Mayor Michael Nutter, who confirmed the deaths, said the scene was horrific.
"It is an absolute disastrous mess," he said. "Never seen anything like this in my life."
Train 188, a Northeast Regional, had left Washington, D.C. The front of the train was going into a turn when it shook.
An Associated Press manager, Paul Cheung, was on the train and said he was watching Netflix when "the train started to decelerate, like someone had slammed the brake."
"Then suddenly you could see everything starting to shake," he said. "You could see people's stuff flying over me."
Cheung said another passenger urged him to escape from the back of his car, which he did. He said he saw passengers trying to escape through the windows of cars tipped on their side.
"The front of the train is really mangled," he said. "It's a complete wreck. The whole thing is like a pile of metal."
The cause of the derailment was unknown, but Amtrak said it was investigating. The area where the derailment occurred is known as Frankford Junction and has a big curve.
Police swarming around the crash site, in Port Richmond, a working-class area, told people to get back, away from the train. They pleaded with curious onlookers: "Do NOT go to scene of derailment. Please allow 1st responders room to work."
Roads all around the crash site were blocked off. Waves of firefighters continued toward the train cars, taking people out.
Several injured people, including one man complaining of neck pain, were rolled away on stretchers. Others wobbled while walking away or were put on city buses. An elderly woman was given oxygen. Dozens of people were hospitalized.
Former U.S. Rep. Patrick Murphy was on the train and said he helped people. He tweeted photos of firefighters helping other people in the wreckage.
"Pray for those injured," he said.
Amtrak said the train was carrying 238 passengers and five crew members. It said rail service on the busy Northeast Corridor between New York and Philadelphia was stopped.
The National Transportation Safety Board said it was gathering information about the derailment. It said it was launching an investigative team, which would arrive at site Wednesday morning.
Another Amtrak train crashed on Sunday. That train, bound for New Orleans, struck a flatbed truck at a railway crossing in Amite, killing the truck's driver and injuring two people on the train.
In March, at least 55 people were injured when an Amtrak train collided with a tractor-trailer that was stuck on the tracks in North Carolina.
Port Richmond, the site of Tuesday's crash, is one of five neighborhoods in what's known as Philadelphia's River Wards, dense rowhouse neighbourhoods located off the Delaware River.
