More than 1,000 police officers started patrols in Shanghai carrying guns on Sunday, Xinhua reported.
The Ministry of Public Security said last month that more armed patrols would be introduced around the country in the wake of the knife attacks at Kunming railway station on March 1.
Twenty-nine people and four assailants were killed during the attacks that officials have blamed on Muslim separatists from Xinjiang. Most police officers on the mainland do not carry firearms.
As China has tight controls on firearms, most violent crime happens with knives, or homemade weapons.
The ministry said security would be stepped up in a number of mainland cities, including Changsha in Hunan province, Urumqi in Xinjiang and Lhasa in Tibet.
Xinhua reported last month that police would increase armed patrols, especially in busy areas like stations, airports, schools, hospitals and tourist attractions, in a drive to crack down on violent crime.
As well as tightened security in Kunming, Urumqi and Lhasa, police in major cities such as Beijing and Shanghai would increase surveillance, the state news agency said.
Police officers around the country are also to receive three months of intensive weapons training.
The ministry said the aim was to improve the skills of officers on the front line, especially those assigned to street patrols or responding to emergencies, particularly in big cities.
The authorities in Guangzhou announced earlier this month that about 3,000 armed police would start patrols in May, the Yangcheng Evening News reported.
The boosted security measures come after the central government had been alarmed by a series of incidents in recent months including the massacre at Kunming railway station last month, where dozens of commuters were killed and more than a hundred others injured when a group of knife-wielding attackers dressed in black went on the rampage through the station, hacking and stabbing people indiscriminately.
The police response was criticised after it emerged that an officer who shot dead the attackers had first to seek official permission before opening fire.
The SWAT team leader sent to the scene said he managed to shoot five of the suspects in 15 seconds, preventing more bloodshed.
After firing two warning shots, the officer shot a masked woman who lunged at him with a knife, before rapidly shooting another four of the attackers, the Legal Daily newspaper reported.
State television said the SWAT team arrived 10 minutes after the attack had started. The entire attack at the railway station lasted for about 25 minutes.