Late on the morning of Hong Kong’s first day under Chinese rule, following an evening of fireworks, pageantry and a torrential downpour, the streets were strangely quiet when I wandered out of my MacDonnnell Road apartment looking for a cab. It was like the entire city was sleeping off a hangover after an all-night bender.

I finally found a taxi driven by an elderly, white-haired driver who quickly started up conversation. Where was I from, he wanted to know. “America,” I told him. He smiled broadly, then said; “And you stayed!”

The fact that so many Westerners – gweilos – stayed on after the handover was a type of reassurance to many Hongkongers, a signal that everything would be OK. But underneath there was another sentiment, the unspoken dirty little secret; despite the trepidations about coming under Beijing’s tutelage, deep down, many Hongkongers were perfectly happy to see the backside of British colonial rule.

During 150 years under British control, Hong Kong went from a barren rock to a prosperous trading port to one of the world’s most prosperous and dynamic cities. The British who descended on the rock in an earlier era were adventurous seafarers, missionaries, taipans and more than a few opium dealers. By the time the territory was handed back to China, 20 years ago on July 1, a large chunk of the British expats still hanging on were unemployed or down-on-their-luck dead-enders in Britain who had decided to start over in the exotic East.

Umbrellas now and then: 20 years since Hong Kong handover, have things really changed?

For all the investment bankers, businessmen and consultants, there were thousands of others working in low-end jobs as bartenders and bouncers, construction workers, food deliverers, even shoeshine boys. Many were transients who came backpacking and bumming around Southeast Asia before taking advantage of visa-free entry and the right to work in one of Her Majesty’s last remaining crown colonies.

As the Chinese population grew more affluent, the Brits took some of the jobs the locals did not want to do. The Chinese sometimes derisively referred to them as “white coolies”, or by the insulting acronym FILTH, for Failed In London, Try Hongkong. But these Brits’ low-end employment rarely stopped them from often displaying a superior attitude toward the locals.

Beijing announced well in advance of the handover that it wanted all relics of British colonialism removed before China took control. That 1995 directive was referring primarily to symbols like the Union Jacks and the Queen’s likeness on the postage stamps. But the change applied to the human relics too; from April, 1997, Brits were granted only six-month tourist visas.

Many older Chinese here also remembered a time when discrimination against locals by the European interlopers was entrenched and systematic, and they consider that period deeply humiliating.

As an American, I always found something a little odd and anachronistic about a British colony in Asia in the 1990s, a relic that should have ended decades earlier. And in the months and years immediately after the handover – before Beijing began tightening its stranglehold over the city – I had the feeling that Hong Kong had become a prouder, more confident, more equal place. Bars and restaurants that were once populated mostly with foreigners soon were filling with well-dressed young Chinese.

A few locals took out their anti-British sentiment in extreme ways. The year before the handover, a local artist named Pun Sing Lui was sentenced to 28 days in jail for splashing red paint on a statue of Queen Victoria in Victoria Park and breaking the statue’s nose with a hammer. Pun claimed to be launching a protest action against “dull, colonial culture in Hong Kong”.

Some localists now have taken to thumbing their nose at China’s heavy-handed meddling by waving the British flag at soccer matches, booing the Chinese anthem and chanting “We are Hong Kong! Hong Kong is not China!” in English. Last year, a small Hong Kong-United Kingdom Reunification Campaign garnered a few headlines if only because their proposition – a return to British rule – seemed so quirky. An unscientific online newspaper survey a few years ago showed the vast majority of respondents preferred returning to British rule than staying under China.

But those were just reactions to Beijing’s transgressions in the territory – like the kidnapping by mainland agents of local booksellers, or the National Peoples’ Congress stepping into local judicial cases. In fact, many here never really forgave the British for what they considered the ultimate act of betrayal, and a vivid illustration of colonial racism; the refusal of the British government to grant British citizenship to 3.5 million Hongkongers born here under the British flag.

Hong Kong handover: what we got right (and wrong) in predictions for 2017

“It’s because we’re not white,” Legislative Council member Emily Lau told me back in 1996 for an article in The Washington Post. “Had the Hong Kong people been white, Britain would have still handed Hong Kong back to China, but we would have gotten a better deal.”

I was speaking with Lau to get her reaction after a visit to Hong Kong by then-prime minister John Major. In a speech here, Major said “Britain’s commitment to Hong Kong will continue well beyond the summer of 1997,” and that “Hong Kong will never have to walk alone.

“If, in the future, there were any suggestion of a breach of the Joint Declaration, we would mobilise the international community and pursue every legal or other avenue available to us,” Major announced.

China did breach the Joint Declaration, as the last British governor, Chris Patten, pointed out in a June 28 interview with The Guardian newspaper. But there was no mobilisation of the international community, and no legal action taken. Commercial interests have taken precedent over any commitment to Hong Kong’s freedoms. Hong Kong has had largely to fend for itself.

“That the joint declaration has been breached [by China] I have no doubt, I have no doubt at all,” Patten told The Guardian. “And occasionally it has been so outrageously breached – for example the abductions of the booksellers – that a British minister and the British government have tut-tutted loudly.”

Patten, in that soul searching interview, sounded like the one British official with regrets about the way the empire left its long-time colony. “That we could have done more, I have got no doubt at all,” he said.