In Hong Kong to address the Asia Society this weekend, one of the things that strikes me again is that amid the profusion of colonial buildings in this teeming city, there is one startling absence. There is no museum to Hong Kong’s colonial history.

The story of Hong Kong for 156 years was the story of colonial rule, but it is not commemorated. Tales may be told of the British magnates who passed through here and made their fortunes in this port city. But a museum that reflects the stories of the Hong Kong Chinese, or of the Indians – Hindus, Parsis, Sikhs, Jews – who played such a major role in the building of Hong Kong, is nowhere to be found. Colonialism is the missing piece of the Hong Kong puzzle. In the city’s headlong rush to into the 21st century, it has left its history behind.

The same is true of India’s somewhat longer and more extensive colonial history under British imperial rule. I recently wrote to the Indian government to propose that one of the country’s most renowned heritage buildings, the Victoria Memorial in Calcutta, be converted into a museum that displays the truth of the British Raj – a museum, in other words, to colonial atrocities.

Does Britain owe reparations to former colonies?

This famous monument, built between 1906 and 1921, stands testimony to the glorification of the British Raj in India. It is time, I argued, that it be converted to serve as a reminder of what was done to India by the British, who conquered one of the richest countries in the world (27 per cent of global GDP in 1700) and reduced it, over two centuries of loot and exploitation, to one of the poorest, most diseased and most illiterate countries on Earth by the time they left in 1947.

It is curious that there is, neither in India nor in Britain or Hong Kong, any museum to the colonial experience. London is dotted with museums that reflect its imperial conquests, from the Imperial War Museum to the India collections at the Victoria and Albert Museum and the British Museum. But none says anything about the colonial experience itself, the destruction of India’s textile industry and the depopulation of the great weaving centres of Bengal, the systematic collapse of shipbuilding, or the extinction of India’s fabled “wootz” steel.

Nor is there any memorial to the massacres of the Raj, from Delhi in 1858 to Amritsar in 1919, the deaths of 35 million Indians in famines caused by British policy, or the “divide and rule” policy that culminated in the horrors of Partition in 1947 when the British made their shambolic and tragic Brexit from the subcontinent. The lack of such a museum is striking.

It is equally striking that the only museum to the British presence in India was one that was, in fact, built to commemorate and celebrate the British Empire. Its progenitor, the then Viceroy, Lord Curzon, had declared the need for “a building, stately, spacious, monumental and grand, to which every newcomer in Calcutta will turn, to which all the resident population, European and Native, will flock, where all classes will learn the lessons of history, and see revived before their eyes the marvels of the past.”

Diwali’s Hindu haze and a weird debate on gunpowder

And of course, as with everything else the British built in India, from the Railways to the Viergal Palace (now, as Rashtrapati Bhavan, the residence of the president of India) the Indians paid for it. The “princes and people of India”, the official website of the Victoria Memorial explains, “responded generously to Curzon’s appeal for funds”, and the total cost of construction of the monument, amounting to one crore and five lakhs of rupees in 1906 (then about £10 million), was entirely derived from their “voluntary” subscriptions. Indians paid for their own oppression too in those days, and even for the conquest of far-off peoples. (Indians even largely financed the British participation in the first world war.)

So isn’t it time Indians got their money’s worth? I can think of no better use for it than to take Curzon’s declared purpose and reverse its intent – to make the monument a place where “all classes will learn the lessons of history”, but rather than the “marvels of the past”, it should depict the horrors of the past. The Victoria Memorial should be converted into a national museum to British colonialism – its exactions and cruelties, its loot and expropriation, its atrocities and racism. In Moscow, there are two state-sponsored national memorials to Stalin’s millions of victims, including a Gulag Museum. There is none in Washington to the horrors of slavery, and none in London to the cruelties of colonialism.

Imagine what China and India can do together

The British basked in the Indian sun and yearned for their cold and fog-ridden homeland; they sent the money they had taken off the perspiring brow of the Indian worker to England; and whatever little they did for India, they ensured India paid for it in excess. And at the end of it all, they went home to enjoy their retirements in damp little cottages with Indian names, their alien rest cushioned by generous pensions supplied by Indian taxpayers.

This is all now forgotten, and it must be remembered. The need for a place to house permanent exhibits about what the British did to India is compelling. If it can’t be in London, it can’t not be in India. An enduring reminder is required, for Indian schoolchildren to educate themselves and for British tourists to visit for their own enlightenment. Historical amnesia is convenient to the British – you can do your “A” levels in history in Britain today and not learn a word about colonial history. It’s time that was set right. The same is true of Hong Kong. Not to memorialise your colonial past is to imply that your economy is more important than your history, your land more important than your people. As I say to young Indians: if you don’t know where you have come from, how will you appreciate where you are going? Isn’t it time to ask the same question in Hong Kong?

Shashi Tharoor, Member of the Indian Parliament, is the author of Inglorious Empire: What the British Did to India