A street in Tsim Sha Tsui bears his name, but few people working along it seem to know anything about him. Many of the traditional shops on Hart Avenue have been superseded by noisy sports bars, but at a Sichuan noodle shop, the elderly owner sips tea and considers the street name for a moment.

He looks vaguely uninterested and suggests the street was named after some “Yingwok yan” (“Englishman”). Further down the avenue, a friendly young barman pouring pints of Guinness in the crowded Hair of the Dog guesses that Hart must have been an English colonial official and seems amused to hear that, in fact, he was Irish – like the drinks in his hands – and worked for the Qing government.

THE SERVICE RUN BY “Our Hart”, as he was fondly referred to by his Chinese masters, collected customs revenue, combatted smuggling and piracy, set up pilotage and maritime services, introduced a modern postal service, built a network of more than 60 lighthouses, and collected and published data on all things Chinese, from meteorology to medical reports.

By the turn of the century, his agency employed nearly 18,000 Chinese and 1,500 foreigners and contributed about 80 per cent of all government revenue. Hart acted as a trusted diplomatic go-between for the Zongli Yamen, which handled all foreign issues for the Qing court, and advised and represented its members in sensitive dealings with the often belligerent Western powers. Hart played pivotal roles in defending Qing interests and often found himself torn between being a Victorian citizen of the British empire and a loyal servant of the Qing court in Beijing.

His customs service grew out of the treaty ports system, which itself had grown out of China’s humiliating defeat in the first and second opium wars and allowed foreign trading privileges and extra-territorial rights in key ports.

The service was a pragmatic compromise accepted by a weakened Qing court, desperate for revenue to combat the massive Taiping Rebellion, and commercially aggressive Western powers, keen to have shipping and trade regulated in turbulent times. Although always a department of the Chinese government, the Customs Service was almost entirely staffed by foreigners in its upper echelons and, right up until December 1949, when the last inspector general resigned in Taiwan, that post was held by a Westerner.

The service originated in Shanghai, and Hart joined to extend it to Guangzhou, in 1859, becoming its official chief in November 1863, after the dismissal of the bellicose and confrontational Horatio Nelson Lay.

Hart’s personality is no less compelling than his achievements; he was no jingoistic empire builder typical of the “old China hands” of that era. Hart’s more reticent style, linguistic skills and preference for logical reasoning over histrionic table thumping were a welcome change for his government colleagues in Beijing.

His origins are to be found far from the pomp of the Forbidden City, though. Hart was brought up a strict Wesleyan in Portadown, Ireland, and as a brilliant student and linguist, was awarded a much coveted job with the consular service in Hong Kong after having studied at Queen’s University Belfast.

The young Hart was suitably enthusiastic about this exciting job offer, partly because of the very generous £200 starting salary and partly because he had developed a taste for the fleshpots of Belfast. There, he had contracted a sexually transmitted disease and needed a fresh start and a chance to recalibrate his wayward lifestyle.

Nineteenth-century Hong Kong, complete with its brothels and opium dens, was possibly the least appropriate destination for a young man seeking some sort of moral redemption, but he was quickly dispatched from here via opium clipper to Ningbo and, later, Guangzhou.

His diaries reveal that it was not long before he was once again being tempted by the pleasures of the flesh. “Now some of the China women are very good-looking: you can make one your absolute possession for 50 to 100 dollars and support her at a cost of 2 or 3 dollars per month,” he wrote on October 29, 1854.

What might seem a morally repugnant proposition to us now was standard practice among government men placed in impersonal consuls in remote treaty ports, isolated from family and friends and with no contact with European women. It was hardly a great shock to his contemporaries, then, when, in 1857, he took a beautiful young mistress called Ayaou. The young woman was probably from a respectable family and procured via the services of a local comprador, and she transferred with Hart to Guangzhou in February 1858. It was by all accounts an affectionate and romantic relationship, although a level of discretion was expected during official visits to conform with the hypocritical social code of the time.

Hart clearly remained tempted by what he referred to as “the pleasures of the couch” for most of his adult life: “Temptation to get a concubine is very strong, I must confess: nothing bothers me so much as my liking for women,” (August 6, 1864).

No wonder, then, that Hart’s colourful early experiences and his struggles with his religious conscience inspired American author Lloyd Lofthouse to write a racy, historical novel called My Splendid Concubine (2007), which has sold more than 16,000 copies. Lofthouse’s fictional version of Hart as a swashbuckling but sensitive soul, finely in tune with Chinese culture and language, and frequently falling into bed with a gorgeous young Chinese woman, is compelling.

“My mother was also extremely religious and spiritual. She was an avid reader but if she was reading a book that had a graphic sex scene in it, she would stop reading and throw it in the fire,” says Lofthouse, who has a home in Shanghai with his wife, the artist and writer Anchee Min. “I think, because of my youth in that environment, I identified with Hart’s struggles with his Wesleyan conscience.”

Hart destroyed the journals that documented his relationship with Ayaou, so “I had to invent the details and intimacy of that relationship”, says Lofthouse.

However, we do know that Hart and Ayaou had three children (Anna, Herbert and Arthur) and that he maintained financial responsibility and an interest in his “charges”, having them educated in England.

Hart’s rising status and prospects demanded a “proper wife”, so the relationship with Ayaou was terminated with a huge £3,000 pay-off. He set off on home leave in 1866 to find a suitable bride, settling quickly on Hester Bredon after a rudimentary, seven-day courtship. Years later, though, his youthful indiscretions were to have damaging consequences on his personal life, threatening his reputation and position.

To fully understand this aspect of Hart’s life, it is necessary to travel far from the treaty ports of the Qing era to a picturesque town in Somerset, southwest England, and the home of former Hongkonger Mary Tiffen. She has spent more than six years researching Hart’s relationships with three generations of women in her family for her book, Friends of Sir Robert Hart. “Hart developed a respect for Ayaou and refuses to condemn her even when the scandal threatens his position,” says Tiffen, referring to a legal declaration made in 1905, at the height of his power, when his illegitimate son Herbert suddenly appeared on the scene, creating a distinct whiff of scandal. Hart needed to protect his official heirs from any claims on his assets or title, but still wrote affectionately in the declaration: “Ayaou was a very good little girl and well-behaved, but we were not married …” Tiffen’s Aunt Kathleen was a daughter of Hart’s private secretary, and was born in the inspector general’s Beijing house, becoming a fondly regarded god-daughter of his. Having access to family correspondence, including letters between Hart and Aunt Kathleen, Tiffen, a sprightly and erudite Cambridge University history graduate now in her 80s, seems to have been able to get under Sir Robert’s skin.

“He was a man who needed both sex and female companionship,” says Tiffen, as she dishes up a light lunch of chicken pie.

Tiffen helps determine whether Hart was a man with an unusual degree of what is now termed “emotional intelligence” in the context of a stylised expatriate environment characterised by sedan chairs and tennis parties, or simply a workaholic whose career was threatened by his tendency to be something of a dirty old man.

“I think Hart developed his emotional intelligence as he got older and had his sex drive more under control,” says Tiffen, who speaks warmly about the man, almost as though he were a friend. “The ten-sion between work goals and emotional needs gave rise to intermittent depression, even as a young man.”

He and Hester also had three children (Evelyn, Edgar Bruce and Mabel) but Hart did not see much of his offspring, as Hester returned to Britain in 1876. After another spell together in Beijing, between 1878 and 1882, she and the children returned to London for good.

Tiffen is convinced that Hester must have found out about Hart’s second family, that this would have severely damaged their marriage and that Hart kept his sexual cravings under control after the split for the sake of his career and reputation, despite what he referred to as the “harems in my head”.

“His intimate female friendships, including that with my Aunt Kath, may have been a need for replacement daughters or even replacement sisters,” suggests Tiffen; they were stand-ins, perhaps, for the wife and children thousands of miles away and comfort against the additional isolation that comes from being the boss of a huge organisation.

Hart’s feelings of acute loneliness are apparent in his distinctive, tightly spaced, scrawling script in the 492 letters he sent to Hester, all of which are contained in the Hart Papers, in the special collections library at HKU.

Many of the earlier letters have a light and affectionate tone, which gradually fades, like the Harts’ marriage, until his writing exhibits a rather laboured and formal Victorian style, which is brought to bear on detailed weather reports and updates on repairs to the furniture.

In the library’s modern glass-walled reading room and sitting among young university students, the reader is quickly transported to the biting isolation of 19th-century overseas officialdom.

“I wish, Hesie, you were here with me. I do not like to be alone, it is not good … what a horrid bore it is to be away from you,” he wrote on August 2, 1866. Just five years later, though, the limitations of a marriage that was to be conducted almost entirely by letter began to surface: “It begins to seem an enormously long time since I heard of or from you. Today is the 23rd and your last letter is dated the 8th, a good fortnight ago,” (July 23, 1871).

The loneliness did at least mean Hart was free to devote himself to his work and he is often described by contemporaries as standing at his desk, firing off dispatches to all parts of his massive department, or immersing himself in local culture.

Some have suggested that it was his early relationship with Ayaou that anchored Hart emotionally in China and helped him develop a unique sensitivity to Chinese ways.

Hart expert Robert Bickers, who grew up in Kowloon Tong and is now professor of history at Britain’s University of Bristol and author of The Scramble for China: Foreign Devils in the Qing Empire, 1832-1914, is not impressed by this theory, however.

“It’s not Ayaou who gives him the unique insight into China; she is just his bed companion,” he says, suggesting there were far more prosaic factors that allowed Hart to be accepted and trusted by the Qing court.”Hart was linguistically gifted and bicultural,” says Bickers, who features in a new film about Hart’s life called On China and the World. “He is also an Irishman with a gift for getting along with people. He would often write of chatting at the Zongli Yamen with senior Chinese government officials about nothing in particular.”

And how did Hart deal with the obvious conflict of interest in being a British citizen and a loyal servant of the Chinese court?

“He believed that both China’s and Britain’s interests were the same, which was a convenient conceit,” says Bickers.

The service was always first and foremost an agency of the Qing, and Hart was criticised, particularly by commercial entities in Hong Kong, for having too Chinese a perspective after so many years in Beijing. The likes of Jardine, Matheson & Company and Dent & Company resented not being able to bribe and bully their way into Chinese ports.

“He was a Chinese civil servant with enormous autonomy running a hybrid institution,” says Bickers.

Perhaps, then, Hart was a hybrid person, too; living in China and immersed in its culture and language but remaining a loyal Victorian advocate of free trade. The result is that he is probably better remembered in Beijing and Shanghai than he is in Hong Kong or in his homeland.

His grave at Bisham, in Berkshire, southern England, where he was buried in 1911, was in such a state of neglect that Bickers and some colleagues obtained funding to restore it and held a rededication ceremony there this year. It was attended by those few who knew about Hart, including Tiffen and descendants of the man from both the British and Chinese sides of his family, who met for the first time. It was also attended by Li Yan, a senior Brussels-based official in China’s customs administration, who sent Tiffen a message a few days later praising Hart’s “leading role in China’s early modernisation and integration into the world that will be rightly recognised and appreciated in China”.

Bickers thinks of Hart not as a villain, but as one of the few Westerners who were active in China during its “century of humiliation” but are now re-taking their place in the country’s history books.

But would Hart have been proud of his enormous legacy? Would he have considered the personal price worthwhile?

On this more personal question, perhaps the verdict should be left to Tiffen, whose family was close to Hart for three generations – he maintained close relationships with both Aunt Kathleen’s mother and grandmother. It can be found in the words that conclude the book she painstakingly researched.

“Hart knew his successes and failures could not be measured only by the outward wealth and fame that he had earned,” she writes. “For me, that adds to his stature.”