'They all went to see their old bedrooms at Government House,' he says. 'Our grandchildren had some difficulty comprehending that we'd lived there.'

You can understand the confusion: this time around, they were in serviced apartments on Robinson Road.

'We had a fantastic time,' says Patten. 'There were so many things for them to do, from eating dim sum to tearing up and down escalators to Ocean Park. It just reminded me of why and how we'd been so happy there. But I don't go back nearly as often as I'm invited just because I think it would be wrong.'

Why wrong?

'Because ...' the 28th, and final, governor pauses for a long moment, sifting his words in advance, '... you shouldn't spend too much time camped on territory that you've left but were once responsible for. It inevitably invites comparisons and the worry that you're trying to interfere in some way. But I adore it. I think Hong Kong is one of my three or four favourite cities and to have had responsibility for it, for five years, was the greatest job and the greatest privilege of my life.'

This is the third interview I've conducted with Patten. It's been like charting the rings of a transplanted tree: the trunk has certainly expanded but the foliage still looks relatively fresh. Rather in the manner of Dr Who, he has a tendency to regenerate himself and pop up in unexpected areas of crisis - a savvy, political Time Lord if ever there was one. (When he was a guest on the BBC radio programme Desert Island Discs, he actually chose Stephen Hawking's A Brief History of Time as the book he'd take with him if he were to be a castaway.)

Our first encounter was in the spring of 1997, in Government House, where Alice slept upstairs in that now-deserted bedroom she showed her two boys at Christmas. Mention to people, here or in London, that it's 15 years since the handover, and they express surprise at the speedy passage of the years; but if you look through the curled newspaper cuttings and the faded faxes of that era, they seem - in every sense - like something beamed in from another century: Passports! Joint Liaison Group! Lu Ping! Passports! Sino-British agreement! Whisky and Soda (the governor's famously aggressive Norfolk terriers)! One-country-two-systems! Passports!

In those febrile days, there was a clock ticking off the minutes up in Beijing; its position, in Tiananmen Square, didn't exactly help to lower the temperature. But a liveried Chinese chauffeur straight out of another of Alice's adventures - he could have been a Tenniel illustration of the Red Queen's footman - still stood by the Daimler and Rolls-Royce at Government House while a Hakka woman tended the garden, squatting imperturbably along the borders. Even then, it was Wonderland, as unreal and perishable as a dream.

Hong Kong is one of my … favourite cities and to have had responsibility for it, for five years, was the greatest job and the greatest privilege of my life

Chris Patten

To be fair, Patten had dispensed with much of the Victorian flummery on his arrival, five years earlier. He'd refused to wear gubernatorial fancy dress, particularly the headgear, which he now likes to describe as 'a topi with a dead pigeon on top'. (Patten's father was a jazz musician who became a publisher of Tin Pan Alley hits, the most notable being She Wears Red Feathers and a Hooly-Hooly Skirt; perhaps that left psychological scars.)

And, clearly, he'd arrived as a politician, not a colonial civil servant. Never mind China's reaction - his CV had excited expectations (subsequently bitterly dashed) among Hong Kong's democrats and suspicions (only too horribly confirmed) among some of the old Sino-hands at the British Foreign Office.

On a tour of Government House that afternoon, 109 days before the handover, he'd pointed to a framed photograph taken outside 10 Downing Street on the morning after the 1992 general election in Britain. In the famous doorway was a triumphant prime minister, John Major, and a crushed Patten, who as chairman of the Conservative Party had just pulled off a spectacular double-whammy: victory for his party but the loss of his own seat, in Bath, by 3,000 votes.

A crowd of mandarins - of the Foreign Office kind - was applauding the duo, none more enthusiastically, his arms aloft, than Percy Cradock, a former ambassador to what was then called Peking and the man who'd helped negotiate the 1984 Joint Declaration. He could hardly have known that the failed member of parliament he was cheering was about to be handed Hong Kong as a consolation prize. By 1997, when Patten was pointing to the photo ('You'll laugh at this'), Cradock utterly detested the governor, whom he accused of having inflicted lasting damage on the colony by incessantly riling China with the D word ('democracy').

'There's a particular English style about being malicious which is rather unattractive,' Patten had murmured. 'I'll tell you this, it's always more tiresome when you can hear the cutlery behind your shoulders than when people are coming at you from the front.' By that standard, Beijing's thundering, full-on fulminations - 'a villain for a thousand antiquities', 'a serpent', 'a triple violator' and (Patten's favourite) 'a whore who opened her legs for President Clinton' - cost him little anguish. Or, as he put it, when asked: 'No, no pangs for Fei Pang ['Fatty Patten', a local nickname].'

On that March day, of course, the handover had effectively already taken place.

'I think we remained in control of the agenda remarkably late,' Patten had said on his verandah, amid the shrieking, squabbling birds in his garden. He was planning to go to his house in France and write ('Not a political memoir about Hong Kong, that would be pretty boring') but beforehand, he was planning to go on a diet. Despite having keeled over with a heart attack a few months after he'd arrived in Hong Kong, and despite the fact that both his parents had died, relatively young, of heart problems, he had a continuing weakness for the restaurants, and egg-tart bakeries, in the territory. 'We've got a list as long as your arm of people who want to give farewell lunches and dinners and before I embark on this jihad - on this culinary jihad - I'm going to spend a week on cabbage soup.'

Five years later, in 2002, Patten was the European Commissioner for External Relations, and the jihad was no longer culinary. By then, that remark - that whole pre-handover interview, in fact - seemed to have come from a simpler planet, where you could set your watch, and your social engagements, by a pre-programmed historical event. The man who sat on the top (i.e. 15th) floor of the Charlemagne Building in Brussels was a greyer, heavier, wearier interviewee: an important besuited bureaucrat in a town stuffed with important besuited bureaucrats.

At one point, his press officer read out his diary. In the previous month, he'd been obliged to troubleshoot in Afghanistan, as well as India and Pakistan (which were, in that spring of 2002, on the brink of nuclear war) and to pontificate about Israel, Georgia, Moscow. Who could successfully patrol such a beat? At first, when he'd talked, he'd sat hunched at his desk and kept his hand over his mouth as if to catch any inadvertent words before they sprang out and caused trouble; he'd only uncoiled as he'd reminisced about Hong Kong and Northern Ireland, where the Patten Report was reforming the police force ('the most intellectually and emotionally demanding assignment I've ever had').

Those two tiny outposts of British colonialism had registered with him more than anywhere else on the vast map in his office. They were populated by real people, leading difficult lives, whom he'd actually met. 'The intimacy of the violence' stunned him about Northern Ireland. 'And I never believed the GDP per capita figures for Hong Kong,' he'd remarked, in that bland office in drizzly Belgium. 'You know, I used to go out every couple of weeks, unannounced, to a housing estate or a settlement. I understood that Hong Kong was a boiler suit, not an Armani suit.'

In London, 10 years later, he's reverted to being a (heftier) version of the man in that Hong Kong garden 15 years ago: droll, relaxed, chatty. Still busy - as well as becoming chairman of the BBC Trust last year, he's been chancellor of Oxford University since 2003, and setting up the interview took months of e-mails - but certainly happier than in Brussels.

Last year, William Waldegrave, a former Conservative minister, said in a BBC interview that the then Labour prime minister, Tony Blair, had simply sent Patten to Brussels because he feared him 'and he fell for it. I don't think there was a proper job to do and it was a bit of a waste'.

'I don't agree with that,' Patten says, mildly. 'It was an important part of my education and I think I made a pretty positive contribution, not least organising the way we distribute aid.' A pause. 'And, anyway - if I hadn't been there, what would I have been doing?'

Is that the psychology behind saying yes to things?

'A bit. My party, don't forget, was in opposition for 13 years. For part of that period, I was the only Conservative with a job.'

Northern Ireland was a classic example of that: the Labour Party made overtures to him about commissioning a report on the police in 1998. Patten had already written his post-Hong Kong book, so he memorably sent back word: 'This great trout has risen to the bait.'

Is that really how he sees himself?

'Yep. I'm just usually tempted by the things that other people say are impossible. The fact that it was so difficult was the biggest part of its attraction and I didn't know if I could do it. But it worked bloody well. In some ways, it's the best thing I've done.'

Although he doesn't say so, it was potentially dangerous for an Englishman, who was also a Catholic, to advise on restructuring the predominantly Protestant Northern Irish police force. His Catholicism is well known. One of the more piquant facts about his time in Hong Kong - during which he kept a crucifix on the wall above his desk in Government House - is that he, the chairman of the Democratic Party, Martin Lee Chu-ming; the then financial secretary, Donald Tsang Yam-kuen; and the then chief secretary, Anson Chan Fang On-sang, were all Catholic. Lee, especially, thought this would mean a greater melding of democratic ideals than turned out to be the case. (There's certainly a piece to be written one day about the political legacy of Catholicism in the final days of the colony's history.)

Of the church he says, thoughtfully, 'I think, on the whole, there's a slightly unhealthy preoccupation with original sin and not enough interest in original virtue,' and adds, almost under his breath, 'I don't mean to criticise church leaders but I thought that when the cardinal in Edinburgh described gay marriage as 'grotesque', it was an awful thing for a church leader to say - an awful thing.'

When Pope Benedict came to Britain in 2010, it was Patten who oversaw his visit. Why?

'The great trout rose again. And it needed to be done.' He says he thought the pope was like 'an elderly German professor, charming', though you feel he doesn't quite enthuse about the pontiff in the way that, say, he describes a project in which he helped raise money to have a granite slab carved with the names of those martyred in Oxford during the English Reformation - both Catholic and Protestant - with no reference to their denominations.

'That was a good thing to do,' he remarks with satisfaction.

He's no meek martyr, however, and certainly there are people out there - and not just in China - who won't be clubbing together to raise a Patten memorial any time soon. (After the interview, I had lunch with a former chairman of the Cambridge University Conservative Association, who asked how Patten was and, on hearing the reply - affable, charming, etc - said, impatiently, 'Yes, yes, but would you trust him?')

Patten didn't, by strict definition, write a memoir after Hong Kong but then he didn't need to: British journalist Jonathan Dimbleby did it for him. The Last Governor veers into near-hagiography as it describes Patten single-handedly battling official duplicity - not all the Daleks, it seems, were north of the border - and the British machinations regarding the need to keep trading with China, that historical constant, are laid unprettily bare. The paperback edition includes single-word summations from Cradock ('Shameful') and the former Conservative grandee Geoffrey Howe ('Lamentable').

Patten's own post-handover book, East and West, has just been the subject of several sessions during the Leveson Inquiry into Rupert Murdoch's media empire, in London. In April, Murdoch told the inquiry, 'I had always taken the view that Mr [sic] Patten was a bad governor of Hong Kong and had raised very false expectations and when I first heard of this book, I said, 'I hope we don't [publish] it' and when I heard that it was about to hit the streets, or very close, I did step in and say, 'Don't do it', which I wish to say now was one more mistake of mine.'

Patten had already told the inquiry that Murdoch, 'having taken the view that publishing a book critical of Chinese leadership wouldn't improve his commercial chances on the mainland', instructed publisher HarperCollins (part of Murdoch's News Corporation) to drop it 'on the grounds it was no good'. In the end, Patten got to keep his HarperCollins advance of GBP50,000 and when East and West was eventually published in the United States, it had a sticker on the front helpfully proclaiming, 'The book that Rupert Murdoch refused to publish' which, as Patten cheerfully told Judge Brian Leveson, 'was worth tens of thousands on the sale of the book'.

'Look, Murdoch has been a great media tycoon,' says Patten, equably, in his BBC office. 'And it's fair to say I think we wouldn't have as many papers in this country as we do had it not been for him. But I think it's disingenuous for him to argue that he doesn't ever lobby or press for his commercial interests.'

The book went down well with Western reviewers ('eloquent', 'searing', 'engaging', 'splendid'), less so with the China Daily ('the portrait of a narrow-minded, retired politician who vents his spleen over those who opposed his perverse actions').

These days, however, all is sunny cordiality between the last governor and the People's Republic of China. About a year into his Brussels job, Patten received a visit from China's foreign minister, Tang Jiaxuan, 'who read very formally from a piece of paper saying that the Chinese leadership now regarded me as a force for concord rather than discord.'

As a result, he travels to the mainland on average once a year and has spoken at the Central Party School.

'I think that the leadership are prepared to put up with my views on civil liberties and democracy because my overwhelming argument is that China's success isn't a threat to the rest of the world - China's failure would be a threat to the rest of the world,' he says. 'And I think they regard that as quite a helpful thing for somebody with my reputation to say.'

IN 1966, as a shiny, new Oxford graduate, Patten turned down a traineeship with the BBC. Instead, he went off to join the presidential campaign of John Lindsay, the Republican mayor of New York. 'And in the States I got injected with this political bug. So it took me some time to get back here ...'

He says his job is 'terrific', though not perfect.

'I was very exercised, very grumpy, about how much senior managers were being paid when I arrived. There are some toxic issues around we have to deal with. It's a great organisation but of course it can get things wrong. It can sometimes be too smug and inward-looking. But there's no broadcaster like it in the world.'

Talking to Patten - with his love of reading and writing, his inconvenient opinions, his itch to be going and doing - you get the impression that he's been a thwarted journalist all along. When asked, at the Leveson Inquiry, what he thought about statutory regulation of the press, he said he was against it 'because we should be able to exercise self-discipline in our plural society which doesn't involve politicians getting involved in determining matters of free speech'. (A good example of this can be heard on the BBC's website, when the chairman gets hammered by Democrat Emily Lau Wai-hing during a recent Radio 4 programme, The Reunion, which relives the handover. 'If we'd done as much as Emily wanted at the time,' he admits on air, 'it would have been morally probably right but it would have produced an even bigger row than there was anyway.')

He still writes frequently, and sometimes unexpectedly, about his travels; you can read about his family Christmas trip to Hong Kong online, for instance, in British Airways' magazine High Life. 'Hong Kong is always a city that cheers you up,' writes the self-described immoderate fan. 'Out of so little, it has created so much.'

He won't be marking the 15th anniversary. 'No - we had a party on the 10th. And we'll have another big one on the 20th, provided I'm spared.' For that 10th anniversary, Alice put together a short compilation of footage from the original night. 'It's the only time I've ever seen any film of those events.' How did he feel, watching it? 'Very emotional. I was more emotional, in some ways, than I was at the time. But I've never watched it again. I don't think I could, really.'