Forgive me, I'm tipsy, but that's only to be expected when one's trying to summon the phantasms I'm chasing.

I am walking along the river bank in Kuala Kangsar, in northern Malaysia's Perak state, looking for hints of Kuala Hantu, the fictitious "ghost town" that served as the backdrop to Anthony Burgess' first novel. Time for a Tiger (1956) is an account of life and alcohol-fuelled multi-ethnic bonding at the tail end of the British occupation of Malaya, and it could be Burgess' less-than-flattering depiction that has saved this pretty colonial town from the ravages of mass tourism.

I start with the obvious: at the gates of the Malay College, a red-roofed, two-storey white building that emerges like a neo-classical molar from a sprawling expanse of cleanly mown English grass. A boarding school for the sons of the Malay elite since 1905, this is where Burgess taught English in the mid-1950s.

The students are all ensconced in their classes, heads bent over textbooks, humming rote-learning mantras. I stroll the campus' empty boulevards, stalking history teacher Victor Crabbe, Time for a Tiger's doubtful protagonist and Burgess' literary alter-ego, to the multi-ethnic Mansor School. Crabbe tries to curb the threat posed by a communist student who runs clandestine night-time indoctrination sessions in the college. Considering the pair of timid boys who pass through the pillars of the building known as Big School and start jumping with a rope on the rugby pitch, it's clear the "red threat" here is now no more than a fictional one.

Burgess described Kuala Kangsar as a spooky and corrupt town, a place where "the dark shrouded the bungalow of the District Officer, the two gaudy cinemas, the drinking shops where the towkays [business owners] snored on their pallets".

Leaving the school grounds I arrive almost immediately at one of those time-warped scars, the Pavilion. Erected in 1920 and once a part of the Iskandar Polo Club, the Pavilion is a cross between a Victorian edifice and a Chinese pagoda, and has stood largely neglected since 1950, when the club was consumed by fire. It glows in the mid-morning sun and points the way to the corner of Jalan Raja Chulan and Raja Idris, and what is supposedly Malaysia's first rubber tree, one grown from the nine seeds introduced in 1877.

As I imagine a swaying police lieutenant Nabby Adams doing a drunken dance beneath the branches, on his way to the novel's infamous watering hole, the Iblis Club, a tourist bus pulls up. A small group of middle-aged Western tourists are ushered down to have a quick peek at the tree. Their shutters click, but do not capture the ghosts of Victor and his displaced, homesick wife, Fenella. I can see them, coming up the road, flanked by the college on one side and the green field in which the Pavilion stands on the other, also heading for the Iblis.

Across the road from the elegant, 19th-century District and Land Office, the Idris, as the club is actually called, sticks out like a sore thumb. Burgess spent many nights here, "socialising" with the colonial well-to-dos, the laborious planters and others of the decadent British social entourage: a taxidermist's take on the empire purposely stuffed for life in the tropics. There's Nabby swaying on a bar stool. The ghosts of Fenella and Victor walk through me, take a seat next to the inebriated lieutenant and order another round of Tiger beers.

I leave those three to it. The ethereal car of Alladad Khan, Victor's Malay driver, whose burning desire for blonde Fenella is thinly disguised, passes as it zooms alongside what Burgess described as "the dirty, drying river" Perak, on his way to pick them all up.

I pass the post office, which dates from the 1930s, its wooden white façade, red tiles and arched windows helping to give an old-world feeling to the town, and the golden-domed clock tower, which resembles a relic from a fantasy Islamic past. The tower was erected in 1939, to commemorate the coronation of King George VI, and today it decorates the roundabout that gives access to the old town.

Down the road from the A-4 Skyhawk fighter plane that is caught mid-flight on a pedestal - a present from the Royal Malaysian Air Force to the Sultan of Perak, in 2004 - rows of Chinese shophouses contrast with the predominantly Malay character of the surrounding countryside. I try the excellent Hainanese barbecued pork pau at Yut Loy, a Chinese coffee house with an original balcony, window frames and bamboo awning that shelters a preserved 1940s shophouse. In the white-tiled interior, customers sit at two rows of round marble tables; despite the pork being served, a group of veiled Malay women sip coffee at the table behind me. This and the Indian Muslim restaurant next door are reminders that modern Kuala Kangsar is as multi-ethnic as Burgess' ghost town ever was.

In the voice of the despicable Nabby, Burgess wrote of colonial Kuala Kangsar as an immoral town, "for the Sultan was in Bangkok with his latest Chinese dance hostess, and the Raja Perempuan at Singapore for the race meetings".

I retrace my steps to where the Kangsar meets the Perak River, walking along the graciously landscaped Cultural Park, past three giant replicas of traditional Sayong pottery; gourd-shaped clay water receptacles. Each bigger than a man, they look as if they are peaceful guardians clad in armour chiselled from gold. The Royal Palace, built in 1903, has been transformed into the Galeri Sultan Azlan Shah, which displays crown jewels and other regal exhibits, and seems well occupied today.

A few minutes later, in front of the Ubudiah mosque, one of Malaysia's most stunning, I realise what has been added since Burgess' times: the weight of Islam. Encased by a collection of finely carved minarets, Ubudiah's spiky golden onion shimmers, placid and majestic under the midday sun. I turn around before entering, and glimpse all of them, standing behind me in single file.

The ghosts of Victor, Nabby, Fenella and Alladad, who's using the opportunity to rub up against the object of his affections, know that their beer-soaked spirits cannot trespass upon this hallowed ground.

They hold still like vampires waiting for an invitation to enter a victim's home - but this is neither the place nor the time for another Tiger.

Getting there: numerous airlines fly from Hong Kong to Penang and Kuala Lumpur, between which Keretapi Tanah Melayu operates a regular train service that calls in at Kuala Kangsar. The town is 125km southeast of Penang and 240km northwest of Kuala Lumpur.