In September 2013, the Metropolitan Museum of Art, in New York, held an exhibition of paintings by a man who liked to call himself Count Balthazar Klossowski de Rola but was better known as Balthus. It was the first overview of his work in an American museum for more than three decades. The show's title was Cats and Girls: Paintings and Provocations, a warning allusion to his artistic predilections, although anyone who'd heard of Balthus knew that what he liked to paint were felines (often sinister) and females (young, and in a state of undress).

The work in the Metropolitan show was mostly produced between the 1930s and 50s but it didn't include Balthus' most notorious painting, The Guitar Lesson, which he painted in 1934. As recently as 2001, The New York Times refused to reproduce it - and you won't be seeing it in Post Magazine either - but the internet has now made it widely available. It features a woman whose right breast is exposed and who looks as if she's about to play, or play with, the naked private parts of the child splayed across her lap whose hair she's viciously pulling.

At the same time as the 2013 Met exhibition, the Gagosian Gallery in New York held its own Balthus show. The Polish-French artist had died in Switzerland in 2001, a couple of weeks before his 93rd birthday, and the Gagosian works were from the last decade of his life. They weren't paintings, however. It turned out, although few people in the art world had known it, that he'd taken nearly 2,000 Polaroids of a girl called Anna Wahli, the youngest daughter of his doctor. They were studies for his paintings, never intended to go on public view: old age had stiffened his hands and he found it difficult to do preparatory sketches.

Every Wednesday afternoon in the 1990s, therefore, from the age of eight until the age of 16, Anna posed for him, usually semi-naked. Afterwards, the pair would settle down to watch American television soap opera The Bold and the Beautiful, which is the sort of shrieking metaphor a director might hesitate to put in the film that is surely waiting to be made about Balthus' extraordinary life.

Anna grew up to become (further cinematic alert) a psychotherapist. She wrote a short piece for the coffee-table book, titled Balthus - The Last Studies, that was published to mark the Gagosian exhibition. In it, she says that she'd been chosen by Balthus because he liked the sound of her humming Mozart, that he was such a "klutz" with the camera she'd had to show him how to operate it and that he made endless, tedious adjustments to her position. "It took such a long time to change what seemed to be a minute detail and, from my point of view, all the photographs looked alike."

You can see her point. Some of those Polaroids are now on view at the Gagosian Gallery here in Hong Kong, and it's hard to distinguish one from another. As they also tend to be murkily lit and, of course, small, they make a peering voyeur out of every visitor. Presumably, this is one of the reasons Museum Folkwang, in Essen, Germany, cancelled a planned exhibition of the Polaroids last year. (It stated that the show "could lead to unwanted legal consequences". German newspaper Die Zeit had already referred to the photos as "documents of paedophile greed".)

Within their glossy gloom, the prepubescent, semi-nude Anna can be discerned: one leg propped up, one leg dangling, arm flung behind her head. It's a classic Balthus pose. He was nothing if not consistent. He was painting it in 1938 (in Thérèse Dreaming, featuring Thérèse Blanchard, aged 12), in 1941 ( The Salon, Georgette Coslin, aged 13) and in 1944 ( The Golden Days, Odile Bugnon, aged 14) and that same languid posture is centre stage in his last work, Jeune Fille à la Mandoline, which is also in the Hong Kong show. He obsessively repainted it, using the Anna Polaroids, until a few months before his death. For Balthus, it was always about the jeune fille: the young girl.

"His aim was not to finish the painting," explains a tiny woman in a kimono, giving the Hong Kong press a guided tour at the preview. "Right to the end, he changed the arms, the fingers, the hair … He could spend two years on a painting, take it all off and start again. He was happy to do that."

Speaking is the Countess Setsuko Klossowska de Rola, the second wife of Balthus. They met in Tokyo in 1962, when she was 20, married five years later and, ever since, she's been the keeper of his flame in occasionally unexpected ways. She was the one who approached Wahli about making public the Polaroids. She looks up at the big, final canvas of the naked girl, finally freed from the artist's ministrations, and says, "The dream he wanted to do is concluded."

HE WASN'T REALLY A count. He was born in Paris as Balthazar Klossowski, son of Erich Klossowski, an art historian; the "de Rola", the title and the claim to Polish nobility came later. But you could easily believe that his second wife is an aristocrat. The Countess, a delicate bird perched in Gagosian's back office, incongruously sipping a Starbucks latte, is a woman of unusual grace. On her lap are spread the anodyne questions I've submitted in advance to the gallery. No one usually pays attention to this hoop-jumping, pre-interview ritual but the Countess has worked her way through the list, penning notes and comments alongside.

"What is your first memory?" she reads aloud. "I love this question. Please write what I say in beautiful English, sometimes I mix in French … I was sent to my grandmother's house in the country as a child because of the war, and in the kitchen, water came directly from the source through a bamboo tube - so beautiful to see, look!"

With her hands, the Countess conjures up a long-ago scene out of the air: the bamboo, the cascading water, the stone trough directing the stream to the garden. She glances back at the list.

"What was your childhood like?" She says, "I returned from the country. On the day of my arrival back in Tokyo, I saw a hydrangea - it was a rainy day, such blue in the garden - and I said, 'What a beautiful flower!' and everyone laughed because now I had a country accent, not a Tokyo accent."

Did she study art? The Countess looks a little taken aback (this isn't question three) but at the suggestion that we just chat and see where it takes us, she puts the list aside and neither of us refers to it again. And a strange thing happens: over the course of what becomes two interviews, the Countess, who is 73, reveals herself to be not exactly a jeune fille but certainly sprightly at heart. At times, she displays such joyousness, such sweet passion, such exultation at life's possibilities, that you wonder if a source within herself had been dammed for many years.

She didn't study art. She was a French student at Tokyo's Sophia University - "a Jesuit university" she murmurs in passing and when asked if she's Catholic, she replies, "I became Catholic on the death of my husband." Then she adds, with a little laugh, "Bono is my godfather." (The frontman for Irish rock group U2, without whom no public event nowadays is truly complete, sang at Balthus' funeral and has expounded a theory that he shares a form of Tourette's syndrome with the artist. "In Balthus' era, the only subject you couldn't approach with any curiosity was puberty," he told the French journalist Michka Assayas. "You weren't allowed to go there, so he had to go there. For me and rock 'n' roll, it was spirituality. You just can't go there, so I had to go there …")

In 1962, while she was a student, a journalist told Setsuko that a man called the Count de Rola was being sent to Japan by André Malraux, then France's first minister of cultural affairs, to choose traditional Japanese artwork for an exhibition in Paris. The journalist invited her to accompany the official visit.

"In that period, there was no Google, no Facebook, no blah blah, nothing. I was satisfied that this man was André Malraux's friend - Malraux, my God! Such a fantastic writer."

They met, they spoke, they quarrelled.

"I'm a typical after-war student. Democracy? Good! Everybody can study? Fantastic! But Balthus was against the obligation of education. Odd, no? Balthus thought artists have to practise their métier and if you don't do it since childhood, it's much too late."

Always the emphasis on childhood, then … "Absolutely."

Balthus told her he was 50. "But it was not true, he was 54."

As he'd been born on February 29, even admitting to 50 was unusual: he tended to divide his age by four. The German poet Rainer Maria Rilke, who had an affair with Balthus' mother, told him that having a leap-year birthday meant he'd slipped through a crack in time, into a "kingdom independent of all the changes we undergo". The lesson from this, obviously, is to be careful what you tell a child; no wonder Balthus was obsessed with mapping the unchartable erotic territory that exists between adolescence and adulthood. (If you're British and literary-minded, the association is reinforced by the fact that Penguin used one of Balthus' paintings of Thérèse on the 1990s paperback cover of Nabokov's Lolita.)

It was Rilke who had encouraged the publication of, and wrote the preface for, a book called Mitsou, which consisted of 40 drawings by the 11-year-old (or almost three-year-old) Balthazar, and told the tale of his beloved lost cat, the first of many artistic felines. The Gagosian show includes a famous 1935 self-portrait called Le Roi des Chats, or The King of Cats (that particular aristocratic title comes from European folklore), but a rather more curious work in the exhibition is an ink drawing dated 1928-29 titled Étude pour "La Place de l'Odéon". It's a sketch of a mother and child; the mother's head is, unmistakably, that of a cat.

And also in the show is a 1932 portrait of his first wife, Antoinette de Watteville, with whom Balthus had two sons, Thaddeus and Stanislas, and to whom he was still married when he met the young Setsuko Ideta in Tokyo. He was soon delaying his departure from Japan.

"We talked and talked," she says. "Balthus knew so deeply about Japan, much more than me. He had a longing for Oriental culture. [French author André] Gide, who was a family friend, had taken him to see Japanese Noh theatre. [Balthus] adores all things Japanese. It's he who made me put on the kimono - yes! It was his artistic view. He said the Japanese woman is made to wear a kimono."

In other words, he created a living artwork out of her: a character for his own Noh drama.

"Balthus dreamt of L'Extreme Orient from childhood," the Countess says, and adds, apparently unconscious of the echo, "When you want to make a photo of something, it comes into focus."

If it was, indeed, a form of extreme Orientalism, she went along with it.

"At that time, there were still arranged marriages - can you imagine, the families go to the restaurant, talk, then one has to decide? I wanted to escape. I told my mother, 'I will really give my life to the most difficult, most dramatic, most impossible love'. I was very into Tolstoy, Stendhal, reading them in translation, and a book, Daddy - " She walks her fingers through the air. Daddy-Long-Legs? She laughs. "Yes!"

That American girls' classic, written in 1912, takes the form of letters from a young student to her older, wealthy benefactor. Did she write to Balthus when he left Japan?

"I never wrote, no. It's very strange. I liked to leave him alone to see everyone he wanted in the outside world, even the girlfriends. But when I came" - and she raps the table smartly - "he should be there."

At the time, Balthus was director of the French Academy in Rome, based at the 16th-century Villa Medici. She went to Italy.

"I should say I had five very, very difficult years until my wedding. The first wife agreed to the divorce. I was very thankful to her. But there was another girl attached to him, a terrible complication." She lowers her voice. "And I had an adventure with another man. He wanted to be with me. But the next day, I receive the proposition of marriage from Balthus."

So there was a letter. "There are two letters. I think very often of that Italian person. He died also." A lengthy silence. "It's a volcanic passion. He believed in me. And brutally I said: 'I will marry Balthus'."

Why? "I know that with Balthus there is something. I give my life to him."

I wondered how she'd initially adapted to the rarefied atmosphere in which she'd found herself. Balthus' dealer was Pierre Matisse, son of Henri; Balthus' brother, also Pierre, had been Gide's secretary and was such a fan of the Marquis de Sade he published a new edition of The 120 Days of Sodom; Balthus' son Stanislas - following family tradition, he styles himself Prince Stash - was arrested on drugs charges in 1967, in the company of the Rolling Stones guitarist Brian Jones, and has been a preening gift to gossip columnists. (Stash's wife, Tan Chan, was present at the Gagosian press preview; she was born in Hong Kong but grew up in Australia.)

"Oh, Stash's world is not at all Balthus' world," says the Countess.

The world, however, does tend to stand in sweeping judgment. Is she offended by what is said of her husband?

"It's other people's opinion and they have a right to that stupid opinion," she replies, evenly. (If she's upset, it's masked: a personal No Drama.) Was she ever worried? "About what? Paedophilia? If it is not true, then what's wrong? Balthus totally has another vision."

But why produce the Polaroids now and reopen the debate?

"You mean, the German museum? We wanted to show the work-in-progress of Balthus. I am of the opinion of Socrates … It's written in my room. I'll get it."

We're now in the Clipper Lounge, at the Mandarin Oriental, where the Countess has been staying. While she goes off on her Socratic search, one of the waiters comes over.

"She is very sincere, very nice," he says, unprompted. "She lives in Switzerland." In what's supposed to be the biggest house in that country, I say, and the waiter, interested, asks if it's a hotel.

In fact, the 18th-century Grand Chalet in Rossinière had been a hotel (French poet and novelist Victor Hugo was a former guest) until Balthus bought it in 1977. Now it's where the Balthus Foundation, which aims to promote his legacy, is based and where his widow still lives.

The couple had two children, Fumio who died aged two (of a rare genetic disorder called Tay-Sachs disease), and Harumi, who was born in 1973 and is now a high-end - Chopard, Boucheron - jewellery designer.

A short film about Harumi, called The Beauty of Spring, was shot at Rossinière. Photographer Stephane Sednaoui captures her, heavily pregnant, in her father's studio. As Sednaoui explains, in the voiceover: "It's Harumi saying, 'See? I'm becoming a mother whereas Balthus' characters remain child-women.'" Harumi - who has her father's nose and her mother's smile - laughs all the way through the shoot, massaging her belly, but a curious sense lingers that it's an exorcism.

The Countess returns. She reads out a quote that Eros is truth and divine nature, then says, firmly: "And Balthus' paintings to me are really about that."

In September, an exhibition of her own paintings and ceramics will take place in Tokyo. Earlier I'd asked her if she knew of the Pygmalion creation-myth and she'd understood. She'd said, "I am a painting by Balthus"; now it's her turn to mould something from clay.

"I didn't have an affair during my marriage, I was fidèle," she offers (this certainly wasn't on the original list of questions but she seems skittishly given to confidences - it's part of her considerable charm). "Afterwards, I had several." One of them was with a man who didn't like her wearing a kimono. It has been her fate to please men.

"Now I'm free! And happy because I have my life. I'm very happy! Plenty of projects, lots of inspiration. Everyone thinks I should stay at home more but I'm like the young girl - always doing the very amazing thing."

"Balthus" continues at Gagosian Gallery, 7/F, Pedder Building, 12 Pedder Street, Central until August 15.