David Waters had read about the atrocities American forces committed while fighting in Vietnam. He knew about the three million Vietnamese who had been deformed and disabled by the dioxin contained in defoliants such as Agent Orange. He was aware that hitherto unexploded ordnance had killed and maimed thousands after the war. He didn't need any more data to know that the American involvement in Vietnam had been a terrible mistake.

And yet, after a short visit to the War Remnants Museum in Ho Chi Minh City, the old soldier is overwhelmed.

"When you look at the things we did, it's impossible not to feel guilty," says the 67-year-old former Green Beret, who fought along the coast of southern Vietnam in 1968 and 1969. "We are not really responsible, we were just 20, and we were following orders, we barely knew anything about life, but it's terrible.

"When I visited that museum I thought some Vietnamese would point at me and say, 'Look, he is one of the guys who did all this,'" he says. In the event, all he encountered were smiles.

Waters was one of 17 war veterans and peace activists from the United States who in April took part in a two-week tour of Vietnam organised by Veterans for Peace. He had returned as a tourist last year, but wanted to come back again, to find out more about the victims of war.

"Sometimes the veterans who come back are taken off guard because people are very friendly. And sometimes they break down and weep in gratitude because they're helping them ease their burden of remorse and confusion, after what we did here," says Chuck Searcy, a member of Veterans for Peace who has been living in Hanoi for 20 years. "Veterans are astonished when the Vietnamese welcome us. They say, 'We are brothers', and they mean it."

Searcy, an intelligence analyst during the war, which ended almost 40 years ago, is one of the organisers of the annual Vietnam tour for veterans, which began in 2011. The group travels across the country, visiting victims of Agent Orange and old ordnance, meeting Vietnamese veterans and authorities, and trying to help: every participant is required to donate at least US$1,000.

"It's a small donation," says Searcy. "It's symbolic but it does help a few families and programmes. And it demonstrates that we are trying to do more."

Even though some veterans are devastated when they visit the war museum or encounter children with severe disabilities caused by Agent Orange, Searcy has no doubt that "the experience is always positive and, in every case, with no exception, visiting Vietnam changes the life of a veteran for the better".

In 1992, together with an army buddy, Searcy became one the first veterans who dared to return to 'Nam.

He had been in Saigon from June 1967 to June 1968, and he still shivers when he recalls the dead bodies that littered the burning streets during the Tet offensive. "Terrible, terrible," he mumbles.

Two decades later, he wanted to know what that infernal place looked like in peacetime. When he began planning the trip over dinner with his friend, it seemed like an exciting idea, but as his plane started its approach into Ho Chi Minh City (as Saigon is now known), he suffered a panic attack.

"I saw my friend was having the same experience, he was looking at the window and he was scared. We were both wondering how these people would react to two former American soldiers coming back on a holiday in their country after all that had happened to them. We were sure they were going to hate us. If we could have turned the plane around, we would have done it."

The next thing Searcy remembers is checking into a hotel downtown and being astonished to see how friendly people on the street were to him.

"There was no anger, no bitterness, no animosity. It was all curiosity and friendliness," he says, sipping a coffee in a Ho Chi Minh City cafe.

Buoyed by the experience, Searcy was asked, in 1994, by the Vietnam Veterans of America Foundation to open an office in Hanoi and start a rehabilitation programme for disabled children. Three years later, he started to work for the Vietnam Veterans Memorial Fund, and in 2001 launched his biggest contribution to the rehabilitation of Vietnam: Project Renew, an initiative to help the victims of wartime ordnance in the central province of Quang Tri. After a slow start, the project now has a staff of 106, including five teams that go into the field every day to clear bombs and mines.

"With the help of donations, we assist around 1,000 victims in total. In the last two years, our teams have removed close to 20,000 bombs. For every bomb we remove, we won't know what could have happened; maybe nothing, and maybe two people would have died. You never know," he says.

Since moving to Hanoi, he has got to know other former GIs who have started a new life in Vietnam. One of the most active is Chuck Palazzo, a 61-year-old New Yorker who has been living in the central city of Da Nang for seven years. Diagnosed with post-traumatic stress disorder (PTSD), he quells his nightmares and a persistent feeling of guilt by helping Agent Orange victims in central Vietnam, one of the country's most affected areas. Coming back to Vietnam and forging relationships with former enemies proved to be the best medicine.

Palazzo also felt uncomfortable when he first returned, in 2001, but he was reassured after talking with former soldiers from both the North and South Vietnamese armies.

"They told me not to worry because war is a part of the Vietnamese culture. They don't take it emotionally, and that made me feel at ease. I am around these former enemies all the time, but now this is home. My life is in Da Nang," he says.

While Palazzo and Searcy banish their demons by helping survivors and their descendants, Australian veteran Brian Cleaver is trying to help the dead. Many veterans experience recurring nightmares. Cleaver's are centred on the 25 days of fierce fighting during the Battle of Coral-Balmoral, in May 1968, in Binh Duong province, about 50km north of Saigon. Twenty-six Australians and more than 300 Vietnamese were killed in the battle. Every day since, Cleaver has relived the terrifying gunfire, the stress of being so hugely outnumbered (700 against 3,000 men), the fear felt while protecting a tank with a jammed rifle and the death of three members of his battalion. Diagnosed with PTSD, he had a hard time adjusting to life back in Perth, Western Australia. There were times when he would break into tears for no obvious reason and other times when he wouldn't want to talk to anyone.

He had not, however, given a thought to the 42 Vietcong whose bodies he and his fellow soldiers had hastily buried in a bomb crater after the battle. Only when he returned to Vietnam as a tourist in 2002, as part of his therapy, did Cleaver learn that the corpses were never retrieved.

Since then, he has travelled back to Vietnam 12 times and dug up almost every crater on the field of battle. Finding the remains of these 42 former enemies has become an obsession, the search a way for Cleaver to clear his conscience.

"I never thought they would still be there … [before 2002] I believed they had been exhumed after the war. But no exact location was recorded and that's what is making it difficult," he explains, in a hotel in Ho Chi Minh City during his latest visit to Vietnam, in March.

Going back to the battlefield for the first time was the hardest, but he was determined to confront his trauma.

"The bombs, the fear, the bloodshed were not there. It had now become a rubber plantation. It was surreal," Cleaver says, 12 years after that first return trip.

As soon as the Vietnamese authorities asked for his help, he gathered all the information he could from his comrades-in-arms and joined the digging. He knew the bodies were buried in a large bomb crater near what had been a machine-gun bunker.

His hopes rose in 2009, when the team retrieved the body of a Vietnamese soldier in the area, but they could not determine whether it was one of the corpses they were searching for.

"After that, we dug every bomb crater we found and we didn't find anything," says Cleaver.

Last year, after he was granted a personal excavation permit by the Vietnamese authorities, he redoubled his efforts.

"I was really obsessed. I had responsibility for the complete dig. I planned it, I went through all the bureaucracy and I sourced material, double checking, trying to find something that would fit." But still Cleaver came up empty handed. He returned to Australia swearing the search was over for him.

Nevertheless, in March, he tried one more time, after two more craters had been discovered.

"This year, I dug the last two craters and there was no stone left unturned," he says.

In all, more than 30 craters have been excavated, and still there has been no sign of the 42 bodies. With the help of the Vietnamese authorities Cleaver has tracked down 33 of the families of these men, but he hasn't dared to contact them directly, for fear of their reaction. He believes he has done all he can do and his conscience is clear. He'll only grab his shovel again if there are significant new clues, he says. Sometimes, he allows himself to be influenced by Vietnamese superstition.

"Maybe," ventures Cleaver, "they just don't want to be found. They fought together, died together, were buried together and they want to stay together."

For Lawrence Johnson, an independent filmmaker from Portland, Oregon, it's his work that helps to pacify the ghosts that have been haunting him since he left Saigon in December 1972.

As a member of the Command Military Touring Show, Johnson's war was a relatively stress-free one, his duty being to entertain the troops. He had plenty of time to enjoy the nightlife and spend long evenings in the cafe at Saigon's Hotel Continental, where he met Candy. Her real name was Truong My Linh, but he called her by her nickname during the stormy and passionate eight months they spent together. Then 22 years old, he fell so deeply in love that, before finishing his tour of duty, he wrote a long letter to his parents announcing his plans to marry her. But red tape prevented Candy from travelling with him to the US and they split up.

Before he left, though, she told him she was expecting his child, but "she didn't look pregnant", he says. "I put it out of my mind right away."

In 2012, after two failed marriages, the documentary maker decided to go back to Saigon and look for Candy.

"I think," he says, "a lot of GIs are entering a time in their life when they see the end is near and they want to reflect on their past and see what is unresolved.

"I felt as responsible as the man who pulled a trigger. It was difficult to go back home and lead a normal life. The experience still chases me."

Johnson decided to make Ghost Money, a documentary about his search and his confrontation with a past depicted through Super 8 footage he shot 40 years ago. The title is a reference to the fake money many Asians burn to make peace with the dead. Candy claimed she was haunted by an angry ghost and, sometimes, he surprises himself by believing that the same spirit has haunted him, too.

The first thing Johnson did when he arrived in Ho Chi Minh City was to place a black and white photo of Candy in the main newspapers. The woman he was looking for would be in her 60s and may have looked very different to the glamorous lady in the old photograph, but that was all he had. And it looked as though it was enough.

A woman got in touch and assured him she knew a lady who used to look like the one in the picture and whose name was Linh, but she was now in the US. She also said Linh had had a son around 1973, soon after Johnson left Vietnam. Enthused by the idea of finding Candy and being now almost certain that he did have a lost descendant, Johnson spent a year looking for the pair throughout the US and Vietnam. But last December, the woman who had initially contacted him found her woman and realised it was the wrong Linh.

"I was really looking forward to meeting her and my child and I was crushed when I realised I had been following the wrong track."

Johnson has been transforming Ghost Money into a reflection on the aftermath of the war and the lives of the Amerasian children left in Vietnam by GI fathers. He thinks one of these children, discriminated against in many parts of Vietnam for their appearance, may be his own. He is still looking for Candy and for a 40-year-old who may or may not exist. He wants to meet them and say he is sorry.

In short, that's why veterans are coming back to Vietnam; because they're sorry. They want to chase away the guilt and remorse from a past they have never managed to forget.

When the remorse threatens to overwhelm him, Palazzo remembers a piece of advice he was given by an old Vietnamese woman who had every right not to offer it.

Last year, he accompanied the Vietnam veterans' tour on a visit to My Lai, a hamlet in central Vietnam ravaged by a US infantry company in 1968. The GIs murdered at least 347 people and raped women and young girls. Palazzo knew he wasn't responsible for the cruelty of other GIs, but as he listened to the experiences of the survivors, he felt ashamed.

"It's one of those places where you can feel the evil, I don't know how to describe it," he says.

The old woman, who had survived the massacre after being buried with her eight dead brothers and sisters, approached him and uttered a sentence he now carries with him always: "We forgave you years ago; now you need to forgive yourselves."