Take a look at history. Spycraft may sound exciting and even romantic. 

In real life, however, it is far more mundane. More secret agents are probably wounded by paper cuts than enemy fire. 

It does not take Francis Bacon to tell us knowledge is power. Man is by nature a social animal and we love to snoop on each other. Ancient Chinese strategist Sun Tzu dedicated an entire chapter of The Art of War to espionage. Knowing your enemy is the first step to overpowering him – this is what all cultures throughout history understand.

WATCH: Beijingers urged to report foreign spies

Yet important as it is, we often fail to see that our fixation on espionage and counter-espionage is often psychological rather than rational. Our fear of uncertainty leads us to assume the worst of others. Gaining knowledge of them is a quick way to domination, while having your secrets exposed generates existential anxiety and insecurity. 

Political leaders tend to spend far more resources than necessary on intelligence (and counter-intelligence). Yet even the most efficiently run spy agency would be hard pressed to justify its bills. In 2010, the US government officially announced for the first time its total tab for intelligence spending: an eye-popping US$80 billion. This included US$53.1 billion on non-military intelligence programmes and US$27 billion on military intelligence.

In comparison, that year the US Education Department’s budget was US$66 billion, The Department of State’s was US$16.38 billion and the Environmental Protection Agency’s was US$10.5 billion. 

Given the colossal bill it paid, Washington had little to show to the public. That year, an amateur 30-year-old terrorist tried to set off a car bomb in the busy streets of Times Square in New York. What could have been a catastrophe was narrowly averted thanks to a combination of luck, the incompetence of the bomber and a vigilant street vendor who spotted smoke coming out of the vehicle and alerted the police. The CIA and the FBI, as it turned out, had no idea what Pakistan-born Faisal Shahzad was up to until the last minute.

WATCH: ‘Soviet chic’ restaurant favoured by KGB spies reopens in Moscow

This paled in comparison with the diplomatic cables leaked by WikiLeaks in the same year, which is truly one of the most serious intelligence breaches in American history. 

As The New York Times reported, US intelligence at the time had little trouble recruiting Chinese informants. Driven by an acute sense of crisis, the new Chinese leadership under  Xi Jinping went into overdrive to clean up the system, resulting in a blanket crackdown on informants and the story of Jerry Lee.

The biggest spying stories in history arguably took place during the second world war and the cold war. Thanks to the collapse of the Soviet Union and the subsequent declassification of wartime documents, some reliable information emerged shedding light on the costs and effectiveness of wartime intelligence work. They did not paint a pretty picture.

Spies and a magic weapon: why are Australia, NZ so suspicious of China?

Many historians of wartime secret services have concluded that spying activities contributed “almost nothing” to the Allied victory in the war. Max Hastings, in his book The Secret War, noted that “perhaps one-thousandth of 1 per cent of material garnered from secret sources by all the belligerents in [the second world war] contributed to changing battlefield outcomes.”

Often the problem is at the top. As many spies have bitterly found, your intelligence can only be as good as your master’s willingness to listen. A primary example is the Soviet Union under Joseph Stalin.

The Soviets had the best intelligence network in the world. Moscow splurged whatever was deemed necessary on buying information. And communism was in vogue among intellectuals, students and freedom fighters around the world before the extent of its true terror was known. Men and women of brains and ideas volunteered to contribute to what they deemed a higher cause. This gave rise to legendary spy rings such as the Red Orchestra and Silvermaster, granting Moscow unrivalled access to intelligence from Berlin to Washington.

One of the most famous Soviet spies was Richard “Ika” Sorge. The child of a German engineer and a Russian mother, Sorge first cut his teeth in Shanghai. The charismatic Soviet spy masqueraded as an American freelance journalist. He fervently studied every aspect of Chinese life and soon became a regular Far East stringer for a prominent Nazi publication. Sorge managed to win the complete trust of his editor and was accepted as a member of the National Socialist German Workers’ Party (the Nazi Party). In 1935, he was sent by the Reich to Tokyo to spy on the Japanese. 

In Tokyo he managed to impress the Japanese so much that he was hired by its prime minister’s office as an expert on China. In 1939, Sorge was sent to Manchuria as an intelligence officer for Japan’s Kwantung Army to spy on the Chinese. He was simultaneously working for three leading spy agencies in the world. Thanks to his intelligence, Moscow gained invaluable insight into Tokyo’s war plan and knew Japan would not attack its eastern front unless Moscow fell under Nazi force. The Red Army swiftly shifted much of its resources in the East to the Western theatre.

Yet for all his spectacular exploitations, Sorge’s career was ultimately a frustrated one. Months before Hitler launched a surprise attack on the Soviets, Sorge learned of the invasion plan. This was independently confirmed by another Soviet spy, Walther Stennes, who at the time acted as Chinese supreme leader Chiang Kai-shek’s adviser. Sorge and Stennes gained so much inside information they could pinpoint the exact period of the planned German attack to between June 20 and 25, 1941. They sent a total of 42 reports to Moscow, detailing the Nazi plan. 

This should have given the Kremlin an incredible advantage and enough time to prepare for a counter-strike. Yet those in Moscow who received and processed the reports were too fearful of offending Stalin, who firmly believed Hitler’s focus was in western Europe. In the end, the Soviets suffered astounding losses in the “surprise” German blitz.

The Soviet Union remained the espionage superpower after the war. In the early days of the cold war, Moscow had far more success than London and Washington in intelligence work. Harry Dexter White, a senior US treasury official and Washington’s chief representative at the landmark Bretton Woods Conference in 1944, was later believed to be a Soviet spy. Ironically, White was also a major architect of two of the most important institutions of the capitalist world – the International Monetary Fund and the World Bank. 

London suffered even worse at the hands of the KGB. Kim Philby, the MI6 chief in charge of countering Soviet espionage, was later found to be a KGB operative. Philby also served as MI6’s go-to guy with the CIA and the FBI. His work greatly compromised the Western intelligence network. Philby defected to Moscow in 1965 and was celebrated as a decorated hero. He was one of the rare few to receive both the Order of the British Empire and the Order of Lenin. 

Similar stories to Sorge’s, if less spectacular, could be found in all capitals throughout the war. Field agents’ reports were routinely ignored by those in the headquarters because they did not fit the dominant political narrative. Most agents were busy just staying alive behind enemy lines – the intelligence they sent home was often banal and of little value. 

One British intelligence officer once drily noted that most of the intelligence he received from Berlin could be gained “by carefully reading local newspapers”. The sentiment was echoed by his counterparts in Berlin. Small wonder that many spies like to disguise themselves as journalists.

Much of the Allied forces’ intelligence success was down not to daredevil agents but brilliant young scientists in the vein of Alan Turing. Through code-breaking technology, the Allied forces managed to break into Nazi Germany’s Enigma and Imperial Japan’s Purple cipher machines. These breakthroughs had far more impact on the outcome of the war than any spy.

Until the 20th century, commanders could only discover their enemy’s movements through individual spies and direct observation. The arrival of wireless communication and radar technology changed the scene forever. Technology, not individual heroism, came to dominate intelligence work.

UNGENTLEMANLY BEHAVIOUR

Like all technological disruptions, the transformation caught many veterans off guard. In 1929, US Secretary of State Henry Stimson was furious when he found that young agent Herbert Yardley had formed a team to intercept and read other countries’ diplomatic telegraphs. Stimson ordered the cipher bureau to be shut and Yardley sacked. “Gentlemen do not read others’ mail,” he said, famously, in his memoir.

Shamefaced and broken, Yardley, in his early 30s, could not find another job. To support his family, he wrote a controversial book, The American Black Chamber, which detailed the internal operation of US intelligence work. The book led to the amendment of the Espionage Act, which Washington used to seize all his remaining materials.

Yardley was later hired by the Chinese Nationalist government to help overhaul its intelligence work. He later wrote another exposé, The Chinese Black Chamber, which was declassified and published in 1983. He was eventually admitted into the Military Intelligence Hall of Fame by the US army for his contribution to the secret service. 

Some 84 years later, another bright young US intelligence officer, Edward Snowden, earned the wrath of the White House. He became an outcast because he could not stand the government reading everyone’s emails. Snowden’s revelation was an eye-opener. He detailed how the NSA and the so-called Five Eyes intelligence alliance (of the US, Canada, Britain, Australia and New Zealand) had been running extensive surveillance programmes around the world. The age of “gentlemen do not read each other’s mail” is a distant dream.

In China, the Communist Party started off as an underground secret organisation. Intelligence and armed forces are core to its operation and form the double helix of its DNA. Run by the meticulous and capable Zhou Enlai, the communist intelligence apparatus scored stunning successes and was critical to its winning of the civil war. Its agents successfully infiltrated almost all levels of the Nationalist government. 

It is said that Mao Zedong often learned details of Chiang Kai-shek’s battle plans before Chiang’s generals got the orders. Li Kenong, one of the three most celebrated Communist agents, was rewarded the rank of General in 1955 – showing the great importance the party attaches to intelligence work.

WATCH: Snowden warns of government spying in first Russia video

Because of its secrecy, the outside world often eyes the Chinese secret service with a mixture of awe and suspicion, imagining it to be an omnipresent hand controlling everything Chinese. 

Internally, Chinese leaders hold quite an opposite view, believing the country’s intelligence work is ill-coordinated and lags behind the US.

Last year, China introduced its first national intelligence law, one of many steps Xi has taken to overhaul the country’s intelligence system and bring it up to date. In the years ahead, Beijing will aggressively restructure its secret service and security apparatus.

From Washington, Moscow and Beijing to Tokyo, governments around the world will continue to squander money on espionage and counter-espionage. It is a great shame for taxpayers – most of these operations are so secretive that they are not accountable to the public nor subject to supervision. They are inherently wasteful and do not always serve the greater good of all. If deployed elsewhere, even a fraction of the resources would solve many problems we are facing today, but it is naive to believe things will be otherwise. It is human nature to be nosy and suspicious; the craving for others’ secrets is in our blood. In God we trust, everything else we spy on. ■

Chow Chung-yan is executive editor of the South China Morning Post, overseeing daily print and digital operations