“In the end, I could not bite my tongue and asked them ‘how could you have got things so wrong?’”

Just a month before the protests began, Vice-Premier Han Zheng – China’s top man in charge of Hong Kong affairs – told the city’s delegates to the national congress that “the political atmosphere in Hong Kong is changing for the better” and “Hong Kong has set on to the right path of development”.

What happened next must have come as a huge embarrassment for Beijing. In response, it dispatched “a record number of people” to the city to collect information and opinions, sources have told the South China Morning Post.

For Christine Loh Kung-wai, a former environment undersecretary who is now a scholar at Hong Kong University of Science and Technology, there is a sense of déjà vu.

Back in 2003, when half a million Hongkongers marched to oppose the proposed national security law, Beijing was equally shocked and did exactly the same thing. At the time, Loh was running a public policy think tank she had founded after serving eight years in Legco.

“After the protest in 2003, a large number of people were sent to [Hong Kong] to talk to all kinds of people about what was happening,” Loh said. “However, we don’t know how their reports were written and analysed; and also what the central authorities thought [was] important.”

Since then, the central government has established numerous channels – official as well as informal ones – to improve its intelligence gathering in the former British colony. Yet, more than a decade later, it apparently was still unable to take the pulse of Hong Kong.

The labyrinthine information network spanning multiple ministries across the central government – each with its own lines of reporting – is confusing even to insiders.

The central government’s liaison office in the city, for example, has a “research office” tasked with monitoring public opinion and sending Beijing daily briefings of Hong Kong media reports across the political spectrum. The office also regularly meets pro-Beijing figures and groups in the political and business circles, as well as in the grass roots of society.

The Hong Kong and Macau Affairs Office (HKMAO) under the State Council, China’s cabinet, has its own team of “researchers” who periodically visit the city to “catch the wind” and submit reports to the leadership. Other departments – such as the National Development and Reform Commission and the Ministry of Commerce – also send people to Hong Kong from time to time, not to mention the Ministry of State Security and its agents.

How Hong Kong government spurned opportunity to listen to public opinion

Official media, such as news agency Xinhua, run bureau offices in Hong Kong. A major daily assignment for them is to write in-depth reports – or “internal references” as they are called – for leaders in Beijing. These reports are not for outside eyes.

Beyond these official channels, scholars and researchers from various universities and think tanks across China make regular visits to Hong Kong, producing reports, papers and databanks for use by the decision makers in the capital. Even major Chinese state-owned companies in Hong Kong have to compile and deliver intelligence reports through their own channels.

Nor does it end there. The Hong Kong government itself is tasked with keeping the central government informed on the situation in the city, and so are the 200 or so Hong Kong members of the Chinese People’s Political Consultative Conference, not to mention the 36 Hong Kong delegates to the National People’s Congress.

This wide range of channels and the vast scale of information collected do not always lead to a clear and comprehensive picture of public sentiment in Hong Kong. Experts blame the failure on a lack of coherent analysis and coordination among the departments.

“What’s lacking is not the sources of information or the amount of information. In fact, they’re probably too abundant right now,” said Lau Siu-kai, vice-chairman of the Chinese Association of Hong Kong and Macau Studies, a semi-official think tank.

“The difficulty lies in how to synthesise and integrate all the information to present a meaningful, full picture that best represents the reality,” said Lau, who once headed the now-defunct Hong Kong government think tank, the Central Policy Unit.

The fact that so many departments are involved but lack cooperation and coordination can make things more difficult by creating “internal strife”, said Tian Feilong, an associate law professor at Beihang University in Beijing.

“As inside experts, during our research we often find that different departments can be sending different signals to the top decision makers – the Politburo Standing Committee and [Chinese President] Xi Jinping himself – which is likely to have an impact on their assessment of the situation [in Hong Kong],” he said.

According to Tian, central government plans to revamp its Hong Kong-related departments and establish a “unified, effective and authoritative” system had got as far as internal discussions and documents, before foundering due to “political difficulties” – given the redistribution of power it would have entailed.

“The ongoing crisis in Hong Kong exposed many structural flaws and complicated problems [in the present system] … that needed to be addressed and resolved,” Tian said.

One middle-ranking mainland official responsible for writing intelligence reports on Hong Kong admitted the central government channels had misread the situation but, he said, they were hardly alone in their mistake.

“To be frank, the reports by the Hong Kong government were just as bad. Nobody saw this coming. Even you media, none of you exactly predicted this two months ago,” he said.

“Back then, [Beijing] had certainly made a misjudgment – they thought 130,000 protesters was not a big deal,” said Lee Cheuk-yan, a long-time pro-democracy activist and former lawmaker in Hong Kong.

The opposition to the extradition bill quickly morphed into a broader, deep-seated discontent against Lam’s administration and what many see as the steady encroachment by Beijing on Hong Kong’s autonomy.

“Beijing could not have guessed all that pent-up anger and deep-rooted disaffection would explode all at once,” Lee said. “Nor could they have guessed that the public would continue to support the protesters after some of them resorted to more radical actions, such as storming Legco.”

Part of the reason may lie in a pro-establishment bias in the information received by Beijing. According to Tian, information collection and reporting is slanted towards the pro-establishment camp while the voices of pan-democrats and young localists are marginalised.

“They know the situation in Hong Kong and what the public wants. But they have a different mindset for governing Hong Kong – what they want is to control Hong Kong to serve their own interests,” said pro-democracy veteran Albert Ho Chun-yan.

Victor Lee, a 23-year-old university graduate who joined one of the police station sieges, said: “I think they’ve known all along what our biggest demand is – which is universal suffrage. We’ve asked for it for so many years.

“They know what we Hongkongers want, but they won’t give it to us because they worry about the implications it would have on the mainland,” he said.

Two months on, what do Hong Kong protesters really want?

In past weeks Beijing has ramped up its propaganda efforts to discredit the movement and fan nationalist anger on the mainland, accusing protesters of launching a “colour revolution” with the help of Western “black hands”.

Some pro-Beijing scholars admit it is not that Beijing does not know the popular will of the pro-democracy movement, but rather that it has neither the will nor the intention to act upon their demands.

“Take the calls for universal suffrage as an example,” said Lau, the former Hong Kong government adviser. “In Beijing’s eyes, there are still anti-communist and anti-China forces in Hong Kong, who are hoping to seize power to rule Hong Kong via general election. How can it respond to such calls?”

The point was laid bare by a meeting between HKMAO director Zhang Xiaoming and Hong Kong’s pro-establishment figures this month, after at least two rounds of extensive intelligence gathering by Beijing in the city since the crisis began.

Zhang, who had summoned the Hong Kong politicians and business elites to Shenzhen, acknowledged there were “all kinds of public opinion and demands in society at the moment”.

But he insisted the prevailing popular will in Hong Kong right now was to “regain stability and peace, and to restore social order as soon as possible”.

Additional reporting by Echo Xie

Other parts of this series have examined what Hong Kong protesters really want, why protesters view the police as the enemy, and how leaders have spurned the chance to listen to public opinion