About 250,000 voters risk being struck off the electoral roll if they fail to validate their addresses by June 29.
They have ignored requests from the Registration and Electoral Office despite being given six weeks to reply, the office said in a document for lawmakers' discussion.
The office embarked on 'improving' the voter registration system after November's District Council polls were dogged by accusations of vote-rigging by people giving false addresses, or being found to be registered at the same address as other voters.
Some gave addresses that were non-existent.
With the Legislative Council polls due in September, the Constitutional and Mainland Affairs Bureau, which oversees the office, is keen to avoid a repeat.
The office said it had tried to confirm addresses in several ways, including checking their details against the Housing Authorities' records.
It also worked with the Buildings Department and the Ratings and Valuation Department to check on recently demolished and soon-to-be demolished buildings in an attempt to identify the scale of the use of false addresses.
About 1.7 million voters, about half of all registered voters, were covered in the exercise.
The office then sent 296,000 letters to voters whose addresses were dubious for one reason or another requesting that they confirm the addresses were correct. But only about 40,000 replied by May 25.
Some lawmakers, including the Labour Party's Cyd Ho Sau-lan, called for further action before removing names from the list ahead of September's election.
The office said it had referred cases involving about 1,500 voters to the police and 600 to the Independent Commission Against Corruption for investigation.
At least 53 people have been arrested by the ICAC since it began investigating the claims.
In March seven Hong Kong men living in Shenzhen were given suspended jail terms for using a false address to register to vote in Yuen Long in the district council elections.
They were the first batch of defendants to be convicted and sentenced since the accusations of vote-rigging in November's polls.
The Independent Commission Against Corruption said that by January it had received 1,182 reports of irregularities in the district council elections, of which 730 involved vote-rigging allegations.
The number of registered voters in Hong Kong
- 45pc of them voted in the last Legco election in 2008