Voters may be biggest poll losers