Polls 'could have been held earlier'