DEV: Do not skip pages when loading polls (PR #13649)

In some conditions, pages were skipped. This was implemented in the past in f490a8d, but then reverted in 04ec543, because sometimes it was stuck reloading the first page.

The code that loads more results was simplified and a lot of duplicate code was removed. The logic to remove users who changed their vote was also introduced again, but just for the regular polls.