How do we know if the average pollster is unbiased?

There is a discussion going on at fivethirtyeight.com about whether Realclearpolitics is cherry picking the polls going into their averages.

Now, a comparison site is pollster.com, which uses all polls.

There is a problem with any of these models:

How do we know if the average pollster is unbiased?

A key parameter in 538's (or anyone's) election modeling has to be the grand mean across all polls.

Speaking in rough terms, the grand mean is estimated as the average across all pollsters, and the grand mean is our best estimate of who is winning. But we don't have that large a universe of pollsters. It's entirely possible the actual sample of pollsters is biased on average. How would we know if they were or were not?

For example, Daily Kos (DK) (Research 2000) recently started a tracking poll. DK appears to have the strongest pro-Obama lean of all tracking polls at the moment. They may be right, they may be wrong, I can't tell. BUT, suppose that prior to DK starting up, we actually had an unbiased set of currently active pollsters. Then, when DK starts up, suddenly the average pollster will lean towards Obama.

One thought: 538 does have some background data on pollster quality from the primaries. Under the assumption that pollster quality carries over to the general election, then it could be possible to estimate the average bias of a set of pollsters. However, the incentives, (really, the utilities) of the many various pollsters are different in the primaries and the general election. Whether you have a democratic or republican bias, you really don't have much preference between primary candidates, (unless you are employed by a particular candidate). In contrast, in the general election, you will have a preference between the two candidates and thus may be inclined to tilt your polling.

If Sean's/Nate's discussions of the Obama ground game are correct, DK could come closest to estimating the final result correctly. And if public predictions in other areas of policy (oil prices for example) are symptomatic, its entirely possible that all pollsters could end up on the same side of the actual result. This usually doesn't happen in the political sphere, but do remember New Hampshire.


I posted a version of this over at fivethirtyeight.com as well.