"Hunt you down"

[Kron3007]

My understanding was that the respondents weren't lying. It was that the pollsters didn't believe there were that many people stupid enough to vote for Trump. So they kept adjusting the "likely voter" numbers.

The phenomenon isn't new to Trump: in any election between a white and a black candidate, for example, the black candidate underperforms in the election compared to the polls. People don't want to appear racist so they just say they'll vote for the black candidate.

But it's also true that "likely voters" can be hard to model when an anti-establishment candidate like Trump runs. There may be a lot of first-time voters who would in any other election be expected to be non-voters. So you need some model of who will actually turn out, even if they haven't voted in the past 20 years, and who they will break for: more for Trump than Clinton, presumably.

But if this is correct, the original model was probably not so far off; it was their "adjustments" that actually made their predictions worse than their original model would have given.  And the whole point of a scientific process is to rely on established evidence to generate results; you don't fudge them by your own whim if they seem off.*

*(Yes, that actually does happen from time to time in science, but it's generally recognized to be a bad thing.)

In science we sometimes remove "outliers" from our data sets, assuming that the unusual data points are artifacts.  Usually this helps improve the accuracy of our analysis, but sometimes the outliers are not artifacts and should not have been removed.  The challenge is determining when removing them is appropriate vs when they are a legitimate part of the results. 

So in this case, things may have been adjusted in an attempt to improve the accuracy using methods that would normally help, but in this case due to unusual circumstances it had the opposite effect.  This dosn't mean the methods are fundamentally flawed, it just means  that the model should be refined based on the new information.  I'm sure the polling agencies are taking a close look at their models as we speak.

[marshwiggle]

But if this is correct, the original model was probably not so far off; it was their "adjustments" that actually made their predictions worse than their original model would have given.  And the whole point of a scientific process is to rely on established evidence to generate results; you don't fudge them by your own whim if they seem off.*

*(Yes, that actually does happen from time to time in science, but it's generally recognized to be a bad thing.)

In science we sometimes remove "outliers" from our data sets, assuming that the unusual data points are artifacts.  Usually this helps improve the accuracy of our analysis, but sometimes the outliers are not artifacts and should not have been removed.  The challenge is determining when removing them is appropriate vs when they are a legitimate part of the results. 

In my labs, I tell students to avoid doing this as much as possible. If there's a point they think is an outlier, they should try to repeat the measurement, and if they're going to ignore the point, they need to be very clear about why they are doing so.

So in a poll, if there seems to be a sampling error, then rather than arbitrarily mess with things they should try a very careful sample focused to verify or correct their earlier results.

The one thing they should absolutely not do is let their own ideology determine what data they think valid.

So in this case, things may have been adjusted in an attempt to improve the accuracy using methods that would normally help, but in this case due to unusual circumstances it had the opposite effect.  This dosn't mean the methods are fundamentally flawed, it just means  that the model should be refined based on the new information.  I'm sure the polling agencies are taking a close look at their models as we speak.

If they lost a lot of business and/or saw their stock prices drop, they might become much more science-driven and less ideology-driven in the future.

[pigou]

There's a pretty extensive science behind polling. Merely random-dialing 1,000 households and reporting the average is nearly guaranteed to do substantially worse than the "adjusted" poll results. So I don't think anyone, and certainly no proper polling agency, will actually report the raw means in their "results."

For one, you have to adjust for the demographics of voters. If your random-dialing poll in a general election has 7% Black respondents, but they make up 12% of actual voters, you're underestimating support for the Democrat. If it has 60% of respondents with a college degree, it's going to overestimate support for the Democrat.

On top of that, asking people who haven't ever voted which candidate they support isn't likely to be informative. Yes, maybe this is the one election they'll go vote in... but probably not. A third of eligible voters don't vote, and non-voters are more likely to be young. So if your sample is nationally representative in terms of age, it's overweighting young voters (and hence will skew left) vs. the actual election.

You can't benchmark a modeling approach with a single election. A model that predicts "the Republican always wins" would have outperformed the most sophisticated polling model, by just assigning 100% to a Trump win and getting it right. But both historically and in all the non-presidential races in 2016, that model would have done much worse.

[mamselle]

Just weighing back in to say that what I was objecting to was not so much a particular choice or words, per se (although I'd appreciate it if people were more careful, indeed).

I was bothered by the underlying assumption that such care is unnecessary because those things "don't really happen," or are very unlikely to exist, and people who request kinder, more considerate speech acts are basing their hoped-for consideration on situations they're lying about, or that have no significance in the wider sense of things.

Discounting the existence or seriousness of such situations does further violence to those of us who have lived those situations. Discounting the effect such triggering events and words can have on individuals disallows the depth of the pain and injury they've caused.

To some degree, civilized, considerate human beings should indeed police their own thoughts and words and deeds--and calling prophetic attention to that need as a present arena in which change can be made to the good may seem like 'nannying' but in fact does what a good nanny does do: raises your consciousness to a more mature level of empathic consideration for others, urging you past socially reinforced cell-phone-autism and mulishly unkind individualism.

Pooh-poohing the idea that the avoidance of flashbacks to such unpleasant events would be helpful to the thriving survivors of violence is just mean and crazy.

Of course people are free to use whatever language they like: I have no interest in controlling others' words. But it is important to create an understandable context for alternatives to be considered as they make their choices  of how to speak and what to say.

My former spouse will probably retire, I think, in the next few years as a respected, tenured prof at a large, well-known R1. I wouldn't really know, or care, or mention it (I deliberately don't follow anything that would tell me about him) but the point is, abusers exist in every setting...even higher ed. 

M.

[marshwiggle]

There's a pretty extensive science behind polling. Merely random-dialing 1,000 households and reporting the average is nearly guaranteed to do substantially worse than the "adjusted" poll results. So I don't think anyone, and certainly no proper polling agency, will actually report the raw means in their "results."

For one, you have to adjust for the demographics of voters. If your random-dialing poll in a general election has 7% Black respondents, but they make up 12% of actual voters, you're underestimating support for the Democrat. If it has 60% of respondents with a college degree, it's going to overestimate support for the Democrat.

On top of that, asking people who haven't ever voted which candidate they support isn't likely to be informative. Yes, maybe this is the one election they'll go vote in... but probably not. A third of eligible voters don't vote, and non-voters are more likely to be young. So if your sample is nationally representative in terms of age, it's overweighting young voters (and hence will skew left) vs. the actual election.

You can't benchmark a modeling approach with a single election. A model that predicts "the Republican always wins" would have outperformed the most sophisticated polling model, by just assigning 100% to a Trump win and getting it right. But both historically and in all the non-presidential races in 2016, that model would have done much worse.

This is exactly the point. Each election you can refine your model after the results are in, but for the the current election, i.e. the one that hasn't happened, you have to stick with the model you have as your best prediction, and wait to revise it until the election is over. 

If, as has been suggested, the polling firms weren't trusting their current model because it predicted more Trump support than they expected, and were trying to adjust it based on their assumptions, rather than after the election, then it's just stupid and unscientific.

At the very least, if you have some very specific reasons you think your model may be unreliable in a specific situation, the honest thing would be to report both "what the model predicts" as well as "what your speculatively revised model predicts". That way if you were right, you'll gain respect for seeing the flaw in the model, and if you're wrong, and the model was right, then it confirms the validity of the model. It's a win-win.

[Kron3007]

But if this is correct, the original model was probably not so far off; it was their "adjustments" that actually made their predictions worse than their original model would have given.  And the whole point of a scientific process is to rely on established evidence to generate results; you don't fudge them by your own whim if they seem off.*

*(Yes, that actually does happen from time to time in science, but it's generally recognized to be a bad thing.)

In science we sometimes remove "outliers" from our data sets, assuming that the unusual data points are artifacts.  Usually this helps improve the accuracy of our analysis, but sometimes the outliers are not artifacts and should not have been removed.  The challenge is determining when removing them is appropriate vs when they are a legitimate part of the results. 

In my labs, I tell students to avoid doing this as much as possible. If there's a point they think is an outlier, they should try to repeat the measurement, and if they're going to ignore the point, they need to be very clear about why they are doing so.

So in a poll, if there seems to be a sampling error, then rather than arbitrarily mess with things they should try a very careful sample focused to verify or correct their earlier results.

The one thing they should absolutely not do is let their own ideology determine what data they think valid.

So in this case, things may have been adjusted in an attempt to improve the accuracy using methods that would normally help, but in this case due to unusual circumstances it had the opposite effect.  This dosn't mean the methods are fundamentally flawed, it just means  that the model should be refined based on the new information.  I'm sure the polling agencies are taking a close look at their models as we speak.

If they lost a lot of business and/or saw their stock prices drop, they might become much more science-driven and less ideology-driven in the future.

There are statistical tests used to remove outliers, so it is not simply a matter of not liking the data point so you remove it to make the data say what you want.  I generally dont like removing outliers very much either, but if you do so, it should be based on the pre-determined statistical analysis to avoid bias unless there was something wrong with the sample in the first place (in our case we would sometimes have a sample that was obviously not treated properly so the data was marked as dubious). 

In the case of polls, I would think/hope the adjustments were made based on standard methods and not someone's personal bias.  Since most of the polls got it wrong, I would assume it was the former since I dont believe all of the polling agencies would choose to modify it based on their gut.