Polling as both art and science

[Baruch Menachem]

An interesting chart from the folks at Pew research

And an interesting and very short article from Scientific American on the danger of misleading poling.

Few seemed to realize that the polls were far from immaculate. In fact, the episode was a good example of what I call disestimation: the act of taking fuzzy numbers way too seriously.

At first, it might seem like a cut-and-dried story: out of 32 quiz questions, atheists and agnostics, on average, got 20.9 correct, higher than any other group and higher than the overall average of 16.0 questions right. But because Pew managed to reach very few atheists and agnostics—only 212 people out of the 3,412 included in the survey—the 20.9 number masks a tremendous amount of imprecision. Small samples don’t give reliable numbers, and if you pre sent the poll results using a standard graphical technique to represent uncertainty (below), you can see that the distinction between atheists/agnostics and Jews and Mormons evaporates.

The story gets even fuzzier because Pew left out one category altogether: those who believe “nothing in particular,” many of whom had specifically said they didn’t believe in God. Interestingly, this group scored worse than the typical American on the religion quiz. Had they been lumped together with atheists and agnostics, the group would have fared a little worse, on average, than evangelical Protestants.

It is Noteworthy that the evangelicals scored better than mainline, and the Mormons did better yet. And if you assume, as I do, that some of those questions might have been included just to make sure these two groups got them wrong, then their numbers might have jumped off the chart for the rest of the test.

And while Hispanic catholic might have the bottom of the chart, that might not be an artifact of religion, but an artifact of immigrants historically suffer, lack of higher education.

Notice Black protestant has the same issue, education for blacks in this country is an embarrassment, with the lousy schools they are stuck with.

And buried in the back of the Pew report is the reality that if you match education attainment with the answers, the religious difference in knowledge shrinks dramatically.

Also not included in the numbers are Buddhist and Shinto. Both religions emphasizing education, as does Judaism. If they had included that, it might have been even more interesting.

So, the poll had two groups of atheist, separated I don't know how. And the large, more accurate sample of that group was the worst scoring among the white population.

Anyway,numbers don't lie. People often do.

 

[Baruch Menachem]

I am going to be a bad boy here.

When this poll was originally posted, it got zillions of looks and posts.

I really hope we can re examine this with better information

 

[Quantum Windbag]

That would require everyone who tried to use this poll to mock religion thinking about their own religious bias.

 

[Madeline]

Did you wish to discuss scientifically sound polling or bias among some pollsters against religious folk, Baruch?

The polls we can most easily measure as to validity are those that allege to predict what a group will do in short order, like a voter opinion poll.

I have a girlfriend who audits the claims handling behavior of insurance companies, which as I am sure you know is heavily regulated. The DOI who hired her will have a standard for randomness, samples size, etc. in her programming and she will run that on the company's actual claims files, and pull the ones selected by that program for actual review. So far as I know, no matter how negative the audit results may be, no insurer has ever litigated the sample size or random files generator -- everyone seems to accept this as valid. (I believe the sample size is 2%, to be adjusted up or down based on a variety of factors.)

Any poll that passively accepts responses is flawed from the get-go because randomness cannot be achieved in this manner. For example, merely hanging up a sign that says "your opinion is wanted" IRL or on the 'net and awaiting whatever responses may be received is "passive" and inherently unreliable. Any poll which draws on a sample size that is too small is also defective. I don't know how many Americans view themselves as atheists and agnostics, but I doubt 212 is a statistically valid sample.

Valid polling data also requires precision in designing questions, so that nothing is vague or prompts a certain reply. It requires a neutral but professional pollster, and a time/place/method that eliminates bias. (E.g., telephone polls are likely to become obsolete in future, as fewer and fewer people have landlines; a poll on a fact drawn from international affairs would likely be invalid if it did not screen out minors, etc.)

My zoo did up an "exit survey" of patrons and administered it live on a single day over the summer. This means the respondents were people willing to be delayed a bit -- likely not the parents of young children. The sample size was 400 -- but 1.5 Million people live in the Cuyahoga County area. Ergo, the sample was far too small to be reliable. The poll made no effort to reach zoo patrons who did not attend that day, or who chose not to participate, nor was there any effort to solicit public opinion in any other manner. I suspect, though I don't yet know, that the poll questions were bias-laden. (This all has to do with banning outdoor smoking on zoo grounds).

The one other thing that bothers me about these polls on the relation between religious belief and brains is, the average person likely has no one fixed belief set all throughout his life. I'd expect religious belief to have peaks around the time most people start their families and again in old age.....if no attempt is made to correct for this meandering, the same person, with presumably the same brains, is bumping the results (or lowering them) for the group he identifies with depending on when during his life he is polled.

Lastly, any poll, no matter how well-designed, is just a "best guess" as to what a population thinks -- guesses can be wrong. And it is zero data about the thoughts or feelings of any particular human....mayhaps "most" Hispanic Catholics are stupid/badly educated/whatever, but that hardly means ALL of them are.

 

[editec]

I have absolute confidnece that scientifically designed polls can be very accurate.

It's the POLLSTERS themselves in whom I have no confidence.

 

[JBeukema]

The story gets even fuzzier because Pew left out one category altogether: those who believe “nothing in particular,” [those are called agnostics] many of whom had specifically said they didn’t believe in God [then they are, by definition, atheists]. .

education for blacks in this country is an embarrassment, with the lousy schools they are stuck with.

Bullshit. With racial quotas in place and the amount of money we throw at blacks who want to go to better schools, blacks need to stop blaming the white man for their own failures. Even blacks know blacks don't want to succeed. That's why successful blacks move away towards white neighborhoods instead of staying behind in the drug-filled culture of violence and unwed mothers that is found in largely black neighborhoods around the country

Also not included in the numbers are Buddhist and Shinto. Both religions emphasizing education, as does Judaism. If they had included that, it might have been even more interesting.

True Buddhism is really more of a philosophy than a religion

 

[Baruch Menachem]

The really problematic part of this was they separated out one group that was homogeneous for belief and separated out the lower performing section of the sample in order to bias the results.

What is interesting is it didn't really show the performance of religious groups in science tests, but it showed the level of education among the groups.