Most bogus study ever?

From a NY Times article (thanks to a reader for emailing me the link):

In the study made public on Thursday, Dr. Friedman and his colleagues compiled a brief test, drawing 20 questions from the verbal sections of the Graduate Record Exam, and administering it four times to about 120 white and black test-takers during last year’s presidential campaign.

In total, 472 Americans — 84 blacks and 388 whites — took the exam. Both white and black test-takers ranged in age from 18 to 63, and their educational attainment ranged from high school dropout to Ph.D.

On the initial test last summer, whites on average correctly answered about 12 of 20 questions, compared with about 8.5 correct answers for blacks, Dr. Friedman said. But on the tests administered immediately after Mr. Obama’s nomination acceptance speech, and just after his election victory, black performance improved, rendering the white-black gap “statistically nonsignificant,” he said.

So approximately 21 different black people took the test each time. Obviously the researchers got lucky in the sense that the random error caused by such a small sample size favored the results they were hoping to find. Why not at least test the same group of people each time?

You can obviously publish crap in a journal if it’s politically correct crap.

This reminds me of a NY Times article from a few days ago, in which the journalist hopes that Obama will lift women in science. That’s not going to happen either. And the connection doesn’t even make any sense. Another man is elected to the nation’s highest political office, but because he’s half black, more women will flock to science?

Comments

[The Undiscovered Jew]

Blacks have the largest penis size of any ethnic group.

So, obviously, the election of the Groove Messiah means white, Asian and Hispanic men will grow larger shlongs.

[RottonPickle]

The connection does make sense if you believe he is a Messiah whom is here to rescue us all.

[Utilitarian]

This certainly is a steaming...

Let me count the ways:

1. Note that they give the scores pre-Obama, but not post-Obama. So there was still a gap, but perhaps only at a 90% significance level rather than a 95% level.
2. Sample size of students is tiny, as you noted.
3. The number of questions is very small.
4. Self-selection effects (it's an internet study, not a representative sample).
5. Different times of year may catch students in or out of school, different people who may have heard about the study (including from the experimenters, who may have sent flyers or emails to different places), etc.

[Peter]

Well heck, what with all the expectations foisted on Obama, people will be disappointed if he doesn't raise the dead.

[Ming]

I wonder why the researchers went to such great lengths to test the students. Wouldn't the "study" be similar if the researchers had asked College Board for SAT scores broken down by racial groups administered after the election instead?

[statguy]

The biggest mistake here: the fact that finding is "statistically nonsignificant" doesn't tell you much at all. If your sample is small, you'll often fail to get statistical significance even if the true gap is large.

Lack of evidence of a gap is NOT the same thing as evidence for lack of a gap.

[statguy]

Also, if they "released" this "study" why can't I find it anywhere on the internet, and why isn't it linked to in any of the stories? Lame.

[Roy]

The Undiscovered Jew is right. I picked up an extra 2" Tuesday afternoon.

BTW, I for one am hoping against all hope that these studies somehow turn out to be correct. If the Obamessiah's election could cause black crime/education/welfare rates to mirror those of whites, my life would become immeasurably better.

Hell, even if they were only raised to mirror those of Hispanics, the murder rate in my city would drop by well over half.

[Brandon Berg]

Failure to reject the null hypothesis doesn't prove it, especially in a study with a small sample size. Also, it's my understanding that releasing the results of a study prior to peer review is generally regarded as a red flag. Though I'm sure that this rule of thumb does not apply to studies proving the messianic qualities of the Lightworker.

[Mr. Mercy Vetsel]

Statguy is right. The real test is whether the CHANGE in the gap is statistically significant. They could have reported the same results if they only tested one black guy and he happened to score above average.

Notice that this study (or even an abstract) doesn't seem to be web accessible.

Aparently the NYT is learning how to be more discrete in spreading blatant lies directed toward statistically illiterate and scientifically ignorant readers.

-Mercy

[Kevin K]

You'll note that article about the hope for increasing women in science doesn't really mean science, but physics. The rest of the natural sciences already have a significant fraction, and sometimes a majority, of women in grad schools and academic positions.

Throughout my career in physics, I have never met a single female who wanted and had the ability to be a physicist who wasn't given every opportunity in the world. Notice how none were interviewed in that article.