Monday, April 10, 2006

She Blinded me with Very Poorly Conceived Science

Take this delightful test from the Harvard psychology department. Come back when you finish - I'll be waiting right here, in the BLOGOSPHERE!

Ok, basically, this test is supposed to measure your association of certain racial groups with negative ideas, in this case black people with weapons. The researchers claim that this is designed to detect the unconcious roots of racial profiling in almost everyone. In the debriefing page for the test they lament the fact that people who take the test most often associate harmless objects with white people and weapons with black people.

The test works in the following way: you can sort an object or person into one of two categories. it starts off with black on the right and white on the left; then it goes to weapons on the right and harmless objects on the left, then to black OR weapons on the right and white OR harmless on the left. Then it switches categories so that blacks are paired with harmless objects and whites with weapons. My test came back that I had a slight tendency to associate blacks with weapons and whites with harmless objects (cellphones were one of the representative examples, although I don't really think of them as harmless).

Here's the thing: the test measures how many mistakes you made and how swiftly you selected categories. I made many more mistakes in the final category (black and harmless objects versus whites and weapons). Is this because watching American History X in high school turned me into a big ol' racist? NO, it's because I got used to where the categories were on the test! It would be like suddenly having the turn signals on your car reversed: what you associate with them (left and right turns) has no intellectual or decision-making content. You simply know to push the stick down for left and up for right, and if these directions suddenly reverse you're going to make many mistakes. I cannot believe that this actually passes for statistically robust research even in a field as decayed and idiotic as psychology. Even in the dim and badly organized sociology and statistics class I took sophomore year they specifically told us not to design surveys like this, since they cause an egregious bias. The study is so flawed and so unimportant to almost every human being that only the blogosphere can pass judgement on it. The sentence? Death by drowning.

3 comments:

tMa said...

I attempted to take the "Asian" test and could not maintain my concentration. Thankfully this test is irrelevant and skewed so my actual hatred for Asians can be excused as a result of this poorly designed test.

emilia said...

Hate to break it to you - assuming you are referring to the experiment that I think you are referring to, what you call a flaw in its setup is actually integral to its design and, indeed, the key to why it works. What the test is measuring is not what you answer, but how long it takes you to answer. You can read about how the test works in a Slate article that came out a few months ago: http://www.slate.com/id/2134921/

Probably you should do a little research before you blog away. If you find flaws with the test (and the creators do acknowledge that it's imperfect) after actually taking the time to understand how it works, that's one thing. Then again, blogging and thoroughness aren't exactly hand in hand.

ams said...

Emilia, I understand your comment, but I still find the same methodological flaw with the test. The Slate article which you reference claims (as the test-makers do) that the test measures tendency to associate blacks with negative concepts and white with positive. If it takes you a very long time to associate a cell phone with a black person or a machete with a white person, the test claims to have discovered a bias.

What I am alleging is that the test first trains and re-affirms that bias. The first half of the test requires that you associate good with white and bad with black, then the second half of the test reverses that. I argue that this creates the bias that it purports to measure. Think of this analogy: let's say you have a test that asks you to sort even and odd numbers into two categories. It also asks you to sort circles into the even category and squares into the odd category. Let's say you do this for 25 iterations, then the test starts asking you to put circles into the odd category and squares into the even. It seems reasonable that you would initially make some mistakes, but my argument is that these mistakes have nothing to do with previous real-world bias, and everything to do with how the test has trained you to act. You should follow the link and take the test and see if you still disagree with me. Or at least send me some racial bias test/ Dr. House M.D. crossover fanfiction.