Every year new studies about women came out. Some “proving” that women “depend” on social media while men just look up the weather and sports scores, others that women are “biologically” less funny than men.  They are both, of course, ridiculous.  But since way too many people give them credibility, let me explain why you shouldn’t.

Method

First of all, many of these studies depend on self-reporting.  In other words, scientists hand people a form, tell them to fill it out, and expect them to be honest.  Ask anybody who has ever worked in marketing: people will lie for candy, forget $50.  Self-reported surveys used in a study should be seen with skepticism, to say the least.

Sample Size

Many of these studies extrapolate off of a few hundred people to generalize about millions.  Some, like the “women no funny” study, involving sitting in on a few corporate meetings: you know, because those are a brilliant measure of human hilarity.  There are statistical reasons for this, but many studies do tend to fall into the overgeneralization trap, especially if they don’t really have the budget to justify a larger study.

Sample Quality

Similarly, many of these studies use college students because, unlike college graduates and other people with jobs and families, college students have lots of time on their hands, need money, and can be ordered to be involved in a study for free by their Psych 101 professor, who doesn’t know or particularly care about them as a person, but does know they want the 10% of their grade the participation requires.

Shockingly, most of these studies tend to uncover that teenagers can have somewhat backwards and/or clueless ideas about gender, which is not surprising since we all grow and change as human beings throughout life.  But hey, half the participants had a vagina and half had a penis, so it’s accurate enough, right?

Interpretation

Finally, there’s the fact that, in many of these studies, the real psychological test is how you interpret the results.

For an example of this, pick any evolutionary psychology study, which tend to be about gender when they hit the larger media.  No, really, just a random one, anyone will do.  It might say something perfectly sensible, it might say something so utterly whacked out you’re amazed it ever got into print.  But it’s the leap, that our brains evolved to prefer this to that, that’s really the problem.  You can’t demonstrate that it’s not, say, social conditioning, or sample bias.

Say your evolutionary psychology study is about how women prefer to stay at home while the menfolk bring home the bacon.  If you ask a bunch of stay-at-home moms whether they’d rather be working or stay at home with their children, of course the majority are going to say they want to stay with their lovely children.  If they do, they’re just being honest.  If they don’t, they might be afraid of being judged as a bad mom for wanting to escape the house.

In short, scientists, especially in the “soft” sciences are as fallible as the rest of us.  So, whenever you see something proclaiming all women this or that…grab your salt shaker, because you’ll need a few grains.