The convenient data set problem.

In an April 21, 2012 Article, “Sexual Strategies,” the Economist describes research by Robert Dunbar that purports to show that the sex of peoples’ best friends varies by age. This research illustrates two problems that journalists, in particular, ought to be sensitive to. I suspect that it is a common sort of research study that arises because of the discovery of an available dataset that the researchers adopt not because it is appropriate but because it's the best the researchers could find or afford, or simply because it seems to support the theory the researchers want to advance.

The first problem is the impossibility of knowing what population the research findings apply to. The population studied by Dr. Dunbar is the customers of an unnamed European mobile-phone operator, a sample that is almost certainly not representative of the population as a whole: the mobile-phone operator may be more popular with the young or with particular nationalities, or with the gay population, for example.  The mere fact that the sample is large is not sufficient to make the results meaningful; it must also be representative of some definable population. The problem is perhaps most obvious if we suppose Dr. Dunbar were to repeat the study in 5 years and identify differences in the results--we would have no way to know whether those differences represented changes in behavior of the subject population or changes in the composition of that population (e.g., that the mobile operator has become more popular among the elderly).  We just cannot know for what population Dr. Dunbar's results apply.

The second problem is that the research claims to provide information about peoples'  best friends, but "best friend" is defined as the person with whom the subject has most frequent cell phone contact.  That's clearly not how one would define "best friend" and it is surely debatable whether “most frequent phone contact" is a good way of identifying a subscriber's best friend.  Maybe most frequent phone contact is more typically with a child, child-care worker, or business colleague.  Maybe “most minutes of phone contact” would be a better definition of best friend, or quite possibly one tends to speak with their best friend mostly in person and relatively infrequently by phone.  The point is that the researchers have studied "most frequent phone contact" and rather arbitrarily labeled that as "best friend." Possibly they did not have information on the length of the phone contacts, so they had no other choice if they were to force this arbitrary data set to fit the subject they wanted to address.

It would be a huge improvement in reporting if journalists could recognize and point out these sorts of questions.