written by Paul Johnson,

So I was excited for the AAPOR webinar on hard-to-reach populations because I really feel like this is the hardest nut to crack in the industry. Unfortunately, I left being underwhelmed probably because of a misalignment of expectations. I came in thinking that hard-to-reach is the same as hard-to-sample so I was expecting the webinar to focus on the hard-to-sample challenges. I am grateful to Dr. Tourangeau for helping me broaden my horizon. As an employee of a sampling company, sometimes I get too focused on the hard-to-sample problem and not enough on the big picture. Still, for this post I want to focus on the hard-to-sample population and open a debate on whether or not respondent-driven sampling can actually produce good estimates that can help a company make informed decisions.

Respondent-driven sampling is a type of snowball or referral sampling. It relies not on standard probability theory, but rather on network theory to estimate the probability of being selected from the population. It makes the jump to say that it doesn’t matter where in the network you start (the seeds of the study), but as long as you map out the network you can make adjustments to account for the probability of selection inside the network. That is a large claim. Taken at face value it is hard to imagine that a convenience sample of a connected convenience sample with somehow turn into a probability sample, but the math works out in theory. Whether theory turns into practice is another issue though. Dr. Tourangeau did a great job of listing the key assumptions of the theory behind respondent-driven sampling:

• The population that connects to each other. After all, if the population doesn’t connect to each other then there is no network to examine and you are not likely to get referrals anyways. This one doesn’t concern me too much as I really believe that most of these hard to reach populations are connected in some way as the whole world become more connected. However, if you are doing respondent-driven sampling and you don’t get any referrals, you have to ask yourself: “Why am I doing respondent-driven sampling anyways?” You might need to reexamine your population or realign your incentives.
• The population is in a single network. Once again, I am not as much of a skeptic here. Even when it is shown that there are distinct clusters inside the network, there is some crossover. As an example, you can look at a picture of the some of the political networks and you will see clear clusters by party but there are still some connections between the two clusters from the people who work on both sides.  http://blog.magicbeanlab.com/networkanalysis/calculating-party-affiliation-us-congress/  NOTE: that this might not be the best example because a lot of Congressman did not tweet each other, but the principal of network crossover still applies.
• Each respondent’s number of network connections needs to be known. I could see some respondent reporting error in here. Frankly though while it might not be exact it should be a close enough approximation to use for our purposes. Also in the future, as social networks develop we should be able to collect some of this data passively depending on the audience we are measuring. This seems to be a solvable problem.
• Respondent need to randomly recruit from their connections in the network. This is the one that gets me. I think that some people’s attributes will make them more likely to be referred than others. At a minimum, the type of connection (friend, family, professional) would have very different chances of being referred. Even if you instructed the respondent to randomly invite the people they know I am not a big believer in the ability of the human mind to create random numbers (our brain isn’t really wired that way in my opinion, but it is a fun topic to Google and of great importance to the computer science community). I am sure that this assumption could be relaxed as long as the probabilities of being referred were known. That is the tricky part though. How do you elicit the probability of referring each of the connections in the network that you have? I don’t have the answer here, but I do think it is an interesting challenge.

All that being said, frankly with most of my work my clients just need a “fit-for-purpose” solution. Respondent-driven sampling might just be that even if it doesn’t approach a probability sample in real life as well as in theory. Look out for my colleague Kristin Cavallaro at AAPOR conference in Boston next month to see how this type of sampling compares to traditional opt-in panel sampling. When you have budget, but you want to make intelligent solutions sometimes “good enough for government work” – or in this case maybe “good enough for non-government work” – is all you need.