In 1936 President Franklin D. Roosevelt was up for re-election, and his Republican challenger was Alfred M. Langdon. Two polling companies set out to determine who would win the election. The Literary Digest sent out 10 million surveys, received more than 2 million of those back, and predicted that Langdon would win by a large margin (57-43). George Gallup, through the newly-founded American Institute of Public Opinion, sampled only 3000 people and also predicted a large margin (54-46), but for Roosevelt. And indeed, Roosevelt did win, with 61% of the popular vote and 98% of the electoral vote. Why was Gallup’s prediction so much more accurate than The Literary Digest‘s, even though the sample size was so much smaller?
The answer has to do with the sampling method. The Literary Digest sent surveys to three groups of people: its own readers, registered automobile owners, and telephone users. But this was 1936, seven years into the Great Depression, and even though the population of the US was only 128 million people, this large sample was not representative of the voting population. It was a wealthier sample, and in this particular election the wealthier people in the United States were more likely to vote Republican. The surveys also relied on volunteer response, which is now viewed as an extremely biased method of sampling because people with strong feelings are much more likely to respond.
According to Ben Wattenberg on PBS’s The First Measured Century, Gallup used interviews instead of paper surveys. He also relied on “Quota Sampling”, a method used by Archibald Crossley and Elmo Roper, in which interviewers were given quota for different categories (e.g. middle-class urban women, lower-class rural men) in order to make sure that the opinions were from a representative sample of the voting population. The results of the election provided powerful evidence that the method of sampling matters more than just the number of people surveyed.
The FMC program segment above has a lot of interesting tidbits about this particular poll. You can also read more at the site History Matters or under The 1936 election and The Literary Digest on Wikipedia (although the Wiki sites say Gallup surveyed 5000 people, and other sources I’ve seen quote 50,000 interviews, both of which contradict Wattenberg above) .
The cover of vol. 68, issue 8 (number 1609) of 19 February 1921 edition of The Literary Digest, shown above, is available here from Wikimedia Commons (where we get most of our images); it is in the public domain in the US because it was first published prior to 1923. Isn’t copyright information fun?