Since 1982, PCMag has been a trustworthy source for more than 26 million computer owners and users. About 30 years ago, the magazine began running the iconic and highly coveted Reader’s Choice Awards. Initially, this annual survey used a number 2 pencil and a Scantron form stitched into the magazine. Over the years, this sizeable annual survey evolved into 12 to 16 surveys which were built by an outside research firm according to the PCMag editors’ specifications. It was getting costly to run more surveys.
Saving $30,000+ in Year One
Features Editor Eric Griffith realized he could bring the survey building and data collection in-house using Alchemer (when it was SurveyGizmo). Now it is a large part of his full-time job. “I did a trial run with SurveyGizmo, built a couple of things, and told my boss, ‘I think we should give this a go,'” explains Eric. SurveyGizmo had just won the PCMag Editors’ Choice Award. “The data we got was very workable compared to what we were paying for, at a fraction of the cost.”
Bringing the research in-house, PCMag saved more than $30,000 in the first year, but it required more work for Eric. “I’m now building 12 to 16 surveys a year,” explains Eric. “But we can turn around surveys for others quickly.” Eric can also build surveys ahead of time, duplicate surveys, use what worked, and try something new when something didn’t work. Transferring logic through on duplicate surveys has saved them a lot of time and saved other departments money.
Anticipating How People Will Complete the Survey
Eric’s freelance author and partner on Readers’ Choice surveys, is Ben Gottesman, a former PCMag editor. He reviews every survey before it goes out to make sure PCMag can collect the best feedback. Before anything goes out to the public, they go through each survey a couple of times to ensure that it will mesh perfectly with the results they expect. “We try not to worry too much about what the numbers will be, as far as what will come in,” Eric notes. “We definitely found the need to anticipate how people will fill out a survey because people will take it in a really strange direction if they have too much leeway. The write-ins can be crazy.”
This means listing every possible option. “We try to anticipate that by making a giant list when we create a survey for, say, laptops,” explains Eric. “We’ll list every possible laptop maker we can even though we know some will get little response. And then Dell and HP will have thousands of people respond. But we have learned that if we don’t include certain names, people will write in things that make it harder to analyze, such as writing Inspiron instead of Dell.”
Processing Massive Feedback
PCMag runs one to two surveys a month with between 1,500 and 8,000 responses per survey, resulting in a lot of feedback data. “I’m no statistician,” reports Eric. “Fortunately, Ben Gottesman was doing Reader’s Choice before I came to PCMag. He understands how to export these numbers from Alchemer and port them into his pivot tables and make it all work. He puts together the final tables that we use for the comparisons to write up the results.”
Because the goal of the survey is to understand how the manufacturers compare to one another and not how one respondent’s experience compares to another’s, the category averages are based on the average of each manufacturer’s rating, not the average of every respondent’s rating. The overall ratings are not based on averages of other scores in the table; they are based on reader answers to the question, “Overall, how satisfied are you with your device?”
There are very few times that Ben and Eric allow the surveys to be just for a specific product or model. However, there’s a big difference between gaming consoles, so you can’t really compare Xbox to PlayStation. PlayStation 4, PlayStation 5, Xbox 360, and Xbox One S are so different that the surveys have to be more product-specific. But for the most part, PCMag surveys focus on comparing brands.
PCMag.com is a leading authority on technology, delivering labs-based, independent reviews of the latest products and services. Their expert industry analysis and practical solutions in categories ranging from PCs and consumer electronics to software, online services, and security, help people make better buying decisions and get more from technology.