I'd like to request that more information be given with surveys such as that in your article, "Game Stores Booming in 2015."
According to the article game store retailers responded to a survey and the results were posted. However there is no information given as to how many stores responded or how they were verified to be actual stores. Without knowing the number of respondents it’s hard to give credence to the "facts" contained, especially since the game industry itself, including the Game Manufacturers Association (GAMA) and the GAMA Retail Division (GRD) does not have an accurate count on the number of independent game stores.
I am in multiple retailer groups with hundreds of national and international game store retailers. There has been much discussion in these groups about this survey. Of the retailers commenting, none were aware of the survey or asked to contribute.
Please provide accurate numbers as to how many retailers responded regarding your article. Posting incomplete information helps none of us.
The opinions expressed in this Talk Back are solely those of the writer, and do not necessarily reflect the views of the editorial staff of ICv2.com.
ICv2 Responds: We had several requests for more information on how the survey was conducted, including this Talk Back and the Rolling for Initiative column by Scott Thorne (see "Rolling for Initiative--Game Stores Booming Survey").
First, a little background. The survey questions on which we reported in the "Game Stores Booming" article were part of an online ICv2 reader survey conducted in November. We’ve been doing those surveys for years, but this year, we did something a little different. Much of the survey was devoted to questions specific to ICv2 (as we’ve done in years past), but at the end of the survey we added a few questions on the industry as a whole. This was done as an experiment to collect some information that might be of interest to our readers, rather than just info that ICv2 uses to run its business.
We suspect that some of the people Rhonda spoke of above, that said they hadn’t seen or heard of the survey, may have not realized that our reader survey included questions about the industry in general so may not have realized they saw promotion of the survey. And now, to give background for the additional info we can provide, here’s what we said about the survey in the original article:
- Survey respondents were recruited via multiple channels (ICv2 email, editorial, and advertising) during November 2016. To qualify for this sub-group, respondents had to self-identify as retailers and describe games as their primary business (biggest category in dollars). Given that the sample is self-selected and relatively small relative to the size of the population, it may differ significantly from the actual situation for game retailers as a whole.
That info was provided at the end of the article, so may have been missed by some.
To provide more background on the description above, respondents were recruited three ways. We ran ads and editorial notices for the survey in the ICv2 Daily Insider, our email newsletter in which we provide subscribers with the daily headlines every business day. On ICv2.com, we served display ads for the survey over 320,000 times. And we also pinned an article promoting participation in the survey in a prominent position on our home page for much of the month.
We used tracking URLs to determine where the survey respondents came from, and the article pinned on the home page (see "ICv2 Launches Industry Survey") was the most successful source of respondents, generating around 70% of the total.
Forty-seven respondents identified themselves as retailers (either brick and mortar, online, or both), said that their largest category in dollars was games, and completed the portion of the survey reported in the article. This is a relatively small subset of the nearly 500 survey respondents, which included all kinds of ICv2 readers. It’s also a subset of retailers: it doesn’t include all retailers that carry games, only those that identify tabletop games as their largest category. Also, the questions on which we reported in this article were over 75% of the way through the survey and as a result had fewer respondents than questions closer to the beginning.
Rhonda asked whether retailers that identify themselves as retailers for the survey are verified as retailers in any way, and the answer is that no, we trust what people tell us. But it’s unlikely that there are material numbers of intentionally false responses that cloud results because there’s a cost (the time it takes to complete a survey) and no benefit for a respondent to give bogus answers. The survey tool we used to collect the survey prevents multiple answers from the same device and has other protections from abuses of the survey.
What percentage of the population of stores does that represent? Based on what distributors tell us, we’d estimate that there are between 1200 and 1800 stores that would meet the definition we used for this question: stores in which tabletop games is the #1 dollar category. That makes the sample that responded to this portion of the survey 3-4% of the population.
This was not the kind of survey that involves random phone dials, it was an online survey. As a result, as we noted in the original article, the sample is "self-selected," meaning the respondents decide whether or not to respond, which may introduce bias into the sample selection. Also, although we might wish otherwise, there are probably retailers out there that don’t read ICv2. We don’t know that the people that responded differ from the retailer population as a whole in material ways, but it’s possible they do.
All those caveats are why we said in the article, "Given that the sample is self-selected and relatively small relative to the size of the population, it may differ significantly from the actual situation for game retailers as a whole."
But we believe the information has value. It represents the views of a significant number of retailers, and the results conform with what we hear from retailers, distributors, and publishers on an anecdotal basis. It’s also a continuation of a persistent growth pattern that’s been running for over half a decade (see "Six Straight Growth Years in Hobby Games"), and had continued through the beginning of 2015 (see "Hobby Games Market Climbs to $880 Million").
The designer of the survey has a degree in sociology (the underlying science behind survey construction and interpretation) and did graduate work in the field, so although it’s undoubtedly not perfect and has the limitations described, it’s also not designed by an untrained amateur. We’ve been using market research here at ICv2 since the mid-00s, and at the company that owned Internal Correspondence back to the early 90s, so have considerable experience in using surveys, both ones we conduct ourselves and independent, random surveys, the gold standard for market research.
Should this survey be the only information on which businesspeople make decisions? Of course not, but it can be useful, along with other information, in understanding the market.
We’re gratified with the many comments and questions we received about this article (thanks Rhonda, among others), and about the accompanying article we did on the comics market (see "Comic Retailers Confident about Customers, Concerned about Publishers"). The reaction tells us that there is considerable interest in survey data from retailers about this market, and that there’s support for efforts to make that data better. As we mentioned at the beginning of this lengthy response, we asked the questions on which we reported as part of a larger reader survey on an experimental basis. Based on the response, we’re going to pursue doing surveys focused more on market conditions and issues rather than primarily on ICv2 reader behaviors. At that time, we’ll start to reach out beyond the ICv2 readers for participants in an effort to increase the survey size and its representativeness. We look forward to your support in finding out more about the market, and to reporting the results.
--Ed.