To a direct marketer, testing is vital. But it’s important to know exactly what you’re testing. If you’re not careful, what you think your test is telling you may not be what it’s saying at all.
It’s not just in direct marketing and business that testing matters, as you’ll see in a minute.
Evan Jones, my good friend and ex-Partner-in-Crime at our old game company, QED Games, Inc., is beginning to make a name for himself in an entirely new field, Climate Change. He’s about to be the “et al” in two scientific papers and was a key researcher in a current report to Congress.
Evan is part of a group led by meteorologist Anthony Watts (who writes a very popular blog, Watts Up With That, aTechnorati Top 5K blog with a ranking of 2083!) that is focused on a major aspect of Global Warming: not whether it’s happening, but whether we know whether it is or not.
Evan and his colleagues have been examining the over 1200 U.S Historical Climate Network (USHCN) surface stations in the US that measure temperature. And what they’ve found is surprising. Nearly 89% of these stations are located in situations that render their data suspect or flawed according to the government’s own standards. Most of these stations were once fine, but encroaching urbanization has frequently turned an isolated station into one surrounded by heat sources. Add to this the fact that temperatures are recorded by citizen volunteers and are roughly 30% incomplete.
There’s more, of course. You can read about it here in this article from WBZ TV in Boston. Better yet, WBZ did an interview with Evan while he was up in Boston assessing some of the surface stations there. You can watch the report here. And be sure to check out Anthony’s blog for even more examples of questionable data.
In the end it all adds up to one thing: the data doesn’t add up. It is, for the most part, not telling us what we think it’s telling us. For instance, when a station formerly situated in the middle of a field registers a temperature increase over the last decade but that station is now situated in a blacktop parking lot with an air conditioning exhaust unit nearby, is the planet getting warmer, or just the station’s readings?
Whether you’re building an online research survey, setting up a 16-cell direct mail testing matrix, optimizing a paid search campaign or collecting temperature data to prove or disprove global warming, you need to check your inputs, check your confidence in the amount and clarity of the data, and structure the test to actually answer the questions you’re asking.
It’s critical that your testing framework not be flawed or all of your results could be useless. It’s equally important that you analyze the testing results accurately. And if you go ahead and act on bad information, whether it’s a new product launch or an attempt to save the planet, you could be doomed to failure before you start.
And speaking of global warming and saving the planet, can someone please explain to me how anyone can be so certain of the truth when the tests themselves are flawed?
Can someone please explain...? 