Tag: testing

Categories
Business Directed Advertising Marketing Media Misleadership PR and News

What does global warming have in common with junk mail?

To a direct marketer, testing is vital. But it’s important to know exactly what you’re testing. If you’re not careful, what you think your test is telling you may not be what it’s saying at all.

It’s not just in direct marketing and business that testing matters, as you’ll see in a minute.

Evan Jones, my good friend and ex-Partner-in-Crime at our old game company, QED Games, Inc., is beginning to make a name for himself in an entirely new field, Climate Change. He’s about to be the “et al” in two scientific papers and was a key researcher in a current report to Congress.

Evan is part of a group led by meteorologist Anthony Watts (who writes a very popular blog, Watts Up With That, aTechnorati Top 5K blog with a ranking of 2083!) that is focused on a major aspect of Global Warming:  not whether it’s happening, but whether we know whether it is or not.

Evan and his colleagues have been examining the over 1200 U.S Historical Climate Network (USHCN) surface stations in the US that measure temperature. And what they’ve found is surprising. Nearly 89% of these stations are located in situations that render their data suspect or flawed according to the government’s own standards. Most of these stations were once fine, but encroaching urbanization has frequently turned an isolated station into one surrounded by heat sources. Add to this the fact that temperatures are recorded by citizen volunteers and are roughly 30% incomplete.

There’s more, of course. You can read about it here in this article from WBZ TV in Boston.  Better yet, WBZ did an interview with Evan while he was up in Boston assessing some of the surface stations there. You can watch the report here. And be sure to check out Anthony’s blog for even more examples of questionable data.

In the end it all adds up to one thing: the data doesn’t add up. It is, for the most part, not telling us what we think it’s telling us. For instance, when a station formerly situated in the middle of a field registers a temperature increase over the last decade but that station is now situated in a blacktop parking lot with an air conditioning exhaust unit nearby, is the planet getting warmer, or just the station’s readings?

Whether you’re building an online research survey, setting up a 16-cell direct mail testing matrix, optimizing a paid search campaign or collecting temperature data to prove or disprove global warming, you need to check your inputs, check your confidence in the amount and clarity of the data, and structure the test to actually answer the questions you’re asking.

It’s critical that your testing framework not be flawed or all of your results could be useless. It’s equally important that you analyze the testing results accurately. And if you go ahead and act on bad information, whether it’s a new product launch or an attempt to save the planet, you could be doomed to failure before you start.

And speaking of global warming and saving the planet, can someone please explain to me how anyone can be so certain of the truth when the tests themselves are flawed?

Categories
Directed Advertising

Are you passing the test?

In Fareed Zakaria’s current bestseller, “The Post-American World,” one of the conclusions he reaches about the American education system compared to that of other nations is that “Other educational systems teach you to take tests; the American system teaches you to think.”

This got me thinking about testing, that critical component of successful direct marketing. Thinking and testing come together in direct marketing. We think, then we test. Then we leverage what we learned to maximize our results.

I started in mail order, and I currently work at a direct marketing agency called Tanen Directed Advertising, where we bring direct marketing disciplines to everything we do. Or at least we try to. Sometimes clients say the budgets aren’t there for testing. Sometimes the universes are so small there’s no point — there’s not enough there to be confident that the results mean what we think they mean, or to leverage whatever we might learn from the test, and the incremental cost of splitting up the universe and printing or creating multiple versions is prohibitive.

But it hurts me not to test. A couple of years ago, at AdTech NY, I heard Roy de Souza, CEO of ZEDO,  an internet e-commerce and ad-serving tech company, share this piece of advice about testing: “2 with one difference.” Roy said that if you buy 2 search ads and change one single item between them, they will
never perform the same.

I think that holds true for just about anything.

The Vice President of Marketing for Trump University is a friend of mine, Josef Katz.  He’s the Marketing Maestro who writes the TrumpUniversity Marketing Blog, and he was recently interviewed by eM+C magazine. Along with discussing behavioral advertising and social marketing, Josef talks about how he used multivariate testing of an event registration page to increase conversion by over 75%. 75%! The biggest factor in the increase:  moving the registration form below the fold. He said the move allows visitors to read more about the event’s content before signing up. Before the move, they were still clicking but converting at a lower rate.

A guaranteed winner. Huge increases in conversions. What’s not to like about testing?

And yet, some people don’t like testing. I remember a former client of mine who wouldn’t go with our proposed testing matrix, and said, “I don’t need to test. I go with my gut.” To which I replied, “I go with my gut, too. I just test it, along with whatever else makes sense.”

By now you’re probably starting to wonder, “where’s the question, Jeff?” Well, here it is. There are plenty of people out there on both the client side and the agency side that never test, that look down on direct marketing as somehow less important than “real advertising.” That are more than happy to throw money at events that can’t be tracked to sales, ads that can’t find their targets, and imprinted premiums like pens and flash drives that don’t work very well as ads and, in a short amount of time, stop working altogether.

So can someone who doesn’t believe in testing please explain to me why, in this day and age when testing is so easy, are you failing to test everything that can be tested?