Southwest Airlines is the top air carrier in the United States. No, wait, it’s American Airlines. Hang on — make that Virgin America.
The best hotel? The Peninsula Chicago. No, no. It’s The Waldorf Astoria in New York.
That’s the consensus — if you can call it that — of the latest round of travel polls and “readers choice” surveys. The results are all over the map — literally and figuratively. None of them seem to agree on anything, leading me to wonder if any of them are believable.
Here’s the problem: Travelers make important vacation decisions based on these questionable surveys. And with more media outlets jumping into the survey game every year, the chances you’ll be at confused or even misled are better than ever.
One new poll claims travel agents voted New York’s John F. Kennedy airport as their favorite. Nonsense, says another — people prefer Las Vegas’ McCarran International Airport. (The correct answer is Orlando International Airport, if I do say so myself.)
Why do these surveys contradict each other? The obvious explanation is audience and methodology. Although in some instances, calling it a “methodology” is generous. None of the travel polls are particularly forthcoming about how the sausage is made.
But there’s more. As someone who has practiced journalism for a little while, I know that the publications sponsoring these awards aren’t necessarily doing this as public service. Awards are marketing tools. Ads are sold around the final results. Award dinners and plaques can be big moneymakers. They also raise the publication’s profile.
Put differently, these aren’t travel industry awards — they are an industry unto themselves. And it’s an industry that isn’t necessarily concerned about your next vacation.
Here’s how to spot a self-serving survey:
Beware of resorts that over-publicize their awards
I realize this isn’t going to make me any friends in the public relations community. But airlines, hotels and cruise lines that snag one of these dubious awards and then blast a press release about their accomplishment to every travel journalist on the planet are hurting themselves. They think they’re sending one message: “Stay with us — we won a five-star/best-of/readers-pick award!” In fact, they’re probably sending another one: “We’re insecure. We need someone else to validate our product. We don’t really believe in ourselves.” Never mind what it says about the award.
Some methodology is better than no methodology
Always look for a few details about how the poll was taken. How many readers were surveyed? What was the response rate? Did they use an independent company to compile the results? Most importantly, how did they determine who was on the ballot? Was it just a group of editors sitting in an office, adding their favorite hotels to the list? Or were the finalists the result of a popular vote? None of the major awards reveal all of these facts, as far as I can tell. The more reputable ones disclose some of their methodology. The bogus ones don’t publish any of it.
A ‘best-of’ list without a ‘worst-of’ list is pointless
Surveys that highlight only the positive aspects of travel are in serious denial. The most credible awards acknowledge both the winners and sinners. For example, SmarterTravel.com recently asked readers to name their least favorite airline and the carrier with the dirtiest cabin. Troubled US Airways won both honors. Handing out an award without having something to compare it to is meaningless. So if your favorite media organization only talks up travel, maybe it’s time to find another favorite media organization.
Big numbers equal big trouble
One of the biggest lies you’ll find in a travel survey is the number of respondents to a poll. I’ve seen claims that range from more than 100,000 (unlikely) to more than a million (impossible). Why exaggerate? Because it makes a poll look credible. When you see an unrealistically big number, you should think the opposite. Remember: if it looks too good to be true, it almost certainly is.
Are the winners running ads in the publication that honored them?
If they are, the survey is probably junk. I can practically guarantee that someone from the advertising department promised a “favorable consideration” in exchange for advertising. Here’s a handy rule: The closer a resort’s ad appears to the poll in which it wins its award, the less credible the poll. Trust me on this.
If something doesn’t look right, the survey is probably wrong
Here’s another sign something may be amiss: the wrong companies are winning awards. For example, you’d expect an airport like Singapore or Hong Kong to take top honors in a reader’s survey. But when an airport that people don’t just dislike, but actively avoid, gets top billing —well, you know something is fishy. I rest my case.
Look for obvious, but unstated, biases
These aren’t ever disclosed, but they have to be considered when you’re reviewing the results. A magazine devoted to luxury travel isn’t going to recommend a hotel with reasonable rates.
An online publication that caters to budget travelers is likelier to vote up a hostel than a five-star resort. Editors can also play a huge part in shaping the results you see. They can exclude categories, downplay entire industry segments and make insignificant businesses seem monumentally important. Boutique hotels, anyone?
Some awards are not meant to be
There’s a class of travel awards that should probably be completely discounted. One well-known site conducts a survey of the cleanest airport restrooms. Think about that. Are you going to base your decision to use an airport on its bathroom facilities? Another travel magazine rates the top airline loyalty programs. But that assumes you have a choice in loyalty. In today’s recklessly deregulated, oligopolistic airline industry, practically no one has that luxury. Oh, and you have to ignore the irony that the award is named after an airline entrepreneur whose carrier never had a frequent flier program.
It would be easy to dismiss every travel award as flawed, if not fraudulent. But then I wouldn’t have any data with which to commit acts of journalism every week.
Instead of ignoring every poll, I’d take a closer look at each one before making an important vacation decision. Is it balanced? Does it acknowledge the reality of travel? Is it using sound methodology? And is it useful?
If the answer to any of those questions is “no” then do yourself a favor. Turn the page.
Every Monday, my column takes a close look at what makes the travel business tick. are always welcome, and if you can’t get enough of my column, for daily insights into the world of travel.