Don’t Listen to Specs. Keep Field Testing.

In practice, the devices we use often perform unpredictably — contrary to their technical specifications. I have often experienced good performance from low end devices, and also experienced bad performance from high end devices.1 My experience seems mysterious and ridiculous. Why should a device that has last years’ specs work better than this years’? Others have told me that they have similar experience, when tech for who-knows-what-reason isn’t working consistently with specs.

This is one of the reasons Apple geeks enjoy MacBooks and iPhones so much more than the comparable devices from other brands. Let’s say that you look at the specs on a PC notebook and observe it is more powerful than your friend’s MacBook. In use how well do they compete? If you turn off your zealous commitment to technical stats (or the Apple brand for that matter) which of the devices seem to operate with more precision? Which of the computers keeps up with your every key press and mouse click? Which of the operating systems pleasantly surprises you? Which notebook intuitively matches your use case?

For many of us, the answer is the Apple computer passes with flying colors — all six colors — and the PC notebook languishes. We aren’t just fan boys swayed by Apple’s brilliant marketing.2 We have personal experience when the Mac and iPhone simply outperform the competition regardless of the competition’s superior specs. I’ve used Apple products for two decades. In that time, most every Apple device I used performed better than the competitor’s in my test drives. Am I lying to myself? I don’t think so. I believe that the quality of performance goes far beyond the technical specs.

Which would you benefit from more? The device that works the way you like? Or the device that some other party declared was better thanks to its hardware and software surpemacy? I would rather use the machine that just works for me in spite of the coolest and latest in gigabytes or gigahertz. Because, time and again, the specs were trumped by the quality of my user experience.

This isn’t limited to the Mac versus PC computer war. This is a phenomenon I encounter with all sorts of technology, where devices and features perform in direct contradiction to their presumable performance (based on the limits of specs). If we are not careful, we let our biases for a brand, scientific mumbo jumbo, or a consumer report mislead our workflows. We would like to believe that computers3 will always work predictably. We don’t stop to compute the changes in real-world experience. Whether we can account for them or not, real-world experience sometimes outweighs scientific stats.

If we want to stay productive, then we should be alert to changes in performance and not assume our devices will always work for us the way they have in the past. We have to re-wire our anticipation: look for the changes in performance on a daily basis and reconsider how we might optimize our devices’ performance in new situations. If your utmost productivity is key to you, then you need to be flexible with the ever-changing behavior of your tools.

All of these thoughts and considerations — and my new-found appreciation for my device’s performance — started with a search for a good Wi-Fi hotspot away from home and the office. Today I’m using the Wi-Fi at a Chick-fil-A restaurant where my younger sister is employed. I thought I would like to work remotely, and I actually need to work remotely because the Internet access is down at home till Comcast restores it tomorrow afternoon.4 Lately, I have been checking out several local hotspots, Chick-fil-A’s being one of them. I have been running a few speed tests to see how the Starbucks, Barnes & Noble, Chick-fil-A and other cafés’ Wi-Fi hold up.5

To test speeds, I use It is not the only way to clock a web connection, but it does the job and the site gets the job done. (If you want to recommend a better speed tester, by all means share it with me.)

I’m not going to factually detail the speed tests results here because I don’t want to pay much attention to the stats. To reinforce my point, what matters is how the Wi-Fi actually performed for me — not what the speed tests told me with the numbers. I clocked the speeds because they should matter, but what matters to me most of all is the performance I get from the Wi-Fi in my experience. It doesn’t matter if B&N says it gives me 4.23 megabits per second if it shows signs of sluggishness (maybe because the Wi-Fi is shared with other customers?). It sounds like a contradiction. How can it be slow if it is scientifically supposed to be fast?

And it doesn’t matter if Starbucks’ Wi-Fi is provided by AT&T or Google. What matters is whether the access they provide me in first-hand experience is snappy and reliable: keeping up with me as I use the web for browsing, writing, banking, shopping, and researching. If I hit long delays for uploads or downloads then who cares what the speed tests say? What matters more is when the rubber meets the road.

Friends told me to rely on the local Starbucks, so that is where I started my field tests. Not surprisingly, the interface for joining Starbuck’s Wi-Fi was the most pleasant to use (to my design sensibilities) but beyond logging in, the experience was very sluggish. In speed testing, it was about 1.2 megabits up and 1.6 megabits down. That’s very low, yes, compared to what you would expect your Wi-Fi at the office and perhaps your home to be. But for the sake of an alternative work environment to keep work afresh, how bad can 1.2 megabits actually be?

Awful! It seemed to me that while I ran multiple open tabs and tried to get work done that nothing was loading or refreshing. I would open a video and everything would come to a screeching halt. I would load a site that I had not previously visited (so there wasn’t anything in my browser’s cache for the site) and nothing would finish loading. Starbucks’ “high speed Internet” might as well have ran at 10 kilobits per second, because I couldn’t get anything done online in five hours.6

Barnes & Noble’s was surprisingly better. The download speed clocked around 4.0 megabits. How it felt was also superior to Starbucks’ Wi-Fi. I got work done over the 3 hours I was there, and I would consider it a viable “office away from the office” in the future. But did the speed feel like 4.0? Not really. Even though the service was four times greater than Starbucks’, it felt like it ran around 2.5 megabits a second, just based on my powers of observation.

As I explain all of this, yes, I admit that I’m using a subjective field test. My reasoning is that the service that works best in my use case is better than the service that claims better performance or clocks better in speed tests. Our human experience with tech is important, because the user and his opinion matters much more than the data. Call me crazy, but this makes sense because at the end of the day we all are lead by our subjective vantage point. We don’t live by objective truth or facts. Now, back to my field tests.

Chick-fil-A’s was the most surprising of them all. As a dining establishment focused on “quick service” with a two lane drive-thru, I would have figured that Chick-fil-A’s free Internet was circa 2009 or 2010 and largely overlooked by corporate and the operator. It’s not like they are known for their free Internet access, like most coffee shops are. I see some people at Chick-fil-A using the Internet, but not as many people as there are at Panera Bread. A constantly rotating collection of professionals work remotely every day at the bakery-café six minutes from my house.

In the speed test, Chick-fil-A was 2.8 megabits down and 0.4 megabits up. Yes, that’s better than Starbucks’ download rate, but significantly slower than B&N’s. Yet, how well did the Wi-Fi perform? For one thing, there were no unexplainable Wi-Fi pauses. Starbucks’ would work erratically every minute, coming and going with interruptions. B&N had fewer interruptions, but there were long pauses for Twitter, YouTube, Squarespace and other sites that I encountered that were unexplainable.7

Besides interruptions, and a general sense of slow loading times, what else is there to grade? The Wi-Fi at Chick-fil-A was not the fastest in the speed tests, but it outpaced the fastest (B&N’s). The restaurant wasn’t the obvious place to work remotely (Starbucks), but it has the most reliable Internet connection of the cafés in my town.

So, the next time someone or some company boasts the best in tech, don’t trust them apart from your personal experience with their device, app, or service. See what you like. See what actually works for you. Did it run reliably? Did it operate the way you expected it would? Would you use the technology in the same fashion if given the choice in the future?

Attempt to discover the right use case — not the service, device, or app that boasts the best specs. Field testing is still important, because the specs don’t lie, but they don’t necessarily reflect the potential performance.

  1. I use “low end” and “high end” terminology based on the present interpretation of available products. It wasn’t that long ago that an iPhone 4s was considered state-of-the-art. Today, it is largely considered obsolete by most people, as it doesn’t fluidly and reliably manage iOS 8 and iOS 8-optimized apps.
  2. Which isn’t consistently brilliant, to say the least. Yes, we Apple geeks notice the less-than specatular ads too, my Android/Windows friends.
  3. Whether they are powerful desktop computers or simplistic smart watches, we like to assume that specs = a rate of the quality of the device’s performance. So if last year’s device has a processor that runs three-fourths the speed of this year’s model, it would seem likely the new device will run 25% faster. But there are other variables to consider, so this is a mistep in deductive reasoning.
  4. The other day, workman adding better urigation for my house’s rain water cut the coax that feeds my house’s Internet service. I called Comcast on Friday. The soonest they could get someone to my place to restore my access was scheduled for Tuesday afternoon.
  5. Over the last few years, Wi-Fi around town hasn’t improved very much. You are going to be fortunate if you find a place with 1.5 megabits down and 0.7 megabits up. If you happen to be blessed with a café offering better speeds than this, good for you!
  6. According to Starbucks, they are supposed to offer fast Internet service at all locations via Google within 18 months of August 2013. There is hope yet for my local Starbucks, but I’m not going to hold my breath with anticipation.
  7. There are so many ways that Wi-Fi can be configured that it is difficult to figure out what was causing the problem. On the surface, it would seem the Wi-Fi was inhibited by the number of users sharing the signal. But who knows for sure.

Joe Darnell

Joe is a UI and graphic designer with prior experience as the creative director for three media-based businesses. Joe’s passionate about web design and graphic design with about 15 years of experience in the media industry. Additionally, Joe is the host of the Top Brew and Techtonic podasts, both featured on iTunes.