Our Big Remote Control Button Tally

One unintended consequence of hosting a bunch of video devices in an over-the-top video lab is the ceaseless clutter of remote controls that line the windowsill. At the moment, our remote count is up to 12, and our windowsill looks like this.

So far, GoogleTV earns the dubious honor of having the most buttons, at 91; AppleTV offers the least, at seven. Our question was this: To what extent does the number of buttons on the remote – and, for that matter, the inclusion or exclusion of a QWERTY keyboard – help us to find and play TV?

Happily, all of the devices in the lab have at one thing in common, besides the fact that they all come with a remote control. The common element: Netflix.

For that reason, we opted to use Netflix as a common point for testing all of our devices. We decided that the best way to measure ease of use was by choosing a piece of content on Netflix, then counting the number of button presses it took to play it on each device.

We also counted the buttons on each remote control, to see if we could find any correlation between usability of the device and complexity of the remote control.

I studied Biology in school, and the biggest lesson I learned is that experiments rarely go exactly as planned. This was no exception.

We tested each device by opening an episode from the AMC series Mad Men, but quickly realized that because Netflix remembered our activity between (most) devices (technically speaking, this is called “preservation of state”), the results would be skewed on our first device.

So we retested the first device, and also amended our test to include more scenarios: Searching for “Mad Men,” tracking when (or if) the predictive search kicked in, then closing the app and counting the steps to open Netflix and play that episode through the fastest method possible (Instant Queue, or a device-specific category.)

One more thing about experiments: Often, you discover unexpected tidbits that make the big picture just a little more complex. For example, when we tested the Netgear NeoTV and Sony SMP, we noticed that they didn’t remember all of our Netflix activity – just the activity specific to that device. So despite the fact that we’d been playing an episode from season 4 of Mad Men on all the other devices, when we tested these two, we wound up earlier in the series and had to scroll through 20-some episodes.

While our data paint a pretty good picture of how efficient each user interface is, relatively speaking, that doesn’t necessarily correlate with ease of use. Several factors, such as the style of the virtual keyboard, the layout of the buttons, and even the tactile feel of the remote control, have a big impact on usability.

Another factor that greatly affects usability is whether or not a streaming device has predictive search – that is, when you are searching for a title does it display search results as you type, or do you have to put in the whole title and hit “enter” first? When using an onscreen keyboard, one character equals several button presses on the remote control – so searching on a device without predictive search can be a really tedious process.

Fortunately for us, predictive search is a pretty standard feature these days. Six of the seven devices we tested have predictive search, making it possible to just search for “Ma” instead of “Mad Men.” Predictably, the device that doesn’t have this feature was the overall loser in our tests.

Roku
Roku took the shortest overall path to our Mad Men episode – using the “Recently Watched” category it took only 5 button presses to start playing the episode. With respect to search, Roku did the best out of the devices that do not have a full QWERTY keyboard (and it was on par with the Boxee Box, which does.)
—
Remote control: 14 buttons; no QWERTY
Steps to search: 20 (predictive search at 2 characters)
Fastest route to episode: 5 steps

Apple TV
Apple came in just behind Roku in our tests, and with the simplest remote control – 7 buttons total, including the directional pad (D-pad) — it achieves a nice balance between simplicity and functionality.
—
Remote control: 7 buttons; no QWERTY
Steps to search: 26 (predictive search at 2 characters)
Fastest route to episode: 7 steps

Sony Google TV (2nd generation)
While the navigation on Google TV is not quite as efficient as Roku and Apple TV, search is where the Google TV devices really shine. Searching for an episode with the full QWERTY keyboard was only a 12-step process.
—
Remote control: 91 buttons, including QWERTY
Steps to search: 12 (predictive search at 2 characters)
Fastest route to episode: 10 steps

Boxee Box
The Boxee Box proved to be quite clunky in terms of navigation, the biggest problem being that you encounter popups each time you launch or exit an app confirming that you actually want to do so.
These extra button presses add up quickly, as evidenced by the fact that Boxee, even with its full and stylish QWERTY keyboard, takes just as many steps to search for an episode as Roku does.
—
Remote control: 45 buttons, including QWERTY
Steps to search: 20 (predictive search at 2 characters)
Fastest route to episode: 10 steps

WDTV Live
The WDTV Live was on the low-to-average end of the scale when it came to the number of steps to start playing an episode – 27 steps when searching, 11 when playing from Instant Queue.
But what those numbers don’t tell you is that the buttons for “Back,” “Options,” “Prev Page,” “Next Page” are interspersed with the 4 directional buttons (which are not fused into a D-pad as they are on Roku and Apple TV.) As a result, it was way too easy to hit the wrong button (and very tempting to chuck this remote out our 11th-story window.)
Also, it’s worth noting that this remote control has nearly as many buttons as Boxee’s, yet no QWERTY keyboard.
—
Remote control: 37 buttons; no QWERTY
Steps to search: 27 (predictive search at 2 characters)
Fastest route to episode: 11 steps

Netgear NeoTV
Because NeoTV didn’t remember our activity on other devices, it required a lot more steps to get to our desired episode. While this probably won’t affect the experience if this is the only device you typically use for Netflix, it is worth keeping in mind.
—
Remote control: 19 buttons; no QWERTY
Steps to search: 54 (predictive search at 2 characters)
Fastest route to episode: 12 steps

Sony Streaming Media Player (2nd-generation)
As in our initial review, this device was a real stinker. Which is a shame, considering it’s the first one to combine all of the major streaming content providers with file-handling capabilities and DLNA (Digital Living Network Alliance) – meaning you can also play media from your home network in addition to Netflix, Hulu, Amazon, and the like.
The Sony SMP wins the dubious award of “Most Buttons without a QWERTY Keyboard.” And like NeoTV, it doesn’t remember your Netflix activity on other devices. Even worse, it drops you at the beginning of the season, instead of the last episode watched, requiring you to scroll through each time. In terms of search, I’ve mentioned the absurdity of the T9 layout of Sony’s onscreen keyboard before – but the lack of predictive search, something all the other devices have, makes the process downright comical. Really, we laughed the whole time.
—
Remote control: 44 buttons; no QWERTY
Steps to search: 59 (NO predictive search!)
Fastest route to episode: 21 steps

Heading into the holiday shopping season, we often get asked: “Which OTT video box/service should I buy?” Know going in that there are lots of different facets to consider.