Last week, Netflix rolled out a new layout for their website that did away many elements, including movie titles, rating stars, and manual scrolling. If the comments on the announcement are any indication, there has been an incredible backlash from the users. I have a few complaints myself. But this article isn’t about the user interface changes, and it’s not a gripe session. This is what we call a “teachable moment.”
Entertainment Weekly reported the following statement from Netflix:
Steve Swasey, vice president of corporate communicates at Netflix, says he’s well familiar with the user backlash. “Most of [the comments] have the same theme, which is ‘what was wrong with before?’ What we’re saying is we’ve tested this extensively, we know the vast, vast majority of people like this. It’s new, it’s easier, it’s cleaner.” When questioned about whether Netflix is planning on keeping the new interface despite the negative response, Swasey said, “Absolutely. We made it and tested it and researched it and tried it out and everywhere we tried it, it had a better reception. Otherwise we wouldn’t have made the change.”
My interpretation of this comment is that Netflix is standing by their usability testing and going forward with the rollout despite the backlash. There is some justification for this position. When Facebook introduced the News Feed, the complaints could be heard from space. Each iteration of Facebook’s layout resulted in a gaggle of groups full of users threatening to leave. But after a few days, the noise died down and people settled back into their routines. Netflix could have looked at these past examples and expected the inevitable push-back to disappear in short order.
But it’s fundamentally naïve to stand by user testing as the sole indicator of a design’s success or failure. As xkcd reminds us, even a 95% confidence interval is wrong 5% of the time, and you don’t need to watch too many episodes of Mythbusters to realize that small-scale testing doesn’t always translate to the real world. Many of the comments on the Netflix blog raise some legitimate usability issues. Swasey’s statement shows a general dismissal of these users because the testing showed it was fine. Even Facebook eventually listened to user concerns over privacy.
Usability testing is a fantastic way to find early interface issues and make sure that your users have a positive interaction with you. But it’s not an excuse to put blinders on. Of course there will always be resistance to change, but don’t assume that all negative feedback is just kvetching. If you have a failure like this, you’re better off with a mea culpa and a designer’s bruised ego than you are with weeks of bad press. Netflix should have said, ”We hear you and we’re working to improve it,” instead of “We know what we’re doing; deal with it.” A successful trial is no reason to commit 100% to a plan. To quote Conan O’Brien, “Don’t be afraid to fail.” At any stage of the game.