It wasn’t that long ago when anyone looking for Real User Measurement (RUM) had to implement it themselves. Thankfully, that is no longer the case. As the RUM space gets more crowded, I thought it might be valuable to make a list of 10 things you should consider when choosing a RUM provider.
1: Ability to correlate your site speed with revenue
Site speed = $$. One of the most powerful things you can do with RUM is show the correlation between your site speed and your business metrics like your bounce rate, conversion rate and revenue. No RUM product is complete without the ability to go to your boss with a case study showing how much performance matters for your business.
2: 100% sampling rate
Every pageview matters. Processing data from billions of pageviews is hard. Many people don’t want to deal with that much data and choose to sample instead. Google Analytics collects some limited performance data, but their default sampling rate is 1%! Behind every pageview statistic is a story – a story of a potential customer, on the brink of clicking away because your site is too slow! You want a RUM provider that understands that every pageview matters and will track you performance data across 100% of your visitors.
3: Percentiles and histograms
Averages lie. Averages can be very misleading when looking at performance data. I regularly see websites with an average load time of 4 seconds, while 10% of their visitors are experiencing 20 second load times! If you believe every pageview matters, you’ll probably agree that 20 second load times aren’t okay. To really understand what is happening on your site, you need to be analyzing your top percentiles and looking at a histogram breakdown.
Live in the moment. It’s not very helpful if you find our tomorrow that your site was slow today. I recently talked to a top news site that used our realtime feature to catch a bad deploy. They didn’t have synthetic tests set up on the affected page, so Keynote reported that everything was fine. Only because they had our real time graph on a wall monitor were they able to catch the issue right away and revert the slow code.
5: Total browser coverage
6: Drilldown capabilities
Actionable data. Our slowest pages tab not only gives developers a prioritized todo list, but also gives actionable suggestions on what to fix. Want to know how fast your site loads in IE 6? Or compare your site speed in NY to your speed in SF? Drilldown capabilities are crucial for doing in-depth analysis. We’ve actually had customers build new data centers after digging into their performance data and realizing how much even a 1-second speed up was worth to them. RUM is great as an analytics tool. It’s best when it drives actionable decisions.
7: Long term data retention
Commitment. At Torbit, we’ll keep your data safe forever. Believe it or not, some analytics tools actually throw your data away over time. The data gets expensive to store over time, so they just toss it out. But what if you want to compare your site speed today to your site speed a year ago? Personally, I want to know that my data is being stored securely and will be available, no matter how far in the future I end up needing it.
8: Support for A/B testing
Measure then optimize. It’s very powerful to be able to test the performance implications of a new feature with your actual visitors. One of the most common uses of Torbit Insight is to evaluate CDN performance. Wayfair used Torbit Insight to discover that Akamai was not delivering a meaningful improvement to their site speed and shared the results on the Wayfair blog.
9: Affordable price
How about free? At Torbit, we’re really passionate about making the web faster. We believe the first step in achieving our mission is to help people get their hands on accurate performance data. We’ve worked hard to make sure our free plan includes tons of useful functionality. If at some point, you choose to upgrade to a paid plan, great! If not, that’s fine too.
10: Proven team and proven product
Don’t get left behind. The idea of RUM has been around for a while, but adoption has only recently started to pick up. We live in a fast changing environment. I personally believe we have only scratched the surface of what is possible with RUM. Resource Timing is already available in IE 10 and will be rolling out to other browsers soon. Make sure you pick a product built by a proven team that you can trust to stay on the cutting edge as new technology is available.
Torbit is used by hundreds of sites including multi-billion dollar corporations, top retailers and leading media properties. Our Insight product has been battle tested with billions of performance metrics. We started Torbit with an audacious mission and we’re dedicated to seeing it through. We live to make the web faster. In spite of all the improvements in browser technology and faster connection speeds, websites are slower than ever. As we head into 2013, I hope you’ll join us in pursing a faster internet for everyone.