Comparing testRTC and callstats.io – How They Fit Into the WebRTC Application Lifecycle

By Carl Blume on June 20, 2019
read

Every once in a while someone asks me how callstats.io compares to testRTC. The answer is simple. They are really two different products, designed for two different audiences and two different purposes. In fact, they are quite complementary.

callstats.io is primarily intended to help operations teams monitor and analyze live WebRTC voice and video quality in production networks. testRTC is primarily intended to help developers test the scalability and performance of WebRTC-based applications in lab environments. Each targets a different stage in the WebRTC application lifecycle.

While testRTC includes monitoring of WebRTC service deployments among its use cases, their approach and capabilities are fundamentally different than ours. Only callstats.io lets you monitor live WebRTC calls and evaluate real end-user experiences.

In this blog, I review the differences between callstats.io and testRTC, especially when it comes to WebRTC monitoring and analytics.

callstats.io is Optimized for Production Operations

callstats.io is built from the ground up to help operations teams monitor and analyze WebRTC service quality in production environments. Enterprises and service providers (e.g. CCaaS, UCaaS, CPaaS providers) use callstats.io to evaluate audio and video quality, gauge user experiences and troubleshoot problems in live networks. The product helps companies improve customer satisfaction and boost business results.

callstats.io is designed with operations teams in mind. It is simple to deploy and simple to use. You can implement callstats.io in a matter of minutes, without writing custom code or complicated scripts. A concise dashboard makes it easy to assess media quality, and identify and isolate problems in production networks (as well as in development and staging environments). Application developers can embed rich monitoring and testing functionality directly into WebRTC endpoints using simple javascript APIs.

testRTC is Optimized for Application Development

testRTC helps developers test the functionality, performance and scalability of WebRTC-based applications. It is a cloud-based service that exercises the customer’s application by simulating WebRTC traffic at varying load levels. QA teams use testRTC to simulate real-world scenarios and evaluate product quality throughout the application development lifecycle.

testRTC is designed to be used with application testing frameworks and browser automation tools like Nightwatch and Selenium. Application developers and test engineers write scripts or custom code to automate WebRTC regression tests, run simulations, and execute stress and sizing tests.

Only callstats.io Monitors Live Calls in Production Deployments

While testRTC is primarily used to test applications under development, you can also use it to run simulations in live networks - if you don’t mind mixing live traffic with load tests. Unlike callstats.io, you cannot use testRTC to monitor live calls or evaluate actual user experiences. Instead testRTC lets you simulate users and generate audio and video traffic using test automation scripts. You can kick-off scripts periodically on geographically distributed browsers in their cloud to confirm service availability.

By contrast, callstats.io lets you monitor live calls and evaluate actual user experiences using a combination of passive endpoint monitoring and smart connectivity testing. We capture key call performance metrics (throughput, packet loss, round-trip time, etc.) from user’s endpoints, storing data in a central repository for analysis and forensics. You can also add visual indicators to WebRTC applications so contact center agents or other users can gauge network conditions before placing or answering calls.

callstats.io provides full visibility into WebRTC sessions, helping you diagnose a wide range of problems, such as network performance constraints, infrastructure and firewall traversal (STUN/TURN/ICE) issues, endpoint misconfigurations, and signaling and media protocol incompatibilities.

Our AI-driven algorithms automatically identify root causes, helping you save time and effort, and avoid guesswork. And our Objective Quality metric makes it easy to gauge call quality from the user’s perspective. We also offer an Amazon Connect integration to help businesses optimize voice quality and deliver better user experiences in Amazon Connect contact center implementations.

Active Network Testing for Different Purposes

The callstats.io Smart Connectivity Test has some similarity with testRTC in that it generates simulated WebRTC traffic, but its purpose is entirely different. SCT runs in the user’s browser and is designed to measure the quality of the user’s local network connection. It doesn’t stress the WebRTC application, as testRTC is intended to do.

The callstats.io SCT can also be used to evaluate the performance of a network connecting a specific geographic site to the Internet. It can be incorporated into a site survey, which is when an enterprise evaluates the infrastructure in a specific location for hosting a contact center.

The table below summarizes the key features and capabilities of callstats.io and testRTC.

 

  callstats.io testRTC
General    
Primary audience Operations teams App dev, test and QA teams
Primary function Monitor and analyze WebRTC service quality Evaluate WebRTC app performance and scalability
Primary use case Production environments Dev and test environments
Key features    
Monitor live calls Yes No
Gather KPIs from real users Yes No
eMOS scores Yes No
AI-driven analysis Yes No
Visual indicators for end-users Yes No
Site survey Yes No
Generate WebRTRC test traffic Yes (from deployment sites) Yes (from the cloud)
Automated stress/sizing testing No Yes
Automated functional testing No Yes

The Devil is in the Details

At first blush testRTC and callstats.io may seem similar. But take a closer look and you’ll see they are actually complementary solutions that target completely different phases of the WebRTC application lifecycle. testRTC lets developers evaluate the performance and scalability of WebRTC applications. It is primarily used during application development. In contrast, callstats.io lets operations teams monitor and analyze WebRTC service quality. It is primarily used in production deployments.

callstats.io is the first and only solution that lets you monitor live WebRTC calls and evaluate real end-user experiences in production networks.