Just a couple of weeks ago, callstats.io presented a scientific paper at the International Conference on Quality of Multimedia Experience — QoMEX. Held in Berlin, Germany, the conference celebrated its 10th anniversary this year.
QoE research has a very active community, and the conference brings together researchers from a variety of backgrounds, from media coding and networking to psychology and economics. It is always very exciting to see what's brewing in the different labs working on these topics, and how it can improve our own work at callstats.io.
This year, callstats.io made two contributions to QoMEX, the paper, Fundamental Relationships for Deriving QoE in Systems, and a talk in the "madness session." This session provides a short time to present ideas that are a bit controversial, or "out there," and meant to spark discussion within the community.
The paper deals with some mathematical aspects of modeling QoE for communications systems. The main message is to show the fundamental relationships between QoE (as experienced by users) and system performance metrics, but that in order to fully exploit them (e.g. for improving the monitoring of media services), subjective QoE studies need to produce more than just MOS (Mean Opinion Score) ratings. The paper also refutes some common practices, such as the use of MOS distributions. In contrast to actual user ratings distributions, MOS distributions do not necessarily deliver the expected results.
Our message to the research community is to consider a systems-centric view on QoE that provides an overall view of quality of a service. This view should be built on top of individual QoE estimates for the users of the service being studied. This is, in a way, a dual of the traditional way of building QoE models based on performance (QoS) parameters, and it is fundamentally what we are doing with callstats.io’s new eMOS.
Our madness session presentation sparked a lively discussion about the meaning of Quality of (a) Service, the generalization of QoE to systems encompassing many users, and the semantics of these all. This affects academics and industry practitioners (network operators, service providers) working on QoE management who need to develop meaningful metrics to understand system performance in terms of QoE.
There were a number of very high-quality works presented at QoMEX this year, some of which presented results relevant to our work on WebRTC quality. Highlights include:
Pérez et al, “Subjective Assessment of Adaptive Media Playout for Video Streaming”, which derives some thresholds for stretching/compression of buffer playout times;
Seufert, “Fundamental Advantages of Considering Quality of Experience Distributions over Mean Opinion Scores”, which builds upon and critiques some of our own previous work on QoE metrics. It furthers the notion that using only MOS as a QoE metric is very limiting, in particular for system-level considerations.
Tiotsop et al presented work in a similar vein, on predicting QoE ranges “Computing Quality-of-Experience Ranges for Video Quality Estimation”, which makes user diversity explicit in quality estimations;
Fiedler et al. presented in “Modeling Instantaneous Quality of ExperienceUsing Machine Learning of Model Trees” a simple and useful way to model instantaneous QoE with model trees, which are ML-models built as a tree of simple linear models defined piece-wise across the parameter space. This allows for a good understanding of which parameters affect the quality the most under given conditions.
Seufert et al. presented a heads-on comparison of TCP vs QUIC in “Is QUIC becoming the New TCP? On the Potential Impact of a New Protocol on Networked Multimedia QoE”.
Of course, there were many other interesting works in QoE subfields apart from WebRTC. This year, Virtual Reality, 360-video, and text analysis were quite prominent in the program.
QoMEX helps us stay up to date with new developments in our field and enables us to collaborate with peers. It enables us to stay on the cutting edge, so we can improve our offering, and deliver better value to our customers.