WO2015073470A1 - Prédiction de qualité d'appel - Google Patents

Prédiction de qualité d'appel Download PDF

Info

Publication number
WO2015073470A1
WO2015073470A1 PCT/US2014/065108 US2014065108W WO2015073470A1 WO 2015073470 A1 WO2015073470 A1 WO 2015073470A1 US 2014065108 W US2014065108 W US 2014065108W WO 2015073470 A1 WO2015073470 A1 WO 2015073470A1
Authority
WO
WIPO (PCT)
Prior art keywords
call
user
model
quality
parameters
Prior art date
Application number
PCT/US2014/065108
Other languages
English (en)
Inventor
Mattias Nilsson
Ando SAABAS
Renat Vafin
Markus Vaalgamaa
Adriana Dumitras
Teele TAMME
Andre VESKI
Original Assignee
Microsoft Technology Licensing, Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from GBGB1320216.3A external-priority patent/GB201320216D0/en
Application filed by Microsoft Technology Licensing, Llc filed Critical Microsoft Technology Licensing, Llc
Priority to CN201480062176.8A priority Critical patent/CN105794187B/zh
Priority to EP14805443.0A priority patent/EP3055983B1/fr
Priority to KR1020167015840A priority patent/KR102257261B1/ko
Publication of WO2015073470A1 publication Critical patent/WO2015073470A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/22—Arrangements for supervision, monitoring or testing
    • H04M3/2227—Quality of service monitoring
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/5003—Managing SLA; Interaction between SLA and QoS
    • H04L41/5019—Ensuring fulfilment of SLA
    • H04L41/5025—Ensuring fulfilment of SLA by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/5061—Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the interaction between service providers and their network customers, e.g. customer relationship management
    • H04L41/5067—Customer-centric QoS measurements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/24—Negotiation of communication capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
    • H04L41/508—Network service management, e.g. ensuring proper service fulfilment according to agreements based on type of value added network service under agreement
    • H04L41/509—Network service management, e.g. ensuring proper service fulfilment according to agreements based on type of value added network service under agreement wherein the managed service relates to media content delivery, e.g. audio, video or TV

Definitions

  • a communication system such as a VoIP system implemented over the Internet may be required to serve billions of calling minutes to people around the globe.
  • Users expect a high quality calling experience.
  • Meeting such high expectations depends on the communication system provider's ability to define, analyze, measure, improve and monitor call quality. This involves the ability to understand the impact and frequency of technical conditions (measured by technical parameters) on the user's subjective call experience; for instance an understanding of the network and media characteristics in categories such as transport quality, quality of service (QoS), quality of media (QoM) and quality of experience (QoE).
  • QoS quality of service
  • QoM quality of media
  • QoE quality of experience
  • the simplest methods use basic engineering metrics such as the signal-to-noise ratio (SNR) commonly used in audio and the peak-signal-to-noise ratio (PSNR) used in video. These simple metrics can also be modified to take more account of perceived quality. For example, refinements to PSNR by adaptation to the spatio-temporal complexity of the video have been proposed, resulting in more correlation with human perception.
  • An alternative to PSNR is the structural similarity index (SSIM) which has a higher correlation with subjective quality. Recent work in video coding has aimed at using SSIM as the encoding distortion measure.
  • the objective test methodologies can be divided into three groups based on the inputs provided to the models: full, reduced and no reference models. This taxonomy takes into account whether a model uses the original audio or video signal as a reference for analysis.
  • the original audio or video signal is compared to the processed (or so-called degraded) audio or video signal. Based on the comparison the model predicts the user perceived quality.
  • the reduced reference models use only part of the original signal properties for quality assessment. Examples in this category of models include the standardized Video Quality Metric (VQM) designed for MPEG-2 quality assessment.
  • VQM Video Quality Metric
  • the no-reference models do not use the original audio or video signal to assess the quality. Instead, these models make assumptions about the properties of the original signal. Perhaps the most well-known no-reference model is the E-model (ITU-T G.107) designed for speech quality assessment. There is a recent extension of the E-model (ITU-T G.1070) that includes both video quality (coding, frame-rate, packet- loss, and display- resolution) and a combination of the audio and video quality (delay and sync) into a total quality score.
  • the audio part in G.1070 is a simplified version of the G.107 model. Both of these models were designed to assist the telecom operators in their network infrastructure design so as to guarantee a specific level of quality.
  • the G.107 E-model has been extended from narrow and wideband usage towards super-wideband usage, supporting the modern speech codecs such as Silk. Further refinements of the G.1070 model also take into account the video content, for instance the spatio-temporal complexity.
  • a communication client application for running on a user terminal such as a desktop or laptop computer, tablet or smart phone in order to conduct one or more calls over a network, e.g. a VoIP call over the Internet (which may include video).
  • the communication client is configured to access a model which models quality of user experience for calls based on a set of technical parameters of each call.
  • the technical parameters could be measurements of properties such as echo, noise, packet round-trip time, received bandwidth, resolution or frame rate.
  • the model is hosted on a server (comprising one or more server units across one or more sites), and accessed by the client over the network.
  • the user terminal may be provided with an instance of the model locally, or that the model could be implemented over a peer-to- peer distributed network.
  • the model itself is based on user feedback indicating the subjective quality of multiple past calls as experienced by multiple users, modeled with respect to the technical parameters collected from each of the past calls.
  • the user feedback may take the form of user feedback scores rating the subjective quality of the calls as experienced by the users.
  • the model is able to define an objective metric predicting the quality of user experience, based on an association between objective technical parameters and subjective user feedback scores.
  • the user feedback scores may for example rate the score out of five, e.g. the user giving the call one to five stars.
  • the predicted score may attempt to predict the call quality on this same scale, e.g.
  • the mean opinion score comprising a discrete or continuous value between one and five, or alternatively the predicted score may be output in a different form, e.g. categorising calls into two categories corresponding to "good” and “bad” calls.
  • any scale for user feedback may be used.
  • the model obtains the technical parameters of the call by receiving measurements from the receive side and/or transmit side (and/or if the call is hosted via the server it could perform some or all of the measurements itself). In response, this causes the model to generate a predicted call quality score predicting the quality of user experience for the current call based on the application of the model to the technical parameters of the call in question. Based on this predicted call quality score, the model also determines whether an adaptation of the call could increase the predicted quality of user experience, i.e. by adapting one or more technical parameters of the call. It returns an indication of any determined adaptation to the client application on at least one end of the call, which uses this indication to adapt the one or more of the technical parameters of the call accordingly.
  • the model returns the predicted score itself which the client could use to experiment with adjusting one or more technical parameters (potentially referring back to the model again to see how changes are affecting the predicted quality). Either way, the current call is thus adapted based on the generation of the predicted quality score by the model.
  • the model may be based on the technical parameters and one or more model parameters of the model itself, and the model may be based on a training of the model parameters by a training algorithm.
  • calls may be adapted dynamically, but calls may be used to obtain further user feedback so as to dynamically adapt the model itself. That is, the model may be automatically adapted in a live fashion, as feedback on calls is collected during real-life system operation or "in the field" (as opposed to just being pre-configured based on a set of prior training or test data).
  • a server comprising the model and a network interface arranged to receive further user feedback scores rating the subjective quality of a plurality of further calls from a plurality of users, and to obtain the technical parameters of the further calls.
  • the model is configured to dynamically incorporate the user feedback scores and the technical parameters of the further calls and thereby dynamically adapt the model as the further user feedback scores are received.
  • the communication client may be configured to receive a user feedback score for the adapted call from a user of the user terminal, and to submit this user feedback score to the model where it is incorporated along with the technical parameters of the respective call in order to adapt the model.
  • the model When the communication client then later conducts a second call over the network, the model generates a predicted call quality score predicting the quality of user experience for the second call based on the adapted model as applied to the technical parameters of the second call.
  • the second call can be adapted based on the prediction by the adapted model, in the same way the first call was adapted based on the prediction by the previous version of the model.
  • the process may continue in this way over many calls conducted between many users: i.e. adapting calls, receiving user feedback on the adapted calls to adapt the model, then adapting further calls based on the adapted model, receiving user feedback on the further calls to further adapt the model, then using this version of the model to adapt yet further calls, and so forth; with all this being performed automatically in the field, dynamically in an ongoing fashion during live system operation.
  • Figure 1 is a schematic block diagram of a communication system
  • Figure 2 is a schematic representation of a user's quality of experience
  • Figure 3 is a schematic representation of a user interface for giving user feedback
  • Figure 4 is a schematic representation of clustering technical parameters in a feature space.
  • Figure 5 is a schematic representation of the clustering of technical parameters in feature space, and prediction of call quality from a model based on such clustering.
  • FIG. 1 shows an example of a communication system in accordance with embodiments disclosed herein.
  • the communication system comprises a plurality of user terminals 102 connected to or being operable to connect to a network 100, in embodiments a packet-based network such as the Internet.
  • Each user terminal 102 is used by a respective user 101.
  • Each user terminal 102 is installed with a respective instance of a communication client application 103 for conducting calls with other user terminals over the network, e.g. a VoIP client.
  • a call is a live, real-time communication between two or more user terminals, comprising media content of one or more media types including at least an audio stream, video stream or both. It is also possible that the call includes one or more other types of media such as a screen sharing stream.
  • the communication system further comprises a server 104 connected to the network 100.
  • the server 104 may be owned and/or operated by a provider of the communication system in question, e.g. a VoIP provider, or by a third party provider operating in conjunction with one or more such communication providers.
  • a provider of the communication system in question e.g. a VoIP provider
  • server as used herein does not necessarily limit to a single server unit or a single physical site.
  • the server 104 comprises a network interface 105 arranged for communicating with the client applications 103 of the user terminals 102 over the network 100.
  • the server 104 is also arranged to run a call quality prediction model 106.
  • Each client 103 is operable to access the model 106 via the network 100 and interface 105.
  • an instance of the model 106 may be provided locally on one or more of the user terminals 102, e.g. being incorporated into the client 103 or provided as an accompanying application such as a plug-in.
  • the model 106 is configured to automate call quality monitoring, based on an objective call quality metric which is able to closely approximate the subjective call quality; and to predict the objective call quality using technical parameters and the objective quality metric. It provides an online (during the call) call quality predictor that runs on the server 104 or in the client 103, in order to identify possible technical trade-offs and/or other actionable items during a call (e.g. to trade off video frame rate vs. resolution or to make a microphone selection).
  • an online (during the call) call quality predictor that runs on the server 104 or in the client 103, in order to identify possible technical trade-offs and/or other actionable items during a call (e.g. to trade off video frame rate vs. resolution or to make a microphone selection).
  • QoE quality of experience
  • Figure 2 gives a high-level representation of the relationships between call quality and call reliability components.
  • modeling the QoE may involve modeling the QoE of the media stream(s) (QoM) and the transport quality. This means that, even though call quality and call reliability are often studied separately, call quality in fact depends on the call reliability.
  • the QoS, scenario and media stack determine the QoE of the media streams (QoM).
  • the set of observable network parameters may comprise one-way network delay, packet loss, available bandwidth, and/or conversational delay.
  • the scenario may comprise visual and/or acoustic parameters of the environment from which the call media is captured, and/or the choice of device(s) capturing the call media (such as a web camera, microphone, etc.).
  • QoS is closely connected to the particular use case or application, which in this disclosure comprises real-time audio and/or video communication.
  • transport quality Various observable network parameters also impact the transport quality, these being parameters that reflect the call reliability.
  • transport quality has an impact on the subjective call quality. For instance, transport properties such as call setup time and number of call drops may have an impact on a user's experience of quality.
  • the QoM and transport quality combine to give the overall QoE.
  • the QoM is the QoE of the media and is affected by various media parameters.
  • parameters affecting audio QoM can include parameters relating to the environment or capture, such as: speech level, noise, echo, room reverberation, and/or microphone frequency response.
  • Parameters affecting video QoM can also include parameters relating to the environment (scene) and capture, such as: noise (particularly when capturing in low light), brightness, contrast, colour, and/or parameters measuring properties of the captured video content (e.g. texture, motion, and/or identification of a face in the video versus other objects).
  • Parameters affecting audio QoM can include parameters relating to encoding and decoding such as: audio frequency range (especially if the frequency range is relatively limited compared to the desired content, e.g. speech), processing distortions and muting, coding distortions, concealment distortions (e.g. time-scale speech modifications and/or speech extrapolations), delay, and/or one or more parameters measuring problems with synchronising with the video.
  • Parameters affecting video QoM can also include parameters relating to encoding and decoding, such as: blockiness, ringing, a measurement of false edges, jerkiness, frame rate, freezes (at loss), delay, and/or one or more parameters measuring problems with synchronising with the audio.
  • Parameters affecting audio QoM can include parameters relating to the rendering, such as loudspeaker frequency response, loudness and/or noise.
  • Parameters affecting video QoM can also include parameters relating to rendering, such as display size, resolution and viewing distance.
  • the set of technical parameters describing each call may comprise: one or more parameters relating to an environment from which the call is being captured, one or more parameters relating to capture of call media of the call one or more parameters relating to media content of the call, one or more parameters relating to encoding and/or decoding of the call, one or more parameters relating to quality of service for the call over the network, one or more parameters relating to rendering of call media of the call, and/or one or more parameters relating to processing resources of a transmitting and/or receiving terminal of the call.
  • the set of technical parameters may comprise one or more parameters relating to video of the call, and/or one or more parameters relating to audio of the call.
  • the set of technical parameters may comprise parameters from one or more of the categories: network features (e.g. network delay, bandwidth, network interface type); audio features (e.g. echo, noise, concealment usage); and video features (e.g. resolution, frame rate).
  • network features e.g. network delay, bandwidth, network interface type
  • audio features e.g. echo, noise, concealment usage
  • video features e.g. resolution, frame rate
  • Other possible parameters may include the version of the client, an indication of the codec, and/or transport stream type (which may have possible values such as: UDP, UDP relay, TCP and TCP relay).
  • the set of technical parameters may be chosen based on an assumption that QoE ⁇ QoM. By making this assumption, the model does not explicitly take into account the impact of the transport quality on the QoE. In embodiments, the model may eliminate the impact of the transport quality (to some extent at least) by only using data that corresponds to successfully established calls and calls without drops. However, it is not excluded that in other embodiments the set may comprise one or more parameters relating to transport quality (reliability).
  • a suitable set may be constructed from any combination of some or all such parameters, and/or other parameters.
  • Speaker change rate e.g. speaker change frame percentage (both sides)
  • Echo - e.g. echo delay or echo to near-end ratio (both sides)
  • QP Quantization parameter
  • Audio extrapolation ratio (in loss and jitter % Near-end concealment)
  • Audio interpolation ratio (in loss and jitter % Near-end concealment)
  • Audio timescale modification ratio (in loss % Near-end and jitter concealment)
  • CPU load CPU load by VoIP client application % Near-end can be values sampled throughout the call at any suitable granularity, and/or can be aggregated parameters aggregated over some or all of a call (e.g. being determined at the end of the call). Any of the above parameters may for example be evaluated in terms of any suitable point estimate such as instantaneous value, maximum, minimum, mean, median, mode, variance and/or standard deviation.
  • the provider gathers feedback of multiple past calls.
  • the client application 103 is configured to provide its respective user 101 with a prompt for a user feedback score at some point during or just after a call, and in response to the user entering the feedback into the client 103 it submits this to the model 106.
  • Figure 3 illustrates an example of a graphical prompt for user feedback, in the form of a screen or window 302 of the client 103.
  • the user is prompted to rate the call from one to five stars.
  • the feedback scores as represented in the model 106 may be integer values from the set ⁇ 1, 2, 3, 4, 5 ⁇ .
  • other subjective scoring metrics and scales are also possible.
  • the model 106 collects the user feedback scores from multiple past calls and stores these in a database. It also collects the technical parameters as measured for each respective call, and stores these in the database in association with the respective user feedback score. The role of the model 106 is then to provide an association between the objective technical parameters and the subjective user feedback scores so that, once a sufficiently large number of past calls has been incorporated into the model, then for a subsequent call the subjective quality of the user's experience can be predicted in a quantified manner (with some reasonable degree of accuracy) based on its objective technical parameters.
  • this can be computed with any suitable statistical measure, for example (but not limited to) root-mean-square error, mean absolute error and/or coefficient of determination (for predicting a continuous variable) or rate of correct classification, a receiver operating characteristics (ROC) curve, and/or confusion matrix (for predicting a discrete score).
  • any suitable statistical measure for example (but not limited to) root-mean-square error, mean absolute error and/or coefficient of determination (for predicting a continuous variable) or rate of correct classification, a receiver operating characteristics (ROC) curve, and/or confusion matrix (for predicting a discrete score).
  • the predicted score may be on the same scale as the user feedback scores, e.g. a score from 1 to 5 (potentially with the possibility of non-integer predictions as in the case of MOS, e.g. 3.9, even if the user feedback was from a set of integers).
  • the predicted score could be on a different scale, e.g. a percentage, or categorised into one of only two possible predicted scores intended to be interpreted as "good” and "bad” or "acceptable quality” and "unacceptable quality”.
  • the set of parameters that is measured and stored for each call defines a multi-dimensional feature space.
  • a given call may be described by a feature vector in the feature space, i.e. a vector comprising the values of the set of technical parameters for that call.
  • the model 106 may involve a partitioning of the feature space to make the processing of the data more manageable in terms of complexity. This may be referred to as clustering, in that the sampled parameters are clustered into multi- dimensional clusters or cells 402. Put another way, the feature space is quantized in each of its multiple dimensions.
  • Figure 4 shows only a three- dimensional feature space defined by a set of three technical parameters (frame rate, resolution and bandwidth) but it will be appreciated that the same principle may be applied to any parameter set of any number of dimensions. Note also the cells don't necessarily have to be box shaped (nor multi-dimensional box shaped in any number of dimensions) - this is just an example.
  • the partitions may be created such that calls in a cell 402 are as similar to each other as possible from a technical standpoint, while the cells 402 themselves are as different from each other as possible. In other words, it may be desired to minimize the within-cell variance while maximizing between-cell variance. It may also be desirable to be able to select the number of clusters 402 and use simple metrics to evaluate the quality of the resulting clusters.
  • clustering algorithms Whilst finding an absolutely optimal partitioning for a dataset of a large size and dimensionality may not be practical, there are a number of clustering algorithms with good heuristics for such cluster creation. Some examples of clustering algorithms are outlined below - clustering by Cartesian product, and vector quantization (VQ).
  • Clustering by Cartesian product one approach to partition the feature space is to simply divide individual technical feature ranges into intervals and then create the clusters by combining these intervals. For instance, if the frame rate range is divided into three intervals (e.g. 0-10, 11-20 and 21-30 fps) and the bandwidth range into two intervals (e.g. 0-200 and 200+ kbps), then six subsets (clusters) are obtained by combining the two sets of intervals.
  • VQ vector quantization
  • the k-means algorithm (also called Lloyd's algorithm) is commonly used in data clustering. It is an iterative algorithm, where each iteration consists of first optimizing k partition regions given k centroids, and then optimizing k centroids given k partition regions. The iterations (called Lloyd iterations) continue until a convergence criterion is met.
  • the performance of the algorithm depends on the choice of the initial k centroids.
  • One commonly used variation is referred to as k-means++. In this algorithm, the initial centroids are selected such that they are well spread in the input space.
  • the LBG algorithm which is well known in data compression, is also based on Lloyd iterations.
  • the selection of the initial centroids is different.
  • the algorithm starts with one centroid and then splits it into two. It then applies Lloyd iterations to optimize the two centroids. It then splits the two centroids into four and optimizes the four centroids. The algorithm continues in this fashion until k centroids are obtained and optimized.
  • the model 106 itself is to be chosen for modeling the relationship between the technical parameters of calls (the feature vector) and the predicted call quality score.
  • the model 106 is initially trained based on the multiple past calls used as training data, then deployed for making live predictions about further calls.
  • the model operates based on the feature space as defined and/or quantized in accordance with techniques such as those discussed above.
  • x t E X denote a feature vector defining a technical state in the space of possible technical configurations. Assuming that the feature vector truly represents the technical condition that users are experiencing, each user will map the objective technical QoM to a subjective experience (opinion) score yi, e.g. x £ ⁇ y £ £ [1,2,3,4,5] .
  • the mean opinion score (MOS) may be defined as an objective measure of the same QoM
  • the prediction may be implemented based on the following approximations. Firstly, the measured (objective) user scores of individual calls that exhibit very similar technical parameters will (likely) be very similar. Secondly, the predicted (objective) mean opinion score of a group of calls with very similar technical parameters is taken as representative of the respective group of calls, and is equal to the expected score y ⁇ .
  • the predicted (objective) mean opinion score will give a reasonable prediction on average over a large number of users (even if not for any one individual user).
  • centroid of a cluster n may be denoted by c n and the corresponding Voronoi region by Vn, and the predicted MOS given a similar objective QoM may then be expressed as
  • the performance of a quality predictor can be measured by comparing the mean predicted score (MPS) in each cluster with the actual mean opinion score (MOS) in each cluster (computed as the mean of the user individual scores).
  • the MPS can be expected to match closely with the MOS, as illustrated in Figure 5.
  • the inputs to the model 106 are the feature vectors extracted from the available data set (i.e. the measurements of the set of technical parameters for each call).
  • the output is a predicted opinion score, describing the predicted quality of a call with the given technical conditions. In embodiments this may for example be a continuous value between 1 and 5 (the mean predicted score MPS) or a categorisation into "good” or "bad” call.
  • the relations between the inputs and the output are defined by a mapping function, which depends on a set of model parameters.
  • the function is initially trained based on a set of training data, i.e. using the technical parameters and user feedback scores of multiple past calls to provide training inputs. In training the goal is to find the model parameters that minimize the prediction error on the training data.
  • the model 106 is then deployed into the field for making live predictions about current calls.
  • the model 106 may be hosted on a server 104 accessible by clients 103 over the network 100, or alternatively an instance of the model could be provided in a new release of the clients 103 themselves.
  • mapping function may for example be based on one of the techniques: Random Forest (RF), Linear Regression (LR) or Vector Quantization (VQ).
  • RF Random Forest
  • LR Linear Regression
  • VQ Vector Quantization
  • Random Forest this is an ensemble learning method for classification and regression, which is based on constructing multiple decision trees, and taking their average (or mode in case of classification) as the final prediction. Random Forest is one of the most powerful learning methods, due to its ability to learn non-linear relationships and being robust to features in different scales.
  • the vector bi comprises the unknown model coefficients to be learned.
  • Linear regression can be extended to model non-linear relationships, using arbitrary non- linear functions on individual predictor variables. For example in polynomial regression, the relationship between the dependent and predictor variables is modeled as an ⁇ ⁇ order polynomial. The model is still linear in terms of the unknowns (coefficients to be learned) and can be effectively computed.
  • Vector quantization this is commonly used in data compression. It partitions an input space into regions, and associates a representation point with each region.
  • the representation points are commonly referred to as centroids, since they are chosen to best approximate all points in their respective regions according to a predefined measure.
  • Each point is described by a vector containing N > 1 values, where N is called the vector dimension.
  • a vector quantizer is trained.
  • the vector quantizer comprises partition regions and centroids. It is trained using N-dimensional feature vectors of technical parameters, each of which corresponds to a call in the training set.
  • the MOS of each region (cluster) is computed as the mean value of the subjective user scores assigned to the calls from the training set, which were quantized to the respective centroid of the cluster.
  • each cluster is represented by a single MOS value.
  • the predicted (objective) score for the call is the MOS value that represents the respective cluster. Note that, with this method all calls that are quantized to the same centroid have the same predicted (objective) score value.
  • model 106 Once the model 106 is deployed then a communication client application 103 running on a user terminal 102 of the system is able to access the model 106, invoking it to determine a predicted call quality for a current call currently being conducted by the client 103 over the network 100 (i.e. to make a prediction while the call is still ongoing). As mentioned, in embodiments the model 106 is hosted on a server 104 and accessed via the network 100.
  • the model 106 When it receives an access request from the client 103 of one of the user terminals 102, the model 106 obtains the respective measurements of the set technical parameters (i.e. the feature vector). This may comprise receiving the measurements from the client 103 making the request, i.e. the receiving end; or from a client 103 on the other end of the call, i.e. the transmitting end; or may comprise taking one or more measurements at the server 104 if it is an intermediary of any aspect of the call; or any combination of these possibilities. Note that the call may be described herein as having a transmitting and receiving end.
  • call media may travel in both directions, but for the purposes of the present discussion the receiving end of the call refers to the client 103 or terminal 102 of the user 101 for whom the quality of user experience is to be predicted, and the transmitting end refers to the client 103 or terminal 102 transmitting the call media of which the experience is to be predicted.
  • the model 106 then inputs the measurements of the technical parameters (the feature vector) for the current call into the mapping function of the model, which has already been trained to a working degree by the training data prior to deployment.
  • the model 106 thus outputs a predicted opinion score for the current call based on the input parameters of the current call (the feature vector) and the trained function of the model being applied to those input parameters in order to generate the output.
  • the predicted score could just be returned to the client application 103 to provide information to the user 101.
  • it can alternatively or additionally be used to adapt the current call while it is still ongoing.
  • this may be performed by determining an adaptation at the model 106 as hosted on the server 104, then returning an indication of the adaptation to the client application 103 on the transmitting and/or receiving user terminal 102 where the respective client 103 acts on the indication to adapt the call.
  • the adaptation comprises adapting the value(s) of one or more of the set of technical parameters of the call. For example, it may be determined that for a given bitrate achievable over the network 100, a better quality of user experience can be achieved in the call video by trading off frame rate for frame resolution or vice versa.
  • the server 104 returns an indication of an increase and decrease in these properties respectively, or target values of these, to the transmitting client 103 and/or to the receiving client 103 (which may in turn make a request to the transmitting client) so that the encoded frame rate and resolution can be adjusted accordingly while the call is still going on.
  • the receiving side may also have to adapt to the new parameter(s) of the received stream, e.g. if the transmitter adapts the frame rate and resolution at which it encodes then the receiver will adapt the frame rate and resolution at which it decodes.
  • the model 106 could be implemented locally on each of one or more of the user terminal 102, e.g. incorporated into a version of the client application 103. Similar principles apply in relation to adapting the call, except that no signalling is necessarily required between server 104 and client 103.
  • the model 106 may be implemented to determine the adjustment or target values to indicate to the client 103.
  • the model 106 may examine neighbouring cells (clusters) 402 or cells within a predetermined distance in the feature space of the cell describing the current call, to check whether any would give a higher predicted score - in effect it looks at alternative candidate values of the call's technical parameters where the candidate values extend within a certain range of the current values in one or more directions of the multidimensional quantized feature space. If it finds any values that would increase the predicted score it signals a corresponding adjustment to the relevant client 103 (possibly subject to a minimum threshold to avoid rapid switching or signalling in response to trivial variations).
  • Another possibility would be to try a random selection of alternative, candidate parameter values (alternative cells in the feature space), to check whether any would give a better score.
  • Another alternative would be to implement an analytical or empirically determined algorithm to determine the path or jump to a higher score from any given point in the feature space.
  • the model 106 may be configured to only attempt to find an adaptation on condition that the current predicted score for the call is below a threshold or categorised as "bad", and may stop once an adaptation has been found which raises the predicted quality to an acceptable level.
  • the search for adaptations could be an unconditional and/or continual process that is applied to throughout the call and/or regardless of its current predicted quality.
  • a call may be adapted dynamically based on a model modeling the relationship between objective call parameters and quality of user experience.
  • the model 106 itself need not remain static after the initial training and deployment. Rather, the model 106 can continue to be trained dynamically "in the field" based on ongoing feedback from users.
  • the feedback he or she provides is not just stored for offline analysis by an operator or developer, but automatically incorporated into the live model 106 along with the respective measurements of the technical parameters for the respective call. That is, the new feedback and parameters are incorporated into the training data, so that the training set itself becomes a dynamic entity.
  • a client When a client requests a quality prediction for a second subsequent call, it will then be based on the currently adapted model including the feedback from the first subsequent call, and so forth. Such feedback will be continually submitted from many users across the globe, so that the accuracy of the model 106 continues to grow automatically with use of the system.
  • the dynamically updated model may be accessed for offline prediction or analysis.
  • a developer of a VoIP client may use the most up-to-date model - as currently updated based on the most recent ongoing feedback from real users - in order to analyze the performance of a planned new release of the VoIP client.
  • the dynamically updated model may be used for live monitoring and/or diagnostic applications in testing systems.
  • a monitoring or diagnostic application run by a communication provider may continuously monitor the system and automatically raise an alarm if there is an abnormal decrease of the predicted score.
  • the server 104 does not necessarily comprise one server unit - in embodiments it may be implemented over many server units networked together across many sites. Wherever a server is mentioned herein, one may read "the cloud" as an embodiment of this.
  • the QoE, QoS and QoM previously discussed in relation Figure 2 do not necessarily always give the full picture.
  • there may be more factors - user expectation, mood and/or other non-call related factors (e.g. UI and/or brand) may also play a part. That is, there may be user-related tendencies that contribute to the perceived experience of the user component 101 in Figure 2, or at least to the feedback given by that user 101.
  • some users have different expectations as to what is a "good" call, while there may also be users who purposefully or unknowingly grade calls in ways that are unrelated to their objective quality.
  • individual user opinion scores are noisy due to user bias and random scoring. Therefore, while the process discussed above may give a higher experienced call quality on average across a group of users, predicting individual user opinion scores from technical parameters alone may not always be as accurate as may be desired for any one individual user.
  • User bias refers to the impact of one or more factors on the perceived or reported call quality. Such factors may include for example the user's geographical location (some users are more forgiving than others depending on their geographic areas and/or cultural inheritance), regular or irregular use of the VoIP product, and the participants and content of a call (a call with one's family is typically perceived as having a higher quality than a call with strangers, while a call with good news is perceived as having a higher quality than a breakup). Random scoring refers to the user's selection of a specific call quality score (typically the highest) irrespective of the underlying technical conditions of the call, e.g. in order to discard additional questionnaires about the technical conditions of the call. Note that, despite its name, the user score selection is not strictly random in this case but rather non-causal (as there is no causal relationship between the technical quality of the call and the resulting user score).
  • the model is configured to model the quality of user experience in further dependence on one or more user-related parameters indicative of one or more characteristics of the user in question.
  • the one or more user-related parameters may comprise any of: a country or geographic region selected by the user, a country or geographic region the user terminal is detected to be located in, a language selected by the user, a language of a user interface of the client application, and/or a language of a user interface of the user terminal (e.g. culture or local climate may affect expectations or perceived quality);
  • a brand of the client application and/or a brand of the user terminal (brand may affect a user's perception of the product);
  • an indication of advertisements in the client and/or an indication of advertisements on the user terminal (adverts may annoy or distract users, leading to a different rating); length of call (users making longer calls may have a different perception, expectation or tendencies than users only using the client for short calls);
  • a contact list of the user a length of time the user has participated in a communication system of which the client is a part and over which the modeled calls were made (e.g. the VoIP system in question), a regularity of participation in the system, and/or an amount of prior feedback given by the user (users who participate or engage to a greater degree may have different expectations or tendencies than more casual users); and/or
  • a relation between the user and one or more other participants of the call e.g. a call with friends or relatives may be perceived differently or have different expectations than a call with a stranger or a work colleague).
  • the model may take into account the user's track record of providing specific call quality ratings, and/or may include only users who have used the VoIP system for a period of time (no new users).
  • different countries or cultures tend to grade calls with similar technical characteristics using different average grades, and UI language has been found to be a good proxy for a users' background and culture. Similar reasoning can be applied for the contact list size - users who are more active tend to rate calls better and the contact list size is a relatively good proxy for measuring engagement.
  • Information such as language, contact list length, age or type of participants may be determined for example from settings of the client and/or a profile of the user within the communication system.
  • Embodiments for accommodating user bias or "random" scoring may be implemented for example by creating separate functions within the model based on separate sets of training data, partitioned according to the one or more user-related parameters, e.g. with different versions of the modeled relationship for different countries or language on the basis that users from different cultures may have different expectations, and/or different versions of the modeled relationship for different lengths of contact list on the basis that users more involved in the communication system tend to score differently than less involved users. That is, the feature space is partitioned into different sub-spaces for a different user characteristics or combinations of such characteristics.
  • a client application 103 submits its technical parameters to the model 106, it queries only the function or sub-space within the model that relates to its own user characteristics, e.g. only the version of the relationship that applies to the same country, region, language, age range and/or contact list length, etc.
  • the predicted opinion score and call adaptation are then determined based on the submitted technical parameters as applied to that part of the model.
  • the model may be configured to take into account all the technical and non-technical parameters together synergistically, including the user-related parameters in the model together with the technical parameters. I.e. the user-related parameters are included as one or more extra dimensions of the multi-dimensional feature space along with the technical parameters.
  • the user-related parameters are input into the model and treated no differently than the technical parameters.
  • a client application 103 submits its technical parameters to the model 106, it also submits the one or more user-related parameters indicative of one or more characteristics of the user 101 of the client application.
  • the predicted opinion score and call adaptation are then determined based on both the technical and user-related parameters being applied to the model with the user-related parameters being treated in the same way as the technical parameters alongside the technical parameters (except that the user-related parameters cannot be adapted as a result).
  • the user feedback takes the form of explicit user feedback scores rating the subjective quality of the calls, i.e. scores submitted explicitly by the users themselves such as opinion scores on a scale of one to five stars.
  • an alternative would be to replace the feedback of scores with feedback of one or more other metrics correlated with call quality.
  • An example would be call duration - poor quality tends to imply shorter calls in general because, whereas better quality calls tend to go on longer.
  • the user may give up on a very bad quality call, and be disinclined to conduct a poor quality call for very long, whilst a good quality call may inspire the user to stay on the line for longer.
  • Such a metric can be used to provide implicit feedback on the quality of a call.
  • one or more other user quality perception related metrics can be used so that the users are not (necessarily) required to manually input a rating.
  • the entire model is estimated based on the user feedback.
  • part of the model is based on user feedback modeling, while another part may be based on other methods such as a hand-tuned function or table, literature, and/or existing objective quality tests or tools.
  • the model may additionally take into account a-priori parameters of the network and/or receiving terminal in order to set initial parameters of the call. That is, the model may comprise additional functionality modeling call quality based on parameters known or assumed about the network or terminal(s) before the call begins, e.g. one or more potential or theoretical network QoS parameters, and/or one or more potential or theoretical CPU resources of the terminal(s).
  • the client on the receiving terminal can access this function in order to obtain an estimate of one or more initial parameters to begin the call with, e.g. to estimate an initial resolution and/or frame rate.
  • any of the functions described herein can be implemented using software, firmware, hardware (e.g. fixed logic circuitry), or a combination of these implementations.
  • the terms “module,” “functionality,” “component” and “logic” as used herein generally represent software, firmware, hardware, or a combination thereof.
  • the module, functionality, or logic represents program code that performs specified tasks when executed on a processor (e.g. CPU or CPUs).
  • the program code can be stored in one or more computer readable memory devices.
  • the terminals and/or server may include an entity (e.g. software) that causes hardware of the terminals to perform operations, e.g., processors functional blocks, and so on.
  • the terminals and/or server may include a computer- readable medium that may be configured to maintain instructions that cause the terminals, and more particularly the operating system and associated hardware of the terminals to perform operations.
  • the instructions function to configure the operating system and associated hardware to perform the operations and in this way result in transformation of the operating system and associated hardware to perform functions.
  • the instructions may be provided by the computer-readable medium to the terminals and/or server through a variety of different configurations.
  • One such configuration of a computer-readable medium is signal bearing medium and thus is configured to transmit the instructions (e.g. as a carrier wave) to the computing device, such as via a network.
  • the computer-readable medium may also be configured as a computer-readable storage medium and thus is not a signal bearing medium. Examples of a computer-readable storage medium include a random-access memory (RAM), read-only memory (ROM), an optical disc, flash memory, hard disk memory, and other memory devices that may us magnetic, optical, and other techniques to store instructions and other data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Multimedia (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Quality & Reliability (AREA)
  • Computer Security & Cryptography (AREA)
  • Telephonic Communication Services (AREA)

Abstract

La présente invention concerne une application client de communication destinée à être exécutée sur un terminal utilisateur pour effectuer des appels sur un réseau. Le client est configuré pour accéder à un modèle qui modélise la qualité de l'expérience utilisateur pour des appels sur la base d'un ensemble de paramètres techniques de chaque appel. Le modèle lui-même est basé sur un retour d'informations utilisateur indiquant la qualité subjective de multiples appels du passé telle que ressentie par de multiples utilisateurs, et modélisé suivant les paramètres techniques récoltés à partir de chacun des appels du passé. Le modèle génère un score de qualité prédite d'appel qui prédit la qualité de l'expérience utilisateur pour l'appel étant donnés ses paramètres d'appel. Sur la base de ce procédé, au moins un des paramètres techniques de l'appel peut être adapté pour essayer d'augmenter la qualité ressentie par l'utilisateur.
PCT/US2014/065108 2013-11-15 2014-11-12 Prédiction de qualité d'appel WO2015073470A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201480062176.8A CN105794187B (zh) 2013-11-15 2014-11-12 预测通话品质
EP14805443.0A EP3055983B1 (fr) 2013-11-15 2014-11-12 Prédiction de qualité d'appel
KR1020167015840A KR102257261B1 (ko) 2013-11-15 2014-11-12 콜 품질의 예측

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
GB1320216.3 2013-11-15
GBGB1320216.3A GB201320216D0 (en) 2013-11-15 2013-11-15 Predicting call quality
US14/199,752 US9558451B2 (en) 2013-11-15 2014-03-06 Adapting parameters of a call in progress with a model that predicts call quality
US14/199,752 2014-03-06

Publications (1)

Publication Number Publication Date
WO2015073470A1 true WO2015073470A1 (fr) 2015-05-21

Family

ID=51999551

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/065108 WO2015073470A1 (fr) 2013-11-15 2014-11-12 Prédiction de qualité d'appel

Country Status (1)

Country Link
WO (1) WO2015073470A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017152932A1 (fr) * 2016-03-07 2017-09-14 Telefonaktiebolaget Lm Ericsson (Publ) Procédé et nœud de notation pour l'estimation de la qualité de confort d'un utilisateur pour un service fourni

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148525A1 (en) * 2010-05-14 2013-06-13 Telefonica, S.A. Method for calculating perception of the user experience of the quality of monitored integrated telecommunications operator services

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130148525A1 (en) * 2010-05-14 2013-06-13 Telefonica, S.A. Method for calculating perception of the user experience of the quality of monitored integrated telecommunications operator services

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017152932A1 (fr) * 2016-03-07 2017-09-14 Telefonaktiebolaget Lm Ericsson (Publ) Procédé et nœud de notation pour l'estimation de la qualité de confort d'un utilisateur pour un service fourni
US20180091386A1 (en) * 2016-03-07 2018-03-29 Telefonaktiebolaget Lm Ericsson (Publ) Method and scoring node for estimating a user's quality of experience for a delivered service
US10666529B2 (en) 2016-03-07 2020-05-26 Telefonaktiebolaget Lm Ericsson (Publ) Method and scoring node for estimating a user's quality of experience for a delivered service
US11178024B2 (en) 2016-03-07 2021-11-16 Telefonaktiebolaget Lm Ericsson (Publ) Method and scoring node for estimating a user's quality of experience for a delivered service
US20220030048A1 (en) * 2016-03-07 2022-01-27 Telefonaktiebolaget Lm Ericsson (Publ) Method and scoring node for estimating a user's quality of experience for a delivered service
US11689433B2 (en) 2016-03-07 2023-06-27 Telefonaktiebolaget Lm Ericsson (Publ) Method and scoring node for estimating a user's quality of experience for a delivered service

Similar Documents

Publication Publication Date Title
EP3055983B1 (fr) Prédiction de qualité d'appel
Tsolkas et al. A survey on parametric QoE estimation for popular services
Charonyktakis et al. On user-centric modular QoE prediction for VoIP based on machine-learning algorithms
Seufert et al. “To pool or not to pool”: A comparison of temporal pooling methods for HTTP adaptive video streaming
US20130290525A1 (en) Service Performance in Communications Network
Tran et al. A multi-factor QoE model for adaptive streaming over mobile networks
Liotou et al. A roadmap on QoE metrics and models
Siris et al. Mobile quality of experience: Recent advances and challenges
CN108390775B (zh) 一种基于spice的用户体验质量评价方法及系统
Aguiar et al. Video quality estimator for wireless mesh networks
CN112752117B (zh) 一种视频缓存方法、装置、设备及存储介质
Wang et al. QoE Management in Wireless Networks
Rehman et al. Perceptual experience of time-varying video quality
Dinaki et al. Forecasting video QoE with deep learning from multivariate time-series
Pal et al. A No‐Reference Modular Video Quality Prediction Model for H. 265/HEVC and VP9 Codecs on a Mobile Device
EP3073736B1 (fr) Procédé et dispositif de mesure de qualité d'expérience de service vidéo mobile
Robitza et al. Measuring YouTube QoE with ITU-T P. 1203 under constrained bandwidth conditions
Suárez et al. Assessing the qoe in video services over lossy networks
Abar et al. Quality of experience prediction model for video streaming in SDN networks
Jelassi et al. A study of artificial speech quality assessors of VoIP calls subject to limited bursty packet losses
WO2015073470A1 (fr) Prédiction de qualité d'appel
Mustafa et al. Perceptual quality assessment of video using machine learning algorithm
Möller et al. Modelling speech service quality: From conversational phases to communication quality and service quality
Tang et al. A survey study on QoE perspective of mobile cloud computing
Baskaran et al. Dominant speaker detection in multipoint video communication using Markov chain with non-linear weights and dynamic transition window

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14805443

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2014805443

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2014805443

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20167015840

Country of ref document: KR

Kind code of ref document: A