JP4327685B2 - User experience quality monitoring method, user experience quality monitoring apparatus, estimation model generation method, and estimation model generation apparatus - Google Patents

Description

  The present invention relates to a monitoring method and a monitoring apparatus that use an estimation model when a real-time application is monitored by a passive measurement for each call on a packet network, and a generation method and a generation apparatus for the estimation model.

  For real-time applications such as IP (Internet Protocol) telephone and video distribution, the quality of the packet network such as the IP network is not guaranteed, so the quality management technology of the application is attracting attention, and a method for estimating the user's quality of experience has been developed. Has been. For example, Patent Document 1 discloses a subjective quality estimation technique for voice of a real-time application such as VoIP in a packet network. A network management method using this technique is disclosed in Patent Document 2. In recent years, such quality control technology uses statistical information such as the amount of data passing through core nodes and edge nodes such as routers, which are devices that hold relays on a large-scale network, and edge nodes, and the line usage rate. In addition to managing quality, a method for monitoring an application for each call by passive measurement has been developed (for example, see Non-Patent Documents 1 to 3).

The applicant has not yet found prior art documents related to the present invention by the time of filing other than the prior art documents specified by the prior art document information described in this specification.
JP 2002-064539 A JP 2004-023594 A ETSI TIPHON TS101 329-5 Annex E, “Method for determinig an equipment impairment factor using passive monitoring,” Nov. 2000. A. Clark, “Modeling the effects of burst packet loss and recency on subjective voice quality,” in IP Telephony Workshop 2001, Apr. 2001. Timur Friedman, Ramon Caceres, and Alan Clark, “Rtp control protcol extended reports (rtcp xr),” in IETF RFC3611, 2003. Telchemy Inc., “Vqmon,” [on line], [searched July 7, 2004], Internet <URL: http://www.telchemy.com/.>

However, the conventional method is not a technique that can take into account the mounting characteristics of the application device, and is not an estimation technique with high estimation accuracy of the user experience quality, that is, the subjective quality. In addition, ITU-TP. Although there are objective evaluation methods that can perform quality estimation that considers the implementation characteristics of the equipment, such as the PSQM algorithm recommended in 862, these techniques are methods for performing active measurement, and quality is determined for each call. It is not a technology that can be monitored, nor is it a technology that can feed back subjective quality estimation results to network quality design and control.
For this reason, there has been a demand for a technique that can estimate the subjective quality of a user's application when the real-time application is constantly monitored by passive measurement for each call in consideration of the mounting characteristics of the application device.
Therefore, the present invention has been made to solve the above-described problems, and a user experience quality monitoring method, which can monitor the subjective quality of a user's application in consideration of the mounting characteristics of the application device, the user It is an object of the present invention to provide a sensory quality monitoring apparatus, an estimation model generation method, and an estimation model generation apparatus.

In order to solve the above-described problems, a user experience quality monitoring method according to the present invention is a monitoring method for monitoring the user experience quality of an application executed in a packet network by passive measurement for each call, and the application user Generation step for generating an estimation model for estimating the quality of experience in consideration of the implementation characteristics of the application, and estimation for estimating the user experience quality of the application based on the estimation model and application quality information including network quality information measured by passive measurement a step, have a monitoring step of continuously monitoring the QoE estimated by the estimating step, generating step includes a calculation step of calculating the quality evaluation value for each quality elements of the application, the network measured by passive measurement quality An estimation model is generated based on an acquisition step for acquiring application quality information including information, a conversion step for converting the application quality information into basic information that is a basis for calculating the quality evaluation value, and the quality evaluation value and the basic information. A model generation step, and a mounting characteristic consideration step for converting the basic information into quality information corresponding to the mounting characteristic of the application based on the relationship between the quality evaluation value and the basic information. An estimation model is generated based on the quality information . Here, the application quality information may include network quality information and application terminal quality information.

In the user experience quality monitoring method, the quality factor of the application includes any one of listening quality and mouth-to-ear delay, viewing quality and mouth-to-ear delay, and the model generation step includes all of the calculated quality factors. An estimation model may be generated based on a value obtained by integrating quality evaluation values.
Here, the listening quality may include the meaning of sound quality. The viewing quality may include image quality.

  In the user experience quality monitoring method, the quality factor of the application is either listening quality or viewing quality, and the calculation step is performed for each application to be monitored, either subjective or objective quality for the quality factor. An evaluation value may be calculated, and the acquisition step may acquire application quality information for each application to be monitored.

In the user experience quality monitoring method, the quality factor of the application includes a mout-to-ear delay, and the calculation step includes a quality including a one-way transfer delay which is a time until the packet reaches the receiving terminal from the transmitting terminal of the application. An evaluation value is calculated for each application to be monitored, and the obtaining step obtains application quality information for each application to be monitored, and a delay estimating step for estimating a mout-to-ear delay from the basic information; a subtraction step of calculating a first value obtained by subtracting the one-way transfer delay from the to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated south-to-ear delay estimated by the delay estimation step. In addition, the model generation step calculates the first value from the second value. To, the implementation characteristics may be generated an estimation model considering.

A user experience quality monitoring apparatus according to the present invention is a monitoring apparatus that monitors the user experience quality of an application executed in a packet network by passive measurement for each call, and determines the user experience quality of an application according to the implementation characteristics of the application. Generation means for generating an estimation model to be estimated in consideration, estimation means for estimating the user experience quality of the application based on application quality information including network quality information measured by the estimation model and passive measurement, and estimation by this estimation means is to have a monitoring means for monitoring continuously the QoE, generation means, a calculation means for calculating the quality evaluation value for each quality elements of the application, the application quality information including network quality information measured by passive measurement Get hands to get Conversion means for converting application quality information into basic information that is the basis for calculating quality evaluation values, model generation means for generating an estimation model based on the quality evaluation values and basic information, quality evaluation values and basic information Mounting characteristic consideration means for converting basic information into quality information corresponding to the mounting characteristics of the application based on the relationship between the model information and the model generation means generates an estimated model based on the quality evaluation value and the quality information It is characterized by that.

  In the user experience quality monitoring apparatus, the quality factor of the application is any one of listening quality and mouth-to-ear delay, viewing quality and mouth-to-ear delay, and the model generation means An estimation model may be generated based on a value obtained by integrating quality evaluation values.

  In the above user experience quality monitoring device, the quality factor of the application is either listening quality or viewing quality, and the calculation means calculates the quality of either subjective or objective for the quality factor for each application to be monitored. The evaluation value may be calculated, and the acquisition unit may acquire application quality information for each application to be monitored.

In the user experience quality monitoring apparatus, the quality factor of the application includes a mout-to-ear delay, and the calculation means includes a quality including a one-way transfer delay that is a time until the packet reaches the receiving terminal from the transmitting terminal of the application. The evaluation value is calculated for each application to be monitored, and the acquisition unit acquires the application quality information for each application to be monitored, and delay estimation unit for estimating the south-to-ear delay from the basic information; subtracting means for calculating a first value obtained by subtracting the one-way transfer delay from the to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated south-to-ear delay estimated by the delay estimation step. further comprising, model generation means, from the second value to calculate a first value, the implementation characteristics considered It may generate an estimated model.

An estimation model generation method according to the present invention is an estimation model generation method for estimating the user experience quality of an application executed in a packet network in consideration of the implementation characteristics of the application. a calculation step for calculating a quality evaluation value for each quality element of the application including at least one of the mout-to-ear delays, an acquisition step for acquiring application quality information including network quality information measured by passive measurement, and an application A conversion step that converts quality information into basic information that is the basis for calculating quality evaluation values, a model generation step that generates an estimation model based on the quality evaluation values and basic information, and the relationship between the quality evaluation values and basic information Based on application implementation characteristics based on basic information And a mounting characteristics considering the step of converting into quality information, the model generating step, and generating an estimation model based on the quality evaluation value and quality information.

  In the above estimation model generation method, the quality factor of the application is either listening quality or viewing quality, and the calculation step is performed for each of the monitoring target applications, either subjective or objective quality. An evaluation value may be calculated, and the acquisition step may acquire application quality information for each application to be monitored.

In the above estimation model generation method, the quality factor of the application includes a mouth-to-ear delay, and the calculation step includes a quality including a one-way transfer delay which is a time until the packet reaches the receiving terminal from the transmitting terminal of the application. An evaluation value is calculated for each application to be monitored, and the obtaining step obtains application quality information for each application to be monitored, and a delay estimating step for estimating a mout-to-ear delay from the basic information; a subtraction step of calculating a first value obtained by subtracting the one-way transfer delay from the to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated south-to-ear delay estimated by the delay estimation step. In addition, the model generation step calculates the first value from the second value. That may be generate estimation model considering the implementation characteristics.

An estimation model generation apparatus according to the present invention is an estimation model generation apparatus that estimates a user experience quality of an application executed in a packet network in consideration of an implementation characteristic of the application. a calculation unit that calculates a quality evaluation value for each quality element of the application including at least one of a mout-to-ear delay, an acquisition unit that acquires application quality information including network quality information measured by passive measurement, and an application Relationship between quality evaluation values and basic information, conversion means for converting quality information into basic information that is the basis for calculating quality evaluation values, model generation means for generating an estimation model based on quality evaluation values and basic information To convert basic information into quality information according to application implementation characteristics And a mounting characteristics considered means that, the model generation means, and generating an estimation model based on the quality evaluation value and quality information.

  In the estimation model generation apparatus, the quality factor of the application is one of listening quality and viewing quality, and the calculation means calculates the quality of either subjective or objective for the quality factor for each application to be monitored. The evaluation value may be calculated, and the acquisition unit may acquire application quality information for each application to be monitored.

In the estimation model generation apparatus, the quality factor of the application includes a mout-to-ear delay, and the calculation means includes a quality including a one-way transfer delay that is a time until the packet reaches the reception terminal from the transmission terminal of the application. The evaluation value is calculated for each application to be monitored, and the acquisition unit acquires the application quality information for each application to be monitored, and delay estimation unit for estimating the south-to-ear delay from the basic information; subtracting means for calculating a first value obtained by subtracting the one-way transfer delay from the to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated south-to-ear delay estimated by the delay estimation step. further comprising, model generation means, from the second value to calculate a first value, taking into account the implementation characteristics It may generate the estimated model.

  According to the present invention, it is possible to estimate subjective quality in consideration of application mounting characteristics by generating an estimation model in consideration of application mounting characteristics. Further, according to the present invention, by using the estimation model, it is possible to constantly monitor the user experience quality of the application by passive measurement for each call. Furthermore, according to the present invention, the subjective quality can be estimated using the existing quality measurement / management technology.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
First, an outline of the present invention will be described. FIG. 1 is a diagram illustrating a configuration of a quality monitoring system based on passive measurement of a conventional IP telephone, and FIG. 2 is a diagram in which the present invention is applied to the quality monitoring system of FIG.
As shown in FIG. 1, the conventional quality monitoring system includes a network quality measuring unit 101, an application terminal quality measuring unit 102, a terminal setting information acquiring unit 103, an intermediate parameter estimating unit 104, a listening quality / Mouth-to. -It has the ear delay estimation part 105 and the comprehensive conversation quality estimation part 106.

The network quality measurement unit 101 measures performance information for each packet such as a packet loss amount, a delay amount, and jitter from the network, and transmits the performance information to the intermediate parameter estimation unit 104.
The application terminal setting information acquisition unit 102 measures performance information of the application terminal such as the echo cancellation amount and the S / N ratio from the application terminal, and transmits it to the intermediate parameter estimation unit 104.
The terminal setting information acquisition unit 103 acquires application terminal setting information such as an encoding type and a buffer size, and transmits it to the intermediate parameter estimation unit 104.

The intermediate parameter estimation unit 104 includes an intermediate parameter quality factor generation unit 104a and an analog quality factor generation unit 104b.
Based on the information received from the network quality measurement unit 101, the application terminal quality measurement unit 102, and the terminal setting information acquisition unit 103, the intermediate parameter factor generation unit 104a has a loss, a one-way transfer delay, a packet discard amount, a jitter loss amount, a buffer Intermediate parameter quality factors such as a delay amount and an echo delay amount are generated and transmitted to the listening quality / Mouth-to-ear delay estimation unit 105.
The analog quality factor generation unit 104b generates analog quality factors such as echo cancellation amount and S / N ratio based on information received from the network quality measurement unit 101, the application terminal quality measurement unit 102, and the terminal setting information acquisition unit 103. And transmitted to the general conversation quality estimation unit 105.

The listening quality / Mouth-to-ear delay estimation unit 105 includes a South-to-ear delay estimation unit 105a and a listening quality estimation unit 105b.
The Mouth-to-ear delay estimation unit 105 a estimates the Mouth-to-ear delay based on the intermediate parameter quality factor received from the intermediate parameter estimation unit 104, and transmits this estimated value to the total conversation quality estimation unit 106.
The listening quality estimation unit 105 b estimates the listening quality based on the intermediate parameter quality factor received from the intermediate parameter estimation unit 104, and transmits this estimated value to the total conversation quality estimation unit 106.

  The overall conversation quality estimation unit 106 estimates the overall conversation quality based on the information received from the intermediate parameter estimation unit 104 and the listening quality / Mouth-to-ear delay estimation unit 105.

In the quality monitoring system described above, various products have been developed for the intermediate parameter estimation unit 104, the listening quality / Mouth-to-ear delay estimation unit 105, and the overall conversation quality estimation unit 106. For example, the intermediate parameter estimation unit 104 and the listening quality / Mouth-to-ear delay estimation unit 105 are disclosed in the cited document 4. Further, the general conversation quality estimation unit 106 has an ITU-T recommendation G.264. 107E-model corresponds.
In addition, the system indicated by reference numeral 100 in FIG. 1, that is, a system including the network quality measurement unit 101, the terminal setting information acquisition unit 103, the intermediate parameter estimation unit 104, and the listening quality / Mouth-to-ear delay estimation unit 105, This corresponds to the system disclosed in cited document 4.

In the quality monitoring system described above, quality estimation considering the mounting characteristics of the application cannot be performed. Therefore, in the present invention, it is possible to estimate the quality in consideration of the mounting characteristics of the application by adopting the configuration shown in FIG.
The quality monitoring system shown in FIG. 2 is further provided with estimation units 201 and 202, a mout-to-ear delay value storage unit 203, and a PESQ value storage unit 204. Therefore, the same name and code | symbol are attached | subjected to the structure equivalent to the quality monitoring system shown in FIG. 1, and description is abbreviate | omitted suitably.

In the quality monitoring system shown in FIG. 2, the intermediate parameter quality factor generation unit 104a of the intermediate parameter estimation unit 104 includes a mouth-to-ear delay quality information generation unit 104c and a media quality information generation unit 104d.
The mouth-to-ear delay quality information generation unit 104 c generates quality information necessary for Mouth-to-ear delay estimation and transmits the quality information to the estimation unit 201.
The media quality information generation unit 104 d generates quality information necessary for media quality estimation and transmits the quality information to the estimation unit 202.

The estimation unit 201 includes information received from the intermediate parameter quality factor generation unit 104a of the intermediate parameter estimation unit 104, a mouth-to-ear delay value stored in the mouth-to-ear delay value storage unit 203, and an estimation model. Based on the above, an estimated value of the mout-to-ear delay is generated and transmitted to the total conversation quality estimation unit 106.
The estimation unit 202 is based on the information received from the intermediate parameter quality factor generation unit 104a of the intermediate parameter estimation unit 104, the PESQ (Perceptual Evaluation of Speech Quality) value stored in the PESQ value storage unit 204, and the estimation model. An estimated value of listening quality is generated and transmitted to the overall conversation quality estimation unit 106.
Note that the system indicated by reference numeral 200 in FIG. 2, that is, a system including the estimation units 201 and 202, the mout-to-ear delay value storage unit 203, and the PESQ value storage unit 204 is the quality estimation of this embodiment described later. -It corresponds to a monitoring device, particularly a device for quality estimation.

  Based on the estimated value of the mouth-to-ear delay received from the estimating unit 201 and the estimated value of the listening quality received from the estimating unit 202, the total conversation quality estimating unit 106 determines the mouth-to-ear delay and the listening quality. Estimate total conversation quality.

  As described above, in the present invention, a conventional method is used for the intermediate parameter estimation unit 104 and the overall conversation quality estimation unit 106. However, an estimation model that takes into account the implementation characteristics of the application is generated, and this model is used for user experience quality estimation. By using it in stages, the user experience quality estimation accuracy is improved.

Next, embodiments of the present invention will be described in detail.
In this embodiment, a quality estimation model for performing quality estimation is calculated in consideration of application mounting characteristics in advance, and the user experience quality is monitored using this estimation model. Therefore, first, an estimation model generation apparatus that calculates a quality estimation model will be described, and then a quality estimation / monitoring apparatus that monitors user experience quality will be described.

[Estimated model generator]
FIG. 3 is a block diagram showing a configuration of the estimation model generation apparatus according to the present embodiment. Hereinafter, a real-time application used between two VoIP-gateways 3 and 4 connected via the network 2 using a network 2 including a packet network such as an IP network, in this case, VoIP (Voice over IP) An example will be described in which a quality estimation model used for monitoring the user experience quality of a voice call using the Internet is calculated.
The estimation model generation apparatus 1 according to the present embodiment includes a media quality measurement unit 10 that measures listening quality and a mouth-to-ear delay, and a network / terminal quality information acquisition unit 20 that acquires quality information of networks and application terminals. And an estimation model generation unit 30 that generates a quality estimation model.

  The media quality measuring unit 10 outputs the evaluation voice signal to the VoIP-gateway 2 on the packet transmission side, and the listening quality based on the evaluation voice signal output from the VoIP-gateway 3 on the packet reception side. and an evaluation value / delay acquisition unit 12 for measuring a mout-to-ear delay.

  The network / terminal quality information acquisition unit 20 is a network and application terminal necessary for estimating the user experience quality and estimating the mouth-to-ear delay based on the evaluation voice signal output from the VoIP-gateway 3 on the packet receiving side Get quality information.

The estimation model generation unit 30 includes an evaluation value / delay reception unit 31, a network / terminal quality information reception unit 32, a parameter calculation unit 33, an estimation model calculation unit 34, and a model storage DB 35.
The evaluation value / delay receiving unit 31 receives the listening quality and the mouth-to-ear delay from the media quality measuring unit 10 and transmits them to the estimated model calculating unit 34.
The network / terminal quality information receiving unit 32 receives the quality information of the network and application terminal from the network / terminal quality information acquiring unit 20 and transmits the quality information to the parameter calculating unit 33.
The parameter calculation unit 33 converts the quality information of the network and the application terminal into information necessary for listening quality estimation and mouth-to-ear delay estimation, and transmits the information to the estimation model calculation unit 34.
The estimation model calculation unit 34 calculates a quality estimation model based on the information received from the evaluation value / delay reception unit 31 and the parameter calculation unit 33. Here, the estimation model calculation unit 34 may calculate a quality estimation model based on a value obtained by integrating information received from the evaluation value / delay reception unit. Further, the estimation model calculation unit 34 may calculate a quality estimation model based on a value obtained by integrating information received from the parameter calculation unit 33.
The model storage DB 35 stores the quality estimation model calculated by the estimation model calculation unit 34.

  The estimation model generation device 1 including the above-described elements includes an arithmetic device such as a CPU, a storage device such as a memory and an HDD, an input / output device such as an I / F device that transmits and receives various information to and from the VoIP-gateways 3 and 4, a keyboard and a mouse. A computer equipped with a display device such as CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), FED (Field Emission Display), or OLED (Electro Luminescence), and a program installed on this computer The hardware device is controlled by a program, that is, each element of the estimation model generation device 1 is realized by cooperation of hardware resources and software.

[Calculation method of quality estimation model]
Next, a method for calculating the quality estimation model by the estimation model generation apparatus 1 described above will be described.
First, the estimated model generation apparatus 1 uses the network / terminal quality information acquisition unit 20 to provide various network qualities for each application such as a target VoIP-gateway product or for each type of encoding in advance of service provision. Under the conditions, the quality information of the network and the application terminal is measured and transmitted to the estimation model generation unit 30. At the same time, the media quality measuring unit 10 measures the listening quality and the mouth-to-ear delay, which are the quality elements of the application, by passive measurement, and transmits them to the estimated model generation unit 30.

  Here, as long as the network / terminal quality information acquisition unit 20 can acquire the quality information of the network and application terminals necessary for the listening quality estimation and the estimation of the mout-to-ear delay, the network / terminal quality information acquisition unit 20 is invalid. Although a packet rate, an equivalent buffer size, or the like may be used, in this embodiment, VQmon technology shown in Non-Patent Document 5 is used. The network / terminal quality information acquisition unit 20 acquires the following data using VQmon technology.

As data used for listening quality estimation, “GFS (with gap filled width)”, “GFP (gap filled with interpolation)”, “GFP (gap filled with interpolation)” and “GFR (Gap filled width)”. get.
“RTT (Round-trip time)” and “RD (Received delay)” are acquired as data used for the mout-to-ear delay estimation.

While the network / terminal quality information acquisition unit 20 acquires these data, the media quality measurement unit 10 measures the listening quality and the mouth-to-ear delay.
Here, the listening quality means a user experience quality evaluation value, and specifically represents a subjective quality evaluation value or an objective quality evaluation value. In this embodiment, ITU-TP is used for objective quality evaluation. The PSQM algorithm recommended in 862 is used.

  In the present embodiment, VoIP is described as an example of a real-time application, but it can also be applied to a real-time application such as streaming. In this case, various parameters relating to viewing quality may be used instead of listening quality. Specifically, the viewing quality evaluation value includes a subjective evaluation value such as MOS (Mean Opinion Score) and DMOS (Delay Mean Opinion Score), or an objective evaluation value such as an ANI parameter, instead of the listening quality evaluation value PESQ. Is used. As with VoIP, the intermediate parameters use information on packet transfer delay and loss, delay fluctuation, and buffer overflow, and include information on lost frames such as I frames, B frames, and P frames in addition to these. The terminal setting information includes the CODEC type and screen size information. In this way, the present embodiment can be applied to real-time applications such as streaming.

Next, the estimation model generation unit 30 of the estimation model generation apparatus 1 converts the quality information of the network and the application terminal into information necessary for listening quality estimation and mout-to-ear delay estimation by the parameter calculation unit 33. And transmitted to the estimated model calculation unit 34.
First, since the listening quality is estimated from the loss rate, the loss rate (gapR: here, corresponding to the Gap rate) is obtained from the data on the Gap obtained by the VQmon technology described above. Since information related to Gap is output when the degraded voice is corrected, the Gap rate is the time when these data were acquired, that is, the speech time (Speech time), as shown in Equation (1). It can be calculated by dividing.

  Moreover, since the RTT outputs the round-trip transfer delay of packet transfer, the mout-to-ear delay is assumed to be 1/2 of the RTT as a one-way transfer delay until it is converted into voice after reaching the receiving terminal. In addition to this, RD, which is a delay of the above, is added and all the time required for packetization and CODEC encoding / decoding is collectively defined as a process delay (PD). Using these assumptions and definitions, a south-to-ear delay is defined as in equation (2). Here, PD defines an arbitrary value.

Next, the estimation model generation unit 30 of the estimation model generation device 1 calculates a quality estimation model for performing quality estimation in consideration of the mounting characteristics of the application device by the estimation model calculation unit 34. Specifically, the estimated model calculation unit 34 calculates the values calculated by the parameter calculation unit 33 using the above equations (1) and (2), the listening quality and the south-to-ear delay measured by the media quality measurement unit 10. Based on the above, a “listening quality estimation model” and a “delay estimation model” are calculated.
Here, VQmon technology uses the obtained value as the ITU-T recommendation G. This is a technique for estimating conversation quality using 07E-model. E-model is expressed by equation (3) and determines the conversation quality with R values from 0 to 100.

  R is composed of each element, and listening quality represents Ie, eff, and delay is Id, which expresses the quality. In the following section, we propose an estimation model to consider the implementation characteristics. The converted Ie, eff and Id are calculated.

(Listening quality estimation model)
First, a method for calculating a listening quality estimation model will be described.
In order to consider the mounting characteristics of the device, an estimation model is calculated by expressing the relationship between the subjective or objective quality evaluation values and the quality information necessary for estimating those values by a relational expression. First, the relationship as shown in FIG. FIG. 4 is a diagram illustrating the relationship between the gap ratio and the PESQ value.
The procedure for calculating the listening quality estimation model taking into account the mounting characteristics of the equipment is shown below.

First, from FIG. 4, to calculate the regression equation for estimating the PESQ (pesq _est) from Gap ratio (gapR). This regression equation is shown in equation (4).

  For example, here, f (gapR) in Expression (4) is as shown in Expression (5).

Next, ITU-T Recommendation G. ITU-T Recommendation G. 107 which converts 107 MOS (here PESQ) to R (R _pesq ) value. The definition of 107 is expressed by the following formulas (6) to (8).

Next, pesq _est estimated by equation (4) is converted to R _pesq by equations (6) to (8). Using 833, conversion into Ie, eff is performed using equation (9).

  The above conversion process can be expressed by the following equation (10), and this equation (10) is applied to the in-service quality monitoring technique as a listening quality estimation model considering the mounting characteristics of the device.

FIG. 5 is a diagram showing the relationship between the estimated values of Ie and eff and the calculated values when the present embodiment is applied, and FIG. 6 shows the estimated values of Ie and eff when the present embodiment is not applied. It is a figure which shows the relationship of a calculated value.
5 and 6, the horizontal axis represents estimated Ie and eff, and the vertical axis represents Ie and eff calculated from the actually measured PESQ values. As can be seen from FIGS. 5 and 6, the higher the estimation accuracy, the more values are plotted on the y = x line. The RMSE that is a measure representing the estimation error is 5.9 in the case of FIG. 5 and 23.1 in the case of FIG. 6, and the error is reduced to about 1/4 by applying this embodiment.

(Delay estimation model)
In order to consider the mounting characteristics of the device, an estimation model is calculated by expressing the relationship between the mouth-to-ear delay and the quality information necessary for estimating those values by a relational expression or the like. First, the south-to-ear delay is defined as the sum of several delays as defined in Equation (2). In the delay estimation, the delay affected by the implementation characteristics is distinguished from the delay not affected. Must. In this embodiment, an object is to provide a method and apparatus for estimating a delay that depends on the implementation. Therefore, an estimation model is created by omitting a delay that does not depend on the implementation characteristics.

  The implementation-independent delay is a one-way transfer delay, which is the time it takes for a packet to reach the receiving terminal from the application transmitting terminal on the network. Therefore, the delay (tPDnwqi) obtained by subtracting the one-way transfer delay (pTD) of the packet from the mouth-to-ear delay (MtETDnwqi) estimated by the network quality information shown in FIG. ) From which the one-way transfer delay (pTD) of the packet is removed (tPDactual) is expressed in a relational expression, thereby making it possible to estimate the mout-to-ear delay considering the mounting characteristics.

The process of calculating a mouth-to-ear delay estimation model that takes into account the mounting characteristics of the device is shown below.
First, a delay (tPD _nwqi ) _obtained by removing a one-way transfer delay (pTD) from a mout-to-ear delay (MtETD _nwqi ) estimated by the network quality information, and a one-way from the actually measured _mout -to-ear delay (MtETD _actual ). The relationship of the delay (tPD _actual ) excluding the transfer delay (pTD) is expressed by equations (11) and (12).

Next, the relationship between tPD _nwqi and tPD _actual is expressed by a relational expression such as a regression equation. This relational expression is shown in Expression (13).

For example, here, g (tPD _nwqi ) is as shown in Expression (14).

Then, by adding a one-way transfer delay over the network to the estimated tPD _est (pTD), estimates mouth-to-ear delay (MtETD _est). This is shown in equation (15).

  When the above-described process is expressed as a relational expression, Expression (16) is obtained. By this expression, the mouth-to-ear delay considering the mounting characteristics can be estimated.

By applying MtETD _est of Expression (16) to E-model, Id can be calculated in consideration of mounting characteristics. FIG. 8 is a diagram showing the relationship between the estimated value of Id and the calculated value when the present embodiment is applied, and FIG. 9 is the relationship between the estimated value of Id and the calculated value when the present embodiment is not applied. FIG.
8 and 9, the horizontal axis indicates the estimated Id, and the vertical axis indicates the Id calculated from the actually measured mout-to-ear delay. As can be seen from FIGS. 8 and 9, the higher the estimation accuracy, the more values are plotted on the y = x line. RMSE, which is a measure representing the estimation error, is 2.7 in the case of FIG. 8 and 4.6 in the case of FIG. 9, and it can be seen that the error is reduced by applying this embodiment.

Next, the estimation model generation device 1 stores the quality estimation model in consideration of the mounting characteristics calculated by the estimation model calculation unit 34 as described above in the model storage DB 35 for each product and CODEC group that created the model. When the subjective quality is monitored during execution of the real-time application using this product and CODEC, the estimated model calculated from this database is called.
This storage location may be incorporated in a product to which VQmon technology is applied, and the value output by VQmon technology is not a terminal that executes an application, but a terminal that monitors the quality of a third party. You may make it store in.

[Quality estimation / monitoring equipment]
FIG. 10 is a diagram showing a configuration of a quality monitoring system to which the quality estimation / monitoring apparatus according to the present embodiment is applied.
The quality estimation / monitoring apparatus 5 according to the present embodiment uses a quality estimation model in consideration of the mounting characteristics calculated by the estimation model generation apparatus 1 described above for each target application device and CODEC group before starting the service. Then, the user experience quality (subjective quality evaluation value) for the real-time application during service execution is estimated. Such a quality estimation / monitoring device 5 monitors the user experience quality of a real-time application used between the VoIP gateways 3 and 4. For this reason, as indicated by reference numeral 5A, quality estimation and monitoring may be performed on the VoIP-gateways 3 and 4 by being incorporated in the VoIP-gateways 3 and 4. Further, as indicated by reference numeral 5B, it may be provided between the VoIP-gateways 3 and 4, and information necessary for quality estimation may be acquired from the VoIP-gateways 3 and 4 to centrally monitor the entire system. In this case, you may make it incorporate in the estimation model production | generation apparatus 1 mentioned above.

Next, the configuration of the quality estimation / monitoring device 5A incorporated in the VoIP-gateways 3 and 4 will be described with reference to FIG. FIG. 11 is a block diagram showing the configuration of the quality estimation / monitoring apparatus 5A.
The quality estimation / monitoring device 5A includes a network / terminal quality acquisition unit 51, a parameter calculation unit 52, an estimated model reading unit 53, an estimated value calculation unit 54, an overall quality estimation unit 55, an estimated data storage unit 56, and an estimated model storage unit 57. It consists of.

The network / terminal quality acquisition unit 51 measures the quality information of the network and the application terminal, and transmits it to the parameter calculation unit 52 and the estimated data storage unit 56.
The parameter calculation unit 52 converts the quality information measured by the network / terminal quality acquisition unit into information necessary for the listening quality and the mouth-to-ear delay estimation, and the estimated model reading unit 53 and the estimated data storage unit 56 Send.
The estimation model reading unit 53 reads the quality estimation model calculated by the above-described estimation model generation device 1 from the estimation model storage unit 57 and transmits the quality estimation model together with the information received from the parameter calculation unit 52 to the estimation value calculation unit 54.
The estimated value calculation unit 54 estimates the user experience quality and delay based on the information received from the estimated model reading unit 57 and transmits the estimated quality to the overall quality estimation unit 55 and the estimated data storage unit 56.
The total quality estimation unit 55 calculates the total quality estimation value by applying the user experience quality and delay calculated by the estimation value calculation unit 54 to the total quality estimation model, and transmits the total quality estimation value to the estimation data storage unit 56.
The estimated data storage unit 56 stores information received from the network / terminal quality acquisition unit 51, the parameter calculation unit 52, the estimated value calculation unit 54, and the total quality estimation unit 55.
The estimated model storage unit 57 stores the quality estimation model calculated by the estimated model generation device 1 described above.

  The quality estimation / monitoring device 5A composed of the above-described elements includes an arithmetic device such as a CPU, a storage device such as a memory and an HDD, an I / F device that transmits and receives various information to and from the network 2 and application terminals, an input such as a keyboard and a mouse. A computer equipped with a display device such as a device, CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), FED (Field Emission Display) or organic EL (Electro Luminescence), and a program installed on this computer, respectively Each component of the quality estimation / monitoring device 5A is realized by the hardware device being controlled by a program, that is, by cooperating hardware resources and software.

Next, referring to FIG. 12, a quality estimation / monitoring device 5B provided between VoIP-gateways 3 and 4 that acquires information necessary for quality estimation from VoIP-gateways 3 and 4 and centrally monitors the entire system. The configuration of will be described. FIG. 12 is a block diagram showing the configuration of the quality estimation / monitoring apparatus 5B.
In the quality estimation / monitoring apparatus 5B, the same components and the same names as those of the quality estimation / monitoring apparatus 5A are denoted by the same names and description thereof will be omitted as appropriate.

  The quality estimation / monitoring device 5B includes a network / terminal quality acquisition unit 51, a parameter calculation unit 52, an estimated model reading unit 53, an estimated value calculation unit 54, an overall quality estimation unit 55, an estimated data storage unit 56, and an estimated model storage unit 57. , An MIB input unit 58, an input process determination unit 59, and a quality state recognition unit 60.

The MIB input unit 58 is processed by the application terminal from the VoIP-gateways 3 and 4, acquires information necessary for quality estimation left as the MIB information using the SNMP protocol, and transmits the information to the input process determination unit 59. To do.
The input process determination unit 59 determines an input process from the MIB information and causes each processing unit to perform a process not processed by the application terminal.
The quality state recognition unit 60 recognizes the current state and determines the next operation.

  In the quality estimation / monitoring apparatus 5B, the network / terminal information acquisition unit 51 reads information acquired by the application terminal, and does not mean that the quality information of the network and the terminal is measured. That is, the quality estimation / monitoring device 5B acquires the quality information before or after processing, and after quality estimation.

  The quality estimation / monitoring device 5B including the above-described elements includes an arithmetic device such as a CPU, a storage device such as a memory and an HDD, an I / F device that transmits and receives various information to and from the network 2, the VoIP-gateways 3 and 4, and application terminals. A computer equipped with an input device such as a keyboard and mouse, a display device such as CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), FED (Field Emission Display) or OLED (Electro Luminescence), and installed on this computer Each component of the quality estimation / monitoring device 5B is realized by controlling the hardware device by the program, that is, by cooperating hardware resources and software.

[User quality monitoring method]
Next, a user experience quality monitoring method by the above-described quality estimation / monitoring devices 5A and 5B will be described.
First, the quality estimation / monitoring devices 5 </ b> A and 5 </ b> B acquire network and application terminal quality information by the network / terminal quality acquisition unit 51, and transmit them to the parameter calculation unit 52. As the quality information, “GFS”, “GFP”, “GFI”, “GFR”, “RTT”, “RD”, and the like are acquired by using the VQmon technology described above.
This data is acquired by the application terminal that is executing the service, and the data is stored in the terminal as an MIB, or transmitted to a terminal that manages a third party network (service) and stored in the management terminal. You may do it.

  Next, the quality estimation / monitoring apparatuses 5 </ b> A and 5 </ b> B use the parameter calculation unit 52 to convert the acquired quality information according to the above-described equations (1) and (2), and transmit the converted information to the estimation model reading unit 53.

  Next, the quality estimation / monitoring apparatuses 5 </ b> A and 5 </ b> B read the estimated model from the estimated model storage unit 57 by the estimated model reading unit 53. Based on the quality information converted by the parameter calculation unit 52 and the estimation model read by the estimation model reading unit 53, the estimated value calculation unit 54 and the listening quality (Ie, eff value) and the mouth-to-ear Delay (Id value) delay is calculated. Here, the estimated model storage unit 57 may be provided in an application to be executed, or may be provided in a quality monitoring terminal that is a third party.

Next, in the quality estimation / monitoring apparatuses 5A and 5B, the total quality estimation unit 55 applies the user experience quality and delay calculated by the estimation value calculation unit 54 to the total quality estimation model, and calculates a total quality estimation value. .
The total quality estimation unit 55 integrates individual quality estimates such as the listening quality (Ie, eff value) and the mouth-to-ear delay (Id value) calculated by the estimated value calculation unit 54, thereby improving the conversation quality (total quality). ) Can be estimated to a value closer to the user's quality of experience. For example, in an IP phone, VQmon technology used in the present embodiment is ITU-T Recommendation G. The total quality (R value) is estimated using 107E-model.
In the present embodiment, the ITU-T recommendation G.264 is used as the overall quality estimation model. Although the total quality estimation is performed by applying 107E-model, another total quality estimation model may be used. The total quality (total conversation quality (R value)) estimation can be calculated by applying the Ie, eff value, and Id value calculated by the estimated value calculation unit 54 to Equation (3). When values other than the Ie, eff and Id values in Equation (3) are measured values (here, default values described in ITU-T Recommendation G.107), and the present embodiment is applied to FIG. FIG. 14 shows a case where the present invention is not applied. FIG. 13 is a diagram showing the relationship between the estimated value of the R value and the measured value when this embodiment is applied, and FIG. 14 is the estimated value and the measured value of the R value when this embodiment is not applied. It is a figure which shows the relationship.

  13 and 14, the horizontal axis represents the estimated R value, and the vertical axis represents the R value calculated from the actually measured mout-to-ear delay. As can be seen from FIGS. 13 and 14, the higher the estimation accuracy, the more values are plotted on the line y = x. The RMSE that is a measure representing the estimation error is 6.2 in the case of FIG. 13 and 21.0 in the case of FIG. 14, and it can be seen that the error is reduced by applying this embodiment.

It is a figure which shows the structure of the conventional quality monitoring system. It is the figure which applied this invention to the quality monitoring system of FIG. It is a block diagram which shows the structure of an estimation model production | generation apparatus. It is a figure which shows the relationship between a Gap rate and a PESQ value. It is a figure which shows the relationship between the estimated value of Ie, eff at the time of applying embodiment of this invention, and a calculated value. It is a figure which shows the relationship between the estimated value of Ie, eff at the time of not applying the embodiment of this invention, and a calculated value. It is a figure which shows the relationship of the delay remove | excluding the one-way transfer delay from the mouth-to-ear delay, and the delay which remove | excluded the one-way transfer delay from the measured mout-to-ear delay. It is a figure which shows the relationship between the estimated value of Id at the time of applying embodiment of this invention, and a calculated value. It is a figure which shows the relationship between the estimated value of Id at the time of not applying embodiment of this invention, and a calculated value. It is a figure which shows the structure of the quality monitoring system to which the quality estimation and monitoring apparatus was applied. It is a block diagram which shows the structure of 5 A of quality estimation / monitoring apparatuses. It is a block diagram which shows the structure of the quality estimation and monitoring apparatus 5B. It is a figure which shows the relationship between the estimated value of R value at the time of applying embodiment of this invention, and a measured value. It is a figure which shows the relationship between the estimated value of R value at the time of not applying embodiment of this invention, and a measured value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Estimated model production | generation apparatus, 2 ... Network, 3, 4 ... VoIP-gateway, 5, 5A, 5B ... Quality estimation / monitoring apparatus, 10 ... Media quality measurement part, 11 ... Evaluation voice output part, 12 ... Evaluation value Delay acquisition unit, 20 ... network / terminal quality information acquisition unit, 30 ... estimation model generation unit, 31 ... evaluation value / delay reception unit, 32 ... network / terminal quality information reception unit, 33 ... parameter calculation unit, 34 ... estimation model Calculation unit, 35 ... model storage DB, 51 ... network / terminal quality acquisition unit, 52 ... parameter calculation unit, 53 ... estimated model reading unit, 54 ... estimated value calculation unit, 55 ... comprehensive quality estimation unit, 56 ... estimated data storage 57 ... Estimated model storage unit, 58 ... MIB input unit, 59 ... Input process determination unit, 60 ... Quality state recognition unit, 100 ... System, 101 ... Network Quality measurement unit 102 ... Application terminal quality measurement unit 103 ... Terminal setting information acquisition unit 104 ... Intermediate parameter estimation unit 104a ... Intermediate parameter quality factor generation unit 104b ... Analog quality factor generation unit 104c ... south-to- ear delay quality information generation unit, 104d ... media quality information generation unit, 105 ... listening quality / Mouth-to-ear delay estimation unit, 105a ... South-to-ear delay estimation unit, 105b ... listening quality estimation unit, 106 ... synthesis Conversation quality estimation unit, 200 ... system, 201, 202 ... estimation unit, 203 ... south-to-ear delay value storage unit, 204 ... PESQ value storage unit.

Claims (14)

A monitoring method for monitoring user experience quality of an application executed in a packet network by passive measurement for each call,
Generating an estimation model for estimating a user experience quality of the application in consideration of mounting characteristics of the application;
An estimation step of estimating a user experience quality of the application based on application quality information including network quality information measured by the estimation model and passive measurement;
Have a monitoring step of monitoring the QoE estimated by the estimating step constantly,
The generating step includes
A calculation step for calculating a quality evaluation value for each quality element of the application;
An acquisition step of acquiring application quality information including network quality information measured by passive measurement;
A conversion step of converting the application quality information into basic information that is a basis for calculating the quality evaluation value;
A model generation step of generating an estimation model based on the quality evaluation value and the basic information;
A mounting characteristic consideration step of converting the basic information into quality information corresponding to the mounting characteristic of the application based on the relationship between the quality evaluation value and the basic information;
Have
The model experience monitoring method according to claim 1, wherein the model generation step generates an estimation model based on the quality evaluation value and the quality information . The quality factor of the application includes any one of listening quality and mouth-to-ear delay and viewing quality and mouth-to-ear delay,
The user experience quality monitoring method according to claim 1 , wherein the model generation step generates an estimation model based on a value obtained by integrating all the calculated quality evaluation values . The quality factor of the application consists of either listening quality or viewing quality,
The calculation step calculates a quality evaluation value of either subjective or objective for the quality factor for each application to be monitored,
The obtaining step, the QoE monitoring method according to claim 1, wherein the acquiring the application quality information for each be monitored application. The quality factor of the application consists of a mout-to-ear delay,
The calculation step calculates a quality evaluation value including a one-way transfer delay that is a time until the packet reaches the receiving terminal from the transmitting terminal of the application for each application to be monitored,
The acquisition step acquires the application quality information for each application to be monitored,
A delay estimation step of estimating a mouth-to-ear delay from the basic information;
A first value obtained by subtracting the one-way transfer delay from the mouth-to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated mout-to-ear delay estimated by the delay estimation step. Subtraction step to calculate and
Further comprising
The model generating step, wherein the second value to calculate a first value, QoE monitoring method according to claim 1, wherein the generating the estimated model with implementation characteristics. A monitoring device that monitors a user experience quality of an application executed in a packet network by a passive measurement for each call,
Generating means for generating an estimation model for estimating a user experience quality of the application in consideration of mounting characteristics of the application;
Estimating means for estimating user experience quality of the application based on application quality information including network quality information measured by the estimation model and passive measurement;
Monitoring means for constantly monitoring the user experience quality estimated by the estimation means;
Have
The generating means includes
A calculation means for calculating a quality evaluation value for each quality factor of the application;
Acquisition means for acquiring application quality information including network quality information measured by passive measurement;
Conversion means for converting the application quality information into basic information that is a basis for calculating the quality evaluation value;
Model generation means for generating an estimation model based on the quality evaluation value and the basic information;
Mounting characteristic consideration means for converting the basic information into quality information according to the mounting characteristic of the application based on the relationship between the quality evaluation value and the basic information;
Have
It said model generating means, the quality evaluation value and the quality information and the features and to Ruyu over The QoE monitoring device that generates an estimated model based on. The quality factor of the application includes any one of listening quality and mouth-to-ear delay and viewing quality and mouth-to-ear delay,
The user experience quality monitoring apparatus according to claim 5 , wherein the model generation unit generates an estimation model based on a value obtained by integrating all the calculated quality evaluation values . The quality factor of the application consists of either listening quality or viewing quality,
The calculation means calculates a quality evaluation value of either subjective or objective for the quality factor for each application to be monitored,
The user experience quality monitoring apparatus according to claim 5 , wherein the acquisition unit acquires the application quality information for each application to be monitored. The quality factor of the application consists of a mout-to-ear delay,
The calculation means calculates a quality evaluation value including a one-way transfer delay that is a time until a packet reaches the receiving terminal from the transmitting terminal of the application for each application to be monitored,
The acquisition means acquires the application quality information for each application to be monitored,
Delay estimation means for estimating a south-to-ear delay from the basic information;
A first value obtained by subtracting the one-way transfer delay from the mouth-to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated mout-to-ear delay estimated by the delay estimation step. Subtracting means to calculate and
Further comprising
The user experience quality monitoring apparatus according to claim 5 , wherein the model generation unit generates an estimation model that considers mounting characteristics and calculates the first value from the second value . A method for generating an estimation model for estimating a user experience quality of an application executed in a packet network in consideration of an implementation characteristic of the application,
A calculation step for calculating a quality evaluation value for each quality element of the application including at least one of listening quality, viewing quality, and mout-to-ear delay;
An acquisition step of acquiring application quality information including network quality information measured by passive measurement;
A conversion step of converting the application quality information into basic information that is a basis for calculating the quality evaluation value;
A model generation step of generating an estimation model based on the quality evaluation value and the basic information;
A mounting characteristic consideration step of converting the basic information into quality information corresponding to the mounting characteristic of the application based on the relationship between the quality evaluation value and the basic information;
Have
The model generating step, the method of generating the estimation model you and generates the estimation model based on the quality information and the quality evaluation value. The quality factor of the application consists of either listening quality or viewing quality,
The calculation step calculates a quality evaluation value of either subjective or objective for the quality factor for each application to be monitored,
The method for generating an estimation model according to claim 9 , wherein the obtaining step obtains the application quality information for each application to be monitored . The quality factor of the application consists of a mout-to-ear delay,
The calculation step calculates a quality evaluation value including a one-way transfer delay that is a time until the packet reaches the receiving terminal from the transmitting terminal of the application for each application to be monitored,
The acquisition step acquires the application quality information for each application to be monitored,
A delay estimation step of estimating a mouth-to-ear delay from the basic information;
A first value obtained by subtracting the one-way transfer delay from the mouth-to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated mout-to-ear delay estimated by the delay estimation step. Subtraction step to calculate and
Further comprising
The method for generating an estimated model according to claim 9 , wherein the model generating step generates an estimated model that considers mounting characteristics, and calculates the first value from the second value . An estimation model generation apparatus that estimates a user experience quality of an application executed in a packet network in consideration of an implementation characteristic of the application,
Calculating means for calculating a quality evaluation value for each quality element of the application including at least one of listening quality, viewing quality, and mout-to-ear delay;
Acquisition means for acquiring application quality information including network quality information measured by passive measurement;
Conversion means for converting the application quality information into basic information that is a basis for calculating the quality evaluation value;
Model generation means for generating an estimation model based on the quality evaluation value and the basic information;
Mounting characteristic consideration means for converting the basic information into quality information according to the mounting characteristic of the application based on the relationship between the quality evaluation value and the basic information;
Have
It said model generating means, generating device of estimation model and generates an estimation model based on said quality evaluation value and the quality information. The quality factor of the application consists of either listening quality or viewing quality,
The calculation means calculates a quality evaluation value of either subjective or objective for the quality factor for each application to be monitored,
13. The estimation model generation apparatus according to claim 12 , wherein the acquisition unit acquires the application quality information for each application to be monitored . The quality factor of the application consists of a mout-to-ear delay,
The calculation means calculates a quality evaluation value including a one-way transfer delay that is a time until a packet reaches the receiving terminal from the transmitting terminal of the application for each application to be monitored,
The acquisition means acquires the application quality information for each application to be monitored,
Delay estimation means for estimating a south-to-ear delay from the basic information;
A first value obtained by subtracting the one-way transfer delay from the mouth-to-ear delay, and a second value obtained by subtracting the one-way transfer delay from the estimated mout-to-ear delay estimated by the delay estimation step. Subtracting means to calculate and
Further comprising
13. The estimation model generation apparatus according to claim 12 , wherein the model generation unit generates an estimation model that considers mounting characteristics and calculates the first value from the second value .

Images