JP2020028082A - Evaluation method, evaluation device, and program - Google Patents

Abstract

To evaluate the estimation accuracy of QoE using a network quality index value.SOLUTION: A computer executes the following procedures: acquiring multiple network quality index values for a predetermined time in the past; calculating an evaluation value indicating the temporal change of the multiple network quality index value using the multiple network quality index values acquired by the acquisition procedure; comparing and judging the magnitude relationship between the evaluation value calculated by the evaluation value calculation procedure and a predetermined threshold value; and outputting a piece of information indicating the quality of estimation accuracy when estimating a given QoE on the basis of the network quality index value according to the judgment result by the judgment procedure.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an evaluation method, an evaluation device, and a program.

  An evaluation index called QoE (Quality of Experience) is known as one of the evaluation indexes for communication services (for example, Non-Patent Document 1). QoE is an evaluation index for service quality experienced by a user, and is also referred to as user experience quality. Generally, in quality control of communication services, a communication service provider or the like is required to evaluate and understand QoE.

  As a method for easily evaluating and grasping QoE, there is a method for estimating QoE from an index value representing network quality such as throughput or RTT (Round Trip Time) (hereinafter, referred to as “network quality index value”). Are known.

Atsushi Okamoto, Ryogo Kubo, Sho Tsugawa, Takeshi Hiraguri, Takahiro Matsuda, "Network Quality Evaluation, Design and Control Technology", IEICE Journal, p.831-836, August 2017.

  However, in the method of estimating QoE from the network quality index value, the estimation accuracy may be significantly different. Therefore, it may not be possible to estimate an appropriate QoE from the network quality index value.

  For example, when "Web page display wait time in a Web browser" is used as QoE and "throughput" is used as a network quality index value, the Web page display wait time increases when the throughput is low, and the Web time is high when the throughput is high. The page display latency is reduced. On the other hand, when the estimation accuracy is low, for example, a situation may occur in which the display wait time of the Web page is long despite the high throughput.

  An embodiment of the present invention aims to evaluate the accuracy of QoE estimation using a network quality index value.

  To achieve the above object, an embodiment of the present invention provides an acquisition procedure for acquiring a plurality of network quality index values during a predetermined time in the past, and a plurality of network quality index values acquired by the acquisition procedure. Using the evaluation value calculation procedure for calculating an evaluation value representing a temporal change of the plurality of network quality index values, and comparing the evaluation value calculated by the evaluation value calculation procedure with a predetermined threshold value A computer executes a determining step of determining, and an output step of outputting information indicating whether or not the estimation accuracy is good when estimating a predetermined QoE from the network quality index value according to a result of the determination by the determining step. It is characterized by.

  It is possible to evaluate the estimation accuracy of QoE using the network quality index value.

It is a figure showing an example of functional composition of an evaluation device in an embodiment of the invention. 5 is a flowchart illustrating an example of an evaluation process according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a hardware configuration of an evaluation device according to the embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail. In the embodiment of the present invention, a description will be given of an evaluation apparatus 10 that can evaluate the estimation accuracy when QoE is estimated using a network quality index value. The network quality index value includes, for example, throughput and RTT. The QoE includes, for example, a Web page display wait time in a Web browser, a MOS (Mean Opinion Score) value in an IP (Internet Protocol) telephone, a value representing smoothness in a video, and the like.

  Here, when estimating QoE from the network quality index value, one of the main factors that lowers the estimation accuracy is that “network quality is unstable”. Unstable network quality refers to a case where network quality index values such as throughput and RTT greatly change in a short time. Therefore, the evaluation device 10 according to the embodiment of the present invention determines whether the estimation accuracy is good or not using the time change of the network quality index value.

<Functional Configuration of Evaluation Apparatus 10>
First, a functional configuration of the evaluation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a functional configuration of an evaluation device 10 according to an embodiment of the present invention.

  As shown in FIG. 1, the evaluation device 10 according to the embodiment of the present invention includes an input unit 101, an evaluation value calculation unit 102, a determination unit 103, and an output unit 104. Each of these functional units is realized by a process of causing a processor such as a CPU (Central Processing Unit) to execute one or more programs installed in the evaluation device 10.

  Further, the evaluation device 10 according to the embodiment of the present invention includes a storage unit 110. The storage unit 110 can be realized using a storage device such as a RAM (Random Access Memory) or an auxiliary storage device. The storage unit 110 may be realized using a storage device or the like connected to the evaluation device 10 via a communication network.

The storage unit 110 stores time-series data of the network quality index value. Hereinafter, the time series data of the network quality index value is represented as Q = {Q _i }, and the network quality index value at the index _i is represented as “network quality index value Q _i ”. Here, the storage unit 110 stores time-series data Q for each range in which QoE is estimated from the network quality index value (hereinafter, referred to as “estimation range”).

The estimation range is a range in which a change or deterioration in network quality is expected to be common, and is represented by, for example, an area including a device group located under a certain device, a certain network, or the like. Is done. In this case, each network quality index value Q _i contained in the time-series data Q, which is an index value representing the estimation range corresponding to the time-series data Q. For example, when the estimation ranges are D ₁ and D ₂ , the time series data corresponding to the estimation range D ₁ is Q ′, and the time series data corresponding to the estimation range D ₂ is Q ″, the time series data Q ′ includes each network quality index value Q _{i 'which} has become an index value representative of the estimation range D _1, time-series data Q'' each network quality index value Q _i'' included in the representative of the estimated range D ₂ It is an index value.

Further, the time interval between each network quality index value Q _i and G. That is, when the time of _{t i} the index i represents the time at which the index i + 1 is expressed was _{t i + 1,} is a _{G = t i} + 1 -t _i.

Thus, the storage unit 110, for each estimation range, the time-series data Q of the network quality index value Q _i every time interval G is assumed to be stored. Incidentally, these network quality index value Q _i, for example, every time interval G, is calculated by the measuring device or the like that measures the network quality of a predetermined estimation range stored in the storage unit 110.

Input unit 101, for each estimation range, and acquires the network quality index value Q _i between the most recent past W time from the storage unit 110. Here, W is a parameter set in advance. For example, when W = 30 [minutes] = 1800 [seconds] and G = 60 [seconds], the input unit 101 outputs 30 network quality indices from the current time to the past 30 minutes before in the estimation range. Acquire the values Q ₁ , Q ₂ ,..., Q ₃₀ .

Evaluation value calculation section 102 uses the parameter τ which is set in advance, calculates an evaluation value X from the network quality index value Q _i obtained by the input unit 101. Here, an integer value is set for τ, which represents a temporal “lag” (ie, a temporal delay). In other words, the time difference between when G × tau calculates the evaluation value X from the network quality index value Q _i. The details of the method of calculating the evaluation value X will be described later.

  The determination unit 103 compares and determines the magnitude relationship between the evaluation value X calculated by the evaluation value calculation unit 102 and a predetermined threshold value P. For example, the determination unit 103 determines that the estimation accuracy is high (that is, the network quality is stable) when the evaluation value X is equal to or smaller than a predetermined threshold P, and determines that the evaluation value X is larger than the predetermined threshold P. It is determined that the estimation accuracy is low (that is, the network quality is unstable).

  The output unit 104 outputs the result of the determination by the determination unit 103. The output destination of the output unit 104 is not limited. The output destination of the output unit 104 may be the storage unit 110, a display device such as a display, or a predetermined device connected via a communication network.

<Evaluation process>
Next, a process (evaluation process) in which the evaluation device 10 according to the embodiment of the present invention evaluates the accuracy of QoE estimation will be described with reference to FIG. FIG. 2 is a flowchart illustrating an example of the evaluation process according to the embodiment of the present invention. The evaluation process shown in FIG. 2 is repeatedly executed for each estimation range, for example. Hereinafter, the evaluation process regarding one certain estimation range will be described.

Step S101: input unit 101 obtains the network quality index value _{Q i} between the most recent past W time in the estimated range from the storage unit 110 (read). In the following, the network quality assessment value _{Q 1,} Q 2, · · ·, it is assumed _{that Q N} is acquired. Here, N = floor (W / G). Floor () is a function that outputs an integer value obtained by truncating the decimal part of the argument. In this case, the network quality evaluation values Q ₁ , Q ₂ ,..., Q _N are represented by vectors composed of N elements.

Incidentally, not limited to the case of obtaining the network quality index value Q _i of the past W period from the storage unit 110, input unit 101, for example, via a communication network, a predetermined device (e.g., network quality index value Q _The network quality index value Qi may be obtained from a storage device that stores _i .

Step S102: the evaluation value calculation unit 102, a network quality index value _Q 1 obtained by the input unit _101, Q 2, · · ·, calculates an evaluation value X from _{Q N.} Here, the evaluation value calculation unit 102 may calculate the evaluation value X by one of the following (method 1) to (method 4), for example.

(Method 1)
Method 1 is a method using an autocorrelation function. The autocorrelation function is generally a function using “time lag” as a variable. In the embodiment of the present invention, a lag τ is set for the time lag.

In the method 1, the evaluation value calculation unit 102 calculates the evaluation value X from the network quality index values Q ₁ , Q ₂ ,..., Q _N by the following equation (1).

ここで、Ｑ_ａｖはＱ_１，Ｑ_２，・・・，Ｑ_Ｎの平均値、Ｑ_ｖａｒはＱ_１，Ｑ_２，・・・，Ｑ_Ｎの分散である。

_{Here, Q av} is _{Q 1, Q 2, ···,} the average value of _{Q N, Q var} is _{Q 1, Q 2, ···,} it is the variance of _{Q N.}

The network quality index value _{Q 1,} Q 2, · · ·, autocorrelation of _{Q N} can also be calculated by using various existing libraries. For example, in Python, the pandas library pandas. Series. The autocorrelation can be calculated by autocorr.

(Method 2)
Method 2 is a method based on covariance. In method 2, the evaluation value calculation unit 102 by the following equation (2), the network quality index value _{Q 1,} Q 2, · · ·, calculates an evaluation value X from _{Q N.}

ここで、Ｑ_ａｖ１はＱ_１，Ｑ_２，・・・，Ｑ_Ｎ−τの平均値、Ｑ_ａｖ２はＱ_τ，Ｑ_２，・・・，Ｑ_Ｗの平均値である。

_{Here, Q av1} is _{Q 1, Q 2, ···,} the average value of _{Q N-τ, Q av2} is the average value of _{Q τ, Q 2, ···,} Q W.

The network quality index value _{Q 1,} Q 2, · · ·, covariance of _{Q N} can also be calculated by using various existing libraries. For example, in Python, numpy. The covariance can be calculated by the cov function.

(Method 3)
Method 3 is a method using a correlation coefficient. In method 3, the evaluation value calculation unit 102 by the following equation (3), the network quality index value _{Q 1,} Q 2, · · ·, calculates an evaluation value X from _{Q N.}

ここで、Ｑ_ａｖ１はＱ_１，Ｑ_２，・・・，Ｑ_Ｎ−τの平均値、Ｑ_ａｖ２はＱ_τ，Ｑ_２，・・・，Ｑ_Ｗの平均値である。

_{Here, Q av1} is _{Q 1, Q 2, ···,} the average value of _{Q N-τ, Q av2} is the average value of _{Q τ, Q 2, ···,} Q W.

The network quality index value _{Q 1,} Q 2, · · ·, correlation coefficients _{Q N} can also be calculated by using various existing libraries. For example, in Python, numpy. The correlation coefficient can be calculated by the corrcoef function.

(Method 4)
Method 4, the network quality index value Q _1, Q _2, · · ·, on which were normalized Q _N, and calculates the evaluation value X by any of the methods of the above methods 1 to Method 3. That is, the evaluation value calculation unit 102 standardizes each Q _i (i = 1,..., N) by the following equation (4).

ここで、「←」は右辺の計算結果で左辺を更新することを表す。すなわち、上記の式（４）は、各ｉに対して、（Ｑ_ｉ−μ）／σでＱ_ｉを更新することを表す。また、μ及びσは、それぞれ標準化オフセット及び標準化偏差であり、任意の値を用いることができるが、典型的には、μとしてＱ_１，Ｑ_２，・・・，Ｑ_Ｎの平均値、σとしてはＱ_１，Ｑ_２，・・・，Ｑ_Ｎの標準偏差が用いられる。

Here, “←” indicates that the left side is updated with the calculation result on the right side. That is, the above equation (4), for each i, indicating that updating the _{Q i} in _(Q i - [mu]) / sigma. Further, the mu and sigma, respectively a standardized offset and standardization deviation can be any value, typically, Q _1, Q _2, · · ·, the average value of Q _N as mu, sigma _{Q 1, Q 2, ···,} the standard deviation of _{Q N} is used as the.

Then, evaluation value calculation unit 102, _{Q 1,} Q 2 after standardization, ..., using a _{Q N,} and calculates the evaluation value X by any of the methods of the above methods 1 to Method 3.

  Step S103: The determination section 103 compares and determines the magnitude relationship between the evaluation value X calculated by the evaluation value calculation section 102 and a predetermined threshold value P. At this time, for example, the determination unit 103 determines that the estimation accuracy is high when the evaluation value X is equal to or less than the predetermined threshold P, and determines that the estimation accuracy is low when the evaluation value X is larger than the predetermined threshold P. .

The determination unit 103 is, for _{example, P 1> P 2> ···} > P M become predetermined plurality of thresholds _{P m (m = 1, ···} , M) and, comparison determining the evaluation value X You may. In this case, the determination unit 103, for example, comparing determines the evaluation value X and the threshold P _1, the estimation accuracy if the evaluation value X is larger than the threshold value P ₁ is determined to be "Level 1", the evaluation value X There is determined the threshold P ₁ or less and the threshold value P ₂ is greater than the estimation accuracy if it is "level 2". Similarly, the determination unit 103 determines the estimation accuracy when the evaluation value X is less than the threshold value P _m and larger than the threshold value P _{m + 1} is to be "level m + 1". That is, the determination unit 103, for example, a plurality of threshold values _{P m (m = 1, ···} , M) of, by identifying the maximum threshold _{P m'less} than the evaluation value X, the estimation accuracy, a particular It is determined that the level is m ′ corresponding to the threshold value _{Pm ′} .

Thereby, the determination unit 103 can determine the level of the estimation accuracy. Incidentally, when the evaluation value X is less than or equal to the threshold value P _M, the determination unit 103, the estimation accuracy may be to be a "level M + 1". Here, the higher the level, the higher the estimation accuracy.

  Step S104: The output unit 104 outputs the result of the determination by the determination unit 103. As described above, the determination result may be “high estimation accuracy” or “low estimation accuracy”, or may be a level of estimation accuracy. The output unit 104 may output information indicating the estimation range together with the determination result.

<Hardware Configuration of Evaluation Apparatus 10>
Next, a hardware configuration of the evaluation device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a hardware configuration of the evaluation device 10 according to the embodiment of the present invention.

  As shown in FIG. 3, the evaluation device 10 according to the embodiment of the present invention includes an input device 201, a display device 202, an external I / F 203, a RAM 204, a ROM (Read Only Memory) 205, a processor 206, , A communication I / F 207, and an auxiliary storage device 208. Each of these hardware is communicably connected via a bus 209.

  The input device 201 is, for example, a keyboard, a mouse, a touch panel, or the like, and is used by a user to input various operations. The display device 202 is, for example, a display or the like, and displays a processing result of the evaluation device 10. Note that the evaluation device 10 may not have at least one of the input device 201 and the display device 202.

  The external I / F 203 is an interface with an external device. The external device includes a recording medium 203a and the like. The evaluation device 10 can read and write the recording medium 203a and the like via the external I / F 203. The recording medium 203a may store one or more programs or the like that implement each functional unit (for example, the input unit 101, the evaluation value calculation unit 102, the determination unit 103, and the output unit 104) included in the evaluation device 10. .

  Examples of the recording medium 203a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

  The RAM 204 is a volatile semiconductor memory that temporarily stores programs and data. The ROM 205 is a nonvolatile semiconductor memory that can retain programs and data even when the power is turned off. The ROM 205 stores, for example, setting information about an OS (Operating System), setting information about a communication network, and the like.

  The processor 206 is, for example, a CPU or the like, and is an arithmetic device that reads a program or data from the ROM 205 or the auxiliary storage device 208 onto the RAM 204 and executes processing. Each functional unit of the evaluation device 10 is realized by, for example, a process of causing the processor 206 to execute one or more programs stored in the auxiliary storage device 208.

  The communication I / F 207 is an interface for connecting the evaluation device 10 to a communication network. One or more programs that realize the respective functional units of the evaluation device 10 may be obtained (downloaded) from a predetermined server or the like via the communication I / F 207.

  The auxiliary storage device 208 is, for example, a hard disk drive (HDD) or a solid state drive (SSD), and is a nonvolatile storage device that stores programs and data. The programs and data stored in the auxiliary storage device 208 include, for example, an OS, an application program, and one or more programs for realizing each functional unit of the evaluation device 10.

  The evaluation device 10 according to the embodiment of the present invention has the hardware configuration illustrated in FIG. 3, so that the above-described various processes can be realized. FIG. 3 shows an example of a hardware configuration in the case where the evaluation device 10 is realized by one computer. However, the present invention is not limited to this, and the evaluation device 10 may be realized by a plurality of computers.

<Summary>
As described above, the evaluation device 10 according to the embodiment of the present invention can determine whether QoE estimation accuracy is good or not using the time change of the network quality index value. Accordingly, when QoE is estimated from the network quality index value, it is possible to determine the reliability (or validity) of the QoE.

  For example, in the case of monitoring the quality of QoE, if the monitoring accuracy of “quality degradation” is detected when the estimation accuracy of QoE is reduced, it is not possible to determine whether or not it is necessary to respond to the monitoring result. Therefore, in this case, the effectiveness of the quality monitoring is reduced. On the other hand, by using the evaluation device 10 according to the embodiment of the present invention, the quality of the QoE estimation accuracy can be obtained. Therefore, when the monitoring result of “quality degradation” is detected, the quality of the QoE estimation accuracy is determined. Accordingly, it is possible to determine whether or not it is necessary to respond to the monitoring result.

  As described above, the evaluation device 10 according to the embodiment of the present invention estimates QoE from the network quality index value in order to determine whether the accuracy of the QoE estimation accuracy is good or not using the time change of the network quality index value. The estimation method used for the estimation is not limited, and the quality of the estimation accuracy of any estimation method can be determined.

  The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications and changes can be made without departing from the scope of the claims.

Reference Signs List 10 evaluation device 101 input unit 102 evaluation value calculation unit 103 determination unit 104 output unit 110 storage unit

Claims (6)

An acquisition procedure for acquiring a plurality of network quality index values during a predetermined time in the past;
Using a plurality of network quality index values acquired by the acquisition procedure, an evaluation value calculation procedure of calculating an evaluation value representing a temporal change of the plurality of network quality index values,
An evaluation value calculated by the evaluation value calculation step, and a determination step of comparing and determining a magnitude relationship between a predetermined threshold value;
An output step of outputting information indicating whether or not the estimation accuracy is good when estimating a predetermined QoE from the network quality index value according to a result of the determination by the determination step;
An evaluation method characterized in that a computer executes the method. The evaluation value calculation procedure,
Using the plurality of network quality index values and a parameter representing a preset time delay, calculate the autocorrelation, covariance or correlation coefficient of the plurality of network quality index values, The evaluation method according to claim 1, wherein a calculation result is calculated as the evaluation value. The evaluation value calculation procedure,
Each of the plurality of network quality index values is standardized, and using the standardized plurality of network quality index values and the parameter, the autocorrelation, covariance or phase of the plurality of standardized network quality index values is used. The evaluation method according to claim 2, wherein one of the relation numbers is calculated, and the calculation result is calculated as the evaluation value. The determination procedure includes:
The evaluation value is compared with a magnitude relationship between each of a plurality of predetermined thresholds,
The output procedure includes:
4. The information according to claim 1, wherein information corresponding to a maximum threshold less than the evaluation value is output as information indicating whether the estimation accuracy is good or not. 5. Evaluation method. Acquiring means for acquiring a plurality of network quality index values during a predetermined time in the past,
Using the plurality of network quality index values obtained by the obtaining unit, an evaluation value calculation unit that calculates an evaluation value representing a temporal change of the plurality of network quality index values,
Evaluation means calculated by the evaluation value calculation means, a determination means for comparing and determining the magnitude relationship between a predetermined threshold,
An output unit that outputs information indicating whether or not the estimation accuracy is good when estimating a predetermined QoE from the network quality index value according to a determination result by the determination unit;
An evaluation device comprising:   A program for causing a computer to execute each procedure in the evaluation method according to claim 1.

Images