JP6063801B2 - Quality evaluation apparatus, quality evaluation method and program - Google Patents

Description

  The present invention relates to a quality evaluation apparatus, a quality evaluation method, and a program. More specifically, the present invention relates to a technique for detecting quality deterioration and identifying deterioration factors when an application is operated in an application used via a network.

  In recent years, applications that are used via a network, such as web applications, have become widespread. In such an application, since a part of the processing is performed on the web server, the user's bodily sensation quality is likely to be affected by the delay of the server and the network, and it is difficult to guarantee a certain bodily bodily quality. Therefore, it is necessary to constantly monitor the end-to-end quality, and to identify the cause of deterioration when quality deterioration is detected. For example, Non-Patent Document 1 and Non-Patent Document 2 propose a method of constantly monitoring server and network quality (response time, delay, loss rate), and specifying a location where they exceed a threshold as a deterioration factor. Yes.

  Recently, web applications including dynamic contents are also increasing. In such an application, the amount of processing at the user terminal increases, and thus the user terminal is a major factor in the quality of experience in addition to the server and the network. Therefore, it is necessary to monitor the quality including the user terminal. For example, in Non-Patent Document 3 and Non-Patent Document 4, the terminal quality (CPU usage rate, memory usage rate, etc.) is monitored and they exceed the threshold. In this case, a method for determining that the terminal is in a high load state has been proposed.

Tomoya Yoshida, "Operation Management Technology for Internet Providers", IPSJ SIG, 2007-7 Atsuo Tachibana, Shigehiro Ano, Masato Tsuru, "Selecting Measurement Paths for Efficient Network Monitoring and Diagnosis under Operational Constraints", INCoS, 2011 Zomut, "Watchdog Task Manager" https://sites.google.com/a/zomut.com/www/watchdog Atsuo Tachibana, Shigehiro Ano, Masato Tsuru, "Network quality measurement system using user PC", IEICE Technical Report, 2010-11 Tenmura Nakamura, Koji Yamamoto, Yasunori Honda, Satoshi Takahashi, "Examination of Operation Feature Extraction Method for Cloud Applications", 10th QoS Workshop, 2012-11 Fujitsu, "CA APM 9" http://systemwalker.fujitsu.com/jp/caapm/ Takashi Ishida and Hiroyuki Kitagawa, "Effective Outlier Detection Method for Time Series Data", IEICE 19th Data Engineering Workshop (DEWS2008), 2008-03

  However, since the value of the quality index (response time or CPU usage rate) that has been used in the past varies depending on the application and operation content to be used, as the types of applications and operations diversify, The gap is getting bigger. For this reason, it may not always be appropriate to detect quality degradation and to isolate degradation factors by the conventional threshold monitoring method. From the viewpoint of enhancing user support, it is necessary to grasp and deal with quality degradation in line with the user experience.

  The present invention has been made in view of the problems in the conventional methods as described above, and an object of the present invention is to detect quality deterioration based on a user experience and to identify a cause of deterioration.

The quality evaluation apparatus according to one aspect of the present invention is
A quality evaluation device that evaluates the quality at the time of execution of an application,
Indicates at least one of a network state and a terminal state at the time of application execution based on a log cut out from an application log output by the application corresponding to a period from the start time to the end time of the user experience waiting time State index calculating means for calculating the state index;
Storage means for storing the sensory waiting time and the calculated state index as a set;
Similar operation extraction means for extracting a set of a sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage means;
A quality map display means for displaying a set of the sensory waiting time and the calculated state index on a map together with a set of the sensory waiting time and the state index corresponding to the extracted similar operation ;
It is characterized by having.

In addition, the quality evaluation method according to one aspect of the present invention includes:
A quality evaluation method in a quality evaluation apparatus that evaluates the quality at the time of execution of an application,
The network at the time of application execution based on a log cut out from the application log output by the application corresponding to the period from the start time to the end time of the user experience waiting time Calculating a state index indicating at least one of a state and a terminal state;
Storing the bodily sensation waiting time and the calculated state index in a storage unit;
Extracting a set of sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage unit;
Displaying the set of the estimated sensation waiting time and the calculated state index on a map together with the combination of the sensation waiting time and the state index corresponding to the extracted similar operations ;
It is characterized by having.

A program according to an aspect of the present invention is
A program for causing a computer to function as a quality evaluation device that evaluates the quality at the time of execution of an application. The computer is output by the application corresponding to the period from the start time to the end time of the user's experience waiting time. A state index calculating unit that calculates a state index indicating at least one of a network state and a terminal state at the time of application execution based on a log cut out from the application log;
Storage means for storing the sensory waiting time and the calculated state index as a set;
Similar operation extraction means for extracting a set of a sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage means, and the estimated sensation waiting time and the calculated state index Quality map display means for displaying the set on the map together with the set of the waiting time and the state index corresponding to the extracted similar operation ,
It is made to function as.

  According to the present invention, it is possible to detect quality degradation based on the user experience and to identify degradation factors.

1 is a configuration diagram of a system including a quality evaluation apparatus according to a first embodiment of the present invention. The block diagram of another system containing the quality evaluation apparatus which concerns on 1st Example of this invention The flowchart of the quality display method in the quality evaluation apparatus which concerns on 1st Example of this invention. Examples of application logs extracted by the quality evaluation apparatus according to the first embodiment of the present invention from the terminal and application logs cut out in accordance with the waiting time for experience Examples of a network log obtained by the quality evaluation device according to the first embodiment of the present invention from a terminal and a network log cut out corresponding to the waiting time Example of sensory waiting time estimated in quality evaluation apparatus according to first embodiment of the present invention Example of processing multiplicity of communication processing calculated by the quality evaluation apparatus according to the first embodiment of the present invention (when time slot = 0.1 seconds) The example which shows the group of the sensation waiting time and state index corresponding to operation similar to a target user extracted in the quality evaluation apparatus which concerns on 1st Example of this invention Example of a quality map displayed in the quality evaluation apparatus according to the first embodiment of the present invention The block diagram of the system containing the quality evaluation apparatus which concerns on 2nd Example of this invention. Flowchart of the quality display method in the quality evaluation apparatus according to the second embodiment of the present invention. Example of the usage environment of the target user used in the second embodiment of the present invention The example which shows the group and usage environment of the waiting time and state parameter | index corresponding to operation similar to an object user extracted in the quality evaluation apparatus which concerns on 2nd Example of this invention Example of quality map displayed in quality evaluation apparatus according to second embodiment of the present invention Example of filtered quality map displayed in quality evaluation apparatus according to second embodiment of the present invention Configuration diagram of a system including a quality evaluation apparatus according to a third embodiment of the present invention Flowchart of the quality display method in the quality evaluation apparatus according to the third embodiment of the present invention.

  Examples of the present invention will be described in detail below.

  In the embodiment of the present invention, a quality evaluation apparatus that evaluates quality at the time of execution of an application used via a network is used. The quality evaluation device estimates the user's experience waiting time from the operation log of the user (hereinafter referred to as the target user) that is the target of quality deterioration detection and deterioration factor isolation, and the network state and terminal during the experience waiting time A state index indicating the state is calculated. Here, the log indicates an application log in which processing at the time of application execution is recorded. Furthermore, a network log in which the amount of transmitted / received data is recorded may be used. The quality evaluation apparatus visualizes the quality by displaying the sensation waiting time and the state index on a map. For example, quality degradation is detected and degradation factors are identified by comparing on the map the sensation waiting time and state indicators of other users corresponding to operations similar to the target user.

  Details of the quality evaluation apparatus will be described below.

<First embodiment>
FIG. 1 shows a configuration diagram of a system including a quality evaluation apparatus according to a first embodiment of the present invention. The system according to the first embodiment of the present invention includes a quality evaluation apparatus 10, a user terminal 20, and an application (AP) server 30. As shown in FIG. 2, each means of the quality evaluation apparatus 10 may be included in the user terminal 20.

  The AP server 30 is, for example, an HTTP server 301 that provides content 303 to the user terminal 20.

  The user terminal 20 requests content from the AP server 30 and displays a web browser 205 that displays the content received from the AP server 30, and log acquisition means 203 that acquires application logs and network logs executed on the user terminal. And a transmission unit 201 for transmitting the application log and the network log to the quality evaluation apparatus 10. The application log is a log in which the type, start time, and processing time (or end time) of processing executed on the user terminal 20 are output. The network log is a log in which the time when data is transmitted / received by the user terminal 20 and the amount of transmitted / received data are output.

  The quality evaluation apparatus 10 includes an experiential waiting time estimation unit 101, a log cutout unit 103, a state index calculation unit 105, a similar operation extraction unit 107, a quality map display unit 109, and a storage unit 111.

  The sensation waiting time estimation unit 101 acquires a log from the user terminal 20 of the target user, estimates the start time and end time of the sensation waiting time of the target user, and stores them in the storage unit 111.

  The log cutout unit 103 cuts out a log that starts after the start time of the sensation waiting time estimated for the target user and ends before the end time, and stores the log in the storage unit 111.

  The state index calculation unit 105 calculates a state index indicating a network state and a terminal state at the time of executing the application from the extracted application log. For example, the extracted application log is divided by a time slot of a predetermined time, the number of processes related to communication executed in the time slot is calculated, and all are added.

  The sensory waiting time estimation unit 101, the log cutout unit 103, and the state index calculation unit 105 acquire not only the log from the user terminal 20 of the target user but also the log from the other user's terminal, and the other user's terminal Is stored in the storage unit 111.

  The similar operation extraction unit 107 extracts from the storage unit 111 a set of a sensation waiting time and a state index corresponding to an operation similar to the operation of the target user. Note that, in order to determine a similar operation, an operation feature amount representing a feature related to the operation is used.

  The quality map display unit 109 displays the combination of the sensation waiting time and the state index of the target user on the map together with the combination of the sensation waiting time and the state index corresponding to an operation similar to the operation of the target user.

  The storage unit 111 is a storage device that stores a combination of a sensation waiting time and a state index and an operation feature amount.

  Next, the operation of the quality evaluation apparatus 10 of the first embodiment will be described in detail with reference to FIGS. FIG. 3 is a flowchart of the quality display method in the quality evaluation apparatus 10 according to the first embodiment of the present invention.

  In step S101, the sensation waiting time estimation unit 101 acquires an application log from the user terminal 20 of the target user. Specifically, the transmission unit 201 of the user terminal 20 acquires a log corresponding to the time zone when the target user performed an operation when executing the application, and transmits the log to the quality evaluation apparatus 10. Note that when the target user performs an operation when executing the application, an application log is output. Therefore, by checking the application log, the time zone in which the target user performed the operation can be recognized. The log received from the user terminal 20 is used as an input to the sensation waiting time estimation unit 101 in the quality evaluation apparatus 10.

  FIG. 4A shows an example of an application log acquired from the user terminal 20. The application log includes the type of processing executed on the user terminal 20, the start time, and the processing time (or end time). FIG. 5A shows an example of a network log acquired from the user terminal 20. The network log includes the time when data is transmitted / received by the user terminal 20 and the amount of transmitted / received data.

  In step S <b> 103, the sensation waiting time estimation unit 101 estimates the start time and end time of the operation sensation waiting time and stores them in the storage unit 111. Specifically, a synchronization process execution section in which the number of processes executed by an application such as the web browser 205 (hereinafter referred to as process multiplicity) exceeds a threshold value is estimated as an experience waiting time. Moreover, the sensory waiting time estimation means 101 may estimate the sensory waiting time by separately measuring the actual sensory waiting time in the user terminal 20 and receiving it.

  FIG. 6 shows an example of the sensation waiting time. Here, it is assumed that the start time of the bodily sensation waiting time is estimated to be 16: 33: 58.4 seconds on October 1, 2012 and the end time is 16: 34: 34.7 seconds on October 1, 2012.

  In step S105, the log cutout unit 103 cuts out a log from the start time to the end time of the sensation waiting time by the operation of the target user. Specifically, the log cutout unit 103 cuts out from the log received from the user terminal 20 of the target user, a log that starts after the start time of the estimated experience waiting time and ends before the estimated end time. .

  For example, as shown in FIG. 4 (B) and FIG. 5 (B), the start time is after 16: 33: 58.4 seconds on October 1, 2012, and the end time is 2012, corresponding to the sensation waiting time. The application log and network log before 16: 34: 34.7 seconds on October 1 will be extracted.

  It should be noted that the log acquisition means 203 for acquiring logs at the user terminal 20 can be replaced with a general application analysis tool, a network analysis tool, or the like. Moreover, the transmission means 201 which transmits the log acquired from the user terminal 20 to the quality evaluation apparatus can be replaced with a general apparatus or a program.

  Here, “operation” refers to an operation performed by the user, such as logging into a web site, clicking a link, and the like. “Processing” refers to processing executed on the user terminal 20 in accordance with a user operation. Usually, a plurality of processes are executed in parallel for one user operation.

  In step S107, the state index calculation unit 105 determines an index indicating the network state and / or an index indicating the terminal state from a cut log that is a log from the start time to the end time of the sensation waiting time by the operation of the target user. Is calculated. In the case of an index indicating the network status, an index that is not affected by the terminal status, such as a total value or an average value of communication processing times, an integrated value of communication processing multiplicity, or a combination thereof is used. The same applies to the index indicating the terminal status, and the index is not affected by the network status, such as the total value or average value of the processing time, the integral value of the processing multiplicity, or a combination thereof in script processing and / or drawing processing. Is used. The calculated state index is stored in the storage unit 111.

  The calculation procedure will be described by taking as an example a case where an integral value of communication processing multiplicity is used as the state index indicating the network state.

  (1) The communication processing log included in the application log is quantized at regular time intervals. In the quantization method, a log is divided into time slots of a predetermined time, and the number of communication processes performed in the time slot is calculated as the processing multiplicity.

  (2) All the processing multiplicity in each time slot obtained in (1) is added.

  For example, when the application log shown in FIG. 4B is input to the state index calculation means 105, the result of quantization in a time slot of 0.1 second is as shown in FIG. 7, and the integrated value of the communication processing multiplicity is 5481. Given. When the integrated value of the communication processing multiplicity is used as the state index, the larger the value, the worse the network state. This is because the area surrounded by the graph of FIG. 7 increases horizontally as a result of the communication processing time extending due to the deterioration of the network state. Thus, a large integrated value of the communication processing multiplicity corresponds to a bad network state. In addition, it is assumed that the good / bad state condition is defined in advance.

  In steps S <b> 101 to S <b> 107, the target user has been described. Similarly, for the logs of other users to be compared with the target user, similarly, from the logs of the terminals of each user equipped with the transmission unit 201 and the log acquisition unit 203, It is assumed that time is estimated, a state index is calculated, and stored in the storage unit 111.

  In step S109, the similar operation extraction unit 107 uses the feature amount representing the operation content calculated from the extracted log (hereinafter referred to as the operation feature amount), and the past experience waiting time stored in the storage unit 111. From the state index, a set of the sensation waiting time and the state index when an operation similar to the operation of the target user is executed is extracted. There may be a plurality of sets of sensation waiting times and state indicators extracted from the storage unit 111.

  The operation feature amount is calculated based on the operation feature amount and calculation method defined in advance by the similar operation extraction unit 107. For example, as the operation feature amount, as in Non-Patent Document 5, the amount of received data, the number of peaks of processing multiplicity for each processing type, height, width, position, or a combination thereof may be used. As in Non-Patent Document 6, a method name, a URL, or the like may be used.

  Further, when the received data amount of Non-Patent Document 5 is used as the operation feature amount, the received data amount is calculated by summing up the received data amount within the sensation waiting time using a network log. The number, height, width, and position of the processing multiplicity peak are determined from the graph shape obtained by quantizing the application log.

  For example, a specific procedure for obtaining the number, height, width, and position of processing multiplicity peaks is as follows.

  (1) Quantize the application log at regular time intervals. Quantization divides the application log into time slots of a certain time in a time series, and for each process of communication processing, script processing, and drawing processing performed in this time slot, the total number of processing executions (processing multiplicity) is quantized. A method of calculating as a conversion value is used. For example, the time slot may be determined by a relative time such as “the length of the waiting time of the experience ÷ 50” or may be determined by an absolute time such as “0.1 second”. Alternatively, a plurality of time slots may be defined, and for each time slot, the multiplicity of each process of the communication process, script process, and drawing process may be quantized as a value.

(2) From the result of quantization in (1) above, the peak of multiplicity of each process of communication processing, script processing, and drawing processing is obtained for each time slot. However, when the interval from the peak end time to the next peak start time is equal to or less than _Sinter , the latter peak is not included in the number of peaks.

In the state where the peak has not been started, the time when the processing multiplicity becomes P _min1 or more for the first time is determined as the peak start time, and the time when the processing multiplicity becomes P _min2 or less after the start of the peak is determined as the peak end time . In addition, a peak height of P _max or less is not counted as a peak. Here, the peak height is the maximum multiplicity of processing from the peak start time to the peak end time. There are several other ways of thinking about peak start and end times. For example, a peak start when the slope of the quantization graph is greater than or equal to a certain angle, and a peak end when the gradient is less than or equal to a certain angle, or a point where the graph has turned from decrease to increase are referred to as peak start and end. There are ways to think.

Note that the values of P _min1 , P _min2 , P _{max, and} S _inter are registered in advance in the storage unit 111 and can be appropriately modified as necessary.

  (3) The peak number, the height, the width, and the position, which are the number of occurrences of the multiplicity peak in each of the communication process, script process, and drawing process obtained in (2) above, are as follows.

  (3-1) Number: The number of peaks is calculated.

  (3-2) Height: The peak height is calculated in the form of (first peak height, second peak height,...).

  (3-3) Width: The peak width is calculated in the form of (first peak width, second peak width,...). The peak width is determined by (peak end time) − (peak start time) at each peak.

  (3-4) Position: The peak position is calculated in the form of (first peak position, second peak position,...).

  As for the operation feature amount, it is assumed that the operation feature amount of another user to be compared with the target user is stored in the storage unit 111 in the same manner as the set of the sensation waiting time and the state index.

  Regarding the similar determination in the similar operation extraction unit 107, when using a combination of the received data amount and the peak number / height of the processing multiplicity as the operation feature amount as in Non-Patent Document 5, the similar operation extraction unit 107 Then, the degree of similarity between the operation feature amounts is evaluated by a predetermined similarity function, and those whose degree of similarity is equal to or higher than the threshold or those higher than the ratio of the value indicated by the threshold based on a preset threshold are extracted. As shown in Non-Patent Document 5, the similarity function is determined by a weighted sum of similarities between operation feature amounts.

  On the other hand, when the URI or method name is used as the operation feature amount as in Non-Patent Document 6, the similar operation extraction unit 107 extracts the operation feature amount that completely matches.

  For example, it is assumed that a combination of a sensation waiting time and a state index corresponding to an operation similar to the target user as illustrated in FIG. 8 is extracted by evaluating the similarity between the operation feature amounts.

  In step S111, the quality map display unit 109 detects the sensation waiting time in the past operation estimated in step S103, the state index calculated in step S107, and the past operation extracted in step S109. A quality map of a user who has performed an operation similar to that of the target user is displayed using a set of the status indicator. Thereby, since the sensation waiting time of another user and a target user can be compared, the quality of the sensation quality of the target user can be relatively determined. Moreover, the quality degradation factor can also be identified by comparing the state index of the target user and the user having a higher quality of experience.

  Specifically, the quality map display means 109 takes the state index on the x-axis on the xy plane and the sensation waiting time on the y-axis, and plots a set of the state index and the sensation waiting time on the plane. When both the network status index and the terminal status index are used as the status index, the network status index is taken on the x-axis, the terminal status index is taken on the y-axis, and a set of two status indices is placed on the plane. Plots may be drawn by connecting the plots with equal bodily sensation waiting lines, and contour lines may be drawn, the network state index on the x-axis in the xyz space, the terminal state index on the y-axis, the bodily sensation waiting time on the z-axis, You may plot the set of 2 types of state parameter | indexes and sensory waiting time on the said space. In any case, the map representation method is not limited to this.

  For example, for example, the sensory waiting time in FIG. 6 (October 1, 2012 16: 34: 34.7 seconds-October 1, 2012 16: 33: 58.4 seconds = 36.3 seconds) and the state index obtained in step S107 ( For example, it is assumed that a set of 5481) and a sensation waiting time and a state index as shown in FIG. 8 are given in step S109. At this time, the quality map of the user who performed an operation similar to the target user is plotted as shown in FIG.

  From the quality map of FIG. 9, it can be seen that the waiting time for the subject user is longer than that for other users, and the quality is deteriorated. Furthermore, since the value of the network status index of the target user is large, it can be understood that the deterioration factor is in the network.

  On the other hand, for example, in the case of user A, the waiting time for sensation is long, but there are users whose sensation waiting time is 1/3 or less even with the same network state index value. You can see that it is also on the terminal. In this way, in the quality map, it is possible to detect quality degradation and to identify degradation factors.

<Second embodiment>
In the second embodiment, in addition to the first embodiment, the related information such as the user's usage environment is displayed on the quality map to detect the quality deterioration and identify the deterioration factor.

  FIG. 10 shows a configuration diagram of a system including a quality evaluation apparatus according to the second embodiment of the present invention. The system according to the second embodiment of the present invention includes a quality evaluation apparatus 10, a user terminal 20, and an application (AP) server 30 as in the first embodiment. The user terminal 20 and the AP server 30 are the same as in the first embodiment. However, it is assumed that the log acquisition unit 203 of the user terminal 20 acquires information on the usage environment when the user performs an operation.

  The quality evaluation apparatus 10 according to the second embodiment further includes a filtering unit 113 in addition to the quality evaluation apparatus 10 according to the first embodiment. The bodily sensation waiting time estimation unit 101, the log cutout unit 103, the state index calculation unit 105, the similar operation extraction unit 107, the quality map display unit 109, and the storage unit 111 are configured in the same manner as in the first embodiment, except as described below. Is done.

  In addition, the storage unit 111 stores information related to the usage environment when the user performs an operation, together with a set of the sensation waiting time and the state index. The related information such as the usage environment may be stored in the storage unit 111 together with the start time and end time of the sensation waiting time by the sensation waiting time estimation unit 101. The usage environment includes, for example, information such as a usage line, a terminal model name, a location, and a time when an operation is performed.

  When the quality map display unit 109 displays a set of the sensation waiting time and the state index of the target user on the map, the quality map display unit 109 tags each pair of the sensation waiting time and the state index with relevant information such as a use environment. To do.

  The filtering unit 113 performs filtering according to a predetermined condition, and displays a set of a sensation waiting time and a state index satisfying a predetermined condition on the map. The predetermined condition is set by the user or the like in the filtering means 113 and can be changed flexibly.

  Next, the operation of the quality evaluation apparatus 10 according to the second embodiment will be described in detail with reference to FIGS. FIG. 11 is a flowchart of the quality display method in the quality evaluation apparatus 10 according to the second embodiment of the present invention.

  Steps S101 to S109 are performed in the same manner as in the first embodiment. However, in step S101, the sensation waiting time estimation unit 101 acquires the user's usage environment acquired by the log acquisition unit 203, and in step S103 stores the usage environment of the target user when the sensation waiting time is estimated. Save in means 111.

  For example, when the target user accesses the AP server 30 from the public wireless LAN in front of Shibuya Station using the smartphone X, the usage environment is stored in the storage unit 111 as shown in FIG.

  Similarly, the usage environment of other users is also stored in the storage unit 111 together with the combination of the waiting time and the state index as shown in FIG.

  In step S111, the quality map display unit 109 detects the sensation waiting time in the past operation estimated in step S103, the state index calculated in step S107, and the past operation extracted in step S109. A quality map of a user who has performed an operation similar to that of the target user is displayed using a set of the status indicator. At this time, the quality map display means 109 not only plots a set of the sensation waiting time and the state index, but also tags each plot with related information such as the usage environment.

  For example, the sensation waiting time (36.3 seconds) and state index (5481) of the target user and the usage environment shown in FIG. 12 are given, and the sensation waiting time and state index extracted by the similar operation extraction unit 107 shown in FIG. Suppose that the group and usage environment are given. At this time, as shown in FIG. 14, the quality map display means 109 displays a quality map tagged with the usage environment.

  In step S113, the filtering unit 113 filters the quality map displayed in step S111 according to the specified condition, and displays only plots that satisfy the condition. For example, as the filtering condition, the usage environment tagged with the quality map display means 109, the waiting time for the experience, the value of the state index, and the like are designated. Thereby, it is possible to grasp not only the quality deterioration factor but also a specific quality improvement method. The filtering condition is set in the filtering unit 113, and the condition can be changed flexibly.

  For example, FIG. 15 shows a result when filtering is performed under the condition of “terminal model name = smart phone X, location = Shibuya station square” which is the same terminal model name and the same location as the target user. From the filtering result of FIG. 15, it can be seen that when the target user continues to use the application in front of Shibuya Station, the quality of experience can be improved by switching the use line from the public wireless LAN to LTE.

<Third embodiment>
In the first and second embodiments described above, quality degradation factors are identified and a quality improvement method is identified by visually observing the quality map. In the third embodiment, this is automatically performed.

  FIG. 16 shows a configuration diagram of a system including a quality evaluation apparatus according to the third embodiment of the present invention. The system according to the third embodiment of the present invention includes a quality evaluation apparatus 10, a user terminal 20, and an application (AP) server 30 as in the first embodiment. The user terminal 20 and the AP server 30 are the same as in the first embodiment. Similarly to the second embodiment, the log acquisition unit 203 of the user terminal 20 may acquire information related to the usage environment when the user performs an operation.

  In addition to the quality evaluation apparatus 10 according to the first embodiment, the quality evaluation apparatus 10 according to the third embodiment further includes a degradation factor isolation unit 115 and an improvement method proposal unit 117. The bodily sensation waiting time estimation unit 101, the log cutout unit 103, the state index calculation unit 105, the similar operation extraction unit 107, the quality map display unit 109, and the storage unit 111 are configured in the same manner as in the first embodiment or the second embodiment. .

  The degradation factor separation unit 115 uses the combination of the sensation waiting time and the state index of the target user and the combination of the sensation waiting time and the state index extracted by the similar operation extraction unit 107 to determine the quality degradation factor of the target user. Carve out.

  The improvement method proposing means 117 proposes a method for improving the quality of the target user according to the result of the degradation factor isolating means 115.

  Next, the operation of the quality evaluation apparatus 10 according to the third embodiment will be described in detail with reference to FIG. FIG. 17 is a flowchart of the quality display method in the quality evaluation apparatus 10 according to the third embodiment of the present invention.

  Steps S101 to S111 are performed in the same manner as in the first embodiment or the second embodiment. As in the second embodiment, in step S101, the sensation waiting time estimation unit 101 acquires the user's usage environment acquired by the log acquisition unit 203, and in step S103, the object when the sensation waiting time is estimated. The user's usage environment may be stored in the storage unit 111.

  In step S115, the degradation factor classification unit 115 classifies the degradation factor of the target user according to the following procedure.

  (1) From the pair of past experience waiting times estimated by the similar operation extraction unit 107 estimated to be similar and the state index, a set having the shortest experience waiting time (hereinafter referred to as the best plot) is extracted.

  (2) Comparing the target user's sensation waiting time with the best plot's sensation waiting time, if the target user is longer than the threshold value α, it is determined that there is quality degradation; otherwise, it is determined that there is no quality degradation.

  (3) When it is determined that there is quality degradation in (2), the state index of the target user is compared with the state index of the best plot. If the best plot is longer than the threshold β, it is a network factor. If so, isolate it as a terminal factor.

  It should be noted that a plurality of sets may be extracted as the best plot (for example, the top n or top x% with the short sensation waiting time). The threshold α may be determined by a relative value such as “130% or more of the target user's state index”, or may be determined by an absolute value such as “100”. Similarly, the threshold value β may be determined as a relative time such as “125% or more of the target user's sensation waiting time”, or may be determined as an absolute time such as “0.5 seconds”. When multiple groups are extracted as the best plot and the network parameters and terminal factors are mixed as a result of comparing each group with the target user's state index, the degradation factors are the network factors and terminal factors. A method of deciding by majority decision, a method of deciding on the one with the larger sum of the differences in sensation waiting time (experience waiting time of best plot-sensation waiting time of target user), and the like are conceivable.

  In addition, as preprocessing, as described in Non-Patent Document 7, an outlier detection method based on statistics (Distribution-Based approach), an outlier detection method based on distance (Distance-Based approach), etc. A set of state indicators may be excluded in advance, or a set that does not satisfy a predetermined condition may be excluded in advance by filtering using information on the usage environment. The distribution-based approach is a method for detecting a plot deviating from a set probability distribution model as an outlier on the assumption that the target plot follows a probability distribution model. The distance-based approach calculates the distance between each target plot Oi and the remaining plots. If M or more plots are found in the vicinity of D, it is determined that Oi is not a DB outlier at that time. And then reject and start processing the next plot. In the distance-based approach, if all the plots in S are scanned and the number of plots near D is M or less, the plot Oi is determined to be an outlier.

  For example, the sensation waiting time (36.3 seconds) and state index (5481) of the target user and the usage environment shown in FIG. 12 are given, and the sensation waiting time and state index extracted by the similar operation extraction unit 107 shown in FIG. Suppose that the group and usage environment are given.

  At this time, the shortest sensation waiting time is 3.6 seconds, and the state index paired therewith is 585, so the deterioration factor separating means 115 extracts this pair as the best plot. Comparing the target user's experience waiting time with the best plot, if the threshold α is set to “125% or more of the target user's experience waiting time”, the target user is judged to have quality degradation from 36.3> 3.6 * 1.25 . Then, comparing the target user with the status index of the best plot, if the threshold is set to `` 130% or more of the target user's status index '', the quality degradation factor of the target user is in the network from 5481> 585 * 1.3 Carved.

  In step S117, the improvement method proposing means 117 proposes a method for improving the quality of the target user according to the result of the degradation factor categorizing means 115, using related information such as the user's usage environment saved in the saving means 111, for example. To do.

  Specifically, when it is separated from the terminal factor in the degradation factor separation means 115, the best plot and the terminal model name of the target user are compared, and in the case of a different model, the best plot is changed to the same model. If it is a model, we suggest closing the background application or uninstalling unnecessary applications. Also, when the degradation factor isolating means 115 identifies the network factor, the best plot is compared with the usage line, location, time, etc. of the target user, and a change to the same usage environment as the best plot is proposed.

  For example, the improvement method proposing means 117 compares the target user with the use line, location, time, etc. of the best plot according to the result of the deterioration factor isolating means 115 in the above example, switches the use line to LTE, and Yoyogi Park We suggest moving to the neighborhood as a quality improvement method. The improvement method proposing means 117 may display the quality improvement method on the screen of the user terminal 20 or may transmit a message indicating the quality improvement method to the user terminal 20.

  In the first to third embodiments of the present invention, in the quality evaluation apparatus 10, the sensation waiting time estimation unit 101 estimates the sensation waiting time, and the log cutout unit 103 records the log from the start time of the sensation waiting time to the end time. The example in which the state index is calculated based on the log extracted by the state index calculation unit 105 has been described. However, the present invention is not limited to the first to third embodiments described above. For example, the state index calculation unit 105 is based on a log cut out by another device or a log cut out by the state index calculation unit 105 itself. The state index may be calculated.

<Effect of Example>
According to the embodiment of the present invention, it is possible to detect quality deterioration based on the user experience and to identify the deterioration factor. By analyzing the log in the user terminal without performing conventional threshold monitoring, it is possible to detect quality degradation and to identify degradation factors. Also, in the embodiment of the present invention, it is possible to grasp the waiting time and network state of other users who are performing similar operations, the use line, the terminal model name, the location, etc. It is also possible to grasp a specific quality improvement method by referring to the use line, terminal model name, location, etc. of the user in good condition.

  As described above, in the embodiment of the present invention, the monitoring threshold value is set in advance by comparing the index and the waiting time for the sensation calculated from the operation log acquired at the user terminal between users performing similar operations. Therefore, it becomes possible to detect quality deterioration and to identify the cause of deterioration.

  For convenience of explanation, the quality evaluation apparatus according to the embodiment of the present invention is described using a functional block diagram. However, the quality evaluation apparatus according to the embodiment of the present invention may be hardware, software, or a combination thereof. It may be realized. In addition, the functional units may be used in combination as necessary. Further, although the method according to the embodiment of the present invention has been described using the flowchart showing the flow of processing, the method according to the embodiment of the present invention may be performed in an order different from the order shown in the embodiment. .

  As mentioned above, although the method for detecting quality degradation based on the user experience and identifying the degradation factor has been described, the present invention is not limited to the above-described embodiments, and various modifications and changes are within the scope of the claims. Application is possible.

101 bodily sensation waiting time estimation means 103 log cutout means 105 state index calculation means 107 similar operation extraction means 109 quality map display means 111 storage means 113 filtering means 115 deterioration factor separation means 117 improvement method proposal means

Claims (7)

A quality evaluation device that evaluates the quality at the time of execution of an application,
Indicates at least one of a network state and a terminal state at the time of application execution based on a log cut out from an application log output by the application corresponding to a period from the start time to the end time of the user experience waiting time State index calculating means for calculating the state index;
Storage means for storing the sensory waiting time and the calculated state index as a set;
Similar operation extraction means for extracting a set of a sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage means;
A quality map display means for displaying a set of the sensory waiting time and the calculated state index on a map together with a set of the sensory waiting time and the state index corresponding to the extracted similar operation ;
Having a quality evaluation device. The storage means stores information about a use environment when an operation is performed together with a set of the sensory waiting time and the calculated state index,
The quality map display means displays a use environment in association with a set of a sensation waiting time and a state index,
The quality evaluation device, to the set of the experience latency and state index displayed in the quality map display means further comprises a filtering means for displaying on the map a predetermined condition is satisfied set to claim 1 The quality evaluation apparatus described. From the extracted set of sensory waiting times and state indicators, one or more sets of sensory waiting times and state indicators are extracted from the shorter sensory waiting times, and the extracted sensory waiting times and states are extracted. based on the comparison of the set of the indices further comprises a degradation factor cut means to isolate the quality deterioration factor for the operation of the user, the quality evaluation apparatus according to claim 1 or 2. The quality evaluation apparatus according to claim 3 , further comprising a quality improvement method proposing unit that proposes a quality improvement method using a use environment stored in the storage unit according to the separated quality deterioration factor. It said state index calculation means divides the cut log at a fixed time of a time slot, summed to calculate the number of processing related to communication performed within the time slot, either one of claims 1 to 4 1 The quality evaluation apparatus according to item. A quality evaluation method in a quality evaluation apparatus that evaluates the quality at the time of execution of an application,
The network at the time of application execution based on a log cut out from the application log output by the application corresponding to the period from the start time to the end time of the user experience waiting time Calculating a state index indicating at least one of a state and a terminal state;
Storing the bodily sensation waiting time and the calculated state index in a storage unit;
Extracting a set of sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage unit;
Displaying a set of the sensory waiting time and the calculated state index on a map together with a set of the sensory waiting time and the state index corresponding to the extracted similar operation ;
Having a quality evaluation method. A program for causing a computer to function as a quality evaluation device that evaluates the quality at the time of execution of an application. The computer is output by the application corresponding to the period from the start time to the end time of the user's experience waiting time. A state index calculating unit that calculates a state index indicating at least one of a network state and a terminal state at the time of application execution based on a log cut out from the application log;
Storage means for storing the sensory waiting time and the calculated state index as a set;
Similar operation extraction means for extracting a set of a sensation waiting time and a state index corresponding to an operation similar to the user's operation from the storage means, and a set of the sensation waiting time and the calculated state index , A quality map display means for displaying on a map together with a set of a sensation waiting time and a state index corresponding to the extracted similar operations ;
Program to function as.

Images