JP5483780B1 - Quality attribute identification system for service quality information and quality attribute identification method for service quality information - Google Patents

Abstract

A service quality information measurement value is specified according to a quality attribute.
A quality attribute specifying system for service quality information disclosed in the present application comprises an evaluation device for evaluating the quality of a service. The evaluation device includes a calculation unit, a classification unit, and a selection unit. The calculation unit calculates the accuracy of the classification result obtained by classifying the data included in the item-specific service quality information obtained by evaluating the quality of the service provided by the service providing system for each item into a plurality of sets based on the function. When the accuracy is equal to or higher than a predetermined threshold, the classification unit classifies data included in other item-specific service quality information into a plurality of sets based on the classification result. The selection unit calculates the accuracy of the classification result by the classification unit based on the function, and selects the classification result that maximizes the calculated accuracy for each item.
[Selection] Figure 1

Description

  The present invention relates to a quality attribute specifying system for service quality information and a quality attribute specifying method for service quality information.

  In recent years, as represented by cloud computing, information systems (service providing systems) that provide services via a network have been widely used. For example, a service providing system includes IaaS (Infrastructure as a Service) that provides a user with a virtual machine via a network.

  There is also a method for measuring service quality information indicating the service quality provided by the service providing system. For example, in IaaS, the service quality information includes items such as a CPU (Central Processing Unit) processing speed, a memory transfer speed, and a network bandwidth. For example, there is a technique in which an operator of a service providing system measures service quality information by executing a benchmark on the service providing system. In addition, for example, there is a technique in which a user who uses a service measures service quality information by executing a benchmark on the service providing system. Further, for example, there is a method of measuring service quality information by a third evaluation organization executing a benchmark. Note that the service quality information disclosed by the service providing system 200 is referred to as public data, and the service quality information measured by the third evaluation organization executing a benchmark is referred to as audit data. In the following, a system operator who acquires and discloses public data is called a publisher, and an evaluation organization or user who acquires and verifies audit data is called an auditor.

  In addition, the service quality of the service providing system may change greatly every time the service is used depending on the internal configuration of the service providing system or the usage status of other users. For example, when using a virtual machine provided by an IaaS service, it is known that the performance of the virtual machine differs depending on the type of hardware (hereinafter referred to as a quality attribute) assigned at the start of use. For example, the quality attribute includes a CPU type, a network adapter type, and the like (Non-Patent Document 1).

  Here, every time the publisher performs quality measurement of the system for public data, for example, the auditor obtains the quality attribute at the time of measurement and publishes the public data in which the quality attribute is associated with each measurement value. The service quality information measurement value can be specified according to the quality attribute. In other words, the auditor can obtain appropriate service quality information obtained by classifying measured values of service quality information according to quality attributes.

Jorg Schad et.al., "Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance", in Proceedings of the VLDB Endowment, Vol. 3, No. 1. Yuichi Suto, Kunio Hado, Hidetsugu Kobayashi, Eiji Kuwana, “Performance Measurement Method in Intercloud”, IEICE Technical Report, Vol. 112, No. 393, IN2012-153, January 2013.

  However, there is a problem that the measurement value of the service quality information may not be specified according to the quality attribute. Specifically, in order to classify the measurement values based on the quality attribute, it is necessary to include the quality attribute in public data or audit data. On the other hand, for example, a publisher may hate to know the hardware configuration of an IaaS system by policy and may not include a quality attribute in public data. Also, for example, the quality attribute may be hidden by the system and cannot be observed when the auditor performs quality measurement.

  Even if the quality attribute is not included in the service quality information, each measurement value may be classified according to the quality attribute by a machine learning clustering technique represented by k-means or the like. For example, when service quality information measurement values differ greatly depending on quality attributes, the time series data obtained by quality measurement of the service providing system has multiple sets of measurement values obtained. It can be specified according to the attribute. However, when the difference in the measured value of the service quality information is small depending on the quality attribute, accurate classification becomes difficult, and thus the measured value of the service quality information cannot be specified according to the quality attribute.

  The disclosed technique has been made in view of the above, and an object thereof is to specify a measurement value of service quality information according to a quality attribute.

  The quality attribute identification system for service quality information to be disclosed includes, in one aspect, a service providing system that provides a service via a network and an evaluation device that evaluates the quality of the service. The evaluation device includes a calculation unit, a classification unit, and a selection unit. The calculation unit calculates the accuracy of the classification result obtained by classifying the data included in the item-specific service quality information obtained by evaluating the quality of the service provided by the service providing system for each item into a plurality of sets based on the function. When the accuracy is equal to or higher than a predetermined threshold, the classification unit classifies data included in other item-specific service quality information into a plurality of sets based on the classification result. The selection unit calculates the accuracy of the classification result by the classification unit based on the function, and selects the classification result that maximizes the calculated accuracy for each item.

  According to one aspect of the disclosed service quality information providing system, there is an effect that the measurement value of the service quality information can be specified according to the quality attribute.

FIG. 1 is a block diagram illustrating an example of a configuration of a service quality information providing system according to the first embodiment. FIG. 2 is a diagram illustrating an example of public data according to the first embodiment. FIG. 3 is a diagram illustrating an example of public data according to the first embodiment. FIG. 4 is a diagram illustrating an example of audit data according to the first embodiment. FIG. 5 is a diagram illustrating an example of audit data according to the first embodiment. FIG. 6 is a diagram illustrating an example of a processing operation performed by the calculation unit. FIG. 7 is a diagram illustrating an example of a processing operation performed by the classification unit. FIG. 8 is a diagram illustrating an example of a classification result by the calculation unit and the classification unit. FIG. 9 is a diagram illustrating an example of a processing operation performed by the selection unit. FIG. 10 is a diagram illustrating a case where the public data according to the first embodiment is classified. FIG. 11 is a diagram illustrating a case where audit data is classified in the first embodiment. FIG. 12 is a flowchart illustrating an example of a process flow by the first measurement unit in the first embodiment. FIG. 13 is a flowchart illustrating an example of a flow of processing by the evaluation apparatus side control unit in the first embodiment. FIG. 14 is a flowchart illustrating an example of a process flow of classification according to the first embodiment. FIG. 15 is a diagram showing that the information processing by the quality attribute specifying program of the service quality information for executing the processing by the service quality information providing system is specifically realized using a computer.

  Hereinafter, embodiments of a quality attribute specifying system for service quality information and a quality attribute specifying method for service quality information to be disclosed will be described in detail based on the drawings. The invention disclosed by this embodiment is not limited. Each embodiment can be appropriately combined as long as the processing contents do not contradict each other.

(First embodiment)
FIG. 1 is a block diagram illustrating an example of a configuration of a service quality information providing system according to the first embodiment. In the example illustrated in FIG. 1, the service quality information providing system includes a user terminal 100, a service providing system 200, and an evaluation device 300. In the example illustrated in FIG. 1, in the service quality information providing system, the user terminal 100, the service providing system 200, and the evaluation device 300 are connected via an arbitrary network 10.

  The user terminal 100 uses a service provided by the service providing system 200. For example, the user terminal 100 uses resources provided by the service providing system 200. Here, the resource provided by the service providing system 200 is, for example, a virtual machine.

  In addition, the user terminal 100 acquires public data disclosed by the service providing system 200 or acquires a test result provided by the evaluation device 300. The details of the public data disclosed by the service providing system 200 and the details of the test result provided by the evaluation apparatus 300 will be described later, and thus the description thereof will be omitted.

  The service providing system 200 provides a service via the network 10. For example, the service providing system 200 provides virtual resources to the user terminal 100 via the network 10. For example, the service providing system 200 provides a virtual machine as a resource.

  Further, the service providing system 200 measures the first item-by-item service quality information obtained by evaluating the quality of the service to be provided at the first frequency for each item, and discloses the measured first item-by-item service quality information. . The service quality information is measured, for example, by executing a benchmark. Here, the service quality includes items such as a CPU (Central Processing Unit) processing speed, a memory transfer speed, and a network bandwidth. In the following description, the first item of service quality information is also referred to as “public data”. Details of the service providing system 200 will be described later.

  The evaluation device 300 evaluates the quality of service provided by the service providing system 200. For example, the evaluation apparatus 300 uses the second item-specific service quality information obtained by evaluating the quality of the service provided by the service providing system 200 for each item at a second frequency that is less frequent than the first frequency. taking measurement. In the following description, the second item-specific service quality information is also referred to as “audit data”.

  Moreover, the evaluation apparatus 300 acquires the publicly disclosed data. Then, the evaluation device 300 evaluates the credibility of the public data by comparing the public data with the audit data, and provides the evaluation result to the user terminal 100. Details of the evaluation apparatus 300 will be described later.

  In the first embodiment, a case where the evaluation apparatus 300 that is a terminal different from the user terminal 100 executes a series of processes described in detail below will be described as an example, but the present invention is not limited thereto. is not. For example, the user terminal 100 and the evaluation device 300 may be the same terminal. In other words, the user terminal 100 using the service provided by the service providing system 200 may verify the authenticity of the public data by executing a series of processes described in detail below.

  As illustrated in FIG. 1, the service providing system 200 includes, for example, a system-side communication unit 201, a system-side public data DB 210, and a system-side control unit 220. The system side communication unit 201 is connected to the system side control unit 220. The system side communication unit 201 controls communication of the service providing system 200. For example, when the system-side communication unit 201 receives an instruction or information from an administrator in the service providing system 200 or receives an instruction or information from the user terminal 100 or the evaluation apparatus 300 via the network 10, the received instruction Or the information is transmitted to the system-side control unit 220. Further, when the system side communication unit 201 receives information from the system side control unit 220, the system side communication unit 201 transmits the received information to the administrator or transmits it to the user terminal 100 or the evaluation apparatus 300 via the network 10. Note that details of instructions and information transmitted and received by the system-side communication unit 201 will be described when each related unit is described as appropriate.

  Further, in the first embodiment, a case will be described in which there are a plurality of types of resources provided by the service providing system 200 and a difference occurs in item-specific service quality information depending on the type of resources. In the following, the case where the service providing system 200 is configured with two types of hardware (for example, CPU1 and CPU2) and the performance of the virtual machine depends greatly on the type of hardware (hereinafter also referred to as quality attribute) will be described. To do. In other words, a case will be described in which a difference occurs in item-based service quality information depending on quality attributes. Here, the quality attribute includes a CPU type, a network adapter type, and the like. Note that the types of hardware are not limited to two, and may be any number. The identification information for identifying the performance of the resource of the service providing system used when measuring the service quality information for each item is called “attribute information”.

  In the first embodiment, in the service providing system 200, the operator does not disclose the hardware type of the sample in the public data for some reason, or the evaluation organization cannot confirm the hardware type of the virtual machine. Is assumed.

  The system side public data DB 210 is connected to the system side control unit 220. The system-side public data DB 210 is, for example, a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. The system-side public data DB 210 stores each first item of service quality information measured at the first frequency by the system-side control unit 220.

  2 and 3 are diagrams illustrating an example of public data according to the first embodiment. In the example shown in FIG. 2, the public data item is the CPU processing speed, and in the example shown in FIG. 3, the public data item is the memory transfer speed. As will be described later, the public data is first item-specific service quality information in which the quality of the service provided by the service providing system 200 by the first measuring unit 221 is evaluated for each item at the first frequency. is there.

  As shown in FIGS. 2 and 3, the system-side public data DB 210 stores a benchmark score, which is a value indicating the quality of service, as item-specific service quality information in association with the measurement time. In the example shown in FIGS. 2 and 3, the benchmark score indicates that the larger the value is, the higher the service quality is, and the smaller the value is, the lower the service quality is.

  2 includes a set in which benchmark score values are distributed between about 20 and about 40 and a set in which benchmark score values are distributed between about 80 and about 120. . As shown in FIG. 2, when two sets are included, it is possible to specify each measured value according to the quality attribute even by a machine learning clustering technique represented by k-means or the like. On the other hand, the item-by-item service quality information shown in FIG. 3 includes a set in which benchmark score values are distributed between about 60 and about 120. As shown in FIG. 3, when there is actually two sets but the difference between the measured values is small and apparently includes one set, the machine learning clustering technique represented by k-means etc. The value cannot be specified according to the quality attribute.

  The system side control unit 220 is connected to the system side public data DB 210 and the system side communication unit 201. The system-side control unit 220 has an internal memory that stores a program that defines various processing procedures and the like, and controls various processes. The system-side control unit 220 is, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a CPU, or a micro processing unit (MPU). In the example illustrated in FIG. 1, the system side control unit 220 includes a first measurement unit 221 and a disclosure unit 222.

  The first measuring unit 221 measures item-specific service quality information indicating the quality of service for each item at a first frequency. The first measurement unit 221 assumes a plurality of evaluation items such as a CPU processing speed, a memory transfer rate, and a network bandwidth, and sets time series data obtained for each evaluation item as X1, X2,. Then, the first measuring unit 221 stores the measured item-specific service quality information in the system-side public data DB 210. Specifically, the first measurement unit 221 periodically executes a benchmark using resources provided by the service providing system 200 and stores the measurement result in the system-side public data DB 210.

  For example, the first measurement unit 221 starts five virtual machines provided by the service providing system 200 four times a day as the first frequency, and executes the benchmark. In other words, in this case, the first measurement unit 221 measures 500 pieces of item-specific service quality information for each item in 25 days.

  To explain with a more detailed example, the first measurement unit 221 causes the virtual machine provided as a resource by the service providing system 200 to execute a benchmark, thereby processing the CPU processing speed and memory transfer speed of the virtual machine, Measure disk read / write speed, network communication speed, etc.

  The disclosure unit 222 publishes first item-by-item service quality information that is each item-by-item service quality information measured at the first frequency by the first measurement unit 221. Specifically, the publishing unit 222 publishes each item-specific service quality information stored in the system-side public data DB 210 to the user terminal 100 or the evaluation apparatus 300. For example, when there is a request from the user terminal 100 and the evaluation device 300, the disclosure unit 222 transmits information stored in the system side public data DB 210. Hereinafter, the first item of service quality information is referred to as “public data”.

  As described above, in the service providing system 200, the system-side control unit 220 periodically executes a benchmark using, for example, resources provided by the service providing system 200, and obtains the obtained measurement data in the system-side public data DB 210. To store. Further, in the system-side control unit 220, the disclosure unit 222 always discloses the contents of the system-side public data DB 210, and when there is a request from the user terminal 100 and the evaluation device 300, the system-side public data DB 210 Send memorized public data. That is, for example, as shown in the first measurement unit 221 and each point in FIG. 2, the performance value of the service quality is periodically measured, and when the public unit 222 has the performance value as shown in FIG. Publish the series data itself.

  Returning to FIG. 1, the evaluation apparatus 300 includes, for example, an evaluation apparatus side communication unit 301, an evaluation apparatus side public data DB 310, an evaluation apparatus side audit data DB 311, and an evaluation apparatus side control unit 320. The evaluation device side communication unit 301 controls communication of the evaluation device 300. For example, the evaluation apparatus-side communication unit 301 transmits various requests to the service providing system 200, receives public data from the service providing system 200 via the network 10, and displays the verification result of the authenticity of the public data on the user terminal. To 100.

  The evaluation device side public data DB 310 is connected to the evaluation device side control unit 320. The evaluation apparatus side public data DB 310 is, for example, a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. The evaluation device side public data DB 310 stores the public data that is disclosed by the service providing system 200 and acquired by the evaluation device 300. In other words, the evaluation apparatus side public data DB 310 stores the first item-by-item service quality information measured at the first frequency by the first measurement unit 221.

  The evaluation apparatus side audit data DB 311 is connected to the evaluation apparatus side control unit 320. The evaluation apparatus side audit data DB 311 is, for example, a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. The evaluation device side audit data DB 311 stores each second item of service quality information measured at the second frequency by the evaluation device side control unit 320.

  4 and 5 are diagrams illustrating an example of audit data according to the first embodiment. In the example shown in FIG. 4, the audit data item is the CPU processing speed, and in the example shown in FIG. 5, the audit data item is the memory transfer speed. As shown in FIGS. 4 and 5, the evaluation apparatus-side audit data DB 311 stores a benchmark score, which is a value indicating the quality of the service, as itemized service quality information in association with the measurement time. Each of the second item-by-item service quality information measured at the second frequency by the evaluation apparatus side control unit 320 is also referred to as “audit data”.

  Here, the public data stored in the system side public data DB 210 and the evaluation apparatus side public data DB 310 and the audit data stored in the evaluation apparatus side audit data DB 311 will be further described. The audit data is measured based on the control and instruction of the evaluation apparatus 300. On the other hand, the public data is voluntarily measured in the service providing system 200. 4 and 5, the second frequency, which is the frequency at which each item of service quality information in the audit data is measured, is the frequency at which each item of service quality information in the public data is measured. Small compared to a certain first frequency. In other words, as shown in FIG. 4 and FIG. 5, the number of item-by-item service quality information per hour is larger in the public data than in the audit data.

  4 includes a set in which benchmark score values are distributed between about 20 and about 40, and a set in which benchmark score values are distributed between about 80 and about 120. . As shown in FIG. 4, when two sets are included, it is possible to specify each measured value according to a quality attribute even by a machine learning clustering technique represented by k-means or the like. On the other hand, the item-specific service quality information shown in FIG. 5 includes a set in which benchmark score values are distributed between about 60 and about 120. As shown in FIG. 5, when there are actually two sets but the difference between the measured values is small and apparently includes one set, the machine learning clustering technique represented by k-means etc. The value cannot be specified according to the quality attribute.

  The evaluation apparatus side control unit 320 is connected to the evaluation apparatus side communication unit 301, the evaluation apparatus side public data DB 310, and the evaluation apparatus side audit data DB 311. The evaluation device side control unit 320 is, for example, an ASIC, FPGA, CPU, MPU, or the like. In the example illustrated in FIG. 1, the evaluation apparatus side control unit 320 includes a collection unit 321, a second measurement unit 322, a calculation unit 323, a classification unit 324, a selection unit 325, and an evaluation unit 326.

  The collection unit 321 collects public data from the service providing system 200. For example, the collection unit 321 collects public data stored in the system-side public data DB 210 by making a request to the service providing system 200. Then, the collection unit 321 stores the collected public data in the evaluation device side public data DB 310.

  Here, when collecting the public data from the service providing system 200, the collection unit 321 does not collect the first item-specific service quality information that has already been collected, but does not collect the first item-specific service quality information. May be collected selectively.

  The second measuring unit 322 measures the second item-specific service quality information at a second frequency that is less than the first frequency. For example, the second measuring unit 322 measures the item-specific service quality information using the resource provided by the service providing system 200 by the same method as the first measuring unit 221. For example, in the second measurement unit 322, the evaluation organization starts one virtual machine once a day, and causes the same benchmark used by the first measurement unit 221 to be executed. In other words, in this case, the second measuring unit 322 measures 25 item-specific service quality information for each item in 25 days.

  The calculation unit 323 calculates the accuracy of the classification result obtained by classifying data included in the item-specific service quality information obtained by evaluating the quality of the service provided by the service providing system 200 for each item into a plurality of sets based on the function. For example, the calculation unit 323 calculates the accuracy of the classification result obtained by classifying the data included in the first item-specific service quality information and the data included in the second item-specific service quality information into a plurality of sets based on the function. .

  Specifically, the calculation unit 323 obtains public data from the evaluation apparatus side public data DB 310, and calculates the accuracy of the classification result obtained by classifying the data included in the service quality information for each item into a plurality of sets based on the function. To do. Also, the calculation unit 323 obtains audit data from the evaluation apparatus side audit data DB 311 and calculates the accuracy of the classification result obtained by classifying the data included in each item of service quality information into a plurality of sets based on the function. Here, the classification result when the data Xi (i = 1 to n) is classified is represented by Cxi (Xi), and the classification accuracy of the classification result is represented by P (Cxi (Xi)).

  FIG. 6 is a diagram for explaining an example of the processing operation by the calculation unit 323. Here, the processing operation of the calculation unit 323 for the first item-by-item service quality information will be described, but the processing operation of the calculation unit 323 for the second item-by-item service quality information also applies to the first item-by-item service quality information. This is the same as the processing operation. The diagram shown on the left side of FIG. 6 (hereinafter referred to as the left diagram) is the first item-specific service quality information. Here, for example, X1 indicates the CPU processing speed, and X2 indicates the memory transfer speed, for example. For example, the calculation unit 323 uses an existing clustering method such as k-means to classify each data Xi into a plurality of sets and obtain a classification result. The diagram shown in the center of FIG. 6 (hereinafter referred to as the center diagram) shows the classification results obtained by the calculation unit 323. For example, the calculation unit 323 classifies the data X1 to acquire the classification result Cx1 (X1), classifies the data X2 and acquires the classification result Cx2 (X2).

  Subsequently, the calculation unit 323 evaluates the classification results of the first item-specific service quality information and the second item-specific service quality information using an existing function for evaluating the classification accuracy such as gap statistics. The figure shown on the right side of FIG. 6 (hereinafter referred to as the right figure) shows the classification accuracy, which is the evaluation value of the classification result. For example, the calculation unit 323 calculates P (Cx1 (X1)) as the evaluation value of the classification result Cx1 (X1), and calculates P (Cx2 (X2)) as the evaluation value of the classification result Cx2 (X2). To do. Note that the function used by the calculation unit 323 for classification is not limited to the gap statistic. In addition, when evaluating the classification accuracy using the gap statistics, it is desirable to normalize all data in advance in order to compare with the classification accuracy of other data.

  Returning to FIG. 1, when the classification accuracy is equal to or higher than a predetermined threshold, the classification unit 324 classifies data included in other item-specific service quality information into a plurality of sets based on the classification result. Note that the predetermined threshold is determined according to a function used such as a gap statistic. Here, if P (Cxi (Xi)) is larger than the threshold Th, the classification unit 324 determines that Cxi (Xi) is a meaningful classification according to some quality attribute of the service providing system 200. Other data is classified based on the classification.

  For example, when the accuracy of the classification result of the first item-specific service quality information is equal to or higher than a predetermined threshold, the classification unit 324 uses the data included in the other first item-specific service quality information based on the classification result. Classify. In addition, when the accuracy of the classification result of the second item-specific service quality information is equal to or higher than a predetermined threshold, the classification unit 324 converts the data included in the other second item-specific service quality information based on the classification result. Classify.

  FIG. 7 is a diagram for explaining an example of the processing operation by the classification unit 324. Here, the processing operation of the classification unit 324 for the first item-by-item service quality information will be described. However, the processing operation of the classification unit 324 for the second item-by-item service quality information also applies to the first item-by-item service quality information. This is the same as the processing operation. FIG. 7 shows a case where P (Cx1 (X1)) and P (Cx3 (X3)) are equal to or higher than the threshold among the classification accuracy shown in FIG. The classification unit 324 classifies data included in the other first item-based service quality information using a classification result having a classification accuracy equal to or higher than a threshold. That is, the classification unit 324 obtains Cxi (Xj) for each data Xj other than Xi. Here, the classification result when the data Xj is classified based on the classification result Cxi (Xi) of the data Xi is represented by Cxi (Xj).

  First, processing for classifying data Xj based on Cxi (Xi) will be described here. The data of each quality item is time series data. Further, since the benchmark is executed at the same time on the same virtual machine, the data of each quality item at a certain time has the same quality attribute for all the quality items. That is, for any i, j, data at time t of Xi and data at time t of Xj have the same quality attribute. The quality attribute includes, for example, a CPU model and a network adapter. “Classifying Xj based on Cxi (Xi)” means classifying the quality attribute of data at each time of Xj to be the same as the quality attribute of data of Cxi (Xi) at the same time. For example, if data at time t is classified as quality attribute a in Cxi (Xi), data at time t is classified as quality attribute a in Cxi (Xj). Further, if the data at time t ′ is classified as quality attribute b in Cxi (Xi), the data at time t ′ is classified as quality attribute b in Cxi (Xj).

  The left diagram in FIG. 7 shows a classification result obtained by the classification unit 324 classifying other first item-based service quality information based on the classification result Cx1 (X1), and the right diagram in FIG. 7 shows the classification unit 324. Shows the classification result obtained by classifying the other first item-based service quality information based on the classification result Cx3 (X3). As shown in the left diagram of FIG. 7, for example, the classification unit 324 classifies X2 based on the classification result Cx1 (X1) to obtain the classification result Cx1 (X2), and based on the classification result Cx1 (X1). Classify X3 to obtain a classification result Cx1 (X3). Similarly, the classification unit 324 obtains a classification result obtained by classifying other first item-based service quality information based on the classification result Cx3 (X3), as shown in the right diagram of FIG.

  In addition, if P (Cxi (Xi)) is lower than the threshold Th, the classification unit 324 does not regard Cxi (Xi) as a meaningful classification according to the quality attribute, and classifies other data based on the classification result. Do not execute.

  FIG. 8 is a diagram illustrating an example of a classification result by the calculation unit 323 and the classification unit 324. FIG. 8 illustrates an example of classifying public data indicating the memory transfer speed using the public data classification result indicating the CPU processing speed. 8a in FIG. 8 is public data indicating the CPU processing speed, and indicates data X1. Further, 8b in FIG. 8 is public data indicating the memory transfer speed, and indicates data X2.

  The calculation unit 323 obtains the classification result 8c by classifying the public data 9a indicating the CPU processing speed by opportunity learning, and calculates the accuracy of the classification result based on the function. Here, a value indicated by a black circle in 8c indicates that the attribute is classified as CPU1, and a value indicated by a triangle in 8c indicates that the attribute is classified as CPU2.

  Subsequently, when the accuracy of the classification result 8c is equal to or higher than a predetermined threshold, the classification unit 324 classifies the public data 8b indicating the memory transfer rate based on the classification result 8c, and acquires the classification result 8d. Here, a value indicated by a black circle in 8d indicates that the attribute is classified as CPU1, and a value indicated by a triangle in 8d indicates that the attribute is classified as CPU2.

  Returning to FIG. 1, the selection unit 325 calculates the accuracy of the classification result by the classification unit 324 based on the function, and selects the classification result that maximizes the calculated accuracy for each item. Here, as for the classification of each data Xj, at most n-1 classifications based on other data are calculated. However, not all classifications are useful as the classification of the data Xj. For example, the classification obtained from the CPU processing speed data corresponds to the quality attribute of the CPU type, and is suitable for the classification of the memory transfer speed data whose measured value depends on the CPU type. On the other hand, for example, even if the classification obtained from the data of the network bandwidth corresponds to the quality attribute of the network adapter, if the data of the memory transfer speed does not depend on the network adapter, the classification of the memory transfer speed Not suitable. For this reason, the selection unit 325 calculates the accuracy of the classification result by the classification unit 324 based on the function, and determines the classification result that maximizes the calculated accuracy for each evaluation item of the first item of service quality information. And it selects for every evaluation item of the service quality information according to 2nd item. Note that the selection unit 325 calculates the accuracy of the classification result by the classification unit 324 using, for example, an existing function that evaluates the classification accuracy such as gap statistics.

  FIG. 9 is a diagram for explaining an example of the processing operation performed by the selection unit 325. The left diagram of FIG. 9 shows the classification accuracy of the classification result based on the classification result Cx1 (X1), and the right diagram of FIG. 9 shows the classification accuracy of the classification result based on the classification result Cx3 (X3). For example, the selection unit 325 employs Cxi (Xj) that maximizes the classification accuracy P (Cxi (Xj)) as the classification of each data Xj. In some cases, j = i. In the example shown in FIG. 9, the maximum value of the classification accuracy of the data X1 is P (Cx1 (X1)), and the maximum value of the classification accuracy of the data X2 is P (Cx1 (X2)). In the example shown in FIG. 9, the maximum value of the classification accuracy of the data X3 is P (Cx3 (X3)), and the maximum value of the classification accuracy of the data Xn is P (Cx1 (Xn)). In such a case, the selection unit 325 selects Cx1 (X1) as the classification of the data X1, and selects Cx1 (X2) as the classification of the data X2. The selection unit 325 selects Cx3 (X3) as the classification of the data X3, and selects Cx1 (Xn) as the classification of the data Xn. By classifying the data Xj in this way, an optimum classification for the data Xj can be obtained based on a meaningful classification of other data. That is, the selection unit 325 can specify the measurement value of the item-specific service quality information according to the quality attribute.

  Returning to FIG. 1, the evaluation unit 326 compares the public data disclosed by the public unit 222 with the audit data measured at the second frequency by the second measurement unit 322, thereby verifying the authenticity of the public data. To evaluate. Here, the evaluation unit 326 includes, for example, a set included in the classification result of the first item of service quality information selected by the selection unit 325, and a set included in the classification result of the second item of service quality information. Are compared, the credibility of the first item of service quality information is evaluated, and an evaluation result is provided.

  Specifically, the evaluation unit 326 determines between the average value of the set included in the classification result of the first item of service quality information and the average value of the set included in the classification result of the second item of service quality information. Is detected by Welch's test to obtain a test result (p value). However, the evaluation unit 326 has a significant difference between the average value of the set included in the classification result of the first item of service quality information and the average value of the set included in the classification result of the second item of service quality information. It is not limited to testing whether there is a difference, but may test whether there is a significant difference in other statistical values such as variance.

  An example of the processing operation performed by the evaluation unit 326 will be described with reference to FIGS. 10 and 11. FIG. 10 is a diagram illustrating a case where the public data according to the first embodiment is classified. FIG. 11 is a diagram illustrating a case where audit data is classified in the first embodiment. In the example illustrated in FIG. 10, the first item of service quality information includes “set A” in which the benchmark score value is distributed between about 60 and about 80, and the benchmark score value of about 80 to about 120. And “set B” distributed between them. In FIG. 10, the set A is represented by a triangle and the set B is represented by a black circle. In the example shown in FIG. 11, the second item of service quality information includes “set C” in which benchmark score values are distributed between about 60 and about 80, and benchmark score values of about 80 to about 80. “Set D” distributed between 120 and 120 is included. In FIG. 11, the set C is represented by a triangle and the set D is represented by a black circle. In this case, the evaluation unit 326 performs Welch's test on each of the “set A” and “set C” pairs and the “set B” and “set D” pairs.

  In addition, the evaluation unit 326 provides an evaluation result to the user. Specifically, the evaluation unit 326 discloses the evaluation result. For example, the evaluation unit 326 provides the user terminal 100 with the p value that is the result of the Welch test.

  As described above, the evaluation apparatus 300 periodically executes the benchmark on the service providing system 200 to obtain the measurement data (audit data) in order to verify the authenticity of the public data. Then, after acquiring the audit data, the evaluation apparatus 300 confirms by a hypothesis test that there is no significant difference between the statistics of the public data and the audit data, and discloses the test result to the user. Specifically, the evaluation apparatus 300 periodically executes a benchmark on the service providing system 200, and stores the obtained audit data in the evaluation apparatus side audit data DB 311. Further, the evaluation apparatus 300 periodically accesses the disclosure unit 222 of the service providing system 200, acquires the latest public data, and stores it in the evaluation apparatus side public data DB 310. The evaluation apparatus 300, for example, every time there is an update in the evaluation apparatus side public data DB 310 and the evaluation apparatus side audit data DB 311, a significant difference in the average value of the CPU speed of the virtual machine between the public data and the audit data. It is confirmed by hypothesis testing that there is no error and the confirmation result is disclosed to the user. However, the present invention is not limited to the case where “CPU speed” is used as the service quality to be measured, and is not limited to the case where “average value” is used as a statistical value for determining a significant difference.

  FIG. 12 is a flowchart illustrating an example of a flow of processing by the first measurement unit 221 in the first embodiment. The first measuring unit 221 measures the item-specific service quality information at the first frequency by executing a series of processes described below for each first frequency.

  As illustrated in FIG. 12, in the service providing system 200 according to the first embodiment, the first measurement unit 221 is preset in the service providing system 200 by an administrator, for example, at the measurement timing (Yes in step S101). When the time comes, the benchmark is executed (step S102). That is, item-specific service quality information is measured.

  Then, the first measuring unit 221 updates the system side public data DB 210 (step S103). That is, the first measurement unit 221 stores the newly measured item-specific service quality information in the system-side public data DB 210.

  In addition, said process procedure is not limited to said order, You may change suitably in the range which does not contradict a process content.

  Next, a processing procedure performed by the evaluation apparatus side control unit 320 will be described with reference to FIG. FIG. 13 is a flowchart illustrating an example of a processing flow by the evaluation apparatus side control unit 320 according to the first embodiment.

  As illustrated in FIG. 13, the calculation unit 323 determines whether it is a processing timing (step S <b> 201). If the calculation unit 323 determines that it is the timing of the process (Yes in step S201), the calculation unit 323 executes a process of classifying the public data and the audit data sample (step S202). The processing for classifying the public data and audit data samples will be described in detail with reference to FIG. On the other hand, when the calculation unit 323 does not determine that it is the processing timing (No in step S201), the calculation processing in step S201 is repeated.

  Subsequently, the evaluation unit 326 verifies the public data and the audit data for each classified sample set (step 203). That is, the evaluation unit 326 compares the public data and the audit data for each classified set, and evaluates the authenticity of the public data for each identification information.

  Thereafter, the evaluation unit 326 discloses the p value of each test (step S204). For example, the evaluation unit 326 discloses the lower, higher, both, or the average of the two obtained p values to the user.

  In addition, said process procedure is not limited to said order, You may change suitably in the range which does not contradict a process content.

  Next, the processing procedure of the classification process will be described using FIG. FIG. 14 is a flowchart illustrating an example of a process flow of classification according to the first embodiment. The process shown in FIG. 14 corresponds to the process in step S202 in FIG.

  As illustrated in FIG. 14, the calculation unit 323 selects quality information Xi of unselected evaluation items (step S301). Subsequently, the calculation unit 323 calculates the classification Cxi (Xi) of the quality information Xi by machine learning or the like (Step S302). Then, the calculation unit 323 determines whether or not the classification accuracy P (Cxi (Xi)) of the classification result is equal to or higher than the threshold (Step S303). Here, when the calculation unit 323 does not determine that the classification accuracy P (Cxi (Xi)) is equal to or higher than the threshold (No in step S303), the calculation unit 323 proceeds to step S301 and selects quality information of an unselected evaluation item. To do.

  On the other hand, when the calculation unit 323 determines that the classification accuracy P (Cxi (Xi)) is equal to or higher than the threshold (Yes in step S303), the classification unit 324 performs all other evaluations based on the classification result Cxi (Xi). The item quality information Xj (j = 1, 2,..., N) is classified (the classification result is Cxi (Xj)) (step S304).

  Further, the calculation unit 323 determines whether or not there is quality information Xi of an unselected evaluation item (step S305). If the calculation unit 323 determines that there is unprocessed quality information (Yes in step S305), the calculation unit 323 proceeds to step S301 and selects quality information of an unselected evaluation item. On the other hand, when the calculation unit 323 does not determine that there is unprocessed quality information (No in step S305), the selection unit 325 determines the classification result Cxl (Xk) that maximizes the classification accuracy for the quality information Xk of each evaluation item. ) Is selected (step S306).

  In addition, said process procedure is not limited to said order, You may change suitably in the range which does not contradict a process content.

  As described above, according to the first embodiment, the evaluation apparatus 300 classifies public data and audit data into sets. In the conventional technique, even if the time series data of a certain evaluation item depends on a certain quality attribute in the existing clustering method, the data cannot be classified unless the dependency is strong. On the other hand, in the first embodiment, even when the time series data A of an evaluation item is weakly dependent on a certain quality attribute, if the time series data B of the evaluation item strongly dependent on the quality attribute exists. Based on the classification of data B, data A is accurately classified according to its quality attribute. Thereby, the evaluation apparatus 300 can specify the measured value of the item-specific service quality information according to the quality attribute even when the attribute information is not disclosed.

  Among the performance items, the CPU processing speed, memory transfer speed, and disk I / O performance are strongly dependent on the CPU type assigned to the virtual machine. Therefore, the coefficient of variation of the measured values of these performance items is greatly reduced by classifying the performance values according to the CPU type. In the test for detecting a significant difference in mean value, the accuracy of the test generally improves as the data coefficient of variation decreases (Non-Patent Document 2). In the first embodiment, the evaluation apparatus 300 classifies public data and audit data into sets, and then evaluates the credibility of the public data for each set. That is, the evaluation apparatus 300 according to the first embodiment can reduce the coefficient of data variation. By doing so, the evaluation apparatus 300 according to the first embodiment can suppress the dispersion of data and improve the normality of the distribution. Thus, according to the first embodiment, it is possible to improve the verification accuracy by classifying by CPU type before verification.

  In the first embodiment, in the service providing system 200, the operator does not disclose the hardware type of the sample in the public data for some reason, or the evaluation organization cannot confirm the hardware type of the virtual machine. However, the present invention is not limited to this. For example, even when the service providing system 200 discloses the hardware type, the service quality information for each item may be classified using machine learning.

  Further, when the dependency relationship between the evaluation item and the quality attribute is known in advance, each measurement value may be classified according to the quality attribute. As an example, a case where the CPU processing speed and the memory transfer speed are used as evaluation items will be described. From the experience of performance measurement to date, it is assumed that these measurement values are strongly dependent on the CPU type commonly assigned to the virtual machine, and that the degree of dependence is higher in the CPU processing speed. Note that the CPU type of each measurement value is not disclosed. The CPU time series data can be classified by the existing clustering method because the measured value strongly depends on the CPU type. On the other hand, time-series data of memory transfer speed is somewhat weak in dependence on the CPU type of measurement values, and if classification is performed using the existing machine learning clustering method, some measurement values may be misclassified. is there. In such a case, the evaluation apparatus 300 may classify each measurement value of the memory transfer speed using the classification result of the CPU processing speed.

  In addition, although the case where the evaluation apparatus 300 includes the evaluation unit 326 has been described in the first embodiment, the present invention is not limited to this. For example, the evaluation apparatus 300 may not have the evaluation unit 326. Further, in this case, the evaluation apparatus 300 specifies the measured value of the item-specific service quality information according to the quality attribute for at least one of the first item-specific service quality information or the second item-specific service quality information. May be.

  Specifically, the calculation unit 323 acquires public data from the evaluation apparatus side public data DB 310, and classifies the data included in the first item-specific service quality information into a plurality of sets based on the accuracy of the classification result. To calculate. Subsequently, when the accuracy of the classification result of the first item of service quality information is equal to or higher than a predetermined threshold, the classification unit 324 includes data included in the other first item of service quality information based on the classification result. Classify. Then, the selection unit 325 calculates the accuracy of the classification result obtained by the classification unit 324 based on the function, and selects the classification result that maximizes the calculated accuracy for each evaluation item of the first item of service quality information. Thereby, the evaluation apparatus 300 can specify the measurement value of the first item-based service quality information according to the quality attribute even when the attribute information is not disclosed.

  Or the calculation part 323 acquires audit data from the evaluation apparatus side audit data DB311, and calculates the precision of the classification result which classified the data included in the item-specific service quality information into a plurality of sets based on the function. Subsequently, when the accuracy of the classification result of the second item-based service quality information is equal to or higher than a predetermined threshold, the classification unit 324 includes data included in the other second item-specific service quality information based on the classification result. Classify. Then, the selection unit 325 calculates the accuracy of the classification result by the classification unit 324 based on the function, and selects the classification result that maximizes the calculated accuracy for each evaluation item of the second item-specific service quality information. Thereby, the evaluation apparatus 300 can specify the measured value of the second item-based service quality information according to the quality attribute even when the attribute information is not disclosed.

  Further, according to the first embodiment, the service providing system 200 measures the item-specific service quality information indicating the quality of the service at the first frequency, and the item-specific service quality measured at the first frequency. Disclose public data that is each piece of information. Further, the evaluation apparatus 300 measures item-specific service quality information at a second frequency that is less than the first frequency, obtains publicly disclosed data, and is measured at the second frequency. The reliability of public data is evaluated by comparing it with audit data, which is service quality information for each item. As a result, item-specific service quality information can be appropriately provided. For example, item-specific service quality information with high credibility can be appropriately used, and item-specific service quality information with high credibility can be used at low cost.

  That is, the service provided by the service providing system 200 generally has to continuously measure the quality regularly and frequently in order to grasp the service quality. Hereinafter, this situation will be described by taking an IaaS system that provides a computing resource such as a virtual machine or storage as a service via a network as an example.

  If it is a normal computer resource, the performance of the resource will hardly change after purchase. However, since the resource provided by the IaaS system shares physical resources with other users, the underlying physical resource Because it is renewed, the performance changes with time. For this reason, when it is desired to know the performance of the resource provided by the IaaS system, it is necessary to continuously measure the performance of the resource.

  In addition, an IaaS system may be composed of physical hardware with different performance, and the physical hardware underneath the resource may change each time it is started. As a result, the resource performance of the IaaS system can vary greatly each time it is used, so the resource performance variation is large compared to conventional computer resources. For this reason, when it is desired to know the performance of the resources provided by the IaaS system, it is necessary to execute a benchmark on a large amount of resources as well as one resource.

  Here, according to the first embodiment, the evaluation unit 326 measures the item-specific service quality information at the second frequency that is less than the first frequency, and uses the publicly disclosed data. The credibility of the public data is evaluated by obtaining and comparing it with audit data that is each item-specific service quality information measured at the second frequency. As a result, item-specific service quality information can be appropriately provided. In addition, the public data disclosed by the service providing system 200 is subjected to a test by the evaluation device 300, and the credibility is increased. In addition, the cost used for the process is the sum of the cost of the operator required to acquire the public data and the cost of the evaluation organization required to acquire the audit data. Here, in acquiring the audit data, since the audit data is used only for the verification of the validity of the public data, it is sufficient to perform the measurement by the evaluation device 300 at a low frequency, and the evaluation device required for acquiring the audit data. The cost at 300 can also be reduced. As a result, highly reliable data can be disclosed and used at low cost. Further, based on the public data and the test result obtained from the service providing system 200 and the evaluation apparatus 300, the user can select the service providing system 200 that is most suitable for the user from the market.

  Further, according to the first embodiment, the first measuring unit 221 measures item-specific service quality information using surplus resources among resources used for providing services in the service providing system. As a result, it is possible to reduce the cost required for obtaining public data.

  Further, according to the first embodiment, when the resources used for providing the service are insufficient, the first measuring unit 221 releases the surplus resources used for measuring the item-specific service quality information. As a result, it is possible to reduce the cost required for obtaining public data. It is also possible to measure service quality information without hindering service provision.

  Further, according to the first embodiment, when the evaluation unit 326 evaluates that the credibility as the evaluation result is lower than the threshold, the second measurement unit 322 measures the item-specific service quality information. Increase the frequency of As a result, when the possibility that the public data is incorrect is higher than the threshold value, it is possible to secure the authenticity of the public data by increasing the frequency of measuring the audit data and increasing the accuracy of the authenticity.

  Note that the functions used by the calculation unit 323 and the selection unit 325 to calculate the accuracy of the classification result may be the same function or different functions.

(Second Embodiment)
Although the embodiments of the present invention have been described so far, the present invention may be implemented in other embodiments besides the above-described embodiments. Therefore, other embodiments will be described below.

(Publication contents)
For example, in the above-described embodiment, the case where the evaluation apparatus 300 publishes the evaluation result is shown as an example, but the present invention is not limited to this. For example, the evaluation apparatus 300 may disclose the audit data itself, may disclose the evaluation result and the audit data together, or may disclose the evaluation result, the audit data, and the public data together.

(Public data and audit data)
Further, for example, the evaluation apparatus 300 may perform evaluation by selectively using public data and audit data from a current time to a predetermined period before. That is, as the number of data samples increases, the accuracy of the test increases, but the freshness of information decreases for older samples. Since the resource performance of the service providing system 200 changes with time, in order to verify the current accurate resource performance, the verification may be performed by paying attention only to a sample of a certain period. For example, the sample of public data and audit data may be used only for the latest two months. As a result, the current service providing system 200 can be evaluated with high accuracy.

(System configuration)
Also, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-mentioned document and drawings (for example, FIGS. 1 to 14) are arbitrarily selected unless otherwise specified. Can be changed.

  Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or a part of the distribution / integration may be functionally or physically distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured.

(program)
FIG. 15 is a diagram showing that information processing by the quality attribute specifying program for service quality information for executing processing by the quality attribute specifying system for service quality information is specifically realized using a computer. As illustrated in FIG. 15, the computer 3000 includes, for example, a memory 3010, a CPU 3020, and a network interface 3070. Each part of the computer 3000 is connected by a bus 3100.

  The memory 3010 includes a ROM 3011 and a RAM 3012 as illustrated in FIG. The ROM 3011 stores a boot program such as BIOS (Basic Input Output System).

  Here, as illustrated in FIG. 15, the hard disk drive 3080 stores, for example, an OS 3081, an application program 3082, a program module 3083, and program data 3084. That is, the service quality information quality attribute specifying program according to the disclosed technology is stored in, for example, the hard disk drive 3080 as a program module 3083 in which a command to be executed by the computer is described. Specifically, a program module describing each procedure for executing the same information processing as each unit of the evaluation apparatus side control unit 320 described in the above embodiment is stored in the hard disk drive 3080.

  Data used for information processing by the quality attribute specifying program for service quality information is stored as program data 3084 in, for example, the hard disk drive 3080. The CPU 3020 reads the program module 3083 and program data 3084 stored in the hard disk drive 3080 to the RAM 3012 as necessary, and executes various procedures.

  Note that the program module 3083 and the program data 3084 related to the quality attribute specifying program of the service quality information are not limited to being stored in the hard disk drive 3080. For example, the program module 3083 and the program data 3084 may be stored in a removable storage medium. In this case, the CPU 3020 reads data via a removable storage medium such as a disk drive. Similarly, the program module 3083 and the program data 3084 related to the quality attribute specifying program of the service quality information are other computers connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). May be stored. In this case, the CPU 3020 reads various data by accessing another computer via the network interface.

(Other)
The quality attribute specifying program for service quality information described in this embodiment can be distributed via a network such as the Internet. The quality attribute specifying program of the service quality information is recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer. You can also

10 network 100 user terminal 200 service providing system 201 system side communication unit 210 system side public data DB
220 system-side control unit 221 first measurement unit 222 disclosure unit 300 evaluation device 301 evaluation device-side communication unit 310 evaluation device-side public data DB
311 Evaluation device side audit data DB
320 Evaluation Device Side Control Unit 321 Collection Unit 322 Second Measurement Unit 323 Calculation Unit 324 Classification Unit 325 Selection Unit 326 Evaluation Unit

Claims (4)

A service quality information providing system comprising a service providing system for providing a service via a network and an evaluation device for evaluating the quality of the service,
The evaluation device is
A calculation unit that calculates, based on a function, the accuracy of a classification result obtained by classifying data included in item-specific service quality information obtained by evaluating the quality of service provided by the service providing system for each item into a plurality of sets;
When the accuracy is equal to or higher than a predetermined threshold, a classification unit that classifies data included in other item-based service quality information into a plurality of sets based on the classification result;
A selection unit that calculates the accuracy of the classification result by the classification unit based on the function, and selects a classification result that maximizes the calculated accuracy for each item;
A quality attribute specifying system for service quality information, comprising: The service providing system includes:
A first measuring unit that measures first item-specific service quality information obtained by evaluating the quality of service provided by the service providing system at a first frequency for each item;
A publishing unit for publishing the first item-specific service quality information by the first measuring unit;
The evaluation device is
An acquisition unit for acquiring the first item-specific service quality information published by the publication unit;
Second measurement for measuring second item-by-item service quality information in which the quality of the service provided by the service providing system is evaluated for each item at a second frequency that is less than the first frequency. And further comprising
The calculation unit uses, as the function, the accuracy of the classification result obtained by classifying at least one of data included in the first item-specific service quality information or data included in the second item-specific service quality information into a plurality of sets. Based on
When the accuracy of the classification result of the first item-based service quality information is equal to or higher than a predetermined threshold, the classifying unit converts the data included in the other first item-specific service quality information based on the classification result. If the accuracy of the classification result of the second item-specific service quality information is equal to or higher than a predetermined threshold, the data included in the other second item-specific service quality information is classified based on the classification result. ,
The selection unit calculates the accuracy of the classification result by the classification unit based on the function, and determines the classification result having the maximum calculated accuracy for each evaluation item of the first item of service quality information or the first The quality attribute specifying system for service quality information according to claim 1, wherein at least one of the evaluation items for the item-specific service quality information is selected.   By comparing the set included in the classification result of the first item of service quality information selected by the selection unit with the set included in the classification result of the second item of service quality information, the first The quality attribute identification system for service quality information according to claim 2, further comprising an evaluation unit that evaluates the credibility of one item of service quality information and provides an evaluation result. A service quality information quality attribute specifying method executed by a service quality information quality attribute specifying system comprising a service providing system for providing a service via a network and an evaluation device for evaluating the quality of the service,
The evaluation device is
A calculation step of calculating, based on a function, accuracy of a classification result obtained by classifying data included in item-specific service quality information obtained by evaluating the quality of service provided by the service providing system for each item into a plurality of sets;
When the accuracy is equal to or higher than a predetermined threshold, a classification step of classifying data included in other item-specific service quality information into a plurality of sets based on the classification result;
A selection step of calculating the accuracy of the classification result by the classification step based on the function, and selecting a classification result that maximizes the calculated accuracy for each item;
A quality attribute specifying method for service quality information, characterized by comprising:

Images