JP2017151614A - Computing machine and analytical index calculation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate an index which properly analyzes prediction accuracy of a predictive model.SOLUTION: A computing machine analyzes prediction accuracy of a predictive model for calculating an estimated value of a prediction item for an object to be observed. The computing machine manages a database which stores a plurality of records comprising estimated values of prediction items of an object to be observed and actual values of the prediction items of the object, sorts the records stored in the database on the basis of the estimated values, calculates predictive errors of the records, on the basis of the estimated values and actual values of the records stored in the database, selects a plurality of records on the basis of a sorting result, and calculates an analytical index on the basis of statistic calculated from the predictive errors of the selected records.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for analyzing prediction accuracy.

  In recent years, in various fields, prediction of an event based on a prediction model generated by machine learning using past data has been performed. Here, the past data indicates a data group such as a record composed of a group of items observed, aggregated, or recorded at an arbitrary time.

  When an arbitrary event is predicted for an observation target using a prediction model, it is necessary to ensure that the accuracy of the prediction model used is high. Therefore, a technique for analyzing the accuracy of the prediction model is required. As a method for analyzing the accuracy of a prediction model, for example, a technique described in Patent Document 1 is known.

Patent Document 1 states that “a single regression equation calculation unit of a prediction index calculation device calculates a single regression equation for a plurality of actual values related to shipment of products recorded as past data and predicted values corresponding to the actual values. The prediction accuracy calculation unit of the prediction index calculation apparatus includes a slope a of a single regression line y = ax represented by the single regression equation, past data as a plurality of actual values, and predicted values corresponding to the actual values. Is calculated on the basis of the difference between the point (X, Y) on the graph and the slope of the straight line connected to the origin. Cos ² θ is calculated for each point (X, Y) for an angle θ formed by a single regression line y = ax and a straight line connecting the point (X, Y) with the origin, and the average of cos ² θ The value is calculated as the prediction accuracy. "

  Here, terms used in this specification will be described.

  An item corresponding to an event to be predicted for an observation target is described as a prediction item. In addition, when the value which a prediction item can take is binary like "0" and "1", it describes also as a prediction event. For example, the incidence of illness and the failure rate of equipment correspond to the predicted events. Items that have a correlation with the prediction item are described as correlation items. Prediction based on past data is a rule (prediction model) that shows the relationship between a prediction item and an observation item using a record group of an arbitrary period, and the observation item at a certain time using that rule is extracted. Indicates that the value of the prediction item is calculated from the value. The value of the prediction item calculated based on the prediction model is described as a predicted value. Moreover, the value of the prediction item actually observed is described as the actual value. Further, an error between the actual value and the predicted value is described as a prediction error. The above is the explanation of terms in this specification.

JP 2009-245032 A

  Conventionally, an index indicating how close the predicted value is to the actual value is used as an index for analyzing the prediction accuracy. In the following description, an index for analyzing prediction accuracy is also referred to as an analysis index. The conventional analysis index is given as the average value of the square of the prediction error of each record or the square root of the average value. The analysis index mentioned above shows that the prediction accuracy of a prediction model is so high that a value is small.

  In the case of a prediction item that has a low correlation between the observation item group and the prediction item and is greatly affected by other factors not reflected in the observation item group, the difference between the prediction value and the actual value becomes large. That is, the difference between the actual value and the expected value of the actual value increases. This is because the prediction model is generated so as to be close to the expected value of the actual value.

  In the case as described above, the conventional analysis index is an index that is not suitable for the analysis of the prediction accuracy because the value becomes very large.

  You can analyze the prediction accuracy of the prediction model by comparing the expected value of the actual value with the predicted value for each record, but the expected value of the actual value cannot be calculated from the value of only one observation target. The method is difficult.

  The present invention provides an apparatus and a method for calculating an index that can appropriately analyze the prediction accuracy of a prediction model.

  A typical example of the invention disclosed in the present application is as follows. That is, a computer that analyzes the prediction accuracy of a prediction model for calculating a prediction value of a prediction item to be observed, the computer being connected to an arithmetic device, a memory connected to the arithmetic device, and the arithmetic device And the computer manages a database that stores a plurality of records composed of the predicted value of the observation target prediction item and the actual value of the observation target prediction item, and the computing device includes: Sorting a plurality of records stored in the database based on the size of the predicted value, storing a sorting result in the memory, the arithmetic unit, the predicted values of the plurality of records stored in the database, and Based on the actual value, a prediction error of each of the plurality of records is calculated, and each of the plurality of records stored in the database is calculated. A measurement error is stored in the memory, and the arithmetic unit selects a plurality of records to be processed based on the sorting result, and calculates a statistical amount calculated from the prediction error of each of the selected plurality of records. Based on this, an analysis index for analyzing the prediction accuracy of the prediction model is calculated, and the analysis index is stored in the memory.

  According to the present invention, it is possible to calculate an index that can analyze the prediction accuracy of a prediction model even when the correlation between the observation item group and the prediction item is low. Problems, configurations, and effects other than those described above will become apparent from the description of the following examples.

6 is a flowchart illustrating an example of processing executed by the computer according to the first embodiment. 6 is an explanatory diagram illustrating a cumulative graph of Example 1. FIG. 1 is a block diagram illustrating an example of a configuration of a computer system according to a first embodiment. FIG. 4 is an explanatory diagram illustrating an example of individual information stored in an individual information storage unit according to the first embodiment. It is explanatory drawing which shows an example of the group information stored in the group information storage part of Example 1. It is explanatory drawing which shows an example of the group statistical information stored in the group information storage part of Example 1. It is explanatory drawing which shows an example of the prediction accuracy analysis graph of Example 1. FIG. 6 is an explanatory diagram illustrating an example of analysis index information according to Embodiment 1. FIG. FIG. 6 is an explanatory diagram illustrating an example of a cumulative graph according to the first embodiment.

  First, an outline of the present invention will be described. Here, a database that stores records including predicted values and actual values of prediction items is assumed. The actual value is a binary value indicating whether or not a certain event occurs. That is, the actual value is either “1” indicating that an event has occurred or “0” indicating that an event has not occurred. In this case, the predicted value is a value between “0” and “1”.

  The present invention analyzes the prediction accuracy of a prediction model by analyzing how close the actual value is to the predicted value. That is, the prediction value based on the prediction model is assumed to be correct, and an error between the prediction value and the actual value is analyzed. However, even if the error between the predicted value and the actual value is calculated for each record, a meaningful analysis cannot be performed. Therefore, the computer 300 (see FIG. 3) of the present invention executes the following processing.

  The computer 300 sorts the records based on the size of the predicted value. Specifically, the computer 300 sorts the records in descending order of the predicted value. The computer 300 calculates a prediction error using the predicted value and actual value of each record. The computer 300 selects a target record group based on the sorting result, and calculates an analysis index based on the prediction error of the selected record group.

  Note that the computer 300 may calculate a prediction value statistical value and a real value statistical value of the selected record group after selecting the record group, and calculate a prediction error based on the statistical amount.

  There are various record selection methods depending on the purpose. For example, in order to analyze the prediction accuracy of the prediction model, the prediction accuracy may be analyzed for a record group in which an event is likely to occur. Therefore, the computer 300 generates a cumulative graph (cumulative information) indicating the cumulative value of the predicted value and the cumulative value of the actual value based on the sorting result.

  Here, the cumulative graph will be described. FIG. 2 is an explanatory diagram illustrating a cumulative graph 200 according to the first embodiment.

  The horizontal axis indicates the sort number assigned to the record. The vertical axis represents the cumulative value. The cumulative value of predicted values can be calculated, for example, by summing the predicted values of each record. The accumulated value of the actual value can be calculated by summing up the actual values of the records. For example, the cumulative value of the predicted values with the sort number “100” is calculated by summing the predicted values of the records with the sort numbers “1” to “100”.

  A record with a large predicted value has a high probability that the actual value is “1”. Therefore, the cumulative value increases rapidly. On the other hand, a record with a small predicted value has a low probability that the actual value is “1”. Therefore, the cumulative value increases gradually. Therefore, the cumulative graph 200 is a curve as shown in FIG.

  When the records are sorted in order of increasing predicted value, the graph of the cumulative value is different from the graph shown in FIG. 2, but is a graph showing the above-described characteristics of the increase amount of the cumulative value.

  In the case of a prediction model with high prediction accuracy, the difference between the cumulative value of the predicted value and the cumulative value of the actual value of the record group whose predicted value is larger than a predetermined threshold value is small. That is, the actual value is close to the predicted value. On the other hand, the difference between the cumulative value of the predicted value and the cumulative value of the actual value of the record group whose predicted value is equal to or less than a predetermined threshold increases monotonously. This is because the difference between the predicted value and the actual value becomes large.

  As described above, it can be seen that the cumulative value of the prediction and the cumulative value of the actual value based on the sorting result are numerical values related to the prediction accuracy of the prediction model. It can also be seen that the statistic of the record group whose predicted value is equal to or less than a predetermined threshold is not suitable for the analysis of the prediction accuracy. Therefore, the prediction accuracy may be analyzed for a record group whose predicted value is larger than a predetermined threshold. As a result, random factors in the prediction accuracy analysis can be eliminated.

  Therefore, the computer 300 selects a record group to be analyzed, and calculates, as an analysis index, a difference between a predicted value statistic and a real value statistic of the record group to be analyzed.

  This is because it is necessary to analyze the difference between the predicted value and the expected value of the actual value in order to analyze the prediction accuracy of the prediction model more accurately. However, in general, the expected value of the actual value is unknown. Further, the expected value of the actual value cannot be calculated from the value of each record.

  It is presumed that records with similar predicted values have similar actual expected values. Therefore, after the records are sorted, the computer 300 groups a plurality of records having similar predicted values. Accordingly, the computer 300 can calculate a statistic corresponding to the expected value of the actual value from the distribution of the actual values of the plurality of records included in the group. Further, the computer 300 calculates a statistic corresponding to the expected value of the actual value of each group and a statistic of the predicted value. The computer 300 calculates a prediction error based on the statistic corresponding to the expected value of the actual value and the statistic of the predicted value. The computer 300 calculates an analysis index based on the prediction error.

  Since the conventional analysis index is calculated from the prediction error or the like of all records, it can be controlled by raising or lowering the predicted value of all records. In this embodiment, the prediction accuracy can be calculated for a group suitable for prediction accuracy analysis.

  In the above-described processing, the analysis index is calculated for a record group in which an event is likely to occur, but the present invention is not limited to this. For example, the analysis index may be calculated for a record group that is unlikely to cause an event. The record group can be selected using the graph of FIG. Further, an analysis index may be calculated for a specific group designated by the user. Therefore, it is possible to analyze the prediction accuracy according to the purpose.

  In the first embodiment, the computer 300 sorts records composed of the predicted values and actual values of the prediction items based on the predicted values, and generates a group composed of a predetermined number of records based on the sorting result. The computer 300 calculates an average value of predicted values and an average value of actual values for each group, and calculates a difference between the average value of predicted values and the average value of actual values as a group prediction error. The computer 300 generates a prediction accuracy analysis graph based on the calculation result, and calculates an analysis index.

  FIG. 3 is a block diagram illustrating an example of the configuration of the computer system according to the first embodiment.

  The computer system includes a computer 300 and a storage device 301.

  The computer 300 analyzes the prediction accuracy of the prediction model of an arbitrary prediction item. The computer 300 according to this embodiment includes an arithmetic device 310, a memory 311, an input device 312, an output device 313, and a storage medium 314. Each component is connected to each other via an internal bus or the like.

  The arithmetic device 310 is an arithmetic device that executes a program stored in the memory 311 and includes, for example, a CPU and a GPU. In the following description, when processing and functions are described using a functional unit as a subject, it indicates that a program for realizing the functional unit is being executed by the arithmetic device 310. The memory 311 stores a program executed by the arithmetic device 310 and information used by the program. The memory 311 may be either a volatile memory or a nonvolatile memory.

  The input device 312 is a device for inputting various information to the computer 300, and includes, for example, a keyboard, a mouse, a touch panel, and the like. The output device 313 is a device that outputs a processing result executed by the computer 300, and includes, for example, a display.

  The storage medium 314 stores programs that realize various functions of the computer 300. In this embodiment, the arithmetic device 310 reads a program from the storage medium 314, loads the read program onto the memory 311, and executes the loaded program. The program stored in the storage medium 314 of this embodiment will be described later.

  Note that the program stored in the storage medium 314 may be obtained from a removable medium such as a CD-ROM and a flash memory or a distribution server connected via a network. When acquiring a program from a removable medium, the computer 300 includes an interface connected to the removable medium.

  The storage device 301 stores various data managed by the computer 300. The storage device 301 can be a general computer, a storage system, or the like. The storage system includes a controller, an external interface, and a plurality of storage media, and a RAID can be configured using a plurality of storage media. The storage system can also provide a plurality of logical storage areas using a RAID volume.

  A database is constructed on the storage device 301, and the database includes an individual information storage unit 351, a group information storage unit 352, and an analysis result storage unit 353.

  The individual information storage unit 351 stores information such as records that hold various types of information on observation targets (people, devices, etc.). The record includes the value of the observation item. Details of the information stored in the individual information storage unit 351 will be described with reference to FIG.

  The group information storage unit 352 stores information on groups generated from a plurality of observation target information. Details of the information stored in the group information storage unit 352 will be described with reference to FIG.

  The analysis result storage unit 353 stores information related to the analysis result of the prediction accuracy of the prediction model. Details of the information stored in the analysis result storage unit 353 will be described with reference to FIGS. 6, 7, and 8.

  Here, the program stored in the storage medium 314 will be described.

  The storage medium 314 stores a program for realizing the record management unit 320, the statistic calculation unit 330, and the prediction accuracy analysis unit 340.

  The record management unit 320 manages records stored in the individual information storage unit 351. In addition, the record management unit 320 sorts the records and groups the records. The record management unit 320 is composed of a plurality of modules. The record management unit 320 of this embodiment includes a record sort unit 321 and a group generation unit 322.

  The record sort unit 321 sorts a plurality of records based on the size of the predicted value. The record sort unit 321 assigns a sort number to each record when the records are sorted. The record sort unit 321 stores the processing result in the individual information storage unit 351. The group generation unit 322 generates a plurality of groups including a predetermined number of records based on the record sorting result. The group generation unit 322 stores the generated group information in the group information storage unit 352.

  The statistic calculator 330 calculates various statistics for each group. The calculated statistic may be an average value of predicted values and an average value of actual values, or an accumulated value of predicted values and an accumulated value of actual values. Note that the present invention is not limited to the type of calculated statistic. The statistic calculation unit 330 includes a plurality of modules. The statistic calculation unit 330 of this embodiment includes a group statistic calculation unit 331 and a prediction error calculation unit 332.

  The group statistic calculation unit 331 calculates the predicted value statistic and the actual value statistic for each group based on the information stored in the group information storage unit 352. The group statistic calculation unit 331 stores the calculated predicted value statistic and actual value statistic in the group information storage unit 352. The prediction error calculation unit 332 calculates a prediction error for each group based on the predicted value statistic and the actual value statistic. The prediction error calculation unit 332 stores the prediction error for each group in the group information storage unit 352.

  The prediction accuracy analysis unit 340 analyzes the prediction accuracy of the prediction model based on the statistics of each group calculated by the statistic calculation unit 330. The prediction accuracy analysis unit 340 includes a plurality of modules. The prediction accuracy analysis unit 340 of the present embodiment includes a prediction accuracy analysis graph generation unit 341 and a prediction accuracy analysis index calculation unit 342.

  The prediction accuracy analysis graph generation unit 341 generates a prediction accuracy analysis graph 700 (see FIG. 7) based on the statistics stored in the group information storage unit 352. The prediction accuracy analysis graph generation unit 341 stores the prediction accuracy analysis bluff in the analysis result storage unit 353. The prediction accuracy analysis index calculation unit 342 calculates an analysis index based on the statistics stored in the group information storage unit 352 and the prediction accuracy analysis graph 700. The prediction accuracy analysis index calculation unit 342 stores the analysis index in the analysis result storage unit 353.

  FIG. 4 is an explanatory diagram illustrating an example of the individual information 400 stored in the individual information storage unit 351 according to the first embodiment.

  The individual information 400 includes a plurality of records corresponding to observation targets. The record includes an ID 401, a predicted value 402, and an actual value 403. In the following description, a record included in the individual information 400 is also referred to as an individual record.

  The ID 401 is identification information for uniquely identifying an observation target, that is, a record. An ID number is stored in the ID 401 of this embodiment. The prediction value 402 is the value of the prediction item calculated based on the prediction model. The actual value 403 is a value indicating the observation result of the predicted event.

  Here, the predicted value 402 and the actual value 403 will be described using information for managing the onset of diabetes as an example.

  In this case, one record corresponds to data of one medical examiner. In the case of a record whose ID 401 is “00001”, the predicted value 402 is “0.32”, and the actual value 403 is “0”. This indicates that the predictive value of the incidence of diabetes of the medical examiner is “0.32,” and the actual onset is “0”. In this embodiment, “1” is stored in the actual value 403 when diabetes occurs, and “0” is stored in the actual value 403 when diabetes does not occur. That is, the medical examiner whose ID 401 is “00001” indicates that diabetes has not occurred.

  FIG. 5 is an explanatory diagram illustrating an example of the group information 500 stored in the group information storage unit 352 according to the first embodiment.

  Group information 500 includes a plurality of records corresponding to observation targets. The record includes an ID 501, a group number 502, and a sort number 503.

  ID 501 is the same as ID 401. The group number 502 is a group identification number assigned by the group generation unit 322. The sort number 503 is a number indicating the sort order assigned by the record sort unit 321.

  In the present embodiment, a sort number is assigned to each record in order from “1” based on the size of the predicted value. The records are sorted in ascending order of the sort number 503.

  In the case of a record whose ID 501 is “00001”, the sort number 503 is “2”. This indicates that among the records included in the individual information 400, the predicted value 402 is the second largest record. The group number 502 of the record is “1”. This indicates that the record is included in the group having the group number “1”.

  FIG. 6 is an explanatory diagram illustrating an example of the group statistical information 600 stored in the group information storage unit 352 according to the first embodiment.

  Group statistical information 600 includes a plurality of records corresponding to groups. The record includes a group number 601, a predicted value average value 602, a real value average value 603, and a prediction error 604.

  The group number 601 is the same as the group number 502. The average value 602 of predicted values is an average value of the predicted values 402 of individual records included in the group. The actual value average value 603 is an average value of the actual values 403 of the individual records included in the group. The prediction error 604 is a value indicating an error between the average value of predicted values and the average value of actual values for each group.

  The average value 603 of the actual values is a value calculated from the distribution of the actual values of the individual records included in the group, and is a statistic corresponding to the expected actual value. In the present embodiment, the computer 300 calculates a prediction error using the average value 603 of actual values and the average value 602 of predicted values.

  In the present embodiment, the statistic calculation unit 330 calculates the first value by subtracting the average value 603 of the actual value from the average value 602 of the predicted value. Further, the statistic calculation unit 330 calculates the second value by dividing the first value by the average value 602 of the predicted values. The statistic calculation unit 330 calculates the third value by multiplying the second value by 100. The statistic calculator 330 stores the third value in the prediction error 604.

  Note that the computer 300 may hold FIGS. 4, 5, and 6 as one piece of information. That is, any column can be used as long as it can manage columns as shown in FIGS. 4, 5, and 6.

  FIG. 7 is an explanatory diagram illustrating an example of the prediction accuracy analysis graph 700 according to the first embodiment. FIG. 8 is an explanatory diagram illustrating an example of the analysis index information 800 according to the first embodiment.

  As will be described later, the prediction accuracy analysis graph 700 and the analysis index information 800 are generated by the prediction accuracy analysis unit 340.

  The prediction accuracy analysis graph 700 is a graph that uses an analysis index of prediction accuracy of a prediction model at the time of calculation. In the prediction accuracy analysis graph 700 of FIG. 7, the vertical axis indicates the prediction error of each group, and the horizontal axis indicates the group number. The average value 602 of predicted values or the average value 603 of actual values may be set on the horizontal axis. The prediction accuracy analysis graph 700 can be appropriately changed according to the purpose of analysis.

  In FIG. 7, the prediction accuracy analysis graph 700 is a graph in which the prediction error of each group is plotted, but may be a curve indicating a change tendency of the prediction error of each group.

  The analysis index information 800 includes an average value 801 of prediction errors and a variance 802 of prediction errors. The average value 801 of prediction errors is an average value of prediction errors calculated from the prediction error of each group. The prediction error variance 802 is a variance of the prediction error calculated from the prediction error of each group.

  Note that the analysis index information 800 may include statistics other than the average value of prediction errors and the variance of prediction errors.

  Next, details of processing executed by the computer 300 will be described.

  FIG. 1 is a flowchart illustrating an example of processing executed by the computer 300 according to the first embodiment.

  The computer 300 executes the sorting process of the individual records of the individual information 400 based on the size of the predicted value (Step S101).

  Specifically, the record sorting unit 321 of the record management unit 320 assigns a sort number to the individual record of the individual information 400 in descending order of the predicted value 402, and sorts the individual record based on the sort number. The record sort unit 321 stores the sort number in the sort number 503 of the group information 500 based on the sorting result. Note that the rules for sorting the individual records are not limited to those described above. For example, a method of sorting the records of the individual information 400 in ascending order of the predicted value 402 can be considered.

  Next, the computer 300 generates a plurality of groups from the plurality of records based on the sorting result of the individual records of the individual information 400 (step S102).

  Specifically, the group generation unit 322 of the record management unit 320 generates a group based on at least one of the predicted value 402 of the individual information 400 and the sort number 503 of the group information 500. At this time, the group generation unit 322 gives a group number to each generated group. Further, the group generation unit 322 sets the group number of the group to which each record belongs in the group number 502 of each record. As a group generation method, for example, the following method can be considered.

  (Generation Method 1) The group generation unit 322 selects 100 records in the descending order of the sort number 503, and generates the 100 records as one group.

  In the generation method 1, since the number of records included in each group is the same, if the expected values of the actual values of all the records included in the group match, the average value of the actual values and the actual value The difference from the expected value can be set to an arbitrary threshold value or less by the law of large numbers. However, since there is little possibility of satisfying the above condition, an error occurs between the average value of the actual values and the expected value of the actual values.

  (Generation Method 2) The group generation unit 322 determines a range of predicted values and assigns a group number to the range of predicted values. The group generation unit 322 determines the individual records to be included in each group by classifying the range in which the predicted value of each individual record is included.

  For example, the predicted value for each record is a range from 0.5 to 0.4, a range from 0.4 to 0.3, a range from 0.3 to 0.2, a range from 0.2 to 0.1, and Divide into four ranges, 0.1 or 0.0. The group generation unit 322 classifies each individual record for each range of predicted values.

  In the generation method 2, an error between the average value of the actual values and the expected value of the actual values can be suppressed. However, the number of records belonging to a group varies from group to group.

  Note that the generation method 1 and the generation method 2 may be combined. The above is the description of the processing in step S102.

  Next, the computer 300 calculates the statistic of the predicted value and the statistic of the actual value for each group (step S103).

  Specifically, the group statistic calculation unit 331 of the statistic calculation unit 330 selects a group. The group statistic calculation unit 331 refers to the group information 500 and the individual information 400, and based on the predicted value 402 and the actual value 403 of the individual record included in the selected group, the predicted value statistic and the actual value statistic Calculate the amount. The group statistic calculation unit 331 sets the calculated statistic in the corresponding record of the group statistic information 600.

  In this embodiment, the group statistic calculation unit 331 calculates the average value of the predicted values 402 and the average value of the actual values 403 as the group statistic. The calculated values are stored in an average value 602 of predicted values and an average value 603 of actual values.

  The group statistic calculation unit 331 may calculate the cumulative value of the predicted value 402 and the cumulative value of the actual value 403 or the variance of the predicted value 402 and the variance of the actual value 403 as the group statistic.

  Next, the computer 300 calculates a prediction error of each group based on the statistical value of the predicted value and the statistical value of the actual value of each group (step S104).

  Specifically, the prediction error calculation unit 332 of the statistic calculation unit 330 selects a group. The prediction error calculation unit 332 refers to the group statistical information 600 and calculates the prediction error of the group based on the average value 602 of the predicted values and the average value 603 of the actual values corresponding to the selected group.

  The following methods can be considered as a calculation method of the prediction miscalculation.

  (Calculation Method 1) The prediction error calculation unit 332 calculates the difference between the average value 602 of predicted values and the average value 603 of actual values. The difference between the average values can be obtained from the absolute value of the value obtained by subtracting the average value 603 of the actual value from the average value 602 of the predicted value. The prediction error calculation unit 332 sets a value obtained by dividing the difference between the calculated average values by the average value 602 of the prediction values as a prediction error.

  The prediction error calculated by the calculation method 1 is a value indicating how close the actual value is to the predicted value when the predicted value is given. When the predicted value is relatively more stable than the actual value, the prediction error calculated by the calculation method 1 is also relatively stable. Therefore, the prediction accuracy of the prediction model can be used as a value suitable for analysis.

  (Calculation Method 2) The prediction error calculation unit 332 calculates the difference between the average value 602 of predicted values and the average value 603 of actual values. The prediction error calculation unit 332 sets a value obtained by dividing the difference between the calculated average values by the average value 603 of the actual values as a prediction error.

  The prediction error calculated by the calculation method 2 is a value indicating how close the predicted value is to the actual value when the actual value is given. When the actual value is relatively more stable than the predicted value, the prediction error calculated by the calculation method 2 is also relatively stable. Therefore, the prediction accuracy of the prediction model can be used as a value suitable for analysis.

  Since the actual value is often random, there are few cases where the actual value is relatively more stable than the predicted value. The above is the description of the processing in step S104.

  Next, the computer 300 generates a prediction accuracy analysis graph 700 based on the group statistical information 600 (step S105).

  Specifically, the prediction accuracy analysis graph generation unit 341 of the prediction accuracy analysis unit 340 sets the prediction error 604 on the vertical axis and sets the group number 601 on the horizontal axis, so that the prediction accuracy as shown in FIG. An analysis graph 700 is generated. The average value 602 of predicted values or the average value 603 of actual values may be set on the horizontal axis. Which column is set on the horizontal axis can be appropriately changed according to the purpose of analysis.

  Next, the computer 300 selects a group used for prediction accuracy analysis based on the group statistical information 600 (step S106).

  Specifically, the prediction accuracy analysis index calculation unit 342 of the prediction accuracy analysis unit 340 selects a group based on the group statistical information 600 and the analysis purpose. As a group selection method, for example, the following method can be considered.

  (Selection Method 1) The prediction accuracy analysis index calculation unit 342 selects a predetermined number of groups based on the group number 601. For example, the prediction accuracy analysis index calculation unit 342 selects five groups with the group number 601 larger than the threshold, or selects three groups with the group number 601 smaller than the threshold.

  (Selection Method 2) The prediction accuracy analysis index calculation unit 342 selects a predetermined number of groups based on the average value 602 of predicted values. For example, the prediction accuracy analysis index calculation unit 342 selects five groups having an average predicted value 602 greater than 0.7, or selects two groups having an average predicted value 602 smaller than 0.2. .

  (Selection Method 3) The prediction accuracy analysis index calculation unit 342 selects a predetermined number of groups based on the average value 603 of actual values. For example, the prediction accuracy analysis index calculation unit 342 selects five groups having an average value of actual values 603 larger than 0.7, or selects two groups having an average value of actual values 603 smaller than 0.2. .

  (Selection Method 4) The prediction accuracy analysis index calculation unit 342 selects a predetermined number of groups based on the prediction error 504. For example, the prediction accuracy analysis index calculation unit 342 selects seven groups in which the prediction error 504 is greater than 20%, or selects five groups in which the prediction error 504 is less than 10%.

  (Selection Method 5) The prediction accuracy analysis index calculation unit 342 refers to the individual information 400 and the group information 500, and generates a cumulative graph 900 in which the vertical axis is the predicted value 402 and the horizontal axis is the group number 502. The prediction accuracy analysis index calculation unit 342 specifies a group in which the increase amount of the predicted value accumulation value or the increase amount of the actual value accumulation value is larger than a predetermined threshold. The prediction accuracy analysis index calculation unit 342 selects the identified group as a group used for analysis of prediction accuracy.

  FIG. 9 is an explanatory diagram illustrating an example of the cumulative graph 900 according to the first embodiment.

  The cumulative value with the group number “5” is the total value of the average values 302 of the predicted values of the groups with the group numbers “1” to “5”. The vertical axis may be a statistic other than the predicted value 402.

  In this embodiment, the records of the individual information 400 are sorted in descending order of the predicted value 402, and a group is generated based on the sorting result. Therefore, the smaller the group number, the larger the average value of the predicted values 402. This indicates that the distribution of the actual values of the group having a large average value of the predicted values 402 is a non-uniform distribution. That is, it indicates that the true value (expected value) of the actual value can be predicted. On the other hand, when the distribution of the actual values is uniform, the actual values are random and the true value of the actual value cannot be predicted.

  The non-uniformity of the distribution of actual values can be regarded as the amount of change in the accumulated value of the accumulated graph 900. When the distribution is a non-uniform real value, the cumulative value increases rapidly. That is, the amount of change in the accumulated value is large. When the distribution is a uniform real value, the cumulative value increases gradually. That is, the amount of change in the accumulated value is constant.

  Features as described above can also be grasped from the cumulative graph 900. The amount of change in the accumulated value of the group with the smaller group number 502 is a curve. On the other hand, the amount of change in the accumulated value of the group having the larger group number 502 is a straight line.

  In the selection method 5, a group is selected based on the characteristics as described above. Specifically, the prediction accuracy analysis index calculation unit 342 uses the cumulative graph 900 to identify a group in which the change amount of the cumulative value is a constant value. Furthermore, the prediction accuracy analysis index calculation unit 342 selects from the first group to the specified group. In the example shown in FIG. 9, groups with group numbers “1” to “10” are selected.

  In the selection method 5, it is possible to appropriately extract a portion with high prediction accuracy. That is, it is possible to identify a portion where an event may occur. As a modification of the selection method 5, the computer 300 can similarly extract a portion with low prediction accuracy.

  A conventional index of prediction accuracy can be controlled by raising or lowering the predicted value of the entire observation target. Therefore, the prediction accuracy of a specific observation target cannot be analyzed. However, in the present invention, the prediction accuracy of a specific observation target can be analyzed according to the selection method.

  The selection method described above is an example, and the present invention is not limited to this. What is necessary is just to select a group according to the objective.

  Next, the computer 300 calculates an analysis index based on the group statistical information 600 (step S107).

  Specifically, the prediction accuracy analysis index calculation unit 342 of the prediction accuracy analysis unit 340 refers to the prediction accuracy analysis graph 700 and calculates an average value of prediction errors and a variance of prediction errors of the selected group as analysis indexes. . The prediction accuracy analysis index calculation unit 342 stores the calculated values in the prediction error average value 801 and the prediction error variance 802 of the analysis index information 800.

  In the first embodiment, the statistic of the predicted value and the statistic of the actual value are calculated for each group, but the present invention is not limited to this. For example, the same processing may be executed for each record. In this case, the process of step S102 is omitted. In addition, after the records are sorted, the computer 300 calculates a prediction error based on the predicted value and the actual value of each record. The computer 300 generates a prediction accuracy analysis graph 700. Further, the computer 300 selects a target record group based on the sorting result, and calculates a prediction error statistic of the selected record group. Further, the computer 300 calculates an analysis index based on the statistical amount of the prediction error. For example, a method of calculating the statistical amount of prediction error as an analysis index as it is is conceivable.

  As described above, according to the first embodiment, the computer 300 sorts records based on the size of the predicted value, and selects a record or a group based on the sorting result, thereby analyzing according to a specific analysis purpose. An index can be calculated. Further, a value corresponding to the difference between the predicted value and the expected value of the actual value can be calculated based on the statistical value of the predicted value and the actual value of the group. Thereby, an analysis index for accurately analyzing the prediction accuracy can be calculated.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. Further, for example, the above-described embodiments are described in detail for easy understanding of the present invention, and are not necessarily limited to those provided with all the described configurations. Further, a part of the configuration of the embodiment can be added to, deleted from, or replaced with another configuration.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. The present invention can also be realized by software program codes that implement the functions of the embodiments. In this case, a storage medium in which the program code is recorded is provided to the computer, and a CPU included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing it constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, SSD (Solid State Drive), optical disk, magneto-optical disk, CD-R, magnetic tape, A non-volatile memory card, ROM, or the like is used.

  The program code for realizing the functions described in the present embodiment can be implemented by a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, Java (Java is a registered trademark).

  Furthermore, by distributing the program code of the software that realizes the functions of the embodiments via a network, the program code is stored in a storage means such as a hard disk or a memory of a computer or a storage medium such as a CD-RW or CD-R. The CPU included in the computer may read and execute the program code stored in the storage unit or the storage medium.

  In the above-described embodiments, the control lines and information lines indicate what is considered necessary for the explanation, and not all control lines and information lines on the product are necessarily shown. All the components may be connected to each other.

300 computer 301 storage device 310 arithmetic device 311 memory 312 input device 313 output device 314 storage medium 320 record management unit 321 record sort unit 322 group generation unit 330 statistic calculation unit 331 group statistic calculation unit 332 prediction error calculation unit 340 prediction accuracy Analysis unit 341 Prediction accuracy analysis graph generation unit 342 Prediction accuracy analysis index calculation unit 351 Individual information storage unit 352 Group information storage unit 353 Analysis result storage unit 400 Individual information 500 Group information 600 Group statistical information 700 Prediction accuracy analysis graph 800 Analysis index information 900 Cumulative graph

Claims (10)

A computer that analyzes the prediction accuracy of a prediction model for calculating a prediction value of a prediction item to be observed,
The calculator includes an arithmetic device, a memory connected to the arithmetic device, and an interface connected to the arithmetic device.
The computer manages a database storing a plurality of records composed of the predicted value of the prediction item to be observed and the actual value of the prediction item to be observed,
The arithmetic device sorts a plurality of records stored in the database based on the size of the predicted value, and stores the sorting result in the memory,
The arithmetic unit calculates a prediction error of each of the plurality of records based on the predicted value and the actual value of the plurality of records stored in the database, and each of the plurality of records stored in the database Storing the prediction error in the memory,
The arithmetic device selects a plurality of records as a target based on the sorting result, and based on a statistic calculated from the prediction error of each of the selected plurality of records, the prediction accuracy of the prediction model The computer calculates an analysis index for analyzing and stores the analysis index in the memory. The computer according to claim 1,
The actual value of the prediction item to be observed is a value indicating whether or not an arbitrary event occurs,
The arithmetic device sorts a plurality of records stored in the database in the descending order of the predicted value or in ascending order of the predicted value,
The arithmetic unit generates a plurality of groups including any number of the records based on the sorting result, stores group information for managing the group in the memory,
The arithmetic device calculates a statistic of the actual value and a statistic of the predicted value of the group based on the actual value and the predicted value of the plurality of records included in the group, and the plurality of groups Storing statistical information including the statistics of each of the actual values and the statistics of the predicted values in the memory,
The arithmetic device calculates the prediction error of each of the plurality of groups based on the statistical information,
The arithmetic unit selects a plurality of target groups based on the sorting result, and calculates the analysis index based on a statistic calculated from the prediction error of each of the selected groups. A computer characterized by The computer according to claim 2,
The arithmetic device calculates, as the analysis index, at least one of an average value of the prediction errors and a variance of the prediction errors based on the prediction errors of each of the plurality of selected groups. calculator. The computer according to claim 2,
The arithmetic unit generates cumulative information related to the cumulative value of the predicted value and the cumulative value of the actual value based on the sorting result, and stores the generated cumulative information in the memory,
The arithmetic device refers to the cumulative information, identifies a group in which either the increase amount of the cumulative value of the predicted value or the increase amount of the cumulative value of the actual value is greater than a predetermined threshold,
A computer that calculates the analysis index based on the prediction error of the identified group. The computer according to claim 2,
The arithmetic unit determines a range of the predicted value,
The computing device generates the plurality of groups by classifying the records based on a range of the predicted values. An analysis index calculation method executed by a computer that analyzes the prediction accuracy of a prediction model for calculating a prediction value of a prediction item to be observed,
The calculator includes an arithmetic device, a memory connected to the arithmetic device, and an interface connected to the arithmetic device.
The computer manages a database storing a plurality of records composed of the predicted value of the prediction item to be observed and the actual value of the prediction item to be observed,
A first step in which the arithmetic device sorts a plurality of records stored in the database based on the size of the predicted value, and stores a sorting result in the memory;
The arithmetic unit calculates a prediction error of each of the plurality of records based on the predicted value and the actual value of the plurality of records stored in the database, and each of the plurality of records stored in the database A second step of storing the prediction error in the memory;
The arithmetic unit selects a plurality of records to be processed based on the sorting result, and analyzes the prediction accuracy of the prediction model based on the prediction error statistic of each of the selected plurality of records. And a third step of storing the analysis index in the memory, and a method for calculating the analysis index. A method of calculating an analysis index according to claim 6,
The actual value of the prediction item to be observed is a value indicating whether or not an arbitrary event occurs,
The first step includes
A fourth step in which the arithmetic device sorts a plurality of records stored in the database in the descending order of the predicted value or in ascending order of the predicted value;
A fifth step of generating a plurality of groups including an arbitrary number of the records based on the sorting result, and storing group information for managing the groups in the memory;
The second step includes
The arithmetic device calculates a statistic of the actual value and a statistic of the predicted value of the group based on the actual value and the predicted value of the plurality of records included in the group, and the plurality of groups A sixth step of storing, in the memory, statistical information including the statistics of each of the actual values and the statistics of the predicted values;
And a seventh step of calculating the prediction error of each of the plurality of groups based on the statistical information.
The third step includes
An eighth step in which the arithmetic unit selects a plurality of target groups based on the sorting result;
The arithmetic unit includes: a ninth step of calculating the analysis index based on a statistic calculated from the prediction error of each of the selected plurality of groups. Calculation method. A method for calculating an analysis index according to claim 7,
In the ninth step, the computing device uses, as the analysis index, at least one of an average value of the prediction errors and a variance of the prediction errors based on the prediction errors of each of the selected groups. A method of calculating an analysis index, comprising a step of calculating. A method for calculating an analysis index according to claim 7,
The eighth step includes
The arithmetic unit generates, based on the sorting result, cumulative information related to the cumulative value of the predicted value and the cumulative value of the actual value, and stores the generated cumulative information in the memory;
The arithmetic device refers to the accumulated information, and specifies a group in which either the increase amount of the accumulated value of the predicted value or the increase amount of the accumulated value of the actual value is larger than a predetermined threshold value. ,
The ninth step includes a step of calculating the analysis index based on the prediction error of the specified group by the arithmetic device. A method for calculating an analysis index according to claim 7,
The fifth step includes
The arithmetic unit determining a range of the predicted value;
The calculation device includes a step of generating the plurality of groups by classifying the records based on the range of the predicted values.

Images