JP2021012594A - System, method, and program - Google Patents

Abstract

To reduce time and effort for maintaining fairness in operating a plurality of learned models.SOLUTION: A system monitors a plurality of models created through machine learning, and comprises: monitoring means that, when a result of prediction with a first model using input including a plurality of values satisfies a predetermined condition, issues a warning about a specific item of the input corresponding to the predetermined condition; and providing means that provides, about the specific item, a message for promoting the setting of a condition to be monitored by the monitoring means to a second model different from the first model. When the predetermined condition is set to the first model, in at least one of a case where the result of prediction with the first model is determined to satisfy the predetermined condition and a case where the second model is registered in the system, the providing means provides the message.SELECTED DRAWING: Figure 1

Description

The present invention relates to systems, methods, and programs for monitoring trained models of machine learning.

In machine learning, the learning model is adjusted using the learning data, and the adjusted learning model is put into production as a learned model. Conventionally, a prediction system that predicts a certain matter using such a trained model has been known. For example, Patent Document 1 discloses that when the user feels uncomfortable with the judgment result of the trained model, the user himself modifies the trained model by changing the weighting of the judgment element. ..

Japanese Unexamined Patent Publication No. 2019-16279

By the way, when a user operates a trained model, there is a case where another trained model is operated for each of a plurality of tasks. The case of operating a plurality of trained models in this way has not been considered in the prior art as in Patent Document 1.
The trained model may have unintended biases and may not be fair. For example, in a system that predicts the approval or disapproval of a loan to a target person, the approval or disapproval of the loan is determined by a learned model based on input data such as the age, gender, income, and years of service of the target person. Such a trained model may unintentionally be biased, such as making it easier to predict that men are approved than women, and easier to predict that older people are approved than younger people. If there is such a bias in the trained model in operation, it is desirable to correct this bias and ensure fairness.
Conventionally, when operating multiple trained models, it is necessary to verify the bias for each of the multiple trained models, and it is necessary to consider and set the items that affect fairness and their criteria. However, there was a problem that it took time and effort. In addition, among multiple trained models, one trained model had an appropriate setting, but another trained model that should have the same setting was missing a setting. It could happen.
An object of the present invention is to provide a system that can reduce the effort for maintaining fairness when operating a plurality of trained models.

The system of one embodiment of the present invention is a system that monitors a plurality of models created by machine learning, and the result of prediction by the first model using an input including a plurality of values satisfies a predetermined condition. In addition, a monitoring means that warns about a specific item of the input corresponding to the predetermined condition and a second model different from the first model are subject to monitoring of the specific item by the monitoring means. The providing means includes a providing means for providing a message prompting the setting of the above-mentioned condition, and when the predetermined condition is set for the first model, the providing means determines the result of prediction by the first model. The message is provided when it is determined that the predetermined condition is satisfied and at least one of the cases where the second model is registered in the system.

According to the present invention, it is possible to reduce the time and effort required to maintain fairness when operating a plurality of trained models.

It is a schematic diagram which shows the whole structure of the system which concerns on Example 1 of this invention. It is a block diagram which shows the internal structure of the information processing apparatus which comprises the system which concerns on Example 1 of this invention. (A) is a block diagram showing an example of the functional configuration of the fairness management system 101, (B) is a block diagram showing an example of the functional configuration of the prediction system 102, and (C) is a block diagram showing an example of the functional configuration of the business system 103. It is a block diagram which shows an example of the functional structure of. (A) and (B) are examples of screens displayed by the display unit 304 of the fairness management system 101. (A) is an example of the life insurance enrollment examination screen displayed by the display unit 311 of the business system 103-1. (B) is an example of the automobile insurance enrollment examination screen displayed by the display unit 311 of the business system 103-2. This is an example. It is a sequence diagram which showed the processing of the business system 103, the fairness management system 101, and the prediction system 102 when the insurance participation examination is predicted. It is a flowchart of fairness monitoring process. FIG. 4 is a diagram showing a state in which the prediction system monitoring screen 400-1 shown in FIG. 4A is displayed by the process of FIG. 7.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

<System configuration>
FIG. 1 is a schematic view showing an overall configuration of a system according to a first embodiment of the present invention.
The system 100 according to the first embodiment of the present invention includes a fairness management system 101, a prediction system 102-1, a prediction system 102-2, a business system 103-1 and a business system 103-2. The fairness management system 101, the prediction system 102-1, the prediction system 102-2, the business system 103-1 and the business system 103-2 are each composed of one or more information processing devices. A plurality of systems may be composed of one information processing device. Here, the information processing device refers to various computers such as personal computers, virtual machines, and the like. A virtual machine is, for example, a virtual computer whose operation is emulated by a software program. The information processing device 200, which is an example of this information processing device, will be described later with reference to FIG. The fairness management system 101 is a system that monitors whether the trained models operating in the prediction system 102-1 and the prediction system 102-2 can ensure fairness. The trained model is created by training the training model with training data. The trained model is a model created by machine learning.

Here, fairness is a viewpoint of whether or not it is fair to, for example, a user who operates a learned model and receives a prediction result, and a user who actually receives a service based on the prediction result. Fairness is, for example, the perspective of fairness to institutional, customary, and cultural standards in the country and environment in which the learned model is actually operated. In other words, the criteria for judging fairness may change depending on the user and the usage environment.
And, in this embodiment, the fact that such fairness is guaranteed for the output using the prediction result of the trained model is a state in which the fairness to such a standard is satisfied. It can be said that. The criteria for determining fairness can be set, for example, by a user who operates a trained model.

The prediction system 102-1 is a system that receives a request from the business system 103-1 and makes a prediction using a trained model. The business system 103-1 is a system that examines insurance coverage for the contents of applications from applicants for life insurance coverage. The trained model of the prediction system 102-1 is an example of the second model.
The prediction system 102-2 is a system that receives a request from the business system 103-2 and makes a prediction using a trained model. The business system 103-2 is a system that examines insurance coverage for the contents of applications from applicants for automobile insurance. The trained model of the prediction system 102-2 is an example of the first model.
The network 104 is a network such as the Internet, and is not particularly limited in type. The fairness management system 101, the prediction system 102-1, the prediction system 102-2, the business system 103-1 and the business system 103-2 can communicate with each other via the network 104.

<Internal configuration of information processing device>
FIG. 2 is a block diagram showing an example of the internal configuration of the information processing apparatus constituting the system 100 shown in FIG.
Each of the fairness management system 101, the prediction system 102-1, the prediction system 102-2, the business system 103-1 and the business system 103-2 shown in FIG. 1 is configured by the information processing device 200 of FIG. ..
The information processing device 200 has a CPU 201 that executes a program that controls the operation of the system 100 according to the present embodiment. CPU is an abbreviation for Central Processing Unit. The information processing device 200 has a system bus 204. The CPU 201 comprehensively controls each hardware connected to the system bus 204. The information processing device 200 includes an HDD 210 and a disk controller 209. HDD is an abbreviation for Hard Disk Drive. The HDD 210 is an example of a storage device, and here, it is a large-capacity storage device. The HDD 210 stores a program executed by the CPU 201. The disk controller 209 controls the HDD 210.

The information processing device 200 has a memory 202. The memory 202 functions as a main memory of the CPU 201, a work area, and the like. The information processing device 200 has a NIC 203. NIC is an abbreviation for Network Interface Card. The NIC 203 bidirectionally transmits and receives data to and from other nodes connected to the network 104 via the network 104. The information processing device 200 has a keyboard 206 and a keyboard controller 205. The user operates the keyboard 206 to input instructions to the information processing device 200. The keyboard controller 205 controls to accept instruction input by the keyboard 206. The information processing device 200 may not have the keyboard controller 205 and the keyboard 206 depending on its role. The information processing device 200 has a display 208 and a display controller 207. The display 208 is a display module composed of a liquid crystal display or the like. The display controller 207 controls the display on the display 208. The information processing device 200 may not have the display 208 and the display controller 207 depending on its role.

Hereinafter, the configurations of the fairness management system 101, the prediction system 102-1, the prediction system 102-2, the business system 103-1 and the business system 1032 will be further described. The configurations of the prediction system 102-1 and the prediction system 102-2 will be described with reference to the prediction system 102 shown in FIG. 3 (B). Further, the configurations of the business system 103-1 and the business system 103-2 will be described with reference to the business system 103 shown in FIG. 3 (C).

<Functional configuration of fairness management system>
FIG. 3A is a block diagram showing an example of the configuration of the fairness management system 101.
The fairness management system 101 includes a control unit 301, a data storage unit 302, a communication unit 303, and a display unit 304. Each configuration of the fairness management system 101 shown in FIG. 3A is realized by the CPU 201 reading the program stored in the HDD 210 of the fairness management system 101 into the memory 202 and executing the program.

The control unit 301 is a software module that receives execution instructions of various processes from the display unit 304 and the like and executes various processes.
The data storage unit 302 is a database configured in the HDD 210.
The communication unit 303 is a software module for communicating with the prediction system 102 of FIG. 3 (B) and the business system 103 of FIG. 3 (C) via the network 104.
The display unit 304 is a software module that displays the UI of the fairness management system 101 and accepts user operations. UI is an abbreviation for User Interface.

<Functional configuration of prediction system>
FIG. 3B is a block diagram showing an example of the functional configuration of the prediction system 102.
The prediction system 102 includes a control unit 305, a data storage unit 306, and a communication unit 307. Each configuration of the prediction system 102 shown in FIG. 3B is realized by the CPU 201 reading the program stored in the HDD 210 of the prediction system 102 into the memory 202 and executing the program.

The control unit 305 is a software module that executes various processes such as prediction processing.
The data storage unit 306 is a database configured in the HDD 210.
The communication unit 307 is a software module for communicating with the fairness management system 101 and the business system 103 via the network 104.

<Functional configuration of business system>
FIG. 3C is a block diagram showing an example of the functional configuration of the business system 103.
The business system 103 includes a control unit 308, a data storage unit 309, a communication unit 310, and a display unit 311. Each configuration of the business system 103 shown in FIG. 3C is realized by the CPU 201 reading the program stored in the HDD 210 of the business system 103 into the memory 202 and executing the program.

The control unit 308 is a software module that receives various processing execution instructions from the display unit 311 and executes various processing.
The data storage unit 309 is a database configured in the HDD 210.
The communication unit 310 is a software module for communicating with the fairness management system 101 and the prediction system 102 via the network 104.
The display unit 304 is a software module that displays the UI of the business system 103 and accepts user operations.

<Fairness monitoring setting process>
In this embodiment, a case where a certain insurance company A deals with life insurance and automobile insurance, and conducts insurance enrollment examination using each learned model will be described. The business system 103-1 in FIG. 1 is a system for performing life insurance business. The business system 103-2 of FIG. 1 is a system for performing automobile insurance business. Further, the prediction system 102-1 of FIG. 1 is a life insurance prediction system that performs enrollment examination using a learned model for life insurance business. The prediction system 102-2 of FIG. 1 is an automobile insurance prediction system that performs enrollment examination using a learned model for automobile insurance business. Insurance company A operates business system 103-1 and forecasting system 102-1, and business system 103-1 and forecasting system 102-1.

The prediction system 102-1 which is a life insurance prediction system uses a trained model trained to determine whether or not to enroll in life insurance by inputting, for example, "age" and "medical history". The trained model used in the prediction system 102-1 is stored in the data storage unit 306 of the prediction system 102-1. The trained model used in the prediction system 102-1 is called a life insurance model.
The prediction system 102-2, which is a car insurance prediction system, is learned to judge whether or not to enroll in car insurance by inputting, for example, "age", "gender", "years of license acquisition", and "driving frequency". Use the completed model. The trained model used in the prediction system 102-2 is stored in the data storage unit 306 of the prediction system 102-2. The trained model used in the prediction system 102-1 is called an automobile insurance model.

The fairness management system 101 monitors the fairness of the prediction system 102-1 and the prediction system 102-2. The fairness management system 101 manages the input items of the trained model of the prediction system 102-1 and the prediction system 102-2 to be monitored.
Table 1 is a table showing an example of the trained model management table. The trained model management table is stored in the data storage unit 302 of the fairness management system 101. In the trained model management table, one record indicates one trained model. The trained model management table has columns for users, models, and items. The trained model management table stores input items trained by each of the life insurance model and the automobile insurance model operated by the user of the fairness management system 101, that is, the insurance company A which is the user of the system 100. ing.

4 (A) and 4 (B) are examples of screens displayed by the display unit 304 of the fairness management system 101. This screen is generated by the display unit 304 of the fairness management system 101 in response to a request from the browser of the personal computer used by the person in charge of the prediction system management of the insurance company A, and is displayed on the browser of the personal computer. Personal computer is an abbreviation for personal computer. This personal computer is connected to the network 104 and can communicate with the fairness management system 101. This personal computer may have a configuration different from that of the fairness management system 101, or may have a configuration of the fairness management system 101.

FIG. 4A shows an example of a prediction system monitoring screen when the fairness management system 101 monitors two prediction systems, the prediction system 102-1 and the prediction system 102-2. The prediction system 102-1 is a life insurance prediction system that predicts the result of the life insurance enrollment examination, and the prediction system 102-2 is an automobile insurance prediction system that predicts the automobile insurance enrollment examination.
The prediction system monitoring screen 400-1 of FIG. 4A has an area 401-1 and an area 401-2, which are areas for each of the prediction systems to be monitored. Area 401-1 is an area for displaying the monitoring status and setting the monitoring for the prediction system 102-1. Area 401-2 is an area for displaying the monitoring status and setting the monitoring for the prediction system 102-2. Even when there are three or more prediction systems to be monitored, the area for each prediction system is displayed.
Each of the area 401-1 and the area 401-2 has a button for setting a monitoring item and an area for displaying the monitoring status. Region 401-1 has a button 402-1 for setting a monitoring item. In FIG. 4A, since the monitoring item for the prediction system 102-1 is not set, the area 401-1 for the prediction system 102-1 does not display the area for displaying the monitoring status. The area 401-2 has a button 402-2 for setting a monitoring item, and an area 403-1 and an area 4032 for displaying the monitoring status. The method of generating an area for displaying the monitoring status such as area 403-1 and area 4032 will be described later.

The person in charge of the prediction system management sets the monitoring for each prediction system from the prediction system monitoring screen 400-1. An example of the procedure for setting the monitoring setting will be described below.
The person in charge of the prediction system manager considered that there should be no discrimination based on gender and age regarding automobile insurance, and determined that gender and age were subject to fairness monitoring among the inputs to the prediction system 102-2. In addition, the person in charge of forecasting system management considered that there was no particular concern about fairness regarding life insurance, and decided that there was no monitoring target. In this case, the person in charge of the prediction system management presses the button 402-2 on the prediction system monitoring screen 400-1, and sets the monitoring for the automobile insurance prediction system.
The display unit 304 of the fairness management system 101 displays the screen of FIG. 4B when the button 402-2 of the prediction system monitoring screen 400-1 is pressed. FIG. 4B shows an example of a monitoring item setting screen displayed when the fairness management system 101 accepts the pressing of a button for setting a monitoring item.

When the control unit 301 of the fairness management system 101 receives the press of the button for setting the monitoring item, the control unit 301 acquires a record about the model of the prediction system associated with the button from the trained model management table (Table 1). .. For example, when the button 402-2 is pressed, the control unit 301 acquires a record in which the "model" column of the learned model management table is automobile insurance. Subsequently, the control unit 301 acquires the value of the “item” of the acquired record, and the display unit 304 of the fairness management system 101 sets the acquired “item” value to the monitoring item in FIG. 4 (B). Displayed on screen 400-2. For example, when the press of the button 402-2 is accepted, the values of the "items" of the record acquired by the control unit 301 are the age, the gender, the number of years of license acquisition, and the driving frequency. The display unit 304 displays the values of these "items" as monitoring item candidates on the monitoring item setting screen 400-2, and determines whether or not to set each of them as a monitoring item and related information by the person in charge of the prediction system management. Accept input.

As shown in FIG. 4B, the monitoring item setting screen 400-2 has a monitoring item setting area 406, a button 407 for confirming the setting of the monitoring item, and a button 408 for canceling the setting of the monitoring item. The display of the monitoring item candidate, whether or not to set the monitoring item candidate as the monitoring item, and the input of the related information of the monitoring item are performed in the monitoring item setting area 406. The information related to the monitoring item includes, for example, information to be monitored, information to be compared, and a threshold value. As for the setting method of the monitoring item, for example, the check box of the item to be monitored is checked, and the related information of the monitoring item is input by "value" or "range". In this input, whether to input by "value" or "range" is a target specification method. When inputting by "value", the "value" to be input may be selected from the pull-down menu.

In the example of FIG. 4B, "age" is selected as the monitoring target item, "18 to 25" years old is monitored, and the result is compared with the prediction result of "26 to 60" years old, and the threshold value is below 0.9. It is set to warn if it is. Further, in the example of FIG. 4B, when "gender" is selected as the monitoring target item, "female" is monitored, and the comparison result is compared with "male", and the threshold value is less than 0.9. It is set to warn. In the comparison method here, the ratio is calculated based on the approval rate of the monitoring target / the approval rate of the comparison target. The "age" of the input data is "18 to 25", which is an example of the first input value. The "age" of the input data is "26 to 60", which is an example of the second input value. The "gender" of the input data is "female", which is an example of the first input value. The "gender" of the input data is "male", which is an example of the second input value. The approval rate of the monitoring target is an example of the first ratio. The approval rate for comparison is an example of the second ratio. If the ratio is below the threshold value, it is judged that the trained model is biased and the monitored target is disadvantageous. "The ratio is below the threshold value" is an example of a predetermined condition. "When the ratio is below the threshold value" is an example of the case where the prediction result satisfies a predetermined condition. "When the ratio does not fall below the threshold value" is an example of the case where the prediction result does not satisfy the predetermined condition. “Approval” of a prediction result is an example in which the prediction result is a predetermined prediction result. The predetermined condition is, for example, between the result of the prediction by the first model using the input including the first input value and the result of the prediction by the first model using the input including the second input value. This is a condition that is satisfied when there is a bias in.
When the display unit 304 receives the pressing of the button 407 by the person in charge of the prediction system management on the monitoring item setting screen 400-2, the display unit 304 stores the monitoring content input on the monitoring item setting screen 400-2 in the fairness monitoring table.
Table 2 is a table showing an example of the fairness monitoring table. The fairness monitoring table is stored in the data storage unit 302 of the fairness management system 101. In the fairness monitoring table, one record indicates one monitoring item. The fairness monitoring table has columns for users, models, items, targeting methods, monitoring targets, comparison targets, and threshold values.

<Insurance application review process>
FIG. 5A is an example of the life insurance participation examination screen displayed by the display unit 311 of the business system 103-1. This screen is generated by the display unit 311 of the business system 103-1 in response to a request from the browser of the personal computer used by the person in charge of examining the application for life insurance, and is displayed on the browser of the personal computer. This personal computer is connected to the network 104 and can communicate with the business system 103-1. This personal computer may have a different configuration from the business system 103-1 or may configure the business system 103-1.
The life insurance participation examination screen 500-1 has a data input unit 501, a button 502, a pull-down 503, a button 504, and a button 505. Button 502 is a button for executing the prediction of the examination. The pull-down 503 is a pull-down that displays the prediction result and selects the judgment result of the person in charge. Button 504 is a button for confirming the entry examination result. Button 505 is a button for canceling the examination. Prediction results include "approval" to allow life insurance and "reject" to refuse life insurance.

FIG. 5B is an example of the automobile insurance participation examination screen displayed by the display unit 311 of the business system 103-2. This screen is generated by the display unit 311 of the business system 103-2 in response to a request from the browser of the personal computer used by the person in charge of examining the application for automobile insurance, and is displayed on the browser of the personal computer. This personal computer is connected to the network 104 and can communicate with the business system 103-2. This personal computer may have a configuration different from that of the business system 103-2, or may configure the business system 103-2.
The automobile insurance participation examination screen 500-2 has a data input unit 501, a button 502, a pull-down 503, a button 504, and a button 505. Button 502 is a button for executing the prediction of the examination. The pull-down 503 is a pull-down that displays the prediction result and selects the judgment result of the person in charge. Button 504 is a button for confirming the entry examination result. Button 505 is a button for canceling the examination. Forecast results include "approval" to allow car insurance and "reject" to refuse car insurance.

FIG. 6 is a sequence diagram showing the processing of the business system 103, the fairness management system 101, and the prediction system 102 when predicting the insurance coverage examination. Here, an example of the automobile insurance enrollment examination will be described, but the processing of the life insurance enrollment examination is the same as the processing of the automobile insurance enrollment examination.
In the case of automobile insurance, the data input unit 501 of the automobile insurance participation examination screen 500-2 has a name and address input fields, and a pull-down for selecting age, gender, number of years of license acquisition, and driving frequency.
The person in charge of examining the automobile insurance enrollment application inputs and selects the applicant's information in the data input unit 501 of the automobile insurance enrollment examination screen 500-2, and presses the button 502.

In step S601, the display unit 311 of the business system 103-2 accepts the press of the button 502, and the control unit 308 receives the press of the button 502, and responds to the prediction system 102-2 via the communication unit 310. Make a forecast request. The prediction request at this time includes the data input to the data input unit 401. Here, the data included in the prediction request does not have to be all the information of the applicant input to the data input unit 401, but only the data required by the trained model of the prediction system 102-2. For example, in this embodiment, the name and address are not included in the forecast request, but the age, gender, years of license acquisition, and driving frequency are included in the forecast request.
In step S602, the communication unit 307 of the prediction system 102-2 receives the prediction request, and the control unit 305 is included in the prediction request by the trained model stored in the data storage unit 306 in response to receiving the prediction request. Prediction is performed using the input data.

In this explanation, it is assumed that the prediction result in step S602 is "rejected".
In step S603, the communication unit 307 responds to the business system 103-2 with the prediction result in step S602.
In step S604, the communication unit 310 of the business system 103-2 receives the prediction result, and the display unit 311 displays the prediction result on the pull-down 503 of the automobile insurance subscription examination screen 500-2 in response to receiving the prediction result. .. In this explanation, since the prediction result is rejected, the display unit 311 sets the pull-down 503 in a state in which "rejected" is selected.

The person in charge of examining the application for automobile insurance conducts the examination with reference to the prediction result displayed in the pull-down 503. The person in charge can also change the prediction result of the prediction system 102-2 and set the pull-down 503 to "approve".
In step S605, the display unit 311 of the business system 103-2 accepts the press of the button 504, and the control unit 308 confirms the current selection state of the pull-down 503 as the examination result in response to the acceptance of the press of the button 504. Is stored in the data storage unit 309.
When the person in charge does not press the button 504 and presses the button 505, the display unit 311 accepts the press of the button 505, and the control unit 308 receives the press of the button 505 and goes to the data storage unit 309. The processing is terminated without saving the examination result of.

When the examination result is confirmed in step S606, the control unit 308 requests the fairness management system 101 for statistics via the communication unit 310. At this time, the data included in the prediction request in step S601 and the prediction result received in step S603 are included in the statistical request.
In step S607, the communication unit 303 of the fairness management system 101 receives the statistical request, and the control unit 301 performs statistics on the prediction result in response to receiving the statistical request. The prediction result statistics is a process of registering the result of the current prediction in the prediction result statistics table of Table 3.
Table 3 is a table showing an example of the prediction result statistical table. The prediction result statistical table is stored in the data storage unit 302 of the fairness management system 101. In the prediction result statistical table, one record indicates one monitoring target. The prediction result statistics table has columns for users, models, items, values, number of applications, and number of approvals. The prediction result statistical table holds the number of applications and the number of approvals for each monitoring target and comparison target of the items stored in the fairness monitoring table of Table 2. The number of applications is an example of the total number of predictions predicted by the first model using the inputs including the first input value. The number of approvals is an example of the number of times that the prediction result predicted by the first model using the input including the first input value is a predetermined prediction result.
Table 3 shows that the number of applications for the past application contents and application results of "18 to 25 years old" is 100 and the number of approvals is 81, and the age is "26 to 60 years old". It shows that the number of applications is 200 and the number of approvals is 180. Table 3 shows that the number of applications for gender "female" is 100 and the number of approvals is 81, and the number of applications for gender "male" is 200. It shows that the number of approvals is 180.

Here, it is assumed that the information of the applicant this time (the applicant whose prediction in step S602 is "rejected" in the above explanation) is "female" in gender and "20 years old" in age. In this case, in step S607, the control unit 301 adds 1 to the number of applications for the record whose age is "18 to 25 years old" in the prediction result statistical table to make 101 people, and does not add the number of approvals for this record. Leave 81 people. Further, the control unit 301 adds 1 to the number of applications for the record whose gender is "female" in the prediction result statistics table to 101 people, and does not add the number of approvals for this record and leaves 81 people.

In this embodiment, an example in which the statistics of the prediction result are performed after the examination result is confirmed (step S605) has been described. This is to exclude from the statistics the duplication when the same input data is predicted multiple times and the prediction for the input error. According to this embodiment, for example, the case where the button 502 is pressed multiple times and the case where the button 502 is pressed with incorrect data input can be excluded from the statistics.
The timing of statistics on prediction results is not limited to this. For example, in the case of an embodiment in which the prediction result cannot be canceled and is determined as the examination result, or in the case of a system in which an input error is not expected, the statistics of the prediction result may be performed at other timings. In such a case, when the prediction system 102 makes the prediction in step S602, the prediction result may be notified to the fairness management system 101, and statistics may be performed at this time.

<Fairness monitoring process>
Hereinafter, with reference to FIG. 7, a process executed by the fairness management system 101 and a process of monitoring the fairness of the prediction system 102 will be described. FIG. 7 is a flowchart of fairness monitoring processing. FIG. 8 is a diagram showing the prediction system monitoring screen 400-1 shown in FIG. 4 (A), showing a state in which the display is displayed by the process of FIG. 7. The prediction system monitoring screen 400-1 of FIG. 8 is generated by the display unit 304 of the fairness management system 101 in response to a request from the browser of the personal computer used by the person in charge of the prediction system management of the insurance company A, and the browser of the personal computer is generated. Is displayed in. This personal computer is connected to the network 104 and can communicate with the fairness management system 101. This personal computer may have a configuration different from that of the fairness management system 101, or may have a configuration of the fairness management system 101. In the prediction system monitoring screen 400-1 of FIG. 8, the same display as the prediction system monitoring screen 400-1 of FIG. 4A is designated by the same reference numerals as those of FIG. 4A.

In the prediction system monitoring screen 400-1 of FIG. 8, the areas 804-1 and 804-2 displayed in the area 401-1 with respect to the prediction system 102-1 are areas for displaying the monitoring setting recommended information described later. The process of displaying the monitoring setting recommended information is an example of the providing means for providing a message prompting the setting of the condition to be monitored by the monitoring means for a specific item. The "age" and "gender" that warned in the automobile insurance forecast are examples of specific items.
First, in the fairness monitoring process, the display unit 304 of the fairness management system 101 receives a display request from the browser of the personal computer. In step S701 of FIG. 7, when the control unit 301 receives the display request, the prediction result statistics are obtained from the prediction result statistics table (Table 3) and the fairness monitoring table (Table 2) stored in the data storage unit 302. Acquire information and monitoring setting information. The prediction result statistical information is the information stored in the prediction result statistical table. The monitoring setting information is the information stored in the fairness monitoring table.

In step S702, the control unit 301 determines fairness using the information acquired in step S701.
In the case of the state of Table 3, the approval rate (number of approvals / number of applications) is 0.81 when the monitoring item "age" is "18 to 25 years old", and the "age" is "26 to 60 years old". The approval rate in the case of is 0.9. Therefore, the ratio of the approval rate of the monitoring target / the approval rate of the comparison target is 0.9, which is not lower than the threshold value of 0.9 in the fairness monitoring table in Table 2. As a result, in this embodiment, the control unit 301 determines that the monitoring item "age" is "18 to 25 years old" in the normal range, and determines that it is "fair" as a judgment of fairness. To do.

Regarding the other monitoring item, "gender", the approval rate is 0.81 when the gender is "female", and the approval rate is 0.9 when the gender is "male". Therefore, the ratio of the approval rate of the monitoring target / the approval rate of the comparison target is 0.9, which is not lower than the threshold value of 0.9 in the fairness monitoring table in Table 2. As a result, in the present embodiment, the control unit 301 determines that the monitoring item "gender" is "female" in the normal range, and determines "fairness" as the determination of fairness.

Furthermore, from the state of Table 3, as described above, the data that the applicant's information is "female" in gender, "20 years old" in age, and "rejected" in the prediction result is added to the prediction result statistics table. The case where it is done will be described. In this case, the approval rate when the monitoring item "age" is "18 to 25 years old" is about 0.8, and the approval rate when the age is "26 to 60 years old" is 0.9. Therefore, the ratio of the approval rate of the monitoring target / the approval rate of the comparison target is about 0.89, which is below the threshold value of 0.9. From this, the control unit 301 determines that the monitoring item "age" is "18 to 25 years old" in a biased state, and determines that the fairness is "unfair".

Regarding the other monitoring item, "gender", the approval rate is about 0.8 when the gender is "female", and the approval rate is 0.9 when the gender is "male". Therefore, the ratio of the approval rate of the monitoring target / the approval rate of the comparison target is about 0.89, which is below the threshold value of 0.9. From this, the control unit 301 determines that the monitoring item "gender" is "female" in a biased state, and determines that the fairness is "unfair".

In step S703, the display unit 304 displays the fairness calculation result and the fairness determination result of the monitoring target for each monitoring item in the area 403-1 and the area 4032 according to the fairness determination result in step S702. Is displayed. This process is an example of a monitoring means that warns about a specific item of the input corresponding to the predetermined condition when the result of the prediction by the first model using the input including a plurality of values satisfies the predetermined condition.
FIG. 4A is a display example when it is determined in step S702 that the learned model of the automobile insurance prediction is in a fair state. In FIG. 4A, as the calculation result of fairness, “0.9” is displayed in the area 403-1 and “0.9” is displayed in the area 4032. In FIG. 4A, as the result of fairness determination, “normal” is displayed in the area 403-1 and “normal” is displayed in the area 4032.
In the case of FIG. 4A, since all the monitoring items are normal and not in the warning state, the control unit 301 determines in step S704 that the monitoring item in the warning state does not exist, and ends the process.

FIG. 8 is a display example when it is determined in step S702 that the learned model of the automobile insurance prediction is in an unfair state. In FIG. 8, as the calculation result of fairness, “0.89” is displayed in the area 403-1 and “0.89” is displayed in the area 4032. In FIG. 8, as a result of determining fairness, “warning” is displayed in the area 403-1 and “warning” is displayed in the area 4032.
In the case of FIG. 8, since all the monitoring items are in the warning state, in step S704, the control unit 301 determines that the monitoring item in the warning state exists, and proceeds to step S705.

In step S705, the control unit 301 uses the trained model management table in Table 1 to determine whether or not there is a trained model managed by the insurance company A other than the trained model in which the fairness warning has occurred. Confirm. In step S705, if the control unit 301 determines that there is no trained model managed by the insurance company A other than the trained model in which the fairness warning has occurred, the control unit 301 ends the process. In step S705, when the control unit 301 determines that there is a trained model managed by the insurance company A other than the trained model in which the warning of fairness has occurred, the control unit 301 proceeds to step S706. In the case of this embodiment, the warning occurred in the trained model of automobile insurance prediction, and there is a trained model of life insurance prediction as another trained model.

In step S706, the control unit 301 sets the monitoring item for which the fairness warning has occurred as a recommended non-display item in the trained model managed by the insurance company A other than the learned model for which the fairness warning has occurred. Check if it is. Recommended hidden items will be described later. In step S705, when the control unit 301 determines that the monitoring item for which the fairness warning has occurred is a recommended non-display item, the control unit 301 ends the process. If the control unit 301 determines in step S705 that the monitoring item for which the fairness warning has occurred is not a recommended non-display item, the process proceeds to step S707.

In step S707, the control unit 301 confirms whether or not a trained model different from the trained model in which the warning has occurred has the same item as the monitoring item in which the warning has occurred as an input item. In step S707, when the control unit 301 determines that the trained model different from the trained model in which the warning has occurred does not have the same item as the monitoring item in which the warning has occurred, the process proceeds to step S710. To do. In step S710, the display unit 304 displays that the training model is recommended to be modified, that is, re-learning, based on the determination result by the control unit 301 in step S707. That is, in step S710, the display unit 304 provides a message urging the trained model to be relearned with the training data including the monitoring item in which the warning has occurred as the input item. In step S707, when the control unit 301 determines that the trained model different from the trained model in which the warning has occurred has the same item as the monitoring item in which the warning has occurred, the process proceeds to step S708. To do.
For multiple trained models managed by the same user, if there is a bias in fairness in a particular item of one trained model, the other trained models created by that user will also be in that particular item. There may be a bias. For this reason, it is conceivable that the monitoring settings are omitted even though the same item needs to be monitored for a plurality of trained models. In the case of this embodiment, the monitoring items for which warnings occurred in the trained model of automobile insurance prediction are "age" and "gender", but the trained model of life insurance prediction has "age" as an item. I don't have "gender". In step S707, the control unit 301 confirms this situation.

In step S708, the control unit 301 confirms whether or not a trained model different from the trained model in which the warning has occurred has the same item as the monitoring item in which the warning has occurred as a monitoring item. In step S708, when the control unit 301 determines that the trained model other than the trained model in which the warning has occurred does not have the same item as the monitoring item in which the warning has occurred, the control unit 301 ends the process. .. In step S708, when the control unit 301 determines that the trained model different from the trained model in which the warning has occurred has the same item as the monitoring item in which the warning has occurred, the process proceeds to step S709. To do. In step S709, the display unit 304 displays that it is recommended to add to the monitoring item based on the determination result by the control unit 301 in step S708.

In this embodiment, in step S708, the control unit 301 confirms whether or not the “age” is possessed as a monitoring item in the learned model of the life insurance prediction. In step S708, the control unit 301 ends the process if it already has the “age” as a monitoring item in the learned model of the life insurance prediction. In this embodiment, in step S708, the control unit 301 does not have the “age” as a monitoring item in the trained model of the life insurance prediction, and therefore determines that it is recommended to add it to the monitoring item. In this embodiment, in step S709, the display unit 304 recommends adding "age" to the monitoring item in the area 804-1 of FIG. 8 based on the determination result by the control unit 301 in step S708. Display.

In this embodiment, in step S707, the control unit 301 adds "gender" as an input item of the trained model of the life insurance forecast, and "gender" to the monitoring item of the trained model of the life insurance forecast. Make a decision to recommend adding. In this embodiment, in step S710, the display unit 304 learns the life insurance prediction so that "gender" is used as an input item in the area 804-2 of FIG. 8 based on the determination result by the control unit 301 in step S707. Display that it is recommended to modify the completed model. Further, in this embodiment, in step S710, the display unit 304 recommends adding "gender" to the monitoring item in the area 804-2 of FIG. 8 based on the determination result by the control unit 301 in step S707. A message to that effect may be displayed.
The trained model of life insurance forecast does not have "gender" as the trained input item. That is, although "gender" does not contribute to the prediction of the prediction system 102-1, as a result, it becomes a learned model that derives a prediction result that makes the applicant suspect that there is a bias due to "gender". It is possible that there is. According to this embodiment, by recommending the review of the trained model itself as described above, it becomes possible for the person in charge of the prediction system manager to grasp the items that are likely to be a problem from the viewpoint of bias. It is possible to provide awareness for building an unbiased trained model.

The person in charge of the prediction system manager confirms the recommended information displayed in the areas 804-1 and 804-1 in FIG. 8, judges whether the recommended contents are appropriate, corrects the omission of the monitoring setting, and has learned. It is possible to rebuild the model.
In this embodiment, when the person in charge of the prediction system manager confirms the recommended information displayed in the areas 804-1 and 804-1 in FIG. 8 and determines that the application of the recommended contents is unnecessary, the recommended display is not displayed. It is possible to set it to be displayed and not to display the same recommendation thereafter. The display unit 304 displays the button 805-1 in the area 804-1 and the button 805-2 in the area 804-2 as a recommended unnecessary button for making such a setting.
When the display unit 304 receives the pressing of the button 805-1, the display unit 304 hides the recommended display regarding "age" for the trained model of the life insurance prediction. Further, when the display unit 304 receives the pressing of the button 805-2, the display unit 304 hides the recommended display regarding "gender" for the trained model of the life insurance prediction. Further, the control unit 301 stores information to that effect in the recommendation non-display table so that the same recommendation is not displayed thereafter. In the determination of step S707 and step S708, the control unit 301 does not recommend adding the recommended hidden items stored in the recommended hidden table as input items of the trained model and adding them to the monitoring items. Judge.
Table 4 is a table showing an example of the recommended hidden table. The recommended hidden table is stored in the data storage unit 302. In the recommended hidden table, one record indicates one recommended hidden item. Table 4 shows the case where "age" is set as the recommended hidden item in the life insurance forecast. The recommended hidden table has columns for users, models, and recommended hidden items.

In this embodiment, a method of determining fairness in response to a request from a browser has been shown, but the fairness is determined when the prediction result statistics are performed in the prediction result statistical processing (step S607) of FIG. You may make a judgment.
In that case, the fairness management system 101 may hold the e-mail address of the person in charge of the prediction system management in advance, and if the value is below the threshold value, the person in charge of the prediction system management may be notified by e-mail or the like.
Further, in that case, the fairness management system 101 may automatically add the monitoring item for which the warning has occurred to the monitoring item of another trained model, instead of displaying the recommendation.

In this embodiment, the fact that a bias warning actually occurs means that the trained model created by the user may be biased in a specific item, and therefore another trained model is trained when the warning occurs. An example is shown that recommends similar monitoring settings for the model. The purpose of this is to make necessary and sufficient monitoring settings while avoiding an excessive increase in the time and effort of the forecasting system manager due to excessive monitoring items. However, the timing for recommending monitoring settings is not limited to this. For example, in the fairness management system 101, when a monitoring item is set in one trained model on the monitoring item setting screen (FIG. 4 (B)), the same monitoring item is set for another trained model. It may be recommended to set. When predetermined monitoring conditions are set for the trained model of the automobile insurance prediction system, which is the first model, it is recommended to set the same monitoring items for the trained model of the automobile insurance prediction system. It may be. In addition, when a new prediction system is registered in the system 100, it is recommended to set the monitoring settings in the trained model of the newly registered prediction system for the items that have been monitored and set in the already registered prediction system. You may do so.

This example shows an example in which the monitoring items of one trained model are recommended to another trained model for all trained models operated by insurance company A, but even if the scope is limited. Good. For example, a plurality of learned models managed by the user may be grouped according to the business content, and the monitoring items may be recommended only within the group.
Further, when the monitoring items of a plurality of trained models managed by the user are similar, it may be recommended to standardize the monitoring items. For example, when the number of monitoring items of each trained model is common to a certain ratio or more, it may be recommended to standardize all monitoring items instead of recommending individual monitoring items.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Fairness management system 102 Prediction system 103 Business system

Claims (9)

A system that monitors multiple models created by machine learning.
When the result of prediction by the first model using inputs including a plurality of values satisfies a predetermined condition, a monitoring means for giving a warning about a specific item of the input corresponding to the predetermined condition, and
A second model different from the first model has a providing means for providing a message prompting the setting of conditions to be monitored by the monitoring means for the specific item.
The providing means includes, when the predetermined condition is set for the first model, when it is determined that the result of prediction by the first model satisfies the predetermined condition, and when the second model A system characterized in that the message is provided in at least one of the cases when is registered in the system. It also has a grouping means for grouping multiple models,
The providing means
When the first model and the second model are grouped, the message is provided.
The system according to claim 1, wherein the message is not provided to the third model that is not grouped with the first model. The system according to claim 1 or 2, further comprising an input means for eliminating the need to set the conditions when the message is provided. The claim is characterized in that the providing means does not provide the message to the second model when the conditions to be monitored by the monitoring means have already been set for the specific item. The system according to any one of 1 to 3. When the input item of the second model does not include the specific item, the providing means provides a message prompting the re-learning of the second model with the learning data including the specific item. The system according to any one of claims 1 to 4, wherein the system is characterized by the above. The predetermined condition is a result of prediction by the first model using an input including a first input value and a result of prediction by the first model using an input including a second input value. This is a condition that is satisfied when there is a bias between them.
The system according to any one of claims 1 to 5, characterized in that. The first model using the input including the first input value for the number of times that the prediction result predicted by the first model using the input including the first input value was a predetermined prediction result. The ratio to the total number of forecasts predicted by
The first using the input including the second input value, the number of times the prediction result predicted by the first model using the input including the second input value was the predetermined prediction result. The second ratio is the ratio to the total number predicted by the model.
The system according to claim 6, wherein the predetermined condition is a condition corresponding to the ratio of the first ratio and the second ratio. There is a way to run on a system that monitors multiple models created by machine learning.
When the result of prediction by the first model using inputs including a plurality of values satisfies a predetermined condition, a monitoring step of giving a warning about a specific item of the input corresponding to the predetermined condition, and
A second model different from the first model has a providing step of providing a message prompting the setting of conditions to be monitored in the monitoring step for the specific item.
In the providing step, when the predetermined conditions are set for the first model, when it is determined that the result of prediction by the first model satisfies the predetermined conditions, and when the second model A method characterized in that the message is provided in at least one of the cases where is registered in the system. A program for operating a computer as each means of the system according to any one of claims 1 to 7.

Images