JP2022097034A - Examination support system and examination support method - Google Patents

Abstract

To keep the accuracy of an examination continuously in association with change of rules of the examination.SOLUTION: An examination support system is configured to: examine classification data by selecting a classification model for examining classification data (application data) as an examination target; verify whether the examination is performed properly according to rules for a classification result which is an examination result; estimate a time when the rules are changed on the basis of the classification result in which it is found that the examination is wrong; update division information showing a sampling range (a division space) used when application data used to generate learning data of the classification model on the basis of the estimated time; generate learning data on the basis of application data belonging to the division space; generate a classification model for each cluster obtained by clustering the learning data for each division space; and generate a classification model of the cluster learned by learning data similar to the classification data as the examination target.SELECTED DRAWING: Figure 1

Description

The present invention relates to an examination support system and an examination support method.

There are various examinations in society, such as determining the right to receive a pension in the public sector, determining the disability grade, whether or not to allow a loan in the private sector, and opening an account. However, such examination work generally involves a huge number of cases and places a heavy load on the person in charge of the work. Therefore, there are increasing expectations for labor saving and automation of examination work.

As a method of automating the examination work, for example, there is a method of constructing a rule-based discrimination program. However, it is often difficult to automate the examination work because the rules are not clear, the rules are changed frequently, and the rules are overly complicated. Therefore, a method of learning rules from data has been proposed.

For example, Patent Document 1 describes an examination support device configured for the purpose of appropriately supporting examination work by adopting learning elements having different properties and viewpoints from personal attribute information. .. The examination support device acquires the information of the application matter, inputs the information of the acquired application matter into the attribute information of the applicant and the information of the examination process, and at least the history of learning the deficiency of requirements is used as training data. By applying the judgment criteria that are the result of machine learning to obtain the estimation result of the requirement deficiency and providing the information of the estimation result of the requirement deficiency to the examination work of the application case, the nature and viewpoint of the personal attribute information Adopt different learning elements.

Further, for example, Patent Document 2 describes a creation device for creating a classifier in which classification accuracy is maintained by using unlabeled learning data in consideration of the time evolution of the classification standard. The creation device uses the labeled data collected up to the predetermined time point in the past and the unlabeled data collected after the predetermined time point as training data at each time point. By learning the classification criteria of the classifier in, learning the time series change of the above classification criteria, and using the learned above classification criteria and the time series change, the classification criteria of the classifier at any time point including the future time point. And predict the reliability of the above classification criteria.

Japanese Unexamined Patent Publication No. 2020-004161 Japanese Unexamined Patent Publication No. 2019-900186

In Patent Document 1, the criteria for deficiencies in requirements are learned using the results of past examinations, and the deficiencies in requirements are estimated based on the results. However, the document maintains the accuracy of the examination against discontinuous changes in the examination criteria and items, such as when the examination criteria and items are changed at some point or when new criteria and items are added at some point. No technology is mentioned.

In Patent Document 2, labeled data collected up to a predetermined time point in the past and unlabeled data collected after a predetermined time point in the past are used as learning data. Learn the classification criteria of the classifier at each time point, learn the time-series changes in the classification criteria, and use the learned classification criteria and time-series changes to classify the classifier at any time point, including future time points. Predict the reliability of the classification criteria and the above classification criteria. However, the techniques described in this document are:
It is constructed on the premise that changes in input trends and rules are continuous in chronological order, and when external factors such as changes in laws and regulations or internal rules of an organization cause discontinuous changes in standards and rules in examination. Cannot be dealt with.

The present invention has been made to solve such a problem, and provides an examination support system and an examination support method capable of continuously maintaining the accuracy of examination in accordance with changes in rules in examination. The purpose is to provide.

One of the present inventions for achieving the above object is an examination support system that supports the examination of an application form, which is configured by using an information processing apparatus, and is configured with each item of the application and the value of each item and the examination. The classification data acquisition unit that acquires the classification data that is the application data to be examined from the database that stores the application data that is the data including the time stamp that is the date and time referred to as the reference of the above. A label storage unit that stores a label indicating the result of the examination when the application data is correctly examined according to the rules applied to the examination of the application form in association with the application form, and a machine learning model that examines the classification data. A model storage unit that stores the classification model, and a classifier that selects the classification model to be used for examining the classification data from the model storage unit and examines the classification data using the selected classification model. , Whether or not the classification result storage unit that stores the classification result, which is the information corresponding to the examination result and the classification data, and the classification result are correctly examined by the classifier according to the rules. The result verification unit that acquires the verification result and the division information that is information about the division space indicating the sampling range when the application data used for generating the training data used for learning the classification model is acquired from the database are stored. A divided information storage unit and a divided information updating unit that estimates the time when the rule changes based on the classification result found to be incorrect by the verification and updates the divided information based on the estimated time. , The learning data generation unit that samples the application data from the database for each divided space and generates the training data based on the application data belonging to each of the divided spaces, and clusters the learning data for each divided space. Each cluster is provided with a learning device that generates the classification model, and the classifier uses the classification model of the cluster learned from the training data similar to the classification data to be examined. Select and examine the classification data according to the selected classification model.

In addition, the problems disclosed in the present application and the solutions thereof will be clarified by the column of the form for carrying out the invention and the drawings.

According to the present invention, even if the criteria and rules in the examination change discontinuously, the accuracy of determining the necessity of amendment to the application contents can be maintained.

It is a figure which shows the schematic structure of the examination support system. This is an example of the application form TBL. This is an example of the classification state TBL. This is an example of model TBL. This is an example of the classification result TBL. This is an example of the label TBL. This is an example of classification information TBL. It is a figure which shows an example of the hardware composition of the information processing apparatus which constitutes the examination support system and the like. It is a flowchart explaining the classification process. It is a flowchart explaining a learning process. It is a flowchart explaining the verification process. It is a flowchart explaining the division information update process. This is an example of the result verification screen. This is an example of the individual result verification screen. This is a configuration example of the examination support system.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are appropriately omitted or simplified for the sake of clarification of the description. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following description, the same or similar configurations may be designated by the same reference numerals and duplicated description may be omitted. Further, in the following description, various information may be described by expressions such as "information" and "table", but various information may be expressed by a data structure other than these. Further, as expressions of identification information, there are expressions such as "identification information", "identifier", "name", "ID", and "number", which can be replaced with each other. Further, in the following description, the "database" may be referred to as "DB" and the "table" may be referred to as "TBL".

FIG. 1 shows a schematic configuration (functional block diagram) of an information processing system (hereinafter, referred to as “examination support system 1”) described as an embodiment. For example, in the examination of pension entitlement and disability grade judgment in the public sector, loan availability in the private sector, account opening, etc., the content of the application form prepared by the applicant by the person in charge of examination (for example, application). Whether or not the contents of the person's name, address, gender, age, job, educational background, career, past record, degree of disability, borrowing record, repayment record, etc. meet the prescribed rules (examination criteria and regulations) Judgment is made. The examination support system 1 supports such work that has been performed manually in the past, and aims to reduce the load on the person in charge and the operation cost.

The examination support system 1 has an application form management unit 200, which is a function of managing digitized data (hereinafter referred to as "application data") of the contents of the application form to be examined, and the contents of the application data are predetermined. The examination support device 100 that automatically examines whether or not the rules are met (hereinafter, also referred to as "classification"), and the individual result verification unit that is a function to verify the result of the automatic examination performed by the examination support device 100. 10, including. The examination support device 100 is communicably connected to the application form management unit 200 and the individual result verification unit 10. All of these functions and devices are realized by using an information processing device (computer).

The examination support device 100 generates the above-mentioned automatic examination for each cluster clustered according to the contents of the application data (the composition of the items of the application form and the values set for each item) to be the target of the automatic examination. This is done using a machine learning model (hereinafter referred to as "classification model"). Further, the examination support device 100 verifies the result of the automatic examination performed on the application data (hereinafter referred to as "classification result"), and the verification result (hereinafter referred to as "verification result") is a classification model. By reflecting it in the method of generating learning data used for learning, automatic examination will be performed appropriately. More specifically, when the rule changes and an error in the automatic examination is detected by the verification, the examination support device 100 estimates the time when the rule changes and refers to it when generating training data used for learning the classification model. Information to be processed (hereinafter referred to as "divided information") is updated.

As shown in FIG. 1, the examination support device 100 includes a classification data acquisition unit 101, a classification device 102, a learning data acquisition unit 103, a learning device 104, a result verification unit 105, and a division information update unit 106.
Each function includes a classification state storage unit 300, a model storage unit 400, a classification result storage unit 500, a label storage unit 600, and a division information storage unit 700.

The classification data acquisition unit 101 acquires application data to be subject to automatic examination from the application form TBL201 managed by the application form management unit 200, and classifies the acquired application data (hereinafter referred to as “classification data”). Input to the device 102. When acquiring the application data, the classification data acquisition unit 101 selects the classification data from the application form TBL 201 with reference to the classification state TBL 301 managed by the classification state storage unit 300.

FIG. 2 shows an example of application data TBL201. The example application data TBL201 is composed of a plurality of entries (records) having each item of the application form ID 2011, the time stamp 2012, and the item group 2013. One entry of application data TBL201 corresponds to one application data (application form). Application data The application data managed by TBL201 is manually input, read from paper media by OCR (Optical Character Recognition), and
It is provided to the application form management unit 200 by data transfer or the like from another information processing system.

Among the above items, the application form ID 2011 is set with an identifier of application data (hereinafter, referred to as "application form ID"). In the time stamp 2012, the date and time referred to as a reference in the examination of the application data (the date and time when the application form of the application data is submitted, the date and time when the application data is received from the applicant, etc.) are set. In the item group 2013, values (character strings, numerical values, enum values, etc.) for each item of the application form to be examined are set. The types and numbers of items constituting the item group 2013 vary depending on the type and contents of the application form. Examples of the above items are, for example, the attributes of the applicant (name, date of birth, relationship, gender, various categories, etc.), the type of application target (application for receiving pension, application for disability grade certification, etc.).

FIG. 3 shows an example of the classification state TBL301. Information indicating the classification status of each application data (whether or not a label (described later) is attached, whether or not automatic examination (classification) has been performed by the classifier 102, etc.) is set in the classification status TBL301. The exemplified classification state TBL301 is composed of a plurality of entries (records) having each item of the application form ID 3011 and the classification state 3012. One entry in the classification state TBL301 corresponds to one application data (application form). The application form ID 3011 is set with the application form ID, which is information for specifying the application data. The classification state of the application data is set in the classification state 3012. The classification state managed by the classification state TBL301 is updated at any time by the functions of the examination support device 100 (classification data acquisition unit 101, result verification unit 105, classifier 102, etc.).

Returning to FIG. 1, the classifier 102 automatically examines the classification data input from the classification data acquisition unit 101. The classifier 102 acquires a classification model corresponding to the classification data to be automatically examined from the model TBL401 managed by the model storage unit 400, automatically examines the classification data using the acquired classification model, and classifies the data. Data with a label attached to the data (hereinafter referred to as "classification result") is generated and input to the classification result storage unit 500. The above label is information indicating the result of automatic examination, and in this example, it is information indicating whether or not the application form (classification data (application data)) needs to be modified.

FIG. 4 shows an example of the model TBL401. The exemplary model TBL401 comprises one or more entries (records) having each item of cluster ID 4011, representative point 4012, distance threshold 4013, label 4014, and target period 4015. One entry in model TBL401 corresponds to one cluster. At the time of introduction of the examination support system 1, for example, an initial value is set in the model TBL401 by a human system.

Among the above items, the cluster ID 4011 is a cluster identifier (hereinafter, "cluster I").
Called "D". ) Is set. The representative point 4012 is an information indicating the representative point of the cluster (a group of items representing the characteristics of the cluster, which is compared with the item group of the classification data when determining which cluster the classification data belongs to (hereinafter,). Information indicating (referred to as "representative point")) is set. In the distance threshold value 4013, a threshold value (boundary value) of the similarity (distance) used when determining which cluster the classification data belongs to is set. The above similarity is, for example, the Euclidean distance or the Mahalanobis distance. In this example, the distance threshold value 4013 is set to a threshold value set for the degree of similarity between the item group constituting the classification data and the representative point 4012. Label 4014 is set with a label to be assigned to the classification data when the classification data to be automatically examined corresponds to the cluster. In the target period 4015, a period (start date and time and end date and time) to which the cluster is applied is set. The above period is compared with the time stamp of the classification data (time stamp 2012 of the application data) when determining which cluster the classification data belongs to. In this example, when the start date and time and the end date and time are not determined, a value meaning undefined (hereinafter referred to as "null value") is set in the corresponding column of the target period 4015.

Returning to FIG. 1, the classification result storage unit 500 manages the classification result input from the classifier 102 in the classification result TBL501. Further, the classification result storage unit 500 inputs the classification result managed by the classification result TBL 501 to the result verification unit 105 at any time according to the request.

FIG. 5 shows an example of the classification result TBL501. The exemplary classification result TBL501 is composed of one or more entries (records) having each item of application ID 5011, label 5012, time stamp 5013, and item group 5014. One entry of the classification result TBL501 corresponds to one classification data (application data (application form)).

Of the above items, the items other than the label 5012 are the same as the items having the corresponding names of the application data TBL201 shown in FIG. 2, and thus the description thereof will be omitted. The label 5012 is set with a label assigned to the classification data by the classifier 102. In this example, the label 5012 has "correction required" indicating that the classification data (application data) needs to be corrected, or "correction" indicating that the classification data (application data) does not need to be corrected. "Not required" is set.

Returning to FIG. 1, the result verification unit 105 individually determines whether or not the classification result input from the classification result storage unit 500 is properly examined according to the rules (whether or not the automatic examination is correctly performed according to the rules). Verification is performed in cooperation with the result verification unit 10. The result verification unit 105 inputs the classification result to the individual result verification unit 10 and acquires the verification result from the individual result verification unit 10.

The individual result verification unit 10 provides a user interface for the human system to verify the classification result input from the result verification unit 105, and returns the verification result received from the user to the result verification unit 105. The user verifies the classification result based on, for example, a document describing the rule or his / her own experience. In this example, the classification result is verified by the human system in this way, but the classification result is automatically verified by the information processing device, such as by using a machine learning model for verification. May be good.

When the verification result returned from the individual result verification unit 10 is incorrect, the result verification unit 105 inputs the division information update instruction to the division information update unit 106. The result verification unit 105 divides the division information update instruction when, for example, the number or ratio of the verification results returned from the individual result verification unit 10 exceeds a preset threshold value. It may be input to the information update unit 106.

The result verification unit 105 inputs a label attached to the classification data (application data) to the label storage unit 600 when the classification data (application data) of the classification result is appropriately examined according to the rules. The label storage unit 600 manages the label of each application data (application form) input from the result verification unit 105 on the label TBL601.

FIG. 6 shows an example of the label TBL601. As shown in the figure, the label TBL601 is composed of a plurality of entries (records) having each item of the application form ID 6011 and the label 6012. One entry on the label TBL601 corresponds to one application data (application form).

Among the above items, the application form ID is set in the application form ID 6011. Similar to the label 5012 of FIG. 5, the label 6012 is set with the label attached to the application form (classification data (application data)).

Returning to FIG. 1, when the result verification unit 105 inputs the division information update instruction, the division information update unit 106 performs processing related to updating the division information, and the division information storage unit 700 manages the division information TBL701. Update the information. The details of the process for updating the division information will be described later.

FIG. 7 shows an example of the division information TBL701. The divided information is information indicating a range (hereinafter, referred to as "divided space") in which the learning data acquisition unit 103 extracts the application data that is the generation source of the learning data. The divided space is defined with a temporal range and a spatial range (characteristics (type of item and domain) of each item of classification data (application data)) as parameters.

The illustrated division information TBL701 is composed of one or more entries (records) having each item of the division space ID 7011 and the range 7012. One entry of the division information TBL701 corresponds to one division space. When the examination support system 1 is introduced, for example, an initial value is set in the division information TBL701 by a human system.

An identifier of the divided space (hereinafter, referred to as "divided space ID") is set in the divided space ID 7011. Information indicating the range of the divided space is set in the range 7012. As shown in the figure, the range 7012 has each item of the date and time 70121 and the item group 70122. Of these, information indicating the temporal range of the divided space is set at the date and time 70121. In this example, at the time of introduction of the examination support system 1, the current date and time (date and time at the time of introduction) is set as the start date and time of the above range, and the null value is set as the end date and time of the above range. Information indicating the spatial range of the divided space is set in the item group 70122. Since the meaning of each item of the item group 70122 is the same as that of the item group 2013 of the application data TBL201, the description thereof will be omitted.

Returning to FIG. 1, the learning data acquisition unit 103 refers to the classification state managed by the classification state TBL301 of the classification state storage unit 300 and the division information managed by the division information TBL701 of the division information storage unit 700. , The application data used for generating the learning data of the classification model is acquired from the application form management unit 200, the learning data is generated based on the acquired application data, and the generated learning data is input to the learning device 104.

The learning device 104 clusters the learning data input from the learning data acquisition unit 103 based on the content of the learning data (the configuration of the items of the application form and the values set for each item), and each cluster obtained by the clustering. A classification model is generated by assigning a label acquired from the label storage unit 600 to the model, and the generated classification model is stored in the model storage unit 400. The learner 104 performs the above clustering by, for example, a hierarchical clustering method. The clustering method is not always limited. Further, the number of clusters obtained by clustering is appropriately set (so as to improve the accuracy of automatic examination) according to the nature of the training data and the like.

FIG. 8 is an example of the hardware configuration of the information processing apparatus used to realize the examination support system 1, the individual result verification unit 10, and the application form management unit 200. As shown in the figure, the exemplary information processing device 20 includes a processor 21, a main storage device 22, a communication device 23, an input device 24, an output device 25, and an auxiliary storage device 26.

It should be noted that the information processing apparatus 20 illustrated is virtual information in which all or part of the information processing apparatus 20 is provided by using virtualization technology, process space separation technology, or the like, for example, like a virtual server provided by a cloud system. It may be realized by using processing resources. Further, all or part of the functions provided by the information processing apparatus 20 may be realized by, for example, a service provided by a cloud system via an API (Application Programming Interface) or the like. Further, all or part of the functions provided by the information processing apparatus 20 are realized by using, for example, SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), and the like. It may be one.

The processor 21 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) chip, an FPGA (Field Programmable Gate Array), a SoC (System on Chip), and an ASIC. (Application Specific Integrated Circuit) etc. are used.

The main storage device 22 is a device for storing programs and data, and is, for example, a ROM (Read).
Only Memory), RAM (Random Access Memory), non-volatile memory (NVRAM (Non Volatile RAM)) and the like.

The communication device 23 is a communication device (smartphone, tablet, notebook computer, etc.) via a communication network (LAN (Local Area Network), WAN (Wide Area Network), the Internet, a dedicated line, a public communication network, etc.). , Various mobile information terminals, etc.), and is a wireless or wired communication module (wireless communication module, communication network adapter, USB module, etc.).

The input device 24 is a user interface that receives input from a user or data input from an external device, and is, for example, a keyboard, a mouse, a touch panel, a card reader, a voice input device (for example, a microphone), or the like.

The output device 25 is a display device that outputs various information as an image, an audio output device that outputs as an audio, a printing apparatus that prints on a paper medium, and the like.

The auxiliary storage device 26 is a device for storing programs and data, for example, an SSD (Solid State Drive), a hard disk drive, and an optical storage medium (CD (Compact Disc)).
, DVD (Digital Versatile Disc), etc.), IC card, SD card, etc. All or part of the auxiliary storage device 26 may be a virtual storage area or the like provided by the cloud. The auxiliary storage device 26 stores programs and data for realizing the functions of the examination support system 1.

The program or data can be read into the auxiliary storage device 26 via the reading device of the recording medium or the communication device 23. Programs and data stored (stored) in the auxiliary storage device 26 are read into the main storage device 22 at any time. The processor 21 realizes various functions by reading the program stored in the auxiliary storage device 26 into the main storage device 22 and executing the program.

For example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), or the like may be introduced in the information processing apparatus 20. The classification state storage unit 300, the model storage unit 400, the classification result storage unit 500, the label storage unit 600, and the division information storage unit 700 manage information (data) in, for example, a table managed by the DBMS or a file related to the file system. do.

Subsequently, the processes performed by the examination support system 1 will be described in order.

FIG. 9 shows a process performed when the examination support device 100 automatically examines (classifies) application data (classification data) provided by the application form management unit 200 and generates a classification result (hereinafter, “classification process S900”). It is a flowchart explaining).

First, the classification data acquisition unit 101 acquires application data to be classified (for example, application data without a label) from the application management unit 200 while referring to the classification state TBL301 (S911), and acquires the application data. The application data is input to the classifier 102 as classification data (S912).

Subsequently, the classifier 102 attempts to acquire the classification data input from the classification data acquisition unit 101 from the model storage unit 400 of the cluster used for automatic examination (classification) (S913). In the classifier 102, the time stamp of the classification data is included in the target period 4015, and the similarity (distance) between the item group (representative point) of the classification model and the item group of the application data is the distance threshold 4013 or less. The cluster is acquired from the model storage unit 400. In calculating the distance, the classifier 102 may perform weighting for each item.

Returning to FIG. 9, in S914, the classifier 102 determines whether or not the cluster can be acquired. When the cluster can be acquired (S914: YES), the classifier 102 generates the information in which the acquired cluster label is attached to the classification data as the classification result (S915). On the other hand, when the cluster could not be acquired (S914: NO), the classifier 102 generates information added to the classification data as a label indicating that the cluster could not be acquired as the classification result (S916).

In S917, the classifier 102 inputs the generated classification result to the classification result storage unit 500.

FIG. 10 is a flowchart illustrating a process (hereinafter, referred to as “learning process S1000”) performed when the examination support system 1 generates (learns) a classification model.

First, the learning data acquisition unit 103 selects one division space 7011 from the division information TBL701 (S1011), and acquires and acquires the labeled application data belonging to the range 7012 of the selected division space 7011 from the application data TBL201. Training data is generated by adding a label acquired from the label TBL601 to the application data, and the generated training data is input to the learner 104 (S1012). The learning data acquisition unit 103 determines whether or not the application data has been labeled with reference to the classification state TBL301.

Subsequently, the learning device 104 clusters the learning data input from the learning data acquisition unit 103 (S1013).

Subsequently, the learner 104 selects one cluster obtained by clustering (S1014), and determines whether or not the labels of all the training data belonging to the selected cluster are the same (S1015). When the learner 104 determines that the labels of all the training data belonging to the selected cluster are the same (S1015: YES), the learner 104 generates and generates a classification model by associating the label with the selected cluster. The classified model is registered in the model TBL401 (S1016). At this time, the learner 104 sets, for example, the value used (or calculated) for the clustering of S1013 as the representative point 4012. Further, a preset value (default value) is set as the distance threshold value 4013. Further, the learning device 104 sets the current date and time at the start date and time of the target period 4015, and sets the null value at the end date and time of the target period 4015. In this example, the determination condition is "whether the labels of all the training data are the same" belonging to the selected cluster, but the determination condition is not necessarily limited to the example, and for example, the selected cluster. The determination condition may be "whether or not a predetermined ratio or more of the labels of the learning data are common" belonging to.

Subsequently, the learner 104 determines whether or not all the clusters have been selected in S1014 (S1017). If all the clusters have not been selected in S1014 (S1017: NO), the process returns to S1014. On the other hand, if all the clusters have been selected in S1014 (S1017: YES), the process proceeds to S1018.

In S1018, the learner 104 determines whether or not all the divided spaces have been selected in S1011. If all the divided spaces have not been selected in S1011 (S1018: NO), the process returns to S1011. On the other hand, if all the divided spaces have been selected in S1011 (S1018: YES), the learning process S1000 ends.

FIG. 11 is a flowchart illustrating a process (hereinafter, referred to as “verification process S1100”) performed by the examination support system 1 when verifying the classification result.

First, the result verification unit 105 selects one division space from the division information TBL701 (S1111).

Subsequently, the result verification unit 105 selects one cluster belonging to the range of the divided space being selected from the model TBL401 (S1112).

Subsequently, the result verification unit 105 acquires one classification result from the classification result TBL501, which belongs to the selected cluster and whose time stamp 5013 is between the final verification date and time and the current date and time (S1113). The final verification date and time is the latest date and time when the verification process S1100 is executed, and is information managed (stored) by the result verification unit 105.

Subsequently, the result verification unit 105 verifies the acquired classification result in cooperation with the individual result verification unit 10 (S1114).

Subsequently, the result verification unit 105 determines the verification result (S1115). When the verification result indicates that the classification result is correct (S1115: NO), the result verification unit 105 executes the process from S1118. On the other hand, when the verification result indicates that the classification result is incorrect (S1115: YES), the result verification unit 105 cooperates with the division information update unit 106 to update the division information (hereinafter, “division information update processing”). S1116 ") is executed. After that, the process proceeds to S1117. The details of the division information update process S1116 will be described later.

In S1117, the result verification unit 105 updates the classification result TBL501 and the label TBL601 based on the result of the division information update process S1116. After that, the process proceeds to S1118.

In S1118, the result verification unit 105 determines whether or not all the clusters have been selected in S1112. If all the clusters have not been selected (S1118: NO), the process returns to S1112. On the other hand, if all the clusters have been selected in S1112 (S1118: YES), the process proceeds to S1119.

In S1019, the result verification unit 105 determines whether or not all the divided spaces have been selected in S1111. If all the divided spaces have not been selected (S1119: NO), the process returns to S1111. On the other hand, if all the divided spaces have been selected in S1111 (S1119: YES), the verification process S1100 ends.

FIG. 12 is a flowchart illustrating the details of the division information update process S1116 shown in FIG.

First, the result verification unit 105 sets a predetermined difference between the time stamp of the classification result determined to be an error in the verification of S1114 (hereinafter referred to as “wrong classification result”) and the final verification date and time. It is determined whether or not it is larger than (S1211). When the result verification unit 105 determines that the difference is larger than the predetermined value (S1211: YES), the process proceeds to S1212. On the other hand, when the result verification unit 105 determines that the difference is equal to or less than a predetermined value (S1211: NO), the process proceeds to S1217.

The processing of S1212 to S1216 is a loop processing that is repeatedly executed until the above difference becomes equal to or less than a predetermined value (until the time difference between the time stamp and the final verification date and time becomes sufficiently small). First, in S1212, the result verification unit 105 determines the unverified classification result that is between the time stamp of the erroneous classification result and the final verification date and time and has the maximum similarity between the erroneous classification result. It is newly obtained from the classification result TBL501. Hereinafter, the newly acquired classification result will be referred to as "similar classification result".

Subsequently, the result verification unit 105 verifies the similar classification result in cooperation with the individual result verification unit 10 (S1213).

Subsequently, the result verification unit 105 determines the verification result (S1214). When the verification result indicates that the classification result is correct (S1214: NO), the result verification unit 105 updates the final verification date and time with the time stamp of the similar classification result (S1215). After that, the process returns to S1211. On the other hand, when it indicates that the verification result is erroneous (S1214: YES), the result verification unit 105 resets the erroneous classification result to the similar classification result (the "wrong classification result" to be processed is set. Substitute with similar classification results) (S1216). After that, the process returns to S1211. When the verification result is erroneous in this way, by resetting the erroneous classification result to the similar classification result and repeatedly executing the loop processing from S1211, the time when the verification result becomes erroneous, that is, the rule It is possible to estimate the time of change. The above loop processing corresponds to searching for a time when the rule changes by a temporal binary search in the past direction).

In S1217, the result verification unit 105 sets the end date and time of the target period 4015 of the selected cluster as an intermediate date and time between the final verification date and time and the date and time of the erroneous classification result. Although the intermediate date and time is set here, it does not necessarily have to be the intermediate date and time in the strict sense, and the date and time set as the end date and time of the target period 4015 of the selected cluster is the final verification date and time. It may be set so that the error rate of the classification result is likely to be low based on the date and time of the erroneous classification result.

Subsequently, the result verification unit 105 acquires unverified classification results whose time stamp is later than the date and time set as the end date and time of the target period 4015 in S1217, and the individual result verification unit for the acquired classification results. Verification is performed in cooperation with 10. (S1218). In this way, by verifying the unverified classification results of the date and time after the end date and time (that is, by verifying the classification results after the date and time estimated to be the date and time when the rule changed, temporally or spatially. It is possible to efficiently identify the time when the rule changes with high accuracy.

Subsequently, the result verification unit 105 sets (generates or updates) the divided space based on the verification result of S1218, and registers the set divided space in the division information TBL701 (S1219). For example, the result verification unit 105 sets a division space in which the date and time when the rule changes is set as the start date and time of the date and time 70121. Further, for example, the result verification unit 105 sets a division space whose end date and time is the date and time when the rule changes.

As described above, the examination support system 1 selects an appropriate classification model for the examination of the classification data, examines the classification data using the selected classification model, and classifies the classification results. Verify whether the instrumental examination is performed correctly according to the rules. If the verification reveals that the examination is incorrect, the time when the rules applied to the application examination change is estimated based on the incorrect classification result, and the division information (division space) is updated based on the estimated time. Then, the training data is clustered for each updated divided space, and a classification model is generated for each cluster. Therefore, for example, even if the rule changes discontinuously, the time of change can be estimated efficiently and with high accuracy, and the classification model can be retrained so that the examination is performed correctly.

Further, since the examination support system 1 can detect the change in the rule regardless of the mode of the change in the rule in the examination in this way, it is possible to continuously maintain the accuracy of the examination in accordance with the change in the rule. can.

In addition, the classification model is generated for each cluster clustered based on the similarity of the training data, and the classification data is automatically examined (classified) by the classification model trained by the training data similar to the classification data. , It is possible to perform automatic examination with high accuracy while considering the spatial difference of application data (classification data).

By the way, the examination support system 1 provides, for example, the following user interface in order to improve the work efficiency of the user who performs the examination.

FIG. 13 is an example of a screen (hereinafter referred to as “verification result screen 1300”) displayed by the examination support device 100 when presenting the verification result in S1114 of FIG. 11 to a user such as a person in charge of examination. The example verification result screen 1300 shows the date and time when an error in the automatic examination was detected (detection date and time 1312) and the date and time when the latest verification was performed for the cluster (final verification date and time 1313) for each cluster specified by the cluster ID 1311. Is displayed. By referring to the verification result screen 1300, the user can visually confirm what kind of automatic examination (cluster) has an error. Further, for the cluster in which the error of the automatic examination is detected, the operation button 1314 for instructing whether or not to update the division information is displayed. By operating the operation button 1314, the user can issue an instruction to update the division information (execution instruction of the division information update process S1116) for the cluster in which the error of the automatic examination is detected.

FIG. 14 is an example of a screen displayed by the individual result verification unit 10 (hereinafter, referred to as “individual result verification screen 1400”) in the verification in S1218 of FIG. The individual result verification screen 1400 exemplified is the application form ID 1411 of the classification data of the classification result to be verified, the time stamp 1412 of the classification data, the image 1413 of the application form which is the source of the classification data, and the classification. The name of each item of the data (item name 1414) and the value 1415 set for each item are displayed. The user can specify a label to be set for the classification data by operating the pull-down list 1416. When the user operates the registration button 1417, the label specified in the pull-down list 1416 is reflected in the label TBL601 in the classification data.

Further, the functions of the application form management unit 200 and the individual result verification unit 10 may be implemented in the same information processing device or in different information processing devices.

FIG. 15 is a configuration example of the examination support system 1 when the functions of the application form management unit 200 and the individual result verification unit 10 are implemented in the same information processing device (hereinafter referred to as “user terminal 220”). As shown in the figure, the illustrated user terminal 220 is examined via a communication network 5 (LAN (Local Area Network), WAN (Wide Area Network), Internet, leased line, public communication network, serial communication device, etc.). It is communicably connected to the support device 100. The application form management unit 200 provides the application data of the application data TBL201 to the examination support device 100 via the communication network 5. The examination support device 100 acquires the application data of the application data TBL201 and the verification result managed by the individual result verification unit 10 by using a protocol such as REST or gRPC, for example. The user terminal 220 presents the screen provided by the examination support device 100 or the screen generated by the user terminal 220 to the user in the form of, for example, a Web page.

Although one embodiment of the present invention has been described above, it is needless to say that the present invention is not limited to the above embodiment and can be variously modified without departing from the gist thereof. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add, delete, or replace a part of the configuration of the above embodiment with another configuration.

Each of the above configurations, functional units, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them with an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

The arrangement form of various functional units, various processing units, and various databases of each information processing device described above is only an example. The arrangement form of various function units, various processing units, and various databases can be changed to the optimum arrangement form from the viewpoint of the performance, processing efficiency, communication efficiency, and the like of the hardware and software included in these devices.

The configuration of the database (schema, etc.) for storing various types of data described above can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

1 Examination support system, 10 Individual result verification unit, 101 Classification data acquisition unit, 102 classifier, 103 Learning data acquisition unit, 104 Learner, 105 Result verification unit, 106 Division information update unit, 200 Application form management department, 201 Application data TBL, 220 user terminal, 300 classification status storage unit, 301 classification status TBL, 400 model storage unit, 401
Model TBL, 500 Classification result storage unit, 501 Classification result TBL, 600 Label storage unit, 601 Label data TBL, 700 Division information storage unit, 701 Division information TBL, 1300 Verification result screen, 1400 Individual result verification screen

Claims (14)

It is an examination support system that supports the examination of applications.
It is configured using an information processing device and
Classification of the application data to be examined from the database that stores the application data, which is the data including each item of the application, the value of each item, and the time stamp which is the date and time referred to as the examination standard. Data acquisition unit for classification to acquire data for
A label storage unit that stores a label indicating the result of the examination when the application data is correctly examined according to the rules applied to the examination of the application form in association with the application form.
A model storage unit that stores a classification model, which is a machine learning model that examines the classification data,
A classifier that selects the classification model to be used for examining the classification data from the model storage unit and examines the classification data using the selected classification model.
A classification result storage unit that stores classification results, which is information in which the examination results and the classification data are associated with each other.
With respect to the classification result, a result verification unit that acquires the result of verification as to whether or not the examination by the classifier is performed correctly according to the rules, and
A division information storage unit that stores division information that is information about a division space indicating a sampling range when the application data used for generating training data used for learning the classification model is acquired from the database.
A division information update unit that estimates the time when the rule changes based on the classification result found to be incorrect by the verification and updates the division information based on the estimated time.
A learning data generation unit that samples the application data for each divided space from the database and generates the learning data based on the application data belonging to each of the divided spaces.
A learning device that generates the classification model for each cluster obtained by clustering the learning data for each divided space.
Equipped with
The classifier selects the classification model of the cluster trained by the training data similar to the classification data to be examined, and examines the classification data by the selected classification model.
Examination support system. The examination support system according to claim 1.
The division information update unit selects another classification result similar to the erroneous classification result retroactively from the time stamp of the erroneous classification result to the final verification date and time, which is the date and time when the previous verification was performed. The time point is estimated by repeatedly executing the process of verifying the selected other classification results until the verification results are correct.
Examination support system. The examination support system according to claim 1.
The result verification unit performs the verification on the classification result between the final verification date and time, which is the date and time when the time stamp was last verified, and the current date and time.
Examination support system. The examination support system according to claim 1.
The result verification unit accepts the verification result for the unverified classification result of the time stamp after the estimated time stamp, and estimates the time point based on the received verification result.
Examination support system. The examination support system according to claim 1.
The learner sets, in the classification model, the target period of the classification model and information for determining the degree of similarity between the classification data and the learning data used for generating the classification model. death,
The classifier uses the classification model for which the time stamp of the classification data is included in the target period and the classification model is determined to be similar to the training data based on the information. Select from the model storage unit as the classification model used for data review.
Examination support system. The examination support system according to claim 1.
An examination support system further comprising a user interface for presenting information indicating the result of the verification to the user for each cluster. The examination support system according to claim 6.
The user interface receives an instruction to update the division information for each cluster, and receives an instruction to update the division information.
The division information update unit updates the division information related to the cluster that has received the update instruction.
Examination support system. The examination support system according to claim 1.
An examination support system further comprising a user interface for presenting the contents of the application data corresponding to the classification data of the classification result found to have been erroneously examined by the verification. The examination support system according to claim 8.
The user interface accepts the correct label for the classification data of the classification result, which was found to have been erroneously examined by the verification.
The label storage unit stores the received label in association with the application form corresponding to the classification data.
Examination support system. It ’s a way to support the examination of the application form.
Information processing equipment
Classification of the application data to be examined from the database that stores the application data, which is the data including each item of the application, the value of each item, and the time stamp which is the date and time referred to as the examination standard. Steps to get the data for
A step of storing a label indicating the result of the examination when the application data is correctly examined according to the rules applied to the examination of the application form in association with the application form.
A step of storing a classification model, which is a machine learning model for examining the classification data,
A step of selecting the classification model to be used for examining the classification data and examining the classification data using the selected classification model.
A step of storing the classification result, which is information corresponding to the result of the examination and the data for classification,
With respect to the classification result, a step of acquiring the result of verification as to whether or not the examination is performed correctly according to the rules, and
A step of storing division information which is information about a division space indicating a sampling range when the application data used for generating training data used for learning the classification model is acquired from the database.
A step of estimating the time when the rule changes based on the classification result found to be incorrect by the verification, and updating the division information based on the estimated time.
A step of sampling the application data from the database for each divided space and generating the learning data based on the application data belonging to each of the divided spaces.
A step of generating the classification model for each cluster obtained by clustering the learning data for each divided space, and
A step of selecting the classification model of the cluster learned by the training data similar to the classification data to be examined, and examining the classification data by the selected classification model.
How to support the examination. The examination support method according to claim 10.
The information processing apparatus selects and selects another classification result similar to the erroneous classification result retroactively from the time stamp of the erroneous classification result to the final verification date and time, which is the date and time when the previous verification was performed. A step of estimating the time point by repeatedly executing the process of verifying the other classification results until the verification result becomes correct.
Examination support system that further executes. The examination support method according to claim 10.
An examination support method in which the information processing apparatus further executes the step of verifying the classification result between the final verification date and time, which is the date and time when the time stamp was previously verified, and the current date and time. The examination support method according to claim 10.
A step in which the information processing apparatus receives a verification result for the unverified classification result of a time stamp after the estimated time stamp, and estimates the time point based on the received verification result.
Examination support method to further execute. The examination support method according to claim 10.
The information processing apparatus provides the classification model with information for determining the target period of the classification model and the degree of similarity between the classification data and the learning data used to generate the classification model. Steps to set and
The classification model determined that the time stamp of the classification data is included in the target period and that the classification data and the learning data are similar based on the information is used for the examination of the classification data. Steps to select as a classification model,
Examination support method to further execute.

Images