JP2021018798A - Information processing method and information processing system - Google Patents

Abstract

To provide an information processing method capable of determining a limit of training of a machine learning model.SOLUTION: An information processing method causing a computer to execute, comprises: inputting acquired evaluation data into a first model to acquire a first inference result; determining abnormality of the first inference result based on the first inference result and reference information of the evaluation data; acquiring a second model by training of machine learning based on the determination result; inputting the evaluation data into the second model to acquire a second inference result; determining abnormality of the second inference result based on the second inference result and the reference information of the evaluation data; acquiring a third model by training of the machine learning based on the determination result; inputting the evaluation data into the third model to acquire a third inference result; determining abnormality of the third inference result based on the third inference result and the reference information of the evaluation data; and, when it is identified that the abnormalities of the third inference result and the first inference result are same, outputting information relating to the training limit of the first model.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an information processing method to be executed by a computer and an information processing system to execute the information processing method.

The object detection technology using deep learning (see, for example, Non-Patent Document 1), for which realization examples with high accuracy have been reported, is expected to be put into practical use in various applications.

As a countermeasure against detection abnormalities (including false detection and undetection) in the object detection technology using machine learning such as deep learning, training data to which the data of the detection processing target in which the detection abnormality has occurred is added is used. Model training is performed by machine learning (see Patent Document 1).

JP-A-2002-342739

Qijie Zhao, 6 outsiders, "M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network", November 2018, [online], arXiv, [Search February 26, 2020], Internet <URL : Https://arxiv.org/abs/1811.04533>

However, the detection abnormality may not be resolved even by training using the added training data. It is difficult to judge whether or not the detection abnormality is resolved, that is, whether or not it is the training limit.

The present disclosure provides an information processing method and the like capable of determining the training limit of machine learning.

The information processing method according to one aspect of the present disclosure is a method executed by a computer, and the evaluation data is acquired, the evaluation data is input to the first model for executing the inference processing, and the first inference result is acquired. Then, based on the first inference result and the reference information of the evaluation data, the first determination for determining the error or omission of the first inference result with respect to the evaluation data is executed, and the first inference result is in error or omission. The second model was acquired by the first training of machine learning in which one or both of the first evaluation data and the data similar to the first evaluation data were included in the training data, and the second model was described as described above. The second determination is made by inputting the evaluation data to acquire the second inference result and determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and the reference information of the evaluation data. One or both of the second evaluation data in which the error or omission in the evaluation data that was not in the first inference result was in the second inference result and the data similar to the second evaluation data. The third model is acquired by the second training of machine learning performed by including the above in the training data, the evaluation data is input to the third model to acquire the third inference result, and the third inference result and the evaluation data are obtained. Based on the reference information of the above, the third determination for determining the error or omission of the third inference result with respect to the evaluation data was executed, and the error or omission in the third inference result and the first inference result were found. It is an information processing method that outputs information regarding the training limit of the first model when it is identified that the error or omission is the same.

Further, the information processing system according to one aspect of the present disclosure includes an inference processing unit, an abnormality determination unit, a model training unit, and a notification control unit, and the inference processing unit acquires evaluation data and performs inference processing. The evaluation data is input to the first model for executing the above, and the first inference result is acquired. The abnormality determination unit of the first inference result is based on the first inference result and the reference information of the evaluation data. The first determination for determining an error or omission in the evaluation data is executed, and the model training unit executes the first evaluation data in which the first inference result has an error or omission, and data similar to the first evaluation data. The second model is acquired by the first training of machine learning in which one or both of the above are included in the training data, and the inference processing unit further inputs the evaluation data into the second model and obtains the second inference result. Upon acquisition, the abnormality determination unit further executes a second determination for determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and the reference information of the evaluation data. The model training unit further includes, among the evaluation data, the second evaluation data in which the error or omission that was not in the first inference result was in the second inference result, and the data similar to the second evaluation data. The third model is acquired by the second training of machine learning in which one or both are included in the training data, and the inference processing unit further inputs the evaluation data into the third model to acquire the third inference result. The abnormality determination unit further executes a third determination to determine an error or omission of the third inference result with respect to the evaluation data based on the third inference result and the reference information of the evaluation data, and performs the notification control. When it is identified that the error or omission in the third inference result and the error or omission in the first inference result are the same, the unit outputs information regarding the training limit of the first model.

In addition to the above methods and systems, these comprehensive or specific embodiments may be realized by a device, an integrated circuit, or a computer-readable recording medium such as a CD-ROM. It may be realized by any combination of integrated circuits, methods, computer programs and recording media.

The training limit of machine learning can be determined by the information processing method or the like according to the present disclosure.

FIG. 1 is a block diagram showing a functional configuration example of an information processing system that executes the information processing method according to the embodiment. FIG. 2 is a flow chart showing an example of an abnormality determination procedure in the information processing method according to the embodiment. FIG. 3 is a flow chart showing a procedure example of the information processing method according to the embodiment.

(Knowledge on which this disclosure was based)
The inventors have found that the following problems occur in the prior art.

For example, when applying object detection technology to applications that require high reliability such as automatic driving, it is indispensable to take measures against detection abnormalities such as false detection and non-detection. This is because such a detection abnormality causes an accident that may affect human life.

In object detection technology that uses machine learning methods such as deep learning, as a countermeasure against detection abnormalities, training of machine learning models is performed using training data that includes data of detection processing targets in which detection abnormalities have occurred. It is common. However, even if the detection anomaly that occurred is resolved in the model obtained as a result of this training, does this model cause other detection anomalies, that is, the accuracy of object detection by the model by such training? At present, the judgment as to whether or not the overall improvement is the limit depends largely on the rules of thumb or intuition of the person in charge of model training. For example, in order to obtain a more accurate object detection model, an experienced person may consider changing the network configuration, but in reality, an inexperienced person may further train by adding training data. It can happen that you choose to go and see the results in the model you get. As described above, there is a problem that the judgment about the limit of machine learning training varies depending on the skill level of the person in charge of the work.

The information processing method according to one aspect of the present disclosure, which was devised in view of such a problem, is a method executed by a computer, and the evaluation is performed on a first model that acquires evaluation data and executes inference processing. Data is input to acquire the first inference result, and based on the first inference result and the reference information of the evaluation data, the first determination for determining an error or omission of the first inference result with respect to the evaluation data is executed. Then, by the first training of machine learning performed by including one or both of the first evaluation data in which the first inference result is erroneous or omission and the data similar to the first evaluation data in the training data, the second The model is acquired, the evaluation data is input to the second model, the second inference result is acquired, and the evaluation data of the second inference result is based on the second inference result and the reference information of the evaluation data. A second determination for determining an error or omission is executed, and among the evaluation data, an error or omission that was not in the first inference result is found in the second inference result, and the second evaluation data and the second inference result. The third model is acquired by the second training of machine learning in which one or both of the data similar to the evaluation data is included in the training data, and the evaluation data is input to the third model to acquire the third inference result. Then, based on the third inference result and the reference information of the evaluation data, the third determination for determining the error or omission of the third inference result with respect to the evaluation data is executed, and the error in the third inference result is found. Alternatively, when it is identified that the leak and the error or leak in the first inference result are the same, it is an information processing method that outputs information regarding the training limit of the first model.

As a result, the limit of improvement in the performance of the inference model by training of machine learning can be judged regardless of the skill level of the person in charge of creating the inference model.

Further, the evaluation data may be image data, the inference process may be an object detection process, and the first inference result, the second inference result, and the third inference result may be object detection results, respectively.

That is, the limit of improvement in the performance of the inference model that performs the object detection process on the image data can be determined regardless of the technical level of the person in charge of creation.

Further, the object detection result includes a first detection frame which is a detection frame of an object as a result of the object detection process, and the reference information includes a second detection frame which is a detection frame of an object as a reference. The error or the omission may be determined based on the first detection frame and the second detection frame.

The abnormality of object detection can be determined in this way.

Further, the output of the information may be executed to notify the possibility that the training limit has been reached, and the notification may be executed via an image or a voice.

As a result, the creator of the inference model is given materials for studying methods other than additional training based on data that were not inferred normally in order to improve the performance of the inference model, which contributes to the efficiency of work.

Further, the information processing system according to one aspect of the present disclosure includes an inference processing unit, an abnormality determination unit, a model training unit, and a notification control unit, and the inference processing unit acquires evaluation data and performs inference processing. The evaluation data is input to the first model for executing the above, and the first inference result is acquired. The abnormality determination unit of the first inference result is based on the first inference result and the reference information of the evaluation data. The first determination for determining an error or omission in the evaluation data is executed, and the model training unit executes the first evaluation data in which the first inference result has an error or omission, and data similar to the first evaluation data. The second model is acquired by the first training of machine learning in which one or both of the above are included in the training data, and the inference processing unit further inputs the evaluation data into the second model and obtains the second inference result. Upon acquisition, the abnormality determination unit further executes a second determination for determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and the reference information of the evaluation data. The model training unit further includes, among the evaluation data, the second evaluation data in which the error or omission that was not in the first inference result was in the second inference result, and the data similar to the second evaluation data. The third model is acquired by the second training of machine learning in which one or both are included in the training data, and the inference processing unit further inputs the evaluation data into the third model to acquire the third inference result. The abnormality determination unit further executes a third determination to determine an error or omission of the third inference result with respect to the evaluation data based on the third inference result and the reference information of the evaluation data, and performs the notification control. When it is identified that the error or omission in the third inference result and the error or omission in the first inference result are the same, the unit outputs information regarding the training limit of the first model.

As a result, the limit of improvement in the performance of the inference model by the training of machine learning can be judged regardless of the skill level of the creator of the inference model.

In addition to the above methods and systems, these comprehensive or specific aspects may be realized by a device, an integrated circuit, or a computer-readable recording medium such as a CD-ROM, and the device, system. , Integrated circuits, methods, computer programs and any combination of recording media.

Hereinafter, an information processing method and an embodiment of an information processing system according to one aspect of the present disclosure will be described with reference to the drawings. The embodiments shown here show a specific example of the present disclosure. Therefore, the numerical values, shapes, components, arrangement and connection forms of the components, steps (processes), order of steps, and the like shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the components in the following embodiments, the components not described in the independent claims are components that can be arbitrarily added. Further, each figure is a schematic view and is not necessarily exactly shown.

(Embodiment)
The information processing method according to the embodiment is to be executed by a computer, and even if further training is executed based on the abnormality (leakage or error) of the processing result by the inference model obtained by the machine learning training, the performance is performed. It is a method for determining that the possibility of improvement is low, that is, a substantial limit (hereinafter, also referred to as a training limit). In the following, this information processing method will be described as an example of a case used for determining the training limit of an inference model that detects an object on an image.

[1. Constitution]
FIG. 1 is a block diagram showing a functional configuration example of an information processing system that executes the information processing method according to the present embodiment.

In the information processing system 100 according to the present embodiment, machine learning training is performed to generate or update an inference model, and the generated or updated inference model is evaluated. Then, based on the result of this evaluation, the information processing system 100 determines whether or not it is the limit of performance improvement by training.

The information processing system 100 includes a server that is composed of one or more computers, each having a processor and a memory, and executes this information processing method. In the example shown in FIG. 1, this server includes a detection processing unit 10, an abnormality determination unit 20, a false detection identification unit 30, an undetected identification unit 40, a notification control unit 50, and a model training unit 60 as functional components.

The detection processing unit 10 causes an object detection model (hereinafter, also simply referred to as a trained model) that has already been trained by machine learning to execute object detection processing for performance evaluation. The detection processing unit 10 that has acquired the evaluation data (image data in this example) used for the performance evaluation of the trained model performs preprocessing on the image used for the performance evaluation. The pre-processing referred to here is a process performed in preparation for object detection, for example, adjustment of image resolution or normalization of pixel values. When the detection processing unit 10 acquires the trained model to be evaluated, it inputs each preprocessed image into the trained model, and the result of the object detection processing which is the inference processing by the object detection model (hereinafter, the object). Output the detection result). Specifically, the object detection result is a detection frame that surrounds an object such as a pedestrian or an obstacle detected by the object detection model in the image. The acquired detection frame information is output from the detection processing unit 10 to the abnormality determination unit 20. This information includes, for example, information that identifies an image that has undergone object detection processing, a range (position and size) of a detection frame in the image, and an accuracy (also referred to as likelihood). The detection processing unit 10 is an example of the inference processing unit in the present embodiment, and the object detection result acquired by the detection processing unit 10 from the object detection result is the first inference result, the second inference result and the second inference result in the present embodiment. This is an example of the third inference result.

The abnormality determination unit 20 acquires the information of the detection frame (hereinafter, also referred to as the correct answer frame) indicating the correct answer of the object detection in this image as a reference, in addition to the information of the detection frame received from the detection processing unit 10. Then, the abnormality determination unit 20 determines the abnormality of the object detection result by the trained model based on the detection frame received from the detection processing unit 10 and the detection frame indicating the correct answer. The anomalies in this example include false detection (error) and undetected (leakage). FIG. 2 is a flow chart showing a procedure example of a determination method for these abnormalities in the object detection result performed by the abnormality determination unit 20.

The abnormality determination unit 20 makes an abnormality determination in descending order of the accuracy of the detection frame acquired for one image indicated by the information input from the detection processing unit 10. First, the abnormality determination unit 20 identifies the detection frame having the highest accuracy among the detection frames for which the abnormality determination has not been performed (step S01).

Next, the abnormality determination unit 20 calculates the IOU (Intersection over Union) between the detection frame specified in step S01 and the correct answer frame of the image (step S02).

Next, the abnormality determination unit 20 determines whether or not the IoU with the correct answer frame calculated in step S02 of each detection frame exceeds 0.5 (step S03). When the IoU with the correct answer frame is a detection frame exceeding 0.5 (Yes in step S03), the abnormality determination unit 20 determines that this detection frame is due to the correct detection, and the correct answer frame with the IoU exceeding 0.5. Pair with (step S04). When the IoU with the correct answer frame is a detection frame that does not exceed 0.5 (No in step S03), the abnormality determination unit 20 determines that this detection frame is due to erroneous detection (step S05).

Next, the abnormality determination unit 20 confirms whether or not the detection frame acquired for this image includes a detection frame for which the series of procedures up to S05 has not been performed (step S06).

If there is a detection frame for which this procedure has not been performed (Yes in step S06), the abnormality determination unit 20 returns to step S01.

When the procedure up to step S05 has been performed for all the detection frames acquired for this image (No in step S06), the abnormality determination unit 20 uses the procedure up to this point in the correct answer frame for this image. It is confirmed in each detection frame whether or not there is a correct answer frame in which the pairing by pairing in step S04 is not established (step S07).

If there is a correct answer frame in which the pair is not established (Yes in step S07), the abnormality determination unit 20 determines that this correct answer frame is due to undetection (step S08), and the series of procedures ends.

When there is no correct answer frame for which no pair is established, that is, when all the correct answer frames are paired with any detection frame (No in step S07), a series for determining an abnormality in the object detection process for this image. The procedure of is completed.

The result of the determination by the abnormality determination unit 20, that is, the information regarding the correctness of each detection frame (hereinafter, also referred to as false detection information) is output to the false detection identification unit 30, and the information regarding undetection (hereinafter, also referred to as undetected information) is output. It is output to the undetected identification unit 40. False detection information and undetected information will be described later.

For the sake of simplicity, the above procedure example is described on the assumption that the detection frame is acquired for only one type of object detection target. When there are a plurality of types of objects to be detected, step S02 may be executed between the detection frame and the correct answer frame for the same type of objects. In this case, for example, information regarding the type of the detected object, which is further included in the information of the detection frame and the information of the correct answer frame acquired by the abnormality determination unit 20, is used.

Further, the reference value 0.5 of IoU in step S03 is an example, and will be used as an example in the following description. This reference value is set so that, for example, only one of the IoUs calculated between one detection frame and each of the plurality of correct answer frames exceeds. However, if a plurality of IoUs still exceed the reference value, for example, the combination of the detection frame having the highest IoU and the correct answer frame may be paired in step S04. Alternatively, based on the idea that the detection result is ambiguous, the detection frame may be determined to be due to erroneous detection.

The detection frame indicated by the information in which the abnormality determination unit 20 receives the input from the detection processing unit 10 is an example of the first detection frame in the present embodiment. Further, the information of the correct answer frame acquired by the abnormality determination unit 20 is an example of the reference information in the present embodiment, and the correct answer frame is an example of the second detection frame. Further, the determination performed by the abnormality determination unit 20 according to the procedure illustrated in FIG. 2 is an example of the first determination, the second determination, and the third determination in the present embodiment.

The erroneous detection identification unit 30 acquires erroneous detection information from the abnormality determination unit 20 and accumulates the erroneous detection information. The false detection information is the details of the false detection that has occurred, such as the image in which the object detection is executed, the correctness of the detection result, and the detection frame of the object detection result determined to be false detection (hereinafter, also referred to as false detection frame). Contains information indicating. The erroneous detection identification unit 30 uses the accumulated erroneous detection information as to whether or not the erroneous detection whose details are shown in the erroneous detection information newly acquired from the abnormality determination unit 20 is a reoccurrence of the erroneous detection already described. To judge. As a specific example, the false detection identification unit 30 has a false detection frame indicated by newly acquired false detection information and a false detection frame due to false detection generated in the object detection process for the same image by an older object detection model whose performance has been evaluated. And IoU is calculated. When the IoU exceeds a predetermined reference value (for example, 0.5), the false detection identification unit 30 indicates that the false detection in which the details are shown in the newly acquired false detection information is a false detection that occurred in an older object detection model. It is determined that the false detection is the same as the detection, and it is determined that the false detection has already occurred. Further, when the IoU is equal to or less than a predetermined threshold value, the false positive identification unit 30 states that the false positive whose details are shown in the newly acquired false positive information is not the existing false positive (a new false positive). judge. Then, the information regarding the result of this determination is output from the erroneous detection identification unit 30 to the notification control unit 50. The information regarding the determination result includes, for example, information for identifying an image in which an erroneous detection has occurred, in addition to the determination result. The information regarding the result of this determination is passed to the model training unit 60, which will be described later, and is used for acquiring training data to be used in the next machine learning training to be executed.

The undetected identification unit 40 acquires information on the undetected from the abnormality determination unit 20, and also accumulates the undetected information. The undetected information includes information indicating the details of the undetected occurrence, such as an image in which object detection has been executed and a correct answer frame determined to be undetected (hereinafter, also referred to as an undetected frame). The undetected identification unit 40 has accumulated undetected information as to whether or not the undetected information whose details are shown in the undetected information newly acquired from the abnormality determination unit 20 is an existing undetected reoccurrence. Judgment is made using. As a specific example, in the undetected identification unit 40, the undetected frame indicated by the newly acquired undetected information is the undetected frame due to the undetected generated in the object detection process for the same image by the older object detection model whose performance has been evaluated. Identify whether it is the same as. When these two undetected frames are the same, the undetected identification unit 40 determines that the undetected details shown in the newly acquired undetected information is the reoccurrence of the previously undetected. Further, when these two undetected frames are not the same, the undetected identification unit 40 indicates that the undetected information whose details are shown in the newly acquired undetected information is not already undetected (new undetected). Is determined. Then, the information regarding the result of this determination is output from the undetected identification unit 40 to the notification control unit 50. The information regarding the determination result includes, for example, information for identifying an image in which a new undetected image has occurred, in addition to the determination result. The information regarding the result of this determination is passed to the model training unit 60, which will be described later, and is used for acquiring training data to be used in the next machine learning training to be executed.

The notification control unit 50 determines whether or not a conflict has occurred based on the information regarding the result of the determination of receiving the input from the false detection identification unit 30 and the undetected identification unit 40, and the model training unit issues a notification according to the result of this determination. Send to 60. The contradictory here means that a false positive or undetected once resolved by machine learning training reoccurs in a trained model acquired after training to resolve another false positive or undetected. As a specific example of the notification, when it is determined that a conflict has occurred, the notification control unit 50 creates a user of the information processing system 100, that is, an object detection model, as information regarding the training limit that the training limit is reached. Notify the person in charge. As information on the training limit, the detection result (detection frame) related to the contradictory determined to have occurred may be output. This notification is made, for example, via an image displayed on a display device (not shown) connected to the computer constituting the server of FIG. 1. Further, when it is determined that no conflict has occurred, the notification control unit 50 may notify the user of the normal end of the training. These notifications may be made via voice output from a speaker (not shown) connected to the computer constituting the server of FIG. 1. When it is determined that no conflict has occurred, the notification control unit 50 models the information regarding the new false detection or the information regarding the new undetection received from the false detection identification unit 30 and the undetected identification unit 40. Output to the training unit 60.

The model training unit 60 executes machine learning training to acquire an object detection model to be evaluated for performance. When the model training unit 60 receives input of information regarding a new false detection from the notification control unit 50, the model training unit 60 acquires training data used for training for eliminating the false detection from the training data D20. The training data used for the training for eliminating the false detection is, for example, the data of the image in which the new false detection occurs, which is specified by the information regarding the new false detection. When the model training unit 60 receives input of new information regarding undetection from the notification control unit 50, the model training unit 60 also acquires training data used for training for resolving the undetection from the training data D20. The training data used for the training for eliminating the undetected is, for example, the data of the image in which the new undetected occurs, which is specified by the information about the new undetected. That is, in addition to the training data D20, the data derived from the evaluation data D10 is used for machine learning training. Further, as the training data used for the training for eliminating the false detection, data of an image similar to the image in which a new false detection has occurred may be used. Similarly, as the training data used for the training for eliminating the undetection, the data of the image similar to the image in which the new false detection has occurred may be used. A similar image is, for example, an image having the same or similar composition as the image and taken by changing the shooting settings such as exposure or shutter speed. Another example of a similar image is an image in which noise is added or reduced by performing image processing on the image, and an image in which brightness, contrast, saturation, white balance, etc. are changed.

In order to execute machine learning training, the model training unit 60 combines the acquired training data with the training data used for training to obtain a trained model in which false detection or undetection has occurred. Prepare. Then, the model training unit 60 uses this data set to perform machine learning training for an object detection model having a predetermined configuration. When the training is completed, the model training unit 60 outputs a new trained model.

These functional components are the computers that make up the server, where the processor executes programs held in memory, stores data in memory, or reads data stored in memory as needed. It will be realized.

In the example of FIG. 1, in the information processing system 100, the evaluation data D10 and the training data D20 are input to the server from the outside of the server, but one or both of them may be arranged in the server.

[2. motion]
Next, an information processing method executed by the information processing system 100 having the above-described configuration will be described. By this information processing method, the performance of the inference model generated or updated through the training of machine learning is evaluated, and based on the result of this evaluation, it is determined whether or not it is the limit of the improvement of the performance by the training. FIG. 3 is a flow chart showing a procedure example of this information processing method. In addition, in order to easily grasp the outline of the information processing method, a part of the procedure is simplified or the whole is explained according to a certain assumed situation, which will be supplemented later. As illustrated in FIG. 3, in the work process of improving the accuracy of the object detection model, image data for evaluation is input to the trained model to acquire the object detection result, and the object detection result and the image for evaluation are obtained. This is the procedure from the point where the result of the abnormality determination (see FIG. 2) by the abnormality determination unit 20 based on the reference information of the data is obtained. Here, the trained object detection model in which the evaluation data is input and the object detection is executed before the procedure shown in FIG. 3 is an example of the first model in the present embodiment, and the object detection result output by this object detection model. Is an example of the first inference result, and the abnormality determination by the abnormality determination unit 20 for this object detection result is an example of the first determination.

Among the results of the first determination, the erroneous detection identification unit 30 that has received the erroneous detection information from the abnormality determination unit 20 compares the erroneous detection information with the erroneous detection information accumulated in the past, and compares the false positive information with the erroneous detection information of the first model. In the object detection process for evaluation, it is determined whether or not a new false detection has occurred (step S10). While no new false positives have occurred (No in step S10), this information processing method does not proceed to the next procedure. That is, while the false positives identified as the same are reoccurring, for example, work is performed to eliminate the false positives.

When a new false positive has occurred (Yes in step S10), this information processing method proceeds to step S11. In step S11, the model training unit 60, which receives information on the result of this determination from the erroneous detection identification unit 30 via the notification control unit 50, acquires a new object detection model by machine learning training. In this training, the existing training data used for training the object detection model that generated the above-mentioned new false positive (referred to as false positive A for convenience) and false positive A One or both of the image data which is the generated evaluation data and the image data similar to this image are used for the training data. The object detection model acquired in step S11 is hereinafter referred to as model a for convenience. The above training by the model training unit 60 in step S11 is an example of the first training in the present embodiment, and the model a is an example of the second model in the present embodiment. Further, among the image data input to the model a as the evaluation data, the data of the image in which the detection is erroneous, that is, the image in which the erroneous detection A occurs is an example of the first evaluation data in the present embodiment.

Next, the detection processing unit 10 inputs image data which is evaluation data to the model a, and acquires the object detection result output by the model a in response to this input (step S20). The object detection result output by the model a is an example of the second inference result in the present embodiment.

Next, the abnormality determination unit 20 illustrates the procedure in FIG. 2 based on the reference information of the evaluation data input to the model a in step S20 and the object detection result output by the model a with respect to the input. The determination is executed (step S21). This abnormality determination in step S21 is an example of the second determination in the present embodiment.

Next, among the results of the second determination, the erroneous detection identification unit 30 that has received the erroneous detection information from the abnormality determination unit 20 compares the erroneous detection information with the erroneous detection information that has been received and accumulated in the past, and erroneously. Identification of whether or not it is the same as detection A, that is, confirmation of whether or not false detection A has been resolved (step S22). If the false positive A is not resolved (No in step S22), the process is restarted from step S11. In this case, step S11 may be executed by changing the training data to be used, or hyperparameters in machine learning training such as batch size, number of iterations or number of epochs, for example, from the time of execution of the previous step S11.

When the erroneous detection A is resolved (Yes in step S22), the undetected identification unit 40 that has received the undetected information from the abnormality determination unit 20 among the results of the second determination receives the undetected information in the past. By comparing with the undetected information accumulated in the above, it is determined whether or not a new undetected has occurred in the object detection process executed for the evaluation of the model a (step S23). When a new undetection has occurred (Yes in step S23), information regarding the result of this determination is passed from the undetected identification unit 40 to the model training unit 60 via the notification control unit 50. The model training unit 60 further acquires a new object detection model by machine learning training (step S24). In this training, the existing training data used for the training of the model a in step S11 and the evaluation data in which new undetected (referred to as undetected B for convenience to refer to this new undetected) are generated. And one or both of the data of the image and the data of the image similar to this image are used for the training data. The object detection model acquired in step S24 is hereinafter referred to as model b for convenience. The above training by the model training unit 60 in step S24 is an example of the second training in the present embodiment, and the model b is an example of the third model in the present embodiment. Further, among the image data input to the model b as the evaluation data, the data of the image in which the detection is omitted, that is, the image in which the undetected B occurs is an example of the second evaluation data in the present embodiment.

Next, the detection processing unit 10 inputs image data, which is evaluation data, to the model b, and acquires the object detection result output by the model b in response to this input (step S30). The object detection result output by the model b is an example of the third inference result in the present embodiment.

Next, the abnormality determination unit 20 illustrates the procedure in FIG. 2 based on the reference information of the evaluation data input to the model b in step S30 and the object detection result output by the model b with respect to the input. The determination is executed (step S31). This abnormality determination in step S31 is an example of the third determination in the present embodiment.

Next, among the results of the third determination, the undetected identification unit 40, which has received the undetected information from the abnormality determination unit 20, has not yet compared the undetected information with the undetected information received and accumulated in the past. It is confirmed whether or not the detection B has been resolved (step S32). If the undetected B is not resolved (No in step S32), the process is restarted from step S24. In this case, step S24 may be executed by changing the training data to be used or hyperparameters in machine learning training such as batch size, number of iterations or number of epochs, for example, from the time of execution of the previous step S24.

When the undetected B is resolved (Yes in step S32), the erroneous detection identification unit 30 that has received the erroneous detection information from the abnormality determination unit 20 among the results of the third determination is based on the erroneous detection information. It is determined whether or not erroneous detection has occurred (step S33). When a false positive has occurred (Yes in step S33), this false positive that occurred in model b in comparison with the false positive information received and accumulated in the past is a reoccurrence of false positive A that has been once resolved. Whether or not it is further determined (step S40). The result of the determination in step S40 is input to the notification control unit 50. When the false detection A reoccurs (Yes in step S40), the notification control unit 50 outputs information on the training limit, for example, outputs predetermined data indicating this information to a display device or a speaker to detect an object. Notify the model creation worker (step S50). Information about the training limit indicates, for example, the possibility of reaching the training limit. Upon receiving such a notification, the worker can know that the improvement in the performance of the model by the machine learning training that he / she was working on has reached the limit.

Further, in the example of FIG. 3, the false detection A is eliminated by the model a obtained through the subsequent training (Yes in step S22), and no undetection occurs in the model a (in step S23). In No), the user is notified that no conflict has occurred (step S51). Similarly, when the undetected B generated in the model a is once eliminated in the model b obtained through the subsequent training (Yes in step S32), and no false detection occurs in the model b (step S33). In No), the user is notified that no conflict has occurred or that the training limit has not been reached (step S51).

The above procedure is an example, and the processing content of the information processing method according to the present embodiment is not limited to this example. For example, in the above example, if the undetected B that occurred when the false detection A that occurred earlier is resolved is eliminated, the training is performed on the condition that the false positive A that has already been resolved reoccurs. It is determined that the limit has been reached. The content of the event that can be used in this way as a condition for determining that the training limit has been reached is not limited to this, and any contradictory occurrence may occur in which a plurality of abnormalities cannot be resolved at the same time. For example, if the false detection and the undetected in the above example are exchanged and the false detection that occurred when the previously generated undetected is once resolved, the undetected that has already been resolved will reoccur. It may be determined that the training limit has been reached. In addition, the occurrence of a plurality of false positives that cannot be resolved at the same time, or the occurrence of a plurality of undetected events that cannot be resolved at the same time may be contradictory conditions that are conditions for determining that the training limit has been reached.

(Modification example and other supplementary items)
The information processing method according to one or more aspects of the present disclosure is not limited to the description of the above embodiment. As long as it does not deviate from the gist of the present disclosure, various modifications conceived by those skilled in the art are also included in the embodiment of the present disclosure. Examples of such modifications and other supplements to the description of embodiments are given below.

(1) In the description of the above embodiment, IoU is used as an evaluation scale for whether or not the object detection result output by the object detection model is erroneous detection, but the evaluation method for the object detection result is limited to this. Not done. As an example of the evaluation scale acquired based on the detection frame and the correct answer frame, which can be used in the information processing method of the present disclosure other than IoU, Precision (also referred to as precision rate, which is used for error evaluation) and Recall (recall rate). Also referred to as (used for leak evaluation) and F value (used for error and leak evaluation).

(2) The output of data to be reproduced by the display device for notification or the speaker by the notification control unit 50 in the description of the above embodiment is an example of one form of output from the information processing system 100. , Other forms may be used. For example, it may be an output of data for storage as a log related to machine learning training in the information system 100 in a storage device (not shown) provided in the information system 100 or external to the information system 100. As another example, the data may be output as data to be processed or referenced data in some information processing inside or outside the information processing system 100. For example, it may be used for information processing to determine hyperparameters for creating an inference model.

(3) The machine learning training for obtaining the model a in step S11 and the model b in step S24 in the above embodiment is training using all training data, and each model is created from scratch. Good. In this case, the first model, the second model, and the third model are acquired as independent models. It may be additionally executed on an existing model using the added training data. The latter will be specifically described with reference to the example of FIG. 3. One of the data of the image in which the object detection by the model a has not been detected and the data of the image similar to this image in order to obtain the model b in step S24. Or both are used to perform additional training on model a. Then, in this case, the updated version of the first model is acquired as the second model, and the updated version of the second model is acquired as the third model.

(4) In the above embodiment, with respect to the occurrence of the contradictory, one abnormality (undetected or erroneous detection) is once resolved and then reoccurs with another abnormality (undetected or erroneous detection). Is explained using. However, the number of abnormalities that occur during this reoccurring abnormality is not limited to one. For example, when repeating a cycle including machine learning training executed to obtain a desired model, object detection, and abnormality determination of the object detection result, information on the content of the abnormality generated in each cycle is accumulated and each When an abnormality occurs in the cycle, the contents are collated with each abnormality. As a result of this collation, if the content of the abnormality that occurred in the latest cycle can be identified as the same as the abnormality that occurred in any of the past cycles, it may be included in the contradictory in the present disclosure. In this sense, the first inference result, the second inference result, and the third inference result in the present disclosure may be any one acquired in this order on the time axis in the repetition of the above cycle, and may be continuously obtained. It is not limited to what can be obtained.

(5) Although the above-described embodiment is described using an inference model that detects an object for image data as an example, the processing target by the inference model is not limited to the image data. Sensor data other than image data, as specific examples, data such as voice, distance point group, pressure, temperature, humidity, odor, etc., or data other than sensor data, for example, language data may be used. Then, when these data are input, the inference model may output the results of, for example, voice recognition, speaker recognition, stylistic analysis, and the like.

(6) A part or all of the functional components included in each of the above-mentioned information processing systems may be configured by one system LSI (Large Scale Integration). A system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip. Specifically, a microprocessor, a ROM (Read-Only Memory), and a RAM (Random Access Memory) are used. It is a computer system that includes such as. A computer program is stored in the ROM. When the microprocessor operates according to this computer program, the system LSI achieves the function of each component.

Although it is referred to as a system LSI here, it may be referred to as an IC, an LSI, a super LSI, or an ultra LSI depending on the degree of integration. Further, the method of making an integrated circuit is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connection and settings of circuit cells inside the LSI may be used.

Furthermore, if an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology or another technology derived from it, functional blocks may be integrated using that technology. The application of biotechnology, etc. is possible.

(7) One aspect of the present disclosure is not limited to the information processing method described above using the flow chart of FIG. 2 or FIG. 3, and may be a program executed by a computer and an information processing system including the computer. .. Also, one aspect of the present disclosure may be a computer-readable, non-temporary recording medium on which such a computer program is recorded.

The information processing method and the like according to the present disclosure can be used for developing a model whose performance is improved through machine learning training.

10 Detection processing unit 20 Abnormality judgment unit 30 False detection identification unit 40 Undetected identification unit 50 Notification control unit 60 Model training unit 100 Information processing system D10 Evaluation data D20 Training data

Claims (5)

It is an information processing method that is executed by a computer.
Get evaluation data,
The evaluation data is input to the first model that executes the inference process, and the first inference result is acquired.
Based on the first inference result and the reference information of the evaluation data, the first determination for determining an error or omission of the first inference result with respect to the evaluation data is executed.
The second model is created by the first training of machine learning in which one or both of the first evaluation data in which the first inference result is erroneous or omission and the data similar to the first evaluation data are included in the training data. Acquired,
The evaluation data is input to the second model to obtain the second inference result.
Based on the second inference result and the reference information of the evaluation data, a second determination for determining an error or omission of the second inference result with respect to the evaluation data is executed.
Training data of one or both of the second evaluation data in which the error or omission that was not in the first inference result in the evaluation data was in the second inference result and the data similar to the second evaluation data. Acquired the third model by the second training of machine learning performed in
The evaluation data is input to the third model to obtain the third inference result.
Based on the third inference result and the reference information of the evaluation data, a third determination for determining an error or omission of the third inference result with respect to the evaluation data is executed.
An information processing method that outputs information regarding the training limit of the first model when it is identified that the error or omission in the third inference result and the error or omission in the first inference result are the same. The evaluation data is image data and
The inference process is an object detection process.
The information processing method according to claim 1, wherein the first inference result, the second inference result, and the third inference result are object detection results, respectively. The object detection result includes a first detection frame which is a detection frame of an object as a result of the object detection process.
The reference information includes a second detection frame, which is a detection frame for an object as a reference.
The information processing method according to claim 2, wherein the error or the omission is determined based on the first detection frame and the second detection frame. By outputting the information, notification of the possibility that the training limit has been reached is executed, and the notification is executed.
The information processing method according to any one of claims 1 to 3, wherein the notification is executed via an image or voice. Inference processing unit and
Abnormality judgment unit and
Model training department and
Equipped with a notification control unit
The inference processing unit acquires the evaluation data, inputs the evaluation data into the first model that executes the inference processing, acquires the first inference result, and obtains the first inference result.
The abnormality determination unit executes a first determination for determining an error or omission of the first inference result with respect to the evaluation data based on the first inference result and the reference information of the evaluation data.
The model training unit includes one or both of the first evaluation data in which the first inference result is erroneous or omission and data similar to the first evaluation data in the training data, and the first machine learning is performed. Obtained the second model by training,
The inference processing unit further inputs the evaluation data into the second model to acquire the second inference result.
The abnormality determination unit further executes a second determination for determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and the reference information of the evaluation data.
The model training unit further includes a second evaluation data in which an error or omission that was not in the first inference result in the evaluation data was found in the second inference result, and data similar to the second evaluation data. Acquire the third model by the second training of machine learning that includes one or both in the training data.
The inference processing unit further inputs the evaluation data into the third model to acquire the third inference result.
The abnormality determination unit further executes a third determination for determining an error or omission of the third inference result with respect to the evaluation data based on the third inference result and the reference information of the evaluation data.
When the notification control unit identifies that the error or omission in the third inference result and the error or omission in the first inference result are the same, the notification control unit provides information on the training limit of the first model. Information processing system to output.

Images