JP7402082B2 - Information processing method and information processing system - Google Patents

Description

The present disclosure relates to an information processing method that is executed by a computer, and an information processing system that executes the information processing method.

Object detection technology using deep learning, which has been reported to be implemented with high accuracy (see, for example, Non-Patent Document 1), is expected to be put to practical use in various applications.

As a countermeasure against detection abnormalities (including false detections and non-detections) in object detection technology using machine learning such as deep learning, it is possible to use training data to which data for detection processing in which detection abnormalities have occurred is added. A model is trained by machine learning (see Patent Document 1).

Japanese Patent Application Publication No. 2002-342739

Qijie Zhao, 6 others, "M2Det: A Single-Shot Object Detector based on Multi-Level Feature Pyramid Network", November 2018, [online], arXiv, [searched on February 26, 2020], Internet <URL : https://arxiv.org/abs/1811.04533〉

However, the detection abnormality may not be resolved even by training using the added training data. It is difficult to judge whether the detection abnormality is resolved, that is, whether it is at the training limit.

The present disclosure provides an information processing method and the like that can determine the training limit of machine learning.

An information processing method according to an aspect of the present disclosure is a method executed by a computer, which acquires evaluation data, inputs the evaluation data to a first model that performs inference processing, and obtains a first inference result. and, based on the first inference result and the reference information of the evaluation data, perform a first determination to determine whether the first inference result is an error or omission with respect to the evaluation data, and determine whether there is an error or omission in the first inference result. A second model is obtained through first training of machine learning, which is performed by including in training data one or both of the first evaluation data that has been evaluated and data similar to the first evaluation data, and the second model is inputting evaluation data to obtain a second inference result, and performing a second determination of determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and reference information of the evaluation data; one or both of second evaluation data in which the second inference result contained an error or omission that was not present in the first inference result among the evaluation data; and data similar to the second evaluation data. A third model is obtained through second training of machine learning performed by including the above in the training data, the evaluation data is input to the third model to obtain a third inference result, and the third inference result and the evaluation data are A third judgment is performed to determine errors or omissions in the evaluation data in the third inference result based on the reference information of the third inference result, and the error or omission in the third inference result is determined based on the error or omission in the third inference result and in the first inference result. The information processing method outputs information regarding a training limit of the first model when errors or omissions are identified as being the same.

Further, an information processing system according to an aspect of the present disclosure includes an inference processing unit, an abnormality determination unit, a model training unit, and a notification control unit, and the inference processing unit acquires evaluation data and performs inference processing. The evaluation data is input to a first model that executes to obtain a first inference result, and the abnormality determination unit determines the first inference result based on the first inference result and reference information of the evaluation data. The model training unit executes a first determination to determine errors or omissions in the evaluation data, and the model training unit determines the first evaluation data in which there is an error or omission in the first inference result, and data similar to the first evaluation data. A second model is obtained by first training of machine learning performed by including one or both of the above in the training data, and the inference processing unit further inputs the evaluation data to the second model to obtain a second inference result. and the abnormality determination unit further executes a second determination for determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and reference information of the evaluation data, The model training unit further includes second evaluation data in which the second inference result has an error or omission that was not present in the first inference result among the evaluation data, and data similar to the second evaluation data. A third model is obtained through second training of machine learning performed by including one or both of them in training data, and the inference processing unit further inputs the evaluation data into the third model to obtain a third inference result. , the abnormality determination unit further executes a third determination for determining an error or omission of the third inference result with respect to the evaluation data based on the third inference result and reference information of the evaluation data, and controls the notification control. The unit outputs information regarding a training limit of the first model when it is identified that an error or omission in the third inference result and an error or omission in the first inference result are the same.

In addition to the methods and systems described above, these comprehensive or specific aspects may be realized by an apparatus, an integrated circuit, or a computer-readable recording medium such as a CD-ROM, and may be realized by an apparatus, a system, It may be implemented in any combination of integrated circuits, methods, computer programs, and storage media.

The training limit of machine learning can be determined by the information processing method and the like according to the present disclosure.

FIG. 1 is a block diagram showing an example of a functional configuration of an information processing system that executes an information processing method according to an embodiment. FIG. 2 is a flow diagram illustrating an example of an abnormality determination procedure in the information processing method according to the embodiment. FIG. 3 is a flow diagram illustrating a procedure example of the information processing method according to the embodiment.

(Findings that formed the basis of this disclosure)
The inventors discovered that the following problems occur with the conventional technology.

For example, when applying object detection technology to applications that require high reliability such as autonomous driving, it is essential to take measures against detection abnormalities such as false detection and non-detection. This is because such a detection abnormality becomes a cause of an accident involving human life.

In object detection technology that uses machine learning methods such as deep learning, as a countermeasure against detection abnormalities, it is recommended to train the machine learning model using training data that includes the data to be processed for detection where detection abnormalities have occurred. Common. However, even if the detection anomaly that occurred is resolved in the model obtained as a result of this training, it is possible to determine whether other detection anomalies will occur in this model. At present, the judgment as to whether there is a comprehensive improvement or a limit depends largely on the empirical rules or intuition of the person in charge of model training. For example, in order to obtain a more accurate object detection model, an experienced person would consider changing the network configuration, but in reality, an inexperienced person may need to perform further training with additional training data. It may happen that you choose to go and see the results in the model you get. As described above, a problem arises in that judgments about the limits of machine learning training vary depending on the technical level of the person in charge of the work.

An information processing method according to an aspect of the present disclosure devised in view of such problems is a method executed by a computer, which acquires evaluation data and applies the evaluation data to a first model that performs inference processing. Input data and obtain a first inference result, and perform a first determination to determine whether the first inference result is an error or omission with respect to the evaluation data based on the first inference result and reference information of the evaluation data. However, the first training of machine learning performed by including in the training data one or both of the first evaluation data in which there was an error or omission in the first inference result and the data similar to the first evaluation data obtain a model, input the evaluation data into the second model to obtain a second inference result, and based on the second inference result and reference information of the evaluation data, the evaluation data of the second inference result A second determination is performed to determine an error or omission in the second inference result, and the second evaluation data includes an error or omission in the second inference result that was not in the first inference result among the evaluation data, and Obtaining a third model through second training of machine learning performed by including one or both of data similar to the evaluation data in the training data, and inputting the evaluation data into the third model to obtain a third inference result. Based on the third inference result and the reference information of the evaluation data, a third judgment is performed to determine an error or omission in the third inference result with respect to the evaluation data, and an error in the third inference result is determined. Alternatively, the information processing method outputs information regarding the training limit of the first model when the omission and the error or omission in the first inference result are identified as being the same.

This makes it possible to determine the limit of improvement in the performance of an inference model through machine learning training, regardless of the technical level of the person in charge of creating the inference model.

Further, the evaluation data may be image data, the inference process may be an object detection process, and each of the first inference result, the second inference result, and the third inference result may be an object detection result.

That is, it is possible to determine the limit of improvement in performance of an inference model that performs object detection processing on image data, regardless of the technical level of the person in charge of creating it.

Further, the object detection result includes a first detection frame that is an object detection frame as a result of the object detection process, and the reference information includes a second detection frame that is an object detection frame as a reference, The error or the omission may be determined based on the first detection frame and the second detection frame.

An abnormality in object detection can be determined in this way.

Further, by outputting the information, notification of the possibility that the training limit has been reached may be performed, and the notification may be performed via an image or sound.

This provides the creator of the inference model with material for considering methods other than additional training based on data that was not successfully inferred, in order to improve the performance of the inference model, contributing to efficiency of work.

Further, an information processing system according to an aspect of the present disclosure includes an inference processing unit, an abnormality determination unit, a model training unit, and a notification control unit, and the inference processing unit acquires evaluation data and performs inference processing. The evaluation data is input to a first model that executes to obtain a first inference result, and the abnormality determination unit determines the first inference result based on the first inference result and reference information of the evaluation data. The model training unit executes a first determination to determine errors or omissions in the evaluation data, and the model training unit determines the first evaluation data in which there is an error or omission in the first inference result, and data similar to the first evaluation data. A second model is obtained by first training of machine learning performed by including one or both of the above in the training data, and the inference processing unit further inputs the evaluation data to the second model to obtain a second inference result. and the abnormality determination unit further executes a second determination for determining an error or omission of the second inference result with respect to the evaluation data based on the second inference result and reference information of the evaluation data, The model training unit further includes second evaluation data in which the second inference result has an error or omission that was not present in the first inference result among the evaluation data, and data similar to the second evaluation data. A third model is obtained through second training of machine learning performed by including one or both of them in training data, and the inference processing unit further inputs the evaluation data into the third model to obtain a third inference result. , the abnormality determination unit further executes a third determination for determining an error or omission of the third inference result with respect to the evaluation data based on the third inference result and reference information of the evaluation data, and controls the notification control. The unit outputs information regarding a training limit of the first model when it is identified that an error or omission in the third inference result and an error or omission in the first inference result are the same.

As a result, the limit of improvement in the performance of an inference model through machine learning training can be determined regardless of the technical level of the creator of the inference model.

In addition to the above-mentioned methods and systems, these comprehensive or specific aspects may be realized by a device, an integrated circuit, or a computer-readable recording medium such as a CD-ROM. , integrated circuits, methods, computer programs, and recording media may be implemented in any combination.

Embodiments of an information processing method and an information processing system according to one aspect of the present disclosure will be described below with reference to the drawings. The embodiment shown here shows one specific example of the present disclosure. Therefore, the numerical values, shapes, components, arrangement and connection forms of the components, steps (processes) and order of steps, etc. shown in the following embodiments are merely examples, and do not limit the present disclosure. Furthermore, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims are constituent elements that can be added arbitrarily. Furthermore, each figure is a schematic diagram and is not necessarily strictly illustrated.

(Embodiment)
The information processing method according to the embodiment is executed by a computer, and based on an abnormality (omission or error) in the processing result of an inference model obtained through machine learning training, performance remains unchanged even after further training. This is a method for determining that the possibility of improvement is low, that is, the practical limit (hereinafter also referred to as training limit). This information processing method will be explained below using an example case in which it is used to determine the training limit of an inference model that performs object detection in an image.

[1. composition]
FIG. 1 is a block diagram showing an example of the functional configuration of an information processing system that executes the information processing method according to the present embodiment.

In the information processing system 100 according to the present embodiment, machine learning training is performed to generate or update an inference model, and the generated or updated inference model is evaluated. Based on the results of this evaluation, the information processing system 100 determines whether there is a limit to the performance improvement due to training.

The information processing system 100 includes a server configured of one or more computers each having a processor and a memory and executing this information processing method. In the example shown in FIG. 1, this server includes a detection processing section 10, an abnormality determination section 20, a false detection identification section 30, an undetected identification section 40, a notification control section 50, and a model training section 60 as functional components.

The detection processing unit 10 causes an object detection model that has already been trained by machine learning (hereinafter also simply referred to as a trained model) to perform object detection processing for performance evaluation. The detection processing unit 10 that has acquired evaluation data (image data in this example) used for performance evaluation of the trained model performs preprocessing on the image used for this performance evaluation. The preprocessing referred to here is processing performed in preparation for object detection, such as adjusting the resolution of an image or normalizing pixel values. When the detection processing unit 10 acquires the trained model to be evaluated, it inputs each preprocessed image into the trained model and calculates the result of object detection processing (hereinafter referred to as object detection processing), which is inference processing by the object detection model. (referred to as the detection result) is output. Specifically, the object detection result is a detection frame surrounding an object, such as a pedestrian or an obstacle, detected in an image by the object detection model. The acquired detection frame information is output from the detection processing section 10 to the abnormality determination section 20. This information includes, for example, information specifying the image on which the object detection process has been performed, the range (position and size) and accuracy (also referred to as likelihood) of the detection frame in the image. The detection processing unit 10 is an example of the inference processing unit in the present embodiment, and the object detection results that the detection processing unit 10 acquires from the object detection results are the first inference result, the second inference result, and the second inference result in the present embodiment. This is an example of the third inference result.

In addition to the information on the detection frame input from the detection processing unit 10, the abnormality determination unit 20 acquires information on a detection frame (hereinafter also referred to as a correct frame) that indicates the correct answer for object detection in this image as a reference. Then, the abnormality determining unit 20 determines whether the object detection result obtained by the trained model is abnormal based on the detection frame input from the detection processing unit 10 and the detection frame indicating the correct answer. Abnormalities in this example include false detections (errors) and non-detections (omissions). FIG. 2 is a flowchart showing a procedure example of a method for determining these abnormalities in the object detection results performed by the abnormality determining section 20.

The abnormality determination unit 20 performs abnormality determination in descending order of accuracy of the detection frames acquired for one image indicated by the information input from the detection processing unit 10. First, the abnormality determination unit 20 identifies the one with the highest accuracy among the detection frames for which abnormality determination has not been performed (step S01).

Next, the abnormality determination unit 20 calculates IoU (Intersection over Union) between the detection frame identified in step S01 and the correct frame of the image (step S02).

Next, the abnormality determining unit 20 determines whether the IoU of each detection frame with the correct frame calculated in step S02 exceeds 0.5 (step S03). If the detection frame has an IoU of more than 0.5 with respect to the correct frame (Yes in step S03), the abnormality determination unit 20 determines that this detection frame is due to correct detection, and uses the correct frame with an IoU of more than 0.5. (Step S04). If the detection frame has an IoU with respect to the correct frame that does not exceed 0.5 (No in step S03), the abnormality determining unit 20 determines that this detection frame is due to erroneous detection (step S05).

Next, the abnormality determination unit 20 checks whether there are any further detection frames that have not been subjected to the series of steps up to S05 among the detection frames acquired for this image (step S06).

If there is a detection frame for which this procedure has not been performed (Yes in step S06), the abnormality determination unit 20 returns to step S01.

If the steps up to step S05 have been performed for all detection frames acquired for this image (No in step S06), the abnormality determination unit 20 determines which of the correct frames for this image have been It is checked whether there is a correct frame in which no pair is established by pairing in step S04 with any of the detection frames (step S07).

If there is a correct answer frame in which no pair has been established (Yes in step S07), the abnormality determination unit 20 determines that this correct answer frame is due to undetection (step S08), and the series of procedures ends.

If there is no correct frame that does not form a pair, that is, all correct frames are paired with any detection frame (No in step S07), a series of steps for determining whether there is an abnormality in object detection processing for this image is performed. The procedure ends.

The results of the determination by the abnormality determining unit 20, that is, information regarding the correctness of each detection frame (hereinafter also referred to as false detection information), are output to the false detection identifying unit 30, and information regarding non-detection (hereinafter also referred to as undetected information) is output to the false detection identifying unit 30. It is output to the undetected identification section 40. Erroneous detection information and non-detection information will be described later.

Note that, for the sake of simplicity, the above procedure example is described assuming that a detection frame is acquired for only one type of object detection target. If there are multiple types of objects to be detected, step S02 may be performed between the detection frame and the correct frame for the same type of object. In this case, for example, information regarding the type of detected object, which is further included in each of the detection frame information and the correct frame information acquired by the abnormality determination unit 20, is used.

Further, the reference value of IoU of 0.5 in step S03 is an example, and will be used as an example in the following description. This reference value is set, for example, so that only one of the IoUs calculated between one detection frame and each of a plurality of correct answer frames exceeds it. However, if a plurality of IoUs still exceed the reference value, for example, a combination of the detection frame and the correct frame with the highest IoU may be paired in step S04. Alternatively, based on the idea that the detection result is ambiguous, it may be determined that the detection frame is due to an erroneous detection.

The detection frame indicated by the information that the abnormality determination unit 20 receives from the detection processing unit 10 is an example of the first detection frame in this embodiment. Further, the information on the correct frame acquired by the abnormality determining unit 20 is an example of reference information in this embodiment, and the correct frame is an example of the second detection frame. Further, the determinations that the abnormality determination unit 20 performs in accordance with the procedure illustrated in FIG. 2 are examples of the first determination, the second determination, and the third determination in this embodiment.

The false detection identification unit 30 acquires false detection information from the abnormality determination unit 20 and also accumulates this false detection information. The false detection information includes details of the false detection that occurred, such as the image on which object detection was performed, the correctness of the detection result, and the detection frame (hereinafter also referred to as false detection frame) of the object detection result that was determined to be false detection. Contains information indicating. The false detection identification unit 30 uses the accumulated false detection information to determine whether the false detection whose details are shown in the false detection information newly acquired from the abnormality determination unit 20 is a reoccurrence of a previous false detection. Make a judgment. As a specific example, the false detection identification unit 30 distinguishes between a false detection frame indicated by newly acquired false detection information and a false detection frame caused by false detection generated in object detection processing for the same image using an older object detection model whose performance has been evaluated. Calculate the IoU with. If this IoU exceeds a predetermined reference value (for example, 0.5), the false detection identification unit 30 determines that the false detection detailed in the newly acquired false detection information is an error that occurred in an older object detection model. It is identified that this is the same false detection as the previous detection, and it is determined that this is a reoccurrence of the previous false detection. Furthermore, if this IoU is less than or equal to a predetermined threshold, the false detection identification unit 30 determines that the false detection detailed in the newly acquired false detection information is not an existing false detection (it is a new false detection). judge. Then, information regarding the result of this determination is output from the false detection identification section 30 to the notification control section 50. In addition to the determination result, the information regarding the determination result includes, for example, information specifying an image in which an erroneous detection has occurred. Information regarding the result of this determination is passed to the model training unit 60, which will be described later, and is used to obtain training data to be used in the next machine learning training.

The undetected identifying unit 40 acquires information regarding undetected from the abnormality determining unit 20, and also accumulates this undetected information. The undetected information includes information indicating details of the undetected items that have occurred, such as images in which object detection has been performed and correct frames that have been determined to be undetected (hereinafter also referred to as undetected frames). The undetected identification unit 40 determines whether the undetected information whose details are newly acquired from the abnormality determination unit 20 is a reoccurrence of the previously reported undetected information. Judgment is made using As a specific example, the undetected identification unit 40 determines that the undetected frame indicated by the newly acquired undetected information is an undetected frame due to undetected objects generated in object detection processing for the same image using an older object detection model whose performance has been evaluated. Identify whether it is the same as or not. If these two undetected frames are the same, the undetected identification unit 40 determines that the undetected item detailed in the newly acquired undetected information is a reoccurrence of the previously reported undetected item. In addition, if these two undetected frames are not the same, the undetected identification unit 40 determines that the undetected whose details are shown in the newly acquired undetected information is not an already detected undetected (it is a new undetected). It is determined that Then, information regarding the result of this determination is output from the undetected identification section 40 to the notification control section 50. In addition to the determination result, the information regarding the determination result includes, for example, information specifying an image in which a new non-detection has occurred. Information regarding the result of this determination is passed to the model training unit 60, which will be described later, and is used to obtain training data to be used in the next machine learning training.

The notification control unit 50 determines whether or not a conflict has occurred based on the information regarding the determination results inputted from the false detection identification unit 30 and the non-detection identification unit 40, and sends a notification according to the result of this determination to the model training unit. Send to 60. The trade-off here refers to the fact that false detections or non-detections that were once resolved through machine learning training reoccur in trained models obtained through training to eliminate other false detections or non-detections. As a specific example of the notification, when it is determined that a conflict has occurred, the notification control unit 50 sends information regarding the training limit to the user of the information processing system 100, that is, to the creation of the object detection model. Notify the person in charge. As information regarding the training limit, a detection result (detection frame) related to the conflict determined to have occurred may be output. This notification is performed, for example, via an image displayed on a display device (not shown) connected to a computer that constitutes the server in FIG. Furthermore, if it is determined that no conflict has occurred, the notification control unit 50 may notify the user that the training has ended normally. These notifications may be made via audio output from a speaker (not shown) connected to a computer constituting the server in FIG. Further, when it is determined that no conflict has occurred, the notification control unit 50 models the information regarding new false detections or the information regarding new non-detections input from the false detection identification unit 30 and the non-detection identification unit 40. Output to the training section 60.

The model training unit 60 executes machine learning training to obtain an object detection model to be subjected to performance evaluation. When the model training unit 60 receives information regarding a new false detection from the notification control unit 50, it acquires training data used for training to eliminate this false detection from the training data D20. The training data used for training to eliminate false detections is, for example, data of an image in which a new false detection has occurred, which is specified by information regarding the new false detection. When the model training unit 60 also receives input of information regarding new non-detection from the notification control unit 50, it acquires training data to be used for training to eliminate this non-detection from the training data D20. The training data used for training to eliminate the non-detection is, for example, data of an image in which the new non-detection has occurred, which is specified by information regarding the new non-detection. In other words, in addition to the training data D20, data derived from the evaluation data D10 is used for machine learning training. Moreover, data of an image similar to the image in which a new false detection has occurred may be used as training data used for training to eliminate false detection. Similarly, data of an image similar to the image in which a new false detection has occurred may be used as training data used for training to eliminate non-detection. A similar image is, for example, an image that has the same or similar composition as the image, and is photographed with different photographic settings such as exposure or shutter speed. Other examples of similar images include images in which noise has been added or reduced by performing image processing on the image, and images in which brightness, contrast, saturation, white balance, etc. have been changed.

In order to perform machine learning training, the model training unit 60 generates a dataset combining the acquired training data and the training data used for training to obtain a trained model in which false detection or non-detection occurred. Prepare. Then, the model training unit 60 uses this data set to perform machine learning training on an object detection model with a predetermined configuration. When the training is completed, the model training unit 60 outputs a new trained model.

These functional components are computers that constitute a server, in which a processor executes a program held in memory, and stores data in memory or reads data stored in memory as necessary. Realized.

In the example of FIG. 1, the evaluation data D10 and the training data D20 are input into the server from outside the server in the information processing system 100, but one or both may be placed inside the server.

[2. motion]
Next, an information processing method executed by the information processing system 100 configured as described above will be described. With this information processing method, the performance of an inference model generated or updated through machine learning training is evaluated, and based on the result of this evaluation, it is determined whether or not there is a limit to the performance improvement due to training. FIG. 3 is a flow diagram showing an example of the procedure of this information processing method. In order to make it easier to understand the outline of the information processing method, some of the procedures will be simplified or the entire procedure will be explained based on a certain assumed situation, and supplementary information will be provided later. What is illustrated in Figure 3 is that in the process of improving the accuracy of the object detection model, image data for evaluation is input to the trained model to obtain object detection results, and this object detection result and the image for evaluation are The procedure starts from the point where the result of the abnormality determination (see FIG. 2) by the abnormality determination unit 20 based on the data reference information is obtained. Here, the trained object detection model that has been input with evaluation data and executed object detection before the procedure shown in FIG. 3 is an example of the first model in this embodiment, and the object detection result output by this object detection model is is an example of the first inference result, and the abnormality determination by the abnormality determination unit 20 with respect to this object detection result is an example of the first determination.

Among the results of this first determination, the false detection identification unit 30 receives the false detection information from the abnormality determination unit 20, and compares this false detection information with the false detection information received and accumulated in the past. It is determined whether a new false detection has occurred in the object detection process performed for evaluation (step S10). As long as no new false detection occurs (No in step S10), this information processing method does not proceed to the next step. In other words, while an erroneous detection that is identified as the same occurs again, work is performed to eliminate the erroneous detection, for example.

If a new false detection has occurred (Yes in step S10), the information processing method advances to step S11. In step S11, the model training unit 60, which has received information regarding the result of this determination from the false detection identification unit 30 via the notification control unit 50, acquires a new object detection model through machine learning training. In this training, we will use the existing training data used to train the object detection model that caused the above-mentioned new false detection (this new false detection will be referred to as false detection A for convenience) and the false detection A. One or both of image data, which is generated evaluation data, and image data similar to this image is used as training data. The object detection model acquired in step S11 is hereinafter referred to as model a for convenience. The above training by the model training unit 60 in step S11 is an example of the first training in this embodiment, and model a is an example of the second model in this embodiment. Further, among the image data input to the model a as evaluation data, data of an image with a detection error, that is, an image in which erroneous detection A has occurred, is an example of the first evaluation data in this embodiment.

Next, the detection processing unit 10 inputs image data, which is evaluation data, to model a, and obtains an object detection result output by model a in response to this input (step S20). The object detection result output by model a is an example of the second inference result in this embodiment.

Next, the abnormality determination unit 20 detects an abnormality, the procedure of which is illustrated in FIG. A determination is made (step S21). This abnormality determination in step S21 is an example of the second determination in this embodiment.

Next, the false detection identification unit 30 that receives the false detection information from the abnormality determination unit 20 among the results of this second determination compares this false detection information with the false detection information received and accumulated in the past. It is determined whether or not it is the same as detection A, that is, it is confirmed whether or not erroneous detection A has been resolved (step S22). If the false detection A has not been resolved (No in step S22), the process starts over from step S11. In this case, step S11 may be executed by changing the training data used or the hyperparameters in machine learning training, such as the batch size, number of iterations, or number of epochs, from the previous execution of step S11, for example.

If the false detection A has been resolved (Yes in step S22), the non-detection identification unit 40, which received the non-detection information from the abnormality judgment unit 20, among the results of the second determination, determines whether the non-detection information has been received in the past. It is determined whether or not a new non-detection has occurred in the object detection process executed to evaluate the model a (step S23). If a new non-detection has occurred (Yes in step S23), information regarding the result of this determination is passed from the non-detection identification unit 40 to the model training unit 60 via the notification control unit 50. The model training unit 60 further acquires a new object detection model through machine learning training (step S24). In this training, existing training data used for training model a in step S11 and images that are evaluation data in which new non-detection (this new non-detection is referred to as non-detection B for convenience) have occurred. and/or data of an image similar to this image are used as training data. The object detection model acquired in step S24 is hereinafter referred to as model b for convenience. The above training by the model training unit 60 in step S24 is an example of the second training in this embodiment, and model b is an example of the third model in this embodiment. Further, among the image data input to model b as evaluation data, data of an image in which detection is omitted, that is, an image in which undetected B occurs, is an example of second evaluation data in this embodiment.

Next, the detection processing unit 10 inputs image data, which is evaluation data, to model b, and obtains an object detection result output by model b in response to this input (step S30). The object detection result output by model b is an example of the third inference result in this embodiment.

Next, the abnormality determination unit 20 detects an abnormality, the procedure of which is illustrated in FIG. A determination is made (step S31). This abnormality determination in step S31 is an example of the third determination in this embodiment.

Next, the undetected identifying unit 40 that receives the undetected information from the abnormality determining unit 20 among the results of this third determination compares this undetected information with the undetected information received and accumulated in the past, and compares the undetected information with the undetected information received and accumulated in the past. It is confirmed whether detection B has been resolved (step S32). If undetected B has not been resolved (No in step S32), the process is restarted from step S24. In this case, step S24 may be executed by changing the training data used or hyperparameters in machine learning training such as batch size, number of iterations, or number of epochs from the previous execution of step S24, for example.

If the non-detection B has been resolved (Yes in step S32), the false detection identification unit 30 that has received the false detection information from the abnormality determination unit 20 among the results of the third determination, based on this false detection information, It is determined whether erroneous detection has occurred (step S33). If a false detection has occurred (Yes in step S33), it is compared with the false detection information received and accumulated in the past, and it is determined that this false detection that occurred in model b is a reoccurrence of the false detection A that was once resolved. It is further determined whether or not (step S40). The result of the determination in step S40 is input to the notification control unit 50. If false detection A has occurred again (Yes in step S40), the notification control unit 50 outputs information regarding the training limit, for example, outputs predetermined data indicating this information to a display device or a speaker, and detects the object. The model creation operator is notified (step S50). Information regarding the training limit indicates, for example, the possibility that the training limit has been reached. By receiving such a notification, the worker can know that the improvement in model performance through machine learning training that they have been working on has reached its limit.

Furthermore, in the example of FIG. 3, if the false detection A is resolved by model a obtained through subsequent training (Yes in step S22), and no non-detection occurs in model a (step S23). If No), the user is notified that no violation has occurred (step S51). Similarly, if undetected B that occurred in model a has been resolved in model b obtained through subsequent training (Yes in step S32), and no false detection occurs in model b (step S33). If the result is No), the user is notified that no violation has occurred or that the training limit has not been reached (step S51).

Note that the above procedure is an example, and the processing content of the information processing method according to the present embodiment is not limited to this example. For example, in the above example, training is performed on the condition that an event occurs in which when the undetected B that occurred when the previously occurring false positive A is resolved, the false positive A that has already been resolved will reoccur. It is determined that the limit has been reached. The content of the event that can be used as a condition for determining that the training limit has been reached is not limited to this, and may be any occurrence of a conflict in which multiple abnormalities cannot be resolved at the same time. For example, if you replace the false detection and non-detection in the above example and eliminate the false detection that occurred when the previous non-detection was resolved, an event will occur where the previously resolved non-detection will reoccur. It may be determined that the training limit has been reached. Furthermore, the occurrence of the event of the existence of a plurality of false detections that cannot be resolved at the same time, or the occurrence of the event of the existence of a plurality of undetected items that cannot be resolved at the same time, may be the content of the contradiction that is the condition for determining that the training limit has been reached.

(Modifications and other supplementary matters)
The information processing method according to one or more aspects of the present disclosure is not limited to the description of the embodiments above. Various modifications to the above-described embodiments that can be imagined by those skilled in the art are also included in the aspects of the present disclosure, as long as they do not depart from the spirit of the present disclosure. Examples of such modifications and other supplementary matters to the description of the embodiments are listed below.

(1) In the description of the above embodiment, IoU is used as an evaluation measure for whether or not the object detection result output by the object detection model is a false detection, but the evaluation method for the object detection result is limited to this. Not done. In addition to IoU, examples of evaluation scales obtained based on the detection frame and the correct frame that can be used in the information processing method of the present disclosure include Precision (also referred to as precision rate, used to evaluate errors), Recall (recall rate), etc. F value (used for evaluating errors and omissions) and F value (used for evaluating errors and omissions).

(2) In the description of the above embodiment, the notification control unit 50 outputs data to be reproduced on a display device or a speaker for notification, which is an example of one form of output from the information processing system 100. , other forms are also possible. For example, the data may be output to be stored as a log related to machine learning training in the information processing system 100 in a storage device (not shown) provided in the information processing system 100 or external to the information processing system 100. As another example, the data may be output as data to be processed or data to be referenced in some information processing within or outside the information processing system 100. For example, it may be used for information processing to determine hyperparameters for creating an inference model.

(3) The machine learning training to obtain model a in step S11 and model b in step S24 in the above embodiment is training using all training data, and each model is created from scratch. good. In this case, the first model, second model, and third model are obtained as mutually independent models. It may be additionally executed on an existing model using additional training data. To explain the latter in detail according to the example of FIG. 3, in order to obtain model b in step S24, one of the data of the image in which non-detection occurred in object detection by model a and the data of an image similar to this image is used. or perform additional training on model a using both. In this case, the updated first model is acquired as the second model, and the updated second model is acquired as the third model.

(4) In the above embodiment, regarding the occurrence of a conflict, there is an example in which one abnormality (undetected or erroneously detected) is once resolved and then reoccurs with another abnormality (undetected or erroneously detected) in between. It is explained using However, the number of abnormalities that occur during this reoccurring abnormality is not limited to one. For example, when repeating a cycle that includes machine learning training, object detection, and abnormality determination of object detection results to obtain a certain desired model, information about the contents of abnormalities that occur in each cycle is accumulated, and each If an abnormality occurs in the cycle, compare each abnormality with the contents. As a result of this verification, a case where the content of the abnormality that occurred in the latest cycle can be identified as the same as the abnormality that occurred in any past cycle may also be included in the conflict in the present disclosure. In this sense, the first inference result, the second inference result, and the third inference result in the present disclosure may be any one obtained in this order on the time axis during the repetition of the above cycle, and may be obtained continuously. You are not limited to what you can get.

(5) Although the above embodiment has been described using as an example an inference model that performs object detection on image data, the object to be processed by the inference model is not limited to image data. Sensor data other than image data, such as voice, distance point group, pressure, temperature, humidity, odor, etc., may be used as specific examples, or data other than sensor data, such as language data, may be used. The inference model may output, for example, the results of voice recognition, speaker recognition, stylistic analysis, etc., when these data are input.

(6) A part or all of the functional components included in each of the above-mentioned information processing systems may be configured by one system LSI (Large Scale Integration). A system LSI is a super-multifunctional LSI manufactured by integrating multiple components on a single chip, and specifically includes a microprocessor, ROM (Read-Only Memory), and RAM (Random Access Memory). It is a computer system that includes the following. A computer program is stored in the ROM. The system LSI achieves the functions of each component by the microprocessor operating according to this computer program.

Note that although it is referred to as a system LSI here, it may also be called an IC, LSI, super LSI, or ultra LSI depending on the degree of integration. Moreover, the method of circuit integration is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used.

Furthermore, if an integrated circuit technology that replaces LSI emerges due to advances in semiconductor technology or other derived technology, the functional blocks may be integrated using that technology. Possibilities include the application of biotechnology.

(7) One aspect of the present disclosure is not limited to the information processing method described above using the flowchart of FIG. 2 or 3, and may be a program executed by a computer and an information processing system including the computer. . Further, one aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a computer program is recorded.

The information processing method and the like according to the present disclosure can be used to develop a model whose performance can be improved through machine learning training.

10 detection processing unit 20 abnormality determination unit 30 false detection identification unit 40 non-detection identification unit 50 notification control unit 60 model training unit 100 information processing system D10 evaluation data D20 training data

Claims (5)

An information processing method for causing a computer to execute,
Obtain evaluation data,
inputting the evaluation data into a first model that executes inference processing to obtain a first inference result;
Based on the first inference result and reference information of the evaluation data, perform a first determination to determine whether the first inference result is an error or omission in the evaluation data;
A second model is created by first training of machine learning, which is performed by including in training data one or both of the first evaluation data in which there is an error or omission in the first inference result, and data similar to the first evaluation data. Acquired,
inputting the evaluation data into the second model to obtain a second inference result;
Based on the second inference result and reference information of the evaluation data, perform a second determination to determine whether the second inference result is an error or omission in the evaluation data;
Among the evaluation data, one or both of second evaluation data in which the second inference result had an error or omission that was not present in the first inference result and data similar to the second evaluation data are used as training data. A third model is obtained by the second training of machine learning, which is performed by including
inputting the evaluation data into the third model to obtain a third inference result;
Based on the third inference result and reference information of the evaluation data, perform a third determination to determine whether the third inference result is an error or omission in the evaluation data;
If an error or omission in the third inference result and an error or omission in the first inference result are identified as being the same, outputting information regarding a training limit of the first model. The evaluation data is image data,
The inference process is an object detection process,
The information processing method according to claim 1, wherein each of the first inference result, the second inference result, and the third inference result is an object detection result. The object detection result includes a first detection frame that is an object detection frame as a result of the object detection process,
The reference information includes a second detection frame that is a detection frame of an object as a reference,
The information processing method according to claim 2, wherein the error or the omission is determined based on the first detection frame and the second detection frame. By outputting the information, a notification of the possibility that the training limit has been reached is executed;
The information processing method according to any one of claims 1 to 3, wherein the notification is performed via image or audio. An inference processing unit;
An abnormality determination section;
model training department,
and a notification control section,
The inference processing unit obtains evaluation data, inputs the evaluation data into a first model that executes inference processing, and obtains a first inference result;
The abnormality determination unit executes a first determination to determine an error or omission of the first inference result with respect to the evaluation data based on the first inference result and reference information of the evaluation data,
The model training unit performs first machine learning by including in training data one or both of first evaluation data in which there is an error or omission in the first inference result and data similar to the first evaluation data. Obtain a second model by training,
The inference processing unit further inputs the evaluation data into the second model to obtain a second inference result,
The abnormality determination unit further performs a second determination to determine an error or omission of the second inference result with respect to the evaluation data based on the second inference result and reference information of the evaluation data,
The model training unit further includes second evaluation data in which the second inference result has an error or omission that was not present in the first inference result among the evaluation data, and data similar to the second evaluation data. A third model is obtained through second training of machine learning, which is performed by including one or both of the above in the training data,
The inference processing unit further inputs the evaluation data into the third model to obtain a third inference result,
The abnormality determination unit further performs a third determination to determine an error or omission of the third inference result with respect to the evaluation data based on the third inference result and reference information of the evaluation data,
When the error or omission in the third inference result is identified as the same as the error or omission in the first inference result, the notification control unit transmits information regarding the training limit of the first model. Information processing system that outputs.

Images