JP2023036131A - Inspection system, method, and program - Google Patents

Abstract

To prevent a possible erroneous determination that a machine learning model has deteriorated even if the machine learning model has not deteriorated.SOLUTION: An inspection system performs an inspection using a machine learning model. The machine learning model is a learned machine learning model having performed machine learning for predicting a predetermined event for an inspection target. An inspection unit repeatedly executes prediction about the inspection target using the machine learning model upon receipt of data about the inspection target as input data, and outputting prediction result data indicating a predicted result. An accumulation unit accumulates the prediction result data output from the inspection unit. An evaluation unit evaluates whether or not the input data falls within a range assumed at the time of learning of the machine learning model from the tendency of time series variation of the result predicted for the inspection target indicated by the prediction result data accumulated in the accumulation unit, and outputs an evaluation result.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to inspection systems, methods, and programs.

In recent years, from the viewpoint of labor saving, machine learning models are used for quality control in the field of manufacturing. For example, a computer on which a machine learning model is built predicts whether a product will meet predetermined inspection criteria. Further, in Patent Document 1, a computer in which a machine learning model is constructed predicts whether or not the quality of a product that has undergone a manufacturing process satisfies a predetermined condition from input equipment data in the manufacturing process. is described.

It is known that the machine learning model deteriorates as the use of the machine learning model continues. As the machine learning model deteriorates, the prediction accuracy decreases. Therefore, Patent Literature 1 describes that when the prediction accuracy of the machine learning model changes, it is determined that the machine learning model has deteriorated and a new machine learning model is generated.

JP 2019-87101 A

However, even if the machine learning model is not actually degraded, other conditions may reduce the prediction accuracy of the machine learning model. For this reason, as described in Patent Document 1, when a change in the prediction accuracy of a machine learning model is uniformly regarded as deterioration of the machine learning model, even if the machine learning model has not deteriorated, machine learning There is a problem that the model may be erroneously determined to be degraded.

The present disclosure can be implemented as the following forms.

(1) An aspect of the present disclosure provides an inspection system that performs inspection using a machine learning model. A machine learning model is a learned machine learning model that has undergone machine learning for predicting a predetermined event with respect to an inspection target. This inspection system includes an inspection unit that repeatedly executes prediction of an inspection object using a machine learning model using data about the inspection object as input data, and outputting prediction result data indicating the predicted result, and an inspection unit. Input data is assumed during training of the machine learning model from the storage unit that stores the output prediction result data and the trend of time-series changes in the prediction result of the inspection target indicated by the prediction result data stored in the storage unit. and an evaluation unit that evaluates whether or not it is within the range and outputs the evaluation result.
According to this aspect, in determining whether or not the machine learning model is degraded, not only the prediction accuracy of the machine learning model but also other conditions such as whether or not the input data is within the assumed range can be considered. . Therefore, even if the machine learning model is not actually degraded, it is possible to prevent erroneous judgment that the machine learning model is degraded when the prediction accuracy of the machine learning model is degraded due to other conditions. be able to.
(2) In the inspection system of the above aspect, the machine learning model may be obtained by performing machine learning for predicting to which of a plurality of preset categories the inspection target is classified. The inspection unit predicts prediction probabilities indicating the possibility of the inspection object being classified into each of the plurality of classifications. The inspection unit determines that the inspection target is classified into the category with the highest predicted probability value. The inspection unit outputs prediction result data including predicted prediction probabilities for the classes determined for the inspection object. The accumulation unit accumulates the prediction result data output by the inspection unit together with time information indicating when the prediction for the inspection object was performed. The evaluation unit evaluates the prediction result data stored in the storage unit for the classification determined for the inspection object, based on the trend of time-series changes in the predicted prediction probability, when the input data is learned by the machine learning model. You may evaluate whether it is in the assumed range.
According to this aspect, whether or not the input data is within the range assumed at the time of learning of the machine learning model from the tendency of the time-series change of the prediction probability predicted for the classification determined for the inspection target. can be evaluated.
(3) In the inspection system of the above aspect, the machine learning model may be obtained by performing machine learning for predicting whether or not the inspection target is within the normal range. The inspection unit predicts whether the object to be inspected is within the normal range. The inspection unit outputs prediction result data including a value representing the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested during the set period. The accumulation unit accumulates the prediction result data output by the inspection unit together with time information indicating when the prediction for the inspection object was performed. The evaluation unit evaluates whether or not the input data is within the range assumed at the time of learning of the machine learning model from the trend of time-series changes in the ratio indicated by the prediction result data accumulated in the accumulation unit. good.
According to this aspect, from the trend of time-series change in the ratio of the number of test subjects predicted to be in the normal range to the total number of test subjects tested, the input data is used when learning the machine learning model It can be evaluated whether it is within the assumed range.
(4) In the inspection system of the above aspect, the input data may be image data representing the appearance of the inspection object acquired by the sensor.
According to such an aspect, the inspection system can use the image data to perform a visual inspection of the inspection target. Furthermore, it is possible to evaluate whether or not the input data is within the range assumed at the time of learning of the machine learning model from the trend of time-series changes in the prediction results predicted by the machine learning model.
(5) The input data may be audio data representing the operating state of the inspection target acquired by a sensor.
According to such an aspect, the inspection system can inspect the object to be inspected for abnormal noise. Furthermore, it is possible to evaluate whether or not the input data is within the range assumed at the time of learning of the machine learning model from the trend of time-series changes in the prediction results predicted by the machine learning model.
Aspects of the present disclosure can also be implemented in various aspects other than inspection systems. For example, it can be implemented by a method executed by a computer that performs inspection using a machine learning model, and a computer program that implements the method.

It is a figure showing functional composition of an inspection system concerning a 1st embodiment. It is a figure which shows the hardware constitutions of the inspection apparatus which concerns on 1st Embodiment. It is a figure which shows the hardware constitutions of the evaluation apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the data accumulate|stored in the accumulation|storage part which concerns on 1st Embodiment. 4 is a flowchart of classification processing according to the first embodiment; 4 is a flowchart of evaluation processing according to the first embodiment; It is a figure showing functional composition of an inspection system concerning a 2nd embodiment. FIG. 11 is a diagram showing an example of data accumulated in an accumulation unit according to the second embodiment; FIG. 9 is a flowchart of inspection processing according to the second embodiment; 9 is a flowchart of evaluation processing according to the second embodiment;

A. First Embodiment FIG. 1 is a diagram showing the functional configuration of an inspection system 1000 according to the first embodiment. The inspection system 1000 includes an inspection device 100 and an evaluation device 200 . The inspection device 100 and the evaluation device 200 are different computers installed in the server room in the factory.

A sensor 50 is connected to the inspection apparatus 100 via a network. In this embodiment, an example in which the sensor 50 is a camera will be described. The sensor 50 captures an image of a component to be inspected by the inspection apparatus 100 at a predetermined position on the production line. The objects to be inspected are, for example, automobile parts such as body panels, engine cylinder blocks, and gears incorporated in transmissions. In this embodiment, the inspection objects are two types of parts, part A1 and part A2. Therefore, the target imaged by the sensor 50 is either the parts A1 or A2. Sensor 50 transmits image data obtained by photographing either component A1 or A2 to inspection device 100 .

The inspection apparatus 100 uses the image data supplied from the sensor 50 as input data, and uses a machine learning model to classify and determine the quality of the inspection object. In this embodiment, the machine learning model is a trained model that has undergone machine learning for predicting a predetermined event with respect to the object to be inspected. The evaluation device 200 evaluates the result of the determination made by the inspection device 100 on the inspection object.

FIG. 2 is a diagram showing the hardware configuration of the inspection apparatus 100. As shown in FIG. The inspection apparatus 100 includes a processor 11 , a main storage section 12 , an auxiliary storage section 13 , an interface 14 and a communication section 15 . Main storage unit 12 , auxiliary storage unit 13 , interface 14 , and communication unit 15 are connected to processor 11 via internal bus 19 .

The processor 11 is, for example, a CPU (Central Processing Unit). Each function of the inspection apparatus 100 is realized by the processor 11 reading the program stored in the auxiliary storage unit 13 into the main storage unit 12 and executing the program.

The main storage unit 12 is, for example, a main storage device including a RAM (Random Access Memory). A program is loaded from the auxiliary storage unit 13 into the main storage unit 12 . The main storage unit 12 is also used as a working area when the processor 11 executes programs.

The auxiliary storage unit 13 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an SSHD (Solid State Hybrid Drive). The auxiliary storage unit 13 stores programs executed by the processor 11 and various data used for executing the programs. For example, the auxiliary storage unit 13 stores a program for the inspection apparatus 100 to inspect an object to be inspected. Auxiliary storage unit 13 supplies data used by processor 11 to processor 11 under the control of processor 11 and stores the data supplied from processor 11 .

The interface 14 is, for example, an I/O (Input/Output) port such as a serial port or a USB (Universal Serial Bus) port. Input devices such as a keyboard, mouse, and touch panel are connected to the interface 14 . The interface 14 outputs information input by the user through the input device to the processor 11 . A display device 61 is also connected to the interface 14 . The display device 61 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The interface 14 outputs the image supplied from the processor 11 to the display device 61 .

Communication unit 15 includes a network interface circuit for communicating with an external device. Communication unit 15 receives a signal from the outside and outputs data indicated by this signal to processor 11 . Also, the communication unit 15 transmits a signal indicating data output from the processor 11 to an external device.

FIG. 3 is a diagram showing the hardware configuration of the evaluation device 200. As shown in FIG. The evaluation device 200 includes a processor 21 , a main storage section 22 , an auxiliary storage section 23 , an interface 24 and a communication section 25 . Main storage unit 22 , auxiliary storage unit 23 , interface 24 , and communication unit 25 are connected to processor 21 via internal bus 29 .

The processor 21 has functions similar to those of the processor 11 of the inspection apparatus 100 . The main storage unit 22 has the same functions as the main storage unit 12 of the inspection apparatus 100 . The auxiliary storage unit 23 has the same functions as the auxiliary storage unit 13 of the inspection device 100 . The interface 24 has functions similar to those of the interface 14 of the inspection apparatus 100 . A display device 62 is connected to the interface 24 . The display device 62 is, for example, a liquid crystal display or an organic EL display. The interface 24 outputs the image supplied from the processor 21 to the display device 62 . The communication unit 25 has functions similar to those of the communication unit 15 of the inspection apparatus 100 .

Next, functions of the inspection apparatus 100 will be described with reference to FIG. Inspection device 100 includes data acquisition unit 110 , classification unit 120 , and inspection unit 130 .

When the data acquisition unit 110 receives the image data of the object to be inspected from the sensor 50, the data acquisition unit 110 stores the image data in a predetermined area. The data acquisition unit 110 transmits the storage destination path of the image data to the classification unit 120 . The function of the data acquisition unit 110 is implemented by the processor 11 executing a program for processing related to data acquisition.

The classification unit 120 uses a learned machine learning model to determine which of the two types of parts the inspection target corresponds to. A machine learning model is generated by arbitrary machine learning. Examples of machine learning methods include random forests and SVMs (support vector machines). The machine learning model used by the classification unit 120 is generated by machine learning for predicting which of the parts A1 and A2 the inspection target corresponds to. This machine learning is performed using a large number of image data obtained by photographing parts A1 and A2 as teacher data.

The reason why many image data are used as training data is as follows. In the manufacturing line, the sensor 50 photographs the parts conveyed by the conveyor. The position where the sensor 50 is installed is fixed. On the other hand, the positions where the parts are arranged on the conveyor are not always the same. For example, when comparing a plurality of images showing parts of the same type, the positions and tilts of the parts in the respective images may differ. Therefore, image data used as teacher data is preferably image data having various variations. For example, an image captured while gradually shifting the placement position of the component, or an image captured while gradually rotating the component may be included. By using image data with various variations as training data, the machine learning model can correctly determine the type of part to be inspected.

In this embodiment, it is assumed that the prediction accuracy of the machine learning model used in the classification unit 120 satisfies a predetermined level. For example, it is a machine learning model that has been used for a short period of time after being generated and whose index value for evaluating prediction accuracy satisfies a predetermined criterion.

The classification unit 120 uses the image data of the inspection object as input data and uses a learned machine learning model to predict the probability that the inspection object may correspond to each of the parts A1 and A2. Hereinafter, this probability may be referred to as prediction probability.

Further, the classification unit 120 determines which of the part A1 and the part A2 the inspection target is from the predicted probability. For example, the classification unit 120 predicts that the probability that the inspection target corresponds to the part A1 is 70% and the probability that the inspection target corresponds to the part A2 is 30%. In this case, the classification unit 120 determines that the inspection target is the part A1. Further, for example, the classification unit 120 predicts that the probability that the inspection target corresponds to the part A1 is 15% and the probability that the inspection target corresponds to the part A2 is 85%. In this case, the classification unit 120 determines that the inspection target is the part A2. The classification unit 120 notifies the evaluation device 200 of the prediction and determination results.

The classification unit 120 also outputs to the inspection unit 130 the result of determination as to which part the inspection target corresponds to and the path of the storage destination of the image data. The function of the classification unit 120 is implemented by the processor 11 by executing application software in which a trained machine learning model is implemented, for example. The classification unit 120 is also called an inspection unit.

The inspection unit 130 uses the learned machine learning model to perform a visual inspection on the inspection target. Appearance inspection is an inspection for checking appearance defects such as foreign matter adhering to the surface of parts and deformation of parts.

A machine learning model used by the inspection unit 130 is generated by machine learning for predicting whether an inspection target is a non-defective product or a defective product. This machine learning is performed using the image data of parts associated with the judgment value of good or bad as teacher data. The function of the inspection unit 130 is realized by the processor 11 by executing application software in which a learned machine learning model is implemented, for example. Note that the machine learning model used by the inspection unit 130 and the machine learning model used by the classification unit 120 are different.

The inspection unit 130 receives the image data and the result of the determination by the classification unit 120 as to which part the inspection object corresponds to as input data, and determines that the parts shown in the image data correspond to good products and defective products. Predict probability. In this embodiment, the inspection unit 130 determines that the inspection target is a non-defective product when the probability of the non-defective product is higher than the probability of the defective product. The inspection unit 130 determines that the inspection target is a defective product when the probability of the product being defective is higher than the probability of being a non-defective product.

For example, the inspection unit 130 predicts that the part A1 to be inspected has an 80% probability of being a non-defective product and a 20% probability of being a defective product. In this case, the inspection unit 130 determines that the inspection target is a non-defective product. The inspection unit 130 outputs to the display device 61 an image that displays the probabilities that the inspection target corresponds to a non-defective product and a defective product, and the determination result that the product is a non-defective product or a defective product.

In this embodiment, it is assumed that the prediction accuracy of the machine learning model used in the inspection unit 130 satisfies a predetermined level. In this embodiment, the evaluation device 200 does not evaluate the determination result of the inspection section 130 . Therefore, the inspection unit 130 does not output the determination result to the evaluation device 200 .

Next, functions of the evaluation device 200 will be described. The evaluation device 200 includes an accumulation section 210 and an evaluation section 220 . In this embodiment, an example in which the evaluation device 200 evaluates the prediction result by the classification unit 120 of the inspection device 100 will be described.

FIG. 4 is a diagram showing an example of data accumulated in the accumulation unit 210. As shown in FIG. The accumulation unit 210 accumulates data supplied from the classification unit 120 . The data supplied from the classification unit 120 to the storage unit 210 includes the date and time, the determination result as to which part the inspection target corresponds to, the predicted probability that the inspection target corresponds to each part, and a path to save the image data. The function of the storage unit 210 is realized by the processor 11 executing software for storing and reading data upon request, as typified by a database management system.

The evaluation unit 220 shown in FIG. 1 determines that the data input to the machine learning model used in the classification unit 120 during the specified evaluation target period is within the range assumed during learning of the machine learning model. Evaluate whether or not

In this embodiment, the evaluation unit 220 detects an abnormality by detecting whether an outlier is included in the time-series data of prediction probability. For example, the evaluation unit 220 detects outliers included in the time-series data of predicted probabilities using existing methods for outlier detection.

When the evaluation unit 220 detects that an outlier is included in the prediction probability time series data, the evaluation unit 220 determines that the data input to the machine learning model is not within the range assumed during learning of the machine learning model. do. As described above, in this embodiment, the prediction accuracy of the machine learning model satisfies a predetermined level. That is, the machine learning model has not deteriorated. Also, the machine learning model outputs result data about prediction when input data is input. Therefore, when an abnormality is detected in the trend of time-series change in the output result data, it is considered that the input data is not within the expected range. The function of the evaluation unit 220 is implemented by the processor 21 executing a program for evaluation processing.

According to the above aspect, the evaluation device 200, when it is determined that the inspection target belongs to one of the classifications, uses the input data as the machine learning It is possible to evaluate whether or not it is within the range assumed during model learning.

FIG. 5 is a flowchart of classification processing in which the inspection apparatus 100 classifies components from image data. Each time the inspection apparatus 100 receives image data from the sensor 50, it performs the following classification processing.

In step S<b>101 , the data acquisition unit 110 receives image data from the sensor 50 . The data acquisition unit 110 performs preprocessing such as noise removal and binarization on the received image data. The data acquisition unit 110 stores the preprocessed image data in a predetermined area within the auxiliary storage unit 13 of the inspection apparatus 100 . The data acquisition unit 110 transmits the storage destination path of the preprocessed image data to the classification unit 120 .

In step S<b>102 , first, the classification unit 120 reads image data stored in the path notified from the data acquisition unit 110 . Using image data as input data, the classification unit 120 predicts the prediction probabilities that a component to be inspected in the image corresponds to each of two types of components. In step S<b>103 , the classification unit 120 determines which part is the inspection target from the predicted probability. Specifically, the classification unit 120 determines that the inspection target corresponds to the part with the higher prediction probability value.

In step S104, the classification unit 120 supplies the evaluation device 200 with the predicted probability that the inspection target corresponds to each part, the determination result as to which part the inspection target corresponds to, and the storage destination of the image data. Send data including path and date and time information. This data is also called prediction result data. The date and time indicate when the classification unit 120 made the prediction. Date and time are also called time information. The above is a series of flow of classification processing.

FIG. 6 is a flowchart of evaluation processing by the evaluation device 200. As shown in FIG. Each time the evaluation device 200 receives the prediction result data from the inspection device 100, it performs the following evaluation process.

In step S111, the accumulation unit 210 receives from the classification unit 120 the prediction probability that the inspection object corresponds to each part, the judgment result as to which part the inspection object corresponds to, and the storage destination of the image data. , and the date and time. In step S<b>112 , storage unit 210 stores the data received from classification unit 120 .

In step S113, the evaluation unit 220 executes the process of step S114 when it is time to evaluate the input data (step S113; YES). On the other hand, in step S113, when it is not the timing to evaluate the input data (step S113; YES), the accumulation unit 210 executes the process of step S211 again.

In step S<b>114 , the evaluation unit 220 detects an outlier in the prediction probability time series data. First, the evaluation unit 220 sorts the predicted probability values of the inspection objects predicted to correspond to the part A1 from the storage unit 210 in the past week by date and time. to read out. Subsequently, the evaluation unit 220 sorts the predicted probability values of the inspection objects predicted to correspond to the part A2 from the storage unit 210 in the past week by date and time. Read in state. The evaluation unit 220 determines whether or not an outlier is included in the time-series data of the prediction probability for the inspection object predicted to correspond to the part A1 in the past one week. Furthermore, the evaluation unit 220 determines whether or not an outlier is included in the time-series data of the prediction probability for the inspection object predicted to correspond to the part A2 in the past one week. When the evaluation unit 220 detects that an outlier is included in the prediction probability time-series data, the evaluation unit 220 determines that the input data of the machine learning model is not within the range assumed during learning of the machine learning model.

In step S115, the evaluation unit 220 presents the evaluation result of the input data based on the outlier detection result. For example, the evaluation unit 220 determines that there is input data that is not within the expected range, the predicted probability value detected as an outlier, and the date and time when the predicted probability value was predicted. and an image displaying the path indicating the storage destination of the input data is output to the display device 62 . This allows the user to know that data outside the range assumed during learning of the machine learning model has been input to the machine learning model. The above is a series of flow of evaluation processing.

As described above, in the first embodiment, from the tendency of the time-series change in the prediction probability that the inspection object is predicted to fall under the classification, the input data is in the range assumed during learning of the machine learning model. Evaluate whether or not there is

In this way, it is evaluated whether or not the input data is within the range assumed at the time of learning of the machine learning model, based on the trend of time-series changes in the prediction results predicted by the machine learning model. As a result, it is possible to consider not only the prediction accuracy of the machine learning model but also other conditions such as whether or not the input data is within an assumed range when determining whether or not the machine learning model is degraded. Therefore, even if the machine learning model is not actually degraded, it is possible to prevent erroneous judgment that the machine learning model is degraded when the prediction accuracy of the machine learning model is degraded due to other conditions. be able to.

B. Second Embodiment FIG. 7 is a diagram showing the functional configuration of an inspection system 1001 according to a second embodiment. The following description focuses on the configuration different from that of the first embodiment. The inspection system 1001 includes an inspection device 100 and an evaluation device 200 .

The inspection device 100 carries out an abnormal noise inspection. An object to be inspected by the inspection apparatus 100 is, for example, an automobile engine. In this embodiment, sensor 50 is a microphone. Sensor 50 is connected to inspection apparatus 100 via a network. The sensor 50 collects the sound of the vehicle's engine prior to shipment at a determined location on the manufacturing line. In this embodiment, voice data representing the operating state of the engine to be inspected serves as input data for the abnormal noise inspection. The sensor 50 transmits collected audio data to the inspection device 100 .

The inspection device 100 includes a data acquisition section 110 and an inspection section 130 . Note that the inspection apparatus 100 does not include the classification unit 120 in the second embodiment.

When the data acquisition unit 110 receives the audio data of the engine sound from the sensor 50, the data acquisition unit 110 stores the audio data in a determined area. The data acquisition unit 110 transmits the path of the storage destination of the voice data to the inspection unit 130 .

The inspection unit 130 uses a learned machine learning model to predict whether the inspection target is normal. The machine learning model used by the inspection unit 130 is generated by machine learning for predicting whether the inspection object is normal. This machine learning uses a large amount of sound data obtained by collecting normal engine sounds as teacher data. In a manufacturing site, many machines are operating, and sounds emitted from these machines are also included in the sound data to be inspected. Therefore, voice data used as teacher data is preferably voice data with various variations. For example, sound data obtained by collecting normal engine sounds in an environment where ambient noise is blocked, and sound data obtained by collecting normal engine sounds in an environment with ambient noise may be included. By using voice data with various variations as training data, the machine learning model can correctly determine the quality of the inspection object.

The inspection unit 130 uses the voice data supplied from the data acquisition unit 110 as input data to predict the degree of abnormality, which is an index indicating how far the engine sound to be inspected differs from the normal engine sound. In the present embodiment, the degree of abnormality is an index indicating that the higher the numerical value, the more likely the object to be examined is in an abnormal state. Furthermore, the inspection unit 130 determines whether or not the inspection target is normal. The inspection unit 130 determines that the object to be inspected is normal when the obtained degree of abnormality is equal to or less than a preset threshold. On the other hand, the inspection unit 130 determines that the object to be inspected is abnormal when the obtained degree of abnormality exceeds a preset threshold value.

Furthermore, the inspection unit 130 counts the number of inspections performed during the set period and the number of inspection targets determined to be normal during that period. The set period is, for example, one day.

The inspection unit 130 calculates, at a predetermined timing, the ratio of the number of inspection targets determined to be normal to the number of inspections performed during a set period. For example, after all the inspections for a day are completed, the inspection unit 130 calculates the ratio of the number of inspection objects determined to be normal to the total number of inspection objects inspected on that day. . Hereinafter, this ratio may be referred to as the normal determination ratio. The classification unit 120 notifies the evaluation device 200 of the calculated ratio. The function of the inspection unit 130 is realized by the processor 11 by executing application software in which a learned machine learning model is implemented, for example.

The evaluation device 200 includes an accumulation section 210 and an evaluation section 220 .

FIG. 8 is a diagram showing an example of data accumulated in the accumulation unit 210. As shown in FIG. The storage unit 210 stores data supplied from the inspection unit 130 . The data supplied from the inspection unit 130 to the storage unit 210 includes the date and the percentage determined to be normal.

The evaluation unit 220 shown in FIG. 7 evaluates whether or not the data input to the machine learning model used by the inspection unit 130 is within the range assumed during learning of the machine learning model. The evaluation unit 220 evaluates the data input to the machine learning model used by the inspection unit 130 during the specified evaluation target period. The evaluation target period is, for example, the past month.

First, the evaluation unit 220 reads, from the storage unit 210, the values of the percentage of normal determinations in the evaluation target period, sorted by date. Therefore, the data read from the storage unit 210 is time-series data of the percentage of normal determinations.

Next, the evaluation unit 220 detects an abnormality in the trend of time-series change in the percentage of normal determinations based on the time-series data of the percentage of normal determinations.

In this embodiment, the evaluation unit 220 detects an abnormality based on the presence or absence of outliers. For example, the evaluation unit 220 uses an existing method for outlier detection to determine whether or not the time-series data of the percentage of normal determinations includes an outlier in the past month.

When the evaluation unit 220 detects that the time-series data of the percentage of normal determinations includes an outlier, the input data of the machine learning model must be within the range assumed during learning of the machine learning model. judge.

According to the above aspect, the evaluation device 200 uses the input data based on machine learning from the trend of time-series changes in the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested. It is possible to evaluate whether or not it is within the range assumed during model learning.

FIG. 9 is a flow chart of inspection processing in which the inspection apparatus 100 inspects an inspection object from image data. Each time the inspection apparatus 100 receives audio data from the sensor 50, it performs the following inspection process.

In step S<b>201 , the data acquisition unit 110 of the inspection device 100 receives voice data from the sensor 50 . The data acquisition unit 110 performs preprocessing on the acquired audio data. The data acquisition unit 110 stores the preprocessed image data in a predetermined area within the auxiliary storage unit 13 of the inspection apparatus 100 . The data acquisition unit 110 transmits the storage destination path of the preprocessed image data to the inspection unit 130 .

In step S202, the inspection unit 130 determines whether the inspection target is normal. Specifically, first, the inspection unit 130 predicts the degree of abnormality for the inspection target using the audio data supplied from the data acquisition unit 110 as input data. Subsequently, the inspection unit 130 determines whether or not the inspection object is normal based on the value of the degree of abnormality. The inspection unit 130 outputs to the display device 61 an image displaying the result of determination as to whether the object to be inspected is normal or abnormal, and the value indicating the degree of abnormality. Furthermore, the inspection unit 130 counts the number of inspections performed during the set period and the number of inspection targets determined to be normal during that period.

In step S203, the inspection unit 130 executes the process of step S204 when it is time to calculate the percentage of the inspection targets determined to be normal (step S203; YES). On the other hand, in step S203, if it is not the timing to calculate the ratio of normal determination (step S203; NO), the inspection unit 130 executes the process of step S201 again.

In step S204, the inspection unit 130 calculates the ratio of the number of inspection objects determined to be normal to the total number of inspection objects inspected during the set period. In step S<b>205 , the inspection unit 130 transmits to the evaluation device 200 data including the calculated ratio of determinations as normal and the date. This data is also called prediction result data. The date indicates when the inspection unit 130 made a prediction as to whether or not it is normal. Date is also called time information.

FIG. 10 is a flowchart of evaluation processing by the evaluation device 200. As shown in FIG. The evaluation device 200 executes the following evaluation process each time it receives prediction result data from the inspection device 100 .

In step S<b>211 , the accumulation unit 210 receives from the inspection unit 130 data including the percentage of the inspection targets determined to be normal and the date. In step S212, the storage unit 210 stores the received data.

In step S213, the evaluation unit 220 executes the process of step S214 when it is time to evaluate the input data (step S213; YES). On the other hand, in step S213, when it is not the timing to evaluate the input data (step S213; NO), the accumulation unit 210 executes the process of step S211 again.

In step S<b>214 , the evaluation unit 220 detects an outlier in the time-series data of the percentage of normal determinations. First, the evaluation unit 220 reads from the accumulation unit 210 the value of the ratio of normal determinations during the past month. The evaluation unit 220 determines whether or not the read time-series data includes an outlier.

In step S215, the evaluation unit 220 presents the evaluation result of the input data based on the outlier detection result. When the evaluation unit 220 detects that the time-series data of the percentage of normal judgments includes an outlier, the evaluation unit 220 detects that there is input data that is not within the assumed range, and the normal judgment detected as the outlier. and the inspection date are output to the display device 62 . This allows the user to know that data outside the range assumed during learning of the machine learning model has been input to the machine learning model. The above is a series of flow of evaluation processing.

In the second embodiment, when performing an abnormal sound test using voice data, from the tendency of time-series changes in the prediction results predicted by the machine learning model, the range of input data assumed during learning of the machine learning model It is possible to evaluate whether or not there is

In this way, it is evaluated whether or not the input data is within the range assumed at the time of learning of the machine learning model, based on the trend of time-series changes in the prediction results predicted by the machine learning model. As a result, it is possible to consider not only the prediction accuracy of the machine learning model but also other conditions such as whether or not the input data is within an assumed range when determining whether or not the machine learning model is degraded. Therefore, even if the machine learning model is not actually degraded, it is possible to prevent erroneous judgment that the machine learning model is degraded when the prediction accuracy of the machine learning model is degraded due to other conditions. be able to.

C. Other Embodiments (C1) In the first embodiment, an example in which the evaluation device 200 evaluates the results of the classification processing of the classification section 120 of the inspection device 100 has been described. However, the evaluation device 200 may evaluate not only the results of the classification processing of the classification section 120 but also the results of the inspection processing of the inspection section 130 .

As described above, the inspection unit 130 uses the learned machine learning model to perform the visual inspection of the inspection target. For example, it is assumed that the inspection unit 130 predicts the probability that an object to be inspected is determined to be a non-defective product and the probability that an object to be inspected is determined to be a defective product. The inspection unit 130 determines that the inspection target is a non-defective product when the value of the probability of being determined as a non-defective product exceeds the threshold value. The inspection unit 130 determines that the inspection target is defective when the value of the probability that the product is determined to be defective exceeds the threshold. The inspection unit 130 transmits to the evaluation device 200 the value of the probability that the inspection target determined as a non-defective product is determined to be a non-defective product. Furthermore, the inspection unit 130 transmits to the evaluation device 200 the value of the probability that the inspection target determined as defective is determined to be defective. The evaluation device 200 uses the input data to perform machine learning based on the trend of change in the time-series data of the probability of being determined to be non-defective and the trend of change in the time-series data of the probability of being determined to be defective. It may be determined whether or not it is within the range assumed during model learning.

According to this aspect, during visual inspection using image data, the range of input data assumed during learning of the machine learning model from the tendency of time-series changes in prediction results predicted by the machine learning model It is possible to evaluate whether or not there is

(C2) In the first embodiment, the prediction result data that the classification unit 120 transmits to the evaluation device 200 includes the date and time when the classification unit 120 made the prediction. In the second embodiment, the prediction result data that the inspection unit 130 transmits to the evaluation device 200 includes the date indicating when the inspection unit 130 made the prediction. Thus, in the first embodiment and the second embodiment, the side that transmits the prediction result data acquires the time information. However, the time information may be acquired by the side that receives the prediction result data.

For example, the classification unit 120 in the inspection system 1000 does not have to include the date and time in the prediction result data to be transmitted to the evaluation device 200 . In this case, the classification unit 120 immediately transmits the prediction result data to the evaluation device 200 after determining which part the inspection target corresponds to. When receiving the prediction result data from the classification unit 120, the storage unit 210 acquires the date and time when the data was received. The accumulation unit 210 accumulates the data received from the classification unit 120 together with information indicating date and time.

Since the classification unit 120 immediately transmits the prediction result data to the evaluation device 200 after the determination, the date and time stored in the storage unit 210 are time information indicating when the prediction was made for the inspection object. can be regarded as Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time-series data of predicted probabilities.

The inspection unit 130 in the inspection system 1001 does not have to include the date and time in the prediction result data to be transmitted to the evaluation device 200 . In this case, the inspection unit 130 immediately transmits the prediction result data to the evaluation device 200 after calculating the ratio of the number of inspection objects determined to be normal to the number of inspections performed in the set period. do. When the prediction result data is received from the inspection unit 130, the accumulation unit 210 acquires the date when the data was received, and accumulates the received data together with information indicating the date.

Since the inspection unit 130 immediately transmits the prediction result data to the evaluation device 200 after calculating the ratio, the date stored in the storage unit 210 is time information indicating when the prediction was made for the inspection object. can be regarded as Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time-series data of the percentage of normal determinations.

Alternatively, neither the side that transmits the prediction result data nor the side that receives the prediction result data may acquire the time information. The accumulation unit 210 accumulates prediction result data in the order in which the prediction result data are received. The data read from the storage unit 210 by the evaluation unit 220 can be said to be time-series data of the percentage of normal determinations.

(C3) In the first embodiment, the classifying unit 120 has explained an example in which the inspection target determines which of the two types of components. The classification unit 120 may determine any one of three or more types of parts to be inspected. In this case, the machine learning model used by the classification unit 120 performs the above-described machine learning for predicting to which of a plurality of preset classifications the inspection target is classified.

(C4) In the first and second embodiments, the inspection systems 1000 and 1001 are configured to include the inspection device 100 and the evaluation device 200 . However, one computer may realize the functions of the inspection apparatus 100 and the evaluation apparatus 200 .

(C5) In the first embodiment, an example in which the data acquisition unit 110 transmits to the classification unit 120 the path of the storage destination of the preprocessed image data has been described. Alternatively, the data acquisition unit 110 may not perform preprocessing. In this case, the data acquisition section 110 may store the image data received from the sensor 50 in a predetermined area within the auxiliary storage section 13 of the inspection apparatus 100 . Alternatively, the data acquisition unit 110 may transmit preprocessed image data or non-preprocessed image data to the classification unit 120 .

In the second embodiment, an example has been described in which the data acquisition unit 110 transmits to the inspection unit 130 the path of the storage destination of preprocessed audio data. Alternatively, the data acquisition unit 110 may not perform preprocessing. In this case, the data acquisition unit 110 may store the audio data received from the sensor 50 in a predetermined area within the auxiliary storage unit 13 of the inspection device 100 . Alternatively, the data acquisition unit 110 may transmit preprocessed audio data or non-preprocessed audio data to the inspection unit 130 .

Computer-readable recording media including magnetic disks, optical disks, magneto-optical disks, flash memories, semiconductor memories, and magnetic tapes are used as recording media for recording programs that implement the functions of the inspection device 100 and evaluation device 200 described above. can do.

Moreover, means for realizing the functions of the inspection apparatus 100 and the evaluation apparatus 200 are not limited to software, and part or all of them may be realized by dedicated hardware. For example, a circuit represented by FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) may be used as dedicated hardware.

The present disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from the scope of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the Summary of the Invention column are used to solve some or all of the above problems, or to Substitutions and combinations may be made as appropriate to achieve some or all. Moreover, if the technical feature is not described as essential in this specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 11, 21... Processor 12, 22... Main memory part 13, 23... Auxiliary memory part 14, 24... Interface 15, 25... Communication part 19, 29... Internal bus 50... Sensor 61, 62... Display device 100 Inspection device 110 Data acquisition unit 120 Classification unit 130 Inspection unit 200 Evaluation device 210 Storage unit 220 Evaluation unit 1000, 1001 Inspection system A1, A2 parts

Claims (7)

An inspection system that performs inspection using a machine learning model,
The machine learning model is a learned machine learning model that has undergone machine learning to predict a predetermined event for the inspection target,
an inspection unit that repeats predicting the inspection target using the machine learning model using data about the inspection target as input data and outputting prediction result data indicating the prediction result;
an accumulation unit for accumulating the prediction result data output by the inspection unit;
Whether the input data is within the range assumed at the time of learning of the machine learning model based on the tendency of the time-series change in the result predicted for the inspection target indicated by the prediction result data accumulated in the accumulation unit an evaluation unit that evaluates whether or not and outputs an evaluation result;
inspection system. An inspection system according to claim 1,
The machine learning model performs the machine learning for predicting which of a plurality of preset classifications the inspection target is classified into,
The inspection unit
Predicting a predicted probability indicating the possibility that the test object will be classified into each of the plurality of classifications;
determining that the inspection target is classified into the classification with the highest predicted probability value;
outputting the prediction result data including the predicted prediction probability for the classification determined for the test object;
the accumulation unit accumulates the prediction result data output by the inspection unit together with time information indicating when the prediction is performed for the inspection target;
The evaluation unit analyzes the input data from the trend of time-series change of the prediction probability predicted for the classification determined for the inspection target indicated by the prediction result data accumulated in the accumulation unit. evaluates whether is in the range assumed when training the machine learning model,
inspection system. An inspection system according to claim 1,
The machine learning model performs the machine learning for predicting whether the test subject is within a normal range,
The inspection unit
predicting whether the test subject is within the normal range;
outputting the prediction result data including a value representing the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested in a set period;
the accumulation unit accumulates the prediction result data output by the inspection unit together with time information indicating when the prediction is performed for the inspection target;
The evaluation unit determines whether or not the input data is within the range assumed at the time of learning of the machine learning model, based on the tendency of the time-series change in the ratio indicated by the prediction result data accumulated in the accumulation unit. evaluate whether
inspection system. An inspection system according to any one of claims 1 to 3,
The input data is image data representing the appearance of the inspection object acquired by a sensor,
inspection system. An inspection system according to any one of claims 1 to 3,
The input data is voice data representing the operating state of the inspection object acquired by a sensor,
inspection system. A computer-implemented method for testing using a machine learning model, comprising:
The machine learning model is a learned machine learning model that has undergone machine learning to predict a predetermined event for the inspection target,
the computer
a step of predicting the inspection target using the machine learning model using data about the inspection target as input data, and outputting prediction result data indicating the prediction result;
accumulating the prediction result data;
Evaluating whether or not the input data is within the range expected at the time of learning of the machine learning model, based on the trend of time-series changes in the results predicted for the inspection object indicated by the accumulated prediction result data. , a step of outputting an evaluation result;
how to run. A program executed by a computer,
The computer performs inspection using a machine learning model,
The machine learning model is a learned machine learning model that has undergone machine learning to predict a predetermined event for the inspection target,
to the computer;
A function of predicting the inspection target using the machine learning model using data about the inspection target as input data, and outputting prediction result data indicating the prediction result;
a function of accumulating the prediction result data;
Evaluating whether or not the input data is within the range expected at the time of learning of the machine learning model, based on the trend of time-series changes in the results predicted for the inspection object indicated by the accumulated prediction result data. , a function to output evaluation results,
program to make it happen.

Images