JP2022175146A - inference device - Google Patents

Abstract

To reduce the processing time from object detection to abnormality detection.SOLUTION: An inference device comprises a first processor which performs inference using a first object detection model, a second processor which performs inference using a second object detection model having higher detection accuracy and more computational complexity than those of the first object detection model, and a third processor which performs inference of abnormality detection of a target object. The third processor selects an abnormality detection model in accordance with a result of inference performed by using the first object detection model and performs inference of abnormality detection in the case of reliability equal to or higher than a reference value, but in the case of reliability lower than the reference value, waits until inference using the second object detection model is performed. The second processor performs inference of detection of the target object in parallel with the third processor, and the third processor performs inference of abnormality detection by using the loaded model in the case of coincidence between results of inference performed by using the two object detection models, but reloads another model in accordance with the result of inference performed by using the second object detection model to perform inference of abnormality detection in the case of non-coincidence between them.SELECTED DRAWING: Figure 4

Description

The present invention relates to an inference device that performs object detection and anomaly detection.

Conventionally, in order to perform inspections of maintenance targets such as electric power equipment at a low cost, there have been attempts to photograph equipment including maintenance targets from an on-board camera, detect objects subject to maintenance, and detect abnormalities appearing in the appearance of maintenance targets. is done.

In Patent Document 1, which is an example of conventional technology, it is possible to efficiently perform patrol inspection work for a large number of utility poles or power distribution facilities, check the patrol inspection results even after the fact, and even an inexperienced worker can perform patrol inspection work. For the purpose of providing a patrol inspection system that can be performed more reliably, the information terminal 3 uses the image of the 360-degree camera 35 and the GPS function to identify the utility pole number of the utility pole to be passed during the patrol. A patrol inspection system is disclosed that automatically identifies equipment to be inspected installed on utility poles from the image and determines whether or not there is an abnormality based on the detection result of an abnormality detection sensor 36 .
In the patrol inspection system, when it is determined that there is an abnormality, the collected and input inspection results for the facility of the utility pole are stored in the storage unit 34, and when it is determined that there is no abnormality, that When the vehicle passes through the utility pole, the inspection result indicating no abnormality is automatically stored in the storage unit 34, and upon receiving an instruction to end the patrol, the inspection result stored in the storage unit 34 and the image captured by the 360-degree camera 35 are stored. The position information at the time of shooting and the associated image are transmitted to the management server 2 .

JP 2019-32684 A

A trained model obtained by machine learning or deep learning is used to detect objects to be maintained and abnormalities appearing in the appearance of the maintenance target from the images obtained by photographing equipment including maintenance targets. It is conceivable to make inferences using
More specifically, in order to identify the area of the object in the input image and the type of equipment (hereinafter referred to as class), "inference by the object detection model" is performed, and depending on the identified class, normal or Perform "inference by anomaly detection model" to judge anomalies.

However, when the number of devices used for inference is small compared to the number of trained models used for inference, it is necessary to load the trained model multiple times for one device, and the trained model is used for inference. There was a problem that the processing speed of the entire system decreased due to the processing time required to load the device.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for shortening the processing time from object detection to abnormality detection.

One aspect of the present invention that solves the above-described problems and achieves the object is an inference device that executes inference for detecting a target object from an image and detecting an anomaly of the target object, the inference device using a first object detection model A first processor that executes inference of detection of the target object; 2 processors, and a third processor that executes inference of anomaly detection of the target object by a plurality of anomaly detection models, wherein the third processor performs a plurality of select an anomaly detection model in descending order of reliability from the anomaly detection models, and if the reliability of the selected model is equal to or higher than a predetermined reference value, execute inference of anomaly detection using the anomaly detection model, If the reliability of the selected model is less than a predetermined reference value, after loading the anomaly detection model, wait for execution of inference by the second object detection model; Inference of detection of the target object from the image is performed in parallel with the selection and inference, and the third processor determines that the inference result of the second object detection model is the inference result of the first object detection model. If the inference result of the second object detection model does not match the inference result of the first object detection model, the inference result of the anomaly detection is executed using the loaded anomaly detection model. is an inference device that selects and reloads an anomaly detection model according to an inference result of the second object detection model, and executes anomaly detection inference using the reloaded anomaly detection model.

In the inference device having the above configuration, the third processor includes a plurality of processors, and each of the plurality of third processors can perform processing in parallel.

According to the present invention, the processing time from object detection to abnormality detection can be shortened.

FIG. 1 is a block diagram showing a schematic configuration of an information processing device used for inference in an embodiment. FIG. 2 is a diagram illustrating an example of a trained model used in an information processing apparatus; FIG. 3 is a diagram illustrating an operation flow of an information processing apparatus in a comparative example; FIG. 4 is a diagram illustrating an operation flow of the information processing apparatus according to the embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings.
However, the present invention is not to be construed as limited by the description of the following embodiments.

<Embodiment>
FIG. 1 is a block diagram showing a schematic configuration of an information processing device 100 used for inference in this embodiment.
The information processing apparatus 100 shown in FIG. , provided.

The main processor 10 includes a built-in graphics processor 11 and scheduler 12 .
The bridge controller 20 is provided between the main processor 10 and the sub-processor group 30 and controls connections between the main processor 10 and the sub-processor group 30 .
Sub-processor group 30 includes sub-processor 31 , sub-processor 32 , and sub-processor 33 .
Input device 40 includes camera 41 and sensor 42 .

The main processor 10, the graphic processor 11, the sub-processor 31, the sub-processor 32, and the sub-processor 33 are processors for inference, and the number of processors provided in the information processing apparatus 100 is limited to these. is not.
Also, the input device 40 is not limited to a camera and a sensor as long as it is configured to output image data to the main processor 10 .

FIG. 2 is a diagram showing an example of a trained model used in the information processing apparatus 100. As shown in FIG.
As shown in FIG. 2, the trained models used in the information processing apparatus 100 include an object detection model A, an object detection model B, an anomaly detection model C, an anomaly detection model D, an anomaly detection model E, an anomaly detection A model F, an anomaly detection model G, and an anomaly detection model H are included.
The object detection model A has high detection accuracy but requires a large amount of calculation, and the object detection model B has medium detection accuracy but requires a small amount of calculation.
A plurality of anomaly detection models C to H are provided for each class detected by object detection model A and object detection model B, and there is no difference in detection accuracy or amount of calculation between them.

FIG. 3 is a diagram showing an operational flow of the information processing apparatus 100 in the comparative example.
Here, the object detection model A is preloaded and held in the graphic processor 11, and the sub-processors 31, 32, and 33 store a plurality of anomaly detection models C~ H is assigned.
Note that loading is a process of developing computational parameters of a trained model on the memory of a processor that performs inference, and the time required for loading increases with the computational complexity of the model.
That is, the smaller the amount of computation of the model, the shorter the time required for loading, and the larger the amount of computation of the model, the longer the time required for loading.

When the processing is started, the graphic processor 11 executes inference by the object detection model A (S1).
The scheduler 12 assigns anomaly detection models to the sub-processors 31, 32, and 33 according to the classes detected in S1 (S2).

The sub-processor 31 loads the anomaly detection model C (S3), executes inference by the anomaly detection model C (S4), loads the anomaly detection model F (S5), and Inference by F is executed (S6).
The sub-processor 32 loads the anomaly detection model D (S7), executes inference by the anomaly detection model D (S8), loads the anomaly detection model G (S9), and Inference by G is executed (S10).
The sub-processor 33 loads the anomaly detection model E according to the assignment of S2 (S11), executes inference by the anomaly detection model E (S12), and makes inference by the anomaly detection model E again without changing the model. (S13).
The processing of S3 to S6 by the sub processor 31, the processing of S7 to S10 by the sub processor 32, and the processing of S11 to S13 by the sub processor 33 are performed in parallel.
When all the processes of S3 to S13 are completed, the inference results of S3 to S13 are output to an output device (not shown) (S14), and the process is terminated.

According to the flow shown in FIG. 3, although highly accurate inference results using object detection model A are obtained and assignment of anomaly detection models is performed based on the highly accurate inference results, during inference processing by object detection model A , the sub-processors 31, 32, and 33 cannot perform inference processing because an anomaly detection model is not assigned.
Further, even if one of the anomaly detection models C to H is preloaded in the sub-processors 31, 32, and 33, depending on the result of the inference processing by the object detection model A, one or more of the necessary anomaly detection models may be loaded into the sub-processors. It is also assumed that the processors 31, 32, and 33 are not loaded frequently, and the calculation time in the entire information processing apparatus 100 varies.

If the number of types of detection targets is small, all anomaly detection models that are assumed to be used can be loaded into the sub-processors 31, 32, and 33, but the types of targets that can be detected are limited. end up
In addition, when the types of objects to be detected are not small, loading all the anomaly detection models that are assumed to be used into the sub-processors 31, 32, 33 requires a huge storage capacity. It is difficult because

FIG. 4 is a diagram showing an operation flow of the information processing apparatus 100 according to this embodiment.
In FIG. 4, the main processor 10 preloads and holds an object detection model B, which is a model with medium detection accuracy but a small amount of computation, and the graphic processor 11 has a high detection accuracy but a small amount of computation. An object detection model A, which is a model with a large , is preloaded and held.
One of the plurality of abnormality detection models C to H is assigned to each of the sub-processors 31, 32, 33 according to the result of inference by the object detection model A, as in FIG.

First, when the process is started, the main processor 10 executes inference by the object detection model B (S101).
Here, it is assumed that classes C, D, E, and F have been detected by inference of object detection model B. Class E has the next highest reliability, and Class F has the lowest reliability.
Next, the scheduler 12 assigns anomaly detection models to the sub-processors 31, 32, and 33 in descending order of reliability according to the class detected in the inference of S101 (S102). .
The graphic processor 11 executes inference based on the object detection model A (S103), and the scheduler 12 detects anomalies with the sub-processors 31, 32, and 33 according to the class detected in the inference of S103. A model is assigned (S104).
As shown in FIG. 4, the process of S102 and the processes of S103 and S104 are performed in parallel.
Execution of inference by the object detection models A and B obtains the region of the detection target object in the image, the class as the type of facility, and the confidence level (%) of the class.
Note that reliability is also called accuracy.
If the reliability (%) of the inference (S101) by the object detection model B performed by the main processor 10 is high and equal to or greater than a predetermined reference value α, the result of the inference (S103) by the object detection model A is waited for. proceed without
In FIG. 4, only the reliability of class C is equal to or higher than the predetermined reference value α.

The sub-processor 31 loads the anomaly detection model C of class C according to the assignment in S102 (S105), and executes inference by the anomaly detection model C (S106).
Here, since the reliability of class C is equal to or greater than the predetermined reference value α, the process continues without waiting for the assignment in S104.
Next, the sub-processor 31 loads the anomaly detection model F of class F (S107).
Here, it is assumed that the assignment of S102 and the assignment of S104 for class F match, and the sub-processor 31 executes inference by the loaded anomaly detection model F (S108).
Note that when the assignment of S102 and the assignment of S104 for class F do not match, the sub-processor 31 reloads the anomaly detection model of another class according to the assignment of S104, and the anomaly detection model perform inference by

The sub-processor 32 loads the anomaly detection model D of class D in accordance with the assignment in S102 (S109).
Here, since the reliability of class D is less than the predetermined reference value α, the assignment of S104 is awaited.
Here, it is assumed that the assignment in S102 and the assignment in S104 are the same for class D, and the sub-processor 32 executes inference by the loaded anomaly detection model D (S110).
Note that when the assignment of S102 and the assignment of S104 for class D do not match, the sub-processor 32 reloads the anomaly detection model of another class according to the assignment of S104, and the anomaly detection model perform inference by
Next, the sub-processor 32 loads the anomaly detection model G of the class G according to the assignment in S104 (S111), and executes inference by the anomaly detection model G (S112).

The sub-processor 33 loads the anomaly detection model E of class E according to the assignment in S102 (S113).
Here, since the reliability of class E is less than the predetermined reference value α, the assignment of S104 is awaited.
Here, it is assumed that the assignment of S102 and the assignment of S104 for class E match, and the sub-processor 33 executes inference by the loaded anomaly detection model E (S114).
Note that if the assignment of S102 and the assignment of S104 for class E do not match, the sub-processor 33 reloads the anomaly detection model of another class according to the assignment of S104, and perform inference by

The processing of S105 to S108 by the sub processor 31, the processing of S109 to S112 by the sub processor 32, and the processing of S113 to S114 by the sub processor 33 are performed in parallel.
Then, when all the processes of S105 to S114 are completed, the inference results of S105 to S114 are output to an output device (not shown) (S115).

As described above, according to the comparative example, all the anomaly detection models are loaded according to the allocation after execution of the inference by the object detection model A. Since the anomaly detection model D and the anomaly detection model F are loaded, the processing time from object detection to anomaly detection can be shortened by the time required for loading the anomaly detection model D or the anomaly detection model F. .
Therefore, according to this embodiment, it is possible to shorten the processing time from object detection to abnormality detection.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications in which components are added, deleted, or converted to the above-described configuration.

10 main processor 11 graphic processor 12 scheduler 20 bridge controller 30 sub-processor group 31, 32, 33 sub-processor 40 input device 41 camera 42 sensor 100 information processing device

Claims (2)

An inference device for detecting a target object from an image and inferring anomaly detection of the target object,
a first processor that performs inference of detection of the target object by a first object detection model;
a second processor for inferring detection of the target object by a second object detection model having higher detection accuracy and a larger amount of computation than the first object detection model;
a third processor that performs inference of anomaly detection of the target object by a plurality of anomaly detection models;
The third processor selects an anomaly detection model from a plurality of anomaly detection models in descending order of reliability according to the inference result of the first object detection model, and the reliability of the selected model is a predetermined reference value. In the case above, inference of anomaly detection is executed using the anomaly detection model, and when the reliability of the selected model is less than a predetermined reference value, after loading the anomaly detection model, the second Wait for the object detection model to perform inference,
the second processor performs inference of detection of the target object from the image in parallel with the selection and inference of the third processor;
The third processor executes inference of anomaly detection using the loaded anomaly detection model when the inference result of the second object detection model matches the inference result of the first object detection model. , if the result of inference by the second object detection model does not match the result of inference by the first object detection model, the anomaly detection model corresponding to the result of inference by the second object detection model is selected and re-selected. An inference device that loads and performs anomaly detection inference using the reloaded anomaly detection model. the third processor comprises a plurality of processors;
2. The reasoning apparatus according to claim 1, wherein each of said plurality of third processors performs processing in parallel.

Images