JPWO2020031422A1 - Object detection method, object detection device and computer program - Google Patents

Abstract

This is an object detection method that detects objects included in a captured image. It detects the type of object included in the captured image, detects the object contained in the captured image in pixel units, and detects the object type and pixel unit. The size and type of the object are determined on a pixel-by-pixel basis based on the pixels of the object.

Description

The present disclosure relates to object detection methods, object detection devices and computer programs.
This application claims priority based on Japanese Application No. 2018-147979 filed on August 6, 2018, and incorporates all the contents described in the Japanese application.

By imaging industrially produced parts and deep machine learning the neural network using the captured images of good and defective products obtained by imaging as teacher data, the quality of the parts is judged using the trained neural network. be able to.

On the other hand, there is a technique for detecting the position and range of an object included in a captured image and the type of the object by using an image recognition technique using Tensorflow (registered trademark). For example, it is possible to detect an image portion of a person, an animal, a vehicle, or the like included in a captured image.
Further, Non-Patent Document 1 discloses a technique for self-learning a feature amount of data by using an autoencoder.

G. E. Hinton and R. Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks”, Science, vol. 313, p. 504-507

The object detection method of the present disclosure is an object detection method for detecting an object included in a captured image, wherein the object is detected by a first method for detecting the type of the object included in the captured image, and the above-mentioned object detection method is performed. The object is detected by the second method of detecting the object included in the captured image on a pixel-by-pixel basis, the type of the object detected by the first method, and the detection by the second method. Based on the object, the size and type of the object are determined in pixel units, and a threshold value for making a predetermined determination regarding the object is selected based on the type of the object and determined in pixel units. The predetermined determination is made based on the object and the threshold selected based on the type.

The object detection device of the present disclosure is an object detection device that detects an object included in the captured image, and includes a first detection unit that detects the type of the object included in the captured image and the said object included in the captured image. The object is based on a second detection unit that detects an object on a pixel-by-pixel basis, the type of the object detected by the first detection unit, and the object detected on a pixel-by-pixel basis by the second detection unit. A determination unit that determines the size and type of the object on a pixel-by-pixel basis, a selection unit that selects a threshold value for making a predetermined determination on the object based on the type of the object, and the object determined on a pixel-by-pixel basis. A determination unit that makes the predetermined determination based on the threshold value selected based on the type.

The computer program of the present disclosure is a computer program for causing a computer to detect an object included in a captured image, and is described by the first method of detecting the type of the object included in the captured image by the computer. The type of the object detected by the second method of detecting the object and detecting the object included in the captured image on a pixel-by-pixel basis, and the second method. Based on the object detected by the method of, the size and type of the object are determined in pixel units, and a threshold value for making a predetermined determination regarding the object is selected based on the type of the object. The process of performing the predetermined determination is executed based on the object determined in pixel units and the threshold value selected based on the type.

The computer program of the present disclosure is a computer program for displaying information obtained by detecting an object included in a captured image on a computer, and causes the computer to detect the type of the object included in the captured image. The information obtained by detecting the object by the first method and the information obtained by detecting the object by the second method of detecting the object included in the captured image in pixel units. Execute the process to display the based information.

The present application can be realized as a semiconductor integrated circuit that realizes a part or all of the object detection device, or can be realized as another system including the object detection device.

FIG. 1 is a block diagram showing a hardware configuration of the incident detection device according to the first embodiment. FIG. 2 is a functional block diagram showing a configuration example of the incident detection device according to the first embodiment. FIG. 3 is a flowchart showing a processing procedure related to object detection. FIG. 4 is a flowchart showing the mapping process. FIG. 5A is an explanatory diagram showing an object detection method by the first detection unit. FIG. 5B is an explanatory diagram showing an object detection method by the first detection unit. FIG. 6 is an explanatory diagram showing an object detection method by the second detection unit. FIG. 7 is an explanatory diagram showing a method of determining the position and type of an object on a pixel-by-pixel basis. FIG. 8 is a flowchart showing a processing procedure related to the dimension measurement of the object. FIG. 9 is an explanatory view showing a method of measuring the dimensions of an object. FIG. 10 is a flowchart showing a procedure for determining the quality of an object. FIG. 11 is an explanatory diagram showing a method of determining the quality of an object. FIG. 12A is an explanatory diagram showing a method of detecting and measuring the dimensions of overlapping objects. FIG. 12B is an explanatory diagram showing a method of detecting and measuring the dimensions of overlapping objects. FIG. 12C is an explanatory diagram showing a method of detecting and measuring the dimensions of overlapping objects. FIG. 12D is an explanatory diagram showing a method of detecting and measuring the dimensions of overlapping objects. FIG. 13 is a block diagram showing a configuration example of the incident detection system according to the second embodiment. FIG. 14 is a flowchart showing a processing procedure related to the additional learning according to the second embodiment. FIG. 15 is a flowchart showing a processing procedure related to the generation of learning data. FIG. 16 is an explanatory diagram showing a method of generating learning data. FIG. 17A is an explanatory diagram showing a method of generating learning data. FIG. 17B is an explanatory diagram showing a method of generating learning data.

[Issues to be solved by this disclosure]
By the way, according to the image recognition technology using Tensorflow (registered trademark), the position and range of the object can be roughly specified by the bounding box surrounding the object, but the position and type of the object are detected on a pixel-by-pixel basis. There is a problem that it cannot be done.

For example, even if the type of an object can be specified, the dimensions of the object cannot be detected accurately. Even if a linear foreign object such as hair attached to a part is detected as an object, the bounding box surrounding the image of the foreign object probably includes a background image, and the size of the foreign object is determined by the size of the bounding box. Cannot be specified. If the size of the foreign matter cannot be specified, it cannot be determined whether or not it is an acceptable abnormality.

An object of the present disclosure is to provide an object detection method, an object detection device, and a computer program capable of detecting the size and type of an object included in a captured image on a pixel-by-pixel basis.

[Effect of the present disclosure]
According to the present disclosure, it is possible to provide an object detection method, an object detection device, and a computer program capable of detecting the size and type of an object included in a captured image on a pixel-by-pixel basis.

[Explanation of Embodiments of the present disclosure]
First, embodiments of the present disclosure will be listed and described. In addition, at least a part of the embodiments described below may be arbitrarily combined.

(1) The object detection method of the present disclosure is an object detection method for detecting an object included in a captured image, in which the position and range of the object included in the captured image are specified by a bounding box and the bounding is performed. The object is detected by the first method of detecting the type of the object surrounded by the box, and the object is detected by the second method of detecting the position of the object included in the captured image in pixel units. , The position and range of the object detected by the first method, the type of the object, and the position and type of the object based on the position of the pixel of the object detected by the second method. Is determined on a pixel-by-pixel basis.

In the present disclosure, objects are detected by two methods. The first method is a detection method in which the position and range of an object can be roughly specified, but the type of the object can be detected. The second method is a detection method in which the position of an object can be accurately detected on a pixel-by-pixel basis, but the type of the object cannot be specified.
The object detection method of the present disclosure includes the position and range of the object specified and detected by the first method, the data of the type of the object, and the pixel position of the object detected in pixel units by the second method. Based on the data, the position and type of the object are determined on a pixel-by-pixel basis.
Therefore, it is possible to detect the position and range of the object included in the captured image on a pixel-by-pixel basis.
The order in which the object detection process by the first method and the object detection process by the second method are executed is not limited, and may be executed in the reverse order or executed in parallel. You may.

(2) A threshold value for making a predetermined determination regarding the object is selected based on the type of the object, the position and type of the object determined in pixel units, and the threshold value selected based on the type. It is preferable to make the predetermined determination based on the above.

In the present disclosure, different threshold values can be selected depending on the type of object, and a predetermined determination can be made based on the position and type of the object detected in pixel units and the threshold value corresponding to the type.
For example, the captured image is an inspection target such as a manufactured part, and the object is an abnormality such as dust, hair, dents, scratches, etc. attached to the part. It is possible to judge whether an object is good or bad.

(3) It is preferable to make the predetermined determination by comparing the total number of pixels of one type of the object with the threshold value selected based on the type.

In the present disclosure, a predetermined determination is made by comparing the total number of pixels constituting one type of object with the threshold value corresponding to the one type.
For example, in the above example of the inspection object, the total number of pixels corresponds to the dimensions of dust, hair, dents, scratches, etc. attached to the part, and the inspection object is good, using different threshold values for each object type. Defects can be determined.

(4) The detection of the object by the second method is performed by a trained autoencoder that outputs the data of the feature extraction image from which the features of the input data are extracted when the data of the captured image is input. The configuration performed by using is preferable.

In the present disclosure, by using the trained autoencoder, the position of the object can be accurately detected on a pixel-by-pixel basis.

(5) In the detection of the object by the first method, when the data of the captured image is input, the trained image that outputs the position and range of the object included in the captured image and the type of the object. A configuration using a recognition neural network is preferable.

In the present disclosure, by using a trained image recognition neural network, it is possible to recognize an object having various characteristics, detect the position and range of the object, and detect the type of the object.

(6) When the detection result by the first method and the detection result by the second method do not match, the type of the object included in the captured image is determined by a third method different from the first method. When the data of the captured image is input to the image recognition neural network after detection, the imaging is performed so that the position and range of the object detected by the third method and the type of the object are output. It is preferable that the image recognition neural network is additionally trained by using the image and the detection result according to the third method.

In the present disclosure, if the object detection result by the first method described in the second method and the object detection result by the second method do not match, the detection of the object type fails, in particular. If so, the type of object is detected by the third method. Then, by additionally learning the image recognition neural network using the detection result by the third method, the detection accuracy of the object by the first method can be automatically improved.

(7) The third method preferably has a configuration in which the type of the object is detected based on the shape pattern of the contour line of the object.

In the present disclosure, an object type that could not be detected by the image recognition neural network can be detected based on the shape pattern of the contour line of the object, and the image recognition neural network can be additionally learned.

(8) When the type of the object cannot be detected by the third method, the type of the object is accepted from the user, and when the data of the captured image is input to the image recognition neural network, the object of the object It is preferable that the image recognition neural network is additionally learned by using the captured image and the received type of the object so that the position and range and the type of the object are output.

In the present disclosure, when the type of the object cannot be recognized even by the third method, the type of the object can be received from the user, and the image recognition neural network can be additionally learned by using the received contents. That is, according to the above aspects (6) and (7), the image recognition neural network is automatically additionally learned as much as possible, and the image recognition neural network is helped by the user for unknown objects that cannot be recognized even by the third method. Perform additional learning.
Therefore, even in an environment where unknown objects can occur, the image recognition neural network is semi-automatically additionally learned so that various objects can be recognized, and the positions and types of various objects can be detected on a pixel-by-pixel basis.

(9) A configuration in which the detection of the object using the image recognition neural network and the additional learning of the image recognition neural network are executed in parallel is preferable.

In the present disclosure, object detection using an image recognition neural network and additional learning of the image recognition neural network can be executed in parallel. Therefore, the object detection process can be continued even during the learning process of the image recognition neural network.

(10) The captured image is an image obtained by imaging an inspection target object, and the object to be detected is preferably a change site in the inspection target object.

In the present disclosure, the abnormal portion and type of the inspection object can be detected on a pixel-by-pixel basis.

(11) The object detection device of the present disclosure is an object detection device that detects an object included in a captured image, and specifies the position and range of the object included in the captured image with a bounding box, and the bounding. A first detection unit that detects the type of the object surrounded by a box, a second detection unit that detects the position of the object included in the captured image on a pixel-by-pixel basis, and the first detection unit that detects the object. A determination unit that determines the position and type of the object on a pixel-by-pixel basis based on the position and range of the object, the type of the object, and the position of the pixel of the object detected by the second detection unit on a pixel-by-pixel basis. To be equipped with.

In the present disclosure, the position and type of the object included in the captured image can be detected on a pixel-by-pixel basis in the same manner as in the aspect (1).

(12) The computer program of the present disclosure is a computer program for causing a computer to detect an object included in a captured image, and causes the computer to set a position and a range of the object included in the captured image in a bounding box. A second method of detecting the object by the first method of detecting the type of the object surrounded by the bounding box and detecting the position of the object included in the captured image on a pixel-by-pixel basis. Based on the position and range of the object detected by the first method, the type of the object, and the position of the pixel of the object detected by the second method. A process of determining the position and type of the object on a pixel-by-pixel basis is executed.

In the present disclosure, the computer can function as the object detection device according to the aspect (11).

(13) The computer program of the present disclosure is a computer program for causing a computer to detect an object included in a captured image and display information obtained by detecting an object included in the captured image, and causes the computer to display the position of the object included in the captured image. And the range is specified by the bounding box, and the information obtained by detecting the object by the first method of detecting the type of the object surrounded by the bounding box, and the object included in the captured image. A process of displaying information based on the information obtained by detecting the object by the second method of detecting the position of the object on a pixel-by-pixel basis is executed.

In the present disclosure, information based on information obtained by detecting an object by the first method and information obtained by detecting an object by the second method, for example, an object detected in pixel units. Information about the position and type of the object can be displayed.

[Details of Embodiments of the present disclosure]
Specific examples of the object detection method, the incident detection device, and the computer program according to the embodiment of the present disclosure will be described below with reference to the drawings. It should be noted that the present disclosure is not limited to these examples, but is indicated by the scope of claims and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

(Embodiment 1)
<Hardware configuration of incident detection device>
FIG. 1 is a block diagram showing a hardware configuration of the incident detection device 1 according to the first embodiment. The incident detection device 1 (object detection device) is a computer having, for example, one or a plurality of CPUs (Central Processing Units), a multi-core CPU, a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), and other arithmetic units 11. A temporary storage unit 12, an image input unit 13, an output unit 14, an input unit 15, a storage unit 16, and a data storage unit 17 are connected to the calculation unit 11 via a bus line. The incident detection device 1 according to the first embodiment acquires the image data of the captured image 5 (see FIG. 5A) obtained by imaging the manufactured inspection object, and the object to be detected included in the captured image 5. 6. For example, it detects abnormal parts such as dust, hair, and dents on the inspection object attached to the inspection object. The object to be inspected is, for example, a connector constituting a wire harness.

The calculation unit 11 controls the operation of each component unit by executing the computer program 16a stored in the storage unit 16 which will be described later. The calculation unit 11 executes a process of detecting a change in the inspection target by detecting the position and type of the object 6 included in the captured image 5 obtained by imaging the inspection target in pixel units. The details of the processing contents will be described later.

The temporary storage unit 12 is a memory such as a DRAM (Dynamic RAM) or a SRAM (Static RAM), and is read by a computer program 16a read from the storage unit 16 when executing the arithmetic processing of the arithmetic unit 11, or by the arithmetic processing. Temporarily store various generated data.

The storage unit 16 is a non-volatile memory such as a hard disk, EEPROM (Electrically Erasable Programmable ROM), and a flash memory. The storage unit 16 executes a process in which the calculation unit 11 controls the operation of each component unit to detect the position and type of the object 6 included in the captured image 5 on a pixel-by-pixel basis and detect an abnormality in the inspection target object. The computer program 16a for the operation is stored. Further, the storage unit 16 recognizes the shape of various objects 6, and information such as a feature amount for detecting the type of the object 6 included in the captured image 5, and the detected object 6, that is, whether an abnormality is allowed. The threshold value for determining whether or not it is stored is stored. The threshold value is different for each type of the object 6, and the storage unit 16 stores different threshold values in association with the types of the plurality of objects 6.
The storage unit 16 may be in a mode of storing the computer program 16a read from the recording medium by a reading device (not shown). Recording media include optical disks such as CD (Compact Disc) -ROM, DVD (Digital Versatile Disc) -ROM, BD (Blu-ray (registered trademark) Disc), flexible disks, magnetic disks such as hard disks, magnetic optical disks, semiconductor memories, etc. Is. Further, the computer program 16a according to the first embodiment may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 16.

The image input unit 13 is an interface to which the image pickup unit 1a is connected. The image pickup unit 1a AD-converts an image pickup element such as a CCD or CMOS that converts an image imaged by a lens into an electric signal, and an electric signal converted by the image pickup element into digital image data, and performs AD conversion. Output image data. The image data output from the image capturing unit 1a is input to the incident detection device 1 via the image input unit 13. The image pickup unit 1a and the incident detection device 1 may be connected by a dedicated cable or may be connected via a network such as a LAN (Local Area Network). The image data is digital data in which each pixel arranged vertically and horizontally is represented by a luminance value of a predetermined gradation. In the present embodiment, it will be described as being monochrome image data.

The output unit 14 is an interface to which the display unit 1b is connected. The display unit 1b is a liquid crystal panel, an organic EL display, electronic paper, a plasma display, or the like. The display unit 1b displays various information according to the image data given by the calculation unit 11. For example, the content of the abnormality detection result, the image of the inspection object having a defect, and the like are displayed. The display unit 1b is an example of an external output device that outputs an abnormality detection result, and may be a buzzer, a speaker, a light emitting element, or another notification device. The factory worker can recognize the state of the inspection object and the like as a result of the abnormality detection from the image displayed on the display unit 1b.

An operation unit 1c such as a keyboard, a mouse, and a touch sensor is connected to the input unit 15. A signal indicating the operating state of the operating unit 1c is input to the incident detection device 1 via the input unit 15. The calculation unit 11 can recognize the operation state of the operation unit 1c via the input unit 15.

Like the storage unit 16, the data storage unit 17 is a non-volatile memory such as a hard disk, EEPROM, or a flash memory. The data storage unit 17 stores data such as image data of the inspection object and an abnormality detection result.

<Functional part of the incident detection device>
FIG. 2 is a functional block diagram showing a configuration example of the incident detection device 1 according to the first embodiment. The incident detection device 1 has a first detection unit 2, a second detection unit 3, and a quality determination unit 4 as functional units. Each functional unit of the incident detection device 1 is realized by hardware such as a calculation unit 11 and a temporary storage unit 12.

The first detection unit 2 image-recognizes the object 6 included in the captured image 5 and detects the position, range, and type of the object 6. Specifically, when the data of the captured image 5 is input, the first detection unit 2 learns to output the position and range of one or more objects 6 included in the captured image 5 and the type of the object 6. The completed image recognition neural network 21 is provided. The position of the object 6 is specified by the position and size of the bounding box 7 surrounding the object 6 (see FIG. 5B). The bounding box 7 has, for example, a rectangular shape, and has a width and a height so that the object 6 is inscribed. The image recognition neural network 21 can be configured using, for example, a known Tensorflow®. Further, the image recognition neural network 21 can be configured by using another known convolutional neural network (CNN).
The image recognition neural network 21 machine-learns the image data of the object 6 and the information indicating the type of the object 6 as teacher data so that the type of the object 6 included in the captured image 5 is output.

The second detection unit 3 includes an autoencoder 31 and a difference data generation unit 32. The autoencoder 31 is a functional unit in which the image data output from the image capturing unit 1a is input and the image data of the feature extraction image 5a extracted from the features of the inspection object included in the captured image 5 is output (see FIG. 6). ). Specifically, when the image data is input, the autoencoder 31 outputs the image data of the feature extraction image 5a showing the features of the normal inspection object. The captured image 5 may include images of dust, scratches, shadows, abnormal parts of the inspection object itself, and the like. The feature extraction image 5a is an image capture image 5 that reproduces an ideal inspection object that can be obtained when these images are removed and a normal inspection object without any abnormality is imaged. Auto-encoding is realized by a neural network. The neural network includes an intermediate layer that dimensionally compresses image data, and inputs and outputs image data having the same number of pixels.

The autoencoder 31 has an input layer 31a, an output layer 31b, a convolution layer (CONV layer) 31c, and a deconvolution layer (DECONV layer) 31d. The input layer 31a is a layer into which data of each pixel value related to image data is input. The convolution layer 31c is a layer that dimensionally compresses image data. For example, the convolution layer 31c performs dimensional compression by performing convolution integration. The feature quantity of the inspection object is extracted by the dimensional compression. The deconvolution layer 31d is a layer that restores the dimension-compressed data in the convolution layer 31c to the original dimension. The deconvolution layer 31d undergoes a deconvolution process to restore the original dimension. By the restoration, the original characteristics of the inspection object, that is, the image data representing the characteristics of the normal inspection object are restored. Although the example in which the convolution layer 31c and the deconvolution layer 31d are two layers is shown, one layer or three or more layers may be used. The output layer 31b is a layer that outputs data of each pixel value related to the feature extraction image 5a from which the features of the inspection object are extracted by the convolution layer 31c and the deconvolution layer 31d.
The autoencoder 31 machine-learns the neural network of the autoencoder 31 so that the input image data and the output image data are the same. That is, the neural network is machine-learned so that the input captured image 5 and the output feature extraction image 5a are the same. Such machine learning is performed using image data obtained by imaging a normal inspection object.

The difference data generation unit 32 calculates the difference between the image data acquired from the imaging unit 1a and the image data output from the autoencoder 31. Specifically, the difference data generation unit 32 calculates the difference between the pixel values of the pixels at the same location in each image data for each pixel. Then, the difference data generation unit 32 binarizes the difference between the pixel values and a predetermined threshold value. By the binarization process, an image including the image of the object 6 detected in pixel units is obtained. Hereinafter, the image including the object 6 detected in pixel units is referred to as a difference image 5b (see FIG. 6).

The quality determination unit 4 includes an object detection processing unit 41, a dimension measurement processing unit 42, and a quality determination processing unit 43.
The object detection processing unit 41 is a functional unit that detects the position and type of one or more objects 6 included in the captured image 5 on a pixel-by-pixel basis.
The dimension measurement processing unit 42 is a functional unit that measures the dimensions of each object 6 detected in pixel units.
The quality determination processing unit 43 executes a process of determining whether or not the change is acceptable, that is, whether or not the change is good or bad, by comparing the dimensions of the object 6 with the threshold value according to the type of the object 6. It is a functional part.

<Abnormal part detection processing>
FIG. 3 is a flowchart showing a processing procedure related to object detection. The calculation unit 11 acquires the image data output from the image pickup unit 1a (step S11). Next, the calculation unit 11 detects the position and type of the object 6 included in the captured image 5 in pixel units by executing the mapping process described later (step S12). Next, the calculation unit 11 measures the dimensions of the detected object 6 (step S13) and determines the quality of the object 6 (step S14).

Next, the calculation unit 11 determines the quality of the inspection object based on the determination result in step S14 (step S15). That is, the calculation unit 11 determines whether or not there is an allowable change. If it is determined to be good (step S15: YES), the calculation unit 11 ends the process. If it is determined to be defective (step S15: NO), the calculation unit 11 executes a predetermined process for dealing with an abnormality of the inspection target object (step S16), and ends the process. For example, the calculation unit 11 executes a process of notifying that the inspection target has an abnormality. Further, the calculation unit 11 executes a process of displaying an image (see the right figure of FIG. 11) showing the quality determination result on the display unit 1b. The information of the image showing the quality determination result is an example of the information based on the information obtained by the detection process by the first detection unit 2 and the information obtained by the detection process by the second detection unit 3.

FIG. 4 is a flowchart showing a mapping processing procedure. The first detection unit 2 specifies the position and range of one or more objects 6 included in the captured image 5 in the bounding box 7, and detects the type of the object 6 surrounded by the bounding box 7 (step S31). ).

5A and 5B are explanatory views showing an object detection method by the first detection unit 2. FIG. 5A shows a captured image 5, and FIG. 5B is a captured image 5 showing a detection result of the object 6. As shown in FIG. 5A, the captured image 5 includes a plurality of objects 6. For example, an object 61 which is an image of dust, an object 62 which is an image of hair, an object 63 which is an image of dents, and the like are included. Then, as shown in FIG. 5B, each object 6 detected by the first detection unit 2 is surrounded by the bounding box 7, and the position of the object 6 is represented by the position and size of the bounding box 7. For example, the bounding box 71 indicates the position and range of the dust object 61, the bounding box 72 indicates the position and range of the hair object 62, and the bounding box 73 indicates the position and range of the dented object 63. There is. Further, in each bounding box 7, the type of the object 6 included in the bounding box 7 is displayed. In this way, according to the first detection unit 2, the approximate position and range of each object 6 included in the captured image 5 and the type are specified.

Next, the second detection unit 3 detects the object 6 included in the captured image 5 in pixel units (step S32).

FIG. 6 is an explanatory diagram showing an object detection method by the second detection unit 3. When the image data of the captured image 5 is input to the autoencoder 31, the image data of the feature extraction image 5a representing the original features of the captured image 5 from which the abnormal portion has been removed is output. Then, the difference data generation unit 32 calculates the difference between the acquired original image data and the image data output from the autoencoder 31. Specifically, the difference data generation unit 32 calculates the difference between the pixel values of the pixels at the same location in each image data for each pixel. Then, the difference data generation unit 32 binarizes the difference between the pixel values and a predetermined threshold value. By the binarization process, an image including the image of the object 6 detected in pixel units is obtained. Hereinafter, the image including the object 6 detected in pixel units is referred to as a difference image 5b. The difference image 5b does not include the inspection object itself, but includes only the object 6 which is an abnormal part. For example, as shown in the right figure of FIG. 6, the difference image 5b includes an object 61 which is an image of dust, an object 62 which is an image of hair, and an object 63 which is an image of dents.

Next, the calculation unit 11 determines the position and type of the object 6 on a pixel-by-pixel basis by the mapping process (step S33), and finishes the process related to the object detection. The calculation unit 11 that executes the process of step S33 functions as a determination unit that determines the position and type of the object 6 on a pixel-by-pixel basis based on the detection results of the first detection unit 2 and the second detection unit 3.

FIG. 7 is an explanatory diagram showing a method of determining the position and type of the object 6 on a pixel-by-pixel basis. The upper left figure is a captured image 5 in which the detection result of the object 6 by the first detection unit 2 is reflected. The position of each object 6 included in the captured image 5 is represented by the position and dimensions of the bounding box 7. The type of each object 6 is represented by a label attached to the bounding box 7. Although the bounding box 7 indicates the type of the object 6, it does not have accurate information on the position, shape, or dimension of the object 6.
On the other hand, the lower left figure is a difference image 5b in which the detection result of the object 6 by the second detection unit 3 is reflected. Each pixel of the object 6 included in the difference image 5b accurately indicates the position or shape of the object 6 on a pixel-by-pixel basis, but does not have information on the type of the object 6.
The figure in the center conceptually shows a state in which the bounding box 7 is mapped to the difference image 5b output from the second detection unit 3. By mapping the bounding box 7 to the difference image 5b, it is possible to determine the type of the object 6 on a pixel-by-pixel basis. That is, the type of the object 6 can be determined on a pixel-by-pixel basis by associating the one type with the pixels of the object 6 included in the one type of bounding box 7.
The figure on the right conceptually shows a state in which the type of the object 6 is determined for each pixel. For example, the type "dust" is associated with each pixel constituting the dust object 61. Similarly, the type "hair" is associated with each pixel constituting the hair object 62, and the type "dent" is associated with each pixel constituting the dent object 63.

FIG. 8 is a flowchart showing a processing procedure for measuring the dimensions of the object 6, and FIG. 9 is an explanatory diagram showing a method of measuring the dimensions of the object 6. The calculation unit 11 selects a pixel group of one type of object 6 (step S51). That is, a group of a plurality of pixels constituting one object 6 is selected. For example, as shown in FIG. 9, a group of plurality of pixels constituting the dust object 61 is selected. A plurality of pixels of different types of objects 6 are not selected as a group of pixels even if they are connected or adjacent to each other. Further, even if a plurality of pixels of the same type of the object 6 are separated from each other by a predetermined pixel or more, they are not selected as a group of pixels.

Then, the calculation unit 11 calculates the total number of pixels of the group of the selected objects 6 as shown in the right figure of FIG. 9 (step S52). For example, the total number of pixels constituting the dust object 61 is 13. Similarly, the total number of pixels that make up the hair is 19. The total number of pixels that make up the object 6 corresponds to the dimensions of the object 6.
The object 6 has various shapes such as a lumpy object and a linear object, but the objects 6 of the same type have substantially the same shape. Therefore, if the type of the object 6 is known, the total number of pixels constituting the object 6 can be used as information indicating the dimensions of the object 6.
For example, the total number of pixels that make up the hair object 62 corresponds to the length dimension of the hair. The total number of pixels constituting the dust object 61 roughly corresponds to the product of the vertical and horizontal dimensions.

Next, the calculation unit 11 determines whether or not the dimensional measurement of all the objects 6 included in the difference image 5b has been completed (step S53). When it is determined that there is an object 6 for which the measurement has not been completed (step S53: NO), the process is returned to step S51, and the same calculation process is executed for the other object 6 whose dimensions have not been measured. When it is determined that the measurement processing of all the objects 6 has been completed (step S53: YES), the calculation unit 11 ends the processing related to the dimension measurement.

FIG. 10 is a flowchart showing a pass / fail determination processing procedure for the object 6, and FIG. 11 is an explanatory diagram showing a pass / fail determination method for the object 6.

The calculation unit 11 selects a threshold value corresponding to the type of pixel group of the object 6 that is the target of the pass / fail judgment (step S71). The storage unit 16 stores different threshold values for each type of the object 6, and the calculation unit 11 selects the threshold value associated with the type of the object 6 to be determined.

Then, the calculation unit 11 compares the total number of pixels of the object 6 to be judged as good or bad with the threshold value selected in step S71 (step S72), and stores the comparison result (step S73). For example, when the total number of pixels constituting one object 6 is equal to or greater than a threshold value, the calculation unit 11 stores information indicating that the object 6 is an unacceptable change. If the total number of pixels is less than the threshold, the object 6 stores information indicating that it is an acceptable anomaly.
As shown in FIG. 11, since the total number of pixels indicating the size of dust (Σ pixels) is less than the first threshold value related to dust, it is determined that it is acceptable. Similarly, the total number of pixels indicating the degree of dents (Σ pixels) is less than the third threshold value related to dents, so it is determined that it is acceptable. However, since the total number of pixels indicating the length of the hair (Σ pixels) is equal to or greater than the second threshold value related to the hair, it is determined that the change is unacceptable.

Next, the calculation unit 11 determines whether or not the pass / fail determination has been completed for all the objects 6 (step S74). When it is determined that there is an object 6 for which the pass / fail judgment has not been completed (step S74: NO), the process is returned to step S71, and the same calculation process is executed for the other objects 6 whose pass / fail has not been measured. When it is determined that the quality determination of all the objects 6 has been completed (step S74: YES), the calculation unit 11 ends the processing related to the quality determination.

The above description has been described by taking as an example a state in which a plurality of objects 6 included in the captured image 5 are separated from each other. It can detect, measure dimensions, and determine abnormalities.

12A, 12B, 12C, and 12D are explanatory views showing a method of detecting and measuring the dimensions of overlapping objects 6. FIG. 12A shows a state in which dust and hair objects 61 and 62 overlap. The second detection unit 3 detects a plurality of pixels constituting the dust and hair objects 61 and 62 as a group of pixels without distinguishing the type of each object 6.
FIG. 12B shows a state mapped by a bounding box 71 showing the position and range of dust. The calculation unit 11 recognizes the pixel group included in the bounding box 71 as the pixels constituting the dust object 61, and assigns the type "dust" to each pixel.
FIG. 12C shows the state mapped by the bounding box 72 showing the position and range of the hair. The calculation unit 11 recognizes the pixel group included in the bounding box 72 as the pixels constituting the hair object 62, and assigns the type "hair" to each pixel.
FIG. 12D shows the result of recognizing the type of the object 6 on a pixel-by-pixel basis by the mapping process shown in FIGS. 12B and 12C. Pixels shown in black are given the type "hair". The hatched pixels are given the type "dust". The white-painted pixels are given both types of "dust" and "hair".
In this way, by assigning both types to the pixels of the portion where the objects 6 overlap, it is possible to measure the dimensions of each object 6 even if the objects 6 overlap. For example, the pixels to which the type of "dust" is given are the pixels with hatching and the white pixels, and the dimension of the object 61 of "dust" is measured by calculating the total number of the pixels. be able to. Similarly, the pixels to which the type of "hair" is given are black-painted pixels and white pixels, and the dimension of the object 62 of "hair" is measured by calculating the total number of the pixels. Can be done. In this way, even if a plurality of types of objects 6 are overlapped with each other, the position and type of each object 6 can be detected on a pixel-by-pixel basis, and the dimensions of each object 6 can be measured.

As described above, according to the present embodiment, the object 6 is detected by the first detection unit 2 and the second detection unit 3, and the mapping process shown in FIG. 7 is performed to obtain the object 6 included in the captured image 5. The position and type can be detected on a pixel-by-pixel basis. Since the position and type of the object 6 can be detected on a pixel-by-pixel basis, the dimensions of the object 6 which is an abnormal portion can be accurately measured.
In addition, the abnormal portion detected on a pixel-by-pixel basis and its quality can be displayed as shown in FIG.
For example, it is possible to detect the abnormal portion and type of the inspection target such as the connector constituting the wire harness on a pixel-by-pixel basis, and accurately calculate the dimensions of the abnormal portion.

In addition, it is possible to determine whether or not the detected object 6 is an acceptable change by using different threshold values depending on the type of the object 6.

Further, it is a simple process of comparing the total number of pixels of the object 6 detected in pixel units with the threshold value, and the above determination can be performed without performing different measurement calculation processes according to the shape of the object 6. That is, it is possible to determine whether or not the size of the abnormal portion is an allowable dimension without calculating the dimension of the abnormal portion by a complicated process.

Furthermore, by using the autoencoder 31, the position of the object 6 can be accurately detected on a pixel-by-pixel basis. In addition, the dimensions of the object 6 can be calculated accurately.

Furthermore, by using the trained image recognition neural network 21, it is possible to recognize an object 6 having various characteristics and detect the position, range, and type of the object 6.

In the first embodiment, an example in which one computer operates as an incident detection device 1 and functions as a first detection unit 2, a second detection unit 3, a quality determination unit 4, etc. has been described, but each functional unit has been described. The cloud computer may be configured to execute a part or all of the processes for realizing the above. Further, it may be configured to be executed in a virtual machine, such as executing a process for realizing each functional unit on a plurality of computers.

Further, in the first embodiment, the system for detecting the quality of the inspection target object is mainly described, but the content of the object 6 to be detected is not particularly limited.

(Embodiment 2)
<Additional learning of image recognition neural network>
FIG. 13 is a block diagram showing a configuration example of the incident detection system according to the second embodiment. The incident detection system according to the second embodiment includes an incident detection device 1 and a machine learning device 9. The incident detection device 1 is installed in, for example, a factory where an inspection object is manufactured, and the machine learning device 9 is installed in a facility outside the factory. The incident detection device 1 has the same components as that of the first embodiment, and further includes a communication unit 18 for transmitting and receiving data to and from the machine learning device 9. The machine learning device 9 is a server device that additionally learns the image recognition neural network 21 that constitutes the incident detection device 1. The machine learning device 9 is a computer, and the basic hardware configuration is the same as that of the incident detection device 1.

The incident detection device 1 and the machine learning device 9 are separate computers, and the programs are executed and operated in parallel. Specifically, the incident detection device 1 provides a copy of the image recognition neural network 21 to the machine learning device 9, and while the incident detection device 1 executes a process of detecting an abnormality of an inspection object, machine learning is performed. The device 9 can execute the additional learning process of the image recognition neural network 21. The incident detection device 1 continues the incident detection process by using the image recognition neural network 21 before learning instead of the image recognition neural network 21 during learning.
The machine learning device 9 transmits various parameters for forming the image recognition neural network 21, for example, parameters such as the number of layers of the image recognition neural network 21, the number of neurons, the type of neural network, and the weighting coefficient to the machine learning device 9. By doing so, a copy of the image recognition neural network 21 is provided.

The incident detection device 1 is added when the position and range of the object 6 detected by the first detection unit 2 and the position of the object 6 detected by the second detection unit 3 do not match during the incident detection process. The machine learning device 9 is requested to perform the processing related to learning.

FIG. 14 is a flowchart showing a processing procedure related to the additional learning according to the second embodiment. Similar to the first embodiment, the calculation unit 11 of the incident detection device 1 acquires image data from the image pickup unit 1a and executes a process related to the detection of the object 6 (step S111). Although the flowchart is simplified by omitting the processing block for acquiring the image data, the specific processing content is the same as that of the first embodiment.

Next, the calculation unit 11 determines whether or not the detection of the object 6 is successful (step S112). Specifically, in the calculation unit 11, the position and range of the object 6 specified by the first detection unit 2 in the bounding box 7 and the position of the pixel of the object 6 detected by the second detection unit 3 are determined. If they are consistent, it is determined that the detection of the object 6 is successful, and if they are not consistent, it is determined that the detection of the object 6 has failed. In the second embodiment, an example of detection failure is a state in which the type of the object 6 cannot be detected by the first detection unit 2 even though the pixels of the object 6 are detected by the second detection unit 3. It is explained as.

When it is determined that the detection of the object 6 has not failed (step S112: NO), the calculation unit 11 executes the processes related to the dimension measurement and the quality determination of the object 6 in the steps S113 and S114 as in the first embodiment. .. Although the flow chart is simplified by omitting the processing block at the time of pass / fail judgment, the specific processing content is the same as that of the first embodiment.

When it is determined in step S112 that the detection of the object 6 has failed (step S112: YES), the calculation unit 11 transmits the current image recognition neural network 21 to the machine learning device 9 by the communication unit 18 (step S115). ). Specifically, various parameters for forming the image recognition neural network 21 are transmitted to the machine learning device 9.

The machine learning device 9 receives the image recognition neural network 21 transmitted from the incident detection device 1 as an additional learning target (step S116).

The incident detection device 1 that has completed the process of step S115 executes the process related to the generation of learning data (step S117).

FIG. 15 is a flowchart showing a processing procedure related to the generation of learning data, and FIGS. 16 and 17 are explanatory views showing a method of generating learning data. First, the calculation unit 11 divides the captured image 5 that failed to detect the object 6 into a grid pattern as shown in the right figure of FIG. 16, and creates an array Y indicating information about the object 6 for each of the divided image blocks. Allocate (step S131). The array Y is represented by, for example, the following formula (1). The initial value of each variable of the array assigned in step S131 is 0.

The variable P in the array Y indicates whether or not the image block contains the object 6. As shown in FIG. 17, P = 1 indicates that the object 6 is included, and P = 0 indicates that the object 6 is not included.
As shown in FIG. 17A, the variables Bx and By indicate the X coordinate and the Y coordinate of the center position of the object 6, for example, the position of the center of gravity of the image. The method of taking the coordinate system is not particularly limited, but as shown in FIG. 17, for example, the origin is the lower left vertex of each image block, the horizontal direction of the captured image 5 is the X axis, and the vertical direction is the Y axis. It is preferable to use a Cartesian coordinate system.
The variables Bw and Bh indicate the width and height of the object 6.
The variables C1, C2, C3, ... Correspond to the type of object 6. For example, C1 corresponds to "dust", C2 corresponds to "hair", C3 corresponds to "dent" and the like. The variable C1 = 1 indicates that the object 6 is "dust", and the variable C1 = 0 indicates that the object 6 is not "dust". The same applies to the other variables C2, C3, .... As for the variables indicating the types of the object 6, it is advisable to prepare an appropriate number of variables of the type "undecided" so as to correspond to the new types of the object 6.

The calculation unit 11 that has completed the process of step S131 specifies the presence / absence, position, and dimension of the object 6 in each image block with reference to the detection result of the second detection unit 3 (step S132). Then, as shown in FIGS. 17A and 17B, the calculation unit 11 substitutes the corresponding numerical values into the variables P, Bx, By, Bw, and Bh of the array Y assigned to each image block according to the specific result (step). S133).
If the image block does not include the object 6, each variable in the array Y becomes 0.

Next, the calculation unit 11 detects the type of the object 6 for which the type has not been specified by executing a known shape recognition process on the captured image 5 (step S134). For example, the calculation unit 11 detects the edge of the object 6, calculates a predetermined feature amount, and compares it with the feature amount stored in the storage unit 16 to specify the type of the object 6 corresponding to the similar feature amount. By doing so, the type of the object 6 is detected. Further, the type of the object 6 may be specified by pattern matching.

Then, the calculation unit 11 that has completed the process of step S134 determines whether or not the detection of the type of the object 6 is successful (step S135). If it is determined to be successful (step S135: YES), the calculation unit 11 substitutes the corresponding numerical values into the variables C1, C2, C3 ... According to the detection result in step S134 (step S137), and the training data. Finishes the generation process of.

When it is determined in step S135 that the object has failed (step S135: NO), the calculation unit 11 accepts the type of the object 6 from the user in the operation unit 1c (step S136). For example, the calculation unit 11 may display the captured image 5 in question on the display unit 1b, and may display an instruction image indicating an image portion in which the detection of the object 6 has failed. For example, it is preferable to superimpose the frame image surrounding the pixel group of the object 6 detected by the second detection unit 3 on the captured image 5 without being detected by the first detection unit 2. The frame image is an example, and an arrow image or the like may be used.
The calculation unit 11 presents a plurality of expected types to the user, and accepts the selection of the type by the user. In the case of a new type, the calculation unit 11 may accept registration of the type name. The calculation unit 11 receives a new type name from the user, and changes and registers the variable name of the type "undecided" to the name.

Then, the calculation unit 11 substitutes the corresponding numerical values into the variables C1, C2, C3 ... According to the contents received in step S136 (step S137), and ends the learning data generation process.
By the above processing, it is possible to generate learning data in which the image data in which the detection of the object 6 fails and the data indicating the position, range, and type of the object 6 included in the image data are combined.

The calculation unit 11 that has completed the process related to the generation of the learning data in step S117 stores the generated learning data in the data storage unit 17 (step S118). Then, the calculation unit 11 determines whether or not a predetermined amount of learning data has been accumulated (step S119). When it is determined that a predetermined amount of learning data has not been accumulated (step S119: NO), the calculation unit 11 returns the process to step S113, and continues the incident detection process and the accumulation of the learning data. When it is determined that a predetermined amount of learning data has been accumulated (step S119: YES), the calculation unit 11 transmits the learning data accumulated in the data storage unit 17 to the machine learning device 9 by the communication unit 18 (step S119: YES). Step S120).

The machine learning device 9 receives the learning data transmitted from the incident detection device 1 (step S121), and additionally learns the image recognition neural network 21 received in step S116 using the learning data (step S122). ). That is, when the image data of the captured image 5 including the unrecognizable object 6 is input to the image recognition neural network 21, the machine learning device 9 outputs data indicating the position, range, and type of the object 6. As described above, the image recognition neural network 21 is additionally trained using the training data. The additional learning is supervised learning.

Next, the machine learning device 9 transmits the image recognition neural network 21 additionally learned in step S122 to the incident detection device 1 (step S123). The incident detection device 1 receives the additionally learned image recognition neural network 21 transmitted from the machine learning device 9 by the communication unit 18 (step S124). Then, the calculation unit 11 of the incident detection device 1 updates the image recognition neural network 21 before the additional learning with the new image recognition neural network 21 that has been additionally learned (step S125), and returns the process to step S111.

According to the incident detection system according to the second embodiment, even if the detection of the object 6 in pixel units using the image recognition neural network 21 and the autoencoder 31 fails, the learning data is automatically accumulated. , The image recognition neural network 21 can be additionally learned.

Specifically, when the type of the object 6 is not detected by using the image recognition neural network 21, the type of the object 6 can be detected by shape recognition, and the image recognition neural network 21 can be additionally learned.

Further, when the type of the object 6 is not specified even in the shape recognition process, by accepting the type of the object 6 from the user, additional learning data can be generated and the image recognition neural network 21 can be additionally learned.

Further, according to the second embodiment, the detection of the object 6 or the abnormality detection process of the inspection target object and the additional learning of the image recognition neural network 21 can be executed in parallel. Therefore, the detection process of the object 6 can be continued even during the learning process of the image recognition neural network 21.

1 Abnormality detection device 1a Imaging unit 1b Display unit 1c Operation unit 2 1st detection unit 3 2nd detection unit 4 Good / bad judgment unit 5 Captured image 5a Feature extraction image 5b Difference image 6, 61, 62, 63 Objects 7, 71, 72 , 73 Bounding box 9 Machine learning device 11 Calculation unit 12 Temporary storage unit 13 Image input unit 14 Output unit 15 Input unit 16 Storage unit 16a Computer program 17 Data storage unit 18 Communication unit 21 Image recognition neural network 31 Autoencoder 31a Input layer 31b Output layer 31c Convolution layer 31d Deconvolution layer 32 Difference data generation unit 41 Object detection processing unit 42 Dimension measurement processing unit 43 Good / bad judgment processing unit

Like the storage unit 16, the data storage unit 17 is a non-volatile memory such as a hard disk, EEPROM (registered trademark), and a flash memory. The data storage unit 17 stores data such as image data of the inspection object and an abnormality detection result.

Claims (12)

This is an object detection method that detects objects contained in captured images.
The object is detected by the first method of detecting the type of the object included in the captured image, and the object is detected.
The object is detected by the second method of detecting the object included in the captured image on a pixel-by-pixel basis.
Based on the type of the object detected by the first method and the object detected by the second method, the size and type of the object are determined in pixel units.
A threshold for making a predetermined determination with respect to the object is selected based on the type of the object.
An object detection method for making a predetermined determination based on the object determined on a pixel-by-pixel basis and a threshold value selected based on the type. The object detection method according to claim 1, wherein the predetermined determination is performed by comparing the total number of pixels of one type of the object with the threshold value selected based on the type. The detection of the object by the second method is performed by using a trained autoencoder that outputs the data of the feature extraction image from which the features of the input data are extracted when the data of the captured image is input. The object detection method according to claim 1 or 2. The detection of the object by the first method is performed by using a trained image recognition neural network that outputs the type of the object included in the captured image when the data of the captured image is input. The object detection method according to any one of claims 3. When the detection result by the first method and the detection result by the second method do not match, the type of the object included in the captured image is detected by a third method different from the first method.
When the data of the captured image is input to the image recognition neural network, the captured image and the detection according to the third method are output so that the type of the object detected by the third method is output. The object detection method according to claim 4, wherein the image recognition neural network is additionally trained using the result. The object detection method according to claim 5, wherein the third method detects the type of the object based on the shape pattern of the contour line of the object. When the type of the object cannot be detected by the third method, the type of the object is accepted from the user.
When the data of the captured image is input to the image recognition neural network, the image recognition neural network is additionally learned by using the captured image and the received type of the object so that the type of the object is output. The object detection method according to claim 5 or 6. The object detection method according to any one of claims 5 to 7, wherein the detection of the object using the image recognition neural network and the additional learning of the image recognition neural network are executed in parallel. The object detection method according to any one of claims 1 to 8, wherein the captured image is an image obtained by imaging an inspection target object, and the object to be detected is an abnormal site in the inspection target object. .. An object detection device that detects objects contained in captured images.
A first detection unit that detects the type of the object included in the captured image, and
A second detection unit that detects the object included in the captured image on a pixel-by-pixel basis,
A determination unit that determines the size and type of the object in pixel units based on the type of the object detected by the first detection unit and the object detected in pixel units by the second detection unit. When,
A selection unit that selects a threshold value for making a predetermined determination regarding the object based on the type of the object, and
An object detection device including the object determined on a pixel-by-pixel basis and a determination unit that makes a predetermined determination based on a threshold value selected based on the type. A computer program that allows a computer to detect objects contained in a captured image.
On the computer
The object is detected by the first method of detecting the type of the object included in the captured image, and the object is detected.
The object is detected by the second method of detecting the object included in the captured image on a pixel-by-pixel basis.
Based on the type of the object detected by the first method and the object detected by the second method, the size and type of the object are determined in pixel units.
A threshold for making a predetermined determination with respect to the object is selected based on the type of the object.
A computer program for executing a process of performing the predetermined determination based on the object determined on a pixel-by-pixel basis and a threshold value selected based on the type. A computer program for displaying information obtained by detecting an object contained in a captured image on a computer.
On the computer
The information obtained by detecting the object by the first method of detecting the type of the object included in the captured image and the second method of detecting the object included in the captured image on a pixel-by-pixel basis. A computer program for executing a process of displaying information based on the information obtained by detecting the object.

Images