JP2022018507A - Image processing apparatus for subject determination, and image processing method - Google Patents

Abstract

To provide an image processing apparatus for subject determination configured to allow for highly accurate determination processing even on a small image size, by applying image processing before inputting a captured image to a discriminator based on a neural network.SOLUTION: An image processing apparatus includes: an input unit for a captured image; an image storage unit for storing the captured image; an image size modification unit which generates a compressed image with a modified size from the captured image; a location information acquisition unit which acquires location information of a subject on the compressed image; a trimming information acquisition unit which extends the location information to the size of the captured image to generate location information for trimming; a trimming image generation unit which generates a trimming image by applying the location information for trimming to the captured image; a determination image generation unit which compresses the trimming image to a determination image size to generate a determination image; and an image output unit which outputs the determination image to an AI discriminator in the next stage.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image determination device for a defective portion of an inspection object, particularly an image processing device for generating a determination image to be input to an image determination device constructed by using an artificial intelligence (AI) network method.

Disclosed by an inspection device that uses a machine learning model based on a neural network to determine the presence or absence of defects appearing on the surface of an article based on an image obtained by photographing the object to be inspected (hereinafter referred to as "inspection object"). Has been done.

As an example of image processing, there is one that uses a so-called AI processing device to which a function of machine learning based on a neural network is applied. For example, Patent Document 1 and Patent Document 2 have been proposed as prior documents.

Patent Document 1 provides an inspection device and an inspection method in which the possibility of erroneously determining a non-defective product satisfying a predetermined standard as a defective product is reduced, and defines a defect in a photographed image of an inspected object. By measuring the length, area, etc. preset as data, it is determined whether or not there is a defect. Furthermore, the judgment is made using a machine learning model based on a neural network.

In Patent Document 2, in order to improve the accuracy of defect inspection, an image of an inspection target is divided into a plurality of blocks, and the presence or absence of defects is inspected for each block by AI processing.

Japanese Patent No. 6630912 JP-A-2017-166929

The discrimination device constructed based on the AI network used as the base of transfer learning for the inspection of defective parts appearing on the appearance of the subject inputs a relatively small square image of about 200 × 200 to 400 × 400 pix as its feature. It is an image. If an image having a larger number of pixels or a rectangular image is input to this network, the processing takes time, and as a result, the learning efficiency does not improve and it becomes difficult to obtain a good judgment result.

Therefore, in order to avoid this, if the captured image is simply reduced to match the image size for input of the AI network, learning is possible, but 2,500 × 2, which was taken at the base 5Mpix. Assuming that a defect area of about 2 × 2 pix exists in the image of 000 pix, if the size is changed as it is (for example, it is reduced to 1/10 in length and width and 1/100 in area), the defect area becomes 0.2 × 0.2 pix. It becomes a very small image and it becomes difficult to distinguish it. As a result, there is a problem that the small defect area is positioned as difficult to detect due to poor learning efficiency.

Therefore, the present invention is for subject determination, which can perform highly accurate determination processing even with a small image size by performing image processing in advance for inputting a captured image to a determination device based on a neural network. The purpose is to provide an image processing device for the above.

In order to solve the above problems, the image processing apparatus for determining a subject according to the present invention is configured as follows.

That is, an image input unit for inputting an image of a subject, an image storage unit for storing the input image in a storage means, and a compressed image of the input to be compressed to a determination image size. An image size changing unit that generates an image, a position information acquisition unit that acquires position information of an object in the resized image, and an image for trimming from an image generated by expanding the position information to the size of the captured image. A trimming information acquisition unit that generates position information, a trimming image generation unit that applies the trimming position information to the corresponding captured image stored in the image storage unit to generate a trimmed image, and the trimmed image. It is characterized by comprising a determination image generation unit that compresses the determination image size to generate a determination image and an image output unit that outputs the determination image to the determination device in the next stage.

Further, it is characterized in that the acquisition of the position coordinate information in the position information acquisition unit is performed by using a means based on a neural network.

In addition, the present invention can be regarded as an invention of a product composed of each component, and can also be regarded as an invention of a method because each component functions in time series.

With the above configuration, the present invention detects a test object from a captured image with a high number of pixels, performs trimming to cut off unnecessary margins, and further compresses this trimmed image to match the input size of a specified AI network. Therefore, even small defects can be left in a size as close to the uncompressed state as possible, and as a result, the effect of improving the detection accuracy is achieved.

Further, according to claim 2, the acquisition of the position information of the test object in the resized image is more stable by combining the conventional binarization, multi-value, edge extraction and other processes. It has the effect of capturing coordinate information and can dramatically increase the processing speed.

It is a flow figure which shows the structure of the Embodiment which concerns on this invention. It is a photographed image diagram of four kinds of test objects which become input images. It is a square-sized compressed image diagram of four kinds of subjects. It is a figure which showed the stretched image which stretched four kinds of subjects to the photographed image size, and the trimming frame. It is a trimming image diagram of four kinds of subjects. It is a judgment image figure of 4 kinds of test objects.

Hereinafter, the present invention will be described with reference to the drawings.
First, as shown in FIG. 1, the image processing apparatus according to the present invention includes an image input unit 1, an image storage unit 2, an image size change unit 3, a position information acquisition unit 4, a trimming information acquisition unit 5, and a trimming image generation. It is composed of a unit 6, a determination image generation unit 7, and an image output unit 8.

In the embodiment of the present invention (hereinafter referred to as “the present embodiment”), image processing performed on the illustrated four types of test objects A, B, C, and D will be described. The size of the captured image Pi (hereinafter referred to as "shooting size") as the input image is 2,048 vertical x 2,448 pix horizontally, and the size of the AI determination image Ji (hereinafter referred to as "judgment size") is set. It will be described below as 224 vertical × 224 horizontal pix.

[Image input section]
First, the test objects A, B, C, and D to be the target in this embodiment are individually photographed by a camera, and the image data of the photographed size is input to the image input unit 1 which is the starting point of the present apparatus ( Step 1, hereinafter, step n is referred to as "Sn"). The shooting size at that time is a rectangular image. This shooting size depends on the specifications of the camera to shoot and is set appropriately.

[Image storage unit]
In order to use the input captured image for later processing, it is temporarily stored in the image storage unit 2 which is a storage means (S2). This storage means is a device that can be recorded as electronic data.

[Image size change part]
The image size changing unit 3 changes the captured image Pi to a compressed image Ci so as to have a determination size (S3). The image change method is performed by using an image compression method. The compression ratio at this time is vertical compression in order to compress the shooting size of the rectangular shot image Pi (vertical 2,048 x horizontal 2,448 pix) to the size of the square compressed image Ci (vertical 224 x horizontal 224 pix). The rate is 0.109 and the compression rate in the lateral direction is 0.092. This compression process uses a general method, and since the compressed image Ci generated by compression is not scheduled to be restored, it is performed by an irreversible method. In this embodiment, compression is performed by taking the average of each pixel in the vertical and horizontal directions of 8.9 pix → 1 pix / 6.7 pix → 1 pix, respectively. The compression process in this embodiment is not limited to this. Further, the compressed image Ci may be irreversible, or restoration information may be added.

[Location information acquisition unit]
Next, the position information acquisition unit 4 acquires the position information of each of the reflected objects A, B, C, and D from the compressed image Ci (S4). In this embodiment, this position information is acquired as an XY coordinate value.

For example, as shown in FIG. 3A, the XY coordinate values at the size (224 × 224pix) of the subject A on the compressed image Ci are points a (45,95), b (180,115), and c. Coordinate values are acquired as position information as points (58, 55), d points (193, 70), and O points (origin).

In this embodiment, the determination of the coordinate values is performed by using a means based on a neural network. That is, the reflected test objects A, B, C, and D are captured as some kind of object regardless of shape, each pixel information is input to the input layer of the neural network, and various shapes are learned in advance for calculation. The output from the output layer is acquired as position information through the intermediate layer (the layer configuration is appropriate) with weight. In this learning, annotations (creation of learning data) are performed using this compressed image, but annotations for judgment learning of defective parts are not performed. In other words, for this learning data, data showing only the shape of the subject in the image is used, and a large amount of this data is input to calculate the weighting to construct a neural network. The position of the test object is acquired as an XY coordinate value. At this stage, the learning data does not include information on the defective portion of the test object.

Since black is included in the color variations of the test objects A, B, C, and D, it is necessary to devise special measures such as lighting at the time of shooting. It was difficult to stably capture the coordinate information of the test object even by combining the conventional binarization, multi-value, edge extraction, etc., but by using this neural network method, the accuracy It has become possible to acquire high-level location information quickly and stably.

In addition, although it does not constitute the present invention, it can be used as a determination device for determining the presence or absence of defects in a test object based on a neural network, which is a next-stage device, by simply changing the above learning content. It has a secondary effect.

It should be noted that the acquisition of the position information of the present invention does not exclude the acquisition by the above-mentioned binarization processing, multi-value processing, various filter processing, edge extraction, center of gravity extraction and combination processing thereof.

[Trimming information acquisition unit]
Next, the trimming information acquisition unit 5 expands the position coordinate information of the subject in the compressed image to the vertical and horizontal sizes of the original captured image Pi to generate an image (see FIG. 4), and this expanded image. The position information converted for trimming based on Ei is acquired (S5). This acquired position information becomes the position information for trimming for creating the next trimmed image.

For example, the XY coordinate values in the size (2,448 × 2,048pix) of the stretched image Ei of the subject A shown in FIG. 4 are point e (491,863), point f (1,967,1051), g. It becomes a point (633,503) and a point h (2,109,660) and is acquired as converted position information. In this embodiment, for example, the reciprocal of the vertical and horizontal compression ratios is multiplied in each of the vertical and horizontal directions to obtain the stretched image Ei.

[Crop image generator]
In the trimming image generation unit 6, the position information for trimming is applied to the captured image Pi of the same size stored in the image storage unit 2, and the peripheral portions of the test objects A, B, C, and D are applied. Is left at a predetermined ratio, and an unnecessary portion is trimmed (cut) to generate a rectangular (square or rectangular) trimmed image Ti (S6).

According to the photographed image Pi shown in FIG. 4 of this embodiment, as an example, the XY coordinate values of the trimming frame 50 corresponding to the subject A are m points (265, 1, 124) and n points (2,). 221,1124), o points (265,266), p points (2,221,266); The trimming frame 50 has a rectangular shape having a size of 858 vertical × 1,956 pix horizontal.

Since the trimmed image Ti is cut out so as to surround each of the test objects A, B, C, and D so as to leave a certain margin, it becomes a photographed image of each area. Since the shapes of the images of the test objects A, B, C, and D reflected in the captured image are polygonal in size and shape as shown in the four types shown in the figure, the trimmed image Ti is also different. It becomes a rectangular shape.

[Image generation unit for judgment]
Finally, the determination image generation unit 7 compresses the trimmed image Ti cut out by the trimming frame 50 to the size of the square determination image (a square image of 224 pix) to generate a determination image Ji (S7). .. As described above, since the cropped image Ti has different sizes, the vertical and horizontal compression ratios are different from each other.

As shown in FIG. 6, the compression ratios in the vertical and horizontal directions in the stretched image Ji for each of the test objects A, B, C, and D are as follows.

(A) Subject A-Vertical compression rate: 224/858 = 0.26
-Horizontal compression rate: 224 / 1,956 = 0.11

(B) Subject B-Vertical compression rate: 224/847 = 0.26
-Horizontal compression rate: 224 / 1,029 = 0.22

(C) Subject C-Vertical compression rate: 224/803 = 0.28
-Horizontal compression rate: 224 / 1,853 = 0.12

(D) Subject D-Vertical compression rate: 224/807 = 0.28
-Horizontal compression rate: 224/1,227 = 0.18

[Image output section]
The image data for determination created through the processing in each of the above units is output from the image output unit 8. The output data is transferred to a determination device for determining the presence or absence of defects in the test object based on the neural network of the next stage.

According to the above embodiment, although it is schematic, the width of a substantially rectangular (or parallelogram) subject as shown in the figure is set to about 35 mm, and this is regarded as a pixel of about 2,100 pix within the shooting size. , 35 mm ÷ 2,100 = 0.017 mm / rectangle. On the other hand, assuming that the minimum size of the defect to be detected is 0.10 mm, the size is 0.10 ÷ 0.017 = 5.88 fix.

When this is applied to the embodiment of the test object A shown in FIG. 6 (A), when the calculation is performed by applying the lateral compression rate of 0.11, the result is 5.88 pix × 0.11 = 0.647 pix, and the defect portion is almost the same. Although it becomes invisible, when the compression rate of 0.26 in the vertical direction is applied, it becomes 5.88 fix × 0.26 = 1.53pix, and the defective image remains, and it becomes a detectable size, which is useful in actual use. It becomes.

1 Image input unit 2 Image storage unit 3 Image size change unit 4 Position information acquisition unit 5 Trimming information acquisition unit 50 Trimming mink frame 6 Trimming image generation unit 7 Judgment image generation unit 8 Image output units A, B, C, D Subject Pi Photographed image Ji Judgment image Ei Stretched image Ti Trimmed image

That is, an image input unit for inputting an image of a subject, an image storage unit for storing the input image in a storage means, and a compressed image of the input to be compressed to a determination image size. An image size changing unit that generates an image, a position information acquisition unit that acquires position information of an object in the compressed image , and a position for trimming from an image generated by expanding the position information to the size of the captured image. The trimming information acquisition unit that generates information, the trimming image generation unit that generates a trimmed image by applying the trimming position information to the corresponding captured image stored in the storage means , and the determination of the trimmed image. It is characterized by comprising a determination image generation unit that compresses the image for determination to generate a determination image and an image output unit that outputs the determination image to the determination device in the next stage.

Further, according to claim 2, the position information of the subject is acquired from the compressed image compressed to the judgment image size by using a method based on the neural network, so that the position information with high accuracy can be acquired. It has the effect of being able to quickly and stably capture the coordinate information of the subject, and can dramatically increase the processing speed.

Claims (4)

It is an image processing device that generates a judgment image for performing a judgment of quality of a test object with a judgment device based on a neural network.
An image input unit that allows you to input an image of the subject,
An image storage unit that stores the input captured image in a storage means,
An image size changer that compresses the input captured image to the image size for judgment and generates a compressed image.
A position information acquisition unit that acquires the position information of the subject in the compressed image, and
The trimming information acquisition unit that generates trimming position information from the image generated by expanding the position information to the size of the captured image and the corresponding captured image that stores the trimming position information in the image storage unit. A cropped image generator that applies and generates a cropped image,
A judgment image generation unit that compresses the trimmed image to the judgment image size and generates a judgment image,
An image output unit that outputs the determination image to the determination device in the next stage,
An image processing device for determining a subject, which is characterized by being composed of. The acquisition of the position coordinate information in the position information acquisition unit is
The image processing apparatus for determining an object according to claim 1, wherein the image processing apparatus is performed by using a means based on a neural network. It is an image processing method that generates a judgment image for performing a judgment of quality of a test object with a judgment device based on a neural network.
An image input step that allows you to input an image of the subject,
An image storage step of storing the input captured image in a storage means, and
The image size change step of compressing the input captured image to the judgment image size to generate a compressed image, and
A position information acquisition step for acquiring the position information of the subject in the resized image, and
The trimming information acquisition step of generating the trimming position information from the image generated by expanding the position information to the size of the captured image and the corresponding captured image stored in the image storage unit with the trimming position information. A cropped image generation step that applies and generates a cropped image,
A judgment image generation step of compressing the trimmed image to the judgment image size to generate a judgment image, and
An image output step of outputting the determination image to the determination device in the next stage,
An image processing method for determining a subject, which is characterized by being composed of. The acquisition of the position coordinate information in the position information acquisition step is
The image processing method for determining a subject according to claim 3, wherein the image processing method is performed by using a means based on a neural network.

Images