JP2022120734A - Learning data generation device, method for generating learning data, and learning data generation program - Google Patents

Abstract

To allow easy generation of learning data of a transmissive image.SOLUTION: The acquisition unit acquires a detection target image as a transmissive image of a detection target object and a transmissive image of an object that is not a detection target object. A display manner changing unit changes the manner of display of the detection target object in the detection target image. A synthesis processing unit generates learning data by synthesizing the detection target image of the detection target object with the changed display manner with the image not to be detected. In that way, changing the display manner of the detection target image by the display manner changing unit and synthesizing the image with the image not to be detected alone makes it possible to generate learning data of transmissive images of different patterns.SELECTED DRAWING: Figure 4

Description

The present invention relates to a learning data generation device, a learning data generation method, and a learning data generation program.

Object inspection devices are known today, for example in airports, factories, etc., which allow visual inspection of stored items, internal structures, etc., without opening or destroying luggage, manufactured products, or the like. In such an object inspection apparatus, as disclosed in Patent Document 1 (Japanese Unexamined Patent Application Publication No. 11-230918), an X-ray transmission image is used as an example.

The transmission image is colored according to the attenuation rate of the X-rays irradiating the inspected object (depending on the atomic number) and the density of the object. In addition, learning data including transmission images obtained by photographing the stored items or the internal structure and information about the transmission images (such as the type of the object being photographed and whether or not it is an object to be detected) is generated. be. Then, the object inspection apparatus learns using a large amount of learning data, judges the contents or internal structure, etc. shown in the transmission image captured during the inspection, and makes a judgment by applying coloring processing etc. according to the contents, etc. Display an image. Based on this determination image, the inspector judges the contents, internal structure, etc. of the package or manufactured product, etc., and conducts an open inspection, etc., as necessary.

JP-A-11-230918

Here, the object inspection apparatus that has learned using the learning data determines what the contents are based on the result of prior learning. Therefore, in order to obtain accurate inspection results, deep learning based on a large amount of learning data is required. However, the learning data of transmission images is generated through a process of capturing transmission images one by one and manually annotating each of the captured transmission images. For this reason, it takes a lot of time and effort to generate one piece of learning data, and there is a problem that the so-called learning cost increases.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a learning data generating apparatus, a learning data generating method, and a learning data generating program that enable easy generation of learning data for transparent images.

In order to solve the above-described problems and achieve the object, the present invention acquires a detection target image that is a transmission image of a detection target, and a detection non-target image that is a transmission image of an object that is not the detection target. An acquisition unit, a display format changing unit that changes the display format of the detection target in the detection target image, and a detection target image of the detection target whose display format has been changed, combined with the non-detection target image to generate learning data. and a synthesis processing unit that

According to the present invention, it is possible to easily generate learning data for transparent images.

FIG. 1 is a diagram showing the appearance of an object inspection apparatus according to an embodiment. FIG. 2 is a hardware configuration diagram of the object inspection apparatus according to the embodiment. FIG. 3 is a functional block diagram of the object inspection device according to the embodiment. FIG. 4 is a flow chart showing the flow of operations for generating teacher image data in the object inspection apparatus according to the embodiment. FIG. 5 is a diagram showing an example of a detection target image and a non-detection target image. FIG. 6 is a diagram for explaining expansion processing such as rotation and movement of the detection target image. FIG. 7 is a diagram for explaining the color tone change processing of the detection target image. FIG. 8 is a diagram for explaining the process of generating a teacher label image. FIG. 9 is a diagram for explaining the operation of generating teacher image data by transparent composition processing in consideration of a transparent image.

Hereinafter, an object inspection apparatus of an embodiment to which a learning data generation device, a learning data generation method, and a learning data generation program of the present invention are applied will be described with reference to the drawings.

(Appearance configuration)
FIG. 1 is a perspective view showing the appearance of an object inspection device 1 according to an embodiment. As shown in FIG. 1, the object inspection apparatus 1 has a main body 16, an entrance 16a for the object 4 to be imaged, and an exit 16b for the object 4 to be imaged. The object inspection apparatus 1 also has a monitor device 14 . The main body 16 irradiates the object 4 to be imaged with, for example, X-rays while the object 4 to be imaged carried in from the entrance 16 a is moved to the exit 16 b by the roller conveyor 22 , forms a transmission image, and displays it on the monitor device 14 . do.

Here, in the case of this object inspection apparatus 1, when an object is not being inspected and the like is not being inspected, a detection target image, which is a transmission image of an object (detection target), such as a knife, and a handbag, etc. A non-detection target image, which is a transmission image of an object that is not a detection target, is captured. Then, the detection target image is subjected to rotation processing, movement processing, color tone change processing, and the like, and a synthesized image is generated by synthesizing the detection target image with the non-detection target image. Then, a synthesized image and teacher image data including information about this synthesized image (such as the type of object being photographed and whether or not it is an object to be detected) are stored as learning data. As a result, a large amount of learning data with various patterns can be generated easily and in a short time. Details will be described later.

The object inspection apparatus 1 performs so-called deep learning based on the learning data thus formed. During object inspection, the presence or absence of an object to be detected in the imaged object 4 shown in the transmission image is determined based on the learned result, and a determination image as the determination result is supplied to the monitor device 14 . Based on the judgment image displayed on the monitor, the inspector judges whether or not there is an object to be detected in the imaged object 4, and performs an open inspection or the like as necessary.

(Hardware configuration of object inspection device)
FIG. 2 is a diagram showing the hardware configuration of the object inspection apparatus 1. As shown in FIG. As shown in FIG. 2, the object inspection apparatus 1 has an X-ray source 21 for irradiating an object 4 to be imaged with X-rays, and a roller conveyor 22 for moving the object 4 carried in from an entrance 16a to an exit 16b. is doing. The object inspection apparatus 1 also forms an X-ray detector 23 that detects X-rays that have passed through the object 4, and an RGB format transmission image (RGB image) corresponding to the X-ray detection output of the X-ray detector 23. It has an arithmetic processing unit 24 . During object inspection, the transmitted image (RGB image) is used to determine the presence or absence of the object to be detected as described above.

In addition, when the object is not inspected, the arithmetic processing unit 24 performs arithmetic operations such as rotation processing, movement processing, deformation processing, and color tone change processing on the RGB image of the detection target image, as well as synthesis processing with the non-detection target image. Processing is applied to generate a composite image. Then, teacher image data including a synthesized image and information about this synthesized image (such as the type of object being photographed and whether or not it is an object to be detected) is generated. The storage unit 25 stores such teacher image data as learning data. The storage unit 25 also stores a learning data generation program for generating such teacher image data. The arithmetic processing unit 24 performs a teacher image generation operation based on the learning data generation program stored in the storage unit 25 .

(Functional configuration)
FIG. 3 is a functional block diagram of each function realized by executing the learning data generation program stored in the storage unit 25 by the arithmetic processing unit 24 when the object is not inspected. As shown in FIG. 3, the arithmetic processing unit 24 executes a learning data generation program to perform an imaging control unit 31, a storage control unit 32, an input processing unit 33, a display control unit 34, a color tone change processing unit 35, It functions as a teacher label image generation unit 36 and a transparency synthesis processing unit 37 . Note that the input processing unit 33, the display control unit 34, and the color tone change processing unit 35 are examples of the display form change unit.

(Generation operation of teacher image data)
FIG. 4 is a flow chart showing the flow of the teacher image data generation operation performed by the arithmetic processing unit 24 executing the learning data generation program when the object is not inspected. When generating teacher image data, an operator captures a transmission image (detection target image) of a detection target such as a knife, and an object other than a detection target such as a handbag (non-detection target image). ) is imaged. The imaging control unit 31 is an example of an acquisition unit, and controls the X-ray source 21, the roller conveyor 22, and the X-ray detector 23 so as to capture the detection target image and the non-detection target image at the timing specified by the operator. etc.

FIG. 5A is an example of a detection target image. The example of FIG. 5(a) shows a transparent image of a kitchen knife with a metal blade fixed to a wooden handle. FIG. 5(b) is an example of a non-detection target image. The example of FIG. 5(b) shows a transparent image of a handbag in which portable devices, subdivided bags, etc. are stored. Note that the non-detection target image is captured in a state in which the detection target such as a kitchen knife is not stored.

When such detection target images and non-detection target images are captured, the flowchart in FIG. 4 starts, and the process proceeds from step S1. In step S1, the input processing unit 33 performs noise removal processing on the detection target image and the non-detection target image by clustering.

Next, expansion processing is performed on the detection target image (step S2). Specifically, the operator changes the display form of the detection target object included in the detection target image by instructing rotation, movement, enlargement, reduction, deformation, etc. of the detection target image via the operation unit. do. The input processing unit 33 acquires instructions such as rotation and movement performed via the operation unit. The display control unit 34 rotates or moves the detection target image displayed on the monitor device 14 in accordance with the instructions for rotation, movement, etc. acquired by the input processing unit 33 . Furthermore, when the deformation is instructed, the shape of the detection object included in the detection target image is deformed and displayed on the monitor device 14 . By subjecting the detection target image to processing such as enlargement, reduction, deformation, etc., it is possible to generate a detection target image of a detection target object having the same material and a different shape based on one detection target image.

FIG. 6(a) is an example in which the detection target image shown in FIG. 5(a) is rotated by a predetermined amount. FIG. 6B shows an example in which the detection target image is rotated by a predetermined amount and the display position is moved to the upper right. By rotating or moving the detection target image, it is possible to easily and quickly mass-produce detection target images of objects that are detection targets in various postures, positions, and shapes.

Next, in step S3, the input processing unit 33 extracts a detection target from the detection target image based on the snake algorithm of "Open CV (registered trademark) (Open Source Computer Vision Library)", which is an open source library for computers, for example. is extracted, and the pixels within the area surrounded by the contour are extracted as the object to be detected. Similarly, in step S4, the input processing unit 33 extracts a handbag or the like from the non-detection image.

Next, the operator instructs to change the color tone (display form) of an object to be detected, such as a knife, via the operation unit. In general, an object inspection apparatus has a color map (color map) corresponding to the material, identifies the material based on the X-ray attenuation rate that differs depending on the atomic number of the object, and also colors the transmitted light according to the attenuation rate. Generate an image. Therefore, the detection target image, which is a transmission image, has a color corresponding to the material of the detection target object that is actually captured. Therefore, for the kitchen knife, which is the detection target image shown in FIG. instruct. The input processing unit 33 acquires information on the material specified via the operation unit and supplies the information to the color tone change processing unit 35 .

Based on the color map, the color tone change processing unit 35 changes and controls the color of the detection target object displayed on the monitor device 14 to a color tone corresponding to the material acquired by the input processing unit 33 (step S5 ). FIG. 7(a) shows the object to be detected (here, the blade of the kitchen knife) in shades corresponding to the material. When the material of the shaded color tone is aluminum and the operator instructs to change it to ceramic, the color tone change processing section 35 changes the material to ceramic as indicated by hatching in FIG. 7(b). The color tone of the object to be detected is changed and displayed on the monitor. As a result, it is possible to greatly reduce the labor, time, and cost of purchasing detection targets made of different materials and capturing detection target images.

Since the color map differs for each object inspection apparatus, the color tone change processing unit 35 has a color map corresponding to each model in advance, so that the detection target image and the non-detection target image captured by other object inspection apparatuses can be changed. By adjusting the colors of the detection target image and the non-detection target image so as to adapt to the color map for the desired object inspection device, the desired object can be detected by the detection target image and the non-detection target image captured by other object inspection devices. A detection target image and a non-detection target image corresponding to the inspection device can be obtained.

Next, in step S6, as shown in FIG. 8, the teacher label image generation unit 36 generates a teacher label image in which information about the detection target image is attached to the detection target image whose color has been changed in step S5. The storage control unit 32 stores this teacher label image in the storage unit 25 .

Next, in steps S7 to S11, the transparency processing unit 37, which is an example of a composition processing unit, performs transparency synthesis processing of the detection target image and the non-detection target image in consideration of the characteristics of the transmission image.

In general, when image synthesis is performed, the brightnesses f ₁ and f ₂ of each image to be synthesized are multiplied by coefficients α and 1−α, respectively, and finally added to calculate the brightness g. A synthesis technique called "alpha blending" is often used.

g=αf ₁ +(1−α)f ₂

However, when synthesizing transmissive images, it is necessary to adjust the transmittance of each object before synthesizing. In the case of the synthesis processing technology disclosed in the reference (Japanese Patent No. 4471032), the luminance y when an object with a different thickness x is imaged for each X-ray apparatus is formulated, and the brightness y is obtained by imaging with a plurality of types of X-ray apparatus. Device-specific parameters are calculated from the results. Then, by normalizing the result of imaging by each X-ray device from this calculation result, it is possible to synthesize transmission images that are imaged by different X-ray devices.

y = ae - ^{bx (a is a coefficient specific to the X-ray device, e -bx is} the X-ray transmittance)

For example, the brightness _y12 when two plates having a thickness of x are superimposed is calculated by the following formula.

y ₁₂ =ae ^−bx e ^−bx

Here, when synthesizing transmission images of X-ray apparatuses of the same model, it is not necessary to strictly obtain the transmittance unlike the synthesis processing technique disclosed in the reference (Japanese Patent No. 4471032). For this reason, in the case of the object inspection apparatus 1 of the embodiment, simple synthesis processing is performed as described below.

That is, in order to suppress the overflow that occurs in the above-described alpha blending when calculating the luminance value, the transparency processing unit 37 sets the luminance of the n-th composite pixel position i and j among the N images to be composited to When In is assumed to be In, before synthesizing each pixel, In is subtracted from the maximum luminance value Imax in advance, the luminance is inverted (step S7), and then the detection target image and the non-detection image are synthesized. A composite image is formed (step S8).

Next, the transparency processing unit 37 adjusts the brightness of the synthesized image (upper limit processing) so that it is equal to or less than the maximum value corresponding to the color map of the object inspection apparatus that performs deep learning using the synthesized image ( step S9).

Next, the transparency synthesis processing unit 37 subtracts the maximum luminance value Imax again to negatively invert the luminance and returns it, thereby generating and outputting a synthesized image with the luminance after synthesis g(i, j). (step S10, step S11). FIGS. 9A and 9B are examples of composite images generated in this way. As shown in FIGS. 9A and 9B, by rotating the detection target image and moving the position of the detection target image and synthesizing it with the non-detection target image, a synthesized image of many patterns can be easily obtained. can be formed. FIG. 9(a) is a composite image of a state in which a kitchen knife, which is a detection target, is housed approximately in the center of a handbag, which is not a detection target. FIG. 9(b) is a composite image in which a kitchen knife, which is a detection target, protrudes from the upper right of a non-detection handbag.

The storage control unit 32 stores and controls the storage unit 25 as learning data, teacher image data obtained by adding information about the generated synthetic image to the synthetic image. This completes the operation of generating teacher image data shown in the flowchart of FIG.

Such synthesis processing can be represented by the following formula.

In this formula, "Imax-In(i, j)" is an operation corresponding to the luminance inversion process (negative inversion) described above. Also, the calculation of min(Σ) is a calculation corresponding to the combining process. Further, the calculation of Imax-min(Σ) is a calculation corresponding to the negative reversal processing in step S10.

(Effect of Embodiment)
As is clear from the above description, the object inspection apparatus 1 of the embodiment prepares an image to be detected and an image not to be detected, which are transmitted images by radiation such as X-rays, and rotates and moves the image to be detected. The orientation, position, size, etc. of the image to be detected are adjusted by applying expansion processing such as deformation. Then, the detection target image adjusted in this manner is combined with the non-detection target image to form teacher image data. As a result, by changing the orientation, position, size, etc. of the detection target image, it is possible to generate various patterns of the detection target image. Many synthetic images (teaching images) can be generated easily and in a short time. Therefore, the learning cost can be kept low, and the learning accuracy can be improved.

Further, when combining the detection target image and the non-detection target image, the detection target image and the non-detection target image are subjected to negative reversal processing before combining. As a result, it is possible to prevent the problem that the white level is saturated due to the synthesis processing of the RGB image, which is the detection target image and the non-detection target image.

Further, the brightness of the synthesized image formed by the synthesizing process is adjusted so that it is equal to or less than the maximum value corresponding to the color map of the object inspection device that performs deep learning using the teacher image (upper limit processing). Since the color map differs for each object inspection device, by adjusting the brightness of the synthesized image using the color map for the desired object inspection device, a teacher image corresponding to the desired object inspection device (corresponding to each model) can be obtained. can be formed.

Finally, the above-described embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention.

For example, in the above embodiment, two-dimensional images are synthesized, but three-dimensional images may be synthesized.

Further, in the above-described embodiment, the case where the learning data generation processing is provided in the object inspection device itself has been described as an example. may be performed by an external device.

Furthermore, in the above-described embodiment, the transmission image obtained by irradiating X-rays is used as an example of the learning data generation processing. You may make it go to the original.

In addition, in the above-described embodiment, the color tone change processing unit 35 adjusts the colors of the detection target image and the non-detection target image before performing the synthesizing process using a desired color map for the object inspection apparatus. Although described, it is not limited to this, and may be performed for a synthesized image stored as teacher image data formed by synthesis processing.

In addition, the embodiments and modifications of the embodiments are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 object inspection device 4 imaging object 14 monitor device 16 main body of object inspection device 16a imaging object entrance 16b imaging object exit 21 X-ray source 22 roller conveyor 23 X-ray detector 24 arithmetic processing unit 25 storage unit 31 imaging control unit 32 Storage control unit 33 Input processing unit 34 Display control unit 35 Color tone change processing unit 36 Teacher label image generation unit 37 Transparency synthesis processing unit

Claims (7)

an acquisition unit that acquires a detection target image that is a transmission image of a detection target and a non-detection target image that is a transmission image of an object that is not the detection target;
a display form changing unit that changes a display form of the detection object in the detection target image;
a synthesis processing unit that synthesizes the detection target image of the detection target whose display form has been changed with the non-detection target image to generate learning data;
A learning data generation device having 2. The learning data generation according to claim 1, wherein the display form changing unit is capable of changing at least one display form out of rotation, movement, deformation, enlargement, and reduction of the detection object. Device. 3. The learning data generation device according to claim 1, wherein the display mode changing unit changes the color tone of the detection object to a color tone corresponding to the specified material. 4. The display mode changing unit performs brightness adjustment processing on the detection target image and the non-detection target image using a color map corresponding to a predetermined model. The learning data generation device according to any one of 4. Among claims 1 to 3, wherein the display form changing unit performs brightness adjustment processing on the synthesized image output by the synthesis processing unit using a color map corresponding to a predetermined model. , The learning data generation device according to any one of the above. an acquisition step of acquiring a detection target image that is a transmission image of a detection target and a non-detection target image that is a transmission image of an object that is not the detection target;
a display form changing step of changing a display form of the detection object in the detection target image;
a synthesizing step of synthesizing the detection target image of the detection target whose display mode has been changed with the non-detection target image to generate learning data;
A learning data generation method having the computer,
an acquisition unit that acquires a detection target image that is a transmission image of a detection target and a non-detection target image that is a transmission image of an object that is not the detection target;
a display form changing unit that changes a display form of the detection object in the detection target image;
A learning data generating program that functions as a synthesizing unit that generates learning data by synthesizing the detection target image of the detection target whose display mode has been changed with the non-detection target image.

Images