JP7034529B1 - Training model generation method, learning model, checking device, checking method and computer program - Google Patents

Abstract

Abnormalities in the inspected object can be detected with high accuracy. The method of generating the learning model is a step of irradiating the object to be inspected with an electromagnetic wave, a step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and a pixel of a pixel of the processed image obtained by image processing the image. The step of identifying the blob contained in the inspected object based on the value, the mask image masked to the position and shape corresponding to the identified blob, the offset information of the mask part in the processed image, and the object corresponding to the blob. A step of acquiring anomaly information including information on an abnormality of an inspected object having a blob label of the type, a step of acquiring teacher data including a mask image and anomaly information, and a step of irradiating the inspected object with an electromagnetic wave to be inspected. When the image acquired according to the electromagnetic wave transmitted through the object is input, the step to acquire the heat map image in which the position and shape of the blob are displayed as a heat map on the processed image based on the teacher data, and the heat. It includes a step of generating a training model that outputs an inspection image including a heat map image by synthesizing the map image with the processed image.

Description

The present invention relates to object detection, in particular, a method for generating a learning model for inspecting fashion accessories, foods, industrial products, etc. as an inspected object and detecting foreign substances mixed in the inspected object, a learning model, an inspection device, and the like. Checking methods and computer programs.

Conventionally, non-destructive inspection waves such as X-rays, terahertz waves, infrared rays, and visible light have been used for inspections of fashion accessories, foods, industrial products, etc., and inspections of packages containing contents in packaging materials. Devices capable of non-contact and high-speed foreign matter inspection / detection / detection are known (see, for example, Patent Document 1 and Patent Document 2). Such a device irradiates the inspection object with an inspection wave such as an electromagnetic wave, measures the amount of the inspection wave transmitted through the inspection object, and acquires the inspection wave transmission image of the inspection object as an inspection image.

In the inspection using inspection waves, for example, X-rays, the concentration of the object to be inspected and the concentration of foreign matter are measured from the X-ray transmission image, and the concentration threshold for foreign matter extraction is set in advance. The presence or absence of foreign matter contamination is determined based on the area, shape, etc. of the singular portion having a concentration exceeding the threshold value existing in the concentration distribution. The presence or absence of abnormalities in the shape of the object to be inspected was determined based on the area and perimeter of the blob (lump) in the inspection image. By counting the pixels that make up the blob, the area of the blob is calculated, and the perimeter of the blob is calculated based on the array of background pixels that form the periphery of the blob. In general, the amount of X-ray transmission correlates with the density of the object to be irradiated with X-rays. The lower the density of the object, the higher the transmission ability of X-rays, and the higher the density of the object, the higher the X-rays. Permeability is low. Therefore, in principle, the larger the density difference between the inspected object and the foreign matter, the larger the density difference between the inspected object and the foreign matter in the X-ray transmission image, and stable foreign matter detection becomes possible.

Japanese Unexamined Patent Publication No. 2018-138899 Japanese Patent No. 5876116

As described above, in the conventional foreign matter inspection method, the presence or absence of foreign matter is determined by comparing with the threshold value, but the difference in pixel value between the normal region and the foreign matter is small depending on the type of the object to be inspected and the foreign matter. There were problems such as inspectability and deterioration of foreign matter detection accuracy.

The present invention has been made in view of such circumstances, and provides a learning model generation method, a learning model, an inspection device, an inspection method, and a computer program capable of detecting an abnormality of an inspected object with high accuracy. The purpose is.

The method for generating a learning model according to the present invention includes a step of irradiating an object to be inspected with an electromagnetic wave, a step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and image processing on the image. A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the processed image, a mask image masked to a position and shape corresponding to the identified blob, and the blob are included. The background image which is not the processed image and the abnormality information including the offset information of the mask portion in the processed image and the information about the abnormality of the inspected object having the blob label which is the type of the object corresponding to the blob are acquired. The step, the step of acquiring the teacher data including the mask image, the background image, and the abnormality information, and the image acquired according to the electromagnetic wave transmitted through the inspected object by irradiating the inspected object with an electromagnetic wave are input. When this is done, a step of acquiring a heat map image in which the position and shape of the blob are displayed as a heat map on the processed image based on the teacher data, and the heat map image are combined with the processed image. This includes a step of generating a learning model that outputs an inspection image including the heat map image, and in the background image, an image different from the shape of the blob is synthesized at an arbitrary position of the processed image. ..

In the learning model according to the uniformity of the present invention, an object to be inspected is irradiated with an electromagnetic wave, an image is acquired according to the electromagnetic wave transmitted through the object to be inspected, and the pixels of the processed image obtained by image-processing the image. Based on the pixel value of, the blob contained in the inspected object is identified, and the mask image masked to the position and shape corresponding to the specified blob, and the background image which is the processed image not including the blob. On the processed image, an input layer into which the offset information of the mask portion in the processed image and the abnormality information including the information regarding the abnormality of the inspected object having the blob label which is the type of the object corresponding to the blob are input. An output layer that outputs an inspection image including the heat map image by acquiring a heat map image in which the position and shape of the blob are displayed as a heat map and synthesizing the heat map image with the processed image. The heat map image is provided with the mask image and an intermediate layer whose parameters are learned based on the abnormality information, and when the mask image and the abnormality information are input to the input layer, the heat map image is calculated by the intermediate layer. The computer is made to function to output the inspection image including the above from the output layer, and when the background image is input to the input layer, the inspection image not including the heat map image is subjected to calculation by the intermediate layer. To function the computer to output from the output layer.

The inspection device according to the present invention includes an irradiation unit that irradiates an object to be inspected with an electromagnetic wave, an image generation unit that generates an image according to the electromagnetic wave transmitted through the object to be inspected, and an image processing on the image. Corresponds to the image processing unit that acquires the processed image, the target identification unit that specifies the blob included in the inspected object based on the pixel value of the pixel of the processed image, and the specified blob. The mask image masked in position and shape, the background image which is the processed image not including the blob, the offset information of the mask portion in the processed image, and the blob label which is the type of the object corresponding to the blob. An acquisition unit that acquires anomaly information including information on an abnormality of the inspected object as teacher data, and an image obtained by irradiating the inspected object with an electromagnetic wave and acquiring the image in response to the electromagnetic wave transmitted through the inspected object are input. In this case, the mask image and abnormality information acquired from the acquisition unit are input to the learning model that outputs information on the abnormality of the object to be inspected, and the position and shape of the blob are used as a heat map on the processed image. The background image is provided with an abnormality information acquisition unit that acquires the displayed heat map image and an output unit that outputs an inspection image including the heat map image by synthesizing the heat map image with the processed image. , It is provided that an image different from the shape of the blob is synthesized at an arbitrary position of the processed image.

The inspection method according to the present invention includes a step of irradiating an object to be inspected with an electromagnetic wave, a step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and a process of performing image processing on the image. A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the image, a mask image masked to a position and shape corresponding to the identified blob, and the process not including the blob. A step of acquiring a background image which is an image, an abnormality information including offset information of the mask portion in the processed image, and information about an abnormality of the inspected object having a blob label which is a type of an object corresponding to the blob. The mask image When the step of acquiring the teacher data including the background image and the abnormality information and the image acquired according to the electromagnetic wave transmitted through the inspected object by irradiating the inspected object with an electromagnetic wave are input. , The step of acquiring a heat map image in which the position and shape of the blob are displayed as a heat map on the processed image based on the teacher data, and by synthesizing the heat map image with the processed image. A computer is made to execute a process including a step of generating a learning model that outputs an inspection image including a heat map image, and an image different from the shape of the blob is synthesized at an arbitrary position of the processed image in the background image. ing.

The computer program according to the uniformity of the present invention includes a step of irradiating an object to be inspected with an electromagnetic wave, a step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and a process of performing image processing on the image. A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the image, a mask image masked to a position and shape corresponding to the identified blob, and the process not including the blob. A step of acquiring a background image which is an image, an abnormality information including offset information of the mask portion in the processed image, and information about an abnormality of the inspected object having a blob label which is a type of an object corresponding to the blob. The mask image When the step of acquiring the teacher data including the background image and the abnormality information and the image acquired according to the electromagnetic wave transmitted through the inspected object by irradiating the inspected object with an electromagnetic wave are input. , The step of acquiring a heat map image in which the position and shape of the blob are displayed as a heat map on the processed image based on the teacher data, and by synthesizing the heat map image with the processed image. A computer is made to execute a process including a step of generating a learning model that outputs an inspection image including a heat map image, and an image different from the shape of the blob is synthesized at an arbitrary position of the processed image in the background image. ing.

According to the present invention, in the inspection image of the object to be inspected, high accuracy is achieved even when the difference in pixel value between the normal part and the abnormal part is small, the shape of the foreign matter is peculiar, and the size of the foreign matter is extremely small. Therefore, the normal part and the abnormal part can be discriminated, and the abnormal state such as foreign matter contamination can be specified.
Therefore, it is possible to accurately detect information regarding the presence or absence of an abnormality such as the presence or absence of a foreign substance.
It should be noted that the present invention frees the user from long-term, repetitive simple work that requires a great deal of concentration, reduces fatigue, frees people from accidents associated with fatigue, and reduces health hazards caused by foreign matter contamination accidents. Therefore, it will be possible to contribute to Goal 3 “Health and Welfare for All” and Goal 8 “Rewarding Work and Economic Growth” of the Sustainable Development Goals (SDGs) led by the United Nations.

It is a schematic diagram which shows the structure of the abnormality inspection system which concerns on Embodiment 1 of this invention. It is a block diagram which shows the structure of the abnormality inspection system which concerns on Embodiment 1 of this invention. FIG. 6 is a schematic diagram showing a known embodiment of a Mask R-CNN convolutional neural network. It is a schematic diagram which shows an example of the record layout of the inspection image DB 151 which concerns on Embodiment 1 of this invention. It is a schematic diagram regarding the generation processing of the abnormality detection model which concerns on Embodiment 1 of this invention. It is a flowchart about the generation process of the abnormality detection model which concerns on Embodiment 1 of this invention. It is a flowchart of the processing procedure of the abnormality detection processing by the abnormality inspection system which concerns on Embodiment 1 of this invention. It is a schematic diagram which shows the processed image of the object to be inspected. It is a schematic diagram which shows the mask image of the object to be inspected. It is a schematic diagram which shows the processed image of the inspected object which is not mixed with foreign matter. It is a schematic diagram which shows the marking image for carrying out the background image processing. It is a schematic diagram which shows the processed image after performing the background image processing. It is a schematic diagram which shows the processed image of the object to be inspected which a foreign substance is mixed. It is a schematic diagram which shows the teacher data creation processing when a plurality of foreign matters exist in a processed image. It is a schematic diagram which shows the inspection image of the object to be inspected after performing the foreign matter detection process. It is a schematic diagram which shows the object to be inspected which is the object of inspection. It is a block diagram which shows the structure of the abnormality inspection system which concerns on Embodiment 2 of this invention. It is a schematic diagram regarding the occurrence condition of the relearning process which concerns on Embodiment 2 of this invention. It is a flowchart which shows the processing procedure of the relearning process which concerns on Embodiment 2 of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
For convenience of explanation, the same reference numerals given to the respective figures indicate the same parts or equivalent parts unless otherwise specified, and are intended to be elements having the same structure and / or function, more than necessary. Detailed explanation may be omitted.
Further, duplicate description will be simplified or omitted as appropriate.
Further, in the present embodiment, the numerical value range may be represented by using the symbol "-", but the numerical values described before and after "-" are included in the numerical value range.
(Embodiment 1)
[Abnormality inspection system]

FIG. 1 is a schematic diagram showing the configuration of the abnormality inspection system 10 according to the first embodiment of the present invention, and FIG. 2 is a block diagram showing the configuration of the abnormality inspection system 10.
The abnormality inspection system 10 includes an information processing device 1 and an X-ray inspection device 2. In the first embodiment, the object 4 to be inspected is placed on the conveyor belt 61 of the transfer unit (conveyor) 6 of the X-ray inspection device 2 in a dispersed state, and the image to be inspected is imaged using X-rays on the object 4 to be inspected. The abnormality inspection system 10 for detecting an abnormality based on the image obtained will be described. Examples of the object 4 to be inspected include shoes, bags, clothing, food and the like. The object 4 to be inspected is not limited to the objects such as shoes listed above, and can be applied as long as the shape can be detected by an image. Further, the object 4 to be inspected is not limited to the object itself such as the shoes listed above, and can be applied to the object stored in the packaging material or the object placed on the tray. The information processing device 1 and the X-ray inspection device 2 can be configured even if they are separated or integrated.
[X-ray inspection device]

The X-ray inspection device 2 includes an irradiation unit 3, a conveyor belt 60, a light emitting unit 24, a light receiving unit 25, a controller 27, and a display unit 22. The display unit 22 includes a first display unit 41 and a second display unit 42.
Further, the X-ray inspection device 2 includes a horizontally long rectangular parallelepiped lower housing 23 and an upper housing 21 provided on the lower housing 23 and smaller than the lower housing 23.
A transport unit 6 is provided in the lower housing 23. The transport unit 6 includes a transport belt 60, a transport belt drive unit 61, and a transport belt control unit 62. The transport belt control unit 62 receives a command including a distance, a speed, and the like for transporting the object to be detected 4 from the control unit 11, and controls the operation of the transport belt drive unit 61 based on the command.

The irradiation unit 3 is housed in the upper housing 21. The irradiation unit 3 has an irradiation body 31 and an irradiation control unit 32. The irradiation control unit 32 receives a command from the control unit 11 including detailed settings such as the X-ray irradiation time for irradiating X-rays and the X-ray intensity, and controls the X-rays emitted from the irradiator 31 based on the command. .. Further, the irradiation unit 3 emits X-rays from an arbitrary angle with respect to the object 4 to be inspected (for example, 0 °, 25 °, 45 °, etc. with respect to the coordinate axes X, Y, Z shown in FIG. 1, respectively). It can be controlled to irradiate. The presence or absence of foreign matter can be confirmed more accurately by the X-ray irradiation image obtained by irradiation at different angles. A display unit 22 is provided on the front surface of the upper housing 21. The display unit 22 displays an X-ray image or the like of the object 4 to be inspected generated by the image generation unit 111, which will be described later. The display unit 22 is, for example, a display device such as an LCD (Liquid Crystal Display), and the first display unit 41 and the second display unit 42 included in the display unit 22 are divided into screens in the same display device. It may be a separate display device.
The upper housing 21 is provided with a detection unit 7. The detection unit 7 is called a TDI (Time Delay Integration) camera or TDI sensor, and has a scintillator that emits fluorescence according to the intensity of X-rays and a large number of photodiodes that detect the fluorescence emitted by the scintillator. is doing.

In the TDI camera, the photodiodes are linearly arranged in the X direction orthogonal to the Y direction, which is the moving direction of the object to be inspected, to form one detection line, and a plurality of these detection lines are provided side by side. The plurality of detection lines are arranged side by side in parallel toward the Y direction, which is the moving direction of the object to be inspected. In the housing of one TDI camera, a plurality of photodiodes are regularly arranged side by side in the X direction and the Y direction, and one line of detection lines is composed of the photodiodes arranged in the X direction.

First, at the time of inspection, the user puts the inspected object 4 on the conveying belt 60 and operates the controller 27 to convey the inspected object 4 until it is located below the irradiation unit 3. When the object 4 to be inspected is conveyed below the X-ray irradiation unit 3, the irradiation unit 3 irradiates the X-ray to the object 4 to be inspected. The irradiated X-rays pass through the object 4 to be inspected and reach the light emitting portion 24 below the transport belt 60. The light emitting unit 24 emits light according to the amount of X-rays that have reached it, and the light receiving unit 25 receives the emitted light. Then, the X-ray inspection device 2 generates an X-ray irradiation image from the light received by the light receiving unit 25 by the image generation unit 111 in the control unit 11 and displays it on the display unit 22.

The user visually observes the X-ray irradiation image displayed on the display unit 22 and determines whether or not foreign matter is mixed in the object to be inspected 4. The basic configuration of the X-ray inspection device 2 is the same as that of the conventionally known X-ray inspection device, and is not limited to the present embodiment.

Next, FIG. 2 shows an outline of a block diagram showing the relationship of each component of the X-ray inspection apparatus 2 according to the present invention. The X-ray inspection device 2 is centrally managed by the control unit 11, and each component included in the X-ray inspection device 2 is controlled based on a command from the control unit 11. The control unit 11 includes an image generation unit 111, an image processing unit 112, a target identification unit 113, an acquisition unit 114, an abnormality information acquisition unit 115, an output unit 116, a control display unit 117, and a discrimination unit 118.
The control unit 11 is mounted on an existing computer (equipped with a CPU, a storage device, or the like). Further, the control unit 11, the image generation unit 111, the image processing unit 112, the target identification unit 113, the acquisition unit 114, the abnormality information acquisition unit 115, the output unit 116, the control display unit 117, and the discrimination unit 118 are realized on software. ing.

The conveyor belt 60 includes a conveyor belt drive unit 61 and a conveyor belt control unit 62. The transport belt control unit 62 receives a command including a distance, a speed, and the like for transporting the object 4 to be inspected from the control unit 11, and controls the operation of the transport belt drive unit 61 based on the command.

The light emitting unit 24 receives X-rays emitted from the irradiating body 31 and passed through the object 4 to be inspected, and emits light. Then, the light receiving unit 25 receives the emitted light and transmits it to the image generation unit 111.

The image generation unit 111 generates an X-ray irradiation image through which the object 4 to be inspected is seen through from the light transmitted from the light receiving unit 25. It is also possible to generate a color imaged X-ray irradiation image by generating a black-and-white image according to the amount of transmitted light and adding color according to a black-and-white density value or the like.

An existing processing method can be used for this image processing, and the processing applied to the image can be appropriately changed so that the user can easily find the foreign matter mixed in the object 4 to be inspected. For example, it is possible to change all kinds of setting information related to image processing, such as changing the color attached to an image, changing the brightness and brightness, changing the resolution, and changing the image processing software.

The data file generation unit 220 (not shown) displays the X-ray irradiation image generated by the image generation unit 111 at least as equipment setting information, image generation processing information, and detected object information of the X-ray inspection apparatus 2 at the time of inspection. , And generate a data file saved with inspection information including any of the user information.

The data file generated by the data file generation unit 220 is stored in a format that can be read by another image display device, for example, a JPEG format or an MPEG format. Therefore, an existing image display device (computer, tablet terminal, etc.) equipped with image software capable of reading these formats can display an X-ray irradiation image as in the conventional case.

Further, the auxiliary storage unit 15 described in detail below can store the data file generated by the data file generation unit 220. The auxiliary storage unit 15 may be an external storage medium (CD, DVD, etc.) or the like, in addition to the main storage unit (RAM, etc.) provided in the control unit 11.

The data reading unit 221 (not shown) can read the data file stored in the auxiliary storage unit 15. Then, the inspection information read by the data reading unit 221 is transmitted to the control unit 11, and the control unit 11 sends a command to the irradiation unit 3, the transport belt 60, or the image generation unit 111.
[Information processing device]

The information processing device 1 is an information processing device capable of transmitting and receiving various types of information processing and information, and is a computer. Computers include personal computers (personal computers), workstations, server computers (servers), mainframes, supercomputers (spacons), ultracomputers, minicomputers (minicons), and office computers, depending on the purpose of use, specifications, specifications, performance, etc. (Off-con), pocket computer (pokecon), microcomputer (microcomputer), mobile information terminal (PDA), sequencer (PLC: programmable logic controller), etc. are classified, but all of these are applicable to information processing equipment. It is possible. The X-ray inspection device 2 is connected to the information processing device 1 via a network N such as a LAN or the Internet, and can communicate with the information processing device 1. It is also possible for a cloud computer, which is communicably connected via a network N such as the Internet, to execute the processing of the information processing apparatus 1.
In the present embodiment, the information processing device 1 is a personal computer, and will be described below.

The information processing apparatus 1 acquires an image obtained by capturing an image of the object to be inspected 4, and performs a process of detecting the presence or absence of an abnormality in the object to be inspected 4 based on the image. In the present embodiment, the information processing apparatus 1 detects the presence or absence of an abnormality by using the abnormality detection model 152, which will be described later, which has been learned so as to detect (identify) the abnormality of the inspected object 4 from the image by machine learning. The anomaly detection model 152 is expected to be used as a program module that is a part of artificial intelligence software.
The information processing apparatus 1 inputs a processed image 50, which is an image processed based on an image obtained by capturing an image of the inspected object 4 using the X-ray inspection apparatus 2, into the abnormality detection model 152, and an abnormality of the inspected object 4 is obtained. The identification result indicating the presence or absence of is acquired as an output. The control unit 11 of the information processing apparatus 1 performs an operation on the mask image 51 and / or the background image 52 and the like input to the input layer according to the command from the abnormality detection model 152, and outputs an identification result indicating the presence or absence of an abnormality. It works like that. The presence / absence of abnormality corresponds to the presence / absence of abnormality (deformation) in the shape of the object 4 to be inspected, the presence / absence of foreign matter, and the like.
Hereinafter, in the present embodiment, the type of abnormality of the object to be inspected 4 is the presence or absence of a foreign substance, and the information processing apparatus 1 inputs a mask image 51 and / or a background image 52 or the like to the abnormality detection model 152, and the object to be inspected 4 is inspected. A case where the position, shape, etc. of the foreign matter are acquired as the heat map image 53 when the foreign matter is mixed in the foreign matter will be described.
Subsequently, the information processing apparatus 1 outputs an inspection image 54 in which the heat map image 53 is combined with the processed image 50.
The information processing apparatus 1 selects whether or not to return the inspected object 4 to the manufacturer according to the acquired abnormality information such as the heat map.
Here, the abnormality information is the offset information of the heat map portion and the object corresponding to the blob in the inspection image 54 including the heat map image 53 at the position and shape corresponding to the blob identified as a foreign object on the processed image 50. Contains information on abnormalities in the inspected object such as blob labels of the type.

The information processing device 1 includes a control unit 11, a main storage unit 12, a communication unit 13, an input unit 14, and an auxiliary storage unit 15.
The control unit 11 has an arithmetic processing unit such as one or a plurality of CPUs (Central Processing Units), MPUs (Micro-Processing Units), GPUs (Graphics Processing Units), and is a program 153 stored in the auxiliary storage unit 15. Is read and executed to perform various information processing, control processing, and the like related to the information processing unit 1. Each functional unit of FIG. 2 is executed by the control unit 11 operating based on the program 153.

The control unit 11 includes an image generation unit 111, an image processing unit 112, a target identification unit 113, an acquisition unit 114, an abnormality information acquisition unit 115, an output unit 116, a control display unit 117, and a discrimination unit 118 as functional units.
The image generation unit 111 passes through the object 4 to be inspected and generates an X-ray image based on the X-rays detected by the detection unit 7.
The image processing unit 112 performs image processing on the X-ray image by a smoothing filter, a feature extraction filter, or the like, and generates a processed image 50.
The target specifying unit 113 analyzes the blob of the inspected object 4 based on the processed image 50, and identifies the blob.
The acquisition unit 114 includes a specific image acquisition unit 124 and a specific information acquisition unit 134. The specific image acquisition unit 124 acquires a mask image 51 masked to a position and shape corresponding to the blob specified by the target identification unit 113. Here, the mask image 51 is an image in which a portion of a foreign substance to be found is masked (painted in white or black, and the others are painted in the opposite color). Further, the specific image acquisition unit 124 acquires a background image which is a processed image 50 that does not include a blob. The method of generating the background image will be described in detail below.
The specific information acquisition unit 134 acquires abnormality information including the offset information of the mask portion in the processed image 50 and the information regarding the abnormality of the inspected object 4 having the blob label which is the type of the object corresponding to the blob. Further, the mask image 51, the background image 52, and the abnormality information are input to the abnormality detection model 152 as teacher data.
The abnormality information acquisition unit 115 irradiates the inspected object 4 with an electromagnetic wave, and when an image acquired according to the electromagnetic wave transmitted through the inspected object 4 is input, the abnormality information acquisition unit 115 outputs information on the abnormality of the inspected object 4. The mask image 51 and the abnormality information acquired from the acquisition unit 114 are input to the model, and the heat map image 53 in which the position and shape of the blob is displayed as a heat map on the processed image 50 is acquired.
The output unit 116 synthesizes the heat map image 53 acquired by the abnormality information acquisition unit 115 on the processed image 50, and outputs the inspection image 54 including the heat map image 53.

The control display unit 117 displays setting input information, setting change information, setting confirmation information, and the like for various functional units constituting the abnormality inspection system shown in FIG. Further, the control display unit 117 can add, delete, and edit the data and the program included in the inspection image DB 151, the abnormality detection model 152, and the program 153 included in the auxiliary storage unit 15.
The discriminating unit 118 selects whether to return the inspected object 4 to the manufacturer based on the inspection image 54 including the heat map image 53 relating to the inspected object 4 and the abnormality information, or stops the transport unit 6. , Marking a mark indicating whether or not an abnormality has been detected in the object 4 to be inspected.

The main storage unit 12 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. Remember. The communication unit 13 is a communication module for performing processing related to communication, and transmits / receives information to / from the outside.

The auxiliary storage unit 15 is a large-capacity memory, a hard disk, or the like, and stores the program 153 and other data necessary for the control unit 11 to execute the process. Further, the auxiliary storage unit 15 stores the inspection image DB 151 and the abnormality detection model 152. The inspection image DB 151 is a database that stores information on the inspected object 4 that is an abnormality detection target.

The program 153 stored in the auxiliary storage unit 15 may be provided from a recording medium in which the program 153 is readablely recorded. The recording medium is, for example, a portable memory such as a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, a micro SD card, and a compact flash (registered trademark). The program 153 recorded on the recording medium is read from the recording medium using a reading device (not shown in the figure) and stored in the auxiliary storage unit 15. When the information processing device 1 includes a communication unit capable of communicating with an external communication device, the program 153 stored in the auxiliary storage unit 15 may be provided by communication via the communication unit.

An external storage device connected to the information processing device 1 can be applied to the auxiliary storage unit 15. Further, a multi-computer composed of a plurality of computers can be applied to the information processing apparatus 1, and a virtual machine virtually constructed by software can also be applied.

FIG. 3 is a schematic view showing a mask image 51 masked to a position and shape corresponding to a blob specified by the target identification unit 113. The target identification unit 113 of the control unit 11 detects blobs by pattern matching, edge detection, R-CNN (Regions with Convolutional Neural Networks), or the like. In the mask image 51, the mask portion is shown as white and the other background is shown as black.
The target identification unit 113 according to the present invention uses the architecture of Mask R-CNN (Mask Regions with Convolutional Neural Networks).
[Mask R-CNN]

The Mask R-CNN network is known and its architecture is shown in FIG. Referring to FIG. 3, the Mask R-CNN is a CNN divided into two sets, and the processing is divided into stage 1 and stage 2.

"Stage 1" allows for preprocessing of the input image and essentially includes a first feature pyramid network (FPN) subnet, the operation of which will be described in more detail below.

"Stage 2" completes and terminates the detection (produces the desired output, ie, the segmentation mask of the detected element of interest, and / or one or more detection boxes and / or classes).

Stage 1 of the Mask R-CNN also includes a region proposal network (RPN) type third subnet that is also a detection network, and a trimming module (“ROI alignment, ROI means“ target region ”). The FPN identifies potential target areas in the FPN output feature map (ie, likely to contain the target element), and the trimming module uses these target areas to facilitate the operation of the detection network. "Realign" the feature map according to the coordinates.

The CNN also includes at least a first FPN type network and a second detection network type subnetwork, and optionally a third RPN type subnetwork and trimming module.

Here, the FPN (first subnetwork) is the most important part of the Mask R-CNN network.

FPN differs in that it consists of an ascending branch (“bottom-up”), a descending branch (“top-down”), and a lateral connection between the ascending and descending branches.

The ascending branch, also known as the backbone of the network as a whole, can be of many types of conventional feature extraction networks, especially conventional CNNs (convolutional layer direct continuous block CONV, batch normalization layer BN, and nonlinear layer NL). Is. The backbone extracts from the input image multiple initial feature maps that represent the input image on different scales. More precisely, the backbone consists of a plurality of convolutional blocks, whereby the first block produces the first initial feature map from the input image, and then the second block is the second initial. A feature map is generated for the first initial feature map, and so on.

Traditionally, for convolutional neural networks, each contiguous map has a smaller scale (ie, lower resolution makes the feature map "smaller" and therefore less detailed), but at an increasingly higher level. It is understood that the semantic depth is further increased because the image of the structure of is captured. Specifically, the initial feature map increases the number of channels as its size decreases.

In practice, the pooling layer is placed between the two blocks, reducing the size by half, and the number of convolutional layer filters used from one block to the other (generally 3x). 3 convolutions) are increased (preferably doubled), for example, in the case of 5 levels, for example, 32, 64, 128, 256, and 512 consecutive channel numbers, and 512x512, 256x256, 128x128, 64x64, And there are 32x32 contiguous map sizes (for 512x512 input images).
[Learning model generation process]

FIG. 4 is a schematic diagram showing an example of the record layout of the inspection image DB 151. The inspection image DB 151 includes a mask image ID string, an offset information column, a blob label column, and a mask image string. The mask image ID indicates information for identifying the mask image 51. The offset information (X, Y, W, H) indicates coordinate information regarding the position of the mask portion on the mask image 51 of the object 4 to be inspected placed on the transport belt 60 in the X-ray irradiation region. Here, X and Y indicate information regarding the X coordinate and the Y coordinate, and W and H indicate information regarding the horizontal and vertical directions of the mask image 51. The blob label (foreign matter type label) indicates the type of abnormal state corresponding to the mask portion, and corresponds to scratches, iron balls, needles, stains, and the like.
Further, the inspection image DB 151 stores a background image 52, which is a processed image 50 that does not include a blob. Further, the inspection image DB 151 records the inspection image 54 including the heat map image 53 as a foreign matter mixed image, and the inspection image 54 not including the heat map image 53 as a foreign matter non-mixed image (background image).

FIG. 5 shows a schematic diagram regarding the generation process of the abnormality detection model 152. FIG. 5 conceptually shows a process of performing machine learning to generate an abnormality detection model 152.

The information processing apparatus 1 inputs the processed image 50 or the like by learning the image feature amount of the abnormality in the processed image 50 of the inspected object 4 as the abnormality detection model 152, and the abnormality in the shape of the inspected object 4 is displayed. And build (generate) a neural network that outputs information indicating the presence or absence of foreign matter. The neural network is MASK R-CNN (Mask Regions with Convolutional Neural Network), and has an input layer that accepts inputs such as a mask image 51 and / or a background image 52, an output layer that outputs an identification result of the presence or absence of an abnormality, and a mask. It has an intermediate layer for extracting an image feature amount such as an image 51 and / or a background image 52.

The input layer has a plurality of neurons that receive the input of the pixel value of each pixel included in the inspection image 54, and passes the input pixel to the intermediate layer. The intermediate layer has a plurality of neurons for extracting the image feature amount of the inspection image 54, and passes the extracted image feature amount to the output layer. In FIG. 6, the number of layers of the intermediate layer is set to 3, but the number of layers is not limited to this. For example, when the abnormality detection model 152 is CNN, the intermediate layer alternates between a convolutional layer that convolves the pixel values of each pixel input from the input layer and a pooling layer that maps the pixel values convoluted by the convolutional layer. It has a configuration connected to the image 54, and finally extracts the feature amount of the image while compressing the pixel information of the inspection image 54. In FIG. 6, the description of the convolution layer and the pooling layer is omitted. The output layer has three neurons that output the identification results that identify the abnormality of the object 4 to be inspected, and the presence or absence of the abnormality in the shape of the object 4 to be inspected and the presence or absence of foreign matter based on the image feature amount output from the intermediate layer. Identify the presence or absence. The first neuron outputs the normal probability value of the inspected object 4, the second neuron outputs the probability value of the abnormal shape of the inspected object 4, and the third neuron outputs the foreign substance in the inspected object 4. Output the probability value to have. The type of abnormality may be 3 or more.

In this embodiment, the anomaly detection model 152 is described as being a CNN, but the anomaly detection model 152 is not limited to the CNN, and is not limited to the CNN, but is a neural network other than the CNN, an SVM (Support Vector Machine), a Bayesian network, and a regression tree. Trained models built with other training algorithms are also applicable.

The information processing apparatus 1 performs learning by using the teacher data in which the plurality of mask images 51 and the information indicating the abnormality of the inspected object 4 in each mask image 51 are associated with each other. Here, the information indicating the above is the presence or absence of an abnormality determined on the processed image 50 of the inspected object 4, and if it is determined that there is an abnormality, the shape of the inspected object 4 is abnormal. And information on the presence or absence of foreign matter. For example, as shown in FIG. 5, the teacher data includes data in which the ID of the mask image 51, offset information, and blob label are labeled with respect to the mask image 51 of the object 4 to be inspected, and / or data such as a background image 52. Is.

The information processing apparatus 1 inputs the mask image 51 and / or the background image 52, which are teacher data, to the input layer, undergoes arithmetic processing in the intermediate layer, and identifies from the output layer whether or not the inspected object 4 is abnormal. Get the result. The identification result output from the output layer is an inspection image 54 including a heat map image 53 when the mask image 51 is input to the input layer, and a heat map image when the background image is input to the input layer. It is an inspection image 54 which does not include 53.

The information processing apparatus 1 compares the identification result output from the output layer with the information labeled with respect to the mask image 51 in the teacher data, that is, the correct answer value, so that the output value from the output layer approaches the correct answer value. , Optimize the parameters used for arithmetic processing in the intermediate layer. The parameters are, for example, a weight between neurons (coupling coefficient), a coefficient of an activation function used in each neuron, and the like. The method of optimizing the parameters is not particularly limited, but for example, the information processing apparatus 1 optimizes various parameters by using the error back propagation method.

The information processing apparatus 1 performs the above processing on each inspection image 54 included in the teacher data, and generates an abnormality detection model 152. When the information processing apparatus 1 acquires the processed image 50 of the inspected object 4, the information processing apparatus 1 detects the presence or absence of foreign matter in the inspected object 4 by using the abnormality detection model 152.

FIG. 6 is a flowchart showing an example of the processing procedure of the generation processing of the abnormality detection model 152 by the control unit 11.
The control unit 11 has data in which the plurality of mask images 51 of the inspected object 4 are associated with the information indicating the abnormality of the inspected object 4 in each processed image 50, and / or the abnormality of the inspected object 4 is found. The data that is the background image 52 that does not exist is acquired as the teacher data (S11).

The control unit 11 uses the teacher data to generate an abnormality detection model 152 (learned model) that outputs the abnormality presence / absence information of the inspected object 4 when the mask image 51 and / or the background image 52 of the inspected object 4 is input. Generate (S12).
Specifically, the control unit 11 inputs the mask image 51 and / or the background image 52, which are teacher data, to the input layer of the neural network, and determines whether or not there is an abnormality in the form of the object 4 to be inspected in the processed image 50. The identification result for identifying the presence or absence of foreign matter mixed in the inspection object 4 is acquired from the output layer. The control unit 11 compares the acquired identification result with the correct answer value of the teacher data (information labeled for the mask image 51 or no abnormality information for the background image 52), and the identification result output from the output layer is obtained. The parameters (weights, etc.) used for the arithmetic processing in the intermediate layer are optimized so as to approach the correct answer value. By increasing the types and numbers of the mask image 51 and the background image 52 and increasing the number of learnings, the abnormality detection model 152 is optimized and the accuracy is improved. The control unit 11 stores the generated abnormality detection model 152 in the auxiliary storage unit 15, and ends a series of processes.
Although it is possible to use only the mask image 51 for the teacher data in the learning model generation process in the present embodiment, it exists in the processed image 50 of the object 4 to be inspected, and although it is very similar to the foreign substance, it is a foreign substance. By training the abnormality detection model 152 as a background image for blobs and the like that do not correspond to the above, the learning effect of the abnormality detection model 152 is improved, and the accuracy of the abnormality detection process by the abnormality inspection system 10 is further improved.
[Abnormality detection processing]

FIG. 7 is a flowchart showing an example of a processing procedure for abnormality detection processing by the abnormality inspection system 10.
The irradiation unit 3 of the X-ray inspection device 2 irradiates the inspected object 4 with X-rays, and based on the X-rays detected by the detection unit 7, the image generation unit 111 of the control unit 11 in the information processing device 1 uses X-rays. A line irradiation image is generated (S21).
The image processing unit 112 of the control unit 11 performs image processing on the X-ray image with a smoothing filter, a feature extraction filter, or the like to generate a processed image 50 (S22).
The target specifying unit 113 of the control unit 11 performs blob analysis of the inspected object 4 based on the processed image 50, and identifies the blob (S23).
The acquisition unit 114 of the control unit 11 acquires the mask image 51, abnormality information, and the like based on the blobs specified in the processed images 50 and S23 generated in S22 (S24).
The abnormality information acquisition unit 115 of the control unit 11 acquires the heat map image 53 based on the mask image 51 or the like acquired in S24 (S25).
The output unit 116 of the control unit 11 synthesizes the heat map image 53 on the processed image 50, outputs the inspection image 54 including the heat map image 53 (S26), and ends the processing.

As described above, according to the present embodiment, in the processed image 50 of the object 4 to be inspected, even when the difference between the pixel values of the normal portion and the abnormal portion is small, the normal portion and the abnormal portion can be accurately discriminated. can.
Therefore, it is possible to accurately detect the presence / absence of abnormality information such as an abnormality in the shape of the object to be inspected 4 and the presence / absence of a foreign substance mixed in the object to be inspected 4.
Further, for a blob or the like that exists in the processed image 50 of the object 4 to be inspected and is very similar to a foreign substance but does not correspond to a foreign substance, the abnormality detection model 152 is trained as a background image to perform an abnormality inspection system. The accuracy of the abnormality detection process according to 10 is further improved.
Therefore, if the accuracy of the abnormality inspection system 10 is further improved, it becomes possible to reduce or eliminate the need for the user (operator) to visually detect the abnormality, and it is possible to improve the efficiency of the abnormality detection work.
[image]

Various images used in carrying out the abnormality detection process of the present invention will be described.
8 (a), (b), and (c) are schematic views showing a processed image 50 of the object 4 to be inspected. The processed image 50 is generated by performing image processing on the X-ray irradiation image.
Next, FIGS. 9A, 9B, and 9C are schematic views showing mask images of the inspected object obtained by performing masking processing on the blobs identified from the processed image 50. Is. In the masking process, the part corresponding to the blob is treated with white, and the area other than the blob is treated with black.
Next, FIG. 10 is a schematic view showing a processed image 50 of the object to be inspected, in which the blob is not identified by the abnormality detection process or the visual inspection of the user (inspector) and it is determined that no foreign matter is mixed. .. The image is subjected to predetermined image processing on the mask image, and is learned by the abnormality detection model 152 as the background image 52. Even if the blob is specified in the image, if the blob is not determined to be a foreign substance, it is processed and learned as the background image 52 in the same manner as described above.

Next, FIGS. 11A and 11B are schematic views showing marking images for performing background image processing. At the upper left of each of FIGS. 11A and 11B, a marking log for converting a normal processed image 50 of the object 4 to be inspected into a background image is displayed. The marking logo is created in a completely different shape from the various types of blobs envisioned so that it will not be judged as a blob by foreign matter detection processing or the user's visual inspection. In FIG. 11A, the marking logo is white, and the area other than the marking logo is black. Note that FIG. 11B is a negative / positive inverted image of FIG. 11A, and is appropriately used according to the two gradation level of the processed image 50. The abnormality detection model 152 learns all the images including the marking logo as background images. By learning the background image 52, even when a blob is specified on the processed image, it is possible to accurately determine whether or not the blob is a foreign substance.
Next, FIG. 12 is a schematic view showing a background image 52 obtained by synthesizing a normal image which is a processed image 50 in which foreign matter shown in FIG. 10 is not mixed and a marking image shown in FIG. ..

Next, FIG. 13 is a schematic view showing a processed image 50 of the inspected object 4 in which foreign matter is mixed. Foreign matter is shown in the circled and oval dotted lines.
Next, FIGS. 14A, 14B, and 14C are images that are previously performed when the foreign matter detection process is performed on the processed image 50 of the object 4 to be inspected in which a plurality of foreign substances are mixed. It is a schematic diagram which shows the process. The processed image 50 of the inspected object 4 in which a plurality of foreign substances are mixed duplicates the same number of images as the number of foreign substances, each of which shows a different foreign substance. FIG. 14A shows a processed image containing two foreign substances. In order to improve the accuracy of the foreign matter inspection process, the processed image 50 shown in FIG. 14 (a) is a processed image 50 containing one foreign substance, which is the process of FIGS. 14 (b) and 14 (c). It is duplicated in the image 50. After that, the mask image 51 is generated for the processed image 50 of FIG. 14 (b) and FIG. 14 (c).

Next, FIGS. 15A and 15B are schematic views showing inspection images of the object to be inspected after the foreign matter detection process is performed. FIG. 15A shows a processed image 50 before the foreign matter detection process, and FIG. 15B shows an inspection image 54 which is a processed image 50 before the foreign matter detection process. After performing the foreign matter detection process, FIG. 15B displays a heat map on the foreign matter portion of the processed image 50, and clearly shows the presence, approximate shape and position of the foreign matter. Specifically, the inspection image 54 of FIG. 15 (b) synthesizes a heat map image generated based on the abnormality information or the like obtained by the foreign matter detection process with the processed image 50 of FIG. 15 (a). Is obtained by.
Next, FIGS. 16A and 16B are schematic views showing an example of an inspected object to be inspected. As can be seen from FIG. 16A, if the foreign matter has an iron ball shape, various sizes such as 0.8 mm, 1.0 mm, 1.2 mm, 1.5 mm, 2.0 mm, and 2.5 mm can be used. It is detectable. Further, FIG. 16B is an example of a foreign substance other than the iron ball shape. As can be seen from FIG. 16B, foreign matter detection targets include gusset needles, staplers, broken needles, hair ties, and lighters. If it is a foreign substance that can be photographed, it can be detected by registering it as a foreign substance in the abnormality detection model 152 and learning it.
(Embodiment 2)
[Re-learning]

FIG. 17 is a block diagram showing the configuration of the abnormality inspection system 20 according to the second embodiment.
In the abnormality inspection system 20 according to the second embodiment, the control unit 11 serves as a functional unit and the re-learning unit 119.
It has the same configuration as the abnormality inspection system 10 according to the first embodiment, and performs the same processing except that the abnormality inspection system 10 is provided. The same parts as those in FIG. 2 are designated by the same reference numerals, and detailed description thereof will be omitted.

In the foreign matter detection process of the present invention, the first display unit 41 displays the processed image 50, and the user (operator) visually confirms the presence or absence of foreign matter in the processed image 50. Further, the second display unit 42 displays the inspection image 54 after the foreign matter detection process by the abnormality inspection system 20, and the user visually confirms the presence or absence of the heat map on the inspection image 54.

At this time, there is a problem that the visual determination of foreign matter by the user and the determination of the abnormality inspection system 20 by the foreign matter detection process are different. Here, as a premise, the user's visual judgment is the correct answer. FIG. 18 is a schematic diagram showing cases where the judgment of the user and the detection result of the abnormality inspection system 20 are different. As can be seen from FIG. 18, the problem is that when the user visually determines that there is a foreign substance, the abnormality inspection system 20 detects that there is no foreign substance, and when the user visually determines that there is no foreign substance, the abnormality inspection is performed. This is the case when the system 20 detects that there is a foreign substance.
That is, the cases where the processed image 50 is relearned are the following (Pattern 1) and (Pattern 2).
(Pattern 1) A case in which the abnormality inspection system 20 according to the present invention does not display a heat map on the inspection image 54 of the object 4 to be inspected, but the user visually determines that foreign matter is mixed.
(Pattern 2) A case where the abnormality inspection system 20 according to the present invention displays a heat map on the inspection image 54 of the inspected object 4, but the user visually determines that no foreign matter is mixed.

When a discrepancy occurs between the output result of the learning model according to the present invention and the visual determination result of the user, the re-learning unit 119 confirms the processed image 50 again by the user's visual inspection and confirms the processed image 50 of the inspected object 4. Perform normal / abnormal status judgment.
The re-learning unit 119 relearns the abnormality detection model 152 by using the teacher data in which the inspection image 54 and the judgment of the quality of the abnormality detection model 152 input by the user are associated with each other.
Here, the second display unit 42 includes an OK button 70 (not shown) which is a first operation button and an NG button 71 (not shown) which is a second operation button.
When the states of (Pattern 1) and (Pattern 2) occur in the images displayed on the first display unit 41 and the second display unit 42, respectively, the control unit 11 has the OK button 70 and NG by the user. The state of pressing the button 71 is detected, the inspection image 54 is associated with the presence or absence of an abnormal state, for example, the presence or absence of foreign matter contamination, and the data is specified as data to be relearned.
The second display unit 42 further includes a selection button 72 that accepts an input of the type of abnormality of the object 4 to be inspected.
In the case of the above (pattern 1), the user presses the selection button in addition to pressing the NG button 71 to select the type of abnormality included in the image displayed on the second display unit 42, for example, the type of foreign matter. You can enter it.
The type of anomaly selected and entered is further associated with the inspection image 54 in addition to the presence or absence of anomalous conditions.
In the above (Pattern 1) and (Pattern 2), accurate information is added to the image that has been erroneously determined, and after re-learning by the abnormality detection model 152, a second display is made for the occurrence of the same abnormality. The unit displays a heat map image based on the type of abnormality on the abnormality image.
That is, the re-learning unit 119 relearns the abnormality detection model 152 for the re-discriminated inspection image 54 by the discrimination unit 118. The re-learning of the abnormality detection model 152 by the re-learning unit 119 is performed, for example, by automatic learning at night. The correct determination result of the corresponding processed image 50 is stored and stored in the inspection image DB 151, and is utilized in the inspection of a new inspected object 4.
If the judgment result of foreign matter contamination by the user and the judgment result of foreign matter contamination by the abnormality inspection system are different, an abnormality is made to the target processed image so that the abnormality inspection system does not make the same misidentification, false detection, or false judgment in the future. It is necessary to retrain the detection model 152.

FIG. 19 is a flowchart showing an example of the processing procedure of the relearning process by the control unit 11 of the information processing apparatus 1.

The control unit 11 determines whether or not the abnormality inspection system 20 determines whether or not the foreign matter detection is good or bad based on the user pressing the OK button 70 or the NG button 71 provided on the second display unit 42 with respect to the selected inspection image 54. Accept (S31).
When the result of foreign matter detection by the abnormality inspection system 20 is different from the visual judgment of the user, the user presses the NG button 71. In the case of the above pattern 1, by pressing the button, a mask image or the like is acquired from the inspection image 54 (S31), these are trained by the abnormality detection model 152 as teacher data, and a new abnormality detection model 152 is generated (S32). , Saved in the inspection image DB 151.
Further, in the case of the above pattern 2, by pressing the NG button 71, the inspection image 54 is combined with the masking image, a background image 52 is acquired (S31), and these are trained by the abnormality detection model 152 as teacher data. A new abnormality detection model 152 is generated (S32) and stored in the inspection image DB 151.
When the foreign matter determination result by the user's visual inspection and the foreign matter detection processing result of the abnormality inspection system 20 are the same, the user presses the OK button 70 to end the foreign matter detection processing.
By these re-learning processes, the anomaly detection model 152 is relearned as the anomaly detection process is continued, and more accurate and accurate anomaly detection can be performed.

The embodiments disclosed this time should be considered to be exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.
For example, the inspection product is not limited to clothing products and food products, but may be packages, industrial products, and the like.
The electromagnetic wave irradiating the object 4 to be inspected is not limited to X-rays, and may be terahertz waves, infrared rays, visible light, or the like.
Further, in the process or operation described above, the process or operation is performed as long as there is no contradiction in the process or operation such as using data that should not be available in that step in a certain step. You can change it freely. Further, each of the examples described above is an example for explaining the present invention, and the present invention is not limited to these examples. The present invention can be carried out in various forms as long as it does not deviate from the gist thereof.

1 Information processing device 2 X-ray inspection device 3 Irradiation unit 4 Inspected object 6 Conveyor unit (conveyor)
7 Detection unit 10, 20 Abnormality inspection system 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Auxiliary storage unit 21 Small upper housing 21 Upper housing 22 Display unit 23 Lower housing 24 Light emitting unit 25 Light receiving unit 27 Controller 31 Irradiating body 32 Irradiation control unit 41 First display unit 42 Second display unit 50 Processing image 51 Mask image 52 Background image 53 Heat map image 54 Inspection image 60 Conveyance belt 61 Conveyance belt drive unit 62 Conveyance belt control unit 70 OK Button (first operation button)
71 NG button (second operation button)
72 Select button 111 Image generation unit 112 Image processing unit 113 Target identification unit 114 Acquisition unit 115 Abnormal information acquisition unit 116 Output unit 117 Control display unit 118 Discrimination unit 119 Learning unit 124 Specific image acquisition unit 134 Specific information acquisition unit 152 Abnormality detection model 153 Program 220 Data file generation unit 221 Data reading unit DB151 Inspection image N network

Claims (12)

The step of irradiating the object to be inspected with electromagnetic waves,
The step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and
A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the processed image that has been image-processed with respect to the image.
A mask image masked to a position and shape corresponding to the specified blob, a background image which is the processed image not including the blob, offset information of the mask portion in the processed image, and an object corresponding to the blob. The step of acquiring anomalous information including information on the anomaly of the inspected object having a blob label of the type of
A step of acquiring teacher data including the mask image, the background image, and the abnormality information, and
When the object to be inspected is irradiated with an electromagnetic wave and an image acquired in response to the electromagnetic wave transmitted through the object to be inspected is input, the position and shape of the blob are displayed on the processed image based on the teacher data. Steps to get the heatmap image displayed as a heatmap,
A step of generating a learning model that outputs an inspection image including the heat map image by synthesizing the heat map image with the processed image is included.
The background image is a method of generating a learning model, characterized in that an image different from the shape of the blob is synthesized at an arbitrary position of the processed image. The electromagnetic wave is an X-ray,
A step of identifying the blob from the images acquired in response to the X-ray and generating a masked image masked to a position and shape corresponding to the blob.
The step of displaying the generated mask image and
The method for generating a learning model according to claim 1, further comprising a step of identifying the abnormality information of the object to be inspected included in the mask image. A step of acquiring the mask image of the inspected object and the abnormality information of the inspected object, and
The method for generating a learning model according to claim 1 or 2, further comprising a step of re-learning the learning model based on the mask image and the abnormality information. The object to be inspected is irradiated with an electromagnetic wave, an image is acquired according to the electromagnetic wave transmitted through the object to be inspected, and the object to be inspected is based on the pixel value of the pixel of the processed image that has been image-processed for the image. A mask image masked to a position and shape corresponding to the identified blob, a background image which is the processed image not including the blob, and offset information of the mask portion in the processed image. And an input layer into which anomalous information including information about anomalies of the inspected object having a blob label, which is the type of object corresponding to the blob, is input.
An inspection image including the heat map image is output by acquiring a heat map image in which the position and shape of the blob are displayed as a heat map on the processed image and synthesizing the heat map image with the processed image. Output layer and
The mask image and the intermediate layer whose parameters are learned based on the abnormality information are provided.
When the mask image and the abnormality information are input to the input layer, the computer is made to function so as to output the inspection image including the heat map image from the output layer through the calculation by the intermediate layer.
A learning model in which, when the background image is input to the input layer, the computer functions to output the inspection image, which does not include the heat map image, from the output layer through calculations by the intermediate layer. The irradiation part that irradiates the object to be inspected with electromagnetic waves,
An image generator that generates an image according to the electromagnetic waves transmitted through the object to be inspected,
An image processing unit that performs image processing on the image and acquires the processed image,
A target identification unit that identifies a blob contained in the inspected object based on the pixel value of the pixel of the processed image.
A mask image masked to a position and shape corresponding to the specified blob, a background image which is the processed image not including the blob, offset information of the mask portion in the processed image, and an object corresponding to the blob. An acquisition unit that acquires anomaly information including information on an abnormality of the inspected object having a blob label of the type as teacher data, and an acquisition unit.
When the image to be inspected is irradiated with electromagnetic waves and an image acquired in response to the electromagnetic waves transmitted through the inspected object is input, the acquisition unit is used in a learning model to output information regarding an abnormality in the inspected object. An abnormality information acquisition unit that inputs the acquired mask image and abnormality information and acquires a heat map image in which the position and shape of the blob are displayed as a heat map on the processed image.
It is provided with an output unit that outputs an inspection image including the heat map image by synthesizing the heat map image with the processed image.
The background image is an inspection device including an image different from the shape of the blob is synthesized at an arbitrary position of the processed image. The electromagnetic wave is an X-ray,
The inspection device according to claim 5, wherein the image generation unit generates an image based on the X-ray. Equipped with a display
The display unit includes a first display unit that outputs an image based on the image acquired after the X-ray irradiation, and a normal image or an image showing a normal state of the object to be inspected, which does not include the heat map image. A second display unit that outputs an abnormal image, which is an image indicating a state in which the object to be inspected is abnormal, including the heat map image, is displayed side by side on the display unit .
The inspection device according to claim 6, wherein the display unit has a first operation button and a second operation button that receive input of normal information or abnormality information to the object to be inspected by the user. A re-learning unit that relearns the learning model based on the image displayed on the second display unit and the normal information or the abnormal information received by the first operation button and the second operation button. The inspection apparatus according to claim 7, wherein the inspection apparatus is provided. The second display unit includes a selection button that accepts an input of the type of abnormality of the object to be inspected.
The seventh or eighth aspect of the present invention, wherein when the selection button accepts the input, the second display unit displays the heat map image based on the type of the abnormality on the abnormality image. Inspection device. The blob is identified by irradiating the object to be inspected with an electromagnetic wave, acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and applying image processing to the image to perform image processing. A second acquisition unit is provided to acquire information regarding an abnormality of the object to be inspected based on a mask image masked to a position and shape corresponding to the identified blob.
The inspection device according to any one of claims 5 to 9, wherein the abnormality information acquisition unit inputs the heat map image according to the information acquired by the second acquisition unit. The step of irradiating the object to be inspected with electromagnetic waves,
The step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and
A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the processed image that has been image-processed with respect to the image.
A mask image masked to a position and shape corresponding to the specified blob, a background image which is the processed image not including the blob, offset information of the mask portion in the processed image, and an object corresponding to the blob. The step of acquiring anomalous information including information on the anomaly of the inspected object having a blob label of the type of
The step of acquiring the teacher data including the mask image, the background image, and the abnormality information, and
When the object to be inspected is irradiated with an electromagnetic wave and an image acquired in response to the electromagnetic wave transmitted through the object to be inspected is input, the position and shape of the blob are displayed on the processed image based on the teacher data. Steps to get the heatmap image displayed as a heatmap,
By synthesizing the heat map image with the processed image, a computer is made to execute a process including a step of generating a learning model that outputs an inspection image including the heat map image.
The background image is an inspection method characterized in that an image different from the shape of the blob is synthesized at an arbitrary position of the processed image. The step of irradiating the object to be inspected with electromagnetic waves,
The step of acquiring an image according to the electromagnetic wave transmitted through the object to be inspected, and
A step of identifying a blob contained in the inspected object based on the pixel value of the pixel of the processed image that has been image-processed with respect to the image.
A mask image masked to a position and shape corresponding to the specified blob, a background image which is the processed image not including the blob, offset information of the mask portion in the processed image, and an object corresponding to the blob. The step of acquiring anomalous information including information on the anomaly of the inspected object having a blob label of the type of
A step of acquiring teacher data including the mask image, the background image, and the abnormality information, and
When the object to be inspected is irradiated with an electromagnetic wave and an image acquired in response to the electromagnetic wave transmitted through the object to be inspected is input, the position and shape of the blob are displayed on the processed image based on the teacher data. Steps to get the heatmap image displayed as a heatmap,
By synthesizing the heat map image with the processed image, a computer is made to execute a process including a step of generating a learning model that outputs an inspection image including the heat map image.
The background image is a computer program characterized in that an image different from the shape of the blob is synthesized at an arbitrary position of the processed image.

Images