JP2021157791A - Method and device for evaluation - Google Patents

Abstract

To provide a device and a method for evaluation that can rapidly evaluate the presence or absence of a specific part for a large number of objects, while maintaining or improving the accuracy of detecting the specific part.SOLUTION: The method includes: acquiring a first image in a first range of an object; extracting a plurality of second ranges as a part of the first range from image data of a first format (for example, gray-scale data) as image data of the first image, which is expressed by one index per one pixel, by using a first method; and evaluating the presence or absence of a specific part in the second ranges from image data of a second format (for example, color data) as image data of a second image in the second ranges, which is expressed by a plurality of indexes per one pixel, by using a second method acquired by a machine learning.SELECTED DRAWING: Figure 4A

Description

The present invention relates to an evaluation method and an evaluation device.

Defect inspection system, abnormality inspection system that evaluates the presence or absence of abnormal conditions that may occur in the object or parts that have different characteristics from others, based on the images obtained by imaging the object. Or, a singular part detection system is known. In recent years, a method has been proposed for evaluating the presence or absence of a site (hereinafter referred to as “singular part”) having characteristics different from most other regions, including defects, using machine learning.

In Patent Document 1, a singular part image including a singular part having an arbitrary feature is extracted from an image of an object captured by the imaging unit, and the type of the singular part is determined by machine learning using the singular part image as an input. A singular part detection system for identification is described.

International Publication No. 2019/003813

However, there is room for improvement in the detection accuracy of the presence or absence of the singular portion. In addition, it may be required to quickly evaluate the presence or absence of singular parts in a large number of objects.

Therefore, an object of the present invention is to provide an evaluation method and an evaluation device capable of quickly evaluating the presence or absence of a singular portion in a large number of objects while maintaining or improving the detection accuracy of the singular portion. ..

The evaluation method according to the present disclosure is to acquire a first image corresponding to the first range of the object, and to use the first method to represent the image data of the first image by one index per pixel. The second range corresponding to the second range is extracted from the image data of the first format to be obtained by extracting the second range which is a part of the first range and using the second method acquired by using machine learning. This includes evaluating the presence or absence of a singular portion within the second range from the image data of the second format, which is the image data of the image and is represented by a plurality of indexes per pixel.

The "first format image data represented by one index per pixel" is image data in which the difference in color of each pixel is represented by one index. Such image data may be image data in which one pixel is represented by a single density value. Examples of such image data include grayscale data. Further, as such image data, since each primary color of RGB has the same density value (brightness) in each pixel, image data capable of expressing each pixel constituting the image data with one density value can be obtained. Can be mentioned. Since each pixel has a density value in which two of the primary colors of RGB have fixed density values, image data capable of expressing each pixel constituting the image data with one density value of the remaining one primary color can be obtained. Can be mentioned. In addition, each pixel may be represented by another primary color such as CMY. When the color of each pixel is expressed by three parameters of hue, saturation, and lightness, two of these three parameters are fixed for each pixel, and the difference in color of each pixel remains. Image data represented by one of the parameters can be mentioned.

As the above-mentioned index, density (luminance), hue, saturation, lightness and the like can be used.

The "second format image data having a plurality of indexes per pixel" includes, for example, color data such as RGB data in which each primary color of RGB can be expressed by an independent density value. It also includes color data represented by different color models such as CMY data. It also includes image data in which the difference between each pixel is expressed by a plurality of parameters among hue, saturation, and lightness.

According to such a method, by using the image data of the first format, which has a smaller amount of information than the image data of the second format, the amount of calculation required for the image processing of the extraction of the second range can be suppressed, and as a result. , It becomes possible to shorten the calculation time. Further, using the second method acquired by using machine learning, the image data expressed in the second format having a large amount of information, such as the color data of the second image corresponding to the second range, is within the second range. Evaluate the presence or absence of singularities. That is, the image processing of the first method is applied to the first image, which is an image in a larger region than the second image of the second format and has a small amount of information in each pixel, thereby. After narrowing down the search range by setting the partial region where the abnormality exists in the first image as the second image, the second image having a relatively large amount of information in each pixel is compared to the first image. Therefore, the image processing of the second method is applied to evaluate whether or not a singular part is present in the second image. According to this, it becomes possible to maintain or improve the detection accuracy by machine learning. As a result, it becomes possible to quickly evaluate the presence or absence of the singular part in a large number of objects while maintaining or improving the detection accuracy of the singular part. For example, the second method may be executed for one object, and at the same time, the first method may be executed for one or more objects different from this object.

The "evaluation" in the present disclosure includes determining the presence or absence of a singular portion, determining the presence or absence of a singular portion together with probability information, and outputting probability information regarding the presence or absence of a singular portion. Further, the "evaluation" in the present disclosure includes outputting information regarding the presence or absence of the singular portion and / or information regarding the type of the singular portion when there are a plurality of types of the singular portion.

Further, the first method is, for example, a method of creating binary data from grayscale data and extracting a second range based on the number of pixels having the predetermined binary data in a region of a predetermined size. May be good. For example, when black-and-white binary data is created based on a threshold value determined from all or part of grayscale data, and the number of black pixels contained in an area of a predetermined size is a predetermined number or more. The method may be a method of extracting the region as a second range. Further, the outline may be emphasized by the binarization filter to emphasize the boundary with the background. In addition, as the first method, a method suitable for extracting a second range that may include the singular part from the grayscale data can be selected based on the characteristics of the singular part to be evaluated. For example, the second range may be extracted by pattern recognition based on the model shape of the foreign matter.

Further, image data in a second format such as color data may be generated based on image data in the first format such as grayscale data. For example, grayscale data may be converted to generate color data. As an example of the method of converting the image data of the first format into the image data of the second format, by adopting the brightness as an index included in the grayscale data of the first format for the brightness of each primary color such as RGB, A method of generating a color image can be mentioned.

With such a configuration, it becomes possible to evaluate the presence or absence of a singular part from the color data by using the second method acquired by using machine learning, and the color data is acquired for the first range. No need.

Further, in machine learning, a plurality of fourth images that are smaller than the third image and have different positions of the one singular portion are generated from the third image including one singular portion, and a plurality of fourth images. It may include optimizing the internal parameters by using the image data represented by the image data of the second format such as the color data of the above as the teacher data.

The third image may be an image having a size larger than the second range. For example, the third image may be an image having the same size as the second range (256 pixels × 256 pixels as an example). The third image may be prepared in advance of a series of arithmetic processes for obtaining a trained model to be used in the second method by machine learning. The fourth image is a part of the third image, smaller than the second range (for example, 224 pixels × 224 pixels), and includes a singular part in a series of arithmetic processes for obtaining the trained model. It may be generated by extracting a plurality of images. As a result, a plurality of fourth images in which the positions of the singular portions are different from each other can be obtained. The plurality of fourth images may include a plurality of images containing the same singular portion and having different positions of the singular portions. These fourth images are used as teacher data. Further, the third image may be an image larger than the second range, and the fourth image may be an image having the same size as the second range.

With such a configuration, it is possible to generate a plurality of patterns of teacher data in which a singular portion exists at a biased position, so that it is possible to maintain or improve the detection accuracy by machine learning. From the third image including one singular part, a plurality of images including one singular part and having different colors may be generated and used as teacher data.

As described above, by increasing the degree of dispersion of the teacher data by changing the mode of the teacher data (position, color, brightness, etc. of the singular part), there is an effect of improving the detection accuracy of the feature of the singular part to be detected. ..

Further, the evaluation device according to the present disclosure uses the means for acquiring the first image corresponding to the first range of the object and the first method, and is the image data of the first image, which is one index per pixel. Corresponds to the second range by using the means for extracting the second range, which is a part of the first range, and the second method obtained by using machine learning from the image data in the first format represented by. The image data of the second image includes means for evaluating the presence or absence of a singular portion within the second range from the image data of the second format represented by a plurality of indexes per pixel.

According to such a device, by using the image data of the first format, which has a smaller amount of information than the image data of the second format, the amount of calculation required for the image processing of the extraction of the second range can be suppressed, and as a result. , It becomes possible to shorten the calculation time. In addition, using the second method acquired by using machine learning, the image data expressed in the second format having a large amount of information, such as the color data of the second image corresponding to the second range, is within the second range. Since the means for evaluating the presence or absence of the singular portion of the above is provided, it is possible to maintain or improve the detection accuracy by machine learning. Therefore, it is possible to quickly evaluate the presence or absence of the singular part in a large number of objects while maintaining or improving the detection accuracy of the singular part.

Further, the computer program according to the present disclosure uses the computer to acquire the first image corresponding to the first range of the object and the first method to obtain the image data of the first image per pixel. The second range is extracted from the image data of the first format represented by one index, and the second range obtained by using machine learning is used. From the image data of the second image corresponding to the image data of the second format represented by a plurality of indexes per pixel, the command to evaluate the presence or absence of the singular part in the second range is executed. include.

According to such a computer program, by using the image data of the first format, which has a smaller amount of information than the image data of the second format, the amount of calculation required for the image processing of the extraction of the second range can be suppressed. As a result, the calculation time can be shortened. In addition, using the second method acquired by using machine learning on a computer, the image data expressed in the second format having a large amount of information, such as the color data of the second image corresponding to the second range, can be obtained from the image data. Since the presence or absence of the singular part within the two ranges is evaluated, it is possible to maintain or improve the detection accuracy by machine learning. Therefore, it is possible to quickly evaluate the presence or absence of the singular part in a large number of objects while maintaining or improving the detection accuracy of the singular part.

Further, the evaluation system according to the present disclosure includes an evaluation device and a server. The server stores the first image data group including different types of singular parts and the second image data group not including the singular parts as teacher data for machine learning. The same label may be given to the first image data. A label different from the label given to the first image data is given to the second image data group. Further, in the evaluation system according to the present disclosure, the execution instructions included in the computer program according to the present disclosure described above include a processor and a memory, and a plurality of arithmetic units connected to each other so as to be able to communicate with each other wirelessly or by wire are linked. Includes aspects to be performed.

According to such an evaluation system, it is possible to exert the same effect as the above-mentioned evaluation device.

Further, in the present disclosure, using the step of acquiring the first image corresponding to the first range of the object and the first image processing method, the image data of the first image is one density value per pixel. Using the first step of performing image processing for extracting the second range, which is a part of the first range, from the image data of the first format having the above, and the second image processing method acquired by using machine learning. , Image processing for evaluating the presence or absence of a singular portion in the second range from the image data of the second image corresponding to the second range and having a plurality of density values per pixel in the second format. Includes 2 steps.

According to such an image processing method, it is possible to reduce the amount of calculation in the first step and shorten the calculation time. Therefore, it is possible to shorten the calculation time for an object that is unlikely to have a singular portion. Further, since the second step uses the image data of the second format having a large amount of information, it is possible to maintain or improve the evaluation accuracy using the second image processing method acquired by using machine learning.

Further, in order to cause the computer to execute such an image processing method, the first software module that implements the first image processing method requests the first image acquired corresponding to the first range of the object. The first step of performing image processing for extracting the second range, which is a part of the first range, from the image data of the first image and the image data of the first format having one density value per pixel. Contains instructions to get the computer to execute. Further, the second software module that implements the second image processing method acquired by using machine learning is the image data of the second image corresponding to the second range and has a plurality of density values per pixel. It includes an instruction for causing a computer to perform a second step of performing image processing for evaluating the presence or absence of a singular portion in the second range from the image data in the second format.

Functional block diagram of evaluation system 10 Block diagram showing the hardware configuration of the evaluation device of the evaluation system 10. Flowchart of evaluation method shown in this embodiment Schematic diagram of the first range AR1 and the second range AR2 to AR4 Second range AR2 and singular part candidate C2 Second range AR3 and singular part candidate C3 Second range AR4 and singular part candidate C4 Graph showing the evaluation result by the evaluation method according to this embodiment Schematic diagram showing how to generate teacher data Schematic diagram showing how to generate teacher data

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments are examples for explaining the present invention, and the present invention is not intended to be limited only to the embodiments.

FIG. 1 is a functional block diagram of the singular part evaluation system 10 according to the present embodiment. The evaluation system 10 is an evaluation system for determining whether or not a foreign substance is mixed in a PET bottle (an example of an "object") containing a liquid. However, as will be described later, the present invention can be applied to various objects and various singular parts.

The evaluation system 10 includes an evaluation device including an imaging unit 12, a singular part detection unit 14, a singular part image extraction unit 16, an image data conversion unit 18, and a singular part presence / absence evaluation unit 20. Further, the evaluation system 10 further includes a server device connected to the singularity presence / absence evaluation unit 20 via the network N. The server device includes a database DB1 that stores images of normal samples and a database DB2 that stores images of abnormal samples. In the present embodiment, the normal sample is a sample that does not contain a singular part, in other words, the object does not have an abnormality, and is a sample corresponding to a non-defective product. Further, the abnormal sample is a sample containing a peculiar portion, in other words, a sample in which the object has an abnormality, and is a sample corresponding to a defective product.

The imaging unit 12 captures a part or all of the object and acquires an image. The imaging unit 12 can be composed of, for example, an image sensor such as a line sensor or an area sensor. For example, in the case of an object having a shape like a PET bottle in which the object is formed rotationally symmetric with respect to the central axis, the object is imaged by a line sensor while rotating the PET bottle around the central axis. An image corresponding to the entire side surface of the object (an example of the "first range") can be acquired. In the case of the present embodiment, the imaging unit 12 is composed of a line sensor, and has an 8-bit (256 gradations) brightness value (an example of "density value") for each pixel of 5000 pixels x 5000 pixels as an image of an object. Grayscale data (an example of "image data of the first format") having the above is acquired. Instead of capturing the image of the object by the imaging unit 12 and acquiring the image, the image of the object may be acquired from a database or the like that stores the image of the object. Further, even if the color data is imaged by the imaging unit 12 and converted into grayscale data, the image data of the first format represented by one index such as one density value per pixel can be acquired. good. For example, the formula (weighting of RGB brightness value) specified in the BT.601 ("Studio encoding parameters of digital television for standard 4: 3 and wide screen 16: 9 aspect ratio") standard established by the ITU (International Telecommunication Union). The image data in the first format may be acquired by converting the color data into grayscale data according to (average). Here, the image data of the first format is image data in which the difference between each pixel is expressed by one index. Therefore, in addition to the case where one pixel is represented by a single index called a density value, image data represented by brightness, hue, saturation or lightness may be used as the image data of the first format. Further, even in the case of RGB data, by having two primary colors of each of the RGB primary colors have fixed density values, each pixel constituting the image data is set to one density value of the remaining one primary color. The image data that can be expressed may be used as the image data of the first format. Further, even when expressed by three parameters of hue, saturation, and lightness, two of these three parameters are fixed, and the image data that can be expressed by the remaining one parameter is the first. It may be used as image data in a format.

The singular part detection unit 14 detects a singular part having an arbitrary feature from the image of the object acquired by the imaging unit 12. Here, the singular part refers to a part having characteristics different from those of the surrounding area from the viewpoint of at least one of shape, brightness and color. The singular part may be a defect, or may be an artificially set mark, pattern, or the like. The singular part related to the shape is a part having characteristics different from the surrounding region in the characteristics (for example, volume, area, height, particle size, aspect ratio, roundness, contour shape, etc.) of the shape. .. Similarly, the luminance refers to a portion having characteristics related to luminance such as brightness and histogram, and the color refers to a portion having characteristics related to color such as spectrum, center wavelength and maximum wavelength which are different from those in the surrounding region. .. The peculiar part in the present embodiment is dust, human hair, dirt on the container, etc. that can be contained in the liquid inside the PET bottle.

Various methods can be adopted as the method for detecting the singular part. The singular part detection unit 14 in the present embodiment uses, for example, grayscale data acquired corresponding to each pixel as an image of the entire side surface of the PET bottle, which is the first range, using an adaptively set threshold value. And binarize, and detect the singular part according to the number of pixels of the binarized data included in the predetermined area (an example of the "first method"). For example, the singularity detection unit 14 acquires the average value of the brightness of a plurality of pixels included in a predetermined peripheral region centered on a predetermined target pixel. Next, the difference between the brightness of the target pixel as the center and the calculated average value of the brightness is acquired. Based on whether or not the value of this difference exceeds a predetermined threshold value, a process of binarizing the target pixel is executed, and this is repeated for each pixel. By setting the threshold value based on the surrounding pixels in this way, it is possible to suppress the influence of the background portion such as the color of the liquid contained in the PET bottle and detect the singular portion more accurately. Then, the singular part detection unit 14 determines that the singular part may exist when the number of black pixels (or white pixels) included in the predetermined area is a predetermined number or more. The threshold value setting method and the singularity detection method can be appropriately selected according to the evaluation purpose. For example, when it is desired to suppress detection omission as much as possible, the threshold value can be set small so that the singular portion can be easily detected.

Further, as a method for detecting the singular part, various methods can be adopted depending on the characteristics of the singular part. For example, when it is known in advance that the singular portion has a characteristic geometric shape, the singular portion detecting unit 14 may be configured to detect the singular portion by pattern matching of the shape.

The singular part image extraction unit 16 extracts a region (an example of the “second range”) including all or a part of the singular part detected by the singular part detection unit 14 from the first range which is the entire side surface of the PET bottle. do. In the present embodiment, a region of 256 pixels × 256 pixels including the singular part detected by the singular part detecting unit 14 is extracted as a singular part image. The number of the extracted region and the corresponding singular part image may be 0 or may be plural.

The image data conversion unit 18 expresses the image data of the first format in which the difference in color of each pixel included in the image is expressed by one index per pixel, and the difference in color of each pixel is expressed by a plurality of indexes per pixel. It is converted into the image data of the second format.

In the present embodiment, the image data conversion unit 18 refers to image data of the first format having one density value per pixel, such as grayscale data, as 8-bit RGB color data (hereinafter, “RGB data”). ) Etc., it is converted into image data of a second format having a plurality of density values per pixel. For example, the predetermined grayscale data expressing dark gray and the image data having a luminance value of 169 has three luminance values of, for example, 169, 169, and 169 in the RGB data and the CMY data. Can be converted. In this way, when the gray scale having one index is converted into the color data having a plurality of indexes, for example, the brightness as one index of the gray scale data is changed to the brightness of the three primary colors expressing the color data. Color data can be obtained by using as. Similarly, the singular part image extraction unit 16 uses a predetermined conversion table, transformation matrix, conversion formula, conversion map, or the like for each pixel included in the second range to obtain the same or the original grayscale data. Convert to approximate RGB data. As described above, the image data conversion unit 18 generates image data of the second format having a plurality of density values per pixel for one or a plurality of second ranges. When each pixel of the image data of the first format is represented by, for example, one parameter of hue, saturation, and lightness corresponding to HSV, a known conversion between the HSV representation and the RGB representation. By using the formula, the image data of the first format may be converted into the image data of the second format.

As the evaluation method in the present embodiment, the second method is applied, and the amount of image data of the second format corresponding to the second range is the first method to which the first method is applied and corresponds to the first range. It is configured to be smaller than the image data in the format. More specifically, in the evaluation method in the present embodiment, the amount of data possessed by each pixel of the image data of the second format corresponding to the second range is each of the image data of the first format corresponding to the first range. The amount of data is larger than the amount of data possessed by the pixels, and the size (number of pixels) of the image data of the second format corresponding to the second range is smaller than that of the image data of the first format. Has been done. At this time, the reduction in the amount of data by making the size of the second range smaller than that in the first range exceeds the increase in the amount of data possessed by each pixel of the image data of the second format. By such arithmetic processing, the image data of the second format corresponding to the second range and the image data of the second format having a smaller amount of data than the image data of the first format corresponding to the first range can be obtained.

The singular part presence / absence evaluation unit 20 evaluates whether or not the singular part is included in the second range based on the model generated by using machine learning. In the present embodiment, the singular part presence / absence evaluation unit 20 includes a convolutional neural network. The convolutional neural network includes an input layer for inputting image data of the second format for the second range, an output layer for outputting the presence / absence of a singular portion, and an intermediate layer connecting both layers. The intermediate layer is a convolution layer that generates a map with features by performing a convolution operation that calculates the sum of products of overlapping parts as a feature by sliding the filter, and the maximum value of the two-dimensional array data output from the convolution layer. It is provided with a plurality of combinations with a pooling layer for extracting the data, and is provided with a coupling layer that combines internal parameters as weighting coefficients and provides an evaluation result to the output layer. In addition to the convolutional neural network, the neural network includes RNN (Recurrent Neural Network), Elman network, Jordan network, ESN (Echo State Network), LSTM (Long Short Term Memory network), and BRNN (Bi-). It can be composed of directional RNN) and so on. Further, the output layer is not limited to the case of outputting two types of results, that is, the presence or absence of a singular portion. For example, it may be configured to output three or more types of results depending on the type of the singular part, and numerical information regarding whether a specific pixel or a collection of specific pixels corresponds to the singular part is output. It may be configured as.

The server device includes a database DB1 that stores a sample image of a normal object and a database DB2 that stores a sample image of an abnormal object. The database DB1 stores image data in a second format that does not include a singular part. In the present embodiment, the database DB1 stores RGB data that does not include a singular portion of 224 pixels × 224 pixels, which is acquired by imaging the side surface of the PET bottle that is the object. For example, if the liquid contained in the PET bottle container contains carbonic acid, the bubbles are not singular parts. Therefore, the database DB1 stores RGB data including bubbles as image data that does not include a singular portion.

The database DB2 stores the image data of the second format including the singular part. In the present embodiment, the database DB2 is RGB data including all or a part of a singular portion of 224 pixels × 224 pixels acquired by imaging the side surface of the PET bottle which is an object. The peculiar part is, for example, human hair, dust, and dirt on the container. Therefore, the database DB2 stores a plurality of RGB data including human hair, a plurality of RGB data including dust, and a plurality of RGB data including dirt on the container. The method of generating the image data stored in the database DB1 and the database DB2 will be described later.

The singular part presence / absence evaluation unit 20 acquires a trained model including internal parameters for evaluating the presence / absence of the singular part by learning the image data of the second format stored in the database DB1 and the database DB2 as teacher data. doing. The internal parameters are, for example, a bias value in the convolution layer, a weighting coefficient in the connection layer, and the like. Internal parameters fluctuate with the addition of teacher data.

The trained model in the present embodiment is a trained model configured to output information regarding the presence or absence of a singular portion included in an input image of a predetermined size. The trained model in the present embodiment is generated by machine learning using teacher data in which the presence or absence of a singular part is labeled for each of the plurality of second-format image data. The trained model may output information regarding the presence / absence of a singular portion included in the input image and / or information regarding the type of the singular portion. Such a trained model is generated by machine learning using teacher data in which the presence / absence of a singular part and / or the type of the singular part is labeled for each of the plurality of second-format image data.

FIG. 2 is a block diagram showing a hardware configuration for realizing an evaluation device constituting the evaluation system 10. As described above, the imaging unit 12 captures an object and acquires a first image corresponding to the first range of the object, and is composed of, for example, a line camera. The processor 22 executes a computer program (including a trained model for evaluating the presence / absence of a singular part) stored in the storage unit 24, and executes each arithmetic process shown in the present embodiment. Therefore, the processor 22 and the storage unit 24 cooperate with each other to function as the singular part detection unit 14, the singular part image extraction unit 16, the image data conversion unit 18, and the singular part presence / absence evaluation unit 20. The processor 22 is composed of, for example, an ASIC (Application Specific Integrated Circuit) having a plurality of arithmetic cores, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a quantum computer, or the like. The storage unit 24 stores each instruction and other information including a computer program (including a learned model for evaluating the presence / absence of a singular part) for executing each arithmetic processing shown in the present embodiment. The storage unit 24 is composed of a non-volatile semiconductor storage element (non-temporary storage element) such as a NAND flash memory, FeRAM, or MRAM that can electrically record and read information, or a magnetic storage element such as an HDD (Hard Disc Drive). Will be done. The RAM 26 is a volatile semiconductor storage element such as a SRAM (Static Random Access Memory) and a DRAM (Dynamic Random Access Memory) for primary storage of data and other information used for each arithmetic processing and the like shown in the present embodiment. Consists of. The display unit 28 includes a display for displaying the calculation result by the processor 22. The calculation result includes the evaluation result by the singular part presence / absence evaluation unit 20. The communication I / F unit 30 is connected to the database DB1 and the database DB2 via the network N, and can receive information from the database DB1 and the database DB2. The input unit 32 includes an input device such as a keyboard for the evaluator to input information into the evaluation system 10. Each of these configurations is connected to each other via a bus so that data can be transmitted and received. However, a partial configuration may be provided at a remote location via the network N, or may be integrated with another configuration. Further, a part of the functions executed by the processor 22 may be executed by another configuration such as the imaging unit 12.

The evaluation method of the singular part using the evaluation system 10 as described above will be described. FIG. 3 is a flowchart of the evaluation method shown in the present embodiment.

First, the imaging unit 12 images an object. Specifically, while rotating the PET bottle, which is an object, around the central axis, the side surface of the PET bottle is imaged by the line camera of the imaging unit 12, and the first image corresponding to the first range is imaged. In the present embodiment, the imaging unit 12 captures an image of 5000 pixels × 5000 pixels as the first image corresponding to the first range. Further, in the present embodiment, the imaging unit 12 acquires grayscale data having a brightness value of 8 bits (256 gradations) as a first image for each pixel of 5000 pixels × 5000 pixels (step S1). Here, the container of the PET bottle has translucency. Therefore, the imaging unit 12 acquires an image of the PET bottle container and the liquid contained therein.

Next, the singular part detection unit 14 detects the singular part from the image of the side surface of the PET bottle acquired by the imaging unit 12 (step S2). Specifically, the grayscale data acquired corresponding to each pixel is binarized using an adaptively set threshold value, and the number of pixels exceeding the threshold value included in the predetermined area is a predetermined number or more. In some cases, the singular part is detected.

Next, the singular part detection unit 14 determines whether or not the singular part is detected in the image of the side surface of the PET bottle acquired by the imaging unit 12 (step S3). In the present embodiment, the singular part detection unit 14 detects the singular part included in the first image corresponding to the first range by rule-based image processing.

If the singular part is not detected and therefore the second range containing the singular part is not extracted (NO), the inspection for the object ends. A simple visual inspection may be performed on the object in which the singular part is not detected.

When the singular part is detected (YES), the singular part image extracting unit 16 starts with the singular part detecting unit 14 from the first image corresponding to the first range (the image captured by the imaging unit 12 in the present embodiment). A region containing all or part of the singular part detected by is extracted (step S4). Specifically, the singular part image extraction unit 16 extracts a region of 256 pixels × 256 pixels including one or a plurality of singular parts detected by the singular part detection unit 14 as a singular part image. FIG. 4A shows a first range AR1 corresponding to the side surface of the PET bottle acquired by the imaging unit 12, and three second ranges AR2 to AR4 extracted from the image data of the first format corresponding to the first range AR1. show. 4B to 4D show images of the second range AR2 to AR4 and the singular part candidates C2 to C4 included therein. The plurality of second ranges may be extracted so that some of them overlap with each other. As a result, in the first image corresponding to the first range, the region where the singular portion exists is narrowed down by a predetermined image processing.

Next, the image data conversion unit 18 converts the grayscale data of each pixel corresponding to the second range AR2 to AR4 into RGB data, and generates RGB data of each pixel corresponding to the second range AR2 to AR4 ( Step S5).

Next, the singular part presence / absence evaluation unit 20 evaluates whether or not the singular part is included in the second range based on the trained model generated by using machine learning (step S6).

When it is evaluated that no singular part is contained in any of the second ranges AR2 to AR4 (NO), it is evaluated that there is no singular part for the object, and the inspection is completed. A simple visual inspection may be performed on the object evaluated as having no singular part.

When it is evaluated that any of the second ranges AR2 to AR4 contains a singular portion (YES), information to that effect is output to encourage the inspector to perform a visual inspection. As a result, a visual inspection is performed by an inspector (step S7). At this time, as shown in FIG. 4A, the display unit 28 includes the first range AR1 and the second range (for example, the second range AR3 and the second range AR4) evaluated to include the singular part. It is configured to display a position within the first range AR1. With this configuration, the inspector can easily identify the region evaluated as containing the singular part. Therefore, it is possible to reduce the visual inspection time. When the position in the first range AR1 of the second range evaluated to contain the singular part is not displayed, the inspector inspects a wide area where it is uncertain whether the singular part is actually included. Must.

The inspector determines whether or not the singular part was actually included in the second range evaluated as containing the singular part. At this time, the inspector can input information indicating whether or not the singular portion is actually included in the second range evaluated to include the singular portion by using the input unit 32. The input information and the corresponding RGB data in the second range can be configured to be stored in the database DB1 or the database DB2 as teacher data.

According to the evaluation system 10 and the evaluation method as described above, the evaluation target area of the second method by machine learning is extracted by using the image data of the first format, which has a smaller amount of information than the image data of the second format. Therefore, the amount of calculation required for the image processing of the extraction of the second range can be suppressed, and as a result, the calculation time can be shortened. In addition, it is possible to reduce image data other than the second range. In the evaluation of the second method by machine learning, necessary information is added and the image data of the second format having a large amount of information is used, so that the evaluation accuracy can be maintained or improved. Therefore, it is possible to quickly evaluate the presence or absence of the singular part in a large number of objects while maintaining or improving the detection accuracy of the singular part.

Further, in the evaluation method in the present embodiment, as a second method, a method using a model obtained by machine learning that can autonomously generate an estimation rule for input data based on teacher data (training data). Is adopted. Further, as the training data used for machine learning, the image data of the second format, which has a larger amount of data indicating the difference between the pixels than the image data of the first format, is used. Thereby, as described with reference to FIG. 5 below, both the oversight rate and the overdetection rate of the singular portion can be reduced.

Further, the image data of the second format is image data having a smaller number of pixels than the image data of the first format (more typically, image data having a smaller size range), so that the amount of data of each pixel can be increased. While increasing, the total amount of data from the image data of the first format is reduced. This prevents an excessive increase in the time (learning time) required to generate the trained model while improving the accuracy of the evaluation of the singular part contained in the second image by applying the second method. be able to.

In particular, assuming that the evaluation method in the present embodiment is implemented on a production line and used for product inspection, a trained model is generated in order to flexibly respond to a lot change or a model change of a product to be manufactured. The time is preferably short. As described above, by suppressing the increase in learning time, it is possible to realize an evaluation method suitable for mounting on a production line.

From the viewpoint of further enhancing the above-mentioned effect, as the first method, there are few elements to be changed according to the lot change or model change of the product to be manufactured, and it can be applied to image processing with a large amount of data. It is preferable to use a rule-based image processing method.

FIG. 5 shows a case where the presence / absence of a singular part is evaluated by grayscale data and a case where the presence / absence of a singular part is evaluated by RGB data for 5000 or more objects using the singular part presence / absence evaluation unit 20. It is a graph which shows the overlook rate and the over-detection rate of.

When the presence or absence of the singular part was evaluated using grayscale data, there was an oversight rate of 0.05%, which was evaluated as having no singular part, even though the singular part was present. On the other hand, when the presence or absence of the singular part was evaluated using RGB data, the oversight rate evaluated as having no singular part could be set to zero even though the singular part was present. Even if a singular part is present in only one of the objects to be mass-produced, a big problem may occur. However, by using the evaluation system 10 and the evaluation method according to the present embodiment, the singular part is used. I was able to reduce the oversight rate to zero. It is presumed that the reason is that the image data of the second format having a large amount of information was used when performing the evaluation by the second method acquired by using machine learning. On the other hand, when executing the first method, it is possible to shorten the calculation time by using the image data of the first format having a small amount of information. In addition, the over-detection rate evaluated as having a singular part even though there is no singular part was about 5.6% in the case of grayscale data, while it was about 3.6% in the case of RGB data. It decreased greatly. When over-detection is performed, a long-time visual inspection is performed, so that the evaluation time can be significantly shortened by reducing the over-detection. In this way, it was confirmed that both the oversight rate and the overdetection rate were improved.

Further, in the present embodiment, since the image data of the second format is generated based on the image data of the first format, the image data of the second format is generated when the second range is not extracted by the first method. You don't have to. Further, since the image data of the second format is generated by using the image data after the image processing executed by the first method, the same image processing may be omitted or simplified when the second method is executed. Becomes possible. Therefore, the calculation time can be further shortened. Therefore, it is possible to efficiently evaluate a large number of objects.

It should be noted that the plurality of objects may be simultaneously imaged by taking an image while moving the plurality of objects so as to pass through the region to be imaged by the imaging unit 12. According to this, it becomes possible to efficiently evaluate a plurality of objects. At that time, the evaluation system 10 may be configured so that the entire side surface can be evaluated by moving the object while rotating it around the central axis. Further, for the first object, the imaging unit 12 and the singular part detection unit 14 are used to perform imaging and detection of the singular part, and at the same time, for the other object, the singular part presence / absence evaluation unit 20 is used. Evaluation of the presence or absence of singularities may be performed. For example, with respect to the second image including the singular part detected by the singular part detection unit 14, the singular part is detected by sequentially evaluating the presence or absence of the singular part by using the singular part presence / absence evaluation unit 20. The evaluation of the presence or absence of the singular part may be performed in parallel.

[How to generate teacher data]
Hereinafter, a method of generating teacher data stored in the database DB1 and the database DB2 will be described.

FIG. 6A is a schematic diagram illustrating a method of generating teacher data based on RGB data of an image (an example of a “third image”) corresponding to the second range AR3 shown in FIG. 4C. As described above, the second range AR3 is a singular part image having a region of 256 pixels × 256 pixels and including the singular part candidate C2. The third image, which is the original image for generating the teacher data, does not necessarily have to be the same size as the second range AR3. For example, teacher data may be generated based on an image larger than the second range AR3.

In the present embodiment, the teacher data is generated by cutting out four fourth images from the second range AR3, which are smaller than the second range AR3 and have different positions of the singular part candidate C3. Specifically, as the first fourth image, an image in the range of 228 pixels × 228 pixels having the apex V1 of the second range AR3 as the apex is cut out. This fourth image is an image in which the vertices V1 and V7 shown in the figure are diagonal vertices and include the singular part candidate C3. Similarly, as the second fourth image, the vertices V2 and V8 of the second range AR3 are diagonal vertices, and as the third fourth image, the vertices V3 and V5 of the second range AR3 are diagonal. The teacher data is obtained by cutting out an image having a size of 228 pixels × 228 pixels, each having the vertices V4 and the vertices V6 of the second range AR3 as the vertices of the second range AR3 as the vertices of 4 RGB data to be generated are generated. When it is confirmed by visual inspection that the singular part candidate C3 is a singular part, the generated teacher data is stored in the database DB2 that stores the abnormal sample. When it is confirmed that the singular part candidate C3 is not a singular part such as a bubble, it is stored in the database DB1 that stores a normal sample.

In this way, it is possible to generate four teacher data based on one third image. The position of the singular part candidate C3 in each teacher data is different. On the other hand, the position of the singular portion in the image extracted by the singular portion image extracting unit 16 is not constant. Therefore, by generating a plurality of teacher data in which the positions of the singular part candidate C3 are different, it is possible to improve the evaluation accuracy by the singular part presence / absence evaluation unit 20.

FIG. 6B is an image obtained by rotating the third image shown in FIG. 6A by 90 degrees. Similarly, by cutting out the four fourth images based on this image, it becomes possible to further generate four teacher data having different positions and angles of the singular part candidate C3. On the other hand, the position and angle of the singular portion in the image extracted by the singular portion image extracting unit 16 are not constant. Therefore, by generating a plurality of teacher data having different angles of the singular part candidate C3, it is possible to improve the evaluation accuracy by the singular part presence / absence evaluation unit 20.

Further, more teacher data may be generated by varying the color of each of the generated teacher data. For example, in each layer of RGB of color data whose color is expressed by the brightness of each primary color of RGB, more teacher data is generated by varying the brightness value and contrast, and the evaluation by the evaluation system 10 is robust. It becomes possible to improve the sex. In particular, if the object is a liquid-containing object, or if the object does not contain a liquid but has a translucent container, the liquid portion or the translucent container will emit light. The brightness value of each layer of RGB may differ depending on the surrounding environment such as how the light hits. Therefore, by generating such teacher data, it is possible to improve the evaluation accuracy by the singular part presence / absence evaluation unit 20. The color variation may be performed by, for example, randomly varying the luminance value and the contrast, or, for example, by adding or subtracting the same value to each of the luminance values of each of the primary colors of RGB. Good. When generating a plurality of fourth images from one third image, the number of generated teacher data of the abnormal sample including the singular part (for example, 12 teacher data) is used as the teacher data of the normal sample not including the singular part. It may be larger than the number of generated numbers (for example, 8 teacher data). With such a configuration, the presence / absence evaluation unit 20 of the singular part can be over-learned so as to perform the evaluation having the singular part, and the possibility of overlooking the singular part can be reduced.

[First modification]
Hereinafter, a modified example of the first embodiment will be described. In the first embodiment, the grayscale data of the object was acquired by using the line sensor included in the imaging unit 12. The imaging unit 12 in this modification acquires the first image of the object by using an area sensor (for example, a CMOS image sensor) that acquires two-dimensional RGB data. The singular part detection unit 14 generates grayscale data of the first image (an example of "image data of the first format") based on the RGB data acquired by the imaging unit 12, and based on this, the singular part is generated. To detect. Further, the singular part presence / absence evaluation unit 20 acquires RGB data (an example of “image data of the second format”) corresponding to the second range including the detected singular part from the imaging unit 12, and based on this, the second Evaluate the presence or absence of singularities in two ranges.

Also in such an aspect, it is possible to provide an evaluation method and an evaluation device capable of rapidly evaluating the presence or absence of a singular part in a large number of objects while maintaining or improving the detection accuracy of the singular part. become. Further, since it becomes possible to evaluate the presence or absence of the singular portion in the second range based on the high-resolution color data acquired by the imaging unit 12 and different color data (for example, color data obtained by the CMYK color model). Depending on the evaluation target, it is possible to improve the evaluation accuracy by the second method.

[Scope of application]
The evaluation system and evaluation device according to the present disclosure can be applied to evaluate various objects. For example, in the process of manufacturing a transparent film of polyvinylidene chloride (PVDC), it can be used to detect foreign substances, scratches, wrinkles and the like that may be contained in the transparent film as peculiar parts.

Further, in the manufacturing process of a polymer film for a flat panel display or the like, it can be used to detect foreign matter, scratches, etc. that may be contained in the polymer film as a peculiar portion.

Further, in a manufacturing process of a glass product or the like, it can be used to detect cracks or scratches that may be contained in the glass as a peculiar portion. In this case, the singular portion can be detected by illuminating the glass product and taking an image, and utilizing the fact that the brightness of the singular portion is higher than that of other portions.

In addition, it can be used to detect deformation of the membrane surface and bubbles as singular portions in the process of manufacturing a separation membrane for chromatography. In this case, the separation film is illuminated and an image is taken, and the singular portion can be detected by utilizing the fact that the brightness of the singular portion is lower or higher than that of other portions.

In addition, the present disclosure may be modified in various ways as long as it does not deviate from the gist thereof. For example, some components in one embodiment may be added to other embodiments within the normal creative abilities of those skilled in the art. Also, some components in one embodiment can be replaced with corresponding components in another embodiment.

For example, the evaluation method according to the present disclosure is
Acquiring an image corresponding to the first range of the object,
Using the first method, an image corresponding to the first range, from the first image in which the difference in color of each pixel is represented by the first index, is a part of the first range. Extracting 2 ranges and
An image corresponding to the second range, in which the difference in color of each pixel is expressed by the first index, is given a second index indicating the difference in color of each pixel. Then, the image of the third image in which the difference in color of each pixel is expressed by the first index and the second index is obtained.
Using the second method obtained by using machine learning, the presence or absence of a singular part in the third image is evaluated, and
It may be an evaluation method including.

Here, the first method is preferably a rule-based method. Further, the first method is preferably a method of extracting a second range including a candidate for a singular portion whose presence or absence should be determined by the second method.

The first image is preferably a grayscale image. At this time, the first index may be the brightness of the pixels expressing the difference in color of each pixel. The second image is preferably a color image. At this time, the second index may be the brightness of each of the three primary colors expressing the color of each pixel. When the second index includes the first index, the second index may be given by removing the first index from the second index.

Further, the second index indicating the difference in color of each pixel is added to the second image within a range in which the amount of data of the third image obtained thereby does not exceed the amount of data of the first image. Is preferable.

For example, the evaluation method according to the present disclosure is
Acquiring the first image of the object and
Using the first method, a part of the first image of the first format in which the color of each pixel is expressed by the first color information in the first image is a candidate region in which a singular portion exists. Extracting as a second image and
By adding a second color information representing the color of each pixel to the second image of the first format, the color of each pixel is expressed by the first color information and the second color information. To obtain the third image of the second format
Using the second method obtained by using machine learning, the presence or absence of the singular part can be identified from the third image of the second format.
It may be an evaluation method including.

For example, the evaluation means according to the present disclosure is
Means for acquiring the first image of the object,
Using the first method, a part of the first image of the first format in which the color of each pixel is expressed by the first color information in the first image is a candidate region in which a singular portion exists. Means to extract as a second image and
By adding a second color information representing the color of each pixel to the second image of the first format, the color of each pixel is expressed by the first color information and the second color information. A means of acquiring a third image of the second format,
A means for identifying the presence or absence of a singular portion from the third image of the second format by using the second method acquired by using machine learning.
It may be an evaluation device including.

Here, the first image may be an image corresponding to the first range of the object, and the second image and the third image may be images corresponding to the second range which is a part of the first range. ..

Further, the first and second color information may be index information indicating the difference in color of each pixel, respectively.

The above-mentioned "index" may be expressed as "color information".

The "evaluation" described above includes identification.

The following additional notes are further disclosed with respect to the above-described embodiment.
(Appendix 1)
Acquiring the first image of the object and
Using the first method, a part of the first image of the first format, which is the first image and is represented by one index per pixel, is extracted as a second image.
Using the second method acquired by using machine learning, the presence or absence of a singular portion is evaluated from the second image of the second format represented by a plurality of indexes per pixel in the second image. When,
Evaluation method including.
(Appendix 2)
To generate the second image of the second format based on the first image of the first format.
The evaluation method according to Appendix 1, further comprising.
(Appendix 3)
In the extraction of the second image, a plurality of the second images are extracted from the first image, and the second image is extracted.
The evaluation of the presence or absence of the singular portion is performed on the plurality of the second images.
The evaluation method according to Appendix 1 or 2.
(Appendix 4)
Judging whether or not to perform visual evaluation based on the evaluation result of the presence or absence of the singular part of the second image.
The evaluation method according to any one of Appendix 1 to 3, further comprising.
(Appendix 5)
When the second image cannot be extracted from the first image by using the first method, it is determined that the singular part does not exist in the range of the first image of the object.
The evaluation method according to any one of Supplementary Provisions 1 to 4, further comprising.
(Appendix 6)
When the extracted second image does not have the singular part, it is evaluated that the object does not have the singular part.
The evaluation method according to Appendix 3, further comprising.
(Appendix 7)
The machine learning
From the third image including the singular part, a plurality of fourth images smaller than the third image and having different positions of the singular part are generated.
Optimizing internal parameters using the fourth image, which is a plurality of the fourth images and is expressed in the second format, as teacher data.
The evaluation method according to any one of Appendix 1 to 6, which comprises.
(Appendix 8)
Means for acquiring the first image of the object,
A means for extracting a part of the first image of the first format, which is the first image and is represented by one index per pixel, as a second image by using the first method.
A means for evaluating the presence or absence of a singular portion from the second image of the second format, which is the second image and is represented by a plurality of indexes per pixel, by using the second method acquired by using machine learning. When,
Evaluation device including.
(Appendix 9)
A means for generating the second image of the second format based on the first image of the first format.
The evaluation device according to Appendix 8, further comprising.
(Appendix 10)
The means for extracting the second image is configured to extract a plurality of the second images from the first image.
The means for evaluating the presence or absence of the singular portion is configured to evaluate the presence or absence of the singular portion with respect to the plurality of the second images.
The evaluation device according to Appendix 8 or 9.
(Appendix 11)
A means for determining whether or not to perform visual evaluation based on the evaluation result of the presence or absence of a singular portion in the second image.
The evaluation device according to any one of Supplementary Provisions 8 to 10, further comprising.
(Appendix 12)
A means for determining that the singular portion is not present in the range of the first image of the object when the second image cannot be extracted from the first image by using the first method.
The evaluation device according to any one of Supplementary Provisions 8 to 11, further comprising.
(Appendix 13)
A means for evaluating that the object does not have the singular part when the extracted second image does not have the singular part.
The evaluation device according to Appendix 10, further comprising.
(Appendix 14)
The machine learning
From the third image including the singular part, a plurality of fourth images smaller than the third image and having different positions of the singular part are generated.
Optimizing internal parameters using the fourth image, which is a plurality of the fourth images and is expressed in the second format, as teacher data.
The evaluation device according to any one of Supplementary Provisions 8 to 13, including the above.

10 Evaluation system 12 Imaging unit 14 Singularity detection unit 16 Singularity image extraction unit 18 Image data conversion unit 20 Singularity presence / absence evaluation unit 22 Processor 24 Storage unit 28 Display unit 30 Communication I / F unit 32 Input unit AR1 First range AR2 2nd range AR3 2nd range AR4 2nd range

Claims (16)

Acquiring the first image corresponding to the first range of the object,
Using the first method, a second range, which is a part of the first range, is extracted from the image data of the first image, which is the image data of the first format represented by one index per pixel. That and
Using the second method acquired by using machine learning, the image data of the second image corresponding to the second range, which is the image data of the second format represented by a plurality of indexes per pixel, is described as described above. Evaluating the presence or absence of singularities within the second range and
Evaluation method including. To generate the image data of the second format based on the image data of the first format.
The evaluation method according to claim 1, further comprising. In the extraction of the second range, a plurality of the second ranges are extracted from the image data of the first image.
The evaluation of the presence or absence of the singular portion is performed on the image data of the plurality of second images corresponding to each of the plurality of second ranges.
The evaluation method according to claim 1 or 2. Judging whether or not to perform visual evaluation based on the evaluation result of the presence or absence of the singular part in the second range.
The evaluation method according to any one of claims 1 to 3, further comprising. When the second range, which is a part of the first range, cannot be extracted from the image data of the first image by using the first method, the singular part is not present in the first range of the object. Judging,
The evaluation method according to any one of claims 1 to 4, further comprising. When the singular part is not present in the plurality of extracted second ranges, it is evaluated that the object does not have the singular part.
The evaluation method according to claim 3, further comprising. The machine learning
From the third image including the singular part, a plurality of fourth images smaller than the third image and having different positions of the singular part are generated.
Optimizing the internal parameters using the image data of the plurality of the fourth images expressed in the second format as the teacher data.
The evaluation method according to any one of claims 1 to 6, which comprises. The generation of the fourth image is
To prepare a plurality of the third images, each containing a different type of singularity,
Including generating a plurality of fourth images from each of the plurality of the third images.
The evaluation method according to claim 7, wherein the plurality of fourth images include the same singular portion and have images in which the positions of the singular portions are different. A means of acquiring a first image corresponding to a first range of an object,
Using the first method, a second range, which is a part of the first range, is extracted from the image data of the first image, which is the image data of the first format represented by one index per pixel. Means and
Using the second method acquired by using machine learning, from the image data of the second image corresponding to the second range, which is the image data of the second format represented by a plurality of indexes per pixel, from the image data of the second format. A means for evaluating the presence or absence of a singular part within the second range, and
An evaluation device equipped with. A means for generating image data in the second format based on the image data in the first format.
9. The evaluation device according to claim 9. The means for extracting the second range includes means for extracting a plurality of the second range from the image data of the first image.
The means for evaluating the presence or absence of the singular portion includes means for performing the evaluation on the image data of the plurality of the second images corresponding to each of the plurality of the second ranges.
The evaluation device according to claim 9 or 10. A means for determining whether or not to perform visual evaluation based on the evaluation result of the presence or absence of a singular portion within the second range.
The evaluation device according to any one of claims 9 to 11, further comprising. When the second range, which is a part of the first range, cannot be extracted from the image data of the first image by using the first method, the singular part is not present in the first range of the object. Judgment means,
The evaluation device according to any one of claims 9 to 12, further comprising. A means for evaluating that the object does not have the singular part when the extracted singular part does not exist in the plurality of the second ranges.
The evaluation device according to any one of claims 11, further comprising. The machine learning
A means for generating a plurality of fourth images that are smaller than the third image and have different positions of the one singular portion from the third image including the one singular portion.
It is configured to be executed by using a means for optimizing internal parameters using a plurality of image data of the fourth image and the image data expressed in the second format as teacher data.
The evaluation device according to any one of claims 9 to 14. The means for generating the fourth image is
Means for preparing a plurality of the third images, each containing a different type of singularity,
A means for generating a plurality of fourth images from each of the plurality of the third images is provided.
The plurality of fourth images include images containing the same singular portion and having different positions of the singular portions.
The evaluation device according to claim 15.

Images