JP2021064215A - Surface property inspection device and surface property inspection method - Google Patents

Abstract

To provide a surface property inspection device and a surface property inspection method for inspecting the surface property of an object with higher accuracy than before.SOLUTION: A surface property inspection device uses a model 141 learned by deep learning to calculate the certainty factor of each defect seed on a pixel-by-pixel basis from a captured image Icam that images the surface of an object, generates a certainty factor map image Imap_i with the certainty factor as a pixel value for each defect seed, performs labeling to extract a defect candidate area blob_j based on the pixel value of the certainty factor map image for each defect seed, calculates, as feature quantities, geometric features of the defect candidate area blob_j, statistic of the pixel value of the captured image Icam in the defect candidate area blob_j, and statistic of the certainty factor in the defect candidate area blob_j, and determines, from the calculated feature quantities, the defect seed on the surface of the object and the score for each defect seed.SELECTED DRAWING: Figure 3

Description

The present invention relates to a surface property inspection device and a surface property inspection method for optically inspecting the surface properties of an object.

Conventionally, techniques for detecting various defects related to surface properties such as irregularities, flaws, color unevenness abnormalities, and roughness abnormalities generated on the surface of an object are well known (for example, Patent Document 1 and Patent Document 2). And Patent Document 3). In recent years, with the improvement of steel sheet quality, the demand for defects existing on the surface of the steel sheet has become stricter, and in order to stably ship the steel sheet satisfying the demand, the surface of the steel sheet is imaged with an imaging device and the captured image is used. It is desired to improve the detection accuracy of an inspection device that automatically detects defects.

Conventional inspection of the surface texture of an object such as a steel plate is generally performed according to the flow shown in FIG. First, the captured image obtained by capturing the surface of the object is binarized by threshold processing (step S1). Next, each region obtained by binarization is labeled and recognized as an aggregate of pixels (blobs), and the blobs are extracted as defect candidate regions, which are regions occupied by defects on the surface of the object. (Step S3). Then, for the extracted defect candidate region, geometric features such as area and statistics of pixel values are calculated as feature quantities (step S5). Next, from the calculated features, the defect type, which is a type of defect that can occur in the object, and the score for each defect type (for example, the score indicating the severity of the defect) are determined. (Step S7).

Japanese Unexamined Patent Publication No. 5-164701 Japanese Unexamined Patent Publication No. 11-45328 Japanese Patent No. 2535257

In the conventional defect determination flow shown in FIG. 1, a defect having a clear outline or a defect having a certain size can be detected, but a defect that appears only very thinly in the captured image or an irregular defect is detected. It's difficult.

For example, in order to detect a defect that appears only very faintly in the captured image, it is conceivable to set the threshold value low in the threshold processing in step S1, but in that case, other than defects such as noise and patterns. A large amount of regions will be detected, and there is a risk that a large number of overdetections will occur in which regions that are not defects are detected as defects. Further, since harmful defects are extracted as defect candidates integrated with noise and patterns, they have a characteristic amount different from the characteristic amount distribution of harmful defects set in the determination logic in step S7, which is harmful. There is a risk that undetected defects will be overlooked.

In addition, since geometric features such as area and shape of amorphous defects are not constant, they may be distributed over a wider range in the feature space than the range normally assumed as the feature of defects. It may overlap with the area of other defect species. Therefore, it is difficult to set the determination logic in step S7, and there arises a problem that undetection or overdetection is likely to occur.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a surface property inspection device and a surface property inspection method for inspecting the surface properties of an object with higher accuracy than before.

The surface property inspection apparatus according to the embodiment of the present invention uses a trained model by deep learning to calculate the certainty of each defect type from the captured image obtained by capturing the surface of the object in pixel units or region units. Based on the deep learning determination processing unit that generates a certainty map image with the certainty as a pixel value for each defect type and the pixel value of the certainty map image for each defect type, labeling is performed to perform a defect candidate region. At least one of a labeling unit for extracting a defect, a geometric feature of the defect candidate region, a statistic of the pixel value of the captured image in the defect candidate region, and a statistic of the certainty in the defect candidate region. It is provided with a feature amount calculation unit that calculates one as a feature amount.

In the surface property inspection method according to the embodiment of the present invention, the certainty of each defect type is calculated for each pixel or region from the captured image obtained by capturing the surface of the object by using the trained model by deep learning. Based on the deep learning determination processing step of generating a certainty map image with the certainty as a pixel value for each defect type and the pixel value of the certainty map image for each defect type, labeling is performed to perform a defect candidate region. At least one of a labeling step for extracting a defect, a geometric feature of the defect candidate region, a statistic of the pixel value of the captured image in the defect candidate region, and a statistic of the certainty in the defect candidate region. It includes a feature amount calculation step of calculating one as a feature amount.

According to the present invention, using a trained model by deep learning, the certainty of each defect type is calculated from the captured image of the surface of the object in pixel units or region units, and the certainty map image is used as the defect type. By generating each feature and calculating the feature amount from the certainty map image, it is possible to improve the detection accuracy of defects on the surface of the object as compared with the conventional case.

It is a figure which shows the flow of the conventional surface property inspection method. It is a schematic diagram of the hardware structure of the surface property inspection apparatus which concerns on 1st to 3rd Embodiment of this invention. It is a figure which shows the flow of the surface property inspection method which concerns on 1st Embodiment. It is a schematic diagram which shows an example of the captured image of the surface of an object. It is a schematic diagram which shows a pixel of interest in a captured image and a region around it. It is a schematic diagram which shows an example of the certainty degree map image of the defect type obtained by the deep learning determination process. It is a figure which shows the flow of the surface property inspection method which concerns on 2nd Embodiment. It is a figure which shows the flow of the surface property inspection method which concerns on 3rd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each of the following embodiments, the same reference numerals are used for the same or similar configurations, and duplicate description will be omitted.

<First Embodiment>
First, the configuration of the surface property inspection device 100 according to the first embodiment will be described. FIG. 2 is a diagram schematically showing a hardware configuration of the surface property inspection device 100. The surface property inspection device 100 is a device that optically inspects the surface properties of an object such as a steel plate, and as shown in FIG. 2, includes an image pickup device 11, a processor 12, a display unit 13, and a storage unit 14. , Equipped with.

The image pickup device 11 includes an optical element such as a lens and an image pickup device such as a Charge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS). Here, the image pickup apparatus 11 may be capable of capturing a gray image or may be capable of capturing a color image. The image pickup device 11 includes an A / D converter, converts an analog signal output from the image pickup device into a digital signal, and uses the digital signal as a signal constituting the captured image obtained by the image pickup device 11. Output to processor 12.

The processor 12 has a Central Processing Unit (CPU) and controls each element constituting the surface texture inspection device 100 in an integrated manner. Further, the processor 12 executes the surface texture inspection method (FIG. 3) according to the program stored in the storage unit 14.

Specifically, the processor 12 uses the trained model 141 by deep learning stored in the storage unit 14 to capture the surface of the object by the imaging device 11, and the degree of certainty for each defect type on a pixel-by-pixel basis. A deep learning judgment processing unit that calculates and generates a certainty map image with the certainty as a pixel value for each defect type, and labeling each pixel of the certainty map image for each defect type based on the pixel value. A labeling unit that extracts defect candidate regions, a feature amount calculation unit that calculates geometric features of defect candidate regions as feature quantities, and a feature quantity that determines the defect type on the surface of the object and the score for each defect type. Functions as a judgment unit. The surface texture inspection method (FIG. 3) executed by the processor 12 will be described in detail later.

It should be noted that the feature amount calculation unit not only uses the geometrical features of the defect candidate region as the feature amount, but also the geometrical features of the defect candidate region, the statistic of the pixel value of the captured image in the defect candidate region, and the statistics. At least one of the statistic of the certainty in the defect candidate region can be used as the feature quantity. Further, the function of the determination unit may be provided in another inspection device or the like connected to the surface property inspection device 100 without including the function of the determination unit in the processor 12.

The display unit 13 includes a display such as a liquid crystal display (LCD), a plasma display, or an organic electroluminescence (EL) display, and displays the processing result by the processor 12.

The storage unit 14 has a Read Only Memory (ROM) and a Random Access Memory (RAM). The ROM stores various programs such as a surface texture inspection processing program for executing the surface texture inspection method (FIG. 3 and the like) by the processor 12 and data necessary for executing these programs. Various programs and data stored in the ROM are loaded into the RAM and executed.

The storage unit 14 stores the trained model 141 by deep learning as data necessary when executing the surface texture inspection processing program. The trained model 141 is used as a defect type of the defect after performing a separate inspection such as a visual inspection by an inspector in advance on the captured image data obtained by imaging various defects on the surface of the object. It is obtained by causing a deep neural network (DNN) to learn a large amount of training data obtained by assigning a label of a defect type that is considered to be appropriate as a judgment result of the inspection to each pixel or region.

The storage unit 14 may include a magnetic memory such as a hard disk drive (HDD) or an optical memory such as an optical disk. Alternatively, various programs and data may be stored in a recording medium that can be attached to and detached from the surface property inspection device 100.

The processor 12 may be configured by dedicated hardware specialized for each function of the processor 12 instead of general-purpose hardware such as a CPU. For example, the above-mentioned functions of the deep learning determination processing unit, the labeling unit, the feature amount calculation unit, and the determination unit may be implemented in an integrated circuit (ASIC) for a specific application. The same applies to the processor 12 according to the following second and third embodiments.

Next, the surface property inspection process executed by the surface property inspection device 100 will be described with reference to FIGS. 3 to 6.

First, the processor 12 uses the trained model 141 by deep learning stored in the storage unit 14 to image the surface of the object by the image pickup device 11, and obtains the captured image Icam (gray image) in pixel units. The certainty of the defect type is calculated, and the certainty map image Imap_i with the certainty as the pixel value is generated for each defect type (step S11). Here, "i" is an index representing a defect type and is a positive integer.

ここで、DNN(i, Icam(q|q∈R(p)))は、撮像画像Icamの画素ｐが、欠陥種ｉ（ｉ＝１、２、…Ｎ）である確信度を与える深層学習モデルであり、R(p)は、画素ｐ（注目画素６０）の近傍領域６１を与える関数であり、ｑは関数R(p)によって規定された近傍領域６１内の画素である。撮像画像Icam内の各画素ｐについて、式（１）により確信度Imap_i（ｐ）を求めることにより、確信度マップ画像Imap_iが生成される。 FIG. 4 shows an example of a captured image Icam (gray image) obtained by capturing the surface of an object. In step S11, as shown in FIG. 5, the trained model 141 is used and exists in the attention pixel 60 based on the brightness value of each pixel in the vicinity region 61 centered on the attention pixel 60 in the captured image Icam. For defects, the certainty is calculated for each defect type. The width W and height H of the neighboring region 61 are determined by the size of the defect to be detected, and are represented by the network structure of the deep learning model. The certainty Imap_i (p) of the defect type i in the pixel p is given by Eq. (1).

Here, DNN (i, Icam (q | q ∈ R (p))) is a deep learning that gives the certainty that the pixel p of the captured image Icam is the defect type i (i = 1, 2, ... N). In the model, R (p) is a function that gives a neighborhood region 61 of the pixel p (pixel of interest 60), and q is a pixel in the neighborhood region 61 defined by the function R (p). The certainty map image Imap_i is generated by obtaining the certainty Imap_i (p) by the equation (1) for each pixel p in the captured image Icam.

An example of the certainty map image obtained from the captured image Icam of FIG. 4 is shown in FIG. In FIG. 6, the confidence map images Imap_1, Imap_2, Imap_3 and Imap_4 correspond to defect types 1, 2, 3 and 4, respectively. Each pixel value of the certainty map image represents a luminance value, and the higher the luminance value, the higher the certainty for each defect type.

In FIG. 6, a region having a high luminance value appears in the certainty map image Imap_1, and in the other certainty map images Imap_2, Imap_3 and Imap_4, the luminance value is extremely low in the entire image. That is, it is shown that the certainty of the defect type 1 is high and the certainty of the defect types 2 to 4 is extremely low for the object reflected in the captured image Icam of FIG. In the certainty map image, the certainty may be expressed by a color instead of a luminance value.

Next, the processor 12 extracts the defect candidate region, which is the region occupied by the defect on the surface of the object, by labeling the certainty map image Imap_i based on the pixel value of each pixel for each defect type i. (Step S13). Specifically, in step S13, the processor 12 concatenates adjacent pixels having similar brightness values (the difference between the brightness values is less than or equal to the specified value) in the certainty map image Imap_i and regards them as one blob, and regards the certainty map. Each blob obtained from the image Imap_i is extracted as a defect candidate region blob_j. Here, "j" is an index that identifies a defect candidate area, and is an integer value.

That is, the defect candidate region blob_j, which is a group for each blob, corresponds to the region occupied by the defects corresponding to each defect type i. Therefore, the geometric features of the defect candidate region blob_j directly correspond to the geometric features of the defect type i.

Next, the processor 12 calculates the geometric feature of the defect candidate region blob_j as a feature amount (step S15). The processor 12 not only obtains the geometrical features of the defect candidate region blob_j, but also the statistics of the pixel values of the captured image Icam in the defect candidate region blob_j and the statistics of the certainty Imap_i (p) in the defect candidate region blob_j. May be calculated as a feature amount.

The geometric features of the defect candidate region blob_j include at least one of the perimeter, area, and shape of the defect candidate region blob_j. The statistic of the pixel value of the captured image Icam in the defect candidate region blob_j includes at least one of the average, variance, maximum, minimum, distribution, etc. of the brightness of the captured image Icam in the defect candidate region blob_j. The statistic of the confidence Imap_i (p) in the defect candidate region blob_j includes at least one of the mean, variance, maximum, minimum, distribution, etc. of the confidence Imap_i (p) in the defect candidate region blob_j. These features are calculated for each defect type i.

ここで、“k”は特徴量の種類（幾何学的特徴、各種の統計量）を識別するインデックスであり、整数値である。Feat_kは、種類kの特徴量を計算する関数である。 The feature amount feat_kj in the defect candidate region blob_j to which the pixel p belongs is given as in Eq. (2).

Here, "k" is an index that identifies the type of feature quantity (geometric feature, various statistics), and is an integer value. Feat_k is a function that calculates the features of type k.

Next, the processor 12 indicates the defect type and the score for each defect type (for example, the rank of the severity of the defect or the severity of the defect from the distribution of the feature amount feat_kj calculated in step S15) by a known method. (Score) is determined (step S17). Specifically, the processor 12 evaluates defects in the captured image Icam from the distribution of the feature amount feat_kj by using a discriminator such as an IF-THEN rule or a support vector machine (SVM).

The processor 12 may end the process by treating it as "all the found defect candidates are defects" when the geometric feature corresponding to the defect candidate region blob_j is obtained as information.

Further, the processor 12 confirms the content of the calculated distribution of the feature amount feat_kj in step S17, and depending on the content, the processor 12 does not have to perform a process of determining the score for each defect type, as appropriate. The operator may be asked to confirm the calculated distribution of the feature amount feat_kj. By doing so, when the characteristics of the defect are known in advance (for example, when a defect of this size always occurs at this position, but it is known that it is a specific defect type), etc. It is possible to directly determine the correct defect type via an operator or the like without having to determine the score.

Further, the processor 12 does not process the distribution of the feature amount feat_kj calculated in step S15 in the surface property inspection device 100, but transmits the distribution to another device connected to the surface property inspection device 100, and the other device In, it is also possible to perform various processes related to the inspection by using the calculated distribution of the feature amount feat_kj. For various processes in the other apparatus, known processes related to inspection can be used, and a discriminator such as an IF-THEN rule or a support vector machine (SVM) can also be used. By doing so, the degree of freedom in device arrangement and combination can be increased.

As described above, according to the first embodiment, the brightness of the vicinity region 61 centered on the pixel 60 of interest in the captured image Icam is used by using the trained model 141 by deep learning without using the conventional threshold processing. From the pattern, the certainty of each defect type in the pixel 60 of interest is calculated to generate a certainty map image, and the defect candidate region is extracted. As a result, it is possible to accurately detect thin defects that were difficult to detect only by threshold processing and irregular defects that were difficult to detect due to the distribution in the feature space. In addition, by calculating the certainty of each defect type from the captured image Icam on a pixel-by-pixel basis, the knowledge of defect determination such that the defect type is divided according to the position where the defect occurs can be directly applied to the defect type determination in the determination logic (step S17). It can be reflected.

Further, according to the first embodiment of the present invention, the defect type is simply applied to the defect itself reflected in the captured image obtained by capturing the surface of the object by applying deep learning using the trained model. Rather than determining what the product is, whether it is a non-defective product or a defective product, deep learning using a trained model is applied to the captured image to calculate the certainty for each defect type. A certainty map image with the certainty as a pixel value is created, and the feature amount is obtained for the "image" of the certainty appearing in the certainty map image.

Therefore, according to the first embodiment of the present invention, by applying deep learning using the trained model, defects that appear only very thinly or irregular defects in the captured image can be detected with high accuracy. Can be done.

<Second Embodiment>
Next, the second embodiment will be described. The hardware configuration of the surface property inspection device according to the second embodiment is the same as that of the surface property inspection device 100 shown in FIG. In the second embodiment below, only the differences from the first embodiment will be described.

The processor 12 of the second embodiment executes the surface texture inspection method (FIG. 7) according to the program stored in the storage unit 14. Specifically, when the surface texture inspection method is executed, the processor 12 not only functions as a deep learning determination processing unit, a labeling unit, a feature amount calculation unit, and a determination unit of the first embodiment, but also with respect to the captured image. A threshold processing unit that generates a binary image by performing threshold processing, a second labeling unit that extracts a second defect candidate region by labeling each pixel of the binary image based on the pixel value, and a second. It functions as a second feature amount calculation unit that calculates at least one of the geometric feature of the defect candidate region and the statistic of the pixel value of the captured image in the second defect candidate region as the second feature amount.

Next, the surface texture inspection method of the second embodiment will be described with reference to FIG. 7. The surface texture inspection method shown in FIG. 7 is realized by combining the flow of the first embodiment (FIG. 3) and the flow using the conventional threshold processing (FIG. 1).

The processor 12 executes the following steps S21, S23 and S25 (corresponding to steps S1, S3 and S5 of FIG. 1, respectively) in parallel with steps S11, S13 and S15 of the first embodiment.

That is, the processor 12 generates a binary image by performing threshold processing on the captured image Icam (step S21). In step S21, in the captured image Icam, 1 is assigned to a pixel having a brightness value equal to or higher than the threshold value, and 0 is assigned to a pixel having a brightness value smaller than the threshold value to generate a binary image having a brightness value of 0 and 1. ..

Next, the processor 12 extracts the second defect candidate region by labeling the binary image generated in step S21 based on the pixel value of each pixel (step S23). In step S23, pixels having a brightness value of 1 are connected in a binary image and recognized as a blob, and the blob is extracted as a second defect candidate region.

That is, the second defect candidate region, which is a group of each blob, corresponds to the region occupied by the defects corresponding to the region. Therefore, the geometric feature of the second defect candidate region directly corresponds to the geometric feature of the defect corresponding to the region.

Next, the processor 12 calculates the geometric feature of the second defect candidate region as the second feature amount (step S25). The processor 12 may calculate not only the geometric feature of the second defect candidate region but also the statistic of the pixel value of the captured image Icam in the second defect candidate region as the second feature amount. The geometric feature of the second defect candidate region includes at least one of the perimeter, area, and shape of the second defect candidate region. The statistic of the pixel value of the captured image Icam in the second defect candidate region includes at least one of the average, variance, maximum, minimum, distribution, and the like of the brightness of the captured image Icam in the second defect candidate region.

The type k of the feature amount feat_kj calculated in step S15 and the type of the second feature amount calculated in step S25 are the same, but in order to distinguish between the two, the processor 12 uses the feature amount feat_kj calculated in step S15. A flag indicating that it is derived from deep learning is set for any of them, and a flag indicating that it is derived from the conventional threshold processing is set for the second feature amount calculated in step S25. This makes it possible to distinguish between the feature amount feat_kj and the second feature amount in step S17, which will be described later.

Next, the processor 12 uses the feature amount feat_kj calculated in step S15 and the second feature amount calculated in step S25 to perform the defect type and the defect type of the defect reflected in the captured image Icam by the same method as in the first embodiment. The score for each is determined (step S17).

The processor 12 may end the process by treating it as "all the found defect candidates are defects" when the geometric feature corresponding to the defect candidate region blob_j is obtained as information.

Further, the processor 12 confirms the calculated distribution of the feature amount feat_kj and the contents of the second feature amount, and depending on the contents, it is not necessary for the processor 12 to perform a process of determining the score for each defect type, as appropriate. , The operator may be asked to confirm the calculated distribution of the feature amount feat_kj and the second feature amount. By doing so, when the characteristics of the defect are known in advance (for example, when a defect of this size always occurs at this position, but it is known that it is a specific defect type), etc. It is possible to directly determine the correct defect type via an operator or the like without having to determine the score.

Further, the processor 12 does not process the calculated distribution of the feature amount feat_kj and the second feature amount in the surface property inspection device 100, but transmits the calculated feature amount feat_kj to another device connected to the surface property inspection device 100. It is also possible to perform various processes related to the inspection by using the calculated distribution of the feature amount feat_kj and the second feature amount in the device of. For various processes in the other apparatus, known processes related to inspection can be used, and a discriminator such as an IF-THEN rule or a support vector machine (SVM) can also be used. By doing so, the degree of freedom in device arrangement and combination can be increased.

As described above, according to the second embodiment of the present invention, deep learning using the trained model is simply applied to the defects themselves reflected in the captured image obtained by capturing the surface of the object. Rather than determining what the defect type is and whether it is a non-defective product or a defective product, deep learning using a trained model is applied to the captured image to determine the degree of certainty for each defect type. A certainty map image is created by calculating and using the certainty as a pixel value, and a feature amount is obtained for the "image" of the certainty appearing in the certainty map image.

Therefore, according to the second embodiment of the present invention, by combining the flow of the first embodiment (FIG. 3) and the conventional flow using the threshold processing (FIG. 1), it is necessary for model learning of deep learning. It is possible to eliminate the need to collect a large amount of teacher data for all defect types, including defect types that can be determined and detected by a conventional flow using threshold processing. As a result, it is possible to reduce the load of surface texture inspection and improve the accuracy of defect type determination and detection.

<Third Embodiment>
Next, the third embodiment will be described. The hardware configuration of the surface property inspection device according to the third embodiment is the same as that of the surface property inspection device 100 shown in FIG. In the following third embodiment, only the differences from the first and second embodiments will be described.

The processor 12 of the third embodiment executes the surface texture inspection method (FIG. 8) according to the program stored in the storage unit 14. Specifically, when the surface texture inspection method is executed, the processor 12 has functions as a deep learning determination processing unit, a labeling unit, a feature amount calculation unit, and a determination unit of the first embodiment, and also for an captured image. A first threshold processing unit that generates a first binary image by performing the first threshold processing, and a second binary image is generated for each defect type by performing the second threshold processing on the certainty map image. It further functions as an integrated unit that generates an integrated image by integrating the second threshold processing unit and the first binary image and the second binary image.

以下、式（３）のＩＤ番号１をｉｄ（０）と表記する。 Next, the surface texture inspection method of the third embodiment will be described with reference to FIG.
First, the processor 12 generates a first binary image by performing a first threshold processing on the captured image Icam (step S31). The first threshold processing in step S31 is substantially the same as the threshold processing in the second embodiment (step S21 in FIG. 7), and as shown in the equation (3), the brightness is relative to the pixel p of the captured image Icam. If the value Icam (p) is equal to or higher than the threshold value T0, the ID number 1 is assigned, and if the brightness value Icam (p) is smaller than the threshold value T0, 0 is assigned, so that the brightness value Ibin_0 (p) consists of 0 and 1. 1 Generate a binary image Ibin_0.

Hereinafter, the ID number 1 of the formula (3) is referred to as id (0).

Further, the processor 12 generates a certainty map image Imap_i from the captured image Icam for each defect type i (step S11), and the processor 12 generates the certainty map image Imap_i with respect to the generated certainty map image Imap_i, as in the first and second embodiments. By performing the second threshold processing, a second binary image is generated for each defect type (step S32).

ここで、閾値Ｔｉは欠陥種ｉ毎に設定されている。また、ＩＤ番号ｉｄ（ｉ）は、欠陥種ｉを識別するため、欠陥種ｉ別に異なる値が与えられる。例えば、ｉｄ（１）＝２、ｉｄ（２）＝３、…、ｉｄ（Ｎ）＝Ｎ＋１のように、ｉｄ（ｉ）には２以上の整数が与えられる。式（４）のＩＤ番号ｉｄ（ｉ）を２以上の値としているのは、ステップＳ３１の第１閾値処理から得られたＩＤ番号ｉｄ（０）＝１と区別するためである。このように、ステップＳ３２では、欠陥種の異なるＮ個の第２二値画像Ibin_i（ｉ＝１、２、…、Ｎ）が生成される。 In step S32, as shown in the equation (4), if the certainty Imap_i (p) is equal to or higher than the threshold Ti, the ID number id (i) is assigned to the pixel p of the certainty map image Imap_i, and the certainty Imap_i ( If p) is smaller than the threshold value Ti, 0 is assigned to generate a second binary image Ibin_i in which the luminance value Ibin_i (p) is 0 and the ID number id (i).

Here, the threshold value Ti is set for each defect type i. Further, the ID number id (i) is given a different value for each defect type i in order to identify the defect type i. For example, id (1) = 2, id (2) = 3, ..., id (N) = N + 1, and so on, id (i) is given an integer of 2 or more. The reason why the ID number id (i) in the formula (4) is set to a value of 2 or more is to distinguish it from the ID number id (0) = 1 obtained from the first threshold processing in step S31. As described above, in step S32, N binary images Ibin_i (i = 1, 2, ..., N) having different defect types are generated.

式（５）に示すように、統合画像Ibinは、画素ｐ毎に確信度Imap_i（ｐ）が最大となる輝度値Ibin_i(p)（ｉ＝０、１、…Ｎ）を与える画像である。ここで、ｉ＝０での確信度、すなわち、確信度Imap_0（ｐ）は、式（３）の輝度値Ibin_0(p)に付与した確信度であり、第１閾値処理（ステップＳ３１）による欠陥検出を優先する場合には相対的に高い値（例えば、規定値よりも大きい値）に予め設定され、深層学習による欠陥検出を優先する場合には相対的に低い値（例えば、規定値よりも小さい値）に予め設定されている。 Next, the processor 12 integrates the first binary image Ibin_0 and the second binary image Ibin_i to generate the integrated image Ibin (step S33). The brightness value Ibin (p) of the integrated image Ibin at pixel p is given as in Eq. (5).

As shown in the equation (5), the integrated image Ibin is an image that gives a luminance value Ibin_i (p) (i = 0, 1, ... N) that maximizes the certainty Imap_i (p) for each pixel p. Here, the certainty at i = 0, that is, the certainty Imap_0 (p) is the certainty given to the luminance value Ibin_0 (p) of the equation (3), and is a defect due to the first threshold processing (step S31). When priority is given to detection, a relatively high value (for example, a value larger than the specified value) is preset, and when priority is given to defect detection by deep learning, a relatively low value (for example, a value larger than the specified value) is set. It is preset to (small value).

Next, the processor 12 extracts a defect candidate region by labeling each pixel of the integrated image Ibin (step S35). In step S35, in the integrated image Ibin, the defect candidate region blob_j is extracted by concatenating the pixels giving the same luminance value Ibin (p) (same ID number id (i)) and regarding them as one blob.

Next, the processor 12 calculates for each defect type i using the geometric feature of the defect candidate region blob_j extracted in step S35 as a feature quantity, as in the first embodiment (step S15). The processor 12 not only obtains the geometrical features of the defect candidate region blob_j, but also the statistics of the pixel values of the captured image Icam in the defect candidate region blob_j and the statistics of the certainty Imap_i (p) in the defect candidate region blob_j. May be calculated for each defect type i as a feature amount (see equation (2)).

Next, the processor 12 determines the defect type of the defect reflected in the captured image Icam and the score for each defect type from the distribution of the feature amount calculated in step S15 by the same method as in the first embodiment (step). S17).

The processor 12 may end the process by treating it as "all the found defect candidates are defects" when the geometric feature corresponding to the defect candidate region blob_j is obtained as information.

Further, the processor 12 confirms the content of the calculated feature amount distribution, and depending on the content, the processor 12 does not have to perform a process of determining the score for each defect type, and the calculated feature amount is appropriately calculated. The operator may be asked to confirm the distribution of. By doing so, when the characteristics of the defect are known in advance (for example, when a defect of this size always occurs at this position, but it is known that it is a specific defect type), etc. It is possible to directly determine the correct defect type via an operator or the like without having to determine the score.

Further, the processor 12 does not process the calculated distribution of the feature amount in the surface texture inspection device 100, but transmits it to another device connected to the surface property inspection device 100, and the calculation is performed in the other device. It is also possible to perform various processes related to the inspection by using the distribution of the feature amount feat_kj. For various processes in the other apparatus, known processes related to inspection can be used, and a discriminator such as an IF-THEN rule or a support vector machine (SVM) can also be used. By doing so, the degree of freedom in device arrangement and combination can be increased.

As described above, according to the third embodiment of the present invention, deep learning using the trained model is simply applied to the defects themselves reflected in the captured image obtained by capturing the surface of the object. Rather than determining what the defect type is and whether it is a non-defective product or a defective product, deep learning using a trained model is applied to the captured image to determine the degree of certainty for each defect type. A certainty map image is created by calculating and using the certainty as a pixel value, and a feature amount is obtained for the "image" of the certainty appearing in the certainty map image.

Further, in the third embodiment of the present invention, the first binary image Ibin_0 obtained by binarizing the captured image Icam and the second binary image Ibin_i obtained by binarizing the certainty map image Imap_i for each defect type i are obtained. Generate and generate integrated image Ibin by integrating the first binary image Ibin_0 and the second binary image Ibin_i so as to give the brightness value that maximizes the certainty for each pixel, and set the integrated image Ibin to the brightness value. Defect candidate area blob_j is extracted by labeling based on. As a result, it is necessary to collect a large amount of teacher data required for deep learning model learning for all defect types, including defect types that can be determined and detected by the conventional flow using threshold processing. Can be eliminated. Therefore, it is possible to reduce the load of the surface property inspection and to improve the accuracy of defect type determination and detection. In addition, two binary images, a first binary image and a second binary image, are combined into one integrated image, and the integrated image is uniformly labeled and the feature amount is calculated. The load can be reduced.

Further, according to the third embodiment of the present invention, by applying deep learning using the trained model, defects that appear only very thinly or irregular defects in the captured image can be detected with high accuracy. Can be done.

Furthermore, according to the third embodiment of the present invention, the features are the same as those handled in the conventional flow using the threshold processing shown in FIG. 1 (so that deep learning using the trained model is not applied). Since the feature amount can be obtained from the certainty, it is possible to divert the technique used when dealing with the feature amount in the conventional flow using the threshold processing. Therefore, as compared with the conventional flow, it is possible to improve the detection accuracy only by adding a simple configuration.

The description contents in the first to third embodiments can be appropriately changed without departing from the spirit of the present invention.
For example, in the surface texture inspection process in each of the above-described embodiments, the certainty of the defect type is calculated directly from the captured image obtained by capturing the surface of the object, but from the captured image obtained by capturing the surface of the object. , Once the defect type is determined, the candidates for the probable defect type are narrowed down, and then the certainty degree for each such defect type may be calculated.

Further, in the surface texture inspection process in each of the above-described embodiments, the processor 12 calculates the certainty of each defect type from the captured image Icam on a pixel-by-pixel basis, but the captured image Icam is set to a specified region (for example, 2). It may be divided into (pixel regions such as × 2, 3 × 3), and the certainty of each defect type may be calculated for each divided region. Further, the adjacent divided regions may partially overlap.

Further, in each of the above-described embodiments, the captured image Icam is a gray image, but a color image may be used. Further, the captured image is not limited to the brightness information on the surface of the object, but may be a depth image in which shape information such as unevenness on the surface of the object is imaged, and the brightness information and the shape information are multiplexed. It may be an image.

11 Imaging device 12 Processor 13 Display unit 14 Storage unit 141 Learned model 60 Attention pixel 61 Near area 100 Surface property inspection device

Claims (8)

Using a trained model by deep learning, the certainty of each defect type is calculated from the captured image of the surface of the object in pixel units or region units, and a certainty map image using the certainty as the pixel value is obtained. A deep learning determination processing unit generated for each defect type,
A labeling unit that performs labeling and extracts defect candidate regions based on the pixel values of the certainty map image for each defect type.
Calculated as a feature quantity at least one of the geometric feature of the defect candidate region, the statistic of the pixel value of the captured image in the defect candidate region, and the statistic of the certainty in the defect candidate region. Feature amount calculation unit and
A surface property inspection device. The surface property inspection apparatus according to claim 1, further comprising a determination unit for determining a defect type on the surface of the object and a score for each defect type from the feature amount. The geometric feature of the defect candidate region includes at least one of the peripheral length, area, and shape of the defect candidate region.
The statistic of the pixel values of the captured image in the defect candidate region includes at least one of the average, variance, maximum, minimum, and distribution of the pixel values of the captured image in the defect candidate region.
The surface texture according to claim 1 or 2, wherein the confidence statistic in the defect candidate region comprises at least one of the mean, variance, maximum, minimum, and distribution of the confidence in the defect candidate region. Inspection device. A threshold processing unit that generates a binary image by performing threshold processing on the captured image,
A second labeling unit that performs labeling and extracts a second defect candidate region based on the pixel values of the binary image, and
Second feature amount calculation in which at least one of the geometric feature of the second defect candidate region and the statistic of the pixel value of the captured image in the second defect candidate region is calculated as the second feature amount. Department and
With
From the feature amount calculated by the feature amount calculation unit and the second feature amount calculated by the second feature amount calculation unit, the determination unit determines the defect type on the surface of the object and each of the defect types. The surface property inspection device according to claim 2, which determines a score. The geometric feature of the second defect candidate region includes at least one of the peripheral length, area, and shape of the second defect candidate region.
The statistic of the pixel value of the captured image in the second defect candidate region includes at least one of the average, variance, maximum, minimum, and distribution of the pixel value of the captured image in the second defect candidate region. The surface property inspection apparatus according to claim 4. A first threshold value processing unit that generates a first binary image by performing a first threshold value process on the captured image, and a first threshold value processing unit.
A second threshold processing unit that generates a binary image for each defect type by performing a second threshold processing on the certainty map image generated for each defect type.
An integration unit that integrates the first binary image and the second binary image to generate an integrated image,
With
The surface property inspection apparatus according to any one of claims 1 to 5, wherein the labeling unit performs labeling based on the pixel values of the integrated image and extracts the defect candidate region. The first threshold processing unit imparts certainty of the defect type to each pixel or each region of the first binary image.
The surface property inspection apparatus according to claim 6, wherein the integrating unit integrates the first binary image and the second binary image so that the certainty of the defect type is maximized for each pixel or each region. .. Using a trained model by deep learning, the certainty of each defect type is calculated from the captured image of the surface of the object in pixel units or region units, and a certainty map image using the certainty as the pixel value is obtained. The deep learning determination processing step generated for each defect type and
A labeling step of performing labeling and extracting a defect candidate region based on the pixel value of the certainty map image for each defect type, and
Calculated as a feature quantity at least one of the geometric feature of the defect candidate region, the statistic of the pixel value of the captured image in the defect candidate region, and the statistic of the certainty in the defect candidate region. Feature calculation steps to be performed and
A method for inspecting surface properties.

Images