WO2020121594A1

WO2020121594A1 - Surface characteristics inspection device and machine learning device for surface characteristics inspection

Info

Publication number: WO2020121594A1
Application number: PCT/JP2019/031698
Authority: WO
Inventors: 長岡　英一
Original assignee: 株式会社堀場製作所
Priority date: 2018-12-14
Filing date: 2019-08-09
Publication date: 2020-06-18
Also published as: JPWO2020121594A1

Abstract

The present invention accurately detects the surface characteristics of a specimen. The present invention comprises a light emission part 2 that emits light at a specimen W, an imaging part 3 that captures an image of reflected light from the specimen W, an image processing part 4 that performs a nonlinear transformation on the image captured by the imaging part 3 and thereby generates a transformed image, and a surface characteristics calculation part 5 that uses the transformed image to calculate the surface characteristics of the specimen.

Description

Surface property inspection device and machine learning device for surface property inspection

The present invention relates to a surface property inspection device that detects reflected light from a test object and calculates the surface property of the test object, and a machine learning device used for this surface property inspection.

Conventionally, as shown in Patent Document 1, from a captured image obtained by irradiating a test object with light and imaging reflected light from the test object, the presence or absence of surface material or surface irregularity is identified. A surface type identification device has been considered.

This surface type identification device calculates a characteristic amount that quantifies the degree of change in the pixel value of the captured image in the spatial direction and compares it with pre-registered dictionary data to determine the presence or absence of surface material or surface irregularities. To identify. Here, the feature amount is, for example, a distribution state of pixel values with respect to a relative angle (relative angle) ψ where the specular reflection direction is zero.

However, the surface type identification device described above is a measurement method that assumes a test object that does not have directivity in surface characteristics, and since the directionality is not considered at all in the method of calculating the relative angle, for example, streak It is difficult to accurately measure directional surface characteristics such as unevenness.

JP, 2002-174595, A

Therefore, the present invention has been made to solve the above problems, and its main object is to accurately detect the surface characteristics of a test object.

That is, the surface property inspection apparatus according to the present invention, a light irradiation unit for irradiating a test object with light, an imaging unit for imaging reflected light from the test object, and a non-linear conversion of the imaged image of the imaging unit. An image processing unit that generates a converted image and a surface characteristic calculation unit that calculates the surface characteristic of the test object using the converted image are provided. Here, as the non-linear conversion, logarithmic conversion, exponential conversion, gamma conversion and the like can be considered.

In such a case, the captured image of reflected light is subjected to non-linear conversion and the converted image obtained by this is used to calculate the surface characteristics of the test object. Can be easily captured, and the surface characteristics of the test object can be accurately detected.

If all the data of the converted image is used, the amount of data is large, so processing time may be required. Therefore, it is preferable that the surface characteristic calculation unit calculates the surface characteristic of the test object by using the luminance distribution along the predetermined direction in the converted image. With this configuration, since processing is performed using part of the data of the converted image, the processing time can be shortened.

When using part of the data of the converted image, it is desirable to use the part where the boundary area between the specular reflection component and the diffuse reflection component in the converted image is large. Therefore, it is desirable that the surface characteristic calculation unit calculates the surface characteristic of the test object by using the luminance distribution along the direction in which the luminance in the converted image extends.

In order to enable the surface property inspection device to automatically and accurately determine the surface property, it is composed of a converted image obtained by performing non-linear conversion of the imaged image of the imaging unit and a surface property label corresponding to the converted image. It is preferable that the apparatus further includes a machine learning unit that generates a learning model by machine learning using a learning data set, and the surface characteristic calculation unit calculates the surface characteristic of the test object based on the learning model.

Further, the machine learning device for surface characteristic inspection according to the present invention includes a captured image acquisition unit that irradiates a test object with light and acquires a captured image obtained by capturing the reflected light from the test object. , A learning model is generated by machine learning using a transformed image generation unit that non-linearly transforms the acquired captured image to generate a transformed image, and a learning data set including the transformed image and a surface characteristic label corresponding to the transformed image. And a machine learning unit for performing the same.

In such a case, the captured image of reflected light is subjected to non-linear conversion, and machine learning is performed using the converted image obtained by this, so a learning model that captures changes in the luminance distribution of reflected light is used. It can be generated, and the surface characteristics of the test object can be detected with high accuracy.

In order to reduce the processing time of machine learning, it is preferable that the machine learning unit uses a brightness distribution along a predetermined direction in the converted image as the converted image of the learning data set.

In order to perform machine learning using the information in which the change in the brightness distribution of the reflected light in the converted image is largely expressed, the machine learning unit uses the direction in which the brightness in the converted image extends as the converted image of the learning data set. It is desirable to use a luminance distribution along.

According to the present invention described above, the surface characteristics of the test object can be detected with high accuracy.

It is the whole schematic diagram of the surface property inspection device concerning one embodiment of the present invention. It is a figure which shows (a) picked-up image and (b) relative brightness|luminance distribution of the same embodiment. It is a figure which shows the (a) conversion image of the same embodiment, the figure which shows the (b) relative brightness|luminance distribution, and the figure which shows the symmetry axis in the (c) conversion image. It is a figure which shows the brightness|luminance distribution of the direction along the symmetry axis in the captured image and conversion image of the same embodiment. It is a schematic diagram which shows the learning data set of the same embodiment.

100... Surface characteristic inspection device W... Object 2... Light irradiation unit 3... Imaging unit 42... Image processing unit 43... Surface characteristic calculation unit 44... Machine learning unit

A surface property inspection apparatus according to an embodiment of the present invention will be described below with reference to the drawings.

<Overall structure>
The surface property inspection apparatus 100 of the present embodiment inspects the surface property of the test object W in a non-contact manner. Here, the surface characteristics serve as an index of the appearance and surface state of the test object W, and are, for example, optical characteristics such as surface reflectance and luminance distribution, or mechanical characteristics such as surface roughness and waviness. It includes a scale indicating the quality of the processed state and an index for determining the quality of a special processed state such as hairline processing.

Specifically, as shown in FIG. 1, the surface property inspection apparatus 100 detects the light irradiation unit 2 that irradiates the test object W with light and the reflected light reflected by the test object W to detect the test object W. The image pickup unit 3 for picking up an image of the surface and the arithmetic unit 4 for processing the picked-up image picked up by the image pickup unit 3 to calculate the surface characteristics are provided.

The light irradiation unit 2 emits infrared light, for example, and emits light toward the imaging range of the imaging unit 3. It is desirable that the light irradiation unit 2 be configured so that the amount of light becomes uniform in the range captured by the imaging unit 3. Further, in order to eliminate the influence of ambient light or the like, it is preferable to provide an enclosure or the like that covers the object W including the light irradiation section 2 and the imaging section 3.

The image pickup unit 3 picks up at least specular reflection light (specular reflection light and diffuse reflection light in the present embodiment) reflected by the test object W, and has an imaging device having a plurality of pixels arranged two-dimensionally. 31 and a captured image generation unit 32 that generates a captured image 50 that is a luminance image of the test object W based on the accumulated charge amount of each pixel in the image sensor 31. The image capturing unit 3 may capture at least specularly reflected light and its surroundings, specifically, at least part of specularly reflected light and diffusely reflected light around it. FIG. 2A shows an example of a captured image 50 obtained by capturing an image of a flat plate-shaped test object W having streaky irregularities. Further, FIG. 2B is a relative brightness distribution diagram in which the two-dimensional array of the image pickup devices 31 is set as the x-axis and the y-axis and the brightness of each pixel in the captured image 50 is displayed three-dimensionally as the height. A sharp peaked specular reflection component 51 can be observed at the center of the xy plane. The brightness values of the other pixels are relatively displayed using the brightness value of the specular reflection component 51 as a reference (for example, 100%). A diffuse reflection component 52 exists around the regular reflection component 51, but since the luminance value is smaller by three digits or more, the height can be regarded as zero. In the boundary area 53 between the specular reflection component 51 and the diffuse reflection component 52, the luminance value is smaller than that of the specular reflection component 51 by one digit or more. Therefore, the change in the boundary region 53 that transits from the regular reflection component 51 to the diffuse reflection component 52 is not noticeable.

The arithmetic unit 4 processes the picked-up image 50 from the image pickup unit 3 to calculate the surface characteristics of the test object W. Specifically, the arithmetic unit 4 is a computer having a structure such as a CPU, a memory, an input/output interface, an AD converter, a display, an input means, and the like, and the CPU and the CPU based on the surface characteristic inspection program stored in the memory. By operating the peripheral devices, as shown in FIG. 1, at least the functions of the reception unit 41, the image processing unit 42, the surface characteristic calculation unit 43, the machine learning unit 44, the data storage unit 45, and the like are exhibited.

The reception unit 41 receives the captured image data indicating the brightness value of each pixel of the captured image 50 from the captured image generation unit 32 of the imaging unit 3.

The image processing unit 42 generates converted image data indicating a converted image 54 obtained by nonlinearly converting the captured image data of each pixel received by the receiving unit 41. Here, as the non-linear conversion, logarithmic conversion is used because the sense amount for human brightness is proportional to the logarithm of brightness according to the Weber-Fechner law. Note that exponential conversion, gamma conversion, or the like may be used as the other non-linear conversion. FIG. 3A shows a converted image 54 obtained by nonlinearly converting the captured image 50. The converted image 54 is an image in which the pixel value is the value of the converted image data that is the conversion result of the captured image data. Further, FIG. 3B is a relative luminance distribution diagram in which the value of the converted image data is displayed three-dimensionally as height. Values of other converted image data are relatively displayed based on the non-linear conversion result of the regular reflection component 51 (for example, 100%). The height of the diffuse reflection component 52 whose luminance value before conversion is three digits or less can still be regarded as zero (very small) even if nonlinear conversion is performed. However, the boundary region 53 is non-linearly converted to a value comparable to the specular reflection component 51 (a small value that cannot be ignored, for example, a value of about half). Therefore, it is possible to clearly observe the change in the boundary region 53 that transitions from the regular reflection component 51 to the diffuse reflection component 52, and it is possible to accurately grasp the characteristics of the boundary region.

In this way, since the captured image data of the captured image 50 is logarithmically converted by the image processing unit 42, the captured image 50 generated by the captured image generation unit 32 of the imaging unit 3 is an image having a dynamic range that can withstand logarithmic conversion. It is necessary to be. Specifically, it is desirable that the captured image generation unit 32 generate the captured image 50 by HDR (high dynamic range) composition. That is, a method of synthesizing a plurality of captured images obtained by changing the exposure time in consideration of the exposure time is used. For example, after exposure for a relatively long time (for example, exposure for 1 second) to acquire the first captured image, the exposure time is reduced to 1/100 to acquire the second captured image, and the first and second captured images are acquired. When images are combined, an image having a dynamic range that is two orders of magnitude wider can be obtained. At this time, the pixels for detecting the specular reflection component of the reflected light are in a so-called “overexposure” state (a state in which the amount of accumulated charge of the image sensor 31 is saturated) during long-time exposure, and thus the second short-time exposure is performed. Use the captured image of. Further, the pixel for detecting the diffuse reflection component is buried in the noise component during short-time shooting (because the accumulated charge amount of the image pickup device 31 is insufficient), so the first shot image which is long-time exposure is used. Then, since the exposure time of the first captured image is 100 times longer, the luminance value of the pixels of the first captured image is multiplied by 100 to be combined. In this way, the one obtained by selecting an appropriate luminance value for each pixel from the first and second photographed images may be used as the photographed image 50 again.

Looking at the converted image 54 obtained by nonlinearly converting the captured image 50 thus obtained, the white bright lines extending in the oblique direction can be clearly observed, as shown in FIG. 3B. Considering a straight line that passes through the apex of the specular reflection component 51 having the maximum pixel value of the converted image 54 and extends in the direction of the white bright line, it can be seen that the pixel values of the converted image 54 are distributed in line symmetry. This is a remarkable conversion result in the test object W having streaky irregularities.

When the object W having unevenness on the surface is irradiated with light from the light irradiating section 2, in addition to the regular reflection component 51 that is reflected in the macroscopic normal direction of the surface, microscopic images corresponding to the fine shape due to unevenness are obtained. There is a component that reflects in the direction corresponding to the specific inclination. The degree increases or decreases depending on the size of the unevenness, and only the specular reflection component 51 remains on the mirror surface having no unevenness.

In addition, in the test object W having streaky unevenness, the directions of the unevenness are aligned with the direction of the lines of the creases, so the microscopic inclination in the direction of the creases is small and the inclination in the direction orthogonal to the creases is large. Therefore, as shown in FIG. 3B, the white bright line extends linearly in a direction substantially orthogonal to the direction of the uneven lines.

That is, a white bright line extends in the direction of the symmetry axis 55 as shown in FIG. The brightness distribution of the converted image 54 in the direction along the symmetry axis 55 remarkably represents the degree of unevenness of the test object W and the degree of inclination.

Therefore, the image processing unit 42 detects the symmetry axis 55 in the conversion image 54 indicated by the conversion image data and extracts the one-dimensional luminance distribution data in the direction along the symmetry axis 55. This one-dimensional luminance distribution As shown in FIG.

Here, the image processing unit 42 detects the symmetry axis 55 by applying an edge detection technique such as Hough transform (Hough transform) known as a general image processing technique to the transformed image 54.

Further, as can be seen from the luminance distribution of FIG. 3B, in the object W having streaky irregularities, in addition to the symmetry axis 55, line symmetry also exists in the direction of the symmetry axis 56 shown in FIG. 3C. Becomes In such a case, the symmetry axis 55 may be determined as the main direction and the symmetry axis 56 may be determined as the sub direction based on the length of the white bright line of the converted image 54, and the one-dimensional luminance distribution data in each direction may be extracted. When the luminance distribution of the converted image 54 is close to point symmetry, the symmetry axis 55 is set in an arbitrary direction passing through the apex of the specular reflection component 51, and then the symmetry axis 56 is formed in the direction orthogonal to the symmetry axis 55. The brightness distribution data in three directions may be extracted by defining the axis 57 that is inclined by 45 degrees from the symmetry axes 55 and 56.

The surface characteristic calculation unit 43 calculates the surface characteristic of the test object W using the converted image 54 converted by the image processing unit 42.

Specifically, the surface characteristic calculation unit 43 calculates the surface characteristic of the test object W using the converted image data which is the brightness in the converted image 54 extracted by the image processing unit 42. In the present embodiment, the surface characteristic of the test object W is calculated using the one-dimensional luminance distribution data in the direction along the symmetry axis 55 in the converted image 54 obtained by the image processing unit 42.

However, the surface property inspection apparatus 100 of the present embodiment further includes the machine learning unit 44 that generates a learning model for surface property detection.

The machine learning unit 44 uses a learning data set including converted image data obtained by performing non-linear conversion (logarithmic conversion here) of the captured image 50 of the imaging unit 3 and a surface characteristic label corresponding to the converted image 54. A learning model is generated by machine learning.

Specifically, the machine learning unit 44 performs machine learning using a learning data set including one-dimensional luminance distribution data in the direction along the symmetry axis 55 in the converted image 54 and a surface characteristic label corresponding to the luminance distribution data. Generate a learning model. The generated learning model is stored in the data storage unit 45.

Here, as the machine learning algorithm by the machine learning unit 44, artificial neural network (ANN), support vector machine (SVM), decision tree, random forest, k-means clustering, self-organizing map, genetic algorithm, Bayesian network. , One or more combinations selected from deep learning methods and the like can be used.

Then, the surface characteristic calculation unit 43 uses the learning model generated by the machine learning unit 44 to convert the captured image data (converted image data obtained by performing non-linear conversion) of the captured image data of the inspection object W into the inspection object. Calculate the surface properties of W.

The captured image 50 accepted by the accepting unit 41, the converted image 54 obtained by the image processing unit 42, various data (for example, brightness distribution data) obtained from the converted image 54, and the surface characteristic obtained by the surface characteristic calculating unit 43 are as follows. , Output on the display, etc.

<Machine learning procedure>
Next, a data processing procedure in machine learning will be briefly described.

An image of the object W (surface to be inspected) whose surface characteristics are known is taken by the imaging unit 3. At this time, the operator inputs the surface characteristic label of the object W imaged by the input means (not shown). The surface characteristic label may be input by selecting from a database prepared in advance. Known surface characteristics include, for example, those obtained by subjecting the metal surface to various hairline treatments, those obtained by subjecting the metal surface to various spin treatments, and various other surface treatments.

The image processing unit 42 non-linearly transforms (logarithmically transforms) the captured image 50 obtained by the imaging unit 3 to generate a transformed image 54. Further, the image processing unit 42 generates the luminance distribution data in the direction along the symmetry axis 55 of the luminance distribution from the converted image 54.

A learning data set (see FIG. 5) including the brightness distribution data thus generated and the surface characteristic label corresponding to the brightness distribution data is input to the machine learning unit 44. Note that FIG. 5 shows an example in which a plurality of brightness distribution data are used for one surface characteristic label A to D. This learning data set was generated by the arithmetic unit 4 (specifically, the image processing unit 42) of the surface property inspection apparatus 100, but the previously prepared learning data set is an arithmetic unit of the surface property inspection apparatus 100. It may be configured such that by inputting to 4, the machine learning unit 44 is input. Alternatively, past surface characteristic inspection data may be stored in the data storage unit 45 as a learning data set and machine learning may be performed using this data.

<Effects of this embodiment>
According to the surface property inspection apparatus 100 of the present embodiment, the captured image obtained by capturing the reflected light is nonlinearly converted, and the surface property of the test object W is calculated using the converted image obtained by this conversion. In the image, the change in the boundary area between the specular reflection component and the diffuse reflection component can be made relatively large, and the surface characteristics of the test object W can be accurately detected.
For example, when a metal surface having streaky irregularities is imaged, only white bright spots, which are specular reflection components, can be confirmed in the captured image, but in the converted image obtained by nonlinear conversion, white bright lines that linearly extend from the specular reflection component The boundary area between the regular reflection component and the diffuse reflection component can be easily confirmed.

<Other modified embodiments>
The present invention is not limited to the above embodiment.

For example, although the surface characteristic inspection device having the machine learning function has been described in the above-described embodiment, the machine learning device for surface characteristic inspection that generates a learning model input to the surface characteristic inspection device also performs machine learning in the same manner as above. To do.

The method of detecting the axis of symmetry 55 is not limited to the edge detection technique such as Hough transform (Hough transform), and the following method may be used.
That is, in the converted image 54, the point at which the converted image data becomes maximum (the brightness becomes maximum) is detected. This point is the center of the specular reflection component 51. Considering a line segment having a predetermined length that is oriented in any direction with this point as the middle point, the converted image data on the line segment is integrated (the cumulative value of the converted image data is obtained in units of predetermined pixel intervals). It is sufficient to repeatedly calculate while changing the direction of the line segment and determine that the direction in which the integrated value (cumulative value) is maximum is the direction of the symmetry axis 55, and a known extreme value search algorithm such as the hill climbing method may be used. The length of the line segment is preferably about the same as the size of the converted image 54 in the vertical and horizontal directions. Further, instead of simple integration, weighting may be performed in proportion to the distance from the midpoint of the line segment. Further, if the square of the distance is used as the weight, it is equivalent to obtaining the variance assuming a one-dimensional Gaussian distribution on the line segment. That is, the direction in which the variance is maximized is searched for, and the direction of the symmetry axis 55 can be accurately detected.

Extending this idea to two dimensions, you may fit to the two-dimensional Gaussian distribution using the least squares method. That is, in the converted image 54, it is assumed that the distribution state of the converted image data can be approximated by a two-dimensional distribution function with respect to the x-axis and the y-axis on the plane of the image sensor 31, for example, a two-dimensional Gaussian distribution function. At this time, the contour lines of the two-dimensional Gaussian distribution function (lines connecting the same function values) are elliptic with the major axis in the direction of the symmetry axis 55 and the minor axis in the symmetry axis 56. The major axis and minor axis directions of this ellipse can be calculated from the coefficients of the two-dimensional Gaussian distribution function. Therefore, in the converted image 54, a set of x-axis and y-axis coordinate values and converted image data may be selected and the coefficient of the two-dimensional Gaussian distribution function may be determined by the least square method. Here, as the data set used for the least squares method, it is preferable to select only pixels whose converted image data is larger than a predetermined threshold value. This is because a pixel having a value smaller than the threshold value acts as noise and deteriorates the ellipse detection accuracy. Further, the two-dimensional distribution function is not limited to the two-dimensional Gaussian distribution function, and a function that can approximate the distribution state of the converted image data with high accuracy may be used.

In addition, various modifications and combinations of the embodiments may be made without departing from the spirit of the present invention.

According to the present invention, it is possible to accurately detect the surface characteristics of the test object.

Claims

A light irradiation unit that irradiates the test object with light,
An imaging unit that images reflected light from the test object,
An image processing unit that generates a converted image by performing non-linear conversion on the captured image of the imaging unit,
And a surface characteristic calculation unit that calculates the surface characteristic of the test object using the converted image.
The surface characteristic inspection apparatus according to claim 1, wherein the surface characteristic calculation unit calculates the surface characteristic of the test object by using a luminance distribution in a predetermined direction in the converted image.
3. The surface property inspection apparatus according to claim 1, wherein the surface property calculation unit is configured to calculate the surface property of the test object by using a brightness distribution in a direction in which brightness in the converted image extends.
A machine learning unit that generates a learning model by machine learning using a learning data set consisting of a conversion image obtained by performing non-linear conversion of the imaged image of the imaging unit and a surface characteristic label corresponding to the conversion image,
The surface characteristic inspection device according to claim 1, wherein the surface characteristic calculation unit calculates the surface characteristic of the test object based on the learning model.
A captured image acquisition unit that irradiates a test object with light and acquires a captured image obtained by capturing the reflected light from the test object,
A converted image generation unit that non-linearly converts the acquired captured image to generate a converted image;
A machine learning device for inspecting surface characteristics, comprising: a machine learning unit that generates a learning model by machine learning using a learning data set consisting of the converted image and a surface characteristic label corresponding to the converted image.
The machine learning device for surface characteristic inspection according to claim 5, wherein the machine learning unit uses, as the converted image of the learning data set, a luminance distribution in a predetermined direction in the converted image.
The machine learning device for surface characteristic inspection according to claim 5 or 6, wherein the machine learning unit uses, as the converted image of the learning data set, a brightness distribution in a direction in which brightness in the converted image extends.