WO2022044673A1

WO2022044673A1 - Image processing device, inspection system, and inspection method

Info

Publication number: WO2022044673A1
Application number: PCT/JP2021/027936
Authority: WO
Inventors: 和真島軒
Original assignee: マクセルフロンティア株式会社
Priority date: 2020-08-27
Filing date: 2021-07-28
Publication date: 2022-03-03
Also published as: JP2023153431A

Abstract

Provided are an image processing device, an inspection system, and an inspection method, with which are capable of reducing the work load of a user. An image processing device 18 processes image data for an inspection subject that has been photographed under illumination. An illumination condition searcher 26 variably sets an illumination condition ILC[i] for the illumination. An image interface 25 acquires image data IMD[i] for each illumination condition ILC[i]. An image analyzer 27 carries out image analysis on the image data IMD[i] for each illumination condition ILC[i]. A comparator 28 compares the image analysis result AR[i] from the image analyzer 27 for each illumination condition ILC[i] to select any one illumination condition ILCx among the plurality of illumination conditions ILC[i].

Description

Image processing equipment, inspection system and inspection method

The present invention relates to an image processing device, an inspection system, and an inspection method, and relates to, for example, a technique for performing inspection based on an image of an inspection object.

For example, Patent Document 1 discloses a lighting device including a projector, a translucent liquid crystal display, an image pickup device unit, and an image processing device, which enables high-definition effect lighting according to the shape and color of an irradiated object. .. Specifically, the light from the projector is irradiated to the irradiated body through the translucent liquid crystal display, and the image pickup apparatus unit images the irradiated body. The image processing device processes the video signal from the image pickup device unit and distinguishes the region of the irradiated body from the region other than the irradiated body. Then, the image processing device controls the translucent liquid crystal display so as to mask the irradiation to the region other than the irradiated body.

Further, Patent Document 2 discloses a lighting control device having a CCD sensor capable of forming an electrical image of a room and a control means for controlling the lighting in the room according to the measurement result of the CCD sensor.

Japanese Unexamined Patent Publication No. 08-271985 Japanese Patent Publication No. 2004-501496

For example, an inspection system including an imaging device (typically a camera) that captures an image of an inspection object and a lighting device that irradiates the inspection object with light to assist the imaging by the camera is known. The lighting device includes, for example, a halogen lamp light source, an LED light source, etc., and can adjust lighting conditions such as brightness and color (R, G, B, IR, etc.) so that an image of an inspection object can be clearly captured. It has become. However, the adjustment of lighting conditions is usually done manually by the user of the device. Therefore, there is a risk that the workload and working time of the user will increase when deriving the optimum value of the lighting condition.

The present invention has been made in view of the above, and one of the objects thereof is to provide an image processing device, an inspection system, and an inspection method capable of reducing a user's workload. ..

The above and other purposes and novel features of the invention will become apparent from the description and accompanying drawings herein.

In one embodiment of the present invention, an image processing device for processing image data of an inspection object captured in a illuminated state is provided, and the image processing device variably sets lighting conditions for lighting. By comparing the image interface that acquires the image data for each lighting condition, the image analyzer that analyzes the image data for each lighting condition, and the image analysis result for each lighting condition by the image analyzer. It may be configured to have a comparator that selects any one of the lighting conditions from a plurality of lighting conditions.

By simply explaining the effects obtained by the representative inventions disclosed in the present application, it is possible to reduce the workload of the user.

It is a schematic diagram which shows the structural example of the main part in the inspection system by Embodiment 1 of this invention. It is a block diagram which shows the structural example of the main part of the image processing apparatus in FIG. It is a flow diagram which shows an example of the processing content of the image processing apparatus of FIG. It is a schematic diagram explaining the operation example of the image analyzer and the comparator in FIG. 2 is a flow chart showing an example of processing contents of the image analyzer and the comparator in FIGS. 2 and 4. It is a supplementary view of FIG. 5A. It is a schematic diagram explaining the operation example of the image analyzer and the comparator in FIG. 2 in the image processing apparatus according to Embodiment 2 of this invention. FIG. 6 is a diagram illustrating an example of an edge extraction method using a sobel filter using an image analyzer. It is a schematic diagram which shows the structural example of the image analyzer in FIG. 2 in the image processing apparatus according to Embodiment 3 of this invention. It is a schematic diagram which shows the structural example of the main part in the inspection system according to Embodiment 4 of this invention. It is a schematic diagram which shows some structural examples in the inspection system according to Embodiment 5 of this invention. It is a schematic diagram which shows the structural example of the main part in the image processing apparatus according to Embodiment 6 of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, in all the drawings for explaining the embodiment, the same members are designated by the same reference numerals in principle, and the repeated description thereof will be omitted.

(Embodiment 1)
<< Outline of inspection system >>
FIG. 1 is a schematic view showing a configuration example of a main part in the inspection system according to the first embodiment of the present invention. The inspection system 10 shown in FIG. 1 includes an image pickup device 15, a lighting device 16, a lighting control device 17, and an image processing device 18. The lighting device 16 includes, for example, a halogen lamp light source, an LED (Light Emitting Diode) light source, and the like, and illuminates the inspection object 20a so that the image of the inspection object 20a can be clearly captured.

The image pickup device 15 includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like, and creates an image data IMD by capturing an image of an inspection object 20a in an illuminated state. The image processing device 18 acquires the image data IMD from the image pickup device 15 via the cable 19a and processes the image data IMD. As an example, the image processing apparatus 18 inspects the mounting state of the electronic component (IC) 20a mounted on the wiring board 20b (for example, the mounting position of the IC, the pitch and number of IC leads, etc.) by image processing. In this case, in the image data IMD, the wiring board 20b is the background, and the electronic component 20a is the inspection target.

Further, although the details will be described later, the image processing device 18 searches for the optimum lighting condition ILC by processing the corresponding image data IMD while variably setting the lighting condition ILC for the lighting device 16. The lighting control device 17 receives the lighting condition ILC from the image processing device 18 via the cable 19b, and transmits the lighting control signal ICS corresponding to the lighting condition ILC. The lighting device 16 receives the lighting control signal ICS from the lighting control device 17 via the cable 19c, generates light based on the lighting control signal ICS, and irradiates the inspection object 20a. The lighting control device 17 may be integrated with the image processing device 18 or the lighting device 16.

Here, in the general inspection system 10, the cable 19b is not provided, and the lighting condition ILC is manually set in the lighting control device 17 by the user. That is, for example, the user visually evaluates the sharpness of the corresponding image data IMD for each type of the inspection target 20a while appropriately changing the illumination condition ILC prior to the inspection of the inspection target 20a. Then, the user derives the illumination condition ILC having the highest sharpness. However, in such a method, the workload and working time of the user may increase.

For example, when the lighting device 16 uses a monochromatic light source and the light amount of 256 gradations can be set, there are 256 options as the lighting condition ILC. Further, when the lighting device 16 uses a ^three -color light source (R, G, B) and the light amount of 16 gradations can be set for each, there are 4096 (= 163) options as the lighting condition ILC. The user needs to visually compare the image data IMDs obtained for each lighting condition ILC option. Further, in addition to such a work load problem, a user's proficiency level is also required in order to obtain the optimum value of the lighting condition ILC. Therefore, it is beneficial to use the image processing apparatus 18 of the embodiment.

<< Details of image processing equipment >>
FIG. 2 is a block diagram showing a configuration example of a main part of the image processing apparatus in FIG. The image processing device 18 shown in FIG. 2 includes an image interface 25, a lighting condition searcher 26, an image analyzer 27, a comparator 28, and a memory 29. The lighting condition searcher 26 variably sets the lighting condition ILC [i] of the lighting device 16. The image interface 25 acquires the image data IMD [i] of the inspection object 20a imaged and created by the image pickup apparatus 15 for each illumination condition ILC [i], and uses the image data IMD [i] as the illumination condition ILC [i]. It is saved in the memory 29 in association with i].

The image analyzer 27 performs image analysis on the image data IMD [i] for each lighting condition ILC [i] stored in the memory 29, and sets the image analysis result AR [i] as the lighting condition ILC [i]. It is stored in the memory 29 in association with each other. The comparator 28 compares the image analysis result AR [i] for each lighting condition ILC [i] stored in the memory 29, and by comparing the image analysis result AR [i], any one of the plurality of lighting conditions ILC [i] is included in the lighting condition ILCx. Select. Then, the comparator 28 determines the selected lighting condition ILCx as the final lighting condition ILCx when inspecting the inspection object 20a.

The image processing device 18 of FIG. 2 has, for example, a single or a plurality of processors (CPU (Central Processing Unit) or GPU (Graphics Processing Unit)), a memory 29 (RAM (Random access memory)), and a non-volatile processor on the motherboard. It is configured in the form of implementing (sex memory). In this case, the image interface 25, the illumination condition searcher 26, the image analyzer 27, and the comparator 28 are implemented by program processing using such a processor.

However, the image processing device 18 is not limited to such an implementation form, and the form using FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or the like instead of the processor, or a PC (Personal Computer). Or the like may be used. Further, although not shown, the image processing apparatus 18 separately includes, for example, an image analyzer for performing image analysis associated with the inspection of the inspection object 20a and performing a quality determination or the like.

FIG. 3 is a flow chart showing an example of the processing content of the image processing apparatus of FIG. The flow of FIG. 3 is executed, for example, for each type of the inspection target 20a prior to the inspection of the inspection target 20a. In FIG. 3, first, the lighting condition searcher 26 sets the lighting condition ILC [i] to the initial value (step S101). Subsequently, the image interface 25 acquires the image data IMD [i] of the inspection object 20a imaged and created by the image pickup apparatus 15 under the illumination condition ILC [i] (step S102). Next, the image analyzer 27 executes image analysis on the acquired image data IMD [i] (step S103), associates the image analysis result AR [i] with the lighting condition ILC [i], and stores the memory. Save in 29 (step S104).

Subsequently, the lighting condition searcher 26 determines whether or not there is another lighting condition ILC [i] (step S105). When there is another lighting condition ILC [i], the lighting condition searcher 26 changes the lighting condition ILC [i] (step S106). After that, as in the case of steps S102 to S104, the image interface 25 acquires the image data IMD [i] (step S107), and the image analyzer 27 executes image analysis (step S108) to perform image analysis. The result AR [i] is saved in the memory 29 (step S109). Then, the image processing apparatus 18 repeatedly executes the processes of steps S106 to S109 until the other lighting condition ILC [i] disappears in step S105.

As a result, the image analysis result AR [i] can be obtained for the number of options of the lighting condition ILC [i]. For example, as described above, when there are 256 options, 256 image analysis results AR [i] are obtained, and when there are 4096 options, 4096 image analysis results AR [i] are obtained. can get. In the case of the specification that the lighting device 16 adjusts the amount of light in an analog manner, for example, the analog value may be quantized in a plurality of stages and then the same processing may be performed.

On the other hand, when it is determined in step S105 that there is no other lighting condition ILC [i], the comparator 28 compares the image analysis results AR [i] for each of the plurality of lighting conditions ILC [i] (step S110). .. Then, the comparator 28 determines the illumination condition ILC [i] at which the image analysis result AR [i] is the best in the final illumination condition ILCx (step S111). After that, the inspection object 20a is inspected under the final lighting condition ILCx.

The flow of the image processing device 18 is not limited to the flow of FIG. 3, and can be changed as appropriate. For example, in the loop processing of steps S105 to S109, the previous image analysis result (AR [i-1]) and the current image analysis result (AR [i]) are compared, and whichever is better. The flow may be such that only the analysis result (and the corresponding lighting condition) is left in the memory 29.

<< Details of image analyzer >>
FIG. 4 is a schematic diagram illustrating an operation example of the image analyzer and the comparator in FIG. In FIG. 4, the image analyzer 27 calculates the contrast for the image data 31a (IMD) and 31b (IMD) for each of the illumination conditions ILC [1] and ILC [2]. Specifically, the image analyzer 27 calculates the contrast between the density of the inspection object 20a and the density of the background 20b for each image data 31a (IMD) and 31b (IMD). In this example, for convenience of explanation, the inspection object 20a is assumed to have a circular shape. Then, the image analyzer 27 outputs the contrast calculated for each image data 31a (IMD) and 31b (IMD) as the image analysis results AR [1] and AR [2].

The comparator 28 compares each image analysis result AR [1] and AR [2] (that is, contrast) to obtain one of a plurality of lighting conditions ILC [1] and ILC [2]. Select. In the example of FIG. 4, the image data 31b under the illumination condition ILC [2] has a larger contrast than the image data 31a under the illumination condition ILC [1]. In this case, the comparator 28 determines that the illumination condition ILC [2] having a large contrast is better than the illumination condition ILC [1], and selects the illumination condition ILC [2].

FIG. 5A is a flow chart showing an example of the processing contents of the image analyzer and the comparator in FIGS. 2 and 4. FIG. 5B is a supplementary view of FIG. 5A. As one of the methods for calculating the contrast described in FIG. 4, a method using a discriminant analysis method can be mentioned. The discriminant analysis method, also called binarization of Otsu, is a method of determining a threshold value TH when binarizing (black and white) all pixels in an image, as shown in FIG. 5B.

FIG. 5B shows a histogram showing the relationship between the pixel value (density) and the number of pixels for each pixel value. In order to binarize an image having such a histogram, a group of pixels having a pixel value equal to or less than the threshold value TH is classified into class [1] (that is, one of 0/1), which is larger than the threshold value TH. The pixel group having the pixel value is classified into the class [2] (the other of 0/1). In the discriminant analysis method, the threshold value TH at which the separation degree S is maximized is searched for by calculating the separation degree S represented by the equation (1) while changing the threshold value TH.
S = σ _b ² / σ _w ² … (1)

In equation (1), σ _b ² is an interclass variance and is given by equation (2). σ _w ² is an intraclass variance and is given by Eq. (3). Further, in the equations (2) and (3), w ₁ , m ₁ , and σ ₁ are the number of pixels of the class [1], the mean value of the pixel values, and the variance, respectively, and w ₂ , m ₂ , σ. Reference numeral ₂ is the number of pixels of the class [2], the average value of the pixel values, and the variance, respectively.
σ _b ² = w ₁ w ₂ (m ₁ − m ₂ ) ² / (w ₁ + w ₂ ) ² … (2)
σ _w ² = (w ₁ σ ₁ ² + w ₂ σ ₂ ² ) / (w ₁ + w ₂ )… (3)

Here, as can be seen from the numerator of the formula (2), the degree of separation S of the formula (1) increases as the difference between the average value m ₁ of the class [1] and the average value m ₂ of the class [2] increases. Become. The difference between the average values represents the contrast. That is, when the discriminant analysis method is used, it is possible to indirectly calculate the index value of the contrast for each image by calculating the maximum degree of separation S for each image.

Using such a discriminant analysis method, the image analyzer 27 and the comparator 28 execute the process as shown in FIG. 5A. First, the image analyzer 27 stores the separation degree S (maximum separation degree Smax) calculated by using the discriminant analysis method as the image analysis result AR [i] for each lighting condition ILC [i] in the memory 29 (. Step S201). After that, the comparator 28 compares the degree of separation (Smax) stored in the memory 29 (step S202). Then, the comparator 28 selects the illumination condition at which the degree of separation (Smax) is maximum (in other words, the contrast is maximum) as the final illumination condition ILCx (step S203).

<< Main effect of Embodiment 1 >>
As described above, by using the method of the first embodiment, it is possible to typically reduce the workload of the user associated with the determination of the lighting conditions. In addition, the work time of the user can be shortened, and eventually the inspection time can be shortened. Further, the user's proficiency level is not required when determining the lighting conditions, and the lighting conditions can be automatically determined based on an objective index (here, contrast). As a result, it becomes possible to suppress variations in inspection accuracy.

(Embodiment 2)
<< Details of image analyzer >>
FIG. 6 is a schematic diagram illustrating an operation example of the image analyzer and the comparator in FIG. 2 in the image processing apparatus according to the second embodiment of the present invention. In FIG. 6, the image analyzer 27 uses an edge extraction filter to inspect the image data 31c (IMD) and 31d (IMD) for each illumination condition ILC [3] and ILC [4] using an edge extraction filter (specifically, inspection). The edge of the object 20a) is extracted, and the

image data

32c and 32d after the edge extraction are created. Further, the image analyzer 27 calculates the intensity values E [3] and E [4] of the extracted edges for the

image data

32c and 32d. Then, the image analyzer 27 outputs the intensity values E [3] and E [4] of the edges as the image analysis results AR [3] and AR [4].

The comparator 28 compares each image analysis result AR [3], AR [4] (that is, edge intensity values E [3], E [4]) to obtain a plurality of lighting conditions ILC [3], ILC. Select any one of the lighting conditions from [4]. FIG. 6 shows the relationship between the pixel position (X coordinate) and the cumulative value (cumulative pixel value) of the pixel value (density) in the Y coordinate direction for each X coordinate in the

image data

32c and 32d. The edge intensity values E [3] and E [4] are determined by, for example, the peak value of the cumulative pixel value.

In this example, the edge intensity value E [4] in the image data 32d of the illumination condition ILC [4] is higher than the edge intensity value E [3] in the image data 32c of the illumination condition ILC [3]. In this case, the comparator 28 determines that the illumination condition ILC [4] having a high edge intensity value is better than the illumination condition ILC [3], and selects the illumination condition ILC [4]. Here, the edge intensity values E [3] and E [4] are used as the image analysis results AR [3] and AR [4], but in addition to this, the sum of the cumulative pixel values and the like are used. You may. Specifically, for example, a value obtained by multiplying the sum of the edge strength values and the cumulative pixel values by a predetermined weighting may be used.

Here, as the edge extraction filter used in FIG. 6, a sobel filter can be typically mentioned. FIG. 7 is a diagram illustrating an example of an edge extraction method using a sobel filter using an image analyzer in FIG. As shown in FIG. 7, the Sobel filter uses a 3-by-3 kernel Kx for extracting row-wise edges and a 3-row-3 kernel Ky for extracting columnar edges. Be done.

Further, the image data IMD includes pixel values (density) I [11] to I [xy] in the x-row and y-columns. The image analyzer 27 creates image data in which edges in the row direction are extracted by convolving the kernel Kx with respect to the image data IMD. Similarly, the image analyzer 27 creates image data in which edges in the Y direction are extracted by performing a convolution operation of the kernel Ky with respect to the image data IMD.

Specifically, the image analyzer 27, for example, convolves the kernel Kx with respect to the pixel value group of 3 rows and 3 columns centered on the pixel value I [22], and then calculates the pixel value I [23]. Spatial filtering processing such as convolution calculation of kernel Kx for a pixel value group of 3 rows and 3 columns at the center is performed for all pixel values. As a result, the image data after the spatial filtering process including the pixel values of x rows and y columns (that is, the image data after edge extraction in the row direction) can be obtained. Further, by performing the same spatial filtering process using the kernel Ky, image data after edge extraction in the column direction can be obtained. The edge extraction filter is not limited to the Sobel filter, and may be various filters such as a differential filter and a Previt filter.
<< Main effect of Embodiment 2 >>
As described above, even by using the method of the second embodiment, the same effect as that of the first embodiment can be obtained. Whether to use the method of the first embodiment (contrast) or the method of the second embodiment (edge strength) depends on, for example, the types of the inspection object 20a and the background 20b in FIG. 1 and the specifications of the lighting device 16. It may be determined as appropriate according to the above. Further, a method may be used in which the determination is made by appropriately combining both the contrast and the edge strength instead of either the contrast or the edge strength.

(Embodiment 3)
<< Details of image analyzer >>
FIG. 8 is a schematic view showing a configuration example of the image analyzer in FIG. 2 in the image processing apparatus according to the third embodiment of the present invention. In the first and second embodiments, contrast and edge intensity are used in order to quantitatively evaluate the lighting condition ILC [i]. However, assuming that the conventional evaluation criteria for images by a proficient user are optimal, there may be a difference between this manual evaluation criteria and the evaluation criteria using contrast and edge strength.

Therefore, in the third embodiment, AI (Artificial Intelligence) is made to learn the evaluation criteria of the image by the proficient user, so that the AI is made to evaluate instead of the proficient user. Specifically, the image analyzer 27 is equipped with an AI (Artificial Intelligence) learned to calculate an evaluation value equivalent to an evaluation value based on a user's evaluation standard. Then, the image analyzer 27 calculates an evaluation value for the image data IMD [i] for each lighting condition ILC [i] using AI, and outputs the evaluation value as an image analysis result AR [i].

The image analyzer 27 shown in FIG. 8 is equipped with a convolutional neural network (CNN). CNN is widely known as AI for performing image recognition. The CNN mainly includes an input layer 35, a convolution layer 36, a pooling layer 37, a fully connected layer 38, and an output layer 39. The convolutional layer 36 and the pooling layer 37 are not limited to one layer, and may be configured to be stacked alternately.

Image data 31 (IMD) including a plurality of pixel values in the matrix direction is input to the input layer 35. The convolution layer 36 performs a convolution operation on the image data 31 in the input layer 35 by using

filters

41a, 41b, 41c of a plurality of channels (3 channels in this example), respectively. As a result, the convolution layer 36 creates

image data

42a, 42b, 42c of a plurality of channels. Weight coefficients W ₀ ¹ , W ₀ ² , and W ₀ ³ are set in the

filters

41a, 41b, and 41c, respectively.

The pooling layer 37 performs predetermined arithmetic processing on the

image data

42a, 42b, 42c of the plurality of channels created by the convolution layer 36, respectively. As a result, the pooling layer 37 creates

image data

43a, 43b, 43c of a plurality of channels with a reduced spatial size. Specifically, in the pooling layer 37, for example, in a state where the image data 42a is partitioned so as to include a plurality of (2 × 2 in this example) pixel values, the maximum value or the average value of the plurality of pixel values is partitioned for each partition. The image data 43a is created by obtaining the image data 43a and the like.

The fully connected layer 38 expands the

image data

43a, 43b, 43c of the plurality of channels created by the pooling layer 37 into a one-dimensional vector. The output layer 39 outputs the evaluation value EV by performing a product-sum operation on each of the one-dimensional vectors in the fully connected layer 38 using a predetermined weighting coefficient Wi.

At the time of learning using such an image analyzer 27, for example, image data 31 and an evaluation value determined by a user who is proficient in the image data 31 are input as teacher data. The image analyzer 27 calculates an error between the evaluation value EV output from the output layer 39 and the evaluation value determined by a proficient user when the image data 31 is used as the input layer 35. Then, the image analyzer 27 uses an error back-propagation method to bring this error closer to zero, and uses a weighting coefficient Wi between the fully coupled layer 38 and the output layer 39 and a weighting coefficient of the

filters

41a, 41b, 41c. Update W ₀ ¹ , W ₀ ² , and W ₀ ³ .

By repeating such learning, for example, the characteristics of the image that a proficient user attaches importance to when determining the evaluation value are mainly the weighting coefficients W ₀ ¹ , W ₀ ² , W of the

filters

41a, 41b, 41c. It is learned as ₀ ³ . Further, the relationship between the feature of the image and the evaluation value EV is mainly learned as the weighting coefficient Wi. As a result, when the predetermined image data 31 is given to the input layer 35 at the time of inference, the image analyzer 27 can output an evaluation value EV equivalent to the evaluation value based on the evaluation criteria of a proficient user from the output layer 39. become. Further, in this case, the comparator 28 may select the lighting condition having the highest evaluation value EV as the final lighting condition ILCx in steps S110 and S111 of FIG.

<< Main effect of Embodiment 3 >>
As described above, even by using the method of the third embodiment, the same effect as that of the first embodiment can be obtained. Furthermore, depending on how the AI is trained, it becomes possible to further optimize the evaluation criteria of the image when determining the lighting conditions. As a result, it becomes possible to improve the inspection accuracy of the inspection system.

(Embodiment 4)
<< Outline of inspection system >>
In the above-described embodiments 1 to 3, the image processing apparatus 18 performs image analysis on the image data IMD obtained for each lighting condition ILC, and compares the image analysis results for each lighting condition ILC. One lighting condition, ILCx, was selected. In addition to this, in the fourth embodiment, the image processing device 18 performs image analysis on the image data obtained for each image pickup condition of the image pickup device 15 in the same manner as in the case of the illumination condition ILC, and the image pickup condition thereof. By comparing the image analysis results for each, one of the imaging conditions is selected.

FIG. 9 is a schematic diagram showing a configuration example of a main part in the inspection system according to the fourth embodiment of the present invention. The inspection system 10a shown in FIG. 9 differs from the configuration example shown in FIG. 1 in the following points. That is, the image pickup device 15 is set with the image pickup condition GLC at the time of image pickup. Along with this, the image processing device 18 determines the image pickup condition GLC for the image pickup device 15, and transmits the image pickup control signal GCS corresponding to the image pickup condition GLC to the image pickup device 15 via the cable 19a.

Here, the image processing device 18 is realized in detail with the same configuration as in the case of FIG. However, unlike the case of FIG. 2, the illumination condition searcher 26 variably sets the imaging condition GLC at the time of imaging for the imaging device 15 in addition to the illumination condition ILC for the lighting device 16. Along with this, the image interface 25 acquires the image data IMD for each image pickup condition GLC. The image analyzer 27 performs image analysis on the image data IMD for each imaging condition GLC. The comparator 28 selects one of the imaging conditions (referred to as GLCx) from the plurality of imaging condition GLCs by comparing the image analysis results for each imaging condition GLC by the image analyzer 27.

As a specific example, it is assumed that the correction value of the brightness correction function is set in the image pickup device 15 as the image pickup condition GLC. First, as a premise of the process using the configuration of FIG. 9, a predetermined lighting condition ILC is set in the lighting device 16. The predetermined lighting condition ILC may be, for example, the final lighting condition ILCx defined by any of the methods 1 to 3, that is, the lighting condition that gives the best image analysis result.

The image processing device 18 acquires the image data IMD while changing the correction value of the brightness correction function under the predetermined lighting condition ILC, and performs image analysis on the acquired image data IMD. Then, the image processing apparatus 18 determines the correction value that gives the best image analysis result, that is, the image pickup condition GLC, as the final image pickup condition GLCx.

<< Main effect of Embodiment 4 >>
As described above, by using the method of the fourth embodiment, in addition to the various effects described in the first embodiment and the like, it is possible to reduce the workload of the user associated with the determination of the imaging condition GLC. Further, as a result of optimizing the imaging condition GLC in addition to the illumination condition ILC, it becomes possible to further improve the inspection accuracy of the inspection system.

(Embodiment 5)
<< Outline of inspection system >>
FIG. 10 is a schematic view showing a part of a configuration example in the inspection system according to the fifth embodiment of the present invention. The inspection system 10b shown in FIG. 10 differs from the configuration example shown in FIG. 1 in the following points. That is, the recognition mark 20c and the component reference mark 20d are written on the wiring board 20b. Although omitted in FIG. 10 for the sake of brevity, the inspection system 10b also includes, in detail, the lighting control device 17 and the image processing device 18 shown in FIG. 1 or 9.

The recognition mark 20c is used as a reference point when determining the orientation of the wiring board 20b. The component reference mark 20d is used as a reference point when the electronic component 20a is oriented on the wiring board 20b. The recognition mark 20c and the component reference mark 20d are formed using various shapes, colors, and materials. The image processing apparatus 18 (not shown) in the inspection system 10b usually detects such a recognition mark 20c and a component reference mark 20d as a preliminary preparation when starting the inspection of the electronic component 20a.

At this time, the image processing apparatus 18 uses the recognition mark 20c or the component reference mark 20d as an inspection target, and uses the method as described in the first to fourth embodiments to add the final lighting condition ILCx and the addition. The final imaging conditions may be determined. That is, the inspection object to be imaged when determining the final illumination condition ILCx and the final imaging condition is not limited to the electronic component 20a, but may be the recognition mark 20c or the component reference mark 20d.

(Embodiment 6)
<< Details of image processing equipment >>
FIG. 11 is a schematic view showing a configuration example of a main part of the image processing apparatus according to the sixth embodiment of the present invention. The image processing device 18a shown in FIG. 11 is mainly realized by a processor such as a CPU and a GPU, and performs image analysis using AI as in the case of FIG. The image processing apparatus 18a includes, as an AI model, a plurality of layers (four layers in this example) convolution layers 36a to 36d, a fully connected layer 38, and an output layer 39.

Image data 31 is input to the convolution layer 36a. The image data 31 is, for example, monochrome image data having a vertical size of 256 pixels and a horizontal size of 256 pixels. For example, the convolution layer 36a performs an operation using an activation function called ReLU (Rectified Linear Unit) after performing a convolution operation of the image data 31 and the weight coefficient obtained by learning, and the maximum value pooling is performed. The required feature quantity is extracted using the method.

The convolution layer 36b performs the same processing as in the case of the convolution layer 36a on the output data from the convolution layer 36a. However, in the convolution layer 36b, the number of convolution operations and the like are different from those in the convolution layer 36a. Similarly, the convolution layers 36c and 36d also perform a convolution operation on the output data from the previous stage to extract necessary features.

The fully connected layer 38 performs a product-sum calculation of the output data from the convolution layer 36d and the weighting coefficient obtained by learning, and outputs 1 × N one-dimensional data. The output layer 39 performs an operation on the output data from the fully connected layer 38 using an arbitrary activation function according to the learning method of AI and outputs the operation result. Examples of the activation function include a sigmoid function, a softmax function, an identity function, and the like.

Here, as a learning method of AI, a method called regression and a method called classification are mainly known. In regression, a regression model is used to infer numerical values such as continuous values. When the regression model is used, one evaluation value is output for each input. In the third embodiment described above, an example in the case of using regression is shown. When regression is used, the configuration example of FIG. 11 is applied to the image analyzer 27 of FIG. Further, the activation function of the output layer 39 may be, for example, an identity function or the like.

On the other hand, in the classification, a classification model for guessing which class the input data belongs to in the learned class is used. When class classification is used, for example, the lighting condition A predetermined as teacher data is output for the input of the image data 31 of the inspection object A at the time of learning, and the lighting condition A is output for the input of the image day 31 of the inspection object B. Each feature amount, that is, a weighting coefficient is learned so as to output the lighting condition B defined in advance as teacher data.

As a result, when the image data 31 of the inspection object A is input to the image processing device 18a at the time of operation, in other words, at the time of deduction, the lighting condition A can be acquired from the image processing device 18a. That is, in this case, the search process by the lighting condition searcher 26 and the comparison process by the comparator 28 shown in FIG. 2 are unnecessary, and the final image data 31 is input to the image processing device 18a. The lighting condition ILCx will be obtained directly. Further, in the classification, the activation function of the output layer 39 may be, for example, a softmax function or the like.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. .. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

10,10a,10b: Inspection system, 15: Imaging device, 16: Lighting device, 17: Lighting control device, 18,18a: Image processing device, 19a-19c: Cable, 20a: Electronic parts (inspection target), 20b : Wiring board (background), 20c: Recognition mark (inspection object), 20d: Parts reference mark (inspection object), 25: Image interface, 26: Lighting condition searcher, 27: Image analyzer, 28: Comparer , 29: Memory, 31, 31a to 31d: Image data, 35: Input layer, 36, 36a to 36d: Folding layer, 37: Pooling layer, 38: Fully coupled layer, 39: Output layer, Filters 41a to 41c, 42a ~ 42c: image data, 43a ~ 43c: image data, AR: image analysis result, E: edge intensity value, EV: evaluation value, GCS: image pickup control signal, GLC: image pickup condition, I: pixel value, ICS: illumination Control signal, ILC: Lighting conditions, IMD: Image data

Claims

An image processing device that processes image data of an inspection object captured under illumination.
A lighting condition searcher that variably sets the lighting conditions of the lighting, and
An image interface for acquiring the image data for each lighting condition, and
An image analyzer that performs image analysis on the image data for each lighting condition, and
A comparator that selects any one of the lighting conditions from a plurality of the lighting conditions by comparing the image analysis results for each of the lighting conditions by the image analyzer.
Have,
Image processing device.
In the image processing apparatus according to claim 1,
The image analyzer calculates the contrast with respect to the image data for each of the lighting conditions, and outputs the contrast as the image analysis result.
Image processing device.
In the image processing apparatus according to claim 1,
The image analyzer extracts an edge from the image data for each lighting condition, calculates the intensity value of the extracted edge, and outputs the intensity value of the edge as the image analysis result.
Image processing device.
In the image processing apparatus according to claim 1,
The image analyzer is equipped with an AI (Artificial Intelligence) learned to calculate an evaluation value equivalent to an evaluation value based on a user's evaluation standard, and the AI is applied to the image data for each lighting condition. The evaluation value is calculated using the above evaluation value, and the evaluation value is output as the image analysis result.
Image processing device.
In the image processing apparatus according to claim 1,
Further, the present invention has an imaging device that creates the image data by imaging the inspection object in the illuminated state and sets the imaging conditions at the time of the imaging.
The illumination condition searcher variably sets the imaging conditions at the time of imaging, and the illumination condition searcher sets the imaging conditions variably.
The image interface acquires the image data for each image pickup condition, and obtains the image data.
The image analyzer performs image analysis on the image data for each image pickup condition, and performs image analysis.
The comparator selects any one of the imaging conditions from the plurality of imaging conditions by comparing the image analysis results for each imaging condition by the image analyzer.
Image processing device.
A lighting device that illuminates the object to be inspected,
An image pickup device that creates image data by taking an image of the inspection object in the illuminated state, and an image pickup device.
An image processing device that processes the image data and
Is an inspection system that has
The image processing device is
A lighting condition searcher that variably sets the lighting conditions of the lighting device,
An image interface for acquiring the image data for each lighting condition from the image pickup device, and
An image analyzer that performs image analysis on the image data for each lighting condition, and
A comparator that selects any one of the lighting conditions from a plurality of the lighting conditions by comparing the image analysis results for each of the lighting conditions by the image analyzer.
Have,
Inspection system.
In the inspection system according to claim 6,
The image analyzer calculates the contrast with respect to the image data for each of the lighting conditions, and outputs the contrast as the image analysis result.
Inspection system.
In the inspection system according to claim 6,
The image analyzer extracts an edge from the image data for each lighting condition, calculates the intensity value of the extracted edge, and outputs the intensity value of the edge as the image analysis result.
Inspection system.
In the inspection system according to claim 6,
The image analyzer is equipped with an AI (Artificial Intelligence) learned to calculate an evaluation value equivalent to an evaluation value based on a user's evaluation standard, and the AI is applied to the image data for each lighting condition. The evaluation value is calculated using the above evaluation value, and the evaluation value is output as the image analysis result.
Inspection system.
In the inspection system according to claim 6,
Imaging conditions for imaging are set in the imaging device, and the imaging conditions are set.
The illumination condition searcher variably sets the imaging conditions at the time of imaging, and the illumination condition searcher sets the imaging conditions variably.
The image interface acquires the image data for each image pickup condition, and obtains the image data.
The image analyzer performs image analysis on the image data for each image pickup condition, and performs image analysis.
The comparator selects any one of the imaging conditions from the plurality of imaging conditions by comparing the image analysis results for each imaging condition by the image analyzer.
Inspection system.
A lighting device that illuminates the object to be inspected,
An image pickup device that creates image data by taking an image of the inspection object in the illuminated state, and an image pickup device.
It is an inspection method for inspecting the inspection object using an inspection system including.
The first step of variably setting the lighting conditions of the lighting device,
A second step of acquiring the image data for each of the lighting conditions from the image pickup apparatus and performing image analysis on the image data for each of the lighting conditions.
A third step of selecting any one of the lighting conditions from the plurality of lighting conditions by comparing the image analysis results for each lighting condition, and
Have,
Inspection method.
In the inspection method according to claim 11,
In the second step, the contrast is calculated for the image data for each of the lighting conditions, and the contrast is output as the image analysis result.
Inspection method.
In the inspection method according to claim 11,
In the second step, an edge is extracted from the image data for each lighting condition, an intensity value of the extracted edge is calculated, and the intensity value of the edge is output as the image analysis result.
Inspection method.
In the inspection method according to claim 11,
In the second step, the evaluation is performed on the image data for each lighting condition by using AI (Artificial Intelligence) learned to calculate an evaluation value equivalent to the evaluation value based on the user's evaluation standard. The value is calculated, and the evaluation value is output as the image analysis result.
Inspection method.
In the inspection method according to claim 11,
Imaging conditions for imaging are set in the imaging device, and the imaging conditions are set.
The above-mentioned inspection method further
A fourth step of variably setting the imaging conditions of the imaging device under the lighting conditions selected in the third step, and
A fifth step of acquiring the image data for each imaging condition from the imaging device and performing image analysis on the image data for each imaging condition.
By comparing the image analysis results for each of the imaging conditions, the sixth step of selecting one of the imaging conditions from the plurality of imaging conditions and the sixth step.
Have,
Inspection method.