JPH0431976A - Image monitoring method - Google Patents

Abstract

PURPOSE:To accurately perform monitoring along a monitoring purpose by performing monitoring decision by calculating the feature quantity of a monitoring area by magnifying a feature area by prescribed number of picture elements after eliminating a noise on a screen. CONSTITUTION:A differential image can be obtained by performing image pickup and binary coding setting a substrate before element packaging as a preceding image and the substrate after packaging as a succeeding image. Thence, one picture element with runlength less than prescribed horizontal and vertical effective runlength Lhe, Lve is eliminated to eliminate noise parts S2 in horizontal and vertical directions in an element packaging part S1 and a noise part S2 in the monitoring area S. Then, the picture element is magnified by prescribed number of expansion picture elements in both horizontal and vertical directions. Therefore, aberration when the element is packaged can be permitted. Thence, when the extraction of the monitoring area T is completed, the block is divided into N inspection blocks Q, and the decision for the normal/defective condition of the block is performed by counting the number of significant picture elements in a normal image i.e. the number of picture elements of binary-coded differential image between the preceding image and the succeeding image.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention monitors whether holes are correctly drilled when IC elements, etc. are correctly mounted on a flat substrate during, for example, a manufacturing and assembly process. The present invention relates to an image monitoring method for.

[Prior Art] To determine the quality of length, pattern, etc., for example, to determine whether the length of the object to be inspected (object to be monitored) is as specified or whether the shape of the object is as specified, Take an image with a camera, etc., specify in a window the characteristic area where the length, shape, etc. appear, specify a narrow inspection area, and count the number of pixels in that area where the length and shape appear. The physical quantity is measured with high precision using the method, and quality determination is performed based on whether the number of pixels is within an allowable value or not.

By the way, in the appearance inspection after processing in the manufacturing and assembly process, for example, in the element mounting process, the accuracy of the installed position of the element is not important, but the accuracy of the element is simply In many cases, it is sufficient if it is installed.

[Problem to be solved] In inspections (hereinafter referred to as monitoring) based on vague and abstract criteria such as visual inspections, the monitoring area is divided into multiple blocks and each block is monitored at the same monitoring level. It is not preferable that the same monitoring level be applied to the characteristic area portion where the device is mounted and other portions which are not processed in the manufacturing process. That is, it is desirable that the characteristic region portion and other non-characteristic region portions have different degrees of contribution to the monitoring region. Further, in order to find out changes in the entire monitoring area, it is necessary to examine the non-characteristic area in detail, which actually deviates from the purpose of monitoring.

On the other hand, if the monitoring area is monitored as a single monitoring area without dividing it, for example, there may be dust attached to a certain area, or there may be scratches in another area, which will appear as different codes during the inspection stage. Therefore, these factors cancel each other out, and there are cases where the object is erroneously determined as normal, and there are also cases where the influence of noise increases and the object is erroneously determined.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide an image monitoring method that allows correct monitoring in accordance with the purpose of monitoring.

[Means for Solving the Problems] In order to achieve the above object, the present invention sets a portion including a characteristic area as a monitoring area in the screen of an imaging means that images an object to be monitored, and In an image monitoring method that generates feature quantities and makes monitoring judgments, the monitoring area is set after removing noise in the screen.
This is performed on a screen in which the characteristic region is enlarged by a predetermined number of pixels.

Further, it is preferable that the monitoring determination is performed by dividing the monitoring area into a plurality of blocks and calculating a feature amount for each divided block.

Furthermore, it is preferable that the feature amount is calculated from a difference image between an unprocessed image and a processed image of the monitored object.

[Example] Hereinafter, an example of the present invention will be described in detail based on the drawings.

First, the overall configuration of the image monitoring device will be explained. Second
In the figure, a monitoring target 1 is in the process of mounting elements (not shown) on a manufacturing line or the like, and under illumination 2, an imaging means 3 images the surface on which the elements are mounted. Imaging is performed for both a preliminary image before mounting and a subsequent image after mounting. Input control signals for completion of mounting preparation and completion of mounting in the element mounting robot are inputted to the apparatus main body 7 through the input line 5. The operating conditions of the device body 7 and the like are input using the keypad 8. When an input control signal is input from the input line 5, the device main body 7 performs pre-imaging or post-imaging.

The pass/fail judgment is executed and the result is outputted as an output control signal from the output line 6. The main body 7 of the apparatus is provided with a CRT 7a capable of displaying an image captured by the imaging means 3, an image signal obtained by binary encoding the image, and the like.

Next, the overall block configuration of the image monitoring device will be explained using FIG. 3.

The ROM 21 stores programs for operating the CPU 22, such as noise removal, subsequent enlargement (expansion) of the feature region, and quality determination.

The RAM 23 stores horizontal effective run length Lhe and vertical effective run length Lve during noise removal and subsequent enlargement of the feature region, the monitoring region extracted according to the present invention, inspection blocks, reference number of significant pixels in each inspection block, Data such as pass/fail judgment reference value 1 is stored. Horizontal effective run length Lhe, vertical effective run length Lve, block size.

Conditions such as allowable values for pass/fail judgment are entered by the operator using the keypad 8, etc. while looking at the CRT 7a, etc.
It is stored in RAM 23 via I/O port 29.

Based on the commands in the ROM 21 and the data in the RAM 23, the CPU 22 performs noise removal, subsequent enlargement of the characteristic region, quality determination, and the like.

The image signal captured by the imaging means 3 is stored in the frame memory 31 via the A/D circuit 30, and is also stored in the frame memory 31.
The stored contents of 1 are displayed on the CRT 7a via the display circuit 32. Note that the frame memory 31 has storage areas for each of a pre-image and a post-image.

Next, the image monitoring method according to the present invention will be explained in detail using FIGS. 1 and 4 to 7.

In FIG. 4, first, monitoring conditions such as a threshold value for binarizing the video imaged by the imaging means 2 and an allowable value for making a pass/fail determination are set (402).

Then, a normal board before device mounting is imaged as a preliminary image, and this is binary encoded. Further, after the elements are normally mounted on the board, the board is imaged as a post-image, and this is binary coded.

Then, a difference image (exclusive OR) between the binary-encoded prior image and the subsequent image is obtained (403). As a result, as shown in FIG. 1(a), a binary reference difference image with a significant difference of 1 is obtained.

Next, a monitoring reference area is extracted (404).

This is done by removing noise and subsequently expanding (bloating) the feature region.

FIG. 5 explains the case of removing noise in the horizontal direction. As shown in FIG. 1(a), the imaging means 1
The monitoring area S (which may be the screen of the imaging means 1 itself, or a part of which may be designated by a window, etc.) set within the screen of is a significant difference pixel area (region of pixels with reference numeral 1). exists, but this is the part SL where elements etc. are mounted.
and a portion S2 formed by noise or the like.

Therefore, in order to eliminate the portion S2 formed by noise or the like, a significant difference pixel portion shorter than the effective horizontal run length Lhe of a predetermined number of pixels in the horizontal direction is removed. Let the pixel point at the upper left corner of the monitoring area S be the reference point, let the rightward and downward coordinates be it J, and let the difference image of point (i, j) be I c (i, j). And (i, j)-(1
,1) Look further to the right. First, j=1. i-1 and 502. 503), and the counter PhO-0 and the counter Ph1-0 (504). The counter PhO counts the number of continuous code 0 pixels immediately before the current pixel position, and the counter Pold counts the number of continuous code 1 pixels immediately before the current pixel position. For convenience of explanation, (
i, j) = 00111110 to the right from (1, 1)
0011101 · and so on. That is, two 0 code pixels follow, then five 1 code pixels, then three 0 code pixels, then three 1 code pixels, and then one 0 code pixel. Assume that one code pixel follows the other, and then one code pixel follows. Also, the effective lateral run length Lh
Let e be 4 pixels, and the number of horizontal pixels and the number of vertical pixels of the monitoring area S be KH and KV, respectively.

In the judgment of Ic(1,1)-1 (505), since the position of (1,1) is a 0 code pixel according to the above assumption, Ph1
Proceed to -0 judgment (506), and at 504 it is YES because it is Ph1-0, and at 511, Ph0-0 +1
-1, and the process goes through steps 515 and 516' to determine the case of Ic (2, 1) (505). And in this case too ('2.
Since the pixel in 1) has a 0 code and Phi remains 0, it is set to 511 as Ph0-1+1-2, and 515.516
The process then proceeds to determination (505) in the case of Ic(3,1).

Here, I c (3, 1) has changed to 1, so 5
Based on the judgment of 05, the process proceeds to the judgment of Pho-0 (512), and Ph
Since it is 0-2, the process goes to 513 and the P of 2 pixels is
Regenerate hO into a noise-removed (reduced) image 1d. Also(
Since PhO is discontinued at position 3.1), it is designated as pho-o. Then, in step 514, it is set to Ph1-01-1.

Next, the process goes through steps 515 and 516 to determine the case of Ic (4, 1) (505). Here, since Ic (4, 1) is 1, the process proceeds to 512 based on the determination in 505, and pho-.

Therefore, proceed to 514, increment the count by 1 as Ph1-1+1-2, and proceed to 515, 51 [5 to I c
The process proceeds to determination (505) in the case of (5, 1). Ic(5,
■) also passes through the same route as Ic(4,1), and P
h1-2 + 1-3 is counted. Ic(6,1
), 1c(7,1), Ph1-
It becomes 5.

Next, move on to the judgment of I c (8, 1), I c (8, 1) -
Since it is 0, it is determined whether it is Ph1-0 (506),
Since Phl-5, it is determined whether Phl≧Lhe (507), and since Phi is greater than or equal to the horizontal effective run length Lhe-4, Phl is reproduced into the reduced image Id (
509), and Phi becomes 0 (510).

Furthermore, Ph0-0+1-1 is counted (511) (
Since the positions of 9,1) and (10,1) are also code 0,
In these, the process proceeds to 506, and in 511, Ph0-
It becomes 3.

Next, in order to become a pixel with code 1 at the position (11, 1) as in the case of the position (3, 1), the pixel with code 0 passes through the same route as in the case of (3, 1). The pixels are reproduced into a reduced image Id. and (12,1), (13,1)
At , the code 1 continues, progresses to the 512 side, and the value of Phi becomes 3.

At (14,1), the code is 0, so the process goes to 506, and since the value of Phl is 3, the process goes to 507. However, since Phi is less than Lhe-4, it is not reproduced into the reduced image Id, and the 3-pixel Phi is changed to pho (508). Then, Phi becomes 0 and PhO is incremented by one.

Ph
It becomes 0-3 + 1-4.

Then, since the code becomes 1 at (15, 1), the process advances to 512, and since it becomes Ph0-4, PLO-4 is reproduced into the reduced image 1d, and PhO becomes 0. Also Ph
1-0+1, and in this way, judgments are made one after another for subsequent pixels. Then, when it reaches the position (KH, 1), it advances to 517, 518, where j becomes 2, and from then on (i
, 2) will be determined sequentially. In this way, determination is made up to j-KV, and the noise removal process ends. Continuous pixels with a code of 1, which is less than the horizontal effective length Lhe = 4 pixels in the horizontal direction, are not reproduced as noise and have a code of 0.
pixel. Although Fig. 5 is for the horizontal direction, Fig. 5 can also be applied to the vertical direction by exchanging i and j and determining an appropriate vertical effective run length Lve. By performing both processes, consecutive pixels with code 1 that are less than horizontal effective run length L he and vertical effective run length Lye are replaced with pixels with code O. This situation is shown in FIG. 1(b), in which one pixel that is less than a predetermined horizontal effective run length Lhe and vertical effective run length Lve is removed from the difference image shown in FIG. 1(a). ing.

Next, the operation of enlarging (expanding) the feature region will be explained.

FIG. 6 is a flowchart regarding expansion (expansion) in the horizontal direction. Here, for convenience of explanation, the above results (001111
100000001...), and the number of pixels to be expanded is es-1.

In FIG. 6, PIO indicates the number of alternating codes up to the pixel immediately before the current pixel position. First, j-1, i=
1. Ph0-0. Ph1-0. Set as PIO-0 (602 to 604). Then, since PLO-0 (
605), and dp becomes e(-1) (607). and since I d(1,1)-0 (60g), 609
Proceed to 611 to be phi-o.
The result becomes Ph0-1, and the same course is followed at Id(2,1), resulting in Ph0-2.

The pixel then passes through 605 and 607 again to determine the pixel at the next position. In 608, since I d(3,1)-1, the process proceeds to 612, and since it is Ph0-2, the process proceeds to 613. pho is set to 2-1-1, and this PhO is reproduced into an expanded image 1e. Also, PIO-0Dou1-1, p
Let's say ho-o. Also, it is set to Ph1-1 (614).

Then it becomes i-4 (616), and at 605 PIO
Since is not 0, the process proceeds to 606 and becomes dp-2e-2.

Since I d(4,1)-1, select 612 at 608, and because it is pho-o, proceed directly to 614;
Ph1-2, I d (5, 1), I d (1
1i,l), Id(7,1) are also similar, and Ph
The score will be 1-5.

Next, since l d(8,1)-1, proceed to 609,
Since it is Ph1-5, proceed to 610 and enter Ph1-Phi.
+dp-5+2=7, and this Phi is reproduced into the expanded image ■e following one code 0 pixel that has already been reproduced. As a result, 8 pixels from the leftmost pixel position have been reproduced, and the 1 code pixels that were 3rd to 7th from the leftmost position before expansion have been increased by 1 on the left and right. There is.

Next, as PIO-1+1-2 and as Ph1-0 (610), and further as Ph0-1 (611) 60
5. Through 606 (9,1), (10°1),...
, proceed to position (14, 1) (608, 609, 611
), which becomes Ph0-7 (611) and (15,1)
Then, the code changes to 1 pixel, so the process goes from 608 to 612, and since it becomes pho-7, the process goes to 613, where it becomes pho-phO-d p -7-2-5, and this is the code being played. It is played back one pixel after another. That is, the code 0 pixel is reproduced as a continuation of the code 1 pixel with two pixels at both ends removed from before being expanded. And as PLO-2+1-3 and as pho-o (61
3) Further, as Ph1-1, proceed to determination of the pixel at the next position.

In this way, when we reach the position of (KH, 1), (615)
, proceeds to judgment (617) whether or not PhQ-0, and pho
is not 0, P hO-P hO-1 is the fat image re
Further, when PhO is 0, it is determined whether Ph1-0 or not (61B), and in step 619, Ph1-Phl+1 is reproduced as the fat image 1e.

Then, the process advances to line i-2, and the same judgment as above is made, and when the expansion process is completed until j=Kv (621), the process ends (623). In this way, in the horizontal direction, the pixel column numbered 1 is reproduced with one pixel added at each end.

If i and j are replaced in FIG. 6, a flowchart of the vertical expansion process is obtained. By performing both horizontal and vertical expansion, the pixel numbered 1 can be expanded by a predetermined number of expansion pixels in both directions. The first one shows this situation.
The image shown in FIG. 3C is an image obtained by enlarging the image shown in FIG. By performing the expansion process in this way,
Positional deviations when mounting elements can be tolerated.

Furthermore, the illuminance of the monitoring surface changes, and therefore the output of the television camera 3 changes, and the binary encoding signal changes, which causes the area of the part marked 1 to change, but this change can be tolerated. . The number e of bulge pixels is set to encompass these fluctuations, and is preferably set to the lowest value that can encompass these fluctuations, thereby making it possible to automatically set a monitoring area close to the object to be monitored.

This image is the monitoring area T obtained by the present invention, and the monitoring area is obtained by removing noise and expanding the important characteristic area consisting of code 1 by a predetermined number of pixels in the horizontal and vertical directions. (404).

Next, when the extraction of the monitoring area T by the above method is completed, N
inspection blocks Q (block numbers jm1.2, 3.
..., N) (404). The width of each block is Kv, and the vertical width is the vertical effective run length Lve.
shall be. By dividing into N test blocks, an optimal test unit can be obtained.

Next, for each block, count the number of significant pixels in the normal image, that is, the number of pixels with code 1, which is the difference image (exclusive OR) between the binary encoded normal prior image and normal subsequent image. This is then set as the pass/fail judgment reference value POr for that block. Here, since the monitoring area obtained above is expanded by a predetermined number of expanded pixels, the pass/fail determination reference value POr is usually smaller than the number of pixels in the monitoring area. Note that each reference value POr defines a permissible range dur, and the permissible upper limit p maxr -p Or+ dur and the permissible lower limit Pm1nr-POr+ dur in each block. This completes the preparation for image monitoring.

Next, it waits for a monitoring operation command (406), and when the command that the preparation for mounting the device is completed is received, the pre-inspection image 1a
Input and convert into binary encoding (407°702). Then, when the device is mounted and a command to end the event is received (40g), the post-image Ib is input and converted into binary encoding (409.703), and as shown in Figs. 7 and 4. , calculate the difference image 1c by exclusive OR10
4.410).

Then, the product of the difference image Ic and the monitoring area 1e obtained above (
AND) Take an image (705.411).

As a result, out of the difference images 1e, those outside the monitoring area 1e are excluded.

Then, a pass/fail judgment is made for each block (4↓2). The number of test significant pixels Pr is calculated for each block Qr (707
), calculate whether each block is within the allowable range, that is, P m1nr≦P ≦P maxr to determine whether it is acceptable or not (708, 709° 710.711)
.

When the quality determination of all blocks is completed, the determination results are output (413). If it is determined to be abnormal, the next mounting process is stopped, and if re-inspection is possible, the block determined to be abnormal is re-inspected. If it is determined to be normal, the next object to be monitored 1 is transported and the same monitoring operation as above is performed.

In the manner described above, image monitoring of the monitored object 1 is performed.

In the above embodiment, the inspection block is divided into N pieces in the vertical direction, but it is further divided into M pieces in the horizontal direction to form NXM lattice blocks. Division methods are possible.

In addition, in the above embodiment, the code 1 that appears as a result of taking a difference image between a prior image and a post image in which significant pixels are binary encoded is
However, if the difference image between the normal pre-output image before binary encoding and the normal post-output image before binary encoding is taken, and this output value is larger than a predetermined threshold value d, If there is a significant difference, that pixel may be determined as a significant pixel.

Further, if the value of the normal post-output image alone before binary encoding is larger than a predetermined threshold value, that pixel may be determined as a significant pixel.

Further, in the above embodiments, if preliminary monitoring is not performed or if a characteristic region exists in the preliminary image, the present invention can also be implemented in preliminary monitoring.

[Effects] According to the present invention, the monitoring area is configured from a feature area expanded by a predetermined number of pixels after noise in the screen is removed, and the feature amount in this monitoring area is calculated to perform monitoring determination. Therefore, monitoring is performed only in the characteristic region and the surrounding area extremely close to it, and monitoring is performed in accordance with the purpose of monitoring.

In addition, since the monitoring area is limited to the characteristic area and the surrounding area extremely close to the feature area, the room for noise to enter the monitoring area can be reduced as much as possible, and more accurate monitoring can be performed.

In addition, by dividing the monitoring area into multiple blocks and calculating the feature values for each block to make monitoring decisions,
The monitoring area extracted above can be further sized to the optimum size for monitoring.

In addition, by calculating the feature amount from the difference image between the pre-processed image and post-processed image of the monitored object, it becomes possible to carry out overall monitoring by focusing on areas of change before and after the monitored object. .

[Brief explanation of the drawing]

FIG. 1(a) is an explanatory diagram showing a difference image in the monitoring area of the imaging screen, FIG. 1(b) is an explanatory diagram showing a state in which noise has been removed from the screen in FIG. ) is the same figure (b
) is an explanatory diagram showing a state in which the characteristic area is enlarged on the screen, FIG. 2 is an external perspective view showing how an object to be monitored is monitored by the image monitoring device, and FIG. 3 is a diagram showing the configuration of the image monitoring device. 4 is a system flow diagram for explaining the image monitoring operation, and FIG. 5 is a flowchart for specifically explaining the noise removal operation in monitoring reference area extraction in the system flow diagram of FIG. 4. Figure 6 is a flowchart for specifically explaining the area enlargement operation in monitoring reference area extraction in the system flow diagram in Figure 4, and Figure 7 is a flowchart for dividing the preliminary image manually in the system flow diagram in Figure 4. FIG. 7 is a flowchart for more specifically explaining the operation up to pass/fail determination for each area. 1. Object to be monitored, 3. Imaging means, Sl. Characteristic region, Q. Inspection block, T. Monitoring region. that's all

Claims (3)

[Claims] (1) In an image monitoring method in which a portion including a characteristic area is set as a monitoring area in the screen of an imaging means that images an object to be monitored, and a monitoring judgment is made by producing a feature amount within this monitoring area, the monitoring area is as follows: An image monitoring method characterized in that the setting is performed on a screen in which the characteristic region is enlarged by a predetermined number of pixels after noise in the screen is removed. (2) The image monitoring method according to claim 1, wherein the monitoring judgment is performed by dividing the monitoring area into a plurality of blocks and calculating the feature amount for each divided block. (3) The image monitoring method according to claim 1 or 2, wherein the feature amount is calculated from a difference image between an unprocessed image and a processed image of the monitored object.