KR20110030275A - Method and apparatus for image generation - Google Patents

Abstract

PURPOSE: A method and a device for creating images are provided to remove the need for the control of an overlapping probability by a user and to ensure a desired composite image. CONSTITUTION: A method for creating images is as follows. A first partial image is taken on a target, which is observed by a microscope(50). A characteristic point is extracted from the first partial image. An overlapping probability of the first partial image and a second partial image is determined. A composite image is obtained by synthesizing the first and second partial images.

Description

Image generating method and image generating device {METHOD AND APPARATUS FOR IMAGE GENERATION}

The present invention relates to an image generating method of dividing a subject into a plurality of partial images to capture an image, and joining the partial images to generate a whole image of the subject.

Today, in industrial microscopes and inspection apparatuses for inspecting and correcting FPD (flat display) substrates, PDP (plasma display) substrates, semiconductor wafers, and the like, as a method of inspecting and identifying defects that cause functional problems of the substrate, The method using an image is generally performed.

In the method of inspecting and identifying a defect of the substrate, in order to inspect and identify fine defects affecting pattern formation of a substrate or the like with high accuracy, it is necessary to compare a pattern to be inspected with a normal reference pattern. . Therefore, not only the image which covers the whole subject at low magnification but also the high-definition image (high resolution image) which can cover the whole subject at higher magnification is increasing in many cases.

However, depending on the size of the subject, a high-precision image of the entire subject cannot be acquired at once. Therefore, as a technique for acquiring a high-definition image, a desired high-definition image is obtained by dividing the entire high-precision image to be obtained into a plurality of regions, imaging each of these divided regions at high resolution, and joining the partial images obtained by the imaging to each other. Methods of obtaining are well known. In the method of obtaining such a high-precision image, first, a high magnification partial image is imaged based on the low magnification whole image, and the picked up partial image is bonded. Such a method is not limited to industrial use, but is used in various fields and for various applications.

As a technique for joining partial images, there is provided a method of searching for a location suitable for joining in an entire image photographed at a low magnification, and joining a partial image photographed at a high magnification based on a search result to generate a high precision image. (For example, patent document 1).

[Patent Document 1] Japanese Patent Application Publication: Japanese Patent Application Publication No. 2006-006525

For example, when joining partial images by the method described in Patent Document 1, in the whole low-magnification image used for searching for a suitable location for joining, it is necessary to enter the range to be captured, that is, the entire subject. There is. Depending on the size of the subject, it may be necessary to switch to a lower magnification to capture the entire image, or to determine that the image cannot be joined if the entire image into which the entire subject enters even if the magnification is lowered. .

In addition, since the low magnification and the high magnification look different from each other, the point obtained by searching in the whole low magnification image is not necessarily suitable for joining the high magnification partial images. In some cases, even if it is attempted at a point obtained by searching, the joining cannot be performed, resulting in a time loss or re-joining from the search process for the entire image, or inability to obtain a desired image. Can be.

In addition, when performing the said joining, it is important to set the "overlapping rate" of the pasting (overlapping) part at the time of joining partial images, ie, a partial image. The larger the overlap rate is set, the longer the processing time is, because more partial images have to be captured. On the other hand, the smaller the overlap rate is set, the smaller the area used for joining in the partial image, and the more difficult it is to join. Since the optimum value of the overlap rate differs according to the object (pattern in the case of inspection of a board | substrate etc.), such as an inspection, there exists a subject which requires a skill for a user to adjust by visual observation.

This invention is made | formed in view of the above-mentioned problem, and an object of this invention is to provide the technique which can obtain a desired composite image, without adjusting the overlapping ratio by an entire image and a user.

In order to solve the above-mentioned problems, the image generating method according to the present invention obtains a first partial image obtained by capturing an image to be observed under a microscope with a predetermined resolution, and based on feature points extracted from the first partial image, The overlapping ratio of the second partial image to be synthesized with the first partial image is determined, and the second partial image obtained by moving the stage of the microscope based on the determined overlapping ratio is synthesized. It is set as the structure which acquires the synthesized image obtained.

According to the present invention, a desired composite image can be obtained while adjusting the overlapping ratio by the entire image and the user is unnecessary.

1 is a block diagram of a microscope system according to an embodiment.
2 is a functional block diagram of an image processing unit according to an embodiment.
3 is a diagram illustrating an example of an imaging object.
4 is a first diagram for explaining a method of bonding images using feature points in the image.
FIG. 5 is a second diagram for explaining a method of bonding images using feature points in the image. FIG.
6 is a first diagram illustrating a method of determining the overlap rate of a junction.
7A is a second diagram illustrating a method of determining the overlap rate of a junction.
FIG. 7B is a third diagram illustrating a method of determining the overlap rate of a junction. FIG.
8 is a diagram for explaining a method of moving a stage.
9 is a view for explaining a method of superimposing images.
10 is a flowchart showing a process of generating a high precision image by the microscope system according to the embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a configuration diagram of a microscope system according to the present embodiment. The microscope system 100 shown in FIG. 1 includes a microscope 50, a stage control unit 6, a microscope Z-axis control unit 8, an image processing unit 5, and a system control unit 7. And the stage 4, the barrel 1, the objective lens 2, and the camera 3, and an imaging object such as a substrate is observed at a predetermined magnification.

The stage 4 has a biaxial movement mechanism which drives in the X-axis direction and the Y-axis direction (left-right direction and depth direction in FIG. 1), and moves relatively with respect to an imaging object. The barrel 1 is located above the imaging object mounted on the stage 4, and the objective lens 2 is attached to the stage 4 side. The barrel 1 is driven in the Z axis direction (up and down direction in FIG. 1) by a vertical drive mechanism not shown.

On the other end side with respect to the objective lens 2 of the barrel 1, the camera 3 is provided. The camera 3 is a CCD (Charge Coupled Device) camera, which picks up an image to be mounted on the stage 4 at a predetermined resolution, outputs the acquired video signal or image signal to the image processing unit 5. send. The CCD camera outputs, for example, grayscale (luminance) data for each pixel corresponding to RGB as image information.

The system control part 7 controls each of the stage control part 6, the microscope Z-axis movement control part 8, and the image processing part 5 as needed. In addition, inputs such as an image size, an image generation range (lower limit value), pixel resolution, and the like are received, and processing such as determination of the necessity of combining a plurality of images is performed.

The stage control part 6 performs the movement control with respect to the biaxial direction of the X-axis and the Y-axis of the stage 4 according to the instruction | command from the system control part 7, and the relative position of the objective lens 2 and the imaging object Adjust it.

The microscope Z-axis movement control unit 8 controls the vertical drive mechanism in accordance with an instruction from the system control unit 7 to move the barrel 1 up and down (Z-axis direction), and is mounted on the stage 4. The focus adjustment is performed on the imaging object.

The image processing unit 5 executes various image processings on the video signal and the image signal transmitted from the camera 3.

The structure of the image processing part 5 among the microscope system 100 shown in FIG. 1 is demonstrated in detail with reference to FIG.

2 is a functional block diagram of the image processing unit 5 according to the present embodiment. The image processing unit 5 illustrated in FIG. 2 includes an imaging control unit 9, a shading, distortion, and luminance correction processing unit (hereinafter abbreviated as "correction processing unit") 10, an image data storage buffer unit 11, and feature points An extraction processing unit 12, an image generation processing unit 13, and a movement amount determination unit 14 are included.

The imaging control unit 9 performs the focus adjustment by the microscope Z-axis movement control unit 8 under the control of the system control unit 7, and performs image information captured by the camera 3 through the objective lens 2. Output to the correction processing unit 10.

The correction processing unit 10 corrects shading, distortion, unevenness in luminance, etc. by the imaging system included in the image information input from the imaging control unit 9, and stores the image data storage buffer 11 in the image data storage buffer 11.

The image data storage buffer unit 11 memorizes the captured image and the intermediate image generated in the process until generating the finally obtained image or the like. The intermediate image refers to an image which has been synthesized in the middle of the image generation process.

The feature point extraction processing unit 12 reads the intermediate image and the image synthesized from the image data storage buffer unit 11 into the intermediate image, and extracts the feature point from the read image. The image synthesized into the intermediate image constitutes a part of the image to be finally acquired, and hereinafter referred to as "partial image". And as a feature point extraction algorithm, well-known techniques, such as a Harris operator and an SOJKA operator, are used, for example.

The movement amount determination unit 14 reads the partial image from the image data storage buffer unit 11 and determines the movement amount of the stage 4 based on the information about the feature points obtained by the feature point extraction processing unit 12. Then, the determined movement amount of the stage 4 is output to the system control unit 7. The system control unit 7 controls the movement in the X-axis direction and the Y-axis direction of the stage 4 by using the movement amount input from the movement amount determining unit 14, and then moves the partial image to a location to acquire a partial image. The stage control section 6 is instructed.

The image generation processing unit 13 performs the inter-images on the basis of the information on the feature points obtained by the feature point extraction processing unit 12 for each image (the image of the region overlapping with the partial image among the partial image and the intermediate image). Detect the best corresponding point in, and create the transformation matrix. Then, using the created transformation matrix, the composition processing between the images is executed. As an algorithm for creating a transformation matrix, for example, a known technique such as a RANSAC (RANdom Sample Consensus) algorithm is used.

When it is necessary to synthesize a plurality of partial images in order to output a necessary image, the image processing unit 5 shown in FIG. 2 performs the synthesis of partial images using the feature points extracted by the feature point extraction processing unit 12. . When acquiring the partial image to be synthesized into the intermediate image, based on the extraction situation of the feature points, it is determined how much of the partial image to overlap with the intermediate image (duplicate rate), and the stage 4 is made to correspond to the determined overlap rate. It moves and acquires a partial image. The obtained partial image is synthesized with the intermediate image according to the determined overlap ratio to obtain an image to be finally obtained.

In the following description, the process of synthesizing an image using the feature point is referred to as a "bonding" process.

3 is a diagram illustrating an example of an imaging object. Hereinafter, the method of generating a bonded image in the case of acquiring a high-definition image using the FPD board | substrate provided with the circuit pattern shown in FIG. 3 as an imaging object is demonstrated.

The microscope system 100 shown in FIG. 1 determines the necessity of the image bonding process from the range of the visual field size and the high precision image of the camera 3, when generating the high precision image with respect to the imaging object shown in FIG.

Specifically, when the range of the visual field size V and the high-precision image is set to P, it is determined that the bonding process is unnecessary when the conditional expression of P = V / 2 is satisfied. On the other hand, when the conditional expression of P> V / 2 is satisfied, it is determined that the bonding process is necessary, and the high precision image generation method according to the present embodiment is executed.

And about the visual field size V and the range P of a high-precision image, it consists of X component and Y component all, and judges the necessity and unnecessary of a joining process using the above-mentioned conditional expression for every component. The visual field size and the range of the high-precision image may be set so as to have a margin with respect to the actual value by considering a moving error or the like.

The high-precision image generation processing in the case where it is determined that the bonding is necessary will be described in detail below.

4 and 5 are diagrams for explaining a method of bonding images using feature points in the image. In the board | substrate shown in FIG. 4, two area | regions RA and RB represent the microscope visual field range, respectively, and the image acquired with respect to area | region RA and RB is partial image A and B shown in FIG. Here, for the partial image A, the bonding process has already been performed to become a part of the intermediate image, and the case where the partial image B is joined is described. In this case, the overlap ratio between the partial image A and the partial image B is 50%.

As shown in Fig. 5, when the overlap ratio is 50%, the overlapping portions of the two partial images A and B become areas of the upper half of the partial image A and the lower half of the partial image B. From the partial image A, four feature points of P1a to P4a are extracted, and from the partial image B, four feature points of P1b to P4b are extracted. The image generation processing unit 13 searches for a correspondence relationship between the feature points extracted by the feature point extraction processing unit 12, and generates a transformation matrix that satisfies the relationship of P1a = P1b, P2a = P2b, P3a = P3b, and P4a = P4b, respectively. Obtain Using the obtained transformation matrix, the partial image B is joined with the partial image A (the intermediate image containing), and an intermediate image M is obtained.

In FIG. 4 and FIG. 5, the case where the overlap ratio is 50% is illustrated, but the microscope system 100 which concerns on a present Example is a partial image according to the extraction situation of the feature point in the feature point extraction processing part 12. FIG. The overlap rate at the time of joining A and partial image B is determined. Next, a method of determining the overlap rate will be described.

6, 7A, and 7B are diagrams for explaining a method for determining the overlap ratio of a junction. The overlapping rate is determined for the partial image A in FIG. 6, and the field of view of the camera 3 is shifted in the direction of the arrow in FIG. 6 based on the determined overlapping rate, and the partial image acquired at the position after the movement is joined to the partial image A. An example is demonstrated.

As shown in Fig. 7A, when the overlap ratio is set to 20%, the feature points exist only in the region RO (20-100% region from the top of the image, hatched regions in Fig. 7A) which are not subject to feature point extraction. In the region of 20% from above, no feature point is detected. In this case, it is judged that the value of the overlap rate is not set appropriately, changes to a larger value, and re-extracts the feature point.

As shown in FIG. 7B. When the overlap rate is set to 50%, four feature points P can be detected from the region of interest for feature point extraction. In this case, it is judged that joining is possible if the overlap ratio is 50%. It is preferable to set it as the structure which a user can set suitably about the number of the characteristic points sufficient for the bonding process.

As described above, the overlap ratio is determined in accordance with the extraction situation of the feature points from the partial image, and the amount of movement of the stage 4 is determined in accordance with the overlap ratio. When the stage 4 is moved in accordance with the determined movement amount, and the viewing direction of the camera 3 is moved, the partial image is picked up at the position after the movement and bonding is performed to the intermediate image.

8 is a diagram illustrating a method of moving the stage 4. The case where the high-precision image is needed about the range R with respect to the board | substrate pattern of an imaging object is illustrated.

In the embodiment, the stage 4 is moved in the order shown in Fig. 8, and P1, P2,... , The partial image is acquired in the order of P6. When the horizontal direction in the drawing is the X-axis direction (row direction) and the vertical direction is the Y-axis direction (column direction), as shown in FIG. 8, the intermediate image is moved by alternately moving the stage 4 in the row direction and the column direction. And the creation of a transformation matrix for concatenation with the newly acquired partial image can be simplified, and a complicated process for image consolidation can be avoided.

According to the above method, the feature points are extracted from each of the two images, and the two images are bonded based on the feature points. However, depending on the pattern of the substrate, for example, the number of feature points necessary for the bonding is extracted from the partial images. It can happen if you can't. Even in such a case, the following pattern matching may be adopted in order to enable the generation of a high-precision image.

9 is a view for explaining a method of superimposing images. As described above, the method of obtaining a synthesized image using a feature point is defined as "joining" of an image, and the method of obtaining a synthesized image using pattern matching as described below is referred to as "overlapping of images." It is defined as ".

Regarding the image superimposition processing, the image generation processing unit 13 uses a pattern matching algorithm such as shape search. For example, when the feature points cannot be extracted as in the partial images A and B shown in FIG. 9, the edges are searched from the partial images A and B by the pattern matching algorithm. And the search area is determined by the overlap rate or initial value previously used for image | video composition similarly to the image bonding process.

In the example shown in FIG. 9, as a result of searching in the image horizontal direction, edge E1a and edge E2a are extracted from partial image A, and edge E1b and edge E2b are extracted from partial image B. As shown in FIG. In the horizontal direction, images are superimposed using these extracted edges.

On the other hand, in the image vertical direction, the edge is not extracted. However, even in such a case, the partial image B is an image acquired by moving the stage 4 in the vertical direction by the overlap ratio from the partial image A. Using this, in the vertical direction, the movement amount of the stage 4 is regarded as the movement amount of the image, and the vertical direction is superposed between the partial image A and the partial image B. FIG. Thereby, the intermediate image M shown in FIG. 9 is obtained. The edge of the part which overlapped the edge E1a and the edge E1b corresponds to E2, and the edge of the part which overlapped E1, the edge E2a, and the edge E2b.

9 shows an example of the case where the edge in the horizontal direction can be extracted, but in addition to this, the case where only the edge in the vertical direction is extracted or the case in which the edge cannot be extracted in either the horizontal direction or the vertical direction can be considered. Can be. In these cases, the superposition can be performed by judging that the desired edge, i.e., the direction in which the position for matching cannot be obtained, may be superimposed at any position, and the movement amount of the stage 4 is regarded as the movement amount of the image.

By performing "overlapping" of the image by the method shown in FIG. 9, even if the feature point suitable for joining is not extracted from an image, execution of the high precision image generation process which concerns on this embodiment can be continued.

10 is a flowchart showing a process of generating a high precision image by the microscope system 100 according to the present embodiment. The process shown in FIG. 10 is started based on the input of the instruction which starts the high precision image generation process via the input device which is not shown in FIG.

First, in step S1, the system control part 7 sets the range of the high definition image required based on the parameter value input by the user through the said input device. And in FIG. 10, the process in the case where it is determined that a bonding process is needed based on the visual field size of the camera 3 and the range of the high precision image set in step S1 is shown after step S2.

In step S2, by the command of the system control unit 7, the imaging control unit 9 of the image processing unit 5 switches the objective lens 2 to the designated magnification, and the microscope Z-axis control unit 8 transmits the barrel ( When the focusing operation is performed by moving 1) up and down, the imaging control unit 9 captures a partial image with the camera 3. In FIG. 10, the stage 4 is exemplified as a process in the case where the high-precision image generation process is started in the state of moving to the imaging start position in advance, but at the time when the generation process of the high-definition image is started, the stage 4 May be a configuration at any position. In this case, in step S2, the stage control part 6 moves the stage 4 to the imaging start position determined according to the range of a high definition image.

The correction processing unit 10 corrects shading, distortion, luminance, and the like to the input image, and stores the image data obtained by the correction in the image data storage buffer unit 11 as one partial image. When the acquired partial image is stored, the system control unit 7 outputs a command to start joining the stored partial image with the joining direction as a parameter.

In step S3, the feature point extraction unit 12 which received the command from the system control unit 7 acquires the partial image stored in the image data buffer unit 11 in step S2, and the area of the predetermined range in the partial image. The feature point is extracted from the image. And the initial value of the predetermined range which extracts a feature point is determined based on the reference redundancy rate set previously.

In step S4, it is determined whether the number of extracted feature points is a predetermined number or more. If the number of feature points is more than a predetermined number, the flow advances to step S5 to start the "bonding" process of the image.

In step S5, the overlapping rate of the junction is determined. The method of determining the overlap rate is as described above with reference to FIGS. 6, 7A, and 7B. In step S6, the stage moving part 6 moves the stage 4 to a predetermined position from the overlap ratio determined in step S5 and the joining direction input to the movement amount determining part 14 in said step S2. .

In step S7, the imaging control unit 9 acquires the partial image at the position of the stage 4 after the movement, and stores it in the image data storage buffer unit 11. In step S8, the image generating processing unit 13 reads the partial image and the intermediate image acquired in step S7 from the image data storage buffer unit 11, performs the joining, and proceeds to step S9. The bonding method of step S8 is as described with reference to FIGS. 4 and 5. And in the initial value, the image joined with the partial image acquired in step S7 becomes a partial image acquired in step S2 instead of an intermediate image.

On the other hand, if it is determined in step S4 that the number of feature points has not reached the predetermined number, the flow advances to step S10 to start the "overlapping" process of the image.

In step S10, the stage control unit 6 moves the stage 4 by a predetermined amount calculated using the overlap ratio prepared in advance, and in step S11, the imaging control unit 9 moves the image after the movement. The partial image is acquired at the position of the stage 4 and stored in the image data storage buffer unit 11. In step S12, the image generation processing unit 13 reads the partial image and the intermediate image acquired in step S11 from the image data storage buffer unit 11, superimposes, and proceeds to step S9. The overlapping method is as described with reference to FIG. 9. In an initial value, the image which overlaps with the partial image acquired in step S11 becomes a partial image acquired in step S2 instead of an intermediate image.

In step S9, the range of the intermediate image generated in step S8 or step S12 determines whether or not the range of the high precision image set in step S1 is satisfied. If it is determined that the set high definition image is not satisfied, the process returns to step S3 and the same processing is repeated. In step S9, if it is determined that the range of the generated intermediate image satisfies the range of the high definition image set in step S1, the intermediate image is output as the high definition image to be finally generated, and the process ends.

As described above, according to the high-precision image generation method according to the present embodiment, since the partial images are synthesized using the feature points of the partial images, the entire image with low magnification becomes unnecessary. In addition, since the overlap rate is determined using the number of extracted feature points, it is unnecessary to adjust the overlap rate by the user.

In addition, although the case where the high precision image is acquired about the imaging object as an example was demonstrated as above, it is not limited to this. Any method of generating one image by synthesizing a plurality of images using feature points is included in the present invention.

1: barrel 2: objective lens
3: camera 4: stage
5: image processing unit 6: stage control unit
7: System control unit 8: Microscope Z-axis control unit
9: imaging controller
10: shading, distortion, luminance correction processing unit (correction processing unit)
11: Image data storage buffer unit 12: Feature point extraction processing unit
13: Image generation processing unit 14: Movement amount determining unit
50: microscope 100: microscope system

Claims (9)

Obtaining the 1st partial image which image | photographed the imaging object observed with a microscope with predetermined | prescribed resolution,
Based on the feature points extracted from the first partial image, the overlapping ratio with the second partial image synthesized with the first partial image is determined,
Acquiring a composite image obtained by combining the first partial image and the second partial image obtained by moving the stage of the microscope based on the determined overlap ratio.
An image generation method comprising a process. The method of claim 1,
And the feature point is extracted based on a preset reference overlap ratio. The method of claim 2,
And if the feature point is not extracted based on the reference overlap rate, re-extracting the feature point by changing the reference overlap rate. The method of claim 3,
When the re-extraction is performed, the reference overlap ratio is changed to a larger value. The method of claim 1,
It is determined whether the number of feature points extracted from the first partial image is equal to or larger than a predetermined number, and when it is determined that the predetermined number has not been reached, the overlapping rate prepared in advance is determined as the overlapping ratio with the second partial image. Image generation method to do. The method of claim 5,
An image generating method for searching for edges of the first partial image and the second partial image and acquiring a composite image using the extracted edges, based on the overlap ratio prepared in advance. The method of claim 1,
It is determined whether the number of feature points extracted from the first partial image is a predetermined number or more, and when it is determined that the predetermined number has not been reached, the amount of movement of the stage of the microscope is determined by the first partial image and the second partial image. The image generating method which acquires a composite image by considering it as the movement amount of. The method of claim 1,
Feature points are extracted with respect to the second partial image to determine whether or not the predetermined number or more of feature points are included in the second partial image, and according to the number of feature points included in the second partial image, the second The composite image obtained by synthesizing the first partial image and the two partial images using an overlap ratio determined based on a feature point of the partial image, or the overlap ratio prepared in advance, and based on the overlap ratio Synthesize the third partial image acquired by moving the stage of the microscope to generate a new composite image,
The above-mentioned process is repeated until the whole image of the said imaging object is acquired. A microscope system comprising a microscope for observing an imaging object at a predetermined magnification and a stage relatively moving with respect to the imaging object,
An imaging unit for acquiring a first partial image of the observation image of the microscope at the predetermined resolution;
A movement amount determining unit determining an overlapping ratio with the second partial image synthesized with the first partial image based on the feature points extracted from the first partial image, and determining a movement amount of the stage based on the overlapping ratio; And
An image generation unit for acquiring a composite image obtained by synthesizing the first partial image and the second partial image acquired by moving the stage according to the determined movement amount;
Including, the microscope system.

Images