WO2021140591A1

WO2021140591A1 - Image processing device, microscope system, and image processing method

Info

Publication number: WO2021140591A1
Application number: PCT/JP2020/000318
Authority: WO
Inventors: 渡辺　伸之; 洋子阿部
Original assignee: オリンパス株式会社
Priority date: 2020-01-08
Filing date: 2020-01-08
Publication date: 2021-07-15

Abstract

An image processing device (1) comprising: a sample image input unit (2) to which first and second images of a sample are inputted, the first image and the second image being microscope images captured respectively through an objective lens having a first magnification and an objective lens having a second magnification that is higher than the first magnification; a reference image input unit (3) to which a reference image corresponding to the first and second images is inputted, the reference image being an image of the sample in a wider range than the imaging range of the first and second images; a positioning unit (4) for estimating a first region in the reference image, the first region corresponding to the imaging range of the first image, by comparing the first image with the reference image; and a position acquisition unit (5) for acquiring the position of a second region in the reference image, the second region corresponding to the imaging range of the second image, on the basis of the estimated first region.

Description

Image processing equipment, microscope system and image processing method

The present invention relates to an image processing apparatus, a microscope system, and an image processing method.

Conventionally, a semiconductor inspection device for inspecting a semiconductor circuit board using a microscope is known (see, for example, Patent Document 1). The semiconductor inspection device of Patent Document 1 has a function of navigating a defective position of a semiconductor circuit board by using CAD data of the semiconductor circuit board, and displays a microscopic image of the semiconductor circuit board superimposed on the CAD data.
On the other hand, as an inspection of a semiconductor circuit board, a visual inspection is also performed in which an operator observes the semiconductor circuit board at a high magnification while manually operating a microscope. In the visual inspection, the operator manually moves the stage of the microscope to move the observation area of the semiconductor circuit board by the microscope, and observes the entire or wide area of the semiconductor circuit board.

JP-A-2007-115991

In order to superimpose a high-magnification microscope image on CAD data and display it, information on the position of the imaging range of the microscope image is required. When the position of the observation area of the microscope is controlled by controlling the electric stage as in Patent Document 1, the imaging range of the microscope image can be calculated from the position of the electric stage. On the other hand, when the operator manually moves the observation area of the sample, it is necessary to acquire the position of the imaging range of the microscope image by aligning the CAD data by image processing with the high-magnification microscope image.

In semiconductor circuit boards such as DRAM or imager, the same or similar regions are regularly arranged. Further, in the visual inspection, a narrow range of the sample is magnified by using a high-magnification objective lens. Therefore, in the case of a high-magnification microscope image containing only the same or similar regions, the alignment of the CAD data and the high-magnification microscope image may fail. Further, in the case of a sample such as a glass substrate, the contrast of a high-magnification microscope image becomes low. Even in the case of such a microscope image, the alignment of the CAD data and the high-magnification microscope image may fail. For these reasons, it is difficult to obtain an accurate position in the imaging range of a high-magnification microscope image.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing apparatus, a microscope system, and an image processing method capable of accurately acquiring the position of a shooting range of a high-magnification image. And.

In order to achieve the above object, the present invention provides the following means.
In one aspect of the present invention, a first image and a second image of a sample are input, and the first image and the second image are passed through objective lenses having a first magnification and a second magnification, respectively. The first magnification is lower than the second magnification, and the optical axis of the objective lens having the first magnification and the optical axis of the objective lens having the second magnification are A sample image input unit, which is the same as or substantially the same as each other, and a reference image corresponding to the first image and the second image are input, and the reference image is the shooting range of the first image and the reference image. By collating the reference image input unit, which is an image of the sample in a range wider than the photographing range of the second image, and the first image with the reference image, the first of the reference images Based on the alignment unit that estimates the first region corresponding to the imaging range of the image and the first region estimated by the alignment unit, the imaging range of the second image of the reference image is This is an image processing device including a position acquisition unit for acquiring the position of the second region corresponding to the above.

According to this aspect, the first image and the second image of the sample are input to the sample image input unit, and the reference image of the sample is input to the reference image input unit. Next, the alignment unit estimates a first region corresponding to the shooting range of the first image in the reference image.
Since the first image and the second image are microscope images taken through an objective lens having the same or substantially the same optical axis, the center position of the first region and the second image in the reference image The center positions of the second region corresponding to the imaging range coincide with or substantially coincide with each other. Therefore, the position acquisition unit can acquire the position of the second region in the reference image by referring to the estimated first region.

Here, since the shooting range of the low-magnification first image is wider than the shooting range of the high-magnification second image, the first image contains many feature points effective for alignment with respect to the reference image. Probability is high. Therefore, the first region can be estimated with high position accuracy. Then, the exact position of the second region in the reference image can be obtained based on the relationship that the center position of the first region and the center position of the second region coincide with each other or substantially match each other. ..

In the above aspect, the position acquisition unit acquires a position deviated from the center position of the first region by a predetermined distance as the center position of the second region, and the predetermined distance is the first magnification. The distance between the second magnification and the second magnification may be a distance corresponding to the amount of displacement of the optical axis of the objective lens.
For example, when the magnification of the objective lens is changed by the rotation of the revolver of the optical microscope or by the zoom function of the objective lens, the optical axis of the objective lens of the first magnification and the optical axis of the objective lens of the second magnification are changed. It can be slightly displaced from each other in the direction perpendicular to the optical axis. In this case, the amount of displacement of the optical axis is substantially constant, and the amount of deviation of the center position between the first region and the second region is also substantially constant. Therefore, the amount of deviation of the center position can be estimated in advance from the amount of displacement of the optical axis, and the position deviated by a predetermined distance from the center position of the first region can be acquired as the center position of the second region. ..

In the above aspect, even if the position acquisition unit estimates the second region by collating the second image with the first region and acquires the estimated position of the second region. Good.
According to this configuration, a more accurate position of the second region is obtained by collating the second image with the first region and searching for the region corresponding to the second image from near the center of the first region. Can be obtained. Further, since the search range is limited to the first region, the time and the amount of calculation required for estimating the second region can be reduced as compared with the case where the second image is collated with the reference image. it can.

In the above aspect, the position acquisition unit corresponds to the second image within a range of a predetermined distance from the center position of the first region in collation of the second image with the first region. A second region is searched, and the predetermined distance is a distance corresponding to a guaranteed range of the amount of displacement of the optical axis between the objective lens of the first magnification and the objective lens of the second magnification. May be good.
Normally, the amount of displacement of the center position between the shooting range of the first image and the shooting range of the second image is the amount of the optical axis between the objective lens of the first magnification and the objective lens of the second magnification. It falls within the guaranteed range of displacement. Therefore, by limiting the search area of the second image with respect to the first area when estimating the second area to the range corresponding to the guaranteed range, the second area is estimated efficiently and surely. be able to.

In the above aspect, the sample may be a semiconductor circuit board, and the reference image may be an image of a design drawing of the semiconductor circuit board.

Another aspect of the present invention is through an objective lens arranged to face the sample, a magnification changing portion for changing the magnification of the objective lens between a first magnification and a second magnification, and the objective lens. A microscope system including an imaging unit for imaging the sample, an optical microscope having the first magnification lower than the second magnification, and an image processing apparatus according to any one of the above.

In the above aspect, an image acquisition unit that acquires an image captured by the imaging unit from the imaging unit is provided, and the image acquisition unit interlocks with the operation of changing the magnification of the objective lens of the magnification changing unit. Even if the first image and the second image are selected from the images acquired from the imaging unit and the selected first image and the second image are input to the sample image input unit. Good.
According to this configuration, it is possible to automate the process of selecting the first image and the second image from the images captured by the imaging unit and inputting them to the sample image input unit.

In the above aspect, the optical microscope and the control unit for controlling the image acquisition unit are further provided, and the image processing apparatus evaluates whether or not the position of the second region is accurately acquired by the position acquisition unit. An evaluation unit is further provided, and when it is evaluated that the position of the second region is not accurately acquired by the evaluation unit, the control unit controls the optical microscope and the image acquisition unit. The first image may be imaged again and input to the sample image input unit, and the alignment unit may estimate the first region using the first image captured again.
According to this configuration, if the position of the second region is not accurately acquired due to a failure in estimating the first region or the like, it is evaluated that the position of the second region is not accurately acquired. The first image is imaged again, the first region is re-estimated, and the position of the second region is re-acquired based on the re-estimated first region. In this way, the evaluation and reacquisition of the position acquisition of the second region can be automated.

Another aspect of the present invention is an image processing method for processing a first image and a second image of a sample, wherein the first image and the second image have a first magnification and a second image. It is a microscope image taken through each of the objective lenses of the magnification, the first magnification is lower than the second magnification, and the optical axis of the objective lens of the first magnification and the second magnification The optical axes of the objective lenses are the same or substantially the same as each other, the first image and the second image are input, and the reference image corresponding to the first image and the second image is input. The reference image is an image of the sample having a range wider than the shooting range of the first image and the shooting range of the second image, and by collating the first image with the reference image, A first region corresponding to the shooting range of the first image of the reference image is estimated, and based on the estimated first region, the second image of the reference image is shot. This is an image processing method for acquiring the position of a second region corresponding to a range.

According to the present invention, there is an effect that the position of the shooting range of a high-magnification image can be accurately acquired.

It is an overall block diagram of the image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the reference image of a sample. It is a figure which shows an example of the 1st image of a sample. It is a figure which shows an example of the 2nd image of a sample. It is a figure explaining the method of determining the 1st magnification. It is a flowchart which shows the operation of the image processing apparatus of FIG. It is an overall block diagram of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the operation of the image processing apparatus of FIG. It is an overall block diagram of the microscope system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the operation of the image processing apparatus of FIG. It is an overall block diagram of the microscope system which concerns on 4th Embodiment of this invention. It is a flowchart which shows the operation of the image processing apparatus of FIG. It is a continuation of the flowchart of FIG. 10A.

(First Embodiment)
The image processing apparatus 1 and the image processing method according to the first embodiment of the present invention will be described with reference to the drawings.
The image processing apparatus 1 displays the microscopic images A and B (see FIGS. 2B and 2C) of the sample S (see FIG. 2A) in the observation of the sample S using an optical microscope, for example, in the visual inspection of the semiconductor circuit substrate S. It processes and presents and notifies the operator or the system of the position of the imaging range of the microscope image B.

As shown in FIG. 1, the image processing device 1 includes a sample image input unit 2, a reference image input unit 3, an alignment unit 4, and a position acquisition unit 5.
An example of an image processing device 1 is a computer including a processor and a storage unit having a RAM, a ROM, and any other storage device. An image processing program is stored in the storage unit. When the processor executes the process according to the image processing program, the functions described later in each

part

2, 3, 4, and 5 are realized.

The sample image input unit 2 is connected to, for example, a camera attached to an optical microscope, and the first image A and the second image B are input from the camera.
The camera captures an optical image of the sample S formed by the objective lens of the optical microscope, and acquires a first image A and a second image B which are digital images of the sample S.

2A to 2C show a reference image C (described later), a first image A, and a second image B, respectively. As shown in FIG. 2A, the sample S is a semiconductor circuit board such as a DRAM or an imager. A large number of identical or similar circuit patterns are regularly arranged on the semiconductor circuit board S. The first image A is an image captured through the objective lens of the first magnification, and the second image B is an image captured through the objective lens of the second magnification. The second magnification is a high magnification for visual inspection of the sample S, and the first magnification is lower than the second magnification. Therefore, the shooting range of the first image A is wider than the shooting range of the second image B. In FIG. 2A, the area R1 is the first area corresponding to the shooting range of the first image A, and the area R2 is the second area corresponding to the shooting range of the second image B.

When the sample S has a characteristic region, the first magnification and the photographing region R1 are set so that the first image A includes a characteristic region. FIG. 3 illustrates a method of determining the first magnification and the photographing region R1. In the example of FIG. 3, the sample S includes a repeating region P1 in which the same or similar circuit pattern is regularly repeated, and a characteristic region P2 located around the repeating region P1 and having a characteristic circuit pattern. The first magnification and the imaging region R1 are determined so that the imaging range of the first image A includes the boundary region P3 between the repeating region P1 and the feature region P2.

For example, an optical microscope includes a first objective lens having a first magnification, a second objective lens having a second magnification, and a revolver holding the first objective lens and the second objective lens. The rotation of the revolver causes the first objective lens and the second objective lens to be selectively arranged on the optical path. Alternatively, the optical microscope may include an objective lens having a zoom function capable of changing the magnification between the first magnification and the second magnification. Therefore, the optical axis of the objective lens of the first magnification and the optical axis of the objective lens of the second magnification are the same as or substantially the same as each other, and therefore, as shown in FIG. 2A, the first image A The center position of the photographing range of the second image B and the center position of the photographing range of the second image B are the same or substantially the same as each other.

The reference image input unit 3 is connected to, for example, an external device of the image processing device 1, and the reference image C corresponding to the first image A and the second image B is input from the external device. Alternatively, the image processing device 1 may include a memory for storing the reference image C, and the reference image C may be input from the memory to the reference image input unit 3.

As shown in FIG. 2A, the reference image C is an image of the sample S having a range wider than the shooting range of the first image A and the shooting range of the second image B. When the sample S is a semiconductor circuit board, the reference image C may be an image of the design drawing of the semiconductor circuit board, and the CAD data of the design drawing may be input to the reference image input unit 3. Alternatively, the reference image C may be a microscope image of the sample S taken through an objective lens having a magnification lower than the first magnification.

The alignment unit 4 receives the first image A and the reference image C from the

image input units

2 and 3, and aligns the first image A with respect to the reference image C. That is, the alignment unit 4 collates the first image A with the reference image C, and searches and finds a region having the highest degree of similarity with the first image A from within the reference image C. Then, the alignment unit 4 estimates that the region in which the first image A is aligned is the first region R1.

An existing alignment algorithm is used to align the first image A with respect to the reference image C.
For example, an edge image is created from the first image A by detecting the edge of the first image A. Edge detection can be done by classical filtering or deep learning edge estimation (eg, Ruohui Wang, Edge Detection Using Convolutional Neural Network, Advances in Neural Network-ISNN2016, 13th International Symposium on Neural Network, pp12-20, 2016). See.) Etc. are used. Next, the kernel of a predetermined size (m pixels × n pixels) of the edge image or the reference image C is moved and rotated, and SAD (Sum of Absolute Difference), which is the sum of the absolute values of the differences between the pixel values, is calculated. Align the edge image based on the minimum position. When it is necessary to enlarge or reduce the kernel, the size of the kernel is adjusted based on the size of the first image A obtained from the magnification of the optical system of the optical microscope and the actual size information of the reference image C.

The position acquisition unit 5 acquires the position of the second region R2 in the reference image C based on the first region R1 estimated by the alignment unit 4.
Specifically, the position acquisition unit 5 has the size of the first region R1 to the second region based on the size ratio between the shooting range of the first image A and the shooting range of the second image B. Calculate the size of R2. Further, the position acquisition unit 5 determines the center position of the first region R1 to the center position of the second region R2. As a result, the position acquisition unit 5 can obtain the position of the second region R2 in the reference image C.

The size ratio of the photographing range is calculated from, for example, information on the actual field of view of the objective lens of the first magnification and the actual field of view of the objective lens of the second magnification. The information regarding the actual field of view is, for example, numerical values of the first magnification and the second magnification, and is input from the optical microscope to the image processing device 1.
The position acquisition unit 5 outputs information on the position of the second image B and the second region R2. The output information on the positions of the second image B and the second region R2 is presented to the operator, for example, by being displayed on a display device connected to the image processing device 1.

Next, the operation of the image processing apparatus 1 corresponding to the image processing method according to the present embodiment will be described with reference to FIG.
The operator observes a wide range of the sample S at a low magnification through the objective lens of the first magnification, and determines the observation area of the sample S. Subsequently, the operator switches the magnification of the objective lens to the second magnification while maintaining the position of the sample S, and observes the observation region at a high magnification. The first image A and the second image B are acquired by the camera before and after the magnification is switched, and the first image A and the second image B are input from the camera to the image processing device 1 (step S1). Further, the reference image C is input to the image processing device 1 (step S2).

In the image processing device 1, the first image A is sent to the alignment unit 4 via the sample image input unit 2, and the second image B is sent to the position acquisition unit 4 via the sample image input unit 2. The reference image C is sent to the alignment unit 4 and the position acquisition unit 5 via the reference image input unit 3.

In the alignment unit 4, the first image A is aligned with the reference image C (step S3), and the first region R1 in the reference image C is estimated (step S4). Next, the position acquisition unit 5 acquires the position of the second region R2 in the reference image C with reference to the estimated first region R1 (step S5). Then, the estimation results of the positions of the second image B and the second region R2 are output from the image processing device 1 and presented to the operator (step S6).

For example, by collating the second region R in the reference image C with the second image B, the operator collates the design drawing which is the reference image C with the second image B which is the actual image of the sample S. It is possible to detect defects and the like by strictly comparing with.

As shown in FIGS. 2A-2C, in the case of sample S containing a large number of regularly arranged identical or similar regions, the high magnification second image B may include only identical or similar regions. When such a second image B is aligned with respect to the reference image C, misalignment is likely to occur. Further, in the case of the sample S such as glass, the contrast of the high-magnification second image B becomes low. Even when such a second image B is aligned with respect to the reference image C, misalignment is likely to occur.
On the other hand, the low-magnification first image A includes, in addition to the same or similar regions, characteristic regions that are effective for alignment with respect to the reference image C. Further, especially in a microscope, the contrast of the low-magnification first image A is often higher than that of the second image B because of the relationship between the optical resolution and the imager's resolution. In an image with high contrast, in the above-mentioned image alignment algorithm, the variation of the evaluation function with respect to the misalignment becomes large. Therefore, the first image A can be aligned with the reference image C with high positional accuracy.

Optical microscopes used for visual inspection use multiple objective lenses or zoom optics with different magnifications, with the optical axes aligned or substantially matched within the guaranteed accuracy of the optical microscope, at two magnifications. Images can be acquired. That is, the positional relationship of the shooting range between the low-magnification first image A and the high-magnification second image B is guaranteed. Further, the size relationship between the first region R1 and the second region R2 in the reference image C is defined by the actual field of view of the objective lens of the first magnification and the actual field of view of the objective lens of the second magnification. Will be done. By utilizing such a positional relationship and a size relationship, it is possible to obtain an accurate position of the shooting range of the second image B with a high magnification from the position of the first region R estimated with high accuracy.

In the present embodiment, the position acquisition unit 5 acquires the center position of the first region R1 as the center position of the second region R2, but instead, from the center position of the first region R1. A position deviated by a predetermined distance may be acquired as the center position of the second region R2.

The predetermined distance is a distance according to the amount of displacement between the optical axes of the objective lens having the first magnification and the objective lens having the second magnification. When the magnification of the objective lens is changed between the first magnification and the second magnification, the optical axis may be slightly displaced in the direction orthogonal to the optical axis. A deviation occurs between the center position of the first region R1 and the center position of the second region R2 by the distance corresponding to the displacement amount of the optical axis. Since the displacement amount and the displacement direction of the optical axis of the objective lens are substantially constant, the displacement amount and the displacement direction between the center position of the first region R1 and the center position of the second region R2 are substantially constant. It can be obtained in advance.

Therefore, a position deviated by a predetermined distance in a predetermined direction from the center position of the first region R1 based on the deviation amount and the deviation direction acquired in advance can be acquired as the center position of the second region R2. This makes it possible to obtain a more accurate position of the second region R2.
The predetermined distance may be fixed, but may be corrected each time. The amount of displacement of the optical axis of the objective lens when the magnification is changed may be different each time the magnification is changed. Therefore, for example, the displacement amount of the optical axis may be measured each time the magnification is changed, and a predetermined distance may be corrected based on the measured displacement amount.

(Second embodiment)
Next, the image processing apparatus 10 and the image processing method according to the second embodiment of the present invention will be described with reference to the drawings.
In the present embodiment, a configuration different from that of the first embodiment will be described, and the same reference numerals will be given to the configurations common to the first embodiment, and the description thereof will be omitted.
As shown in FIG. 5, the image processing device 10 includes a sample image input unit 2, a reference image input unit 3, a first alignment unit 41, and a second alignment unit 51. As shown in FIG. 6, the operation of the image processing device 10 corresponding to the image processing method according to the present embodiment is that the image processing device 1 of the first embodiment performs step S7 instead of step S5. It's different.

The first alignment unit 41 is the same as the alignment unit 4 of the first embodiment.
The second alignment unit 51 is a position acquisition unit that acquires the position of the second region R2 in the reference image C, but the second region is different from the position acquisition unit 5 of the first embodiment. Acquire the position of R2. The second alignment unit 51 aligns the second image B with respect to the first region R1 in the reference image C by using the same method as the alignment unit 4 (step S7). That is, the second alignment unit 51 collates the second image B with the first region R1 and searches for the region having the highest degree of similarity between the first region R1 and the second image B. And find out. Then, the second alignment unit 51 estimates that the region in which the second image B is aligned is the second region R2, and acquires the estimated position of the second region R2 (step S6). ).

According to the present embodiment, the search range of the second region R is within the first region R1 as compared with the case where the second image B is collated with the reference image C to estimate the second region R2. Be restricted. Therefore, the time required for the alignment of the second image B can be shortened. In addition, it is possible to prevent misalignment due to a repeating pattern of the same or similar regions in the sample S, and accurately estimate the second region R2.

In the present embodiment, the search range in the alignment of the second image B may be the entire first region R1, but is narrower than the first region R1 centered on the center of the first region R1. It may be a range.
For example, the second alignment unit 51 searches for the second region R2 within a predetermined distance from the center position of the first region R1 in collation with the first region R1 of the second image B. You may. The predetermined distance is a distance corresponding to the guaranteed range of the displacement amount of the optical axis of the objective lens 62 between the first magnification and the second magnification.

Normally, the amount of deviation of the center position between the shooting range of the first image A and the shooting range of the second image B is the displacement of the optical axis of the objective lens between the first magnification and the second magnification. It fits within the guaranteed amount. Therefore, by limiting the search area of the second image B in the alignment of the second region B to the range corresponding to the guaranteed range, the calculation time and the load required for the alignment can be suppressed, and the calculation time and the load required for the alignment can be suppressed, which is efficient and reliable. In addition, the second region R2 can be estimated.

(Third Embodiment)
Next, the microscope system 100 and the image processing method according to the third embodiment of the present invention will be described with reference to the drawings.
In this embodiment, configurations different from those of the first and second embodiments will be described, and configurations common to the first and second embodiments will be designated by the same reference numerals and description thereof will be omitted.
As shown in FIG. 7, the microscope system 100 is connected to the optical microscope 60, the image processing device 20, the optical microscope 60, and the image processing device 20, and controls the optical microscope 60 and the image processing device 20. A control unit) 80 is provided. As shown in FIG. 8, the operation of the image processing apparatus 20 corresponding to the image processing method according to the present embodiment is that the image processing apparatus of the first embodiment performs steps S8 to S10 instead of step S1. Different from 1.

The optical microscope 60 is a magnification changing unit that changes the magnification of the stage 61, the objective lens 62 arranged to face the sample on the stage 61, and the objective lens 62 between the first magnification and the second magnification. A 63 and a digital camera (imaging unit) 64 that captures the sample S through the objective lens 62 are provided.
The optical microscope 60 notifies the main controller 80 of the timing of changing the magnification by the magnification changing unit 63.

In the example of FIG. 7, the magnification changing unit 63 is a revolver that holds the objective lens 62 of the first magnification and the objective lens 62 of the second magnification, and by rotating the revolver 63, it faces the sample and observes the sample. The magnification of the objective lens 62 used is changed.
When the objective lens 62 is a zoom optical system having a zoom function, the magnification changing unit 63 may be the objective lens 62. In this case, a mechanism for detecting the state of the objective lens 62 may be provided in order to obtain information on the magnification.

The image processing device 20 includes an image acquisition unit 6 in addition to the sample image input unit 2, the reference image input unit 3, the alignment unit 4, and the position acquisition unit 5. The image processing device 20 may include a first alignment unit 41 and a second alignment unit 51 in place of the alignment unit 4 and the position acquisition unit 5.

As shown in FIG. 8, the image acquisition unit 6 acquires an image from the camera 64 (step S8), and in conjunction with the operation of changing the magnification of the objective lens 62 of the magnification changing unit 63, the first image is taken from the image. Image A and the second image B are selected (step S9), and the selected first image A and the second image B are transferred to the sample image input unit 2 and input (step S10).
For example, the image acquisition unit 6 has a FIFO (first in, first out) memory 6a. The camera 64 captures the sample S at regular time intervals and transmits the acquired image to the FIFO memory 6a. The FIFO memory 6a holds the latest plurality of images for a certain period of time.

The main controller 80 causes the image acquisition unit 6 to select the first image A and the second image B in response to the notification from the optical microscope 60.
For example, the operator observes a wide range of the sample S at the first magnification to determine the observation area, switches from the first magnification to the second magnification, and observes the observation area at a high magnification. In this case, based on the notification that the first magnification is changed to the second magnification at time T, the image acquisition unit 6 selects the time immediately before time T from the images stored in the FIFA memory 6a. The image acquired at T-τ1 is selected as the first image A, and the image acquired at time T + τ2 immediately after the time T is selected as the second image B. In this way, a pair of images A and B whose centers of the photographing range coincide with each other or substantially coincide with each other can be specified and selected based on the timing of changing the magnification by the magnification changing unit 63.

The images A and B selected do not necessarily have to be the images immediately before and after the time T, and are within the period in which the objective lens 62 of the first magnification and the objective lens 62 of the second magnification are used, respectively. Any acquired image may be used.

As described above, according to the present embodiment, the pair of first images required for the processing of the image processing device 1 from the images acquired by the camera 64 in conjunction with the change of the magnification by the magnification changing unit 63. A and the second image B are automatically selected, and a pair of the first image A and the second image B are automatically input to the sample image input unit 2. This eliminates the need for the operator to select the first image A and the second image B, and can improve the efficiency of observing the sample S.

In the present embodiment, the image processing device 20 does not necessarily have to include the image acquisition unit 6.
For example, the sample image input unit is caused by an operator operating the camera 64 to execute acquisition of the first image A and the second image B immediately before and immediately after switching from the first magnification to the second magnification. Images A and B may be input to 2. Alternatively, the operator selects a pair of the first image A and the second image B from the images acquired by the camera 64, and the selected images A and B are input to the sample image input unit 2. It may be configured in.

(Fourth Embodiment)
Next, the microscope system 200 and the image processing method according to the fourth embodiment of the present invention will be described with reference to the drawings.
In this embodiment, configurations different from those of the first to third embodiments will be described, and configurations common to the first to third embodiments will be designated by the same reference numerals and description thereof will be omitted.
The microscope system 200 evaluates whether or not the position of the second region R2 has been accurately acquired by the position acquisition unit 51, and if it is not accurate, the first image A is automatically reacquired. It is different from the third embodiment.

As shown in FIG. 9, the microscope system 200 includes an optical microscope 60, an image processing device 30, and a main controller (control unit) 80 connected to the optical microscope 60 and the image processing device 30. As shown in FIGS. 10A and 10B, the operation of the image processing apparatus 30 corresponding to the image processing method according to the present embodiment is that steps S11 to S16 are added after the alignment of the second image in step S7. Is different from the third embodiment.

The image processing device 30 includes an evaluation unit 7 in addition to the sample image input unit 2, the reference image input unit 3, the first alignment unit 41, the second alignment unit 51, and the image acquisition unit 6.
The second image B is not properly aligned with respect to the first region R1 due to a failure of the alignment of the first image A with respect to the reference image C, and as a result, the position of the second region R2 is changed. It may not be acquired accurately. As shown in FIG. 10A, the evaluation unit 7 evaluates whether or not the position of the second region R2 is accurately acquired by the second alignment unit 51 (step S11). Specifically, the evaluation unit 7 has a second with respect to the first region R1 based on the degree of similarity between the second image B and the second region R2 estimated by the second alignment unit 51. It is evaluated whether or not the alignment of the image B of 2 is performed accurately.

For example, the evaluation unit 7 calculates the SAD between the second image B and the second region R2.
When the SAD is less than a predetermined value, the evaluation unit 7 evaluates that the alignment of the second image B with respect to the first region R1 has been performed accurately (YES in step S11). In this case, as shown in FIG. 10A, the evaluation unit 7 notifies the second alignment unit 51 of the accurate evaluation result, and the second alignment unit 51 provides information on the position of the second region R2. Output (step S6).
On the other hand, when the SAD is equal to or higher than a predetermined value, the evaluation unit 7 evaluates that the alignment of the second image B with respect to the first region R1 has not been performed accurately (NO in step S11). In this case, as shown in FIG. 10B, the evaluation unit 7 notifies the main controller 80 of the inaccurate evaluation result (step S12), and the second image B is held in the memory 8 (step S13). ..

The main controller 80 responds to the notification of the inaccurate evaluation result and controls the magnification changing unit 63 and the camera 64 of the optical microscope 60 to re-image the first image A and input it to the sample image input unit 2. Let me. That is, the main controller 80 transmits a magnification change signal for changing the magnification of the objective lens 62 to the first magnification to the optical microscope 60. The magnification changing unit 63 changes the magnification of the objective lens 62 to the first magnification in response to the magnification change signal, and then the camera 64 re-images the first image A. The image acquisition unit 6 selects the first image A in conjunction with the change to the first magnification and inputs it to the sample image input unit 2 (steps S14 to S16).

The first alignment unit 41 re-estimates the first region R1 using the re-imaged first image A (steps S3 and S4), and the second alignment unit 51 is re-estimated. Using the first region R1 and the second image B held in the memory 8, the position of the second region R2 is acquired again (step S7).
As described above, according to the present embodiment, the estimation accuracy of the second region R2 is evaluated, and when the estimation accuracy is low, the image imaging, the estimation of the first region, and the estimation of the second region are re-executed. Will be done. As a result, the exact position of the second region R2 can be reliably obtained.

In the present embodiment, after it is evaluated that the position of the second region R2 is not accurately acquired, the optical microscope 60 automatically switches the magnification of the objective lens 62, but instead of this, the work The person may manually switch the magnification of the objective lens 62 based on, for example, an inaccurate evaluation result displayed on the display device.

In the present embodiment, only the first image A is imaged again, but instead, both the first image A and the second image B may be imaged again. In this case, the second region R2 is re-estimated using the pair of recaptured first images A and second images B.
In the present embodiment, the image processing device 30 is provided with the first alignment unit 41 and the second alignment unit 51, but instead of this, it is provided with the alignment unit 4 and the position acquisition unit 5. You may.

1,10,20,30 Image processing device 2 Sample image input unit 3 Reference image input unit 4 Alignment unit 5 Position acquisition unit 41 First alignment unit (alignment unit)
51 Second alignment section (position acquisition section)
6 Image acquisition unit 7 Evaluation unit 60 Optical microscope 64 Camera (imaging unit)
80 Main controller (control unit)
100,200 Microscope system A 1st image B 2nd image C Reference image R1 1st region R2 2nd region S Sample

Claims

A first image and a second image of the sample are input, and the first image and the second image are microscopic images taken through objective lenses of the first magnification and the second magnification, respectively. Yes, the first magnification is lower than the second magnification, and the optical axis of the objective lens of the first magnification and the optical axis of the objective lens of the second magnification are the same or substantially the same as each other. There is a sample image input part,
A reference image corresponding to the first image and the second image is input, and the reference image is a range of the sample that is wider than the shooting range of the first image and the shooting range of the second image. The reference image input section, which is an image,
An alignment unit that estimates a first region of the reference image corresponding to the shooting range of the first image by collating the first image with the reference image.
A position acquisition unit that acquires the position of a second region corresponding to the shooting range of the second image of the reference image based on the first region estimated by the alignment unit is provided. Image processing device.
The position acquisition unit acquires a position deviated from the center position of the first region by a predetermined distance as the center position of the second region, and the predetermined distance is the first magnification and the second. The image processing apparatus according to claim 1, which is a distance according to the amount of displacement of the optical axis of the objective lens with respect to the magnification of.
The first aspect of the present invention, wherein the position acquisition unit estimates the second region by collating the second image with the first region, and acquires the estimated position of the second region. Image processing equipment.
In collation of the second image with the first region, the position acquisition unit corresponds to the second image within a range of a predetermined distance from the center position of the first region. The predetermined distance is a distance corresponding to a guaranteed range of the amount of displacement of the optical axis between the objective lens having the first magnification and the objective lens having the second magnification. The image processing apparatus according to 3.
The sample is a semiconductor circuit board.
The image processing apparatus according to any one of claims 1 to 4, wherein the reference image is an image of a design drawing of the semiconductor circuit board.
An objective lens arranged so as to face the sample, a magnification changing unit that changes the magnification of the objective lens between the first magnification and the second magnification, and an imaging unit that images the sample through the objective lens. With an optical microscope, the first magnification of which is lower than the second magnification.
A microscope system comprising the image processing apparatus according to any one of claims 1 to 5.
An image acquisition unit for acquiring an image captured by the image pickup unit from the image pickup unit is provided.
The image acquisition unit
In conjunction with the operation of changing the magnification of the objective lens of the magnification changing unit, the first image and the second image are selected from the images acquired from the imaging unit.
The microscope system according to claim 6, wherein the selected first image and the second image are input to the sample image input unit.
A control unit for controlling the optical microscope and the image acquisition unit is further provided.
The image processing apparatus further includes an evaluation unit that evaluates whether or not the position of the second region has been accurately acquired by the position acquisition unit.
When it is evaluated that the position of the second region is not accurately acquired by the evaluation unit, the control unit controls the optical microscope and the image acquisition unit to re-image the first image. An image is taken and input to the sample image input unit.
The microscope system according to claim 7, wherein the alignment unit estimates the first region using the first image captured again.
An image processing method for processing a first image and a second image of a sample, wherein the first image and the second image are passed through objective lenses of a first magnification and a second magnification, respectively. It is a microscopic image taken, the first magnification is lower than the second magnification, and the optical axis of the objective lens of the first magnification and the optical axis of the objective lens of the second magnification are mutual. Is the same or almost the same as
The first image and the second image are input,
A reference image corresponding to the first image and the second image is input, and the reference image is a range of the sample that is wider than the shooting range of the first image and the shooting range of the second image. It is an image
By collating the first image with the reference image, a first region of the reference image corresponding to the shooting range of the first image is estimated.
An image processing method for acquiring the position of a second region of the reference image corresponding to the shooting range of the second image based on the estimated first region.