JP3956584B2 - Omnifocal image composition method and apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a charged particle beam apparatus, and more particularly to an omnifocal image synthesis method and apparatus for synthesizing an image that is entirely in focus from a plurality of images with different in-focus portions.
[0002]
[Prior art]
As a technique for observing a three-dimensional structure with respect to an electron microscope image, there are JP-A-5-128989, JP-A-5-299048, JP-A-9-92195, and the like.
[0003]
In addition, the prior art of the omnifocal image composition method for synthesizing an image that is entirely in focus from a plurality of images having different in-focus portions is a technique for optically photographed images. -333271, JP-A-9-26312, JP-A-6-113183, JP-A-2000-39566, JP-A-5-227460, JP-A-10-290389, JP-A-11-261797, JP-A-11- 283035, JP-A-11-287618, and the like.
[0004]
As a conventional technique for evaluating the amount of noise, John Immerkaer, “Fast Noise
Variance Estimation ", Computer Vision and Image Understanding, Vol.64, No.2, 1996, p.300-302.
[0005]
[Problems to be solved by the invention]
Along with the increase in the number of semiconductor wafers, the sample has a three-dimensional extent, and the depth of focus has become shallow due to the improved resolution of the electron microscope, and the need for observing the three-dimensional structure is increasing. Conventional techniques for observing a three-dimensional structure with respect to an image obtained with an electron microscope are those that observe a three-dimensional image (Japanese Patent Laid-Open No. 5-12989), and those that use contour lines and bird's-eye views (Japanese Patent Laid-Open No. 5-299048). Etc.) and not a simple method. The omnifocal image composition technology, which synthesizes an image that is entirely in focus from a plurality of images with different in-focus parts, is a simple method and there is a growing need to apply it to electron microscope images. Yes.
[0006]
However, since the image of the electron microscope is extremely noisy, it is essential to consider the noise. Since an optically photographed image has relatively little noise, conventional omnifocal image synthesis techniques for optical images do not consider noise. When an electron microscope image is synthesized by a conventional method, an artificial artifact is generated, resulting in an unnatural omnifocal image.
[0007]
An object of the present invention is to provide a natural omnifocal image synthesizing method and apparatus that do not look like an artificial synthesized image even for a noisy input image.
[0008]
[Means for Solving the Problems]
The present invention reduces noise in a plurality of input images read at different focal points, evaluates the noise amount of the noise-reduced image from which noise has been reduced, and evaluates the focus matching degree of the noise-reduced image to thereby change the amount of signal change. An evaluation value is calculated, a signal change amount evaluation maximum value and synthesis information are generated based on the calculated signal change amount evaluation value, and a noise influence level is determined from the signal change amount evaluation maximum value and the noise amount evaluation value. A plurality of omnifocal image synthesis means for generating goodness information, and a method for generating a composite image based on the plurality of goodness information generated from a plurality of omnifocal image synthesis means and the synthesis information, or A device using
[0009]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a diagram showing the concept of omnifocal composition according to the present invention. 110, 120, and 130 are a plurality of input images taken with different focal points, 140 is a signal change amount evaluation, 141, 142, and 143 are signal change amounts corresponding to 110, 120, and 130, respectively. A composite image 160 is a depth image. It is assumed that one corresponding pixel of 110, 120, and 130 is taken, 110 is blurred in the vicinity thereof, 120 is in focus, and 130 is blurred. In the blurred part, the signal in the vicinity does not change much, whereas in the focused part, the signal in the vicinity changes greatly. When the signal change amount is evaluated by 140, the signal change amount is 141, 142, 143, the signal change amount 142 of the focused image 120 is the largest value. Therefore, 120 pixels corresponding to 142 having the largest signal change amount are set as pixels at corresponding positions in the composite image 150. Further, the depth information for 120 images corresponding to 142 having the largest signal change amount is set as a pixel at a corresponding position of the depth image 160. Depth information is preferable if the focal length of the input image is expressed as a pixel value, so that the actual depth of the imaged object can be known, but for example, the input images 110, 120, and 130 are # 1, # 2, and # 3, respectively. A number may be assigned and the number may be used as depth information. In short, it is only necessary to distinguish which input image has been selected. When the above operation is performed on all the pixels, the synthesized image 150 is finally synthesized into an image that is entirely in focus.
[0010]
FIG. 2 shows a second embodiment of the present invention.
[0011]
The input image 200 is processed by the preprocessing unit 210 and then input to the omnifocal image synthesizing unit 220 and 230, and different omnifocal image synthesis is performed to generate synthesis information and goodness information 221 and 231. The The synthesizing unit 240 selects, for each pixel, a pixel suitable for the purpose of synthesis from the corresponding information of the pre-processed input image 211 from the synthesis information and the goodness information 221, 231, and the synthesized image 250, depth An image 260 is generated. As described above, it is possible to combine a plurality of omnifocal images and combine preferable points to combine a more preferable image. In this example, two examples of the omnifocal image synthesis means are shown, but it is easy to expand to three or more.
[0012]
FIG. 3 shows a third embodiment of the present invention. In the present embodiment, the implementation configuration in the case where the “preference” in the second embodiment is limited to “being less susceptible to noise” is disclosed in more detail.
[0013]
Since the omnifocal image composition unit 220 and the omnifocal image composition unit 230 have the same configuration, the operation of the omnifocal image composition unit 220 will be described. The input image is preprocessed by the preprocessing unit 210, and then noise is reduced by the noise reduction unit A300. Specific examples of the noise reduction means include a noise reduction filter and smoothing reduction. Here, the smoothing reduction will be described as an example. In the noise reduction image 301, a noise amount evaluation value 311 is calculated by the noise amount evaluation unit 310, and at the same time, a signal change amount evaluation value 321 representing the degree of coincidence of focus is calculated by the signal change amount evaluation unit A320. The focus determination unit 340 generates a signal change maximum value 342 and synthesis information 341 from the signal change evaluation value 321. The signal change maximum value 342 is sent to the noise determination unit 330, and the noise determination unit outputs goodness information 331 indicating the degree of influence of noise. The omnifocal image synthesizing unit 230 operates in the same manner. However, if the noise reduction unit B350 is smoothed and reduced larger than the noise reduction unit A300, the result of the omnifocal image synthesis unit 230 is the result of the omnifocal image synthesis unit 220. Compared to this, the noise reduction effect is great, but the spatial resolution is reduced. Therefore, the omnifocal image synthesizing unit 220 obtains a result with a high spatial resolution with few good parts not affected by noise, and the omnifocal image synthesizing unit 230 obtains a space with good parts without noise. Results with low resolution are obtained. By synthesizing a synthesized image with less noise influence by selecting the pixel of the synthesized image with less noise influence for each pixel by the synthesizing unit 240, there are many good portions without noise influence. In addition, the synthesized image 250 having high spatial resolution can be synthesized. In this example, two examples of the omnifocal image synthesis means are shown, but it is easy to expand to three or more.
[0014]
FIG. 4 shows a configuration example of the focus determination unit 340.
[0015]
The maximum value storage means 410 contains a signal change amount maximum value 411 up to the previous image. The maximum value calculation means 420 causes the signal change amount maximum value 411 up to the previous image and the signal change amount 321 related to the current image to be displayed. The larger one is selected to calculate the maximum signal change amount 342 including the current image, and the maximum value stored in the maximum value storage means 410 is updated. On the other hand, the subtracting means 430 obtains the difference between the signal change amount maximum value 411 up to the previous image and the signal change amount 321 relating to the current image, and the binarizing means 440 determines whether or not it is greater than zero. Therefore, when the signal change amount 321 is represented by f, the signal change amount maximum value 411 up to the previous image is represented by fmax, and the composite information 341 is represented by g,
When f> fmax, g = 1
g = 0 when f <fmax
Thus, when the signal change amount 321 related to the current image is larger than the signal change amount maximum value 411 until the previous time, g = 1. That is, the composite information 341 has g = 1 for a pixel that should select a pixel of the current image as a pixel of the composite image.
[0016]
FIG. 5 shows a configuration example of the noise determination unit 330. The noise determination unit 330 receives the noise amount evaluation value 311 and the signal change amount maximum value 342 and outputs the goodness degree information 331. When the goodness information 331 is represented by v, the signal variation maximum value 342 is represented by fmax, and the noise amount evaluation value 311 is represented by N, when the signal variation maximum value fmax is larger than the noise amount evaluation value N, the influence of noise is small. Since the degree of goodness is high, v is 1. Further, when the maximum signal change amount fmax is smaller than the noise amount evaluation value N, the influence of noise is large and not good, so v is set to 0. However, since there is no guarantee that the signal change maximum value 342 and the noise amount evaluation value 311 have the same dimensional scale, the noise amount evaluation value 311 is corrected by applying a proportional constant K. From the above, when the functions of the noise determination means 330 are summarized,
When f (x, y)> K ^* n, v (x, y) = 1
When f (x, y) ≦ K ^* n, v (x, y) = 0
It becomes. The goodness information 331 to be output is information that becomes 1 when the influence of noise is small and the goodness is high, and becomes 0 when the influence of noise is not large and good.
[0017]
FIG. 6 shows a configuration example of the synthesizing unit 240. The synthesizing unit 240 receives the synthesis information 341, 343, the goodness information 331, 333, and the pre-processed input image 211, and outputs the synthesized image 250 and the depth image 260. The goodness information 331 and 333 is information indicating the amount of influence of noise, and represents a portion where the influence of noise is small, such as 1 (white), and a portion where the influence of noise is large, such as 0 (black). In FIG. 3, if the noise reduction unit B350 is smoothed and reduced more greatly than the noise reduction unit A300, the 1 (white) portion where the influence of the noise of the goodness information 333 is small is affected by the noise of the goodness information 331. It should be larger than the small 1 (white) part. Assuming that the composite information 341 and 343 are in the form as shown in the figure, it is desirable to use the composite information 341 when the goodness information 331 is 1, and the goodness information 331 is 0 and the goodness information 333 is 1. In the part, it is desirable to use the synthesis information 343. Furthermore, the portion where the goodness information 331 is 0 and the goodness information 333 is 0 is an unbelievable portion, but since something must be selected, the information of the goodness information 333 is used. Then, the final synthesis information 600 created by combining only the parts with little influence of noise is as shown in the figure. In the composite image 250, the portion where the final composite information 600 is 1 is updated to the pixel of the post-process input image 211, and the portion where the final composite information 600 is 0 is not updated the original pixel of the composite image 250. leave it as it is. Further, in the depth image 260, the portion where the final composition information 600 is 1 is updated to the depth information of the pre-processed input image 211, and the portion where the final composition information 600 is 0 is the original of the depth image 260. Leave the pixel as it is without updating it. In this way, the part that is less affected by noise maintains a high spatial resolution, and the part that is affected by large noise is reduced in spatial resolution to reduce noise. be able to.
[0018]
FIG. 7 shows a flowchart when the second embodiment is realized by software. The noise reduction 200 or 250, the noise evaluation 210, the signal change amount evaluation 220 or 260, the noise determination 230, and the focus determination 240 are repeated while changing the degree of noise reduction by the noise reduction change 700. Then, after performing composition 250, the target is changed to the next image in the next image 710, and the whole is repeated until there are no more images.
[0019]
That is, the omnifocal image synthesis 220 and 230 in FIG. 3 is expressed as a separate part when described as a configuration as shown in FIG. 3, but in the flowchart when realized by software as shown in FIG. It is possible to configure the same module as a changed one. Even in such a case, since it is repeatedly executed, it must be considered that a plurality of omnifocal image synthesis means exist.
[0020]
FIG. 8 shows a first configuration example of the preprocessing unit 210. In the case where the positions of the captured objects are shifted in the input image 201 and the input image 202, even if the omnifocal image synthesis is performed on the pixels directly corresponding to the input image 201 and the input image 202, the correct result is not obtained. Therefore, an area 801 having an appropriate size, for example, at the center of the input image 201 is taken, and this is used as a template to perform template matching on the input image 202.
[0021]
As a result, if the region 802 is matched, the region 801 and the region 802 are overlapped, and a rectangular region (AND region) 803 where the input image 201 and the input image 202 overlap is considered, and the AND region of the input image 201 and the input image 202 is considered. The parts that do not overlap with 803 are cut off to obtain input images 804 and 805 after alignment. By performing the omnifocal image composition using the input images 804 and 805 after the alignment as inputs, the omnifocal image composition can be performed for the correctly corresponding pixels. In this example, two input images are shown, but it is easy to expand to three or more.
[0022]
FIG. 9 shows a second configuration example of the preprocessing unit 210. As in FIG. 8, when the positions of the captured objects are shifted in the input image 201 and the input image 202, the correct result is obtained even if the omnifocal image synthesis is performed on the pixels directly corresponding to the input image 201 and the input image 202. Must not. Therefore, an area 801 having an appropriate size, for example, at the center of the input image 201 is taken, and this is used as a template to perform template matching on the input image 202. As a result, if the region 802 is matched, the region 801 and the region 802 are overlapped, and a rectangular region (OR region) 903 including both the input image 201 and the input image 202 is considered. Portions that do not overlap with the OR region 903 are added, and the pixel values are filled with 0 or an average value of the respective input images to obtain input images 904 and 905 after alignment. By performing omnifocal image composition using the post-positioning input images 904 and 905 as inputs, it is possible to perform omnifocal image composition for correctly corresponding pixels. In this example, two input images are shown, but it is easy to expand to three or more.
[0023]
FIG. 10 shows a third configuration example of the preprocessing unit 210. In the input image f1 and the input image f2, when the omnifocal composition is performed without matching the luminance levels such that the average value of the pixel value of f1 is a and the average value of the pixel value of f2 is b, the pixel of f1 is changed. The boundary between the selected portion and the portion where the pixel of f2 is selected is clearly seen, and a preferable synthesis result is not obtained. Therefore, luminance conversion is performed on the input image f1 and the input image f2 so that the average pixel value is c.
[0024]
f1 (x, y) = f1 (x, y) + c−a
f2 (x, y) = f2 (x, y) + c−b
By doing so, since the average values of the pixel values of the input image f1 and the input image f2 are both equal to c, when the omnifocal composition is performed using the input image f1 and the input image f2, f1 The boundary between the portion where the pixel is selected and the portion where the pixel of f2 is selected is not conspicuous, and a preferable synthesis result is obtained. In this example, two input images are shown, but it is easy to expand to three or more.
[0025]
FIG. 12 shows a third embodiment of the present invention.
[0026]
The input image is pre-processed by the pre-processing unit 210 and then noise is reduced by the noise reduction unit 300. Specific examples of the noise reduction means include a noise reduction filter and smoothing reduction. In the noise reduction image 301, the noise amount evaluation value 311 is calculated by the noise amount evaluation unit 311, and at the same time, the signal change amount evaluation unit 320 calculates the signal change amount evaluation value 321 representing the degree of coincidence of the focus. The focus determination unit 340 generates composite information 341 from the signal change amount evaluation value 321. The synthesizing unit 1240 synthesizes the synthesized image 250 and the depth image 260 from the synthesized information 341.
[0027]
As described above, in the omnifocal image synthesis, before the signal change amount evaluation unit 320 evaluates the signal change amount, the noise included in the input image is reduced by the noise reduction unit 300, so that the influence of the noise is reduced. It is possible to synthesize a composite image 250 with a small amount.
[0028]
FIG. 13 shows a fourth embodiment of the present invention.
[0029]
The omnifocal image synthesizing unit 230 includes only default synthesis information generating unit 1301. The input image is pre-processed by the pre-processing unit 210 and then noise is reduced by the noise reduction unit 300. In the noise reduced image 301, the noise amount evaluation value 311 is calculated by the noise amount evaluation unit 311. At the same time, the signal change amount evaluation unit A320 calculates the signal change amount evaluation value 321 representing the degree of coincidence of the focus. The focus determination unit 340 generates a signal change maximum value 342 and synthesis information 341 from the signal change evaluation value 321. The signal change maximum value 342 is sent to the noise determination unit 330, and the noise determination unit outputs goodness information 331 indicating the degree of influence of noise. The default composite information generation unit 1301 displays composite information 343 in which all pixels are set to 1 when a default image is input, and all pixels when a non-default image is input so that a fixed default image is always selected. The composite information 343 with 0 being output is output.
[0030]
Therefore, the omnifocal image synthesizing unit 220 obtains a result with few good portions that are not affected by noise but high spatial resolution, and the omnifocal image synthesizing unit 230 selects a fixed default image. These are synthesized by a synthesizing unit 240, and a synthesized image is selected by selecting, for each pixel, a pixel as a result of the omnifocal image synthesizing unit 220 for a part not affected by noise and a pixel of a default image for a part affected by noise. As a result, it is possible to synthesize the composite image 250 having many good portions that are not affected by noise and high spatial resolution.
[0031]
FIG. 11 shows a fifth embodiment. The omnifocal image synthesis 1110 receives the input image 201, the signal change maximum value image 1101, the synthesis result image 1102, and the depth image 1103, and the signal change amount maximum value 1111, the synthesis result image 1112, and the depth image 1113. Output. The signal change maximum value image 1101 corresponds to the signal change maximum value fmax 411 in FIG. 4, the synthesis result image 1102 corresponds to the composite image 250 in FIG. 2, and the depth image 1103 corresponds to the depth image 260. The input / output of the omnifocal image synthesis 1120 and 1130 is the same as the input / output of 1110. Next, the operation will be described. First, the omnifocal image synthesis 1110 checks the signal change amount for the input image 201 and compares it with the signal change maximum image 1101, and updates the synthesis result image 1102 to 1112 and the depth image 1103 to 1113. . Next, the omnifocal image composition 1120 checks the signal change amount with respect to the input image 202 and compares it with the signal change amount maximum value image 1111, and updates the composition result image 1112 to 1122 and the depth image 1113 to 1123. To do. Further, the omnifocal image composition 1130 checks the signal change amount with respect to the input image 203 and compares it with the signal change amount maximum value image 1121, and updates the composition result image 1122 to 1132 and the depth image 1123 to 1133. . In this way, by inputting the input image one by one and generating the composite result in which one input image input this time is reflected on the previous composite result, an input for inputting the next input image is made. The process and the omnifocal image synthesis process for the input-completed image can be performed in parallel, and the processes of the omnifocal image synthesis 1110, 1120, and 1130 can be hidden in the process of inputting the input image. For example, in the case where the input image is composed of 10 or more images, the processing is completed in a significantly shorter time than when all 10 or more images are input and then the omnifocal image synthesis processing for 10 or more images is processed. It becomes possible.
[0032]
Next, an embodiment of a charged particle beam apparatus to which the above-described omnifocal image synthesis method and apparatus are applied will be described with reference to FIG.
[0033]
In the description of this embodiment, a scanning electron microscope, which is one of charged particle beam apparatuses, will be described as an example. However, the present invention is not limited to this. For example, an FIB (scanning an ion beam on a sample to obtain a sample image) Other charged particle beam devices such as a Focused Ion Beam device may be used.
[0034]
FIG. 14 is a diagram showing a scanning electron microscope to which the present invention is applied. This scanning electron microscope incorporates an automatic focus control function. In FIG. 14, reference numeral 501 denotes a sample stage, 502 denotes a sample to be photographed on the sample stage, 504 denotes a cathode, 505 denotes a scanning coil, 506 denotes an electronic lens, 508 denotes a scanning coil control circuit, and 509 denotes a lens control circuit.
[0035]
The electron beam 514 is scanned on the sample 502 by the scanning coil 505, and the electrons emitted from the sample 502 are detected by the detector 503. A signal S1 from the detector 503 is input to the AD converter 507 and converted into a digital signal S2.
The digital signal of S2 is input to the image processor 510, image processing such as image differentiation processing and feature amount extraction are performed, and the result is sent to the control computer 511.
[0036]
The processed image is sent to the display device 512 and displayed. The focus control signal S3 from the control computer 511 is input to the lens control circuit 509, and the focus control can be performed by adjusting the excitation current of the electronic lens 506.
[0037]
Reference numeral 513 denotes input means connected to the control computer 511. The automatic focus control in the scanning electron microscope configured as described above is a control for automatically setting the focus condition of the electron lens to an optimum value, and the method can be performed while changing the condition of the electron lens. A focus evaluation value is calculated and evaluated from detection signals of secondary electrons and reflected electrons obtained by scanning one frame, and an optimum value is set as a condition of the electron lens.
[0038]
By applying the omnifocal image synthesis of the present invention to such a charged particle beam apparatus, a charged particle beam apparatus capable of obtaining an image without blurring over the entire image can be provided.
[0039]
【The invention's effect】
According to the present invention, it is possible to provide a natural omnifocal image synthesizing method and apparatus that do not look like an artificial synthesized image even for a noisy input image.
[Brief description of the drawings]
FIG. 1 is a diagram showing a concept of omnifocal image synthesis according to the present invention.
FIG. 2 is a diagram showing a first embodiment of omnifocal image synthesis according to the present invention.
FIG. 3 is a diagram showing a second embodiment of omnifocal image synthesis according to the present invention.
FIG. 4 is a diagram showing a configuration example of a focus determination unit of the present invention.
FIG. 5 is a diagram showing a configuration example of a noise determination unit of the present invention.
FIG. 6 is a diagram showing a configuration example of a synthesizing unit of the present invention.
FIG. 7 is a flowchart of a second embodiment of omnifocal image synthesis according to the present invention.
FIG. 8 is a diagram showing a second embodiment of omnifocal image synthesis according to the present invention.
FIG. 9 is a diagram showing a configuration example of pre-processing means 210 for omnifocal image synthesis according to the present invention.
FIG. 10 is a diagram showing another configuration example of the pre-processing means 210 for omnifocal image synthesis according to the present invention.
FIG. 11 is a diagram showing a fifth example of omnifocal image synthesis according to the present invention.
FIG. 12 is a diagram showing a third example of omnifocal image synthesis according to the present invention.
FIG. 13 is a diagram showing a fourth example of omnifocal image synthesis according to the present invention.
FIG. 14 is a diagram showing an example of a charged particle beam apparatus according to the present invention.
[Explanation of symbols]
200: input image, 210: preprocessing means, 211: input image after preprocessing, 220, 230 ... omnifocal image composition means, 221, 231 ... composite image and goodness information,
240 ... compositing means, 250 ... composite image.

Claims (7)

Reduce the noise of multiple input images read at different focal points,
Evaluate the amount of noise in the noise-reduced image with reduced noise, evaluate the focus matching degree of the noise-reduced image, and calculate a signal change amount evaluation value,
Based on the calculated signal variation evaluation value, a signal variation evaluation maximum value and synthesis information are generated,
A plurality of omnifocal image synthesis means for determining the degree of influence of noise from the signal change amount evaluation maximum value and the noise amount evaluation value and generating goodness information;
An omnifocal image composition method for generating a composite image based on a plurality of the goodness information and the composite information generated from a plurality of omnifocal image composition means. The omnifocal image synthesis method according to claim 1,
An omnifocal image composition method for selecting, from a plurality of pieces of goodness information and the combination information, combination information having a low noise influence for each pixel from among a plurality of pieces, and combining the selected combination information to generate a combined image. The omnifocal image synthesis method according to claim 1,
An omnifocal image composition method for performing alignment and luminance adjustment of a plurality of input images before reducing noise of the plurality of input images. The omnifocal image synthesis method according to claim 1,
At least one of the plurality of omnifocal image synthesis means is an omnifocal image synthesis method in which a predetermined default image in the input image is selected. A noise reduction means for generating a noise reduced image by reducing noise of a plurality of input images read at different focal points; a noise amount evaluation means for evaluating a noise amount of the noise reduced image and calculating a noise amount evaluation value; A signal change amount evaluation unit that calculates a signal change amount evaluation value by evaluating a focus matching degree of the noise reduction image together with the calculation of the noise amount evaluation value, and a signal change amount evaluation maximum by evaluating the signal change amount evaluation value. A plurality of omnifocals comprising: a focus determination unit that generates a value and composite information; and a noise determination unit that determines a noise influence level from the maximum signal change amount evaluation value and the noise amount evaluation value to generate goodness information Image synthesis means;
An omnifocal image synthesizing apparatus including a synthesizing unit that generates a synthesized image based on the plurality of goodness information generated from the plurality of omnifocal image synthesizing units and the synthesized information. The omnifocal image composition apparatus according to claim 5, wherein
An omnifocal image synthesizing apparatus comprising preprocessing means for performing alignment, luminance adjustment, wrinkle alignment and luminance adjustment of the plurality of input images before the plurality of input images are input to the noise reduction means. The omnifocal image composition apparatus according to claim 5, wherein
An omnifocal image synthesizing apparatus that performs an omnifocal image synthesis process on an input image input in parallel with an input process of inputting the plurality of input images to the omnifocal image synthesis unit.

Images