JPH10243289A - Image adder - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image adder with which images, including high luminance and low luminance, can be settled within the same picture and simultaneously observed as high-depth images. SOLUTION: An image of observed sample of a microscope is picked up by an exposure time controllable television camera 12, the focusing of this picked-up sample image is discriminated by a focusing discriminating part 1412 and while waiting this focusing discriminated result, the exposure time of television camera 12 is changed at the same sample point. At the same time, the sample images are pickled up respectively by the television camera 12, these picked-up sample images are added by an arithmetic adder 142, these added images are recovered by a spatial filter 145 and displayed on a display part 16. Further, within the focusing range of sample image discriminated by the focusing discriminating part 1412, this sample is moved in an optical axis direction by a prescribed distance each time.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image adding apparatus for adding a plurality of images on a plurality of focal planes of a sample image observed by an optical microscope to generate a high-depth image.

[0002]

2. Description of the Related Art Generally, in order to optically capture an image having excellent resolution and high brightness, an image forming optical system using an optical element having a large aperture is required. However,
The imaging optical element represented by a lens has a large aperture and a shallow depth of focus.
In the case of using an image device such as an endoscope, it is important that the image has an excellent resolution and brightness and also has a large depth of focus.

[0003] By the way, in specimen inspection observation with an optical microscope, a specimen having a step, such as a pattern of a semiconductor wafer, a biological specimen having irregularities, and an LCD panel inspection, has a plurality of focal planes. If the image is shallow, blurred images are mixed in the observed image, which may hinder the inspection process based on the observed image.

Therefore, conventionally, as a method of expanding the depth of focus, a plurality of images having different in-focus positions are added, and the added image is subjected to an appropriate restoration process, so that the resolution and brightness are not impaired. It is considered that an image with a large depth is generated.
As disclosed in JP-A-83478, the position of the object surface to be focused and the exposure amount are set based on the position of the lens to be focused and the brightness of the object with respect to the object of the partial image. Focusing on the subject surface, inputting an image of each subject based on the set exposure amount, and adding these input images, and those described in Japanese Patent Application No. 7-123788. As described above, the specimen is moved in the optical axis direction, a plurality of images are taken in through the TV camera, and the observation image data in the in-focus range is added and processed. There is one that obtains an image.

[0005] However, when a plurality of images having different focal positions are added in this manner, a good added image cannot be obtained from an image captured by a TV camera or the like among the plurality of images. Sometimes. That is, a solid-state imaging device represented by a CCD used in a TV camera is
The dynamic range is narrower than a normal photographic system.
For this reason, when high- and low-luminance images are mixed in a plurality of images, overexposure and underexposure may occur in the added image. This is because the light receiving cell of the CCD has a certain amount of received light, and if the amount of light that exceeds this amount causes halation, the image is sufficiently captured by the input system such as a TV camera in such a case. An impossible portion is generated, and an overexposed image and an overexposed image are obtained. Further, even in a display device typified by an output CRT, an LCD monitor, or the like, the same occurs because the display capability of gradation is limited.

[0006] By the way, the specimen observed with an optical microscope is
Some specimens have very dark underexposure, very bright overexposure, or a mixture of these conditions. Of these specimens, either very dark or bright specimens can be dealt with by adjusting the light intensity.However, if there is a mixture of bright and dark areas, simply adjusting the light intensity will lose the original dynamic range of the image light to be observed. Observing both light and dark original images cannot be supported.

For example, in microscopic observation, there is a case where the light is insufficient due to insufficient light in the original halogen light source observation. In such a case, a high-brightness xenon light source is used. Then, the light amount is adjusted to improve the observation state. However, since the light amount is adjusted by using a neutral density filter, the light amount cannot be finely adjusted. In some cases, it is not possible to observe an image including the gray scale.

In order to solve this, it has been considered to perform observation using a TV camera capable of controlling the exposure time (electronic shutter). For example, as a representative example, an exposure time of a TV camera is controlled, a dynamic range is artificially expanded, and an image in which light and dark portions are mixed, that is, an image including high and low luminance portions can be observed. Japanese Patent Application Laid-Open No. 63-306779 discloses a technique for synthesizing a plurality of images with different exposure amounts obtained from the same subject. This uses a dedicated CCD capable of controlling the exposure time and a control unit, and divides one frame into two fields (ODD, EDD).
VEN), the exposure time is shortened in the ODD field to obtain image information of a high-luminance component, and in the EVEN field, image information of the low-luminance component is obtained in a normal exposure time to synthesize images. It is a method. As a result, the high-brightness and low-brightness portions can be observed simultaneously.

Furthermore, as a method for preventing the image quality of a synthesized image from being deteriorated, Japanese Patent Application Laid-Open No. 5-153473 discloses a method of synthesizing a plurality of images having different exposure conditions. in this case,
Since a high-brightness image and a low-brightness image for one frame instead of one field are synthesized using a TV camera capable of controlling exposure, the image quality is improved as compared with the above-described one.

[0010]

SUMMARY OF THE INVENTION However, Japanese Patent Application No. Hei.
In the case of the specimen described in the specification of JP-A No. 123788, a good and deep image can be obtained for a sample having a step in a microscope observation, but a high-luminance and low-luminance bright and dark portion are mixed, and the dynamic range of a TV camera is further reduced. In the case of a sample exceeding such a size, a good image in which high luminance and low luminance are mixed cannot be obtained by capturing an image using a normal TV camera.

Japanese Patent Application Laid-Open No. Sho 63-30677 discloses a means for expanding the dynamic range of an image in which light and dark portions are mixed.
No. 9, there is no noticeable deterioration in image quality when observing a moving image, but the image quality is noticeable when observing a still image. That is, the image information obtained by this method is not information of one frame but information of half a frame, that is, information of one field, so that the resolution and image quality are about half that of a normal observation image. . This means that a high image quality such as a microscope observation image cannot provide a sufficient image, and further, when used for obtaining a high-depth image, the configuration here has the same focal point but different positions. Since the operation of increasing the depth of the observation image is not taken into account, it is not possible to obtain an image in which the depth of focus in the optical axis direction is enlarged. Further, Japanese Patent Application Laid-Open No. 5-153473 can improve the image quality, but cannot obtain an image with an increased depth of focus as described above.

The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an image adding apparatus in which an image including high luminance and low luminance can be stored on the same screen and simultaneously observed as a high-depth image. .

[0013]

According to the first aspect of the present invention,
An imaging unit that captures an image of an observation sample of a microscope; an exposure time control unit that controls an exposure time of the imaging unit; a focus determination unit that determines a focus of the sample image captured by the imaging unit; Waiting for the focus determination result of the focus determination unit, changing the exposure time of the imaging unit by the exposure time control unit for the same sample position, and changing the sample image captured by the imaging unit at each exposure time Control means for capturing, image adding means for adding the sample images captured by the control means, recovery processing means for performing recovery processing by spatial frequency filtering on the image added by the image addition means, and recovery processing Display means for displaying an image restored by the means, and a predetermined direction of the sample within a focus range of the sample image determined by the focus determination means. Constitute a driving means for releasing movement.

According to a second aspect of the present invention, in the first aspect, there is further provided a luminance conversion means for converting a luminance range of a sample image picked up by the image pickup means according to the exposure time.

According to a third aspect of the present invention, in the second aspect, the luminance conversion by the luminance conversion means is further used together with the added image of the same image by the image addition means.
As a result, according to the first aspect of the present invention, by using the imaging means capable of controlling the exposure time, it is possible to obtain high luminance and low luminance that have irregularities in the optical axis direction and that exceed the dynamic range of the imaging means. Since the depth of focus can be expanded and contained on the same screen even for the sample image that contains it, it is possible to observe information in the depth direction at once, and to secure a dynamic range of the bright and dark parts of the effective sample image. A processed image is obtained.

Further, according to the second aspect of the present invention, it is possible to match the levels of the luminance ranges of the sample images picked up at different exposure times. An image with a tonal change is obtained.

Further, according to the third aspect of the present invention, an image having an improved S / N can be obtained by further using the added image of the same image by the image adding means together with the luminance conversion by the luminance converting means. The image quality is improved as the final added image.

[0018]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a schematic configuration of an image adding apparatus to which the present invention is applied. In FIG. 1, reference numeral 11 denotes an optical microscope, 12 denotes a television (TV) camera for photographing a microscope specimen whose exposure time can be controlled, 13 is a lighting device, 14
Is an image input / output unit that performs input addition of a microscope image, recovery processing, and various controls; 16 is a display unit that displays a processed image;
Denotes a drive control unit that generates a signal corresponding to the moving direction and the moving distance in accordance with an instruction of a CPU 148 described later. I have.

Here, the image input / output unit 14 includes an A / D converter 141 for converting a video signal from the television camera 12 from analog to digital, and this A / D converter 14
A first luminance conversion unit 149 that performs optimal luminance conversion on the input image digitally converted in step 1 according to the exposure amount, and adds and adds two image data of the input image from the first luminance conversion unit 149 and the previous image An arithmetic adder 142 that outputs a result,
First image memory 143 for storing the image data added by operation adder 142, first image memory 143
A second brightness conversion unit 1413 that performs brightness conversion of the added image stored in the storage unit, and a spatial filter 14 that performs an image restoration process on the added image whose brightness has been converted by the second brightness conversion unit 1413.
5. Storage for controlling the second image memory 146, the first image memory 143, and the second image memory 146 for storing the display image data subjected to the image restoration processing by the spatial filter 145, and performing synchronization control of the image data. Control unit 14
4. D / A converter 14 for converting display image data stored in second image memory 146 into analog data
7. Third image memory 1410 for storing an input image for focus determination, focus determination unit 14 for determining whether or not the image is in focus
11, an exposure time control unit 1412 for controlling the exposure time of the TV camera 12, and a CPU 148 for controlling the entire image processing unit and the drive control unit.

The image addition and recovery processing in the image input unit 14 is performed as follows. First, a video signal captured by the television camera 12 is A / D converted by the A / D converter 141 at a predetermined clock cycle, and the first
The luminance conversion unit 149 performs luminance conversion according to the exposure time and supplies the result to the arithmetic adder 142. Then, the arithmetic adder 142 performs image addition of the image data with the previous image already stored in the first image memory 143, and the result is stored in the first image memory 143 again. In this case, reading and writing of the first image memory 143 are performed by the memory control of the storage control unit 144.

The sample image data after addition stored in the first image memory 143 is compressed by the second luminance conversion unit 1413 into a luminance value suitable for a predetermined display system, and the spatial frequency of the spatial filter 145 Recovery processing is performed to remove the blurred portion after the addition by filtering and to restore only the in-focus image by emphasizing an edge portion having a high frequency component. The result is stored in the second image memory 1
The image data stored in the memory 46 and subjected to the recovery processing is converted into analog data by the D / A converter 147 and displayed on the display unit 16 as a processed sample image.

In this manner, the image of the required specimen is sequentially captured and added while changing the depth direction (optical axis direction) of the specimen. In this process, the specimen image exceeding the dynamic range is captured. In the case where the request has been made, the following processing is executed.

In this case, the current image data of the observation sample is fetched into the third image memory 1410, and the focus determination section 1411 determines whether or not the image data is in focus.
Based on the result of this determination, it is determined whether the addition of the current image is continued or stopped.

Here, as a simple example of the focus judgment by the focus judgment unit 1411, a difference value between each pixel of the observed image is used.
A method is used in which a determination is made based on a value (contrast value) obtained by integrating these values. For example, assuming that the contrast value is the sum of the absolute values of the difference values for each of the x and y lines as shown in FIG. This may be any part of the image. For example, an oblique line or a lattice shape is possible.

[0025]

(Equation 1)

Here, x _n and yn are the position coordinates of the image, and D _n
(X, y) indicates image data at that position, and k indicates an arbitrary line position. Then, by these calculations, the contrast value C at the Z position (= Z0) in the optical axis direction is obtained.
X ₁₁ and CY ₁₁ are obtained, and the sum of these CX ₁₁ and CY ₁₁ is set as the contrast value at the Z position (= Z0). Then, this value is sequentially calculated in the Z direction, that is, in the optical axis direction, and a contrast curve of the sample in the optical axis direction is obtained.

FIG. 3 shows an example of the contrast characteristic in the optical axis direction. In this contrast curve, when the contrast value is larger than a certain contrast threshold value C0 at which the image is in focus or out of focus, it means that the image is within the in-focus range A; It means that it is within the focusing range B. If it is within the focusing range A, the CPU 148 receives the addition permission signal, makes a determination, and shifts to the operation of addition permission and the next image capture.

In this case, the images are captured and added during the normal exposure time of the TV camera 12 (NTSC 1/60 s). For example, in the case of the frequency characteristic of the input luminance value as shown in FIG.
Since the signal can be output as a sufficient signal of the CCD of the V camera 12, the signal at this time does not need to be converted and is taken in as it is.

However, as shown in FIG. 5, an image which exceeds the dynamic range of the CCD and in which high luminance and low luminance are mixed is used for the ordinary TV camera 1.
2 (at the input MAX, 1/60 s), and the CCD cannot capture high-luminance information. In the input state, the output of the TV camera 12 loses the input luminance data above the input MAX value as shown in FIG.
There is saturated image data, and an overexposed image is obtained.

In such a case, the low-luminance side and the high-luminance side are separately taken in. That is, the TV camera 12 can change the range of the amount of incident light that can be captured by the CCD depending on the difference in the exposure time as shown in FIG.
The exposure time of the V camera 12 is changed. In this case, the input light amount up to the input MAX can cope with the normal exposure time of 1/60 s as shown by the line a in the figure, but the input over the input MAX that causes overexposure is not shown. This can be dealt with by shortening the exposure time and widening the input range as in lines b, c and d.

Next, a method of separately dividing and capturing images having a wide dynamic range exceeding the input range in accordance with the above concept will be described. In this case, the low luminance side is taken in first. At this time, the information on the high luminance side is saturated and cannot be distinguished from the surrounding image, so it is necessary to exclude it. Therefore,
As shown in FIG. 8, for the higher luminance of the IO, the first luminance converter 149 performs luminance conversion to obtain 1 / 60s of the normal luminance.
Then, an image in the vicinity of the saturated area or its luminance is removed, and the addition is performed by the arithmetic adder 142. It is desirable to change the setting of the IO value according to the frequency distribution of the luminance of the sample. For example, it is preferable to set the IO value at a valley in a portion where the frequency of data on the high luminance side is low. This emphasizes that the gradation continuity of an image can be maintained when data on the high-luminance side is fetched and synthesized next. FIG. 9 shows data after the low-luminance data is actually taken in as 8-bit data and converted. In this case, the right part from IO
It is 0.

In the case of digital processing, the first luminance converter 149 is generally constituted by a look-up table as shown in FIG. In the look-up table, an address is defined by a high / low luminance conversion signal and data input, and a data conversion output is read from this address. Generally, a RAM, a ROM memory, or the like is used. When a RAM is used here, the input / output characteristics can be arbitrarily changed by the CPU 148.

Next, for the same observation position on the specimen, the exposure control unit 1412 changes the exposure time of the camera to a short time, that is, sets the electronic shutter at a high speed to capture image information on the high luminance side. In this case, as shown in FIG. 11, data on the low luminance side is lost, but information on the high luminance side can be captured. In this case, in order to match the range in which the exposure time on the high luminance side is shortened with the original range on the low luminance side, the data luminance conversion is performed by the first luminance conversion unit 1.
49.

In such a luminance conversion, the conversion rate is made to correspond to how short the time is compared to the normal exposure time (1/60 s). For example, in the case of 1/1000 s, since the time is reduced to 1/16 times, the first luminance conversion unit 149 sets 1 time.
The range is adjusted by 6 times. FIG. 12 shows an example of data actually taken in. High-luminance-side data exceeding the high-luminance-side input MAX that cannot be accommodated within the dynamic range in a normal standard exposure time can be obtained.

The low-luminance data and the high-luminance data after the luminance conversion are image-added by the arithmetic adder 142 to synthesize and capture an image exceeding the dynamic range as shown in FIG. Can be.

Next, the microscope stage 18 is moved by a predetermined distance in the optical axis direction, that is, the Z direction according to a control command from the drive control unit 17, and the same image capturing as described above is performed. By continuing such a process, image information within the necessary focus range of the sample in the optical axis direction can be obtained.

Thereafter, if it is determined that the calculated value of the contrast by the focusing determination unit 1411 is not within the focusing range, the processing is terminated. In this case, the added image obtained by adding the required image, which is originally the input image of 8 bits, increases as the dynamic range (gradation) increases with the number of added images and the value of the exposure time for high luminance. . For example, when the exposure time is 1/1000 s, information for 4 bits is increased and 1
It becomes 2-bit data and becomes 13-bit data because it is further added to 8 bits of a normal input image. Further, when adding four light beams by moving the optical axis, 1
It expands to 5-bit data.

After that, this image is displayed on the display unit 16, and for this display, it is compressed by the second luminance conversion unit 1413 into display data, for example, 8-bit data. This is to compress the low, middle and high contrast information evenly and to keep it within the displayable dynamic range of the output device. Then, the compressed image is provided to the spatial filter 145, and the blurred portion of the image is removed, and an in-focus image is finally formed. The final image is recorded in the second image memory 146 for temporary display, converted into analog data by the D / A converter 147, displayed as a processed sample image on the display unit 16, expanded in dynamic range, and A deep image can be obtained.

In the above description, the two-image synthesis is described. However, depending on the sample, not only the two-image synthesis but also individual tone exposure time and conversion table can be prepared for each of a plurality of images to improve the gradation continuity. Good and good images can be obtained. For example, in the case of three-screen combining, low-, high-, and ultra-high-brightness combining as shown in FIG.

In the case of an image in which the S / N is deteriorated by shortening the exposure time, not only the luminance conversion by the first luminance converter 149 but also the image addition may be used to improve the S / N. it can. For example, instead of multiplying the high-luminance side image with an exposure time of 1/1000 s by 16 at once by the first luminance conversion unit 149, the image is quadrupled by the first luminance conversion unit 149.
If the same screen is added four times for the high-brightness image by the arithmetic adder 142, the result is similarly multiplied by 16. In this case, the operation takes four times as long. However, unlike the case where the data including noise is amplified by 16 times as it is, the amplification factor of the noise is reduced to 4 times, and the random noise component is obtained by adding four times. Is reduced, so that data with good image quality can be obtained.

In the above description, a case of a monochrome image has been described, but the present invention can be applied to a case of a color image. In this case, as shown in FIG. 15 in which the same parts as those in FIG. 1 are denoted by the same reference numerals, the decoder 191 divides the video composite signal into R, G, and B image data components, and To perform parallel processing on R, G, and B image data. In this case, focus detection is performed using an image of a G component that contains a large amount of luminance components. As a result, addition input processing can be performed on a color image. Further, depending on the sample, it may be performed for each color component or for the color component of interest. When it is desired to more reliably perform the operation on the luminance component, the luminance storage memory 192 may be prepared, and the above-described processing may be independently executed based on the stored luminance data.

Further, in the configuration shown in FIG. 1, it goes without saying that an image other than an image sometimes exceeding the dynamic range of the input can be used as a normal input / output adder. Others
The TV television camera 12 may be a PAL or SECAM signal type television camera, and it goes without saying that a high-resolution image can be obtained by using a high-vision type camera capable of capturing a higher band image. Further, the imaging device need not be a CCD, and there is no limitation as long as the exposure time can be controlled.

[0043]

As described above, according to the present invention, the use of the image pickup means capable of controlling the exposure time makes it possible to obtain a high luminance having irregularities in the optical axis direction and exceeding the dynamic range of the image pickup means. The depth of focus can be extended to fit the same screen even for sample images containing low brightness, so that information in the depth direction can be observed at a time, and a more effective dynamic range of the bright and dark parts of the sample image is secured. A good processed image can be obtained.

Further, since the levels of the luminance ranges of the sample images picked up by different exposure times can be matched, an image having a natural gradation change without a gradation jump in the final added image can be obtained.

Further, the brightness conversion by the brightness conversion means is as follows:
Further, by using the added image of the same image together in the image adding means, an image with improved S / N can be obtained, and the image quality can be improved as the final added image.

[Brief description of the drawings]

FIG. 1 is a diagram showing a schematic configuration of an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of focus determination according to one embodiment;

FIG. 3 is a diagram illustrating an example of a contrast characteristic with respect to an optical axis direction according to the embodiment;

FIG. 4 is a view showing an example of the frequency of an input luminance value for explaining one embodiment;

FIG. 5 is a view showing an example of the frequency of an input luminance value for explaining one embodiment;

FIG. 6 is a view showing an example of the frequency of an input luminance value for explaining one embodiment;

FIG. 7 is a view showing an example in which the exposure time of a TV camera is changed with respect to input luminance data according to the embodiment;

FIG. 8 is a diagram illustrating an example of conversion characteristics of a luminance conversion unit according to the embodiment;

FIG. 9 is a diagram illustrating a state of luminance data after conversion by the luminance conversion unit according to the embodiment;

FIG. 10 is a diagram illustrating an example of a luminance conversion unit used in one embodiment.

FIG. 11 is a diagram illustrating an example of a luminance conversion unit used in one embodiment.

FIG. 12 is a diagram illustrating a state of luminance data after conversion by the luminance conversion unit according to the embodiment;

FIG. 13 is a diagram illustrating a state of luminance data after conversion by the luminance conversion unit according to the embodiment;

FIG. 14 is a diagram showing an example of a conversion table by a brightness conversion unit according to another embodiment of the present invention.

FIG. 15 is a diagram showing a schematic configuration of another embodiment of the present invention.

[Explanation of symbols]

 11: Optical microscope, 12: TV camera, 13: Illumination device, 14: Image input / output unit, 141: A / D converter, 142: Operational adder, 143: First image memory, 144: Storage control unit, 145 ... Spatial filter, 146 ... Second image memory, 147 ... D / A converter, 148 ... CPU, 149 ... First luminance conversion unit, 1410 ... Third image memory, 1411 ... In-focus judgment unit, 1412 ... Exposure Time control unit, 1413: second brightness conversion unit, 16: display unit, 17: drive control unit, 18: microscope stage, 191: decoder, 192: memory for storing brightness.

Claims (3)

[Claims] An imaging unit configured to capture an image of an observation sample of a microscope; an exposure time control unit configured to control an exposure time of the imaging unit; and a focusing unit configured to determine a focus of the sample image captured by the imaging unit. Determining means, and waiting for a focus determination result of the focus determination means, changing the exposure time of the imaging means with respect to the same sample position by the exposure time control means, and imaging by the imaging means at each exposure time Control means for taking in a sample image to be taken, image adding means for adding the sample image taken in by the control means, and restoration processing means for performing restoration processing by spatial frequency filtering on the image added by the image adding means Display means for displaying an image recovered by the recovery processing means; and the mark within a focus range of the sample image determined by the focus determination means. Image addition unit and characterized by including a drive means for a predetermined distance in the predetermined direction. 2. The image adding apparatus according to claim 1, further comprising a luminance conversion unit that converts a luminance range of a sample image picked up by the image pickup unit according to the exposure time. 3. The image adding apparatus according to claim 2, wherein the luminance conversion by said luminance conversion means is performed in combination with an addition image of the same image by said image addition means.