SE516744C2 - Optical system for wide-field microscope, determines optical image corresponding to optimal focus of objective for each frequency band separately and aligns determined optimal images - Google Patents

Abstract

An automatic focus controller (98) determines optical image corresponding to optimal focus of objective (72) for each frequency band, separately. A registration controller (104) aligns the determined optimal images. An objective (72) modifies the electromagnetic radiation received from an object and emits the modified electromagnetic radiation. A camera (32) detects separate images for each frequency band of modified electromagnetic radiation of image emitted by objective. An Independent claim is also included for optical system operating method.

Description

25 30 35. . «. - a 516 744 v .. ff: 2 värd. For example, such traditional microscopes provide illumination 14 by using a light source 12 with white light, such as a filament lamp. Such lamps are not energy efficient. In addition, the color temperature of the lamp changes, which affects the color balance of the image as the lamp ages.

Another requirement, which increases the cost, of traditional microscopes is to provide a high quality electronic image, usually using a relatively expensive wooden chip RGB (red, green, blue) camera 32. Such a camera has internal prisms and filters for separating colors, one color for each of three black / white sensors. However, maintaining the relative positions and directions of the prisms, filters, and sensors during camera life is complicated.

In addition, the most complicated component of a traditional microscope is the lens (lens) 20 between the lamp 12 and the camera 32. Since the lamp provides white light, a traditional lens 20 is usually color compensated to provide an image for an eyepiece. In other words, the lens 20 must produce reasonably sharp images 22 for the entire spectrum of interest at the same time. This spectrum is typically the entire visible spectrum.

To meet the requirements of such a microscope, the color-compensated light field lens 20 of a traditional microscope 10 is a compromise for providing: (1) color compensation; (2) wide field of view; (3) magnification of 50-100 times; and (4) a numerical aperture typically 0.9 for a dry (air immersion) lens and 1.3 for an oil immersion lens.

To achieve the compromise of a variety of conditions, traditional microscope systems are often equipped with a number of different lenses, for example to adjust to the thickness of a possible coverslip or to provide overview images by using a small magnification lens. . Such different lenses er- 10 l5 20 25 30 35. . 1 K 4 v 516 744 »} - ~~ '-n 3 requires that the individual lens elements be aligned with small error tolerances.

In addition, even with the best possible lenses for visible light, some details of the image remain just beyond visibility. Therefore, "image sharpening" of recorded images 64, which is a common image processing step, is performed with an image processor 66. For example, small details and edges appear better in the image 68 with enhanced sharpness, by amplifying the higher spatial frequencies in an image. . Selection of such frequency-dependent gain for the sharpening filter can be optimized if the properties of the lens are known. However, the lens properties are wavelength dependent and therefore, ideally, each wavelength should have its own sharpening filter.

Unfortunately, such separate sharpening filters are difficult to achieve with an RGB camera because each color component of the camera responds to light with a spectrum width of typically +/- 50 nm which causes overlap between the colors. Thus, for example, the green component may correspond to light with wavelengths ranging from about 500-600 nm. Thus, once a sensor of the camera 32 has been exposed to white light, the contribution of each wavelength to the blur of the image caused by the lens cannot be determined, and therefore the characteristics of a sharpening filter optimized for a wavelength, not determined. Thus, a sharpening filter for such a system can only be a compromise.

Thus, traditional microscopes are unable to provide a low cost system that has a computer that produces high performance digital microscope images without filament lamps and three chip cameras. Such traditional microscopes also fail to allow the use of a simple lens suitable for wavelength-adjusted image sharpening filters. 10 l5 20 25 30 35,,. . . n 5 7 .. »a - ~ If" '. » o '16 "". *. "" .., »~ _v ~ _,» ... v-- x. .. n <1 .- H- w- 4 Summary of the Invention The problems of the prior art are completely solved or in part with an optical system according to claim 1 and a method according to claim 15.

The present invention provides a new design for a low cost digital image microscopy system. The system has a performance comparable to that of much more expensive traditional microscopes. In addition, the invention can be used to improve the performance of an existing microscope system.

An object of the invention is to generate high quality microscope images on a computer screen to eliminate the need for a user to look through the eyepiece. Thus, the invention does not provide a direct optical path from the sample to the user's eyes, but instead allows the image to pass through a camera and an image processing computer.

Another object of this invention is to provide an optical system having: (1) a lens which receives electromagnetic radiation from a sample, modifies the electromagnetic radiation and emits the modified electromagnetic radiation as an image of the sample; (2) a focusing mechanism that controls the movement of the lens along at least one path for modifying the image; (3) direct images for each of a plurality of frequency bands one or more cameras that detect the separation of the modified electromagnetic radiation of the image emitted by the sample; and (4) an autofocus controller, the detected images; (a) arranged to determine, in accordance with control parameters, to the focusing mechanism; and (b) separately for each frequency band, an optical image corresponding to an optimal focus of the lens for that frequency band and an alignment controller for aligning a plurality of optimal images. The images can be detected / registered at different times. In addition, the camera can be one or more black and white cameras.

.H n lO 15 20 25 30 35 _..-. _ _ I ~ '"e I;"' _, i f ~ o 0 '. å, | v 0 '1 a f V',; . - .- ..X. I,,. ; A further object of this invention is to provide an automatic focusing control unit having: (1) a filter calculation unit for receiving the detected images as image signals and for generating filtered image signals such that noise components of the image signals has been reduced, whereby the noise components are reduced by an increase in the energy contributions from parts of the image signals, which contribute to a relatively larger part of the image components than the noise components, and by a decrease in the energy contributions from other parts of the image signals, which contribute to a relatively larger part of the noise components other than the image components; (2) an energy calculation unit for receiving the filtered image signals and determining the energy levels of the filtered image signals; and (3) a control calculation unit for receiving the energy levels and for generating the control parameters in accordance with the energy levels.

A further object of the present invention is to provide an alignment controller having: (1) a transformation unit for receiving a first optimal image and a transformation and for generating a transformed image, the transformation having the ability to translate, rotate and / or enlarge the first optimal image; (2) a compilation unit for combining a second optimal image and the transformed image for generating a composite image; (3) an energy calculation unit for receiving the composite image and for determining an energy level of the composite image; and (4) for receiving the energy level and for generating a transform generator the transformation in accordance with the energy level so that the selected transformation corresponds to a focus of the composite image.

A further object of the present invention is to provide one or more sharpening filters, each filter being optimized for a particular frequency band. 10 15 20 25 30 35. . . . . a 516 744. . . . . . Another object of this invention is to provide an electromagnetic radiation source which can emit electromagnetic radiation in different frequency bands and selectively emit electromagnetic radiation from only one of the frequency bands. This source may comprise a plurality of separate sources, each separate source corresponding to one or more of the different frequency bands.

A further object of the present invention is to select: (1) one or more of the frequency bands to provide a response from only the corresponding parts of the sample and / or (2) at least three of the frequency bands to respond to red, green and blue color components in the visible light.

A further object of the present invention is to provide a lens which: (1) lacks significant color compensation; (2) is selected so that the optimal focus for each frequency band appears in different positions along the path of the lens; (3) provides optimal focusing positions which are monotonically related to the frequency bands; and / or (4) the lens along the path to reveal the optimum is moved in a single motion in a direction of focus for each frequency band.

In addition, it is an object of the present invention to provide a converter for transforming each n x 1 pixel of a composite image by using a weight matrix to produce a respective 3 x 1 pixel for an RGB image, where n is the number of frequency bands. In addition, the RGB image can be generated to simulate cameras that are different from the camera or cameras used in the optical system.

Another object of the present invention is to provide a method of using an optical system comprising the steps of: (1) illuminating a sample with electromagnetic radiation from a source; (2) modifying electromagnetic radiation from the sample through a lens to form an image of the sample; (3) directing the movement of the lens along at least one path for modifying the car 15 20 25 30 35. . v «1- 516 744 .m- .n 7 den; (4) detecting separate images for each of a plurality of frequency bands of the electromagnetic radiation; (5) provide control parameters for controlling the movement of the lens; (6) determining from the detected images separately for each frequency band an optimal image corresponding to an optimal focus of the lens for this frequency band; and (7) aligning a plurality of optimal images. This method may also include the steps of: (1) moving the lens only in one direction to reveal the optimal focus for each frequency band; and (2) by using a filter optimized to increase the sharpness of one or more optimal images and frequency bands, respectively.

These and other objects, advantages and features of the invention will become apparent to those skilled in the art upon study of the following description of the invention.

Brief Description of the Drawings Fig. 1 is a block diagram showing a traditional light field microscope system with selectable modules, which include an RGB video camera, a video monitor, a computer, an autofocus unit and a computer monitor; Fig. 2 is a block diagram showing an embodiment of this invention in which the red, green and blue color components of an RGB camera are captured at different times and lined up using an image alignment controller; and Fig. 3 is a block diagram illustrating a second embodiment of the present invention in which the light source is divided into several spectra with narrower bandwidth and which uses a simple lens and a camera for black and white photography.

Detailed Description of the Invention Fig. 2 shows a traditional microscope system that has been modified in accordance with this invention. More specifically, the beam splitter 24 and the eyepiece 28 in Fig. 1 have been removed, which reduces the size of the system and reduces the cost of production. In addition, the reduced 10 15 20 25 30 DJ U '| 516 744 8 the number of optical components in the optical path image quality. For reasons explained below, in this embodiment, an image storage unit 94 and an alignment controller 104 have been added to operate with or in the computer 90 to control the system.

In a traditional microscope, which has a lens with good color compensation such as the Olympus "UplanFl l00x / 1.30 oil", different color components in an image from an RGB camera have slightly different vertical positions for optimal focus. More specifically, the position of the blue color component is further away than the other color components, green and red. Nevertheless, all the colors in an image from a traditional microscope are viewed simultaneously through the eyepiece 28 or the camera 32.

In contrast to the simultaneous consideration in a traditional microscope system 10, the optical system 80 according to an embodiment of the present invention captures (views) each of the three color components 34 as separate images sequentially through the "image capture" 62. Each of the The captured color component images 92 can be stored separately in an image storage unit 94 in accordance with a synchronization signal 100 from the autofocus controller 98. Although the optical system 80 in this embodiment of the invention uses illumination 14 from a white light source 12, the time-separated image capture of each RGB component 34 with automatic refocusing between the captures an optical system 80 having a color compensation for rhyme computer solution, as opposed to an optical solution. In Fig. 2, the width of each color frequency spectrum is determined by the wavelength-dependent response of the color components of the RGB camera 32. A typical three-chip RGB camera has color components with bandwidths having a width of approximately +/- 50 nm.

Since each image capture requires an autofocus step by the autofocus controller 98, a fast image content autofocus procedure, which is insensitive to noise, is used by the autofocus controller 98 for each of the 10 15 25 25 30 35 516 744 | | -, u .U n, »| ». = = 9 captured color component images 96. In particular, one embodiment of the present invention, although other autofocus procedures may be used, uses an autofocus procedure that requires a single unidirectional scan of the vertical position of the lens to store each color component image at the optimal focus.

U.S. Patent Application No. 09/212,720, which is incorporated herein by reference, relates to such a unidirectional autofocus search used in an embodiment of the present invention. This application describes an autofocusing procedure that takes advantage of the fact that at the optimal focusing modes, the response in the spatial frequency spectrum is most evident at the low and intermediate spatial frequencies, while at higher spatial frequencies the response is usually hidden in noise. By treating the response as a signal with noise and by filtering the image with an optimal Wiener filter to remove the higher spatial frequencies, a focusing function is achieved with an improved signal (focusing response) noise ratio. This patent application also discusses the use of an approximate, linear, digital convolution filter instead of the optimal Wiener filter for faster calculations.

The approximate filter can be a linear vector that has only three non-zero elements, the 2, 0, 0, 0, the filter or the approximate filter requires curve- for example [1, 0, 0, 0, 1,]. Since neither the optimal Wiener fit, ie a procedure that uses positions on both sides of the optimal focusing position, a lens repositioning mechanism with high repeatability or that is free from hysteresis is not required. Instead, the optical system only needs to retain the image that has the highest value of the focusing function found during a unidirectional scan in the image memory 94. Thus, the optimal focus for each of the optimal color component images appears one after the other with negligible overlap by selection. of the RGB camera, the optical sys- 10 15 20 25 30 35 -. . . u 516 744 1 1 «. . The theme, the direction of the vertical scan and the order in which the color components are focused and captured.

The result is a fast capture of all the color components.

The procedure of capturing a color component and refocusing produces sharper color components from the RGB camera 32. Since the color component images 92, 102 are captured and stored at different times and at different focusing modes, however, the color component images 92 must be laterated, rotated and enlarged with with respect to each other when used together. If the color component images 102 are only used separately, such adjustments may be unnecessary. However, for example, for a grass scale image or a presentation of color images 68 for a user, alignment (alignment) of the color component images is typically necessary. Such alignment can be problematic because different color components respond differently to different types of details in the sample.

U.S. Patent Application No. 09/212,730, which is incorporated herein by reference, relates to a robust feature-free similarity criterion for aligning two different images, i.e., images that do not have identical details, used in an embodiment of this invention. . A similarity criterion is globally optimal when the images are best aligned. A similarity criterion that is feature-free does not depend on the extraction of particular image details. The criterion criterion consists of measuring the focus of an image, which (feature) before the alignment. This still image is formed from the two images to be lined up, for example by adding the two images pixel by pixel.

By using one of the images as a reference and by transforming the other relative to the reference image, the similarity criterion is evaluated and optimized over a number of combinations of transformation parameters to determine the optimal focus, which corresponds to the optimal orientation of the images. lO 15 20 25 30 UJ U '| . ,. , -ø. - | u a u. . While an embodiment of this invention utilizes this alignment procedure for color component images 102 in the alignment controller 104, other alignment procedures may be used. One reason why alignment is needed is that the three-chip camera itself, due to, for example, vibration, shock, temperature changes, temperature gradients and aging, has components that are not in line with each other and thus causes the lens to move images that are not precisely targeted.

By removing the beam splitter 24 and the eyepiece 28 and at the same time adding procedures and controllers for autofocusing and aligning different images, in accordance with this invention an optical system 80 is provided which is smaller and years cheaper, is more reliable and has higher performance. .

In the modified system of Fig. 2, the benefits of color compensation of a traditional color compensated lens 20 are not needed since each color component is focused separately. Thus, a lens 72 with less color compensation can be used. Such a lens also reduces the overlap of the focusing peaks of different colors when scanning in the vertical direction. For example, in one embodiment of this invention, a GELTECH 350140 lens is used. This lens is a non-color compensated single lens, which has a focusing depth of the size 1 μm while the center of the optimal focusing mode monotonically changes by 5-10 μm for every 50 nm change in illumination wavelength.

Another embodiment of this invention utilizes the fact that an optical system produces sharp images over a wide range of wavelengths if only a limited spectral width at a time is received by the lens. According to this embodiment, the spectrum of the lighting 118 is controlled. Such an optical system 110 is shown in Fig. 3. It operates essentially in the same way as the system in Fig. 2, except for, among other things, the differences: (1) the color control takes place in the illumination 118, which is provided by the light source 116, instead of 25 30 35. . Q. - I 516 744 «- *. HRS . 1 «| n 12 let for a camera; (2) more than three color component images 134, 144 can be observed; and (3) a plurality of narrow spectra of the color component images 134, 144 allow better path length adjustment and therefore more efficient sharpening filters.

In Fig. 3, the light source 116 is color separated over time.

A number of narrow spectrum light sources 116, such as LEDs, laser diodes and / or filtered light bulbs, are turned on and off and electronically turned on and off under the control of the computer / microprocessor 130. In one embodiment, the switching of a signal 114 from the autofocus control unit 138. The color separation can also be achieved with, for example, a light bulb and a filter wheel. in a light response 120 having a narrow spectrum and the Light 118 illuminates the sample 16, which results in object information. The light 124 emitted from the lens 122 has as narrow a spectrum as the light 118 from the illumination source 116. Thus, the lens 122 need not be color compensated and may have a simple structure. Thus, in one embodiment in Fig. 3, a GELTECH 350140 lens which is non-color compensated is used with good results. Although the fluorescence of the object 16 may widen, duplicate, or shift the spectral light response, the camera or cameras 126 will still detect the image 124 provided that the camera or cameras 126 can detect the frequency band created by the fluorescence. In addition, one or more additional autofocusing steps may be required to achieve an optimal focus for the fluorescent contribution to the image, if the fluorescent response is at a different frequency band (s) than the other frequency bands detected.

The image 124 coming out of the lens 122 hits a black and white camera 126, resulting in a black and white image 128, camera 126 can be used because in this embodiment captured in an image capture 132. A black and white illumination source 116 performs color separation. In addition, 10 15 20 25 30 35 516 744. . . »1 1 13 camera 126 be analog or digital. The image capture input 136 is provided for the autofocus controller 138. The position of the lens 122 The fixation (an image signal) is set by the focusing mechanism 54 which is controlled via a switch 50 from the autofocus controller 138 or from a manual focus controller 46. The lens 122 scans along the optical system 110. axis so that the image 134, the tion of this image, can be determined and stored in the image corresponding to the optimal focusing position memory 142 by means of a storage synchronizing signal 140 from the autofocusing controller 138. After optimizing the focus for a given color component image 134, the next color component image is selected. until the desired color component images have been captured and stored.

The alignment of the color component images 144 can begin as soon as the first two color component images have been captured. The alignment controller 146 aligns each of the "n" image components in an "n" component aligned image 148. In the image processor 150, one or more path length dependent sharpening filters may be applied to each color component of the recorded image 148.

Because this filtering is applied to separate colors, ie the narrow color components (frequency bands) instead of a full color RGB image, to be more precise and efficient. the sharpening filters can be selected Since different frequency bands can be selected from the source 12, 116, the number of frequency bands and therefore frequency responses can be greater than the three typical responses red, green and blue.

For example, an image detected from a frequency band corresponding to a wavelength of 525 nm can be optimally focused and sharpened, and a frequency band corresponding to a wavelength of 565 nm can be optimally focused and sharpened.

Since both of these frequency bands correspond to the color green, these optimal images can be combined to express the color green. lO 15 20 25 30 35 fi. in; 516 744 - -. . «. 14 If a color image 68 is to be presented to an operator on the computer screen 70, the sharpened color component images 152 are to be transformed into the standard RGB format. This transformation can be accomplished, for example, by pixel-wise linear combinations, by multiplying each 'n' x 1 pixel vector of the recorded image 148 by a 3 x 'n' weight matrix in a weight matrix calculator 154, resulting in a 3 x 1-RGB vector for each pixel.

The weights can be adjusted empirically or calculated so that the resulting image appears as a specific combination of an RGB camera and a light source.

The weight matrix also allows simulation of cameras other than the camera or cameras 126 used by the optical system. The conversion is performed by multiplying each pixel of each optimal image by the corresponding constants. Each constant is the response of the sensor to be simulated divided by the response of the sensor in the camera 126. The simulated responses are color dependent and can be measured or derived from a data sheet. Then, frequency (red, converted, or unconverted friend bands, corresponding to each of the colors green or blue) are added. to together form the green component of the RGB image 68. In order not to cause overflow in a possible integer representation of the output RGB image, the calculation unit 154. The whole procedure of multiplying the sum for each pixel by weight matrix , add and scale can be implemented using a single weight matrix.

With the optical system 80 shown in Fig. 2, the object 16 provides at least one response from each visible wavelength in either the red, green or blue component to the camera 32. In the divided illumination system 110, however, the number of color components as well as their nant path lengths and spectral widths are selected to provide a total spectrum sufficient to obtain a response from at least the wavelengths of interest for the particular application. In fact, a good choice of light source provides a better color discrimination than RGB systems 10, 80, which are illuminated with white light, since some wavelengths in the illumination with white light or other broad-spectrum illumination do not have any information about the object 16. Thus, the photos from these wavelengths of the camera's sensors without providing any information. Therefore, in the worst case, the integration time may be kept short to avoid saturation of a white light optical system requiring the sensor a camera sensor.

Another aspect of an embodiment of this invention provides the merging of adjacent and / or overlapping optimal images captured from different parts of the object to provide a wider view of the object using known techniques. This aspect is especially useful for lenses that have a narrow field of view.

While the embodiments of this invention have 132 required 142 may be described including an image capture device 62, not an image capture device 94, accepting a detected image directly from the camera or cameras 32, 126 or whether the camera or cameras may store at least one image.

Although this invention has been described by means of particular embodiments, this invention as well as this description and appended claims are not limited thereto and are to be understood in accordance with the basic idea of the invention which includes alternatives and modifications which have been made apparent to those skilled in the art. _. . . . .

Claims (17)

10 15 20 25 30 35 516 744 gyn; 16 PATENT REQUIREMENTS An optical system comprising an objective (72; 122) magnetic radiation from an object, modifying it to receive the electro-electromagnetic radiation and emitting the modified electromagnetic radiation as an image of the sample; a focusing mechanism (54) for controlling the movement of the lens along at least one path for modifying the image; at least one camera (32; 126) images for each of a plurality of frequency bands for detecting the modified electromagnetic radiation of the image emitted by the lens; characterized by (98; 138) which is in accordance with the detected images, an automatic focusing control unit arranged to, on the one hand, provide control parameters to the focusing mechanism (54): optimal image corresponding to an optimal focus for object determination, separately for each frequency band , a tivet for this frequency band; and an alignment controller for aligning a plurality of optimal images. The optical system of claim 1, wherein the autofocus controller further comprises; a filter calculation unit for receiving the detected images as image signals and for generating filtered image signals, so that noise components of the image signals have been reduced, the noise components being reduced by increasing energy contribution from parts of the image signal with a relatively larger contribution to the image components than the noise components and by reducing energy contributions from other parts of image signals with a relatively larger contribution to the noise components than the image components; 10 15 20 25 30 35 516 744. . -; m. »>. \. : v «| n. 17 an energy calculation unit for receiving the filtered image signals and determining energy levels for the filtered image signals; and a control calculation unit for receiving the energy levels and for generating control parameters in accordance with the energy levels. An optical system according to claim 1 or 2, wherein the alignment controller further comprises: 4 a transformation unit for receiving a first optimal image and a transformation and for generating a transformed image, the transformation having the possibility to translate, rotate and / or enlarge the first, optimal image; a merging unit for combining a second optimal image with the transformed image to generate a merged image; an energy calculation unit for receiving the merged image and for determining an energy level of the merged image, and a transformation generating unit for receiving the energy level and for generating the transformation in accordance with the energy level so that the selected transformation corresponds to a focus for the merged image. the picture. The optical system of claim 1, further comprising an electromagnetic radiation source (116) having the ability to emit electromagnetic radiation in different frequency bands and selectively emitting electromagnetic radiation from only one of the frequency bands. The optical system of claim 4, wherein the source further comprises a plurality of separate sources (116), each separate source corresponding to one or more different frequency bands. An optical system according to claim 1, 122) in which the lens (72; lacks significant color compensation. 10 15 20 25 30 35 516 744. An optical system according to claim 1, in which the detection of the images is arranged to take place at different times. The optical system according to claim 1, (72; 122) frequency bands appear at different positions along the lens in which the lens is selected so that the optimal focus for each path. An optical system according to claim 1, (72; 122) ring positions which are monotonically related to frequency in which the lens is arranged to provide optimal focusing bands. The optical system of claim 1, wherein a single one (72; 122) discloses the optimal focus for each of the unidirectional movement of the lens along the path of the frequency bands. The optical system of claim 1, further comprising one or more sharpening filters, each filter being optimized for a particular frequency band. The optical system of claim 1, wherein the camera comprises a single black and white camera. The optical system of claim 1, wherein at least three of the frequency bands correspond to red, green and blue color components of visible light. An optical system according to any one of the preceding claims, wherein the optical system is intended for a microscope. A method of an optical system comprising the steps of: illuminating a sample with electromagnetic radiation from a source; modifying electromagnetic radiation from the sample through a lens to create an image of the sample; control the movement of the lens along at least one path to modify the image; detecting separate images for each of a plurality of frequency bands in the electromagnetic radiation; provide control parameters for controlling the movement of the lens; 10 516 744 1 .. n. - «=. u 19 know the steps of determining, from the detected images separately for each frequency band, an optimal image corresponding to an optimal focus of the lens for this frequency band; and to align a plurality of optimal images. The method of claim 15, wherein the step of controlling the movement of the lens further comprises the step of: moving the lens only in one direction to reveal the optimal focus for each frequency band. The method of claim 16, further comprising the step of: providing increased sharpness to one or more of the optimal images by using a filter optimized for the corresponding frequency band.