Classifications

• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B21/00—Microscopes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
  • H04N23/676—Bracketing for image capture at varying focusing conditions

Definitions

• the present invention relates to an optical system comprising an objective which receives electromagnetic radiation from an object, modifies the electromagnetic radiation, and emits the modified electromagnetic radiation as an image of the object, and at least one camera which detects separate images for each of a plurality of frequency bands of the modified electromagnetic radiation of the image emitted by the objective. Furthermore, the invention relates to a method for operating an optical system.
• Fig. 1 Traditional high performance wide-field microscopes 10 as shown in Fig. 1, achieve a high-resolution color image of a specimen (object) 16 with a wide field of view of the object information 18. Also, by means of a beam splitter 24, the image 22, 26, 30 is fed simultaneously to both a video camera 32 and to an eyepiece 28.
• the focusing 48, 52 is often performed using a manual focus controller 46.
• autofocus controllers 42 which use the digital image context or additional optics can also be employed to send focusing commands 44 through a selector 50 to control the focusing mechanism 54.
• a user can look: (1) through the eyepiece 28; (2) at a video monitor 40 connected to the camera 32; or (3) at a computer monitor 70 connected through a frame grabber 62 and an image processor 66 to the camera 32.
• looking through an eyepiece strains the eyes, neck, and shoulders of the user.
• viewing the image through a monitor 40, 70 is clearly superior. Therefore, a high quality image on a monitor screen is preferable.
• a computer 60 in addition to grabbing, storing, processing and presenting images, can use the output 34 from a camera 32 for image analysis.
• such traditional microscopes provide illumination 14 using a white light source 12 such as a filament bulb.
• a white light source 12 such as a filament bulb.
• Such bulbs are not energy efficient.
• the color temperature of the bulb which affects the color balance in the image, changes as the bulb ages.
• RGB red, green, blue
• a traditional objective 20 is usually color compensated to supply an image for an eyepiece.
• the objective 20 has to produce reasonably sharp images 22 for the whole spectrum of interest at the same time. This spectrum is typically the entire visible spectrum.
• the color compensated wide field objective 20 of a tradi- tional microscope 10 is a compromise to achieve: (1) color compensation; (2) wide field of view; (3) magnification of 50-100 times; and (4) a numerical aperture which typically is 0.9 for a dry (air immersion) objective and 1.3 for an oil immersion objective.
• a common image processing step is 'image sharpening' of captured images 64 performed by an image processor 66. For example, by amplifying the higher spatial frequencies in an image, small details and edges become enhanced in the sharpened image 68. Selection of such frequency dependent amplification for the sharpening filter can be optimized if the characteristics of the objective are known. However, the objective characteristics are wavelength dependent, and therefore, ideally, each wavelength should have a respec- tive sharpening filter.
• One object of the present invention is thus to provide a low cost optical system which could be used in a microscope to generate high quality images on a computer screen.
• Another object is to provide an inexpensive method for generating high quality images on a screen of a computer which is connected to a microscope.
• the invention thus provides an optical system having an objective which receives electromagnetic radiation from an object, modifies the electromagnetic radiation, and emits the modified electromagnetic radiation as an image of the object; at least one camera which detects separate images for each of a plurality of frequency bands of the modified electromagnetic radiation of the image emitted by the objective; an automatic focus controller which, in accordance with the detected images determines separately for each of the frequency bands, an optimal image which corresponds to an optimal focus of the objective for that frequency band; and a registration controller which aligns the optimal images.
• the basic idea of the present invention is thus to separately focus images for each of a plurality of frequency bands and then to align the optimally focused images to a composite image.
• the optical system can use an RGB camera, which detects the images simultaneously, which are then sequen- tially focused.
• the at least one camera however detects the images at different times, which makes it possible to use a single black and white camera, i.e. a camera with a single sensor without filters.
• the automatic focus controller may be any means which can separately determine optimal images from the detected images.
• the controller may e.g. control the optical path between the object and the camera in diffe- rent ways. One way of doing this is to control the optical characteristics of the objective.
• the optical system however further comprises a focusing mechanism which controls the movement of the objective along at least one path to modify the image; wherein the automatic focus controller provides control parameters to the focusing mechanism.
• the automatic focus controller has (1) a filter calculator for receiving the detected images as image signals and for generating filtered image signals such that noise components of the image signals have been reduced, the noise components being reduced by increasing energy contributions from parts of the image signals which contribute a relatively larger proportion to image components than the noise components and by decreasing energy contributions from other parts of the image signals which contribute a relatively larger portion to the noise components than to the image components; (2) an energy calculator for receiving the filtered image signals and determining energy levels of the filtered image signals; and (3) a control calculator for receiving the energy levels and for generating the control parameters in accordance with the energy levels .
• This embodiment of the automatic focus controller is advantageous in that it only requires a single one- directional scan of the vertical position of the objective for storing each optimal image.
• the registration controller has (1) a transformer for receiving a first optimal image and a transformation, and for generating a transformed image, the transformation capable of translating, rotating, and/or magnifying the first optimal image; (2) a compositor for combining a second optimal image and the transformed image to generate a composite image; (3) an energy calculator for receiving the compo- site image and for determining an energy level of the composite image; and (4) a transformation generator for receiving the energy level and for generating the transformation in accordance with the energy level such that the transformation selected corresponds to a focus for the composite image.
• the optical system may comprise a source of electromagnetic radiation capable of emitting electromagnetic radiation in different frequency bands and of selectively emitting electromagnetic radiation from only one of the frequency bands so that the separation into different frequency band can be performed already at the source of electromagnetic radiation and so that a single black and white camera (single sensor without filters) can be used.
• This source can include a plurality of separate sources, each separate source corresponding to one or more of the different frequency bands.
• the optical system can preferably use an objective which lacks significant color com- pensation, which results in a less expensive optical system and even sometimes in images of higher quality.
• the optical system can be arranged such that the optimal focus for each frequency band occurs at a different position along the path of the objective; and/or such that the objective provides optimal focus positions which are monotonically related to the frequency bands; and/or such that a single unidirectional movement of the objective along the path of the objective reveals the optimal focus for each of the frequency bands. The result is that all the images of the different frequency bands can be swiftly focused.
• At least three of the frequency bands are preferably selected to correspond to red, green, and blue color components of visible light, since these are the color components traditionally used in e.g. an RGB camera.
• the optical system may comprise one or more image sharpening filters, each filter optimised for a particular frequency band.
• the invention relates to a method for operating an optical system having the steps of illuminating an object with electromagnetic radiation from a source; modifying electromagnetic radia- tion from the object through an objective to form an image of the object; detecting separate images for each of a plurality of frequency bands of the electromagnetic radiation; determining from the detected images separately for each of the frequency bands, an optimal image which corresponds to an optimal focus of the objective for that frequency band and aligning the optimal images with each other.
• Fig. 1 is a block diagram illustrating a traditional wide-field microscope system with optional modules including an RGB video camera, a video monitor, a computer, an auto-focus controller and a computer monitor;
• Fig. 2 is a block diagram illustrating one embodiment of the present invention in which the red, green and blue color components of an RGB camera are grabbed at different instants of time and are aligned using an image registration controller;
• Fig. 3 is a block diagram illustrating another embodiment of the present invention in which the light source is divided into several spectra of smaller widths and which employs a simple objective and a black and white camera.
• Fig. 2 shows a traditional microscope system that has been modified in accordance with this invention.
• the beam splitter 24 and the eyepiece 28 of Fig. 1 have been removed which makes the system smaller and reduces the cost of production.
• the reduced number of optical components in the optical path improves the image quality.
• an image store 94 and an image registration controller 104 have been added to operate with or on the computer 90 controlling the system.
• the different color components of an image from an RGB camera have slightly different vertical positions of optimal focus; in particular, the position for the blue color component is further offset from the other color components, green and red. Nevertheless, all the colors of an image from a traditional microscope are viewed simultaneously through the eyepiece 28 or the camera 32.
• the optical system 80 grabs (views) each of the three color components 34 as separate images sequentially through the frame grabber 62.
• Each of the grabbed color component images 92 can be stored separately in image store 94 in accordance with a synchronization signal 100 from the autofocus controller 98.
• the optical system 80 of this embodiment of the invention uses illumination 14 from a white light source 12 the time separated grabbing of each of the RGB components 34 with automatic refocusing between grabs allows for an optical system 80 having a computer solution, as opposed to an optical solution, for reasonable color compensation.
• the width of each color's frequency spectrum is determined by the wavelength de- pendent response of the color components of the RGB camera 32.
• a typical 3-chip RGB camera has color components with bandwidths that are approximately +/- 50 nanometers wide.
• each frame grab requires an auto- focus step by the autofocus controller 98
• a quick and noise-insensitive image content auto- ocus procedure is employed by the autofocus controller 98 for each of the grabbed color component images 96.
• one embodiment of this invention uses an auto-focus procedure that requires a single one-directional scan of the vertical position of the objective for storing each color component image at the optimal focus.
• Pending PCT/SE98/02379 incorporated herein by this reference, relates to such a one-directional auto-focus search used in one embodiment of this invention.
• an auto-focus procedure which exploits the fact that at the optimal focus positions, the response in the spatial frequency spectrum is most clearly seen at the low and middle spatial frequencies, while the response at higher spatial frequencies are usually hidden in noise.
• By treating the response as a signal plus noise and by filtering the image with an optimal Wiener filter to remove the higher spatial frequencies, a focus function with an improved signal (focus response) to noise ratio is achieved.
• That patent application also discusses using an approximate linear digital convolution filter instead of the optimal Wiener filter to provide for quicker calculations.
• the approximate filter can be a linear array having only three non-zero elements, for example [1,0,0,0,-2,0,0,0,1,]. Because neither the opti- mal Wiener filter nor the approximate filter require curve-fitting, that is, a procedure using positions on both sides of the optimal focus position, a highly repeatable or hysterisis free objective positioning mechanism is not required. Instead, the optical system only needs to retain in the image store 94 the image having the highest value of the focus function as encountered during a one-directional scan. Thus, by selecting the RGB camera, the optical system, the direction of the vertical scan, and the order in which the color components, i.e.
• the images of the different frequency bands are focused and grabbed, the optimal focus for each of the optimal color component images appear one after another with negligible overlap.
• the result is a swift grab of all the color components.
• the procedure of grabbing a color component and refocusing provides sharper color components from the RGB camera 32.
• the color component images 92, 102 are grabbed and stored at different times and at different focus positions, the color component images 92, 102 when used together should be translated, rotated and magnified with respect to one another. If the color component images 102 are only used separately such adjustment may not be needed. However, for example, a gray scale image or a presentation of color images 68 to a user, alignment (registration) of the color component images is typically necessary.
• Pending PCT/SE98/02380 incorporated herein by this reference relates to a robust feature-free similarity criterion for registering two dissimilar images, that is, images which do not have identical details, used in one embodiment of this invention.
• a similarity criterion is globally optimal when the images are best aligned.
• a feature-free similarity criterion does not depend on extracting special image details (features) prior to the registration.
• This similarity criterion consists of measuring the focus of an image composed by the two images to be registered, for example, by adding pixel -wise the two input images. By having one of the images as a reference and by transforming the other one relative to the reference image, the similarity criterion is evaluated and optimized over a number of combinations of transform parameters to determine the optimal focus which corresponds to the optimal alignment of the images.
• While one embodiment of the invention applies this registration procedure to the color component images 102 in registration controller 104, other registration procedures can be employed.
• One reason that registration is needed is that the 3 -chip camera 32 itself, due to for example vibration, shock, temperature changes, tempera- ture gradients and ageing, has components which are out of alignment and, thus, movement of the objective produces images which are not precisely aligned.
• a smaller, lower cost, more reliable and higher performance optical system 80 is provided by this invention.
• the color compensation benefits of a traditional color compensated objective 20 are not required because each color component is separately focused. Accordingly, a less color compensated objective 72 can be employed.
• Such an objective also reduces the overlap of the focus peaks of the different colors when scanning in the vertical direction.
• a GELTECH 350140 lens is used. This lens is a non-color compensated single lens having a focus depth of a magnitude of 1 micrometer while the center of the optimal focus position monotonically changes 5-10 micrometers for each 50 nanometer change in illumination wavelength.
• FIG. 3 Another embodiment of this invention exploits the fact that an optical system yields sharp images over a wide range of wavelengths if only a limited spectral width is received by the objective at a time.
• the spectrum of the illumination 118 is controlled.
• Such an optical system 110 is shown in Fig. 3. It operates mainly in the same way as the system in Fig. 2 except, among other differences: (1) the color control is in the illumination 118 provided by the light source 116 instead of a camera; (2) more than three color component images 134, 144 can be observed; and (3) mul- tiple narrow spectra of the color component images 134, 144 allow for a more wavelength tailored and therefore more efficient sharpening filter.
• the light source 116 is color separated over time.
• a number of narrow spectrum illumination sources 116 such as light emitting diodes (LEDs) , laser diodes, and/or filtered bulbs, are lit one at a time and electronically switched on and off under control of the computer/microprocessor 130. In one embodiment, the switching is determined by a signal 114 from the auto- focus controller 138.
• the color separation can also be achieved with, for example, one bulb and a filter wheel.
• the light 118 illuminates the object 16, resulting in a spectrally narrow light response 120 which contains object information.
• the light 124 emitted from the objective 122 is as spectrally narrow as the light 118 from the illumination source 116. Accordingly, the objective 122 is not required to be color compensated and can have a simple construction.
• a GELTECH 350140 lens which is non-color compensated, was used with good results.
• fluorescence in the object 16 can broaden, duplicate, or shift the spectral light response
• the camera or cameras 126 will still detect the image 124 provided that the camera or cameras 126 can detect the frequency band caused by the fluorescence.
• one or more additional auto- focus steps can be required if the fluorescent response is at different frequency band(s) than the other frequency bands being detected.
• the image 124 emerging from the objective 122 impinges on a black and white camera 126 resulting in a black and white image 128, which is grabbed in a frame grabber 132.
• a black and white camera 126 can be used because, in this embodiment, the illumination source 116 performs color separation. Also, the camera 126 can be analog or digital.
• the frame grabber information (an image signal) 136 is provided to the autofocus controller 138.
• the position of the objective 122 is determined by the focusing mechanism 54 which is controlled through selector 50 from the autofocus controller 138, or from a manual focus controller 46.
• the objective 122 scans along the z-axis of the optical system 110 so that the image 134 corresponding to the optimal focus position for that image can be determined and stored in image store 142 by means of a storage synchronization signal 140 from the autofocus controller 138. After optimizing the focus for a given color component image 134, the next color component image is selected and the procedure repeats until all the desired component images have been grabbed and stored.
• the registration of the color component images 144 can begin as soon as the first two color component images have been grabbed.
• the registration controller 146 registers each of the ' n' image components into an 'n' component registered image 148.
• one or more wavelength dependent sharpening filters can be applied to each of the color components of the registered image 148. Since this filtering is applied to separate colors, that is, the narrow color components (frequency bands) , rather than a full color RGB image, the sharpening filters can be selected to be more precise and efficient. Because different frequency bands can be selected from the source 12, 116, the number of frequency bands and therefore, frequency responses can be greater than the three typical responses of red, green, and blue.
• an image detected from a frequency band corre- sponding to the 525 nanometer wavelength can be optimally focused and sharpened and a frequency band corresponding to the 565 nanometer wavelength can be optimally focused and sharpened. Since both of these frequency bands correspond to the color green, these optimal images can be combined to express the color green.
• the sharpened color component images 152 should be transformed into the usual RGB format.
• This transformation can be achieved, for example, with pixel-wise linear combinations by multiplying each 'n' by 1 pixel vector of the registered image 148 with a 3 by 'n' weight matrix in a weight matrix calculator 154, resulting in one 3 by 1 RGB vector for each pixel.
• the weights can be adjusted by trial and error or calculated in order to make the resulting image behave much like a specific combination of an RGB camera and light source.
• the weight matrix also permits simulation of different cameras than the camera or cameras 126 used by the optical system. The conversion is done by multiplication of each pixel of each optimal image with corresponding constants.
• Each constant is the response in the sensor to be simulated divided by the response in the sensor of the camera 126.
• the simulated responses are color dependent and can be measured or derived from a data sheet.
• the frequency bands corresponding to each of the colors (red, green, or blue) converted or unconverted, are added together.
• the pixels for a 525 wavelength frequency band and a 565 wavelength frequency band would be added to together to form the green component of the RGB image 68.
• the sum for each pixel can be scaled by the weight matrix calculator 154.
• the complete procedure of multiplying, adding, and scaling can be implemented using a single weight matrix.
• the object 16 yields at least a response from each visible wavelength in either the red, green, or the blue component to the camera 32.
• the number of color compo- nents as well as their dominant wavelengths and spectral widths must be selected to provide a total spectrum that is sufficient to obtain a response from at least the interesting wavelengths of the actual application. Indeed, a good choice of light sources provides better color discrimination than from white light illuminated
• RGB systems 10, 80 because some wavelengths in the white light illumination or any wide spectrum illumination, do not carry any information about the object 16. Thus, the photons from these wavelengths expose the camera sensors without contributing any information. Therefore, in the worst case, a white light optical system can require that the sensor integration time be kept short to avoid saturating a camera sensor.
• Another aspect of one embodiment of the invention provides for the amalgamation of adjoining and/or over- lapping optimal images captured from different portions of the object to obtain a wider view of the object using techniques known in the art. This aspect is particularly useful for objectives which have a small field of view. Also, while the embodiments of this invention have been described as including a frame grabber 62, 132, a frame grabber is not required if an image store 94, 142 can accept a detected image directly from the camera (s) 32, 126 or if the camera (s) can store at least one image. While this invention has been described in terms of specific embodiments, this invention, including this disclosure and appended claims, is not so limited and is to be construed in accordance with the full spirit and scope of the invention including alternatives and modifications made apparent to those skilled in the art.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Chemical & Material Sciences (AREA)
• Analytical Chemistry (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Microscoopes, Condenser (AREA)
• Color Television Image Signal Generators (AREA)

Applications Claiming Priority (5)

Publications (1)

Family

ID=26663540

Family Applications (1)

Country Status (4)

Cited By (1)

Families Citing this family (3)

Family Cites Families (2)

• 2000
  • 2000-03-17 EP EP00919213A patent/EP1169662A1/de not_active Withdrawn
  • 2000-03-17 JP JP2000605243A patent/JP2002539497A/ja active Pending
  • 2000-03-17 WO PCT/SE2000/000529 patent/WO2000055667A1/en not_active Application Discontinuation
  • 2000-03-17 AU AU39921/00A patent/AU3992100A/en not_active Abandoned

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events