AU O ATIC FOCUS SYSTEM
Field of the Invention
The present invention relates to automatic focus systems, and more particularly to an automatic focus system for the microscopic examination of tissue or tissue components in medicine and biology.
Background of the Invention
The microscopic examination of tissue or tissue components is a common and valuable practice in both medicine and biology. In the art, microscopic examination is used to view blood smears under magnification to establish the density of certain types of blood components or the presence of disease. Such procedures typically rely on the visual appearance of the tissue which is often enhanced by the use of specialized stains that bind to certain tissue components, foreign bodies or the products of cellular processes.
With the advent of computer technology, it has now become possible to automate many of the manual examination procedures by digitizing the images and placing them into memory of a computer for analysiε, display and storage. However, the success of known automated imaging systems critically depends on the ability of the system to focus its optics on the tissue components of interest without operator intervention. The utilization of image processing equipment for the diagnostic analysis of cellular tissue often requires a detailed examination of the nuclei within the cells. Specifically, it is the evaluation of the granulation or the "texture" of the nuclei together with the accurate representations of the borders of the nuclei that plays an important role in the final diagnostic decision.
One method known in the art for enhancing the "edge" information comprises applying the Laplacian operator. The Laplacian operator comprises a mathematical operation represented by the following expression:
■) Kχ,y) (i) δx2 δy2 where I(x,y) is the continuous, planar distribution of light intensity that makes up an image.
Summary of the Invention
It has been found that the Laplacian operator can be adapted to enhance the texture content of digitized images if the texture is considered to be composed of many edges. In the case of cells examined in a diagnostic procedure, the texture of the nuclei is typically ascribed to the presence of many small, thin strands of genetic material. Such material can be interpreted as a system of many edges and thereby lends itself to the enhancement provided by the Laplacian operation.
In addition to the enhancement of edge/texture information, it has been found that the Laplacian operator can be modified to yield a quantitative measure of the texture content of a digitized image.
Accordingly, the present invention provides an automated focus system for adjusting the positioning of magnifying optics utilized for microscopic examination of tissue or tissue components in medical and biological applications. It is a feature of the present invention that texture characteristics of the captured images are enhanced and the texture characteristics are also utilized in the determination of optimal focusing.
According to the invention, the Laplacian operator is modified to provide enhanced edge/texture information and thereby a quantitative measure of the texture content of a digitized image. According to the invention, the Laplacian operator is modified in four respects. Firstly, the absolute value of the Laplacian operation is used rather than its calculated value. Secondly, the local value of the Laplacian operation is calculated and added to a running total. Thirdly, the two- dimensional Laplacian operator (expression (1) above) is modified to a one-dimensional Laplacian operator as shown below:
where I(x,yk) represents a single line of video data generated by the digitizing means of the system. In another aspect, the one- dimensional Laplacian operation is generalized for discreet digitized images to yield a "second difference" equation as shown below:
I(x_,yk) s (I 3+1,k L3,k' (I], ^3-1,k' (3) δx2
In another aspect of the present invention, expression (3) allows the calculation of the texture content to be done at high computing speeds with reduced memory requirements. Fourthly, the Laplacian operation is modified according to a weighting curve. The weighting curve can be selected, for example, to give darker regions of the image greater weight in the running total than lighter regions . The weighting according to the weighting curve allows darker regions, i.e. those typically associated with nuclei, to dominate the texture content calculation so that the method preferentially favours nuclear texture content.
Therefore, according to one aspect of the preferred embodiment of the present invention, there is provided an automated focus system comprising an intelligent controlled electro-mechanical actuation system for manoeuvring a microscope objective. The system brings a stained biological material into optimal focus for image acquisition. The system is operable without human intervention and utilizes a merit function based on the "texture" of a dark stained biological material in the field of view of the microscope objective. The system utilizes a maximization procedure using a feedback technique related to the merit function in order to control the position of the objective lens. The merit function, in turn, is based on a series of calculations performed on a set of digitized images captured at different focal positions. The use of an intelligent control routine to issue instructions to the motion control
system allows the device to avoid the usual focus and pitfalls associated with microscopic image capture.
In a first aspect, the present invention provides an automatic focus system for focusing a microscope objective for viewing a specimen located on a carrier, said automatic focus system comprising: (a) actuator means coupled to said microscope objective for moving said microscope objective in response to control signals; (b) image capture means for capturing images of the specimen located on said carrier,- (c) means for calculating focus numbers for said images; and (d) a controller for controlling said actuator means, said controller including means for generating an optimum focus position from said focus numbers and having means for issuing control signals to said actuator means for moving said microscope objective co said optimum focus position.
In a second aspect, the present invention provides an automatic focusing method for focusing a microscopic objective for viewing a specimen located on a carrier, said method comprising the steps of: (a) capturing a plurality of images of said specimen; (b) calculating a focus number for each of said images; (c) determining an optimum focus position for said microscope objective from said focus numbers,- (d) moving said microscope objective to said optimum focus position.
A preferred embodiment of the present invention will now be described by way of example, with reference to the following specification, claims and drawings.
Brief Description of the Drawings
Fig. 1 is a block diagram of an automatic focus system according to this invention;
Fig. 2 is a block diagram of a "second difference" calculation according to the invention,-
Fig. 3 is a graphical representation of a second difference weighting function according to the present invention,-
Fig. 4 is a flow chart illustrating a method for calculating the Focus Number according to this invention,-
Fig. 5 is a flow chart illustrating a method for initially focusing a microscope objective according to the present invention; and
Fig. 6 is a flow chart illustrating a method for tracking the focus of the microscope objective lens according to this invention.
Detailed Description of the Preferred Embodiment
Reference is made to Fig. 1, which shows an automatic focus system according to the present invention and denoted by 10. The automatic focus system 10 according to the present invention is suitable for integration with an image acquisition module of known microscopic examination instruments comprising a digital camera and an objective lens. The details of such microscopic examination instruments are within the understanding of those skilled in the art .
As shown in Fig. 1, the automatic focus system 10 comprises a digital camera 12, a focus number calculation module 14, and a controller 16. The digital camera 12 is coupled to a mechanical actuator 18 which in turn iε coupled to an objective lens 20 of a microscope (not shown) .
The mechanical actuator 18 is of known design and carries the objective lens 20 and translates it vertically under instructions from the controller 16. In one embodiment, the mechanical actuator comprises a voice-coil actuator with a long range (1 mm) movement. The voice-coil actuator 18 may also include a LVDT (Linear Variable Differential Transformer) position senεor (not shown in Fig. 1) . The LVDT sensor as will be familiar to those skilled in the art provides precise positioning information to the controller 16, although such a position sensor is not required for operation of the system 10.
The automatic focus system 10 according to the present invention includes a texture calculation routine. Preferably, the texture calculation routine is embedded in electronic hardware. For the system 10 shown in Fig. 1, the texture calculation routine resides in the focus number calculation module 14. As shown in Fig. 1, digitized images 22 generated by
the digital camera 12 are transferred to the focus number calculation module 14 for processing. As will be described in more detail below, the texture calculation routine analyzes the digitized images produced by the digital camera 12 and generates a figure, i.e. Focus Number, that measures the texture content of the dark-stained regions of the image.
The controller 16 preferably also comprises an electronic hardware module. The primary function of the controller 16 is to receive and store the texture figures, i.e. Focus Numbers, generated by the focus number calculation module 14 for different focal positions of the objective lens 20. Using the Focus Numberε, the controller 16 determines how far and in what direction the objective lens 20 must be moved in order to maximize the texture, i.e. focus, of the dark-stained regions in the field of view. After a decision is generated, the controller 16 issueε appropriate motion commandε 26 to the mechanical actuator 18 for re-positioning of the objective lens 20. Preferably, the controller 16 includes logic for receiving senεor and position data 28 from the mechanical actuator 18.
The process steps for analyzing the texture content and controlling the positioning of the objective lens 20 will now be described with reference to Figs. 2 to 6. According to the invention, the automated focusing system 10 is operable in two modes and the mode of operation depends upon the anticipated distance to focus for the current position of the objective lens 20. If the distance from the current position of the objective lens 20 is determined to be large, for example, as might be the case on initial start-up or when a large lateral excursion has been executed, then the automatic focusing system 10 applies an "initial focusing protocol". On the other hand, if the distance from the current position to the optimum focal point is likely to be small, then the automatic focusing system 10 uses a "tracking focusing protocol".
The method according to the present invention operates on the digitized images 22 produced by the digital camera 12. In known manner, the digital camera 12 digitizes the image of the stained biological material (located on a specimen plate or
slide) captured through the objective lens 20. Fig. 2 depicts in diagrammatic form a digitized image 50. The digitized image 50 is generated by the digital camera 12 and transferred to the focus number calculation module 14 for processing. A typical digitized image will comprise 512 scan lines produced by known CCD-type imaging cameras. Fig. 2 depicts a partial (i.e. not all 512 lines are shown) digitized image 50 comprising a series of digital scan lines 52 shown individually as 52a, 52b, 52c and so on. Each digital scan line 52, in turn, comprises a sequence of digital scan values 54 denoted individually as 54a, 54b....54n. Each scan value 54 corresponds to the binary value of the digitized pixel as will be understood by one skilled in the art. The method for providing automatic focusing according to the present invention utilizes a one-dimensional Laplacian operator given by the expression:
δ2 i(χ,yk ) (i) δx2
where l(x,yk) representε a single line of video data 52 (Fig. 2) generated by the digitizing means, i.e. digital camera 12, of the system 10. According to this aεpect of the present invention, the one-dimensional Laplacian operation given in expresεion (1) iε generalized for discreet digitized images to yield a "second difference" equation as shown below:
I(xD,yk) ≡ (I.+l,k - I 3,kJ (I D.k "3-1,k' (2) δx2
The "second difference" equation given by expresεion (2) allows the calculation of the texture content to be done at high computing speeds and also reduces the memory requirements for the focus number calculation module.
In another aspect of the present invention, the absolute value of each "second difference" calculation iε weighted according to a weighting function or curve. A weighting curve denoted generally by reference 40 iε εhown in Fig. 3. The vertical axis of the weighting curve 40 representε a weighting
factor "w" and the horizontal axis represents pixel intensity "x" for each pixel in the scan line 52. The weighting curve 40 depicted in Fig. 3 has been selected to give darker regions of the image 50 greater weight in the running total than lighter regions. In other words, the weighting according to the weighting curve 40 allows darker regions, i.e. those typically associated with nuclei, to dominate the texture content calculation so that the method preferentially favours nuclear texture content.
Reference is next made to Fig. 4 which shows in flow¬ chart form a method for calculating the Focus Number according to the present invention. The method denoted generally by 100 operates on a digitized grey-level image 50 (as depicted in Fig. 2) which iε generated by the digital camera 12. The method 100 iε preferably implemented as a routine embedded in electronic hardware in the focus number calculation module 14. In the preferred embodiment, a series of three digitized images are captured by the digital camera 12 with each image at a different focal position. In addition, a fourth image designated a "test image" or "confirmation image" iε also utilized.
The method 100 calculateε a Focus Number for each of the imageε where each Focuε Number provides a meaεure of the quantity of "texture" in the image and utilizing the weighting function 40 of Fig. 3 particular attention is given to the dark- εtained regions. The darker regions are of interest becauεe they are generally associated with the objectε of diagnostic intereεt such as nuclei in the biological material. However, it will be understood that the weighting of the Focus Number or texture calculation can be altered to accommodate alternative staining schemes.
Referring to Fig. 4, the firεt step m the focus number calculation routine 100 comprises a decision block 102 which checks if the last line 52, e.g. line 512 in 512 line image, in the image 50 has been reached. If yes, the routine 100 stops or returns (block 104) . If the laεt line 52 (i.e. line 512 of the image 50) haε not been reached, the routine 100 determineε (deciεion block 106) if a new line, e.g. line 52k (Fig. 2), in
the image 50 is being processed. If a new line is being processed, the routine 100 inputs the first and second pixels 54a, 54b (Fig. 2) as shown in block 108. If the processing of the line 52k is already in progresε, the routine 100 inputs the next pixel, e.g. pixel 54h in the image line 52 shown in Fig. 2.
Referring to Fig. 4, the routine 100 in block 112 executes the "second difference" calculation according to the "second difference" equation given above in expression (2) . In block 114, the routine 100 takes the absolute value of the second difference calculation. Next, the routine 100 performs the weighting assignment in block 116. In this εtep, the weighting function 40 (Fig. 3) iε applied to the abεolute value so that the value is weighted inversely with respect to the centre pixel's value. In block 118, the routine 100 adds the weighted value determined in step 116 to a running total . The running total repreεentε the Focuε Number for the image under consideration.
The routine 100 then moves to step 120 where the pixel values are updated. As will be understood by one skilled in the art, the routine 100 utilizes four memory locations for procesεing. Three memory locations store the active picture element or pixel values, i.e. pixel Iη_, , I-ι,k, and I-+lιk in the "second difference" equation. The fourth memory location stores the running total value (updated in step 118) . In step 120, pixel 2 (i.e. I. k) becomes new pixel 1 (i.e. ID-ι,k) and pixel 3 (i.e. pixel I-+1 κ) becomes new pixel 2 (i.e. I._k) . New pixel 3 is updated in step 110 as the next pixel in the line 52k (described above) . The stepε of the routine 100 are repeated until all of the lineε, e.g. 512, in the image 50 have been proceεεed. At completion of the processing (step 104) , the running total value calculated for the image represents the Focuε Number or texture quantity for the image.
Both the initial focusing routine and the tracking focusing routine utilize the above described focus number calculation routine.
Reference is made to Fig. 5 which showε in flow-chart form the initial focusing routine 200. In operation, the initial focusing routine 200 utilizes a series of three digitized images
taken at different focal positions (i.e. vertical translation steps for the objective lens 20) . The Focus Numbers are calculated for the images and the controller 16 analyzes the Focus Numbers as follows. First, the direction of motion for the next series of three images is determined from the direction of increasing Focus Number. Second, if the Focus Numbers show a middle figure bracketed by two lower values, then a test is performed to ensure that a local maximum has been found. Third, the step εize of the image series iε gradually decreaεed to efficiently determine the maximum and alεo avoid first-surface reflections from the cover slip for the microscope slide. Once the maximum Focuε Number has been determined, the routine 200 takes the four Focus Numbers corresponding to the three captured images and the test or confirmation image, and fitε the data to a quadratic polynomial function. The precise position of the maximum is established from the maximum in the polynomial and the controller 16 moves the objective lens 20 to this poεition which repreεents the optimal focus for the image.
Referring back to Fig. 5, at step 202 the lens 20 is advanced one step and a digitized image iε captured by the digital camera 12. A εuitable εtep size at this stage in the routine is 50 microns. The image is εtored in memory and the focuε number calculation routine 100 (Fig. 4) is called in step 204 to calculate the Focus Number for the image. In step 206, the Focus Number is compared to a maximum value. If the Focus Number is greater than the maximum value, the Focus Number is stored as the maximum Focus Number in block 208. The routine 200 next determines in step 210 if the range limit has been reached. The range limit 210 keeps count of the three images and additional confirmation image. If the range limit has not been reached, the lens 20 is advanced one step (block 202) and the sequence is repeated i.e. another image is captured at the next step and the focus number is calculated. Once the three images have been captured, a test or confirmation image is taken to verify that previously determined maximum Focus Number is not due to noise.
After the maximum Focus Number is determined, the initial focusing routine 200 proceeds to step 212. At step 212, the lens 20 is moved to the maximum Focus Number position. Next, the focusing routine 200 goes back one step (block 214) and then reduces the step size (e.g to 25 microns) for the image series (block 216) . With the step size reduced, the routine 200 advances the lens 20 by one step in block 218. The Focus Number is calculated for the image at block 220 by calling the Focus Number calculation routine 100 (Fig. 4) . The Focus Number calculated at step 220 is compared to the last Focus Number in decision block 222. If the Focus Number is not greater than the last Focus Number, the routine 200 proceeds to decision block 224 to check if the range limit has been reached. If the range limit is reached (i.e. three imageε and a test image have been captured and procesεed) , the routine 200 abortε processing (block 226) . If the range limit is not reached, the routine 200 returns to step 218 and advances the lens 20 one more step and captures the next image and calculates the Focus Number.
Referring to Fig. 5, if the Focus Number is greater than the last Focus Number at step 222, the routine 200 moves to block 228 and executes a vertical translation step for the lens 20 and an image is captured at this new position. Next at step 230, the Focus Numbers for the image is calculated. The routine 200 then determines at decision block 232 if the calculated Focus Number is the maximum. If yes, then the routine 200 checks if the range limit has been reached at block 224 (i.e. three images and a test image have been processed) , and if the range limit has not been reached, steps 218 to 222 are repeated as described above. If the Focus Number (determined at step 230) is not greater than the last Focus Number, i.e. the maximum has not been reached, the routine 200 moveε to decision block 234 to determine if the step size can εtill be reduced. If the minimum εtep size has not been reached, the routine 200 goeε back one vertical step at block 214, and steps 216 through 222 are repeated as described above.
On the other hand, if the minimum step size (e.g. 10 micronε) haε been reached as determined in step 234, the search
for maximum Focus Number, i.e. texture, is complete and the routine 200 turns to determining the precise position of the texture maximum. At step 236, the initial focusing routine 200 takes the Focus Numbers for the three images and the additional test or confirmation image and fits a quadratic polynomial to the data. The routine 200 then establiεhes the position of the Focus Number maximum from the maximum in the polynomial at step 238. One skilled in the art will understand the implementation of the quadratic polynomial. In response to the poεition determined by the focusing routine 200, the controller 16 isεueε commands to the actuator 18 to move the objective lens 20 to the position which is the optimal focus for the image (block 240) . The routine 200 uses sensor information 28 from the actuator 18 to verify the position of the objective lens 20 (block 242) . If the positioning of the objective lens 20 is correct, the initial focusing routine 200 is complete (block 244) , otherwise the position of the lens 20 is adjusted through the actuator 18 (block 240) .
Reference is next made to Fig. 6 which showε the tracking focusing routine 300. The tracking focusing routine 300 is appropriately used after the system 10 has executed a small lateral move in which the anticipated focal position is not very far from the current poεition of the lens 20. In thiε context, "not far" means that it is likely that a series of three of the smallest vertical steps will be sufficient to bracket the focal position and allow the use of a quadratic polynomial fit in order to find the exact focal position.
In operation, the tracking focusing routine 300 executes three small vertical translation steps and calculates a Focus Number for each position. The routine 300 then uses these points to calculate the position of the maximum in a theoretical parabola. The parabola describes the variation of the Focus Numbers with respect to vertical position. The controller 16 then instructs the mechanical actuator 18 to move the lens 20 to this position in order to complete the focus routine.
Referring to Fig. 6, the tracking focusing routine 300 first checks if the vertical step size (e.g. 10 microns) is at the minimum in decision block 302. If the step size is not at the minimum, it is reduced (block 304) . Once the minimum step size is set, the tracking routine 300 advances the lens 20 one step at block 306 and an image is captured, and then calculates the Focus Numbers for the captured image at the step in block 308. The results of the Focus Number calculation are stored at block 310 and the lens 20 is advanced one more step at block 312 and another image is captured. The Focus Number calculation is repeated for the new vertical step (block 314) and stored in memory (block 316) . The lens 20 is then advanced another vertical εtep (block 318) and a third image and confirmation image are captured. The Focuε Number calculations are performed for the captured images taken at the εtep (block 320) and stored in memory (block 322) .
As shown in Fig. 6, the tracking focusing routine 300 then uses a quadratic polynomial to fit the Focus Numbers to a parabola (block 324) . The parabola provides a relation between the Focus Numbers and the vertical position of the objective lens 20. In step 326, the routine 300 attempts to ascertain the maximum focus position from the parabola. If the maximum position for the optimal focus cannot be ascertained, the tracking focusing routine 300 reverts to the initial focusing routine 200 (block 328) . If the maximum is ascertainable, the maximum is determined in step 330 and the controller 16 issues commands to the actuator 18 to move the lens 20 to the position for optimal focus. The position of the lens 20 is checked (block 334) and adjuεted if necessary (block 332) , otherwise the tracking focusing routine 300 returns control (block 336) to the calling routine running for example on a central control computer (not shown) .
If the operation of the tracking focusing routine 300 cannot generate a maximum with the three measurementε taken at each vertical εtep aε described above, then the automatic focus system 10 automatically reverts to the initial focusing routine 200 described above with reference to Fig. 5.
It is to be understood that the foregoing description of the preferred embodiment of this invention is not intended to be limiting or restricting, and that various rearrangements and modifications which may become apparent to those skilled in the art may be resorted to without departing from the scope of the invention as defined in the appended claims.