RU2794050C1 - Autofocus method for digitizing a microscopic preparation - Google Patents

Abstract

FIELD: digital microscopy.

SUBSTANCE: in the method of autofocusing when digitizing a microscopic preparation, which includes moving the microscope object table along the vertical Z axis to a position corresponding to the maximum value of image sharpness of the sample of the microscopic preparation, autofocusing is carried out in two stages. The first stage includes continuous movement of the object table with the sample along the Z axis towards the microscope objective at the maximum speed by a given value, then the object table is continuously moved away from the lens at a given speed while obtaining images, calculating the sharpness S of the resulting images, assigning object table coordinates along the Z axis and the calculated sharpness values for the obtained images during the movement of the object table. The sharpness values S and coordinates along the Z axis for all images are sent to the memory, the focused image is fixed by selecting the coordinate of the object table with the sample on the Z axis, which satisfies the condition under which the sharpness decreases during the passage of the object table along the specified range along the Z axis on three images in a row before the “peak” of sharpness and after the “peak” of sharpness. Moreover, if the specified condition is met, the first stage of autofocusing is considered complete, and the movement of the object table with the sample is stopped before reaching the end of the specified range, and if the specified condition is not met, the second stage is carried out with an increased focusing range, the speed of movement of the object table and a given distance to obtain a focused image while simultaneously performing image sharpening calculations.

EFFECT: increased speed of autofocusing.

1 cl, 6 dwg

Description

The invention relates to the field of digital microscopy, namely to the field of continuous autofocusing of digital photosystems, and can be used in medicine for digitizing microscopic preparations.

At the moment, in digital microscopy systems, when digitizing each individual field of view, digital microscopy systems are forced to focus in order to obtain the sharpest image for its subsequent digitization. In the context of devices for automatically digitizing microscopic preparations, this procedure is also called autofocus. Since a single field of view obtained at a high magnification (for example, 40x) has dimensions of the order of 300 x 200 μm, and depending on the nature of the digital preparation, such fields of view can be in the hundreds, the time required to autofocus each field of view significantly affects the duration of the entire microscopic specimen digitization procedures. In this regard, reducing the time required for autofocusing seems to be one of the ways to reduce the time required to digitize the entire microscopic preparation.

A known method of automatic focusing (RU2389050, publ. 10.05.2010), which is carried out as follows:

- the focusing device is moved to one of the extreme positions;

- calculate the sharpness parameter for a given position of the focusing device by performing wavelet analysis on a plurality of images;

- take the calculated value for the preliminary maximum sharpness value;

- move the focusing device from one extreme position to another with a given step;

- calculate the sharpness parameter for a given position of the focusing device;

- the calculated value of the sharpness parameter is taken as the preliminary maximum value of the sharpness parameter of the captured image;

- move the focusing device from one extreme position to another step by step to determine the preliminary maximum value of the sharpness parameter;

- comparing in pairs the current value of the sharpness parameter of the captured image with the preliminary maximum sharpness value;

- determine the maximum value of the sharpness parameter for a predetermined number of stages of refinement, for which the focusing device is moved to a position corresponding to the previously maximum value of the sharpness parameter;

- move the focusing device one step to the left and right of the preliminary maximum value;

- calculate the current value of the sharpness parameter of the captured image;

- compare the current value of the sharpness parameter with the previously maximum.

The disadvantage of the known method is the need to move the focusing device from one extreme position to another, which significantly increases the time required for focusing. Another disadvantage is the step-by-step nature of the movement, which adversely affects the speed of focusing, and also puts an additional burden on the focusing device, since it is necessary to stop the focusing device at each step. Another disadvantage is the need to perform refining movements of the focusing device, which consists in moving the focusing device, after moving from one extreme position to another position with the highest value of the sharpness parameter, and moving to the left and right of it, which also increases the time required for focusing.

A known method of automatic focusing (RU 2528582, publ. 20.09.2014), which is carried out as follows:

- step by step scanning the focus area, while removing the characteristics of changes in the contrast (sharpness) of the image;

- move the observed object to the point of the best contrast, which is determined by the results of comparing the contrast change function obtained in the first phase and the current contrast value.

The disadvantages of this method are the need to scan the entire focus area, which increases the time required for scanning, as well as the step-by-step method of movement, which reduces the resource of the focusing mechanism and increases the time required to scan the focus area. Another disadvantage of this method is the need to return the object of observation to the point of the best contrast, which also increases the scanning time.

The closest analogue to the claimed invention is a method of autofocusing for a microscope (US7027221B2, publ.11.04.2006). The microscope is configured to provide a relatively continuous movement between the object table and the objective and perform the following sequence of actions:

a) acquisition of an image from the camera during continuous movement along the Z-axis of the object table relative to the lens;

b) determining the position along the Z-axis, reached during the relative movement along the Z-axis, in which the image from the camera was received;

c) calculating the contrast value of the resulting image;

d) a record of the Z-axis position reached during the relative Z-axis movement, together with the contrast value for that position;

e) executing steps a) to d) until a certain end position is reached when moving along the Z axis;

f) calculating a continuous function of contrast depending on the positions along the Z-axis, and assigning contrast values to the corresponding positions along the Z-axis;

g) calculation, using a continuous function of contrast, of the maximum contrast value and its corresponding z-position;

h) moving to the position along the Z-axis, which corresponds to the largest contrast value.

The disadvantage of the known method is the low speed of autofocusing when working with microscopic samples due to the need to move the microscope object table with the preparation to the position corresponding to the highest value of the image sharpness, as well as the use of multi-stage mathematical transformations to plot the curve of sharpness dependence on the position of the object table along the Z axis.

Since one field of view, obtained at a high magnification (for example, 40x), has dimensions of the order of 300 x 200 microns, and depending on the nature of the microscopic preparation, such fields of view can be in the hundreds. The time required for autofocusing of each field of view significantly affects the duration of the entire procedure for digitizing a microscopic preparation. In this regard, reducing the time required for autofocusing seems to be one of the ways to reduce the time required to digitize the entire microscopic preparation.

The technical task is to reduce the time required for autofocusing and digitization of a microscopic preparation.

The technical result is an increase in the speed of autofocusing when working with microscopic samples due to the absence of the need to move the microscope object table to the position corresponding to the highest value of the image sharpness of the microscopic preparation.

The technical result is achieved by the fact that in the method of autofocusing when digitizing a microscopic preparation, including moving the microscope object table along the vertical Z axis to a position corresponding to the maximum value of the image sharpness of the sample of the microscopic preparation, according to the invention, autofocusing is carried out in two stages. The first stage includes continuous movement of the object table with the sample along the Z axis towards the microscope lens at maximum speed (the so-called "jerk") by a predetermined value, then the object table is continuously moved away from the lens at a predetermined speed with simultaneous acquisition of images, their transmission to personal computer (PC), calculation of the sharpness S of the obtained images, assignment of object table coordinates along the Z axis and calculated sharpness values to the obtained images in the process of moving the object table. In this case, the resulting image in the RGB color model is converted into the YUV color model and the sharpness S of each image is calculated using the formula:

Where

– порядковый номер текущего пикселя изображения по горизонтали,

– serial number of the current image pixel horizontally,

– порядковый номер текущего пикселя изображения по вертикали,

– serial number of the current pixel of the image along the vertical,

Y is the value of the brightness component of the image pixel in the YUV color model;

– разность значений яркостного компонента соседних пикселей по горизонтали,

is the difference between the values of the brightness component of adjacent pixels horizontally,

- разность значение яркостного компонента соседних пикселей по вертикали,

- difference value of the brightness component of adjacent pixels vertically,

– количество пикселей по горизонтали,

- the number of pixels in the horizontal direction,

- количество пикселей по вертикали.

- number of vertical pixels.

The sharpness values S and coordinates along the Z axis for all images are sent to the PC memory, the focused image is fixed by selecting the coordinates of the object table with the sample on the Z axis, which satisfies the condition under which, during the passage of the object table through the specified range along the Z axis on three images in a row, sharpness decreases before the “peak” of sharpness and after the “peak” of sharpness. Moreover, if the specified condition is met, the first stage of autofocusing is considered complete, and the movement of the object table with the sample is stopped before reaching the end of the specified range, and if the specified condition is not met, the second stage is carried out with an increased focusing range, the speed of movement of the object table and a given distance ”) until a focused image is obtained, while simultaneously performing the calculation of the sharpness of the images, while the second stage is carried out until the specified condition is met.

The invention is illustrated by the following figures.

Fig. 1. Schematic representation of the "peak" sharpness S _max .

Fig. 2. Schematic representation of a slight change in sharpness.

Fig. 3. Increasing sharpness when passing through the focus range.

Fig. 4. Schematic representation of sharpening.

Fig. Fig. 5. Graph of the movement of the object table (a) and graph of the sharpness of the obtained images (b) during the first stage of focusing.

Fig. Fig. 6. Graph of the movement of the object table (a) and graph of the sharpness of the obtained images (b) during the first and second stages of focusing.

The method is carried out as follows.

A microscopic preparation is placed on the object table and the autofocusing procedure is started.

First stage

. В данный момент цифровая камера, подключенная к микроскопу, не отправляет получаемые изображения на ПК на анализ с целью определения резкости S изображений, соответственно в этот момент отсутствует обратная связь между получаемым изображением и движением предметного стола.The object table starts moving upwards, i.e. towards the lens of a digital camera with the maximum possible speed V _l , with which its mechanization allows the object table to move. The object table "drives" up by a predetermined value of "jerk" -

. At the moment, the digital camera connected to the microscope does not send the received images to the PC for analysis in order to determine the sharpness S of the images, respectively, at this moment there is no feedback between the received image and the movement of the object table.

на длину заранее выбранного диапазона фокусировки

или до момента получения сфокусированного изображения.Then the object table starts moving away from the lens, i.e. down axis

by the length of a preselected focus range

or until a focused image is obtained.

The image with the highest sharpness value is taken as the in-focus image.

During a continuous downward movement from the lens, the camera connected to the microscope through a photographic tube takes an image and sends them to a PC in order to determine the sharpness S of each received frame.

The number of images taken per second depends on the characteristics of the camera used. The frame refresh rate of the camera and the speed of movement of the object table must be such that the distance between two adjacent images does not exceed the value of the depth of field of the optical system used.

During a continuous downward movement, the received images from the camera are analyzed in order to calculate the sharpness S of the image. To calculate the sharpness of the image S, the sum of the squared differences of adjacent pixels vertically and horizontally is calculated. For this, the YUV (Y component) color model is used. To convert the components of the RGB color model to the YUV color model, use the following formula for each image:

In the resulting YUV color space, the Y component is used, i.e. image brightness.

Sharpness is calculated as the square root of the sum of the squared differences of adjacent pixels horizontally and vertically:

(1)

(1)

Where

– порядковый номер текущего пикселя изображения по горизонтали,

– serial number of the current image pixel horizontally,

– порядковый номер текущего пикселя изображения по вертикали,

– serial number of the current pixel of the image along the vertical,

Y is the value of the brightness component of the image pixel in the YUV color model;

– разность значений яркостного компонента соседних пикселей по горизонтали,

is the difference between the values of the brightness component of adjacent pixels horizontally,

- разность значение яркостного компонента соседних пикселей по вертикали,

- difference value of the brightness component of adjacent pixels vertically,

– количество пикселей по горизонтали,

- the number of pixels in the horizontal direction,

- количество пикселей по вертикали.

- number of vertical pixels.

In the prototype method, a different, complex and multi-stage image processing algorithm is used, according to which the image is obtained in gray tones (black and white image), then it is converted using an operator into the so-called gradient image. From the gradient image, a K value is obtained, which is the contrast value of each image. K is obtained by determining the low-frequency distribution in a black and white image, which is stored as an array H[1,2….N] , and calculated according to the formula presented in the description of the prototype. Based on the values of K build a graph of the value of K from the position along the Z-axis (figure 7 of the prototype). In order to make the graph continuous, the values are additionally interpolated using the Lagrange formula (formula 5 of the prototype).

In the claimed method of image processing, the same goal is pursued, which is to obtain the dependence of the sharpness value for constructing a sharpness curve on the position along the Z axis. However, in the claimed method, the number of mathematical manipulations with the image is much less, which also allows real-time calculation. However, the prototype uses a camera that shoots 25 frames per second, while the claimed method uses a camera that shoots 75 or more frames per second. This means that the number of images received for analysis is three or more times greater, and the calculation must be done much faster.

During the autofocusing process, for all acquired images, the PC memory stores their sharpness S values and the Z-coordinates at which they were acquired, and stores them until autofocusing is completed.

будет постоянно увеличиваться, так как был сделан «рывок» на значительное расстояние

. от возможного сфокусированного изображения. После того, как сфокусированное изображение будет достигнуто, резкость дальнейших проанализированных изображений будет уменьшаться. Таким образом, точка фокусировки – это координата по оси Z, при которой достигается сфокусированное изображение, она находится на «пике» резкости, т.е. в точке, после прохождения которой первоначальное увеличение значений резкости S изменится в сторону уменьшения.In the first images acquired, it is expected that the sharpness

will constantly increase, as a "jerk" has been made for a considerable distance

. from a possible in-focus image. Once the in-focus image is reached, the sharpness of further analyzed images will decrease. Thus, the focus point is the Z-coordinate at which a focused image is achieved, it is at the "peak" of sharpness, i.e. at the point after which the initial increase in sharpness values S will change downward.

In order to eliminate digital noise, when calculating the increase or decrease in sharpness in the resulting images, a change, up or down, is considered to be a change that exceeds 0.5% of the sharpness value of the current frame from the previous one. Also, during focusing, false “peaks” of sharpness can occur, when sharpness is increased due to dust that has fallen into the focus area, stains on the coverslip, or other artifacts. With false “peaks”, the sharpness will decrease no more than twice in a row, since the object table will approach the area of the focused image of the preparation, and the sharpness will begin to increase. Thus, the “peak” of sharpness should be taken only as the one at which, after sharpening in three images, a decrease in sharpness was observed in three images in a row.

Next, you need to follow the following conditions:

Condition 1. The value of sharpness S decreased on three images in a row and during the passage of the range there was an increase in sharpness in the amount of three times (Fig. 1). Condition 1 allows you to determine that autofocusing is successful, if the sharpness value has been reduced more than three times in a row, and there has been an increase in sharpness three times during the entire movement of the stage, then the sharpest image is taken as the in-focus image. If this condition is met, then the first stage of focusing is considered completed, and the object table, before reaching the end of the specified focusing range, stops moving to complete the autofocusing process.

Condition 2. The value of sharpness decreased on three images in a row and during the passage of the range there was no increase in sharpness in the amount of three times (Fig. 4). Condition 2 allows you to determine that the “peak” of sharpness was not reached, that is, the movement of the object table was carried out in the opposite direction from the sharpest image. When this condition is met for the first time, the first step must be repeated in the opposite direction. If condition 2 is met for the second time during the autofocus process, then the first stage is considered not fulfilled.

Condition 3. The value of sharpness did not change on twenty images in a row (Fig. 2). Condition 3 allows you to determine that the glass slide with the preparation is missing. If this condition is met, the first stage is considered not fulfilled.

Condition 4. Passed the entire specified range and not met any of the above conditions 1,2,3 (figure 3). The condition allows you to determine that the slide with the preparation is at a distance from the lens that is greater than the specified range. When this condition is met, the focus is refocused from the sharpest point for the first, second, or third time. If the “peak” of sharpness was not detected after three repeated focusings, then the first stage is considered not completed.

If necessary, even if the first stage is successfully carried out, it can be repeated. This may be necessary to eliminate errors due to external factors, such as vibration of the base on which the microscope system rests, or unintentional impact on the stage. Thus, obtaining a falsely focused image is excluded and the reliability of the method is increased.

Second phase

The second stage of autofocusing (hereinafter referred to as long focus) is characterized by an increased focusing range, an increased speed of movement of the object table V _l , and also the length of the initial "jerk" J _l . The second focusing step is performed if condition 1 was not met within the framework of the first step and a focused image was not obtained.

, которое превышает значение «рывка» на первом этапе. Диапазон длинного фокуса

превышает диапазон на первом этапе. После «рывка» в режиме длинного фокуса предметный стол начинает движение навстречу объективу со скоростью

, превышающую скорость движение на 1 этапе, до получения сфокусированного изображения, или на длину, соответствующую диапазону длинного фокуса, и одновременно осуществляют анализ резкости полученных изображений.The second stage of focusing begins with a “jerk” towards the lens at the maximum possible speed V _l for a predetermined distance

, which exceeds the value of "jerk" at the first stage. Long focus range

exceeds the range at the first stage. After a “jerk” in the long focus mode, the object table begins to move towards the lens at a speed

, exceeding the speed of movement at 1 stage, until a focused image is obtained, or by a length corresponding to the range of a long focus, and at the same time the analysis of the sharpness of the obtained images is carried out.

At the second stage, the movement of the object table occurs until one of the conditions 1-4 prescribed earlier is met.

When conditions 2, 3, 4 are met, autofocusing of the second stage is considered unsuccessful.

When condition 1 is met, autofocusing of the second stage is considered successful. The fulfillment of each of the above conditions determines the further performance of autofocus.

During the second stage, the distance between images that the camera receives during continuous movement is greater than when focusing within the first stage due to the higher speed of the object table. In such a case, the most focused image may be at a z-axis position between two adjacent frames. Then, in order to improve the quality of the focused image, it is necessary to autofocus within the first stage, taking as a starting point the position along the Z axis of the sharpest image obtained in the second stage.

If condition 1 is met in the first stage, then autofocus is considered successful. If autofocusing was performed to scan a microscopic preparation, then when moving to the next field of view simultaneously with movement along the X and Y axes, the object table is moved to the position along the Z axis, which was assigned the maximum sharpness value in the previous field of view. This is done to save time, since adjacent fields of view are more likely to have a focused image at close z-coordinates. In this case, the image that has the highest sharpness value is used to obtain a digitized microscopic preparation or it is displayed on a monitor for further analysis by a clinical laboratory diagnostics doctor.

Example 1 of the implementation of the method.

To implement this method, a commercially available Zeiss microscope was used, equipped with a mechanized object table, a 10x objective and a Luminera Lt345R digital camera capable of shooting 75 frames per second and connected to a personal computer (PC) with installed software of the Vision family from West Medica.

A glass slide with a smear of peripheral blood applied to it was placed on an automated microscope stage. Then the autofocus procedure was started. The initial coordinates of the object table were recorded. As part of the first stage, the object table made a “jerk” towards the lens at a speed V _l =12 mm/s to a distance J _l . =20 µm. After that, the object table began to move continuously from the lens at a speed of 77 μm/s, at the same time, the digital camera sent image frames for analysis on a PC to determine the sharpness value S of each frame, the sharpness calculation was carried out according to formula (1). The focusing range was 50 µm. After passing in the first direction and not fulfilling condition 1, that is, without obtaining the sharpest image, the object table began to move in the opposite direction in accordance with condition 2.

The object table moved in the opposite direction until the PC received a focused image. Since the sharpness S grew rather quickly, that is, condition 3 was not met, the transition to a long focus did not occur, and a focused image was obtained within the first stage. Autofocus was considered complete. AF time per field of view using the first stage was approximately 0.52 seconds.

The object table motion graph and the sharpness graph are shown in Fig. 5 (a, b) according to example 1.

Example 2 of the implementation of the method.

A glass slide with a smear of peripheral blood applied to it was placed on an automated microscope stage. Then the autofocus procedure was started. The initial coordinates of the object table were recorded. As part of the first stage, the object table made a “jerk” towards the lens at a speed V _l =12 mm/s to a distance J _l . = 20 µm. After that, the object table began to move continuously from the lens at a speed of 77 µm/s, at the same time, the digital camera sent image frames for analysis on a PC to determine the sharpness value of each frame using formula (1). The focusing range was 50 µm.

After passing in the first direction, without obtaining the sharpest image, and without fulfilling condition 1, the object table began to move in the opposite direction in accordance with condition 2. After passing the range in the opposite direction and also without fulfilling condition 1, a transition to the second stage of focusing occurred .

The table returned to its original position. Then the table made a "jerk" towards the lens at a speed V _l =12 mm/s to a distance J _l . = 150 µm. After that, it started moving away from the lens at a speed of 350 µm/s. After obtaining the sharpest image and thus fulfilling condition 1, the stage was moved to the position where the sharpest image was obtained to repeat step 1. The autofocus time per field of view was approximately 3.5 s.

In FIG. 6 (a, b) shows the table movement graph and the sharpness graph obtained by implementing the above method according to example 2.

Claims (11)

A method for autofocusing when digitizing a microscopic preparation, which includes moving the microscope object table along the vertical Z axis to a position corresponding to the maximum image sharpness of the sample of the microscopic preparation, characterized in that autofocusing is carried out in two stages, the first stage includes continuous movement of the object table with the sample along the Z axis in the direction of the microscope lens at a maximum speed by a given value, then the object table is continuously moved from the lens at a given speed with simultaneous acquisition of images, transferring them to a personal computer, calculating the sharpness S of the images obtained, assigning the coordinates of the object table along the Z axis and the calculated values sharpness of the obtained images in the process of moving the subject table, while the resulting image in the RGB color model is converted into the YUV color model and the sharpness S of each image is calculated according to the formula:

, Where

– serial number of the current image pixel horizontally,

– serial number of the current pixel of the image along the vertical,

Y is the value of the brightness component of the image pixel in the YUV color model,

is the difference between the values of the brightness component of adjacent pixels horizontally,

- the difference between the values of the brightness component of neighboring pixels along the vertical, n - number of horizontal pixels, m - the number of vertical pixels, sharpness values S and coordinates along the Z axis for all images are sent to the memory of the personal computer, the focused image is fixed by selecting the coordinates of the object table with the sample on the Z axis, satisfying the condition under which during the passage of the object table of the specified range along the Z axis on three images in a row sharpness decreases before the “peak” of sharpness and after the “peak” of sharpness, and if the specified condition is met, the first stage of autofocusing is considered completed, and the movement of the object table with the sample is stopped before reaching the end of the specified range, and if the specified condition is not met, the second stage is carried out with increased focusing range, the speed of movement of the object table and a given distance to obtain a focused image, while calculating the sharpness of the images, while the second stage is carried out until the specified condition is met.

Images