CN114967093A

CN114967093A - Automatic focusing method and system based on microscopic hyperspectral imaging platform

Info

Publication number: CN114967093A
Application number: CN202210687806.7A
Authority: CN
Inventors: 李庆利; 孙星宇; 王妍
Original assignee: East China Normal University
Current assignee: East China Normal University
Priority date: 2022-06-16
Filing date: 2022-06-16
Publication date: 2022-08-30
Anticipated expiration: 2042-06-16
Also published as: CN114967093B

Abstract

The invention discloses an automatic focusing method and system based on a microscopic hyperspectral image, wherein the method comprises the following steps: acquiring a down-sampling image; the method comprises the steps of obtaining field definition based on a down-sampled image, judging the field definition, obtaining an image sequence based on a judgment result, obtaining a forward difference sequence and a reverse difference sequence based on the image sequence, and obtaining the clearest image position based on the forward difference sequence and the reverse difference sequence. By the technical scheme, the accuracy and the real-time performance of acquiring the hyperspectral image can be improved, and the optimal focusing position of the positioned object can be acquired.

Description

Automatic focusing method and system based on microscopic hyperspectral imaging platform

Technical Field

The invention relates to the technical field of microscopic imaging, in particular to an automatic focusing method and system based on a microscopic hyperspectral imaging platform.

Background

The hyperspectral image has extremely rich spatial information and spectral information, and a new idea is provided for classification and identification of objects. However, the spectral range of the conventional microscope is very limited, so that it is necessary to apply the microscopic hyperspectral imaging technology to the medical field. However, due to the uneven thickness of the object to be imaged, it is a common practice to acquire hyperspectral image data field by using a digital scanning microscope. The position of the quasi-focal planes of different regions of the object being imaged is unknown, but the quasi-focal planes in adjacent regions of the object are within a certain range. Therefore, there is a need for improvement in obtaining sharp images of objects and in the time to obtain sharp images.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides an automatic focusing method and system based on a microscopic hyperspectral imaging platform, so that the accuracy and the real-time performance of acquiring a hyperspectral image are improved, and the optimal focusing position of a positioned object can be acquired.

In order to achieve the technical purpose, the invention provides an automatic focusing method based on a microscopic hyperspectral image, which comprises the following steps:

acquiring a down-sampling image; and acquiring the definition of a field of view based on the downsampled image, judging the definition of the field of view, acquiring an image sequence based on a judgment result, acquiring a forward difference sequence and a reverse difference sequence based on the image sequence, and acquiring the position of the clearest image based on the forward difference sequence and the reverse difference sequence.

Optionally, the process of acquiring the down-sampled image includes:

setting collection conditions, wherein the collection conditions comprise a wave band range, wave band number, exposure time, objective lens multiple and an image label;

and performing down-sampling on the image of the single view field based on the acquisition condition to obtain a down-sampled image.

Optionally, the process of acquiring the definition of the field of view includes:

and calculating the down-sampling image through a Laplace operator to obtain the field definition.

Optionally, the process of acquiring the image sequence includes:

judging the definition of the view field, if the definition of the view field is larger than an expected value, moving the electric objective table of the mobile microscope up and down, and acquiring an image sequence in a first range, otherwise, moving the lens up and down, acquiring a downsampling image sequence in a second range, calculating a Laplace evaluation value based on the downsampling image sequence, moving the electric objective table of the mobile microscope to the maximum value of the Laplace evaluation value, moving the electric objective table of the mobile microscope up and down, and acquiring the image sequence in the first range.

Optionally, the process of obtaining the clearest position includes:

and filtering the forward difference value array and the reverse difference value array by using a Laplacian operator, and taking the median of the filtering result and the first image position in the image sequence as the clearest image position.

On the other hand, in order to better achieve the technical object, the invention provides an automatic focusing system based on a microscopic hyperspectral image, which comprises:

the control computer, the optical microscope, the electric objective table of the mobile microscope, the light splitter, the acousto-optic tunable filter, the gray level camera and the color camera;

the grayscale camera and the color camera are used for acquiring a down-sampling image in the optical microscope, wherein the down-sampling image comprises a color image and a grayscale image;

the control computer is used for controlling the electric objective table of the mobile microscope to move, acquiring field definition based on a down-sampled image, judging the field definition, acquiring an image sequence based on a judgment result, acquiring a forward difference sequence and a reverse difference sequence based on the image sequence, and acquiring the clearest image position based on the forward difference sequence and the reverse difference sequence;

the acousto-optic tunable filter is used for displaying a down-sampling image and an image sequence in an optical microscope in real time.

Optionally, the method includes setting a grayscale camera and a color camera, wherein the acquisition conditions include a band range, a band number, exposure time, objective lens multiples and an image label; and performing down-sampling on the image of the single view field based on the acquisition condition to obtain a down-sampled image.

Optionally, the control computer includes a calculation module, and the calculation module is configured to calculate the downsampled image by using a laplacian operator to obtain the field definition.

Optionally, the control computer includes a determination module, the determination module is configured to determine the field of view definition, and if the field of view definition is greater than the expected value, move the electric stage of the mobile microscope up and down and acquire an image sequence in a first range, otherwise move the lens up and down and acquire a downsampling image sequence in a second range, calculate a laplacian evaluation value based on the downsampling image sequence, move the electric stage of the mobile microscope to a maximum value of the laplacian evaluation value, move the electric stage of the mobile microscope up and down and acquire the image sequence in the first range.

Optionally, the control computer includes a processing module, and the processing module is configured to filter the forward difference sequence and the reverse difference sequence by using a laplacian operator, and use a median between a filtering result and a first image position in the image sequence as a clearest image position.

The invention has the following technical effects:

according to the automatic focusing method based on the microscopic hyperspectral image, the image down-sampling is adopted, the calculation amount of definition is effectively reduced, and the efficiency is further improved through difference value calculation. The cost input of manual focusing is reduced, a clear view field can be efficiently and accurately obtained, and the microscopic digital image shooting efficiency is efficiently and accurately improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

FIG. 1 is a schematic flow chart of a method provided by an embodiment of the present invention;

fig. 2 is a schematic diagram of a system according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example one

As shown in fig. 1, the present invention provides an automatic focusing method based on a microscopic hyperspectral image, including:

moving the electric objective table of the mobile microscope to the interested view field, acquiring a down-sampling image of the single view field and storing the down-sampling image into a memory;

calculating the definition of the current field of view through a Laplacian operator, directly executing the step 4 if the definition of the current field of view is more than 6, and otherwise, continuously executing the step 3;

moving up and down by taking 24um as a step length, acquiring 5 downsampling picture sequences in a range of +/-48 um, calculating a Laplace evaluation value and moving to a maximum value position;

moving up and down by taking 6um as a step length, obtaining 9 image sequences in the range of +/-24 um in total, calculating two groups of forward and reverse difference value series of the image sequences, and searching for the minimum value in the series;

and filtering a group which does not meet the condition by using a Laplace operator to obtain the clearest position.

In the process of calculating the definition of the current field of view through a Laplace operator, setting data such as a wave band range, wave band number, exposure time, objective lens multiple of a shot image, image name and the like required by hyperspectral image acquisition, and manually or using an industrial controller to control a three-axis electric objective table of a microscope to move to an interested area; the image of 2048 pixels by 2048 pixels from the monoscopic field is down-sampled at 0.08 magnification and stored in memory. Calculating a Laplacian evaluation value of the image after the down-sampling, and directly carrying out the fine focusing in the step 4 if the evaluation value is more than 6;

in the process of calculating the Laplace evaluation value and moving to the maximum position, the Z axis of the moving object stage respectively stores the down-sampled images into a memory by taking 24um as a step length; respectively calculating corresponding Laplace definition evaluation values, and finally moving to an objective table coordinate corresponding to the highest evaluation value;

in the process of obtaining the clearest position, taking 6um as a step length, moving the Z axis of the objective table to respectively store the down-sampled images into a memory; obtaining 9 picture sequences, sequentially obtaining a group of number sequences A from a first picture according to coordinates, wherein the index of the first picture is 0, and subtracting the first picture to the last picture; sequentially subtracting the last sheet from the first sheet according to the coordinates to obtain another array B; finding the minimum number in the number sequence, and filtering out smaller points in the number sequence through a Laplacian evaluation value; the resulting point (if present in array a) and the median of the first image are the position of the sharpest image.

There are many considerations for achieving auto-focus by controlling the microscope motorized stage 270. The method comprises the selection of a coarse focusing step length, the number of sequences of coarse focusing images, the selection of a fine focusing step length, the number of sequences of fine focusing images and the selection of a range containing clear images. Since the distance between the quasi-focal planes of adjacent regions will vary depending on the thickness of the slice fabrication, but is typically maintained within ± 50 um. The step length of the rough focusing can be determined by the fine focusing, the step length of the fine focusing is selected to be 6um, at least 9 images are needed by adopting the image difference, so the step length of the rough focusing is set to be 24um, and the number of the image sequences of the rough focusing can be set to be 5 due to the limitation of the range.

The invention is based on the dual reduction of the image down-sampling and the image difference value on the automatic focusing operation amount of the microscopic hyperspectral image, and can realize accurate real-time automatic focusing on the basis.

The invention provides an automatic focusing method based on a microscopic hyperspectral image, the specific algorithm flow of the method can be seen in figure 1, and the experimental result data of the embodiment are as follows:

based on the step 100, the acousto-optic tunable filter 210 of the hyperspectral image is firstly set, the wave band range is set to be the expected wave band range, the real-time display wave band of the current industrial control machine is set to be 640nm, and a series of initialization settings are carried out according to the exposure time, the light intensity objective lens multiple of the halogen lamp light source 200 and the sample name. An interested view field is selected through an electric objective table 270 of a manual or industrial controller control microscope, or a shooting area is defined and moved to the first view field of the defined shooting area, and a downsampling image of the view field is obtained, namely, the current single view field 2048 × 2048 pixels and a grayscale image of 640nm wave band are downsampled at the multiplying power of 0.08;

then, step 110 is executed, the sharpness evaluation value (mean value) of the downsampled image obtained in step 100 is calculated through the laplacian operator, if the sharpness evaluation value is greater than 6, the fine focusing process in step 120 is directly executed, and if the sharpness evaluation value is not greater than 6, the step 115 is continuously executed;

in step 115, the electric stage 270 of the microscope is moved up and down with 24um as a step length, the image acquired by the grayscale camera 240 is refreshed in real time, down-sampling is performed with 0.08 magnification, and the down-sampled images are stored in the memory together. 5 picture sequences of the gray level images of the band 640nm of downsampling can be obtained in the range of +/-48 um, the Laplace evaluation values of the 5 images are respectively calculated to form an array of 5 elements, the array is traversed to find out the height of the microscope electric objective table 270 corresponding to the maximum value, and the height is moved to the position; the calculated resolution after Z-axis motion for a field of view is shown in table 1.

TABLE 1

Position(mm)	Sharpness
		-48	2.436
-24	3.369
		0	9.522
24	2.573
		48	2.409

With 24um as a step length, moving the object stage 270Z axis to respectively store the down-sampled images in the memory; the corresponding laplacian sharpness evaluation values are calculated, and finally the stage 270 coordinate corresponding to the value with the highest evaluation value is moved.

Since the maximum value in table 1 is 9.522, at the current position, no movement is required;

in step 120, the electric stage 270 of the microscope is moved up and down in steps of 6um, the similar operation as in step 115 is performed, the image acquired by the grayscale camera 240 is refreshed in real time, down-sampled at 0.08 magnification, and the down-sampled images are stored in the memory. An image sequence formed by 9 pieces of single-waveband gray level images of microscopic hyperspectrum can be acquired within the range of +/-24 um. In step 125 and step 130, we calculate two sets of difference arrays of a forward difference array with a length of 8 obtained by sequentially subtracting the ith image (1< i ≧ 9) from the first image in the image sequence and a reverse difference array with a length of 8 obtained by sequentially subtracting the ith image (9> i ≧ 1) from the last image in the image sequence, and find the minimum value in the arrays. The specific implementation data of this step is shown in table 2:

TABLE 2

Difference in positive direction	0-1	0-2	0-3	0-4	0-5	0-6	0-7	0-8
									Mean value	4.768	10.255	12.991	7.856	3.187	2.265	4.351	6.076
Difference in reversal	8-7	8-6	8-5	8-4	8-3	8-2	8-1	8-0
									Mean value	2.787	5.243	7.922	11.761	15.939	13.672	8.383	6.076

And 6um is used as the step length, and the Z axis of the moving object stage 270 stores the down-sampled images into the memory respectively. Obtaining 9 picture sequences, and sequentially reducing the first picture sequence to the last picture sequence according to coordinates to obtain a group of image mean value sequence A; sequentially reducing the last image to the first image according to the coordinates to obtain another group of image mean value array B;

according to the step 140, traversing the two arrays, finding out the down-sampling gray level image in the picture sequence corresponding to the minimum value, respectively calculating the definition evaluation values of the two images by using a laplacian operator, filtering out one image with a smaller evaluation value, and finally obtaining the position of the microscope electric objective table 270 corresponding to the image, namely the clearest position. The specific implementation of this example is:

finding the minimum number in the number sequence, and filtering out smaller points in the number sequence through a Laplacian evaluation value; the resulting point (if present in array a) and the median of the first image are the locations of the sharpest images. Excluding the position corresponding to 2.787 with low sharpness evaluation value, the position of the clearest image is the position corresponding to 0-6 in the above table, i.e. the midpoint between the 6 th image and the first image.

The automatic focusing method based on the microscopic hyperspectral image provided by the invention has the advantages that the operation is carried out by integrally using the down-sampled image, and the operation amount is greatly reduced. And because the difference value operation is adopted to replace the traditional definition operation, the calculation complexity is further reduced, and the real-time automatic focusing can be realized under the condition of ensuring the definition of the image. The single-view-field automatic focusing can be finished within 0.9s at the fastest speed. The cost investment of manual focusing is reduced, and the shooting efficiency of the digital microscopic hyperspectral image is efficiently and accurately improved.

Example two

As shown in fig. 2, an embodiment of the invention is based on a microscopic hyperspectral imaging system, the system comprising: an industrial control computer, an optical microscope, a programmable three-axis motorized stage 270, a beam splitter, an acousto-optic tunable filter 210, a grayscale camera 240 and a color camera 250. The industrial control computer provides a user-friendly operation interface and executes corresponding codes for user operation. In the above operations, one of the operations may be to perform iterative scanning on the acquired image based on the current position, and finally move to the in-focus position based on the scanning result, acquire the image, and refresh the real-time preview interface of the industrial control machine. In addition, the method also carries out self-adaptive optimization aiming at real-time performance, the optimization is mainly based on a calculated value of image definition evaluation, focusing calculation of finer granularity is directly carried out when the calculated value exceeds a certain threshold, and otherwise, focusing operation of coarse granularity is carried out firstly.

The embodiment of the present invention specifically describes an automatic focusing method based on the above system. The method comprises the steps of conducting down-sampling on the whole single-view-field collected image, directly entering fine-grained focusing calculation if the definition evaluation value of the down-sampled image at the current position is larger than a threshold value, and otherwise, obtaining a group of down-sampled image sequences through continuously moving an objective table 270, and collecting the calculated out-of-focus data. And finding the maximum value to reach the position close to the quasi-focus position, acquiring a group of down-sampled image sequences based on the current position with smaller step length, and comparing the difference values to obtain the real quasi-focus position. The automatic focusing method is effective to both color images and gray level images of the microscopic hyperspectral acquisition system.

The method for realizing the automatic focusing of the hyperspectral image mainly comprises the following steps of: setting data such as a wave band range, wave band number, exposure time, objective lens multiple of a shot image, image name and the like required by hyperspectral image acquisition; the method comprises the steps that BSQ image data combination mode is adopted for shooting, namely, images are shot according to wave bands, and an industrial control computer can display microscopic hyperspectral gray level images under the current wave bands and under the current position of a microscope objective table 270 in real time through an acousto-optic tunable filter 210; the image is transmitted into the automatic focusing algorithm 260, the current single view field is subjected to down-sampling, and the evaluation value of the definition of the down-sampled image is calculated to judge whether the image is greater than a threshold value; if the difference is larger than the preset value, fine focusing is directly carried out, otherwise, the large-step objective table 270 is moved to acquire an image sequence, the maximum value is found, and the maximum value is moved; and moving the object stage 270 in a small step length to obtain an image sequence, performing image difference to obtain a real focusing position, and moving the real focusing position to the position to finally finish the automatic focusing of the microscopic hyperspectral image.

The foregoing illustrates and describes the principles, general features, and advantages of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims

1. An automatic focusing method based on a microscopic hyperspectral image is characterized by comprising the following steps:

acquiring a down-sampling image; the method comprises the steps of obtaining field definition based on a down-sampled image, judging the field definition, obtaining an image sequence based on a judgment result, obtaining a forward difference sequence and a reverse difference sequence based on the image sequence, and obtaining the clearest image position based on the forward difference sequence and the reverse difference sequence.

2. The method of claim 1, further comprising:

the process of acquiring a downsampled image includes:

3. The method of claim 1, further comprising:

the process of acquiring the definition of the field of view comprises the following steps:

4. The method of claim 1, further comprising:

the process of acquiring a sequence of images includes:

5. The method of claim 1, further comprising:

the process of obtaining the clearest position comprises the following steps:

6. An automatic focusing system based on a microscopic hyperspectral image is characterized in that: comprises the steps of (a) preparing a mixture of a plurality of raw materials,

7. The system of claim 6, wherein:

acquiring conditions including a wave band range, wave band number, exposure time, objective lens multiple and an image label by setting a gray camera and a color camera; and performing down-sampling on the image of the single view field based on the acquisition condition to obtain a down-sampled image.

8. The system of claim 6, wherein:

the control computer comprises a calculation module, and the calculation module is used for calculating the down-sampling image through a Laplace operator to obtain the definition of a view field.

9. The system of claim 6, wherein:

the control computer comprises a judging module, wherein the judging module is used for judging the definition of a view field, if the definition of the view field is larger than an expected value, the electric objective table of the mobile microscope is moved up and down, an image sequence is obtained in a first range, otherwise, the lens is moved up and down, a down-sampling image sequence is obtained in a second range, a Laplace evaluation value is calculated based on the down-sampling image sequence, the electric objective table of the mobile microscope is moved to the position of the maximum value of the Laplace evaluation value, the electric objective table of the mobile microscope is moved up and down, and the image sequence is obtained in the first range.

10. The system of claim 6, wherein:

the control computer comprises a processing module, wherein the processing module is used for filtering the forward difference value array and the reverse difference value array through a Laplacian operator, and taking the filtering result and the median of the first image position in the image sequence as the clearest image position.