CN111462075A - Rapid refocusing method and system for full-slice digital pathological image fuzzy area - Google Patents
Rapid refocusing method and system for full-slice digital pathological image fuzzy area Download PDFInfo
- Publication number
- CN111462075A CN111462075A CN202010242168.9A CN202010242168A CN111462075A CN 111462075 A CN111462075 A CN 111462075A CN 202010242168 A CN202010242168 A CN 202010242168A CN 111462075 A CN111462075 A CN 111462075A
- Authority
- CN
- China
- Prior art keywords
- image
- full
- digital pathological
- slice
- slice digital
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000001575 pathological effect Effects 0.000 title claims abstract description 191
- 238000000034 method Methods 0.000 title claims abstract description 42
- 230000007170 pathology Effects 0.000 claims abstract description 38
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 238000012216 screening Methods 0.000 claims abstract description 10
- 238000012549 training Methods 0.000 claims description 36
- 238000013136 deep learning model Methods 0.000 claims description 28
- 238000012795 verification Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 230000011218 segmentation Effects 0.000 claims description 2
- 238000011496 digital image analysis Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 7
- 238000011156 evaluation Methods 0.000 description 6
- 238000010200 validation analysis Methods 0.000 description 4
- 238000003745 diagnosis Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000001000 micrograph Methods 0.000 description 2
- 238000011179 visual inspection Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000010186 staining Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a quick refocusing method of a full-slice digital pathological image fuzzy area, which is used for detecting a full-slice digital pathological image through a full-slice digital pathological image fuzzy area detection unit, rapidly obtaining the focusing state of each image subblock in the full-slice digital pathological image, screening out defocusing image subblocks according to the focusing state, converting the focusing state of the defocusing image subblocks into an actual defocusing distance index, and finally adjusting the moving speed of a carrier table in a digital pathological section scanner along a Z axis in real time according to the defocusing distance index to realize refocusing of a pathological section area corresponding to the defocusing image subblocks. The refocusing method provided by the invention has high precision and high speed, avoids subjective dependence of artificial focusing, and can effectively assist a pathology expert in quickly detecting a full-slice digital pathological image fuzzy area and analyzing by a subsequent computer image analysis system.
Description
Technical Field
The invention relates to the technical field of digital image processing, in particular to a rapid refocusing method and a rapid refocusing system for a fuzzy area of a full-slice digital pathological image.
Background
In recent years, with the rapid development of pathology and computer technology, full-slice digital pathology images are widely used in the fields of clinical diagnosis and pathology research. In addition, the full-slice digital pathological image is processed and analyzed through an image algorithm or a machine learning method, and the method has great reference value for assisting a pathologist in quick diagnosis and computer automatic diagnosis.
The digital pathological image is obtained by converting pathological sections into digital sections through a digital pathological section scanner. The core function of the digital pathological section scanner is to acquire digital images of pathological sections for a pathologist to read, store, analyze and the like. Due to various uncertain factors existing in the processes of slice making and microscopic scanning, the acquired digital image usually has local unclear/out-of-focus conditions. This reduces the number of pathological sections available to a pathologist for accurate diagnosis and, further, hinders the performance of computer image analysis systems. Therefore, the fast and effective digital pathology image fuzzy region refocusing method is crucial to the development of digital pathology.
The existing refocusing method for the blurred region of the full-slice digital pathological image generally adopts a visual inspection method to refocus the blurred region of the digital image, the time spent is long, and the visual inspection can cause subjective evaluation, thereby causing high variability between the inside of an observer and the observer.
Disclosure of Invention
The invention provides a rapid refocusing method and a rapid refocusing system for a fuzzy area of a full-section digital pathological image, which are used for overcoming the defects of long time consumption, low accuracy and the like in the prior art.
In order to achieve the above object, the present invention provides a fast refocusing method for a blurred region of a full-slice digital pathological image, comprising:
obtaining a pathological section;
collecting a full-section digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy area detection unit, and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy area detection unit;
screening out the out-of-focus image sub-blocks in each image sub-block according to the focusing state;
converting the focus state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index to realize refocusing of the pathological section area corresponding to the defocus image sub-block.
In order to achieve the above object, the present invention further provides a fast refocusing system for a blurred region of a full-slice digital pathological image, comprising:
the image acquisition module is used for acquiring pathological sections; collecting a full-section digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
the focusing state judging module is used for inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy area detection unit and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy area detection unit;
the screening module is used for screening out the out-of-focus image subblocks in each image subblock according to the focusing state;
the conversion module is used for converting the focusing state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and the focusing module is used for adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index so as to realize refocusing of the pathological section area corresponding to the defocus image sub-block.
To achieve the above object, the present invention further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.
Compared with the prior art, the invention has the beneficial effects that:
the method for rapidly refocusing the fuzzy area of the full-slice digital pathological image detects the full-slice digital pathological image through the fuzzy area detection unit of the full-slice digital pathological image, rapidly obtains the focusing state of each image subblock in the full-slice digital pathological image, screens out the defocused image subblock according to the focusing state, converts the focusing state of the defocused image subblock into an actual defocused distance index, and finally adjusts the moving speed of a carrying table in a digital pathological section scanner along the Z axis in real time according to the defocused distance index to realize refocusing of the pathological section area corresponding to the defocused image subblock. The full-slice digital pathological image is only defocused in a partial visual field area to cause partial area blurring, so the refocusing method provided by the invention utilizes the full-slice digital pathological image blurring area detection unit to divide the full-slice digital pathological image into non-overlapping image sub-blocks with fixed pixel sizes, respectively judges the focusing state of each image sub-block, and quickly screens the defocused image sub-blocks. The refocusing method provided by the invention has high precision and high speed, avoids subjective dependence of artificial focusing, and can effectively assist a pathology expert in quickly detecting a full-slice digital pathological image fuzzy area and analyzing by a subsequent computer image analysis system.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
Fig. 1 is a flowchart of a fast refocusing method for a blurred region of a full-slice digital pathological image provided by the invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.
The invention provides a rapid refocusing method for a blurred region of a full-slice digital pathological image, which comprises the following steps of:
101, obtaining a pathological section;
the pathological sections comprise pathological sections of different patients and different staining mechanisms.
The pathological section may be a cell sample or a tissue section.
102, acquiring a full-slice digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
the digital pathological images of the whole section of the pathological section are collected by a digital pathological section scanner, and the pathological section is carried on an objective table of the digital pathological section scanner in the collecting process.
And acquiring digital pathological images with different focusing degrees so as to be beneficial to model training of the fuzzy region detection unit of the full-slice digital pathological image.
103, inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy region detection unit, and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy region detection unit;
the full-slice digital pathological image fuzzy area detection unit divides the full-slice digital pathological image into a plurality of image sub-blocks, and then detects the focusing state of each image sub-block.
104 screening out the out-of-focus image sub-blocks in each image sub-block according to the focusing state;
the focusing state comprises focusing and different degrees of defocusing, and all the image sub-blocks in the different degrees of defocusing are defocused image sub-blocks.
105 converting the focus state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and converting, namely correspondingly endowing different defocus distance index values for different focusing states.
106, adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index, and realizing refocusing of the pathological section area corresponding to the defocus image sub-block.
The general microscopic image focusing process is: starting to collect microscopic images of pathological sections carried on the object stage from the position of the object stage, and then calculating the focusing evaluation value (namely the focusing degree) of the images; then the objective table moves to the next position along the Z axis at a constant moving speed, and a microscopic image of the position is collected, and the focusing evaluation value of the image is calculated; by continuously moving the stage at a constant movement speed, a microscope image is acquired at each position and the focus evaluation value of the microscope image is calculated, and in this process, a maximum focus evaluation value is found, and the position of the Z axis corresponding to the maximum focus evaluation value is the best focal plane position. In the existing method, the movement of the objective table moves at a constant speed, and when the objective table is far away from the focusing position, the objective table needs to spend a long time for focusing.
The method for rapidly refocusing the blurred region of the full-slice digital pathological image adopts a depth learning model to calculate the focusing degree of the microscopic image collected at each position, converts the focusing degree at the position into an out-of-focus distance index, adjusts the moving speed of the objective table along the Z axis by using the distance index, moves the objective table to the next position along the Z axis at the adjusted speed, collects the microscopic image at the position, calculates the focusing degree of the microscopic image collected at each position by using the depth learning model, converts the focusing degree at the position into the out-of-focus distance index, and readjusts the moving speed of the objective table along the Z axis; this process is repeated until the defocus distance index corresponding to the acquired microscopic image is 0, that is, the Z-axis moving speed is 0, and this position is the focus position. According to the refocusing method provided by the invention, the depth learning model is adopted to calculate the focusing degree of the microscopic image acquired at each position, the calculation speed is high, and the precision is high; meanwhile, the moving speed of the object stage along the Z axis is not constant, but the moving speed of the object stage along the Z axis is adjusted in real time through the defocus distance index obtained through real-time calculation, when the object stage is far away from the focal plane, the moving speed is high, the object stage can be quickly moved to a position close to the focal plane, and when the object stage is close to the focal plane, the moving speed is low, and the position of the focal plane can be accurately found. Compared with uniform movement, the method can greatly reduce the number of moving steps and find a clear focal plane position.
In one embodiment, for step 102, acquiring a full-slice digital pathology image of the pathology slice comprises:
201, scanning the central area of the pathological section by using a digital pathological section scanner, focusing and collecting a focused digital pathological image of the pathological section;
focusing, which includes manually selecting a number of pre-focus points and then fine-tuning the auto-focus values of the selected pre-focus points to achieve focusing.
202, setting a plurality of deviation values, and disturbing the focused focal position according to the deviation values;
the offset value is a distance value for moving the stage of the digital pathological section scanner relative to the focus point position.
The stage of the digital pathological section scanner is movable along the microscope Z-axis to adjust focus. The magnification of the microscope objective lens of the digital pathological section scanner in this embodiment may be 20X (i.e., 20 times) or 40X.
The offset value includes a positive value indicating that the stage is moving upward and a negative value indicating that the stage is moving downward.
203 acquiring a fuzzy digital pathological image of the central area of the pathological section at the focus point position after each deviation value disturbance.
The central region is generally a square region with a side length S, but is not limited to a square region, and may be a region of another shape.
The larger the absolute value of the offset value is, the farther the offset value is from the focus point, and the more blurred the acquired digital pathological image is.
In a next embodiment, for step 202, a number of offset values are set, including:
setting a plurality of deviation values delta according to the size of the central area of the pathological section, wherein the deviation values delta
Δ∈{-3.5μm，-3μm，-2.5μm，-2μm，-1.5μm，-0.5μm，0.5μm，1.5μm，2μm，2.5μm}。
In this embodiment, the offset value Δ has 10 values, and one full-slice digital pathological image is acquired under each offset value Δ, so that 10 full-slice digital pathological images with different degrees of blur and 1 clear full-slice digital pathological image can be acquired for one pathological slice.
In another embodiment, for step 103, the full-slice digital pathological image is input into a full-slice digital pathological image fuzzy area detection unit which is constructed in advance, wherein the built-in program of the full-slice digital pathological image fuzzy area detection unit is as follows:
301 obtaining a full-slice digital pathology image;
all the acquired full-slice digital pathological images contain digital pathological images in different focusing states so as to enhance the comprehensiveness of the training set and the verification set and improve the detection precision of the deep learning model obtained by training.
302, dividing the full-slice digital pathological image into non-overlapping image sub-blocks with fixed pixel sizes, randomly extracting a plurality of image sub-blocks to form a training set and a verification set, and marking the image sub-blocks in the training set and the verification set;
303, inputting the training set into a pre-constructed deep learning model, and training the deep learning model to optimize the hyper-parameters of the deep learning model;
the hyper-parameters include the kernel size, number of layers, learning rate, etc. of the model.
304, inputting the verification set into the trained deep learning model, and verifying the trained deep learning model; if the verification is passed, the next step is carried out to detect the full-slice digital pathological image to be detected; if the verification fails, returning to the previous step to perform model training again;
305, preprocessing a full-slice digital pathological image to be detected to obtain a characteristic diagram of the full-slice digital pathological image to be detected; and dividing the characteristic graph into a plurality of non-overlapping image sub-blocks with fixed pixel sizes to be detected, and inputting the image sub-blocks into the trained deep learning model for classification to obtain the focusing state of each image sub-block to be detected.
In a certain embodiment, for step 302, segmenting the full-slice digital pathology image into non-overlapping image sub-blocks of fixed pixel size, randomly extracting a number of image sub-blocks to form a training set and a validation set, and labeling the image sub-blocks in the training set and the validation set, comprises:
3021 segmenting the focused digital pathology image into non-overlapping sub-blocks of fixed pixel size images, obtaining a first set of images; dividing the digital pathological images with different fuzzy degrees into non-overlapping image sub-blocks with fixed pixel sizes to obtain a second image set;
if the magnification of the microscope objective of the digital pathological section scanner is 20X, the size of the image sub-block is 300 × 300;
if the magnification of the microscope objective of the digital pathological section scanner is 40X, the size of the image sub-block is 128 × 128.
3022 randomly extracting image sub-blocks from the first image set and the second image set, and mixing to obtain a sample set;
when training the model, the number of the positive samples and the number of the negative samples are equivalent, so that the training effect of the model is good. When the images are collected, the fuzzy images are far larger than the clear images, if the images are not extracted separately, the probability of the fuzzy images in the extracted image sub-blocks is very high, so that most of the training set can be the fuzzy images, and the training of the model is not facilitated.
3023 randomly extracting 8-15% of image sub-blocks from the sample set to form a validation set, and forming a training set by the rest image sub-blocks;
3024 labeling the image sub-blocks in the training set and the verification set, specifically:
marking the image subblocks with the focus offset value delta between [ -0.5 μm,0.5 μm ] as focus image blocks, wherein the corresponding labels are 0;
marking image subblocks with focus offset values delta between 0.5 mu m and 1.5 mu m as first positive blurred image blocks, wherein the corresponding labels are 1;
marking the image subblocks with the focus offset value delta between [1.5 μm and 2.5 μm ] as second positive blurred image blocks, wherein the corresponding label is 2;
marking image sub-blocks having a focus offset value Δ between [ -1.5 μm, -0.5 μm ] as a first negative blurred image block, the corresponding label being-1;
marking image sub-blocks having a focus offset value Δ between [ -2.5 μm, -1.5 μm ] as second negative blur image blocks, the corresponding label being-2;
an image sub-block having a focus offset value Δ between-3.5 μm and-2.5 μm is labeled as a third negative blurred image block, with the corresponding label-3.
In a further embodiment, before training the model, the image sub-blocks in the training set and the validation set are preprocessed, including:
firstly, linearly scaling the color intensity value of each image subblock in the training set and the verification set to 0-1;
the image sub-block color intensity values are then converted to have an average value of zero.
In another embodiment, for step 303, inputting the training set into a pre-constructed deep learning model, and training the deep learning model, wherein the deep learning model is a convolutional neural network model and sequentially includes six convolutional layers, two fully-connected layers, and a softmax classification layer;
a largest pooling layer is arranged behind the third, fourth, fifth and sixth convolutional layers in the six convolutional layers;
and dropout layers with the probabilities of 0.7 and 0.5 are respectively arranged behind the two full-connection layers.
In this embodiment, the input to the deep learning model is a 128 × 128 pixel size image sub-block, and if the input image sub-block is not 128 × 128, the model automatically scales its size to 128 × 128.
In the deep learning model in this embodiment, the convolution kernel size of the first convolution layer (conv1) is 5 × 5 and the step size is 1, the convolution kernel sizes of the following five convolution layers (conv2, conv3, conv4, conv5 and conv6) are 3 × 3 and the step size is 1, and the convolution kernel size of the maximum pooling layer (maxpool) is 3 × 3 and the step size is 2.
The feature map numbers of the top 10 layers (conv1, conv2, conv3, maxpool, conv4, maxpool, conv5, maxpool, conv6, maxpool) of the deep learning model in the present embodiment are 64, 128, 256, respectively.
The last layer of the deep learning model is a softmax classification layer, and the probability that the image subblocks belong to each category (the focused image block, the first positive blurred image block, the second positive blurred image block, the first negative blurred image block, the second negative blurred image block and the third negative blurred image block) and the labels (0, 1, 2, -1, -2 and-3) of the categories are output by adopting a cross entropy function.
In the next embodiment, for step 304, inputting the verification set into the trained deep learning model, and verifying the trained deep learning model; if the verification is passed, the next step is carried out to detect the full-slice digital pathological image to be detected; if the verification fails, returning to the previous step to perform model training again, wherein the model training comprises the following steps:
3041 inputting the image sub-blocks in the verification set into the trained deep learning model, and comparing the output labels with the labels of the image sub-blocks to obtain the precision of the trained deep learning model;
such as: if there are 100 image sub-blocks in the verification set and 95 image sub-blocks pass the verification (i.e., the output label corresponds to the true mark), the classification accuracy of the deep learning model is 95%.
And (5) verifying the quality of the trained model, namely the classification precision of the model. If the classification accuracy is too low, more training data sets are needed to retrain the model.
3042 if the precision is not less than the preset precision threshold, the verification is passed, and go to step 305;
3043 if the accuracy is less than the preset accuracy threshold, the verification fails and the process returns to step 303.
In the training process, an initial learning rate is set to be 0.1, then when the error does not decrease, the learning rate is reduced to be 0.01, and the iteration number is set to be 60 × 104。
The output feature map x of a certain layer of the model can be represented as x ═ g (wu + b), where w and b are parameters to be learned during model training, g (·) is an activation function, specifically, g ═ max (0, wu + b), and u is an input vector of the current layer.
The embodiment optimizes hyper-parameters (kernel size, number of layers, learning rate, etc.) of the convolutional neural network model using grid search and cross validation.
In a next embodiment, for step 305, the pre-processing of the full-slice digital pathology image to be detected comprises:
3051, zooming the to-be-detected full-slice digital pathological image to a magnification ratio of 1 to obtain an original digital pathological image;
3052 setting pixel threshold, and extracting cell region or tissue region of the original digital pathological image by using threshold segmentation method;
3053, mapping the cell region or the tissue region to the full-slice digital pathological image to be detected to obtain a characteristic diagram of the full-slice digital pathological image to be detected, specifically:
and zooming the cell area or the tissue area to the magnification (20X or 40X) of the full-slice digital pathological image to be detected, namely obtaining the characteristic map of the full-slice digital pathological image to be detected.
The size of the characteristic image is the same as that of the full-section digital pathological image, and the characteristic image is a binary mask image, namely the background is filled with black, and the foreground is a cell region or a tissue region.
In another embodiment, for step 105, the focus state of the out-of-focus image sub-block is converted into an actual out-of-focus distance index, specifically:
if the out-of-focus image sub-block is the first positive blurred image block, the numerical value of the assigned out-of-focus distance index dis is 1, that is, dis is 1;
if the out-of-focus image sub-block is the second positive blurred image block, the numerical value of the assigned out-of-focus distance index dis is 2, that is, dis is 2;
if the out-of-focus image sub-block is the first negative blurred image block, the numerical value of the assigned out-of-focus distance index dis is-1, namely dis is-1;
if the out-of-focus image sub-block is the second negative blurred image block, the numerical value of the assigned out-of-focus distance index dis is-2, namely dis is-2;
if the out-of-focus image sub-block is the third negative blur image block, the numerical value of the assigned out-of-focus distance index dis is-3, that is, dis is-3.
In a next embodiment, for step 106, adjusting the speed of the stage moving along the Z-axis in the digital pathological section scanner according to the defocus distance index to achieve refocusing of the pathological section region corresponding to the defocus image sub-block includes:
601, calculating and obtaining the moving speed v of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index;
v＝0.2×V×dis (1)
wherein, V is the default moving speed of the objective table; dis is the defocus distance index.
602 moving the objective table at the moving speed v, and acquiring digital pathological images of pathological section areas corresponding to the out-of-focus image sub-blocks;
603 inputting the digital pathological image into the full-slice digital pathological image fuzzy region detection unit to obtain the focusing state of the digital pathological image; if the digital pathological image is in a focusing state, finishing refocusing; if the digital pathological image is in the out-of-focus state, returning to the previous step, specifically:
if the digital pathological image is in a focusing state (namely the output label of the fuzzy region detection unit of the full-slice digital pathological image is 0), refocusing is finished, and a refocused image of the fuzzy region of the full-slice digital pathological image is obtained;
if the digital pathological image is out of focus, the method returns to the step 105 until the digital pathological image is in focus.
The invention also provides a rapid refocusing system for the fuzzy area of the full-slice digital pathological image, which comprises the following components:
the image acquisition module is used for acquiring pathological sections; collecting a full-section digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
the focusing state judging module is used for inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy area detection unit and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy area detection unit;
the screening module is used for screening out the out-of-focus image subblocks in each image subblock according to the focusing state;
the conversion module is used for converting the focusing state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and the focusing module is used for adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index so as to realize refocusing of the pathological section area corresponding to the defocus image sub-block.
The invention further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A fast refocusing method for a blurred region of a full-slice digital pathological image is characterized by comprising the following steps:
obtaining a pathological section;
collecting a full-section digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy area detection unit, and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy area detection unit;
screening out the out-of-focus image sub-blocks in each image sub-block according to the focusing state;
converting the focus state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index to realize refocusing of the pathological section area corresponding to the defocus image sub-block.
2. The method for fast refocusing of blurred regions of full-slice digital pathology image of claim 1, wherein acquiring full-slice digital pathology image of said pathology slice comprises:
scanning the central area of the pathological section by using a digital pathological section scanner, focusing and collecting a focused digital pathological image of the pathological section;
setting a plurality of deviation values, and disturbing the focused focal position according to the deviation values;
and acquiring a fuzzy digital pathological image of the central area of the pathological section at the focus point position disturbed by each deviation value.
3. The method for fast refocusing of blurred regions of full-slice digital pathology images of claim 2, wherein setting a number of offset values comprises:
setting a plurality of deviation values delta according to the size of the central area of the pathological section, wherein the deviation values delta
Δ∈{-3.5μm，-3μm，-2.5μm，-2μm，-1.5μm，-0.5μm，0.5μm，1.5μm，2μm，2.5μm}。
4. The method for rapidly refocusing the blurred region of the full-slice digital pathological image according to claim 1, wherein the full-slice digital pathological image is input into a pre-constructed blurred region detection unit of the full-slice digital pathological image, and the built-in program of the blurred region detection unit of the full-slice digital pathological image is as follows:
acquiring a full-slice digital pathological image;
dividing the full-slice digital pathological image into non-overlapping image sub-blocks with fixed pixel sizes, randomly extracting a plurality of image sub-blocks to form a training set and a verification set, and marking the image sub-blocks in the training set and the verification set;
inputting the training set into a pre-constructed deep learning model, and training the deep learning model to optimize the hyper-parameters of the deep learning model;
inputting the verification set into a trained deep learning model, and verifying the trained deep learning model; if the verification is passed, the next step is carried out to detect the full-slice digital pathological image to be detected; if the verification fails, returning to the previous step to perform model training again;
preprocessing a full-slice digital pathological image to be detected to obtain a characteristic diagram of the full-slice digital pathological image to be detected; and dividing the characteristic graph into a plurality of non-overlapping image sub-blocks with fixed pixel sizes to be detected, and inputting the image sub-blocks into the trained deep learning model for classification to obtain the focusing state of each image sub-block to be detected.
5. The method of claim 4, wherein the method of fast refocusing a blurred region of a full-slice digital pathology image comprises dividing the full-slice digital pathology image into non-overlapping image sub-blocks of fixed pixel size, randomly extracting a plurality of image sub-blocks to form a training set and a verification set, and labeling the image sub-blocks in the training set and the verification set, comprising:
dividing the focused digital pathological image into non-overlapping image sub-blocks with fixed pixel sizes to obtain a first image set; dividing the digital pathological images with different fuzzy degrees into non-overlapping image sub-blocks with fixed pixel sizes to obtain a second image set;
randomly extracting a plurality of image sub-blocks with the same quantity from the first image set and the second image set respectively, and mixing to form a sample set;
randomly extracting 8-15% of image sub-blocks from the sample set to form a verification set, and forming a training set by the rest image sub-blocks;
and marking the image subblocks in the training set and the verification set respectively.
6. The method for fast refocusing a blurred region of a full-slice digital pathological image as claimed in claim 4, wherein the training set is input into a pre-constructed deep learning model, and the deep learning model is trained, wherein the deep learning model is a convolutional neural network model and sequentially comprises six convolutional layers, two full-link layers and one softmax classification layer;
a largest pooling layer is arranged behind the third, fourth, fifth and sixth convolutional layers in the six convolutional layers;
and dropout layers with the probabilities of 0.7 and 0.5 are respectively arranged behind the two full-connection layers.
7. The method for fast refocusing blurred regions of full-slice digital pathological images of claim 4, wherein the preprocessing of the full-slice digital pathological image to be detected comprises:
zooming the to-be-detected full-slice digital pathological image to a magnification ratio of 1 to obtain an original digital pathological image;
setting a pixel threshold value, and extracting a cell area or a tissue area of the original digital pathological image by using a threshold value segmentation method;
and mapping the cell area or the tissue area to the full-slice digital pathological image to be detected to obtain a characteristic diagram of the full-slice digital pathological image to be detected.
8. The method for fast refocusing a blurred region of a full-slice digital pathological image according to claim 1, wherein the refocusing of the pathological section region corresponding to the out-of-focus image sub-block is realized by adjusting the moving speed of a carrier in the digital pathological section scanner along the Z-axis according to the out-of-focus distance index, comprising:
calculating and obtaining the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index;
moving the objective table at the moving speed, and acquiring digital pathological images of pathological section areas corresponding to the out-of-focus image sub-blocks;
inputting the digital pathological image into the full-slice digital pathological image fuzzy region detection unit to obtain the focusing state of the digital pathological image; if the digital pathological image is in a focusing state, finishing refocusing; and if the digital pathological image is in the out-of-focus state, returning to the previous step.
9. A fast refocusing system for blurred regions of full-slice digital pathology images, comprising:
the image acquisition module is used for acquiring pathological sections; collecting a full-section digital pathological image of the pathological section; the full-slice digital pathology image comprises a focusing digital pathology image and digital pathology images with different fuzzy degrees;
the focusing state judging module is used for inputting the full-slice digital pathological image into a pre-established full-slice digital pathological image fuzzy area detection unit and obtaining the focusing state of each image sub-block in the full-slice digital pathological image through the full-slice digital pathological image fuzzy area detection unit;
the screening module is used for screening out the out-of-focus image subblocks in each image subblock according to the focusing state;
the conversion module is used for converting the focusing state of the out-of-focus image sub-block into an actual out-of-focus distance index;
and the focusing module is used for adjusting the moving speed of the object carrying table in the digital pathological section scanner along the Z axis according to the defocus distance index so as to realize refocusing of the pathological section area corresponding to the defocus image sub-block.
10. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program implements the steps of the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242168.9A CN111462075B (en) | 2020-03-31 | 2020-03-31 | Rapid refocusing method and system for full-slice digital pathological image fuzzy region |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242168.9A CN111462075B (en) | 2020-03-31 | 2020-03-31 | Rapid refocusing method and system for full-slice digital pathological image fuzzy region |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111462075A true CN111462075A (en) | 2020-07-28 |
| CN111462075B CN111462075B (en) | 2023-12-15 |
Family
ID=71685122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010242168.9A Active CN111462075B (en) | 2020-03-31 | 2020-03-31 | Rapid refocusing method and system for full-slice digital pathological image fuzzy region |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111462075B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112037153A (en) * | 2020-09-08 | 2020-12-04 | 哈尔滨工业大学 | Full-slice digital imaging quasi-focus restoration method based on quasi-focus distance prior |
| CN112037154A (en) * | 2020-09-08 | 2020-12-04 | 哈尔滨工业大学 | High-precision quasi-focus restoration method for full-slice digital imaging and double photographing |
| CN112053304A (en) * | 2020-09-08 | 2020-12-08 | 哈尔滨工业大学 | Rapid focusing restoration method for single shooting of full-slice digital imaging |
| CN112070661A (en) * | 2020-09-08 | 2020-12-11 | 哈尔滨工业大学 | Full-slice digital imaging rapid automatic focusing method based on deep learning |
| CN112069735A (en) * | 2020-09-08 | 2020-12-11 | 哈尔滨工业大学 | Full-slice digital imaging high-precision automatic focusing method based on asymmetric aberration |
| CN112464727A (en) * | 2020-11-03 | 2021-03-09 | 电子科技大学 | Self-adaptive face recognition method based on light field camera |
| CN117115241A (en) * | 2023-09-06 | 2023-11-24 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1996004566A1 (en) * | 1994-08-02 | 1996-02-15 | British Technology Group Limited | A method of magnetic resonance imaging employing velocity refocussing |
| CN103605203A (en) * | 2013-11-07 | 2014-02-26 | 麦克奥迪实业集团有限公司 | Automatic focusing method in digital slicing scanning process |
| CN104390981A (en) * | 2014-12-12 | 2015-03-04 | 宁波江丰生物信息技术有限公司 | Method for rapidly and accurately focusing and scanning pathological section tissue on basis of image collection device |
| US20160182813A1 (en) * | 2014-12-19 | 2016-06-23 | Thomson Licensing | Method and apparatus for generating an adapted slice image from a focal stack |
| CN107369134A (en) * | 2017-06-12 | 2017-11-21 | 上海斐讯数据通信技术有限公司 | A kind of image recovery method of blurred picture |
| CN107945156A (en) * | 2017-11-14 | 2018-04-20 | 宁波江丰生物信息技术有限公司 | A kind of method of automatic Evaluation numeral pathology scan image image quality |
| US20180182099A1 (en) * | 2016-12-27 | 2018-06-28 | Definiens Ag | Identifying and Excluding Blurred Areas of Images of Stained Tissue To Improve Cancer Scoring |
| CN109191457A (en) * | 2018-09-21 | 2019-01-11 | 中国人民解放军总医院 | A kind of pathological image quality validation recognition methods |
| US20190050980A1 (en) * | 2017-08-11 | 2019-02-14 | Konfoong Biotech International Co., Ltd. | Digital pathological section scanning system |
| CN110411946A (en) * | 2019-08-05 | 2019-11-05 | 山东志盈医学科技有限公司 | The focus method of camera image in a kind of digital slices scanner |
| US20190340468A1 (en) * | 2018-05-07 | 2019-11-07 | Google Llc | Focus-Weighted, Machine Learning Disease Classifier Error Prediction for Microscope Slide Images |
-
2020
- 2020-03-31 CN CN202010242168.9A patent/CN111462075B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1996004566A1 (en) * | 1994-08-02 | 1996-02-15 | British Technology Group Limited | A method of magnetic resonance imaging employing velocity refocussing |
| CN103605203A (en) * | 2013-11-07 | 2014-02-26 | 麦克奥迪实业集团有限公司 | Automatic focusing method in digital slicing scanning process |
| CN104390981A (en) * | 2014-12-12 | 2015-03-04 | 宁波江丰生物信息技术有限公司 | Method for rapidly and accurately focusing and scanning pathological section tissue on basis of image collection device |
| US20160182813A1 (en) * | 2014-12-19 | 2016-06-23 | Thomson Licensing | Method and apparatus for generating an adapted slice image from a focal stack |
| US20180182099A1 (en) * | 2016-12-27 | 2018-06-28 | Definiens Ag | Identifying and Excluding Blurred Areas of Images of Stained Tissue To Improve Cancer Scoring |
| CN107369134A (en) * | 2017-06-12 | 2017-11-21 | 上海斐讯数据通信技术有限公司 | A kind of image recovery method of blurred picture |
| US20190050980A1 (en) * | 2017-08-11 | 2019-02-14 | Konfoong Biotech International Co., Ltd. | Digital pathological section scanning system |
| CN107945156A (en) * | 2017-11-14 | 2018-04-20 | 宁波江丰生物信息技术有限公司 | A kind of method of automatic Evaluation numeral pathology scan image image quality |
| US20190340468A1 (en) * | 2018-05-07 | 2019-11-07 | Google Llc | Focus-Weighted, Machine Learning Disease Classifier Error Prediction for Microscope Slide Images |
| CN109191457A (en) * | 2018-09-21 | 2019-01-11 | 中国人民解放军总医院 | A kind of pathological image quality validation recognition methods |
| CN110411946A (en) * | 2019-08-05 | 2019-11-05 | 山东志盈医学科技有限公司 | The focus method of camera image in a kind of digital slices scanner |
Non-Patent Citations (2)
| Title |
|---|
| REMY CLAVEAU ET AL.: "Digital refocusing and extended depth of field reconstruction in Fourier ptychographic microscopy", 《BIOMEDICAL OPTICS EXPRESS》, vol. 11, pages 215 - 226 * |
| 牛原野: "压缩感知光场重建及深度估计的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112037153A (en) * | 2020-09-08 | 2020-12-04 | 哈尔滨工业大学 | Full-slice digital imaging quasi-focus restoration method based on quasi-focus distance prior |
| CN112037154A (en) * | 2020-09-08 | 2020-12-04 | 哈尔滨工业大学 | High-precision quasi-focus restoration method for full-slice digital imaging and double photographing |
| CN112053304A (en) * | 2020-09-08 | 2020-12-08 | 哈尔滨工业大学 | Rapid focusing restoration method for single shooting of full-slice digital imaging |
| CN112070661A (en) * | 2020-09-08 | 2020-12-11 | 哈尔滨工业大学 | Full-slice digital imaging rapid automatic focusing method based on deep learning |
| CN112069735A (en) * | 2020-09-08 | 2020-12-11 | 哈尔滨工业大学 | Full-slice digital imaging high-precision automatic focusing method based on asymmetric aberration |
| CN112069735B (en) * | 2020-09-08 | 2022-08-12 | 哈尔滨工业大学 | Full-slice digital imaging high-precision automatic focusing method based on asymmetric aberration |
| CN112464727A (en) * | 2020-11-03 | 2021-03-09 | 电子科技大学 | Self-adaptive face recognition method based on light field camera |
| CN117115241A (en) * | 2023-09-06 | 2023-11-24 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
| CN117115241B (en) * | 2023-09-06 | 2024-03-29 | 北京透彻未来科技有限公司 | Method for searching central focus of digital pathological image in zooming process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111462075B (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111462076B (en) | Full-slice digital pathological image fuzzy region detection method and system | |
| CN111462075B (en) | Rapid refocusing method and system for full-slice digital pathological image fuzzy region | |
| CN108447062B (en) | Pathological section unconventional cell segmentation method based on multi-scale mixed segmentation model | |
| CN111007661B (en) | Microscopic image automatic focusing method and device based on deep learning | |
| CN108537751B (en) | Thyroid ultrasound image automatic segmentation method based on radial basis function neural network | |
| CN111083365B (en) | Method and device for rapidly detecting optimal focal plane position | |
| CN111046880A (en) | Infrared target image segmentation method and system, electronic device and storage medium | |
| US20220012884A1 (en) | Image analysis system and analysis method | |
| GB2466818A (en) | Cell image segmentation using binary threshold and greyscale image processing | |
| CN113689374B (en) | Method and system for determining surface roughness of plant leaf | |
| CN110736747A (en) | cell liquid based smear under-mirror positioning method and system | |
| CN110987886A (en) | Full-automatic microscopic image fluorescence scanning system | |
| CN112135048B (en) | Automatic focusing method and device for target object | |
| CN112348059A (en) | Deep learning-based method and system for classifying multiple dyeing pathological images | |
| CN110837809A (en) | Blood automatic analysis method, blood automatic analysis system, blood cell analyzer, and storage medium | |
| JP2020085546A (en) | System for supporting inspection and repair of structure | |
| CN116012291A (en) | Industrial part image defect detection method and system, electronic equipment and storage medium | |
| CN115641345A (en) | Multiple myeloma cell morphology fine segmentation method based on deep learning | |
| CN115909006A (en) | Mammary tissue image classification method and system based on convolution Transformer | |
| CN113569679B (en) | Method, device and system for measuring elongation at break | |
| CN112464802B (en) | Automatic identification method and device for slide sample information and computer equipment | |
| CN113705298A (en) | Image acquisition method and device, computer equipment and storage medium | |
| CN116648723A (en) | Method and device for analyzing microstructure of material | |
| CN114897823B (en) | Cytological sample image quality control method, system and storage medium | |
| US20220058369A1 (en) | Automated stereology for determining tissue characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |