CN111105415A - White blood cell large-field-of-view image detection system and method based on deep learning - Google Patents
White blood cell large-field-of-view image detection system and method based on deep learning Download PDFInfo
- Publication number
- CN111105415A CN111105415A CN201911410640.9A CN201911410640A CN111105415A CN 111105415 A CN111105415 A CN 111105415A CN 201911410640 A CN201911410640 A CN 201911410640A CN 111105415 A CN111105415 A CN 111105415A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- sample
- low
- white blood
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000000265 leukocyte Anatomy 0.000 title claims abstract description 69
- 238000001514 detection method Methods 0.000 title claims abstract description 39
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000013135 deep learning Methods 0.000 title claims abstract description 17
- 238000001228 spectrum Methods 0.000 claims abstract description 54
- 210000003743 erythrocyte Anatomy 0.000 claims abstract description 27
- 238000005286 illumination Methods 0.000 claims abstract description 4
- 238000013527 convolutional neural network Methods 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 21
- 238000002372 labelling Methods 0.000 claims description 18
- 238000012360 testing method Methods 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 17
- 230000006870 function Effects 0.000 claims description 15
- 238000003384 imaging method Methods 0.000 claims description 14
- 238000007781 pre-processing Methods 0.000 claims description 11
- 238000012795 verification Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 8
- 239000013589 supplement Substances 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 238000007405 data analysis Methods 0.000 claims description 4
- 238000003062 neural network model Methods 0.000 claims description 2
- 238000004458 analytical method Methods 0.000 abstract description 6
- 238000004519 manufacturing process Methods 0.000 abstract description 4
- 230000000694 effects Effects 0.000 abstract description 3
- 238000000605 extraction Methods 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract description 2
- 238000013528 artificial neural network Methods 0.000 abstract 1
- 230000000875 corresponding effect Effects 0.000 description 21
- 238000004820 blood count Methods 0.000 description 8
- 210000004027 cell Anatomy 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000000386 microscopy Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 208000024172 Cardiovascular disease Diseases 0.000 description 1
- 102000001554 Hemoglobins Human genes 0.000 description 1
- 108010054147 Hemoglobins Proteins 0.000 description 1
- 206010061218 Inflammation Diseases 0.000 description 1
- 210000003651 basophil Anatomy 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 238000004159 blood analysis Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 210000003979 eosinophil Anatomy 0.000 description 1
- 238000000684 flow cytometry Methods 0.000 description 1
- 230000004054 inflammatory process Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 210000004698 lymphocyte Anatomy 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 210000001616 monocyte Anatomy 0.000 description 1
- 210000000440 neutrophil Anatomy 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 208000001072 type 2 diabetes mellitus Diseases 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses a white blood cell large-view-field image detection system and method based on deep learning, wherein the system consists of an image acquisition module, an image reconstruction module and an intelligent detection module based on deep learning, and detection samples under different illumination are imaged through a low-power microscope objective; the image acquisition module records the set of low-resolution images; the image reconstruction module acquires a high-resolution image by a frequency spectrum iteration method; the detection module uses a pre-trained neural network to perform feature extraction and recognition on the generated high-resolution image, and finally outputs a detection result. The invention can intelligently identify white blood cells, and count the white blood cells and red blood cells in a visual field respectively, thereby helping an inspector to quickly and accurately obtain an analysis result; meanwhile, the requirement of combining high resolution with a large view field is met, the effect of obtaining higher contrast and resolution by using a low power lens is realized, and the manufacturing cost of the system is reduced.
Description
Technical Field
The invention belongs to the field of computational microscopy imaging and the field of intelligent medical detection, and particularly relates to a system and a method for detecting a white blood cell large-view-field image based on deep learning.
Background
Blood analysis is one of the clinical examination methods commonly used in the medical field, and includes white blood cell counting and classification, red blood cell counting, platelet counting, hemoglobin counting, and the like. Among them, the white blood cell count is closely related to inflammation, and is considered as an important predictor for early disease diagnosis. Multiple studies show that the white blood cell count is positively correlated with cardiovascular diseases, type 2 diabetes and other diseases; in addition, the ratio of white blood cells to red blood cells is of medical value. Therefore, white blood cell count is of great medical importance.
The white blood cell count can be performed automatically by a flow cytometer or manually by a medical practitioner with the aid of a microscope. Flow cytometry is efficient in cell detection, but cannot analyze cell morphology, so microscopy remains an important medical detection means. In order to improve the diagnostic accuracy, it is usually necessary to examine a large area of sample area, i.e. a microscope is required to be able to achieve large field-of-view imaging, high resolution imaging.
The conventional microscope is limited by diffraction, and the resolution is mainly determined by the NA (numerical aperture) of the microscope objective, and the resolution of the microscope is higher when the NA value is larger. However, the field range of the objective lens with a high NA value is small, which is not beneficial to observing the overall appearance of the sample. In order to obtain a large-field microscopic image, the conventional microscope mainly adopts methods such as mechanical scanning splicing and the like, and the methods usually require a precise mechanical scanning device and have higher cost.
The fourier stacked microscopy imaging technique scans the fourier spectral information of the object by varying the angle of the illuminating wavefront through a programmable LED array, and then recovers a high resolution image from the fourier domain. Because the Fourier laminated microscopic imaging technology does not need a high-coherence light source, a moving part and a detector with a high dynamic range, the manufacturing cost is greatly reduced while the requirement of large field of view and high resolution is met, and the Fourier laminated microscopic imaging technology becomes an ideal means for large field of view microscopy.
However, similar to the conventional microscope, the microscope based on the fourier transform imaging technology lacks corresponding system support, cannot realize intelligent detection, can only depend on the naked eye detection of medical workers, not only wastes time, but also has a high false detection rate influenced by the subjective state of an inspector.
Disclosure of Invention
The invention aims to provide a white blood cell large-view-field image detection system and method based on deep learning for overcoming the problems in the prior art, which can meet the requirement of combining high resolution and large view field, realize the effect of obtaining higher contrast and resolution by using a low power lens and reduce the manufacturing cost of the system; meanwhile, the specially designed convolutional neural network is used for intelligently identifying the white blood cells and respectively counting the white blood cells and the red blood cells in the visual field, so that multiple functions of white blood cell detection, white blood cell classification, white blood cell counting, red blood cell counting and the like are realized, and an inspector is helped to quickly and accurately obtain an analysis result.
The purpose of the invention is realized by the following technical scheme:
a white blood cell large-field image detection method based on deep learning at least comprises the following steps: s1: a database establishing step, namely acquiring a low-resolution sample image, reconstructing the low-resolution sample image to generate a high-resolution reconstructed image, labeling the generated image and storing the labeled image into a database; s2: training a convolutional neural network model, namely dividing a training set, a verification set and a test set according to the ratio of 6:2:2 of the database acquired in the step S1, and using the test set, the verification set and the training data set, adjusting network hyper-parameters and checking the model until a loss function convergence model is obtained; s3: and a step of collecting a sample to be detected and completing leukocyte detection, wherein leukocyte labeling in the image of the sample to be detected is completed based on the convolutional neural network model trained in the step S2, comparison with a preset standard value is completed, and condition judgment of the sample to be detected is realized.
According to a preferred embodiment, the step S3 specifically includes: and (4) acquiring a low-resolution image of the sample to be detected, reconstructing a high-resolution image, taking the reconstructed image of the sample to be detected as the input of the convolutional neural network model trained in the step (S2), outputting the white blood cell positioning, the type and the number, the red blood cell number and the white/red blood cell ratio value in the sample to be detected by the convolutional neural network model, comparing the white blood cell positioning, the type and the number with a prestored standard value, and judging whether the sample is abnormal or not.
According to a preferred embodiment, the step S1 and the step S3 of reconstructing the low resolution image to generate the high resolution image specifically include the following steps: step a: carrying out denoising pretreatment on the low-resolution image; b, selecting a low-resolution image with vertical incidence for interpolation processing, taking the image after the interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started; c, corresponding different apertures in the frequency domain at different incidence angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incidence angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; replacing the intensity value of the complex amplitude distribution to be updated with the intensity value of the low-resolution image acquired by the incident angle, and keeping the phase information unchanged; d, solving the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating the corresponding frequency spectrum component in the high-resolution frequency spectrum by using the low-resolution frequency spectrum; and e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e after all the incident angles are updated until the cost function is converged, thereby obtaining the high-resolution image to be detected.
According to a preferred embodiment, in the step a, the denoising preprocessing step is performed on the low-resolution image, and specifically includes: in the low-resolution image acquisition process, shading treatment is carried out on the image acquisition microscope, and a dark field image is acquired under the condition that the light supplement LED is completely extinguished; programming the LED array, sequentially lightening each LED, and collecting corresponding sample low-resolution intensity images under different incidence angles; and calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, so that the influence of noise is reduced.
A white blood cell large-view-field image detection system based on deep learning at least comprises an image acquisition module, an image reconstruction module and an intelligent detection module, wherein the image acquisition module is connected with the image reconstruction module, the image reconstruction module is connected with the intelligent detection module, and the intelligent detection module realizes analysis and judgment of an object to be identified based on a reconstructed image completed by the image reconstruction module; wherein, the intellectual detection system module includes: a database storing labeled Fourier stacked micro-reconstructed images of leukocytes; the convolutional neural network model is a loss function convergence model obtained on the basis of a training set, a verification set and a test set which are divided by a database in a ratio of 6:2:2, wherein the test set and the verification set are used for training the data set and adjusting network hyper-parameters, and the test set is used for verifying the model; the convolutional neural network model is used for completing the white blood cell labeling of the image of the sample to be detected, which is sent by the image reconstruction module; and the data analysis unit is used for finishing comparison with a preset standard value based on the white blood cell marking in the sample image to be detected, so that the condition judgment of the sample to be detected is realized.
According to a preferred embodiment, the image labeling of the sample to be detected by the convolutional neural network model includes: and outputting the white blood cell location, the type and the number in the sample image to be detected, the red blood cell number and the white/red blood cell ratio value by the convolutional neural network model.
According to a preferred embodiment, the image reconstruction module comprises an image preprocessing unit and a spectrum iteration unit; the image preprocessing unit is configured to perform denoising preprocessing on a low-resolution image, and specifically includes: in the low-resolution image acquisition process, shading treatment is carried out on the image acquisition microscope, and a dark field image is acquired under the condition that the light supplement LED is completely extinguished; programming the LED array, sequentially lightening each LED, and collecting corresponding sample low-resolution intensity images under different incidence angles; and calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, so that the influence of noise is reduced.
According to a preferred embodiment, the spectrum iteration unit is configured to perform an iteration process and obtain a high-resolution reconstructed image by adopting the following processing steps; step b: selecting a vertically incident low-resolution image for interpolation processing, taking the image subjected to interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started; step c: b, corresponding different apertures in the frequency domain at different incident angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incident angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; replacing the intensity value of the complex amplitude distribution to be updated with the intensity value of the low-resolution image acquired by the incident angle, and keeping the phase information unchanged; step d: calculating the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating corresponding frequency spectrum components in the high-resolution frequency spectrum by using the low-resolution frequency spectrum; step e: and e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e until the cost function is converged after all the incident angles are updated, so as to obtain the high-resolution image to be detected.
According to a preferred embodiment, the image acquisition module comprises an LED array, a microscope, a digital camera and an image acquisition card, the LED array is used to complete the illumination of the acquisition target of the microscope, the microscope is used to complete the imaging of the target to be acquired, and the digital camera and the image acquisition card are used to sample the corresponding low-resolution intensity images of the sample at different incident angles to obtain the digital image data which can be directly processed by the computer.
According to a preferred embodiment, the microscope is a low-power objective imaging structure, has a focusing and condensing structure, and is provided with a stage capable of carrying a sample to be observed.
The main scheme and the further selection schemes can be freely combined to form a plurality of schemes which are all adopted and claimed by the invention; in the invention, the selection (each non-conflict selection) and other selections can be freely combined. The skilled person in the art can understand that there are many combinations, which are all the technical solutions to be protected by the present invention, according to the prior art and the common general knowledge after understanding the scheme of the present invention, and the technical solutions are not exhaustive herein.
The invention has the beneficial effects that:
(1) the system or the method realizes the acquisition of high-resolution images through the low-power lens, meets the requirement of large field of view, and better meets the actual requirement of microscopic examination.
(2) The system or the method of the invention not only can intelligently detect the white blood cells, but also can count and analyze the white blood cells in form, output corresponding analysis results, save manpower and inspection time and improve inspection efficiency.
(3) The low power microscope and the LED lighting source used by the system of the invention are common components, and compared with a microscope system provided with a high power objective lens and a microscope system adopting a mechanical scanning device, the low power microscope and the LED lighting source have low manufacturing cost and are more beneficial to popularization and application of the system.
Drawings
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic workflow diagram of the system of the present invention;
FIG. 3 is a flow chart of sample image acquisition and reconstruction based on Fourier stack imaging theory in the method of the present invention;
FIG. 4 is a block diagram of a branched convolutional neural network of the present invention;
FIG. 5 is a schematic diagram of labeling white blood cells and red blood cells using the convolutional neural network model of the present invention.
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that, in order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments.
Thus, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate orientations and positional relationships that are conventionally used in the products of the present invention, and are used merely for convenience in describing the present invention and for simplicity in description, but do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed in a particular orientation, and be operated, and therefore, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Furthermore, the terms "horizontal", "vertical", "overhang" and the like do not imply that the components are required to be absolutely horizontal or overhang, but may be slightly inclined. For example, "horizontal" merely means that the direction is more horizontal than "vertical" and does not mean that the structure must be perfectly horizontal, but may be slightly inclined.
In the description of the present invention, it should also be noted that, unless otherwise explicitly specified or limited, the terms "disposed," "mounted," "connected," and "connected" are to be construed broadly and may, for example, be fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In addition, it should be noted that, in the present invention, if the specific structures, connection relationships, position relationships, power source relationships, and the like are not written in particular, the structures, connection relationships, position relationships, power source relationships, and the like related to the present invention can be known by those skilled in the art without creative work on the basis of the prior art.
Example 1:
referring to fig. 1 and 2, the invention discloses a white blood cell large-field image detection system based on deep learning, which at least comprises an image acquisition module, an image reconstruction module and an intelligent detection module. The image acquisition module is used for realizing low-resolution image acquisition of the sample image. The image reconstruction module is used for realizing the reconstruction of a high-resolution image. The intelligent detection module is used for realizing the analysis and judgment of abnormal conditions of the white blood cells based on the reconstructed image with high resolution.
In this embodiment, the sample to be detected is a human blood smear, and the detection object is a leukocyte. The labeled image sets of the white blood cells and the red blood cells are stored in the database, and the convolutional neural network is trained by taking the image sets as a training set. The convolutional neural network used for detection is a neural network model designed specifically for the system.
Preferably, the image acquisition module is connected with the image reconstruction module, the image reconstruction module is connected with the intelligent detection module, and the intelligent detection module realizes analysis and judgment of the object to be identified based on the reconstructed image completed by the image reconstruction module.
Preferably, the image acquisition module comprises an LED array, a microscope, a digital camera and an image acquisition card.
Preferably, the illumination of the acquisition target of the microscope is done by the LED array. And the microscope is used for finishing the imaging of the target to be acquired. And simultaneously, the digital camera and the image acquisition card sample the corresponding sample low-resolution intensity images under different incidence angles to acquire digital image data which can be directly processed by a computer.
Preferably, the LED array may be a programmable LED array. The light source wavelength was 629nm and the distance between each adjacent LED in the array was 8.128 mm. The distance from the light source to the sample may be set to 98 mm. The resolution of the sensors of the digital camera may be 2160 × 2560.
Further, the microscope is a low-magnification objective lens imaging structure, has a focusing and condensing structure, and is provided with an objective table capable of bearing a sample to be observed. For example, the microscope is a 4-fold mirror configuration with a corresponding numerical aperture of 0.13.
Preferably, the image reconstruction module comprises an image preprocessing unit and a spectrum iteration unit.
The image preprocessing unit is configured to perform denoising preprocessing on a low-resolution image, and specifically includes: firstly, shading the image acquisition microscope in the low-resolution image acquisition process, and acquiring a dark field image in the state that the light supplement LED is completely extinguished. And then programming the LED array, sequentially lightening each LED, and collecting the corresponding sample low-resolution intensity images under different incidence angles. And finally, calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, so that the influence of noise is reduced.
The spectral iteration unit is configured to perform an iterative process and obtain a high resolution reconstructed image using the following processing steps. The method specifically comprises the following steps:
step b: selecting a vertically incident low-resolution image for interpolation processing, taking the image subjected to interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started;
step c: b, corresponding different apertures in the frequency domain at different incident angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incident angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; replacing the intensity value of the complex amplitude distribution to be updated with the intensity value of the low-resolution image acquired by the incident angle, and keeping the phase information unchanged;
step d: calculating the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating corresponding frequency spectrum components in the high-resolution frequency spectrum by using the low-resolution frequency spectrum; step e: and e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e until the cost function is converged after all the incident angles are updated, so as to obtain the high-resolution image to be detected.
Preferably, wherein the smart detection module comprises: the system comprises a database, a convolutional neural network model and a data analysis unit.
Preferably, the database stores labeled fourier-stacked micro-reconstructed images of leukocytes for providing raw data for training and testing of convolutional neural network models.
Preferably, the convolutional neural network model is configured as a model of loss function convergence obtained based on a training set, a validation set, and a test set into which the database is divided at a ratio of 6:2: 2. The test set and the verification set are used for training the data set and adjusting the network hyper-parameters, and the test set is used for testing the model performance.
Preferably, referring to fig. 4, the convolutional neural network model has two branch structures: detecting branches and counting branches. The two branch networks share the convolutional layer extraction features of the front end. After extracting features, the detection branches are added into an RPN network to generate a plurality of candidate areas; mapping the characteristics of all candidate regions to the same dimensionality through ROI pooling; and finally, obtaining the category fraction and the coordinate estimation of the white blood cells through a classifier. And the counting branch fuses the multilayer features through 1-by-1 convolution and linear interpolation upsampling, and the feature channel number is compressed by using the convolution layer on the basis of the fused features, so that the density map is further fitted.
Further, the trained convolutional neural network model is used for completing the leucocyte labeling of the image of the sample to be detected, which is sent by the image reconstruction module.
Specifically, the image labeling of the sample to be detected by the convolutional neural network model includes: and outputting the white blood cell location, the type and the number in the sample image to be detected, the red blood cell number and the white/red blood cell ratio value by the convolutional neural network model.
Preferably, referring to FIG. 5, a schematic illustration of labeling of white blood cells and red blood cells is shown. The three images are a cell original image, a red blood cell density image and a white blood cell labeling image from left to right in sequence. In this embodiment, labeling red blood cells requires calibrating a center point of the red blood cells in the original image, converting the center point into an area with a sum of 1 through gaussian transformation, and the value of the unmarked area is 0. The image generated by transformation is called a red blood cell Gaussian density image, and the sum of the numerical values of all the pixel points in the image is the sum of the number of the red blood cells in the original image. Labeling of white blood cells is characterized by a rectangular box with the same size as the cell, and the coordinates of the four corners of the rectangular box are recorded as true values.
Preferably, the data analysis unit is used for completing comparison with a preset standard value based on leukocyte labeling in the sample image to be detected, so as to judge whether the leukocytes of the sample to be detected are abnormal.
Example 2
On the basis of the embodiment 1, the invention also discloses a white blood cell large-field image detection method based on deep learning.
Preferably, the image detection method includes at least the steps of:
step S1: and establishing a database. And acquiring a low-resolution sample image, reconstructing to generate a high-resolution reconstructed image, marking the generated image and storing the marked image into a database.
Preferably, the labeling of the generated image in step S1 specifically includes: for white blood cells, the functions of detection, classification and counting are realized. That is, the size, location and specific classes of leukocytes (neutrophils, eosinophils, basophils, lymphocytes and monocytes) need to be labeled; for red blood cells, a counting function is required. That is, only the location of the red blood cells need be noted.
Step S2: and (3) training a convolutional neural network model, namely dividing the database acquired in the step S1 into a training set, a verification set and a test set according to the ratio of 6:2:2, and using the test set, the verification set and the training data set, adjusting network hyper-parameters and checking the performance of the model until a model with a loss function convergence is obtained.
Step S3: and (5) collecting a sample to be detected, completing a leukocyte detection step, and completing leukocyte labeling in the image of the sample to be detected based on the convolutional neural network model trained in the step S2. And the comparison with a preset standard value is completed, so that the condition judgment of the sample to be detected is realized.
Preferably, the step S3 specifically includes: and acquiring a low-resolution image of the sample to be detected, and reconstructing a high-resolution image. And the reconstructed image of the sample to be detected is used as the input of the convolutional neural network model trained in the step S2, and the white blood cell location, category and number, the number of red blood cells and the white/red blood cell ratio value in the sample to be detected are output by the convolutional neural network model. And comparing with a pre-stored standard value to judge whether the sample is abnormal or not.
Preferably, as shown with reference to fig. 3. In the steps S1 and S3, the reconstructing the low-resolution image to generate the high-resolution image specifically includes the following steps.
Step a: and carrying out denoising pretreatment on the low-resolution image.
Preferably, in the step a, the step of performing denoising preprocessing on the low-resolution image specifically includes: firstly, shading the image acquisition microscope in the low-resolution image acquisition process, and acquiring a dark field image in the state that the light supplement LED is completely extinguished. And then programming the LED array, sequentially lightening each LED, and collecting the corresponding sample low-resolution intensity images under different incidence angles. And finally, calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, thereby reducing the influence of noise.
B, selecting a low-resolution image with vertical incidence for interpolation processing, taking the image after the interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started.
C, corresponding different apertures in the frequency domain at different incidence angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incidence angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; the intensity values of the complex amplitude distribution to be updated are replaced by the intensity values of the low-resolution image acquired at the incident angle, and the phase information remains unchanged.
And d, solving the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating the corresponding frequency spectrum component in the high-resolution frequency spectrum by using the low-resolution frequency spectrum.
And e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e after all the incident angles are updated until the cost function is converged, thereby obtaining the high-resolution image to be detected.
The foregoing basic embodiments of the invention and their various further alternatives can be freely combined to form multiple embodiments, all of which are contemplated and claimed herein. In the scheme of the invention, each selection example can be combined with any other basic example and selection example at will. Numerous combinations will be known to those skilled in the art.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.
Claims (10)
1. A white blood cell large-field image detection method based on deep learning is characterized by at least comprising the following steps:
s1: a database establishing step, namely acquiring a low-resolution sample image, reconstructing the low-resolution sample image to generate a high-resolution reconstructed image, labeling the generated image and storing the labeled image into a database;
s2: training a convolutional neural network model, namely dividing a training set, a verification set and a test set according to the ratio of 6:2:2 of the database acquired in the step S1, and adjusting network hyper-parameters and a test model by using the test set, the verification set and the training data set until a loss function convergence model is obtained;
s3: and a step of collecting a sample to be detected and completing leukocyte detection, wherein leukocyte labeling in the image of the sample to be detected is completed based on the convolutional neural network model trained in the step S2, comparison with a preset standard value is completed, and condition judgment of the sample to be detected is realized.
2. The method for detecting large-field-of-view white blood cell images based on deep learning of claim 1, wherein the step S3 specifically includes:
collecting a low-resolution image of the sample to be tested, reconstructing a high-resolution image, using the reconstructed image of the sample to be tested as an input of the convolutional neural network model trained in the step S2,
and the white blood cell location, category and number in the sample to be tested, the number of red blood cells and the white/red blood cell ratio value are output by the convolution neural network model,
and comparing with a pre-stored standard value to judge whether the sample is abnormal or not.
3. The method for detecting white blood cell large-field image based on deep learning as claimed in claim 1 or 2, wherein the step S1 and the step S3 of reconstructing the low-resolution image to generate the high-resolution image specifically include the following steps:
step a: carrying out denoising pretreatment on the low-resolution image;
b, selecting a low-resolution image with vertical incidence for interpolation processing, taking the image after the interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started;
c, corresponding different apertures in the frequency domain at different incidence angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incidence angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; replacing the intensity value of the complex amplitude distribution to be updated with the intensity value of the low-resolution image acquired by the incident angle, and keeping the phase information unchanged;
d, solving the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating the corresponding frequency spectrum component in the high-resolution frequency spectrum by using the low-resolution frequency spectrum;
and e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e after all the incident angles are updated until the cost function is converged, thereby obtaining the high-resolution image to be detected.
4. The method for detecting the large-field-of-view image of white blood cells based on deep learning as claimed in claim 3, wherein in the step a, the step of denoising the low-resolution image comprises:
in the low-resolution image acquisition process, shading treatment is carried out on the image acquisition microscope, and a dark field image is acquired under the condition that the light supplement LED is completely extinguished;
programming the LED array, sequentially lightening each LED, and collecting corresponding sample low-resolution intensity images under different incidence angles;
and calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, so that the influence of noise is reduced.
5. The system for detecting the white blood cell large-view-field image based on deep learning is characterized by at least comprising an image acquisition module, an image reconstruction module and an intelligent detection module, wherein the image acquisition module is connected with the image reconstruction module;
wherein, the intellectual detection system module includes:
a database storing labeled Fourier stacked micro-reconstructed images of leukocytes;
the convolutional neural network model is a loss function convergence model obtained by a training set, a verification set and a test set which are divided based on a database in a ratio of 6:2:2, wherein the test set and the verification set are used for training the data set and adjusting network hyper-parameters, and the test set is used for verifying the performance of the model; the convolutional neural network model is used for completing the white blood cell labeling of the image of the sample to be detected, which is sent by the image reconstruction module;
and the data analysis unit is used for finishing comparison with a preset standard value based on the white blood cell marking in the sample image to be detected, so that the condition judgment of the sample to be detected is realized.
6. The system for detecting the large-field-of-view image of the white blood cells based on the deep learning of claim 5, wherein the convolutional neural network model for image labeling of the sample to be detected comprises:
and outputting the white blood cell location, the type and the number in the sample image to be detected, the red blood cell number and the white/red blood cell ratio value by the convolutional neural network model.
7. The system for detecting white blood cell large-field-of-view images based on deep learning of claim 5, wherein the image reconstruction module comprises an image preprocessing unit and a spectrum iteration unit;
the image preprocessing unit is configured to perform denoising preprocessing on a low-resolution image, and specifically includes:
in the low-resolution image acquisition process, shading treatment is carried out on the image acquisition microscope, and a dark field image is acquired under the condition that the light supplement LED is completely extinguished;
programming the LED array, sequentially lightening each LED, and collecting corresponding sample low-resolution intensity images under different incidence angles;
and calculating a noise threshold according to the collected dark field image, and carrying out noise processing on each image to enable the gray value lower than the noise threshold in the image to be zero, so that the influence of noise is reduced.
8. The system according to claim 7, wherein the spectrum iteration unit is configured to perform an iteration process by adopting the following processing steps and obtain a high-resolution reconstructed image;
b, selecting a low-resolution image with vertical incidence for interpolation processing, taking the image after the interpolation processing as an initial high-resolution light intensity image of a sample, and setting the phase to be zero initially; then Fourier transform is carried out on the spectrum to obtain an initial high-resolution spectrum, and iteration is started;
c, corresponding different apertures in the frequency domain at different incidence angles, intercepting corresponding sub-aperture frequency spectrums from the high-resolution frequency spectrum obtained in the step b aiming at a certain incidence angle, and obtaining complex amplitude distribution to be updated through inverse Fourier transform; replacing the intensity value of the complex amplitude distribution to be updated with the intensity value of the low-resolution image acquired by the incident angle, and keeping the phase information unchanged;
d, solving the frequency spectrum of the updated sub-aperture by utilizing Fourier transform, and updating the corresponding frequency spectrum component in the high-resolution frequency spectrum by using the low-resolution frequency spectrum;
and e, repeating the step c and the step d until all the incident angles are updated once, and repeating the step e after all the incident angles are updated until the cost function is converged, thereby obtaining the high-resolution image to be detected.
9. The system of claim 5, wherein the image capturing module comprises an LED array, a microscope, a digital camera and an image capturing card,
the illumination of the collection target of the microscope is completed through the LED array, the imaging of the target to be collected is completed through the microscope,
and simultaneously, the digital camera and the image acquisition card sample the corresponding sample low-resolution intensity images under different incidence angles to acquire digital image data which can be directly processed by a computer.
10. The system as claimed in claim 9, wherein the microscope is a low-power objective imaging structure with focusing and condensing structure and is equipped with a stage for carrying the sample to be observed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911410640.9A CN111105415A (en) | 2019-12-31 | 2019-12-31 | White blood cell large-field-of-view image detection system and method based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911410640.9A CN111105415A (en) | 2019-12-31 | 2019-12-31 | White blood cell large-field-of-view image detection system and method based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111105415A true CN111105415A (en) | 2020-05-05 |
Family
ID=70425631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911410640.9A Pending CN111105415A (en) | 2019-12-31 | 2019-12-31 | White blood cell large-field-of-view image detection system and method based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111105415A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113327233A (en) * | 2021-05-28 | 2021-08-31 | 北京理工大学重庆创新中心 | Cell image detection method based on transfer learning |
| TWI769647B (en) * | 2021-01-04 | 2022-07-01 | 鴻海精密工業股份有限公司 | Method and device for determining cell density, computer device and storage medium |
| CN115220211A (en) * | 2022-07-29 | 2022-10-21 | 江南大学 | Microscopic imaging system and method based on deep learning and light field imaging |
| CN116630294A (en) * | 2023-06-08 | 2023-08-22 | 南方医科大学南方医院 | Whole blood sample detection method and device based on deep learning and storage medium |
| CN116704503A (en) * | 2023-08-01 | 2023-09-05 | 昕传生物科技(北京)有限公司 | Machine learning-based three-dimensional lung slice cell classification and protein quantification method |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107065159A (en) * | 2017-03-24 | 2017-08-18 | 南京理工大学 | A kind of large visual field high resolution microscopic imaging device and iterative reconstruction method based on big illumination numerical aperture |
| CN108169173A (en) * | 2017-12-29 | 2018-06-15 | 南京理工大学 | A kind of big visual field high-resolution three dimensional diffraction chromatography micro imaging method |
| CN108362643A (en) * | 2018-02-10 | 2018-08-03 | 北京工业大学 | A kind of double altitudes illumination Fourier's lamination micro imaging method |
| CN109815919A (en) * | 2019-01-28 | 2019-05-28 | 上海七牛信息技术有限公司 | A kind of people counting method, network, system and electronic equipment |
| CN110188685A (en) * | 2019-05-30 | 2019-08-30 | 燕山大学 | A kind of object count method and system based on the multiple dimensioned cascade network of double attentions |
| WO2019170464A1 (en) * | 2018-03-08 | 2019-09-12 | Carl Zeiss Microscopy Gmbh | Microscope and method for viewing a sample using a microscope in order to display images with an extended focal depth or three-dimensional images |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
| CN110501278A (en) * | 2019-07-10 | 2019-11-26 | 同济大学 | A kind of method for cell count based on YOLOv3 and density estimation |
| CN110516584A (en) * | 2019-08-22 | 2019-11-29 | 杭州图谱光电科技有限公司 | A kind of Auto-counting of Cells method based on dynamic learning of microscope |
-
2019
- 2019-12-31 CN CN201911410640.9A patent/CN111105415A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107065159A (en) * | 2017-03-24 | 2017-08-18 | 南京理工大学 | A kind of large visual field high resolution microscopic imaging device and iterative reconstruction method based on big illumination numerical aperture |
| CN108169173A (en) * | 2017-12-29 | 2018-06-15 | 南京理工大学 | A kind of big visual field high-resolution three dimensional diffraction chromatography micro imaging method |
| CN108362643A (en) * | 2018-02-10 | 2018-08-03 | 北京工业大学 | A kind of double altitudes illumination Fourier's lamination micro imaging method |
| WO2019170464A1 (en) * | 2018-03-08 | 2019-09-12 | Carl Zeiss Microscopy Gmbh | Microscope and method for viewing a sample using a microscope in order to display images with an extended focal depth or three-dimensional images |
| CN109815919A (en) * | 2019-01-28 | 2019-05-28 | 上海七牛信息技术有限公司 | A kind of people counting method, network, system and electronic equipment |
| CN110188685A (en) * | 2019-05-30 | 2019-08-30 | 燕山大学 | A kind of object count method and system based on the multiple dimensioned cascade network of double attentions |
| CN110501278A (en) * | 2019-07-10 | 2019-11-26 | 同济大学 | A kind of method for cell count based on YOLOv3 and density estimation |
| CN110378981A (en) * | 2019-07-19 | 2019-10-25 | 中国科学院长春光学精密机械与物理研究所 | Fourier's lamination microscope pupil restoration methods neural network based |
| CN110516584A (en) * | 2019-08-22 | 2019-11-29 | 杭州图谱光电科技有限公司 | A kind of Auto-counting of Cells method based on dynamic learning of microscope |
Non-Patent Citations (3)
| Title |
|---|
| SHAOSHUAI SHI 等: "Part-A^2 Net：3D Part-Aware and Aggregation Neural Network for Object Detection from Point Cloud", 《ARXIV》, 8 July 2019 (2019-07-08), pages 1 - 10 * |
| XING WANG 等: "SO-YOLO Based WBC Detection With Fourier Ptychographic Microscopy", 《IEEE ACCESS》, vol. 6, 23 August 2018 (2018-08-23), pages 51566 - 51576, XP055903026, DOI: 10.1109/ACCESS.2018.2865541 * |
| 王铭群: "傅里叶叠层显微成像系统误差校正算法研究", 《中国优秀博硕士学位论文全文数据库（硕士）信息科技辑》, 15 January 2019 (2019-01-15), pages 138 - 4718 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI769647B (en) * | 2021-01-04 | 2022-07-01 | 鴻海精密工業股份有限公司 | Method and device for determining cell density, computer device and storage medium |
| CN113327233A (en) * | 2021-05-28 | 2021-08-31 | 北京理工大学重庆创新中心 | Cell image detection method based on transfer learning |
| CN115220211A (en) * | 2022-07-29 | 2022-10-21 | 江南大学 | Microscopic imaging system and method based on deep learning and light field imaging |
| CN115220211B (en) * | 2022-07-29 | 2024-03-08 | 江南大学 | Microscopic imaging system and method based on deep learning and light field imaging |
| CN116630294A (en) * | 2023-06-08 | 2023-08-22 | 南方医科大学南方医院 | Whole blood sample detection method and device based on deep learning and storage medium |
| CN116630294B (en) * | 2023-06-08 | 2023-12-05 | 南方医科大学南方医院 | Whole blood sample detection method and device based on deep learning and storage medium |
| CN116704503A (en) * | 2023-08-01 | 2023-09-05 | 昕传生物科技(北京)有限公司 | Machine learning-based three-dimensional lung slice cell classification and protein quantification method |
| CN116704503B (en) * | 2023-08-01 | 2023-10-13 | 昕传生物科技(北京)有限公司 | Machine learning-based three-dimensional lung slice cell classification and protein quantification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111105415A (en) | White blood cell large-field-of-view image detection system and method based on deep learning | |
| US9684960B2 (en) | Automated histological diagnosis of bacterial infection using image analysis | |
| US9349036B2 (en) | System and method for quality assurance in pathology | |
| US6181414B1 (en) | Infrared spectroscopy for medical imaging | |
| US9064304B2 (en) | Image quality assessment of microscopy images | |
| US8611620B2 (en) | Advanced digital pathology and provisions for remote diagnostics | |
| US20210217190A1 (en) | Analyzer for three-dimensionally analyzing a medical sample by means of a light field camera | |
| JPH07504283A (en) | How to confirm a normal biomedical specimen | |
| CN110736748A (en) | Immunohistochemical nuclear plasma staining section diagnosis method and system | |
| CN115032196B (en) | Full-scribing high-flux color pathological imaging analysis instrument and method | |
| CN108388853B (en) | Stepwise reconstruction and counting method for leucocyte and platelet coexistence hologram | |
| Caya et al. | Detection and counting of red blood cells in human urine using canny edge detection and circle hough transform algorithms | |
| CN112432902A (en) | Automatic detection system and method for judging cell number through peripheral blood cell morphology | |
| JP2007516428A (en) | A system for determining the staining quality of slides using a scatter plot distribution | |
| WO2022183078A1 (en) | Computational refocusing-assisted deep learning | |
| KR20010017092A (en) | Method for counting and analyzing morphology of blood cell automatically | |
| CN113327233A (en) | Cell image detection method based on transfer learning | |
| CN114518362A (en) | Sperm quality analysis device, system, method and readable storage medium | |
| Ghazali et al. | Microscopy image processing analaysis for automatic detection of human intestinal parasites ALO and TTO | |
| CA2279945A1 (en) | Infrared spectroscopy for medical imaging | |
| CN108387553A (en) | The piecemeal that hologram coexists for leucocyte and blood platelet is rebuild and differential counting method | |
| CN115050024B (en) | Intelligent real-time identification method and system for granulocytes with interpretability | |
| TW202238610A (en) | Optical system and optical image processing method by applying image restoration | |
| WO2023119267A1 (en) | Method and system for visualization of the structure of biological cells | |
| CN111751266A (en) | Mark-free cell nucleus scattering inversion method based on multi-scale gray level co-occurrence matrix |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200505 |