CN113158996A - Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence - Google Patents
Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence Download PDFInfo
- Publication number
- CN113158996A CN113158996A CN202110570264.0A CN202110570264A CN113158996A CN 113158996 A CN113158996 A CN 113158996A CN 202110570264 A CN202110570264 A CN 202110570264A CN 113158996 A CN113158996 A CN 113158996A
- Authority
- CN
- China
- Prior art keywords
- diatom
- electron microscope
- scanning electron
- image
- rois
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 30
- 238000001000 micrograph Methods 0.000 title claims abstract description 23
- 241000206761 Bacillariophyta Species 0.000 claims abstract description 38
- 238000001514 detection method Methods 0.000 claims abstract description 34
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 28
- 238000004458 analytical method Methods 0.000 claims abstract description 18
- 210000001519 tissue Anatomy 0.000 claims description 48
- 238000013527 convolutional neural network Methods 0.000 claims description 34
- 238000012549 training Methods 0.000 claims description 13
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 10
- 238000011156 evaluation Methods 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 8
- 210000005228 liver tissue Anatomy 0.000 claims description 7
- 210000005084 renal tissue Anatomy 0.000 claims description 7
- 230000004931 aggregating effect Effects 0.000 claims description 6
- 238000013145 classification model Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000007635 classification algorithm Methods 0.000 claims description 5
- 210000004072 lung Anatomy 0.000 claims description 5
- 241000894007 species Species 0.000 claims description 5
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims description 4
- 239000010931 gold Substances 0.000 claims description 3
- 229910052737 gold Inorganic materials 0.000 claims description 3
- 238000000120 microwave digestion Methods 0.000 claims description 3
- 238000005507 spraying Methods 0.000 claims description 3
- 238000003828 vacuum filtration Methods 0.000 claims description 3
- 238000010200 validation analysis Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000012552 review Methods 0.000 claims description 2
- 238000007689 inspection Methods 0.000 abstract description 5
- 206010013647 Drowning Diseases 0.000 description 10
- 238000011840 criminal investigation Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000017531 blood circulation Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 210000003734 kidney Anatomy 0.000 description 1
- 210000004185 liver Anatomy 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000005464 sample preparation method Methods 0.000 description 1
- 238000001878 scanning electron micrograph Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000003319 supportive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
Abstract
The invention discloses a diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence, which is based on the images of a scanning electron microscope, adopts two types of artificial intelligence algorithms to automatically identify the position of diatoms and automatically classify the types of diatoms through two steps respectively, deeply integrates the process with control software of the scanning electron microscope through software automation, presents the analysis result of the algorithms to forensic personnel experts in the software process, and finally rechecks the analysis result of the artificial intelligence algorithms through the experience of the forensic personnel so as to combine the artificial intelligence algorithms with forensic experience, thereby obtaining more accurate and efficient diatom inspection results. The invention can greatly reduce the input of manpower, improve the detection accuracy, can realize the acquisition of the analysis result and the performance which are equivalent to the traditional forensic manual detection with higher efficiency and lower cost, can greatly improve the detection capability and efficiency of human operators and saves time.
Description
Technical Field
The invention belongs to the field of death identification in the forensic industry, and particularly relates to a diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence.
Background
In the field of forensics, the identification of causes of death of putrefactive bodies in water is one of the world-recognized forensics problems. When a medic investigates a corpse salvaged from water, victims belong to real drowning or dead-behind entry, and the key problem is often not easy to solve. Although many biological and chemical methods are described in the prior related literature to address these problems, "inspection by observation of diatom microscopic images" is still considered the gold standard.
Diatoms are an aquatic plankton, also known as phytoplankton, and are one of the most common plankton species found in the world's oceans, waterways, and soils. During the immersive drowning of the victim, if these diatoms are inhaled by the victim, they will be distributed through the blood circulation into the various organs of the victim. By identifying the water area in which the drowning person is located, and the types of diatoms present in the lung, kidney and liver tissues, experts can analyze for supportive evidence of drowning.
Diatoms typically range in size from 2 μm to 0.5 mm. Therefore, the identification method of diatoms is to observe the appearance and characteristics of diatoms through microscopic images, and can be performed by using an optical microscope or a Scanning Electron Microscope (SEM). In current workflow, a forensic person typically scans water, lung, liver and kidney tissue samples using a Scanning Electron Microscope (SEM) to obtain microscopic images of the tissue samples, as shown in fig. 1, and then the forensic person analyzes the resulting microscopic images through the human eye to identify and count the various types of diatoms and the amount present in the tissue.
However, finding and classifying diatoms with the aid of human eyes in scanning electron microscope images is a very expensive and burdensome task, because forensic staff needs to find the location of diatoms and judge their species in thousands of microscopic images at high magnification (hundreds to tens of thousands) through human eyes. It usually takes several hours for many experts to identify and calculate the various types of diatoms and classify the diatoms in the SEM images of the scanning electron microscope, and therefore the process is cumbersome and time consuming to operate and is extremely inefficient.
Therefore, the application of the electron microscope in diatom is to solve the scanning problem firstly, no omission is achieved, rapidness is achieved, automatic scanning is the first step, on the basis, automatic identification of diatom is achieved through artificial intelligence image identification, efficient and convenient tools are provided for criminal investigation personnel, and the method is embodied in a patent of ' a high-precision diatom detection and identification method and system based on scanning electron microscope images ' (patent number: 202010495609.6) ' which is applied in the company earlier.
However, in practical applications, the above method is often limited in practical applications because speed and accuracy cannot be compatible. In order to really realize the practical application of an electron microscope in diatom criminal investigation, the invention provides a two-step method for realizing automatic high-precision identification and classification of diatom, and provides a more practical tool for criminal investigation and environmental detection.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence, which is used for helping a forensic to automatically analyze and identify whether diatoms exist in a drowned water area and the types of subjects of the diatoms in the drowned water area in a tissue sample of the dead, and further provides forensic evidence for the forensic to identify whether the dead is drowned or enters water after death.
In order to solve the technical problems and achieve the technical effects, the invention is realized by the following technical scheme:
a diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence comprises the following steps:
s1, collecting a tissue sample, preprocessing the tissue sample, and preparing the tissue sample into a tissue sample for observation by a scanning electron microscope;
step S2, the prepared tissue sample is loaded into a sample bin of a scanning electron microscope for diatom analysis, and the parameter setting of the scanning electron microscope is completed;
step S3, the scanning electron microscope automatically scans and photographs the tissue sample according to the set magnification factor to obtain the scanning image of the tissue sample;
step S4, sending the obtained scanned image of the tissue sample into a convolutional neural network AI 1, wherein the convolutional neural network AI 1 is a diatom intelligent recognition algorithm AI, and analyzing the scanned image of the tissue sample to automatically recognize the positions of diatom target areas ROIs, and the diatom target areas ROIs represent the positions of suspected diatoms on the microscope image with the set magnification;
step S5, automatically transmitting the position coordinates of the identified diatom target areas ROIs back to the scanning electron microscope, and the scanning electron microscope scans and photographs the diatom target areas ROIs at a higher magnification according to the returned position coordinates, so as to obtain high-magnification scanning images of the diatom target areas ROIs;
step S6, sending the obtained high-magnification scanned image of the diatom target area ROIs into a convolutional neural network AI2 independent of a convolutional neural network AI 1, wherein the convolutional neural network AI2 is a diatom classification algorithm AI, analyzing the high-magnification scanned image of the diatom target area ROIs, automatically confirming whether the high-magnification scanned image of the diatom target area ROIs really contains diatoms, automatically classifying the correct diatom species visible on the high-magnification scanned image of the diatom target area ROIs, and giving a matching and classifying result;
and step S7, collecting and summarizing the diatom identification and classification results of all the diatom target areas ROIs, and outputting or displaying the diatom identification and classification results of all the diatom target areas ROIs.
Further, in step S1, the tissue sample includes one or more of a water body sample, a lung tissue sample, a liver tissue sample, or a kidney tissue sample.
Further, in step S1, the tissue sample is pretreated by a series of operations including microwave digestion, vacuum filtration, and gold spraying.
Further, in step S2, typical parameters of the scanning electron microscope are set to be 800 times amplification factor, backscatter detector, voltage 10kV, vacuum degree 10 Pa. The parameter is recommended based on experience, other magnification factors can be used, and the fixed parameter is mainly used for obtaining scanning electron microscope pictures under unified conditions, so that AI analysis is facilitated.
Further, in step S3, the scanning electron microscope automatically scans and photographs the tissue sample at a magnification of 800 times to obtain a scanned image of the tissue sample, in which case the resolution of the obtained scanned image of the tissue sample is about 0.33 μm or the FOV of the field of view is about 336 μm.
Further, in step S4, the specific method for identifying the positions of the diatom target areas ROIs by the convolutional neural network AI 1 is as follows:
s4.1, preprocessing an original image collected from the scanning electron microscope;
s4.2 if the image processed in step S4.1 is an image of liver tissue or kidney tissue, a complete diatom is randomly selected on the image: a uniform random sampling mode is adopted on the sampling area of the diatom, so that the extracted image blocks with specific sizes certainly and completely contain the diatom;
s4.3, inputting the image blocks extracted in the step S4.2 into a general diatom detection network for training, and adopting the sum of Focal local functions
Combining Loss functions forming the general diatom detection network, wherein the Focal local function is used for defining probability prediction error,
the function is used for defining the position prediction error;
s4.4, preprocessing the image to be detected according to the method of the step S4.1, dividing the image to be detected into a plurality of overlapped sub-images with the same image size to ensure the detection integrity of the boundary area, simultaneously recording the offset information of each sub-image on the original image to be detected, inputting the data of the sub-images into a trained general diatom detection network, outputting to obtain the diatom position prediction result of each sub-image, then adding the offset information of the sub-image relative to the original image to be detected to the diatom position prediction result of each sub-image to obtain a prediction set
Wherein p isnTo predict the probability of an object being judged as a foreground target,
positions of the upper left corner and the lower right corner of the prediction object are determined;
s4.5 aggregating the predictions
The following operations are performed:
aggregating the predictions
According to pnSorting from big to small;
the first prediction result Pr1And each of the remaining prediction results Prn≠1Calculating the position intersection ratio according to the following formula:
wherein A is the predicted result Pr1The position of diatom in the solution B is the predicted result Prn≠1The diatom location of (a);
calculating the predicted probability mean, e.g. the predicted probability mean is greater than a given threshold TDiatomCalculating a position mean value, and taking the position on the original image to be detected corresponding to the position mean value as a diatom detection target;
removing IoU values greater than a threshold T from the prediction setIoUAre predicted as the results of prediction of (1) Prm and Pr1；
S4.6 repeating the steps and sequentially iterating until the prediction result does not remain in the prediction set, and finally obtaining a group of diatom detection targets { Prdet_m}m＝1,...,M。
Further, in step S5, the scanning electron microscope automatically scans and photographs the diatom target area ROIs at a magnification of about 3000 to 20000 times according to the returned position coordinates, so as to obtain a high-magnification scanning image of the diatom target area ROIs, which is equivalent to using a higher resolution or a smaller field of view FOV; the automatic calculation method of the magnification comprises the following steps: field of view FOV/suspected diatom size at 800 x 800 and not higher than 20000 x.
Further, in step S6, the specific method for the convolutional neural network AI2 to confirm whether the high-magnification scanned image of the diatom target areas ROIs indeed contains diatoms is as follows:
s6.1, extracting each diatom image from the high-definition sample image, and manually marking to determine the category of the diatom; the various diatoms were divided into 2 groups, one for training and the other for validation;
s6.2, constructing a convolutional neural network AI2 model, and carrying out automatic diatom classification model training on the model by using a training sample;
s6.3, performing model evaluation and error analysis on the trained classification model based on the verification set;
s6.4, if the model evaluation index is not reached, continuing to optimize from multiple aspects, and iteratively updating the model;
s6.5, if the model evaluation index is reached, implementing model deployment and application;
s6.6, the basic network selects network structures such as MobileNet v2, inclusion v3 or EfficientNet b4 and the like to extract features, so that the balance of detection accuracy and detection speed is realized.
Further, in step S7, the output or displayed diatom identification and classification results of all diatom target regions ROIs are presented to the forensic staff for review, and the final result obtained by combining the artificial intelligence algorithm and the forensic expert experience is obtained as an evidence for forensic identification.
Further, the automatic diatom identification process of the convolutional neural network AI 1 and the automatic diatom classification process of the convolutional neural network AI2 are deeply integrated with control software of a scanning electron microscope through software automation.
It should be noted that "diatom identification" in the present invention refers to finding the location of diatom from a large whole tissue microscopic image (giga pixels) or a large number of (1024 × 1024) images obtained by dividing the whole microscopic image into smaller images, without analyzing the type of diatom. The "classification of diatoms" in the present invention refers to identifying the type (or sub-type) of diatoms based on high resolution images (typically thousands to tens of thousands) of specific diatoms.
Automatic identification or automatic classification of diatoms in microscopic images has been the subject of a number of prior academic research papers. Regarding diatom identification, a few people are currently known to automatically identify the number of diatoms in microscopic images of the whole sample in a drowning forensic case by using artificial intelligence, particularly by using a convolutional neural network, but at present, artificial intelligence is only used for identification of diatoms, but not for classification of diatoms.
While diatom classification is a complete and complex research field, although there are several successful diatom classification algorithms, since diatom identification algorithms usually depend to a large extent on the specific properties and parameters of microscope hardware and the background image (tissue) of the diatom in these microscopic images, it is a challenging task to develop an end-to-end diatom identification and classification algorithm. In fact, to our knowledge, such algorithms have not been developed at present.
The method provided by the invention is based on the image of the scanning electron microscope, two types of artificial intelligent algorithms are adopted to automatically identify the position of the diatom and automatically classify the type of the diatom through two steps respectively, the process is deeply integrated with the control software of the scanning electron microscope through software automation, the analysis result of the algorithms is presented to a forensic personnel expert in the software process, and finally the analysis result of the artificial intelligent algorithm is rechecked through the experience of the forensic personnel, so that the artificial intelligent algorithms and the forensic experience are combined once, and the more accurate and efficient diatom inspection result is obtained.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the method, the diatom identification artificial intelligence algorithm and the diatom classification artificial intelligence algorithm are divided into two steps to perform deep cooperation, and the two artificial intelligence algorithms are integrated into the control software of the scanning electron microscope, so that the diatom full-automatic identification and classification of the scanning electron microscope are realized, the supporting evidence for forensic drowning case analysis and inspection is obtained, the manual investment is greatly reduced, and the detection accuracy is improved.
2. In practical application, the type and the quantity of the diatom are the most important information for judging drowning, and the most basic criteria are provided for drowning places and drowning reasons. Therefore, the analysis result obtained by the diatom recognition and classification artificial intelligence algorithm is combined with the expert experience of the forensic personnel, and the obtained final result can be used as the evidence of the forensic drowning case, so that the equivalent analysis result and performance of the forensic artificial detection can be obtained with higher efficiency and lower cost, and the detection capability and efficiency of human operators can be greatly improved.
3. The diatom identification AI of the present invention is a convolutional neural network, and is a boundary object detector, such as an object detection algorithm (SSD). Neural networks of this type are common in, for example, automotive applications, but applications in diatom detection have not been discovered at present. Such boundary target detectors are based on the currently most advanced neural networks, such as YOLO, RetinaNet or mobilenenet.
4. The diatom classification AI algorithm is not novel when being independently seen, but the idea that the diatom identification AI algorithm and the diatom classification AI algorithm are combined as a two-step flow is the first creation of the invention and is bright ahead of sight, and the result of the two-step method can be completely automated and integrated into software of a scanning electron microscope, which is the greatest innovation of the invention.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to be implemented according to the content of the description, the following detailed description is given with reference to the accompanying drawings, which are a preferred embodiment of the present invention. The detailed description of the present invention is given in detail by the following examples and the accompanying drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
figure 1 is three adjacent diatoms found in high resolution (several thousand fold) scanning electron microscope images.
FIG. 2 is a schematic diagram of the diatom two-step identification and classification method of the present invention.
FIG. 3 is a schematic diagram of the training and deployment process of the diatom two-step identification and classification method of the present invention.
Fig. 4 is a scan image of 4 tissue samples with locations of diatom target areas ROIs identified by convolutional neural network AI 1 of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Referring to fig. 2-3, a diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence comprises the following steps:
and step S1, according to the conventional sample preparation method of microwave digestion-filter membrane enrichment-scanning electron microscope, the forensic personnel collects tissue samples of water, lung, liver, kidney and the like, and prepares the tissue samples for observation by a scanning electron microscope after a series of pretreatment operations such as microwave digestion, vacuum filtration, gold spraying and the like.
Step S2, the prepared tissue sample is loaded into a sample bin of a scanning electron microscope for diatom analysis, and the parameter setting of the scanning electron microscope is completed;
typical parameters of the scanning electron microscope are set as 800 times of amplification factor, a back scattering detector, 10kV of voltage and 10Pa of vacuum degree. The parameter is recommended based on experience, other magnification factors can be used, and the fixed parameter is mainly used for obtaining scanning electron microscope pictures under unified conditions, so that AI analysis is facilitated.
Step S3, the scanning electron microscope automatically scans and photographs the tissue sample according to the set magnification factor to obtain the scanning image of the tissue sample;
for example, a scanning electron microscope may automatically scan and photograph a tissue sample at a magnification of 800 to obtain a scanned image of the tissue sample, in which case the resolution of the scanned image of the tissue sample obtained is about 0.33 μm or the FOV of the field of view is about 336 μm.
Step S4, sending the obtained scanned image of the tissue sample into a convolutional neural network AI 1, wherein the convolutional neural network AI 1 is a diatom intelligent recognition algorithm AI, and analyzing the scanned image of the tissue sample to automatically recognize the positions of diatom target areas ROIs, and the diatom target areas ROIs represent the positions of suspected diatoms on the microscope image with the set magnification;
referring to fig. 4, fig. 4 shows the locations of diatom target areas ROIs in the scanned images of 4 tissue samples successfully identified by convolutional neural network AI 1, and the coordinates of the box marked in fig. 4 represent the smallest rectangle containing suspected diatoms.
The specific method for identifying the positions of the diatom target areas ROIs of the convolutional neural network AI 1 is as follows:
s4.1, preprocessing an original image collected from the scanning electron microscope;
s4.2 if the image processed in step S4.1 is an image of liver tissue or kidney tissue, a complete diatom is randomly selected on the image: a uniform random sampling mode is adopted on the sampling area of the diatom, so that the extracted image blocks with specific sizes certainly and completely contain the diatom;
s4.3, inputting the image blocks extracted in the step S4.2 into a general diatom detection network for training, and adopting the sum of Focal local functions
Combining Loss functions forming the general diatom detection network, wherein the Focal local function is used for defining probability prediction error,
the function is used for defining the position prediction error;
s4.4, preprocessing the image to be detected according to the method of the step S4.1, dividing the image to be detected into a plurality of overlapped sub-images with the same image size to ensure the detection integrity of the boundary area, simultaneously recording the offset information of each sub-image on the original image to be detected, inputting the data of the sub-images into a trained general diatom detection network, outputting to obtain the diatom position prediction result of each sub-image, then adding the offset information of the sub-image relative to the original image to be detected to the diatom position prediction result of each sub-image to obtain a prediction set
Wherein p isnTo predict the probability of an object being judged as a foreground target,
positions of the upper left corner and the lower right corner of the prediction object are determined;
s4.5 aggregating the predictions
The following operations are performed:
aggregating the predictions
According to pnSorting from big to small;
the first prediction result Pr1And each of the remaining prediction results Prn≠1Calculating the position intersection ratio according to the following formula:
wherein A is the predicted result Pr1The position of diatom in the solution B is the predicted result Prn≠1The diatom location of (a);
calculating the predicted probability mean, e.g. the predicted probability mean is greater than a given threshold TDiatomCalculating a position mean value, and taking the position on the original image to be detected corresponding to the position mean value as a diatom detection target;
removing IoU values greater than a threshold T from the prediction setIoUAre predicted as the results of prediction of (1) Prm and Pr1；
S4.6 repeating the steps and sequentially iterating until the prediction result does not remain in the prediction set, and finally obtaining a group of diatom detection targets { Prdet_m}m＝1,...,M。
Step S5, automatically transmitting the position coordinates of the identified diatom target areas ROIs back to the scanning electron microscope, and the scanning electron microscope scans and photographs the diatom target areas ROIs at a higher magnification according to the returned position coordinates, so as to obtain high-magnification scanning images of the diatom target areas ROIs;
for example, the SEM automatically scans and photographs the diatom target areas ROIs at a magnification of about 3000 to 20000 times (the magnification is automatically calculated by the method of 800 times of FOV/suspected diatom size 800 and not more than 20000 times) based on the returned position coordinates, thereby obtaining high magnification scanned images of the diatom target areas ROIs, which is equivalent to using a higher resolution or a smaller FOV.
Step S6, sending the obtained high-magnification scanned image of the diatom target area ROIs into a convolutional neural network AI2 independent of a convolutional neural network AI 1, wherein the convolutional neural network AI2 is a diatom classification algorithm AI, analyzing the high-magnification scanned image of the diatom target area ROIs, automatically confirming whether the high-magnification scanned image of the diatom target area ROIs really contains diatoms, automatically classifying the correct diatom species visible on the high-magnification scanned image of the diatom target area ROIs, and giving a matching and classifying result;
the specific method for confirming whether the high-magnification scanning image of the diatom target areas ROIs actually contains diatoms by the convolutional neural network AI2 is as follows:
s6.1, extracting each diatom image from the high-definition sample image, and manually marking to determine the category of the diatom; the various diatoms were divided into 2 groups, one for training and the other for validation;
s6.2, constructing a convolutional neural network AI2 model, and carrying out automatic diatom classification model training on the model by using a training sample;
s6.3, performing model evaluation and error analysis on the trained classification model based on the verification set;
s6.4, if the model evaluation index is not reached, continuing to optimize from multiple aspects, and iteratively updating the model;
s6.5, if the model evaluation index is reached, implementing model deployment and application;
s6.6, the basic network selects network structures such as MobileNet v2, inclusion v3 or EfficientNet b4 and the like to extract features, so that the balance of detection accuracy and detection speed is realized.
And S7, collecting and summarizing the diatom identification and classification results of all the diatom target areas ROIs, outputting or displaying the diatom identification and classification results of all the diatom target areas ROIs for being presented to forensic personnel for rechecking, and obtaining a final result combining an artificial intelligence algorithm and forensic expert experience to be used as a corroboration of forensic identification.
Further, the automatic diatom identification process of the convolutional neural network AI 1 and the automatic diatom classification process of the convolutional neural network AI2 are deeply integrated with control software of a scanning electron microscope through software automation.
It should be noted that "diatom identification" in the present invention refers to finding the location of diatom from a large whole tissue microscopic image (giga pixels) or a large number of (1024 × 1024) images obtained by dividing the whole microscopic image into smaller images, without analyzing the type of diatom. The "classification of diatoms" in the present invention refers to identifying the type (or sub-type) of diatoms based on high resolution images (typically thousands to tens of thousands times as shown in fig. 1) of a specific diatom.
The method provided by the invention is based on the image of the scanning electron microscope, two types of artificial intelligent algorithms are adopted to automatically identify the position of the diatom and automatically classify the type of the diatom through two steps respectively, the process is deeply integrated with the control software of the scanning electron microscope through software automation, the analysis result of the algorithms is presented to a forensic personnel expert in the software process, and finally the analysis result of the artificial intelligent algorithm is rechecked through the experience of the forensic personnel, so that the artificial intelligent algorithms and the forensic experience are combined once, and the more accurate and efficient diatom inspection result is obtained.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence is characterized by comprising the following steps:
s1, collecting a tissue sample, preprocessing the tissue sample, and preparing the tissue sample into a tissue sample for observation by a scanning electron microscope;
step S2, the prepared tissue sample is loaded into a sample bin of a scanning electron microscope for diatom analysis, and the parameter setting of the scanning electron microscope is completed;
step S3, the scanning electron microscope automatically scans and photographs the tissue sample according to the set magnification factor to obtain the scanning image of the tissue sample;
step S4, sending the obtained scanned image of the tissue sample into a convolutional neural network AI 1, wherein the convolutional neural network AI 1 is a diatom intelligent recognition algorithm AI, and analyzing the scanned image of the tissue sample to automatically recognize the positions of diatom target areas ROIs, and the diatom target areas ROIs represent the positions of suspected diatoms on the microscope image with the set magnification;
step S5, automatically transmitting the position coordinates of the identified diatom target areas ROIs back to the scanning electron microscope, and the scanning electron microscope scans and photographs the diatom target areas ROIs at a higher magnification according to the returned position coordinates, so as to obtain high-magnification scanning images of the diatom target areas ROIs;
step S6, sending the obtained high-magnification scanned image of the diatom target area ROIs into a convolutional neural network AI2 independent of a convolutional neural network AI 1, wherein the convolutional neural network AI2 is a diatom classification algorithm AI, analyzing the high-magnification scanned image of the diatom target area ROIs, automatically confirming whether the high-magnification scanned image of the diatom target area ROIs really contains diatoms, automatically classifying the correct diatom species visible on the high-magnification scanned image of the diatom target area ROIs, and giving a matching and classifying result;
and step S7, collecting and summarizing the diatom identification and classification results of all the diatom target areas ROIs, and outputting or displaying the diatom identification and classification results of all the diatom target areas ROIs.
2. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S1, the tissue sample includes one or more of a water body sample, a lung tissue sample, a liver tissue sample, or a kidney tissue sample.
3. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S1, the tissue sample is pretreated by a series of operations including microwave digestion, vacuum filtration, and gold spraying.
4. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S2, the parameters of the scanning electron microscope are set to be 800 times of magnification, the backscatter detector, voltage 10kV, and vacuum 10 Pa.
5. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S3, the scanning electron microscope automatically scans and photographs the tissue sample at a magnification of 800 times to acquire a scanned image of the tissue sample, in which case the resolution of the acquired scanned image of the tissue sample is 0.33 μm or the FOV of the field of view is 336 μm.
6. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S4, the specific method for identifying the positions of the diatom target areas ROIs by the convolutional neural network AI 1 is as follows:
s4.1, preprocessing an original image collected from the scanning electron microscope;
s4.2 if the image processed in step S4.1 is an image of liver tissue or kidney tissue, a complete diatom is randomly selected on the image: a uniform random sampling mode is adopted on the sampling area of the diatom, so that the extracted image blocks with specific sizes certainly and completely contain the diatom;
s4.3, inputting the image blocks extracted in the step S4.2 into a general diatom detection network for training, and adopting the sum of Focal local functions
Function(s)To combine the Loss functions that make up the universal diatom detection network, wherein the Focal local function is used to define the probabilistic prediction error,
the function is used for defining the position prediction error;
s4.4, preprocessing the image to be detected according to the method of the step S4.1, dividing the image to be detected into a plurality of overlapped sub-images with the same image size to ensure the detection integrity of the boundary area, simultaneously recording the offset information of each sub-image on the original image to be detected, inputting the data of the sub-images into a trained general diatom detection network, outputting to obtain the diatom position prediction result of each sub-image, then adding the offset information of the sub-image relative to the original image to be detected to the diatom position prediction result of each sub-image to obtain a prediction set
Wherein p isnTo predict the probability of an object being judged as a foreground target,
positions of the upper left corner and the lower right corner of the prediction object are determined;
s4.5 aggregating the predictions
The following operations are performed:
aggregating the predictions
According to pnSorting from big to small;
the first prediction result Pr1And each of the remaining prediction results Prn≠1Calculating the position intersection ratio according to the following formula:
where A is the predicted outcome Pr1The position of diatom in the solution B is the predicted result Prn≠1The diatom location of (a);
calculating the predicted probability mean, e.g. the predicted probability mean is greater than a given threshold TDiatomCalculating a position mean value, and taking the position on the original image to be detected corresponding to the position mean value as a diatom detection target;
removing IoU values greater than a threshold T from the prediction setIoUAre predicted as the results of prediction of (1) Prm and Pr1；
S4.6 repeating the steps and sequentially iterating until the prediction result does not remain in the prediction set, and finally obtaining a group of diatom detection targets { Prdet_m}m＝1,...,M。
7. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S5, the scanning electron microscope automatically scans and photographs the diatom target regions ROIs at a magnification of 3000 to 20000 times according to the returned position coordinates, so as to obtain high-magnification scanned images of the diatom target regions ROIs, which is equivalent to using a higher resolution or a smaller field of view FOV;
the automatic calculation method of the magnification comprises the following steps: field of view FOV/suspected diatom size at 800 x 800 and not higher than 20000 x.
8. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S6, the specific method for the convolutional neural network AI2 to confirm whether the high-magnification scanned images of the diatom target areas ROIs actually contain diatoms is as follows:
s6.1, extracting each diatom image from the high-definition sample image, and manually marking to determine the category of the diatom; the various diatoms were divided into 2 groups, one for training and the other for validation;
s6.2, constructing a convolutional neural network AI2 model, and carrying out automatic diatom classification model training on the model by using a training sample;
s6.3, performing model evaluation and error analysis on the trained classification model based on the verification set;
s6.4, if the model evaluation index is not reached, continuing to optimize from multiple aspects, and iteratively updating the model;
s6.5, if the model evaluation index is reached, implementing model deployment and application;
s6.6, the basic network selects network structures such as MobileNet v2, inclusion v3 or EfficientNet b4 and the like to extract features, so that the balance of detection accuracy and detection speed is realized.
9. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: in step S7, all the outputted or displayed diatom identification and classification results of all diatom target regions ROIs are presented to forensic personnel for review, and the final result obtained by combining the artificial intelligence algorithm and the forensic expert experience is used as a corroboration for forensic identification.
10. The diatom two-step identification and classification method based on scanning electron microscope images and artificial intelligence of claim 1, wherein: the automatic diatom identification process of the convolutional neural network AI 1 and the automatic diatom classification process of the convolutional neural network AI2 are deeply integrated with control software of a scanning electron microscope through software automation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110570264.0A CN113158996A (en) | 2021-05-25 | 2021-05-25 | Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110570264.0A CN113158996A (en) | 2021-05-25 | 2021-05-25 | Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113158996A true CN113158996A (en) | 2021-07-23 |
Family
ID=76877399
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110570264.0A Withdrawn CN113158996A (en) | 2021-05-25 | 2021-05-25 | Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113158996A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117422696A (en) * | 2023-11-08 | 2024-01-19 | 河北工程大学 | Belt wear state detection method based on improved YOLOv8-Efficient Net |
-
2021
- 2021-05-25 CN CN202110570264.0A patent/CN113158996A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117422696A (en) * | 2023-11-08 | 2024-01-19 | 河北工程大学 | Belt wear state detection method based on improved YOLOv8-Efficient Net |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7198577B2 (en) | Image analysis method, device, program, and method for manufacturing trained deep learning algorithm | |
| JP7076698B2 (en) | Image analysis method, image analysis device, program, learned deep learning algorithm manufacturing method and learned deep learning algorithm | |
| CA2921325C (en) | Systems and methods for comprehensive multi-assay tissue analysis | |
| US8600143B1 (en) | Method and system for hierarchical tissue analysis and classification | |
| US6463425B2 (en) | Neural network assisted multi-spectral segmentation system | |
| CN111488921A (en) | Panoramic digital pathological image intelligent analysis system and method | |
| JP2006333710A (en) | Automatic system for judging quality of cell | |
| WO2017150194A1 (en) | Image processing device, image processing method, and program | |
| CN109903282B (en) | Cell counting method, system, device and storage medium | |
| CN110736748A (en) | Immunohistochemical nuclear plasma staining section diagnosis method and system | |
| CN103793709A (en) | Cell recognition method and device, and urine analyzer | |
| US20040014165A1 (en) | System and automated and remote histological analysis and new drug assessment | |
| CN113158996A (en) | Two-step diatom identification and classification method based on scanning electron microscope images and artificial intelligence | |
| EP3837664A1 (en) | Generating annotation data of tissue images | |
| WO2020007587A1 (en) | Method and apparatus for training a neural network classifier to classify an image depicting one or more objects of a biological sample | |
| CN110807754A (en) | Fungus microscopic image segmentation detection method and system based on deep semantic segmentation | |
| CN116757998A (en) | Screening method and device for CTC cells and CTC-like cells based on AI | |
| Jiménez et al. | Image quality metrics applied to digital pathology | |
| CN110084270A (en) | Pathological section image-recognizing method and equipment | |
| Alhammad et al. | Efficient dnn-based classification of whole slide gram stain images for microbiology | |
| Li et al. | A Deep Learning based Method for Microscopic Object Localization and Classification | |
| US20230291992A1 (en) | Method and system for event-based imaging | |
| CN117975453A (en) | Image analysis method, device, program and method for manufacturing deep learning algorithm after learning | |
| Bonam et al. | Toward automated quantification of biological microstructures using unbiased stereology | |
| Morera et al. | MIMO YOLO-A Multiple Input Multiple Output Model for Automatic Cell Counting |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210723 |