CN113539476A

CN113539476A - Stomach endoscopic biopsy Raman image auxiliary diagnosis method and system based on artificial intelligence

Info

Publication number: CN113539476A
Application number: CN202110613183.4A
Authority: CN
Inventors: 季敏标; 刘至杰; 胡皓; 敖建鹏; 周平红
Original assignee: Fudan University
Current assignee: Fudan University
Priority date: 2021-06-02
Filing date: 2021-06-02
Publication date: 2021-10-22

Abstract

The invention belongs to the technical field of medical equipment, and particularly relates to a gastric endoscopic biopsy Raman image auxiliary diagnosis method and system based on artificial intelligence. The invention firstly uses the artificial intelligence technology in the auxiliary diagnosis of the gastroscope stimulated Raman scattering endoscopic biopsy tissue image, obtains histopathology image information by using the stimulated Raman scattering microscopic imaging technology, and then constructs the auxiliary diagnosis system of the stomach endoscopic biopsy Raman image by adopting image classification and image omics data analysis based on deep learning neural network and machine learning; the system comprises a gastric tissue Raman image data preprocessing module, an algorithm module containing a neural network model and training thereof, a neural network fine tuning module and a testing module; compare in present traditional endoscope system of diagnosing, its advantage embodies: real-time, rapid and intelligent diagnosis support in the endoscopic examination process is realized without the need of explanation by a pathologist.

Description

Stomach endoscopic biopsy Raman image auxiliary diagnosis method and system based on artificial intelligence

Technical Field

The invention belongs to the technical field of medical equipment, and particularly relates to a gastric endoscopic biopsy Raman image auxiliary diagnosis method and system based on artificial intelligence.

Background

Gastric cancer is a serious disease affecting human health. According to the latest annual report of national tumor registration centers, the number of new gastric cancer patients is over 68 ten thousand, about 50 ten thousand of gastric cancer patients die, and the new gastric cancer patients become the second most lethal tumor in China. The key point of the prevention and control of the gastric cancer lies in early diagnosis and early treatment. Early cancer is found in time, precancerous lesion is delayed and reversed, and the incidence rate and the fatality rate of gastric cancer can be effectively reduced. The stomach precancer and precancerous lesion can not be diagnosed by observing the form by an endoscope, and the stomach precancer and precancerous lesion must be examined by pathological tissues at present, namely, an endoscope and biopsy mode is adopted. However, this model has significant time lag and subjectivity and is not compatible with current medical service requirements. For doctors, the pathological diagnosis result cannot be obtained while performing endoscopic examination, and quick diagnosis and treatment service and treatment guidance cannot be provided for patients; for patients, secondary treatment, queuing and registration increase the economic cost and time cost. Meanwhile, pathological diagnosis depends heavily on the diagnosis level of a pathological doctor, and the subjectivity is strong. For the diagnosis of the early cancer and other difficult diseases, the technical requirements of a diagnostician are high. Therefore, a real-time rapid diagnosis system of the gastric cancer which can be compared with the pathological gold standard is established, a diagnosis method which is suitable for rapid examination is innovated, and the method has great scientific significance and practical requirements for changing the current gastric cancer endoscope diagnosis mode.

The current gastric cancer endoscope diagnosis mode lacks obvious morphological characteristics due to gastric cancer and precancerous lesions, and the performance cannot be clear by only depending on the endoscope, so that the diagnosis must depend on histopathological diagnosis. The latter requires multiple steps of dehydration, embedding, sectioning, staining, scoring, etc., with significant time lag. However, the gastric cancer and precancerous lesion lack obvious morphological characteristics, and the performance cannot be clarified under the condition of single endoscope, so that the diagnosis of histopathology is required. The latter requires multiple steps of dehydration, embedding, sectioning, staining, scoring, etc., with significant time lag. In large hospitals, pathological reports are often obtained after endoscopic examination for several days, and an endoscopic doctor cannot realize quick diagnosis of lesions while performing endoscopic examination, so that an instant diagnosis and treatment suggestion cannot be provided for a patient.

With the development of machine learning theory and the breakthrough of GPU technology, especially the progress of neural network, artificial intelligence has been developed rapidly in recent years, which greatly saves manpower and improves working efficiency and accuracy. The data is abstracted at a high level through a multilayer neural network structure formed by multiple nonlinear transformations, and a more abstract high-level representation attribute category or feature is formed by combining low-level features through a multilayer perceptron, so that artificial intelligence can simulate a certain human brain processing mechanism, particularly extraction of complex images, discrimination and fine features. Such as "certificate recognition", "number plate recognition", "face recognition", etc., which are well known.

In recent two years, the potential of AI in the field of medical diagnostics has become increasingly important. In 2016, Google collaborated with the university of california to release results in the journal of the top level of medicine, JAMA, which was trained by 13 million photographs of the retina, enabling the AI society to automatically detect diabetic retinopathy and macular edema. Its specificity and sensitivity for recognizing lesions is comparable to ophthalmologists. Recently, Stanford university published a research report on skin cancer diagnosis using a deep learning algorithm in Nature, and skin cancer was diagnosed using a deep learning algorithm. The result shows that the specificity and the sensitivity of the AI in the aspect of skin cancer diagnosis are even higher than those of medical experts, and the diagnosis accuracy can reach 91%. In the early stage, the early research results of 85% accuracy and 87.5% accuracy of transfer learning on the molybdenum target intelligent diagnosis of the breast cancer are realized by members of a project team by using a deep learning and transfer learning method. Meanwhile, the team members adopt a CNN network to realize the detection of various tiny hidden objects, the accuracy rate reaches 95.6%, the accuracy rate of recall reaches 0.976%, and the F-measure reaches 0.969, and the research results provide a method basis for the analysis and intelligent identification of the stimulated Raman image.

Because the images and spectral data of raman imaging are complex and there is no data accumulation and summarization of mature large samples, clinical interpretation is very difficult. The "blindness" of clinicians is currently a major factor impeding the clinical advancement of raman technology. Therefore, the artificial intelligence technology and the big data analysis summarization are helpful for converting the Raman result into a diagnosis conclusion which can be read and understood by a doctor, and are also the key for promoting the Raman technology and the artificial intelligence technology to serve the clinic.

Disclosure of Invention

The invention aims to provide a real-time and rapid gastric endoscopic biopsy Raman image auxiliary diagnosis method and system for gastric cancer endoscopic diagnosis.

The invention firstly provides a stomach endoscopic biopsy Raman image auxiliary diagnosis method based on artificial intelligence, which comprises the following specific steps:

s1, preprocessing the Raman image data of the stomach tissue: cutting the stimulated Raman histopathology image generated by the stimulated Raman microscopic imaging technology to meet the requirement of inputting the required size of a subsequent convolution neural network, and then using a data enhancement method to perform operations such as overturning, rotating, blurring and brightness adjustment on the Raman image so as to further increase a database; splitting a training set, a verification set and a test set from the stomach tissue Raman image data set; preparing for subsequent training;

s2, algorithm design: selecting a proper convolutional neural network model including a deep convolutional layer, a pooling layer and a full-link layer mapped to several categories according to the gastric tissue Raman image data set generated in S1, and designing a classification algorithm aiming at classifying different pathological categories according to the model; inputting the training set into an algorithm model for training, and tracking the convergence speed and accuracy of the algorithm in real time by using a verification set;

s3, further fine-tuning parameters used by the neural network on the basis of the trained network, such as the type of an optimizer, the learning rate, the weight deviation and the like, so as to improve the accuracy of the algorithm; and then, exchanging a part of the training set with the verification set to form a new training set and a new verification set, inputting the new training set into a neural network classification algorithm for training to form a cross-validation algorithm mode so as to prove the relevance and consistency between the training set and the verification set.

S4, testing the trained neural network model in the S3 by using an external test set, and detecting whether the untrained image can be correctly identified by an algorithm; on the basis of correct identification, an algorithm is designed to convert the identification result into auxiliary diagnosis of pathology by combining with a gold standard, specifically, a Raman histopathology image is input into a network, a judgment category is generated for each cut small image, the judgment category is integrated into the whole pathology image, and the judgment category is visualized for a doctor to read. And comparing the diagnosis result with the diagnosis result of the traditional histopathology mode, and verifying the consistency.

In step S1, if the number of data sets is still small and the normal data enhancement method is already used to a proper extent, the step of cutting the raman histopathology image can be improved by moving the cutting by a certain step distance, i.e. the cutting can be regarded as the step distance being one picture size long and wide, and after the cutting is shortened, more data sets can be obtained in the same raman histopathology image, and the edge features caused by cutting the picture can be preserved, so that the completeness of the features learned by the neural network is enhanced.

Based on the auxiliary diagnosis method for the Raman image of the endoscopic biopsy in the stomach, the invention also provides a rapid auxiliary diagnosis system for the Raman image of the endoscopic biopsy in the stomach, which specifically comprises the following steps: the system comprises a stomach tissue Raman image data preprocessing module, an algorithm module, a neural network fine-tuning module and a testing module; these four modules perform the content of the four steps in the secondary diagnostic method in sequence.

The invention carries out pathological diagnosis analysis on the stimulated Raman histopathological image through artificial intelligence, verifies the resolution recognition capability of the artificial neural network on the stimulated Raman image, compares the resolution recognition capability with the diagnosis result of the traditional diagnosis mode, verifies the consistency of the combination of the stimulated Raman imaging and the artificial intelligence diagnosis compared with the traditional diagnosis mode, and provides verification for the practicability of the invention.

The technical effect of the present invention is innovative compared to the conventional gastric cancer diagnosis process. The invention provides a rapid diagnosis method of stimulated Raman histopathology images based on artificial intelligence, which aims at the requirements of gastroscope and incisal margin operation on rapid diagnosis of intraoperative tissue imaging images, transplants image classification and deep learning technologies widely applied in the field of artificial intelligence. The innovation is realized as follows: after the stomach biopsy or incisal marginal tissue is subjected to rapid stimulated Raman imaging, a pathologist does not need to read the film, the subjective judgment of the pathologist is not brought, and based on a unified gold standard, a diagnosis result, the distribution situation of the cancer tissue in the whole sample image and the canceration situation can be given within seconds or tens of seconds.

Drawings

FIG. 1 is a flow chart of Raman image generation and network model training for gastric biopsy according to an embodiment of the present invention.

FIG. 2 is a process of sample graph segmentation and input into a network model.

FIG. 3 is a flow chart of the generation of a cancerous profile in an embodiment of the present invention.

FIG. 4 is a graph of model parameters as a function of cycle steps in network model training.

FIG. 5 is a complementary image of a sample tissue map after inversion amplification.

Fig. 6 shows the region segmented by the whole tissue sample when generating the canceration distribution map, and the generated canceration distribution map.

Reference numbers in the figures: 1-11 is a lipid channel stimulated Raman image, 1-2 is a protein channel minus lipid channel stimulated Raman image, 1-3 is a collagen second harmonic image, 2 is a Raman pseudo color image formed by synthesizing three images, 3 is a spliced Raman pseudo color image, 4 is a cut Raman pseudo color image, 5 is a data set formed by the cut Raman pseudo color image, 6 is a training set divided from the data set, 7 is a test set divided from the data set, 8 is a data set used as training after five equal divisions of the training set, 9 is a data set used as verification after five equal divisions of the training set, 10 is a convolutional neural network model, 11 is a model parameter finished by training of the training set, and 12 is prediction of the test data set by a network after five times of cross verification.

Detailed Description

The invention is further illustrated with reference to the following specific embodiments and the accompanying drawings.

Example 1

The invention provides a rapid diagnosis method of a stomach endoscopic biopsy Raman image based on artificial intelligence. In a specific embodiment, a process flow as shown in fig. 1 is set up. In the process, firstly, two channel images formed by a stimulated Raman imaging system, namely lipid 1-1, protein minus lipid 1-2 and second harmonic channel image collagen 1-3 are respectively mapped to different colors of a stomach and then are superposed to form a stimulated Raman histopathology image 2, then a splicing algorithm is used for generating a complete sample Raman pathology image 3, all the sample images are divided into small pictures 4 which accord with the input size of a neural network by using a division algorithm, as shown in figure 2, the input size in a network input-rest-v 2 used in the invention is 300x300x3, so that the sample pictures are cut into 300x300, then data enhancement operation is carried out by means of blurring, rotating, overturning and the like, an image with more stomach histopathology characteristics is formed into a data set 5, the data set 5 is divided into a training set 6 and a test set 7 according to cases, mixing all cases in the training set 6, performing five-fold cross validation, randomly dividing the training set 6 into five parts, wherein four parts are used as the training set 8, the other part is used as the validation set 9, performing five-fold cross validation, inputting the four training sets 8 of each cross validation into a convolutional neural network 10, performing real-time detection on model loss values and accuracy rates through the validation set 9 in real time by using a network 11 obtained through training, giving a real-time detection result, storing obtained model parameters to form a network model after the network 11 converges to be stable, performing five-fold cross validation to form five network models, and finally inputting the five network models into the network by using the test set 7 to obtain a prediction result 12. After training, the loss function values and classification accuracy of the training set and the validation set corresponding to each training cycle step number shown in fig. 4 are obtained.

In the embodiment, the deep learning algorithm of the convolutional neural network is utilized to classify the Raman images of the stomach biopsy tissue, the feasibility of the artificial neural network for carrying out pathological classification on the Raman imaging images of the fresh tissue of the stomach biopsy is verified, and a foundation is laid for generating the canceration distribution images of the samples in the embodiment 2.

Example 2

In this embodiment, a suitable classification test algorithm is selected. With reference to example 1, the algorithm for generating the distribution of fresh tissue carcinoma in gastric biopsy comprises the following steps:

s1, because the number of times of calculation of the image edge is small due to the direct use of the canceration distribution generation algorithm, firstly, the image is subjected to flip amplification according to the input size of the convolutional neural network, as shown in FIG. 5, the upper part and the lower part of the image are subjected to flip amplification, then, the left part and the right part of the image are subjected to flip amplification, and the amplified image is subjected to size extension in four directions;

s2, dividing the large graph of the picture according to the moving step distance of each selection of the small graph of the size of the network input image;

and S3, inputting the small graph sequences into the network, and generating the classification of the network model for the small graph sequences.

S4, filling the sorted results into a matrix, averaging the number of times each small image is calculated, and finally intercepting the part before inversion amplification as the result of the last cancerous distribution, as shown in fig. 6, where the red region is the cancerous distribution map in the fresh tissue of the stomach biopsy.

In the examples, a biopsy from a gastroscope is selected as a sample to illustrate the experimental ideas and features of the present invention. The protective scope of the invention is not limited to the embodiments described above. The invention is in the protection scope when applied to the artificial intelligent diagnosis of the lesion condition of the stimulated Raman image of the stomach biopsy tissue.

Claims

1. An artificial intelligence-based Raman image aided diagnosis method for endoscopic biopsy in stomach is characterized by comprising the following specific steps:

s1, preprocessing the Raman image data of the stomach tissue: cutting the stimulated Raman histopathology image generated by the stimulated Raman microscopic imaging technology to meet the requirement of the input of the subsequent convolution neural network on the required size, and then using a data enhancement method to perform turning, rotating, blurring and brightness adjustment operations on the Raman image so as to further increase a database; splitting a training set, a verification set and a test set from the stomach tissue Raman image data set;

s3, further fine-tuning parameters used by the neural network on the basis of the trained network, wherein the parameters comprise optimizer types, learning rates and weight deviation, so that the accuracy of the algorithm is improved; then, exchanging a part of the training set with the verification set to form a new training set and a new verification set, inputting the new training set into a neural network classification algorithm for training to form a cross-validation algorithm mode so as to prove the relevance and consistency between the training set and the verification set;

s4, testing the trained neural network model in the S3 by using an external test set, and detecting whether the untrained image can be correctly identified by an algorithm; on the basis of correct identification, an algorithm is designed to convert the identification result into auxiliary diagnosis of pathology by combining with a gold standard, specifically, a Raman histopathology image is input into a network, a judgment category is generated for each cut small image, the judgment category is integrated into the whole pathology image, and the judgment category is visualized for a doctor to read;

and comparing the diagnosis result with the diagnosis result of the traditional histopathology mode, and verifying the consistency.

2. The raman image-assisted diagnosis method for endoscopic biopsy according to claim 1, wherein in step S1, if the number of data sets is still small and the normal data enhancement method is already used to a proper degree, the step of cutting the raman histopathology image is changed to move by a certain step distance, i.e. the cutting can be regarded as the step distance is one picture size, and after the step distance is shortened, more data sets are obtained from the same tensilman histopathology image, and the edge features caused by cutting the picture are kept, so as to enhance the completeness of the features learned by the neural network.

3. An auxiliary diagnostic system for Raman image of endoscopic biopsy in stomach based on the auxiliary diagnostic method for Raman image of endoscopic biopsy in stomach of claim 1, comprising: the system comprises a stomach tissue Raman image data preprocessing module, an algorithm module, a neural network fine-tuning module and a testing module; the four modules sequentially execute the contents of steps S1, S2, S3, S4 in the auxiliary diagnostic method.