CN112741651B - Method and system for processing ultrasonic image of endoscope - Google Patents
Method and system for processing ultrasonic image of endoscope Download PDFInfo
- Publication number
- CN112741651B CN112741651B CN202011585174.0A CN202011585174A CN112741651B CN 112741651 B CN112741651 B CN 112741651B CN 202011585174 A CN202011585174 A CN 202011585174A CN 112741651 B CN112741651 B CN 112741651B
- Authority
- CN
- China
- Prior art keywords
- image
- mode
- fusion
- modes
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
- A61B8/5238—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image
- A61B8/5246—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image combining images from the same or different imaging techniques, e.g. color Doppler and B-mode
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
- A61B8/08—Detecting organic movements or changes, e.g. tumours, cysts, swellings
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Biophysics (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention provides a method and a system for processing an endoscope ultrasonic image, wherein the method comprises the following steps: acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F; and classifying the ultrasonic images in the three modes by adopting a multi-modal neural network to obtain a processing result for identifying the lymph nodes. The invention uses the image data of three modes in the CP-EBUS imaging technology, processes the images through the multi-modal neural network, can more accurately identify the lymph nodes of the ultrasonic images, and is favorable for improving the accuracy of lymph node diagnosis.
Description
Technical Field
The invention relates to the technical field of medical image processing, in particular to a method and a system for processing an endoscope ultrasonic image.
Background
The ultrasonic endoscope is an endoscope which fills some special indications which can not be covered by common endoscopes, body surface ultrasound, CT and the like. CP-EBUS (convex probe ultrasonic bronchoscope) is a minimally invasive intrathoracic lymph node diagnosis technique, which uses a probe equipped with a biopsy needle, an ultrasonic device and an endoscope to accurately enter the vicinity of a lymph node to be detected from a bronchus, and uses the biopsy needle to obtain the tissue components of the target lymph node, and can accurately diagnose the disease of a patient through pathological analysis of the extracted tissue. However, due to the small amount of tissue taken from a biopsy, the diagnosis of a biopsy is false negative by 20%. In the process of obtaining biopsy tissues, the ultrasonic device can obtain ultrasonic images of lymph nodes under different modalities, and the prior researches show that the ultrasonic images have important value for diagnosing the benign and malignant lymph nodes. According to the diagnosis result of the ultrasonic image, a doctor can be helped to select a proper lymph node for puncture in the biopsy process, and the diagnosis method can be used as a supplement of biopsy diagnosis to make up for the defects of the biopsy diagnosis.
Existing diagnostics for CP-EBUS images are classified as semi-quantitative or quantitative methods, where semi-quantitative methods rely primarily on physician observation of certain features in the image, e.g., the contour of lymph nodes in gray-scale B-mode images, which tend to be malignant if sharp and benign otherwise. The limitation of this type of method is that the accuracy of the diagnosis depends on the personal experience of the doctor, since the sharpness of the same B-mode image varies from person to person. The method is difficult to be applied in the areas with deficient medical resources, and simultaneously, the stability of the diagnosis accuracy rate is difficult to be ensured. Another type of quantitative method is to set a certain threshold value, usually using some statistical features of the image as indicators, and exceeding or falling below the threshold value, the lymph node is considered to be malignant or benign. The limitation of this method is that the setting of the threshold is often very close to the relation of the data set, and there is also a large difference between different subdivided disease categories. In addition to the limitations described above, the diagnostic accuracy of the above methods is often limited.
Through retrieval, the chinese patent CN201911310175.1 provides an image recognition method, apparatus and storage medium based on deep learning, the method is: firstly, acquiring an ultrasonic image in real time, and intercepting an image area in the ultrasonic image as an ultrasonic image; and then, automatically identifying the ultrasonic image by adopting a deep convolutional neural network algorithm, and judging whether the ultrasonic image contains the central lymph node. The method can improve the accuracy of the central lymph node image identification. The patent is only used for judging whether the ultrasonic image contains the central lymph node, cannot identify and process benign and malignant lymph nodes, and has little significance for assisting diagnosis of doctors.
Disclosure of Invention
Aiming at the problems, the invention provides a method and a system for processing an endoscope ultrasonic image, which can more accurately identify the lymph node of the ultrasonic image, thereby being beneficial to improving the accuracy of lymph node diagnosis and being suitable for CP-EBUS imaging technology.
In a first aspect of the present invention, there is provided a method for processing an endoscopic ultrasound image, including:
acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;
and classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes.
Optionally, the classifying the ultrasound image in three modes by using a multi-modal neural network includes:
extracting the image characteristics of the ultrasonic image of each mode;
performing feature fusion on the image features of each mode to obtain image fusion features of the three modes;
and classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.
Optionally, the extracting the image feature of the ultrasound image of each mode includes:
inputting the ultrasonic image of each mode into a neural network;
the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.
Optionally, the performing feature fusion on the image features of each mode includes:
and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.
Optionally, the method further comprises an image preprocessing performed before the extracting of the image features of the ultrasound image of each mode, so that the ultrasound image of each modality is suitable as an input of a neural network after being preprocessed.
Optionally, the image preprocessing comprises:
and automatically positioning the scanning frames in the elastic E mode image and the Doppler blood flow F mode image, and selecting a key analysis area by using a minimum rectangular frame according to the positioning.
Optionally, the image preprocessing further comprises:
and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.
In a second aspect of the present invention, there is provided an endoscopic ultrasound image processing system including:
the image acquisition module is used for acquiring ultrasonic images of three modes in the CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;
and the image processing module is used for classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes.
Optionally, the image processing module includes:
the characteristic extraction module is used for extracting the image characteristics of the ultrasonic image in each mode;
the characteristic fusion module is used for carrying out characteristic fusion on the image characteristics of each mode to obtain the image fusion characteristics of the three modes;
and the classifier is used for classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.
In a third aspect of the present invention, there is provided a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the endoscopic ultrasound image processing method.
In a fourth aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program characterized in that the program is executed by a processor for executing the processing method of endoscopic ultrasound images.
Compared with the prior art, the embodiment of the invention has at least one of the following advantages:
according to the processing method and system for the endoscope ultrasonic image, the image data of three modes in the CP-EBUS imaging technology are used, the images are processed through the multi-modal neural network, the lymph node of the ultrasonic image can be identified more accurately, and therefore the accuracy of lymph node diagnosis is improved.
The processing method and the system of the endoscope ultrasonic image can be used for identifying the benign and malignant lymph nodes, can process multi-mode ultrasonic images, namely three modes of gray scale, blood flow and elasticity, and integrates different mode information to assist in improving the diagnosis rate.
Compared with the prior art, the processing method and the processing system for the endoscope ultrasonic image are easy to popularize, can be quickly and efficiently arranged in a place with CP-EBUS equipment, and do not need manual intervention.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flowchart of a method for processing endoscopic ultrasound images according to an embodiment of the present invention;
FIG. 2 is a flow chart of the processing of a multi-modal neural network in a preferred embodiment of the present invention;
FIG. 3 is a flowchart of a process for extracting image features in a preferred embodiment of the present invention;
FIG. 4 is a flowchart of the process of image quality identification in a preferred embodiment of the present invention;
FIG. 5 is a block diagram of a system for processing endoscopic ultrasound images in accordance with an embodiment of the present invention;
FIG. 6 is a block diagram of a system for processing endoscopic ultrasound images in accordance with a preferred embodiment of the present invention.
Detailed Description
The present invention will be described in detail below with reference to specific embodiments and the accompanying drawings. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the concept of the invention. All falling within the scope of the present invention.
Fig. 1 is a flowchart of a processing method of endoscopic ultrasound images according to an embodiment of the present invention. Referring to fig. 1, the method for processing an endoscopic ultrasound image in the present embodiment includes:
s100, acquiring ultrasonic images of three modes in a CP-EBUS imaging technology;
and S200, classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes.
In the above CP-EBUS imaging technique, the three modes of the ultrasound image are an elastic mode E, a gray-scale mode B, and a doppler blood flow mode F. By adopting the multi-mode image based on the convex array probe ultrasound bronchoscope (CP-EBUS), the CP-EBUS image of the target lymph node is processed by using the multi-mode neural network, manual intervention is not needed, the lymph node of the ultrasound image can be more accurately and automatically identified, and the accuracy of lymph node diagnosis is improved.
FIG. 2 is a flow chart of the processing of the multi-modal neural network in a preferred embodiment of the present invention. Referring to fig. 2, in the preferred embodiment, the classifying the three-mode ultrasound images by using the multi-mode neural network specifically includes:
s201, extracting image characteristics of the ultrasonic image of each mode;
s202, performing feature fusion on the image features of each mode to obtain image fusion features of three modes;
and S203, classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.
According to the preferred embodiment, the image characteristics of the ultrasonic images in the three modes are extracted and fused and are used for classification, so that the final identification of the lymph nodes of the ultrasonic images, including the identification of the malignancy and the benign of the lymph nodes, can be improved, the diagnosis accuracy can be improved according to the processing result of the CP-EBUS images, and a doctor can be assisted in puncturing the lymph nodes.
FIG. 3 is a flowchart of a preferred embodiment of the present invention for extracting image features. In order to better extract the image features of the ultrasound image of each mode, as shown in fig. 3, in a preferred embodiment, the method may include:
s2011, inputting the ultrasonic image of each mode into a neural network; the neural network adopts a convolution neural network;
and S2012, the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.
Preferably, in a specific embodiment, the image feature of the ultrasound image of each mode is extracted according to the following detailed operations:
and (3) convolution operation: the input image signal x of the ultrasonic image of each mode is divided into three branches, wherein one branch is subjected to dimension reduction convolution of 1*1, the dimension reduction convolution result is respectively subjected to dimension increase convolution of 1*1 and dimension increase convolution of 3*3, the output results of the two dimension increase convolutions are respectively added to the other two branches of the input signal, and then the two branches are connected together. Expressed mathematically as:
wherein c is 1 ,c 2 ,c 3 Respectively 1*1, 1*1 and 3*3,
a join operation in the channel dimension.
An attention mechanism is as follows: the features extracted by the convolution operation are weighted by channel. Specifically, the input image signal x is divided into two branches, one branch of the input image signal x is subjected to adaptive average pooling in spatial dimension, then the input image signal x is subjected to dimension reduction convolution operation of 1*1, then the input image signal x is subjected to linear rectification function (ReLU) r, then the input image signal x is subjected to dimension ascending convolution operation of 1*1, then the weighting of each channel is obtained through softmax operation sigma, then the weighting value is subjected to channel dot multiplication with the other branch of the input image signal x, and the result of the dot multiplication is output. Expressed mathematically as: x is the number of out ＝σ(r(p(x)*c 1 )*c 2 ) X, wherein
Where p is a spatial adaptive pooling function,
"·" is a profile weighting operation along the channel dimension.
In the above preferred embodiment, in order to implement feature fusion on the image features of each mode, it may be: and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.
The feature fusion is realized by adopting a plurality of feature fusion modules, and the fusion operation of each feature fusion module is divided into four paths. Feature extraction output for three mode images
The received original signals are directly output by three paths of the signals respectively. And the fourth path has slightly different output according to the relative position of the characteristic fusion module.
For the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths, and the mathematical form of the fourth path is as follows:
wherein w 1 ,w 2 ,w 3 Are the weight parameters that can be learned and,
a join operation in the channel dimension.
For fusing subsequent feature fusion modules, e.g. the l-th feature fusion module (l)>1) Firstly, the three characteristics are weighted and connected, and then the inter-layer characteristics are carried out on the fusion characteristics of the previous layer and the current fusion characteristicsFusion, which is the mathematical form of integration:
wherein w 4 ,w 5 As are learnable parameters. In the preferred embodiment, the four-way fusion features are output for subsequent classification based on the multi-modal images.
In addition, in order to ensure that the image input by the neural network meets the requirement, the image can be preprocessed before being input. That is, the image feature of the ultrasound image of each mode is extracted before, and the ultrasound image of each mode is adapted as an input of the neural network by preprocessing.
FIG. 4 is a flow chart of the process of image quality identification in a preferred embodiment of the present invention. Referring to fig. 4, the preprocessing may specifically include scan frame positioning, where the scan frame positioning is to automatically position the scan frames in the elastic E-mode and doppler flow F-mode images, and select the key analysis area using the minimum rectangular frame according to the positioning. In order to better reduce the data processing work of the subsequent neural network, on the basis of the preprocessing, the method can further comprise the following steps of identifying the image quality: and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.
For the CP-EBUS image (elastic mode E, doppler blood flow mode F) in the above embodiment, the specific operations of positioning the preprocessed scan frames may be: giving an image of elastic E-mode or Doppler flow F-mode
And obtaining a mask matrix m set according to the corresponding modal imaging rule
By removing extraneous information in this step, a preprocessed intermediate signal is obtained
Then, the minimum rectangle containing the detection frame is determined through a frame detection operator G,
wherein
Is the original signal contained by the smallest rectangle. Of course, the operation of the preferred embodiment is not limited thereto, and other techniques may be used to achieve the above-described effects.
For the CP-EBUS image (elastic mode E, gray scale mode B, doppler blood flow mode F) in the above embodiment, the specific operations of the pre-processing image quality identification may be:
(1) For the image of the elastic E mode, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold value T e The image quality is considered to be unqualified, otherwise, the average intensity of the image is calculated, and if the intensity is larger than the threshold value T I If the image quality is qualified, otherwise, the image quality is unqualified.
(2) For Doppler blood flow F-mode images, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold value T f The image quality is considered to be qualified, otherwise, the image quality is not qualified.
(3) For a grayscale B-mode image, the quality is qualified by default.
In the above embodiments, the classifier performs global pooling on the features of the four branches output by the feature fusion module, and then multiplies the feature by the classification matrix to give a final classification result.
FIG. 5 is a block diagram of a system for processing endoscopic ultrasound images in accordance with an embodiment of the present invention.
Referring to fig. 5, the system for processing an endoscopic ultrasound image in the present embodiment includes: the device comprises an image acquisition module and an image processing module, wherein the image acquisition module acquires ultrasonic images in three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F; the image processing module adopts a multi-modal neural network to classify the ultrasonic images in the three modes to obtain an image processing result for identifying the lymph nodes.
In the above embodiment, the image processing module may further include: the system comprises a feature extraction module, a feature fusion module and a classifier, wherein the feature extraction module extracts image features of the ultrasonic image of each mode; the feature fusion module performs feature fusion on the image features of each mode to obtain image fusion features of the three modes; and classifying the image fusion characteristics by a classifier to obtain a processing result for identifying the lymph nodes.
FIG. 6 is a block diagram of a system for processing endoscopic ultrasound images in accordance with a preferred embodiment of the present invention.
Referring to fig. 6, in the preferred embodiment, there is provided an endoscopic ultrasound image processing system including:
the data preprocessing module is used for preprocessing the data according to the characteristics of each modal data to enable the data to be suitable for being used as the input of a neural network;
a feature extraction module which performs feature extraction on an input image signal by using convolution operation, attention mechanism, cross-layer connection operation and the like, and applies the extracted features to a subsequent feature fusion module;
the characteristic fusion module fuses the characteristics of different modes by utilizing learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image;
and a classifier for classifying the features output by the feature fusion module and outputting a result of the predicted diagnosis.
Specifically, the data preprocessing module is composed of a scanning frame positioning module and an image quality judging module, wherein:
a scanning frame positioning module, which automatically positions the scanning frames in the elastic E mode image and the Doppler blood flow F mode image and selects a key analysis area by using a minimum rectangular frame according to the positioning;
and the image quality judging module evaluates the quality of the image according to the characteristics of the mode, and sends out a prompt aiming at the image with unqualified quality so as to avoid useless analysis.
Furthermore, in the scan frame positioning module, an image of an elastic E mode or a Doppler blood flow F mode is given
And obtaining a mask matrix m set according to the corresponding modal imaging rule
By removing extraneous information in this step, a preprocessed intermediate signal is obtained
Then, the minimum rectangle containing the detection frame is determined through a frame detection operator G,
wherein
Is the original signal contained by the smallest rectangle.
Further, in the image quality identification module, for the image of the elastic E mode, the variance of three channel values at each position is firstly calculated, and if the variance at a certain position is larger than the threshold value T c Judging that the pixel point is a colored pixel point, and if the proportion of the colored pixel point is less than the threshold value T e The image quality is considered to be unqualified, otherwise, the average intensity of the image is calculated, and if the intensity is larger than the threshold value T I If the image quality is qualified, otherwise, the image quality is unqualified. For Doppler blood flow F-mode images, judging colored pixel points according to the variance of three channel values at each position, and if the proportion of the colored pixel points is smaller than a threshold value T f The image quality is considered to be qualified, otherwise, the image quality is not qualifiedAnd (5) passing the test result. For the grayscale B mode image, the quality is qualified by default.
Further, the feature extraction module comprises: a convolution module and a channel attention module.
The convolution module is used for extracting the characteristics of the image signal. Specifically, in the module: the input signal x is divided into three branches, one branch is input into a dimensionality reduction convolution module 1*1, the output results of the module are respectively input into a dimensionality increase convolution module 1*1 and a dimensionality increase convolution module 3*3, and the output results of the two dimensionality increase convolution modules are respectively added to the other two branches of the input signal and then connected together. Expressed mathematically as:
wherein c is 1 ,c 2 ,c 3 Respectively 1*1, 1*1 and 3*3,
a join operation in the channel dimension.
The channel attention model is used to weight the features extracted by the convolution module by channel. Specifically, in the module, an input signal x is divided into two branches, wherein one branch is subjected to adaptive average pooling in a spatial dimension, then the adaptive average pooling is performed through a dimensionality reduction convolution operation of 1*1, then a linear rectification function (ReLU) r is performed, then an dimensionality increasing convolution operation of 1*1 is performed, then the weighting of each channel is obtained through a softmax operation sigma, and then the weighting value is dot-multiplied with the other branch of the input signal according to the channels to output a dot-multiplied result. Expressed mathematically as: x is the number of out ＝σ(r(p(x)*c 1 )*c 2 ) X, wherein
Wherein p isA spatially adaptive pooling function is used to generate a spatial adaptive pooling function,
"·" is a feature map weighting operation along the channel dimension.
Further, there are a plurality of feature fusion modules. The fusion operation of each module is divided into four ways, and the output of the feature extraction module for three mode images
Three of the three paths are used to directly output the received original signals. And the fourth path has slightly different output according to the relative position of the characteristic fusion module.
For the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths, and the mathematical form of the fourth path is as follows:
wherein w 1 ,w 2 ,w 3 Are the weight parameters that can be learned and,
a join operation in the channel dimension.
For fusing subsequent feature fusion modules, e.g. the l-th feature fusion module (l)>1) Firstly, the three features are weighted and connected, then the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion, and the comprehensive mathematical form is as follows:
wherein w 4 ,w 5 As are learnable parameters.
The number of the feature extraction modules and the number of the feature fusion modules can be adjusted by specific scenes.
Further, the classifier respectively performs global pooling on the four paths of features output by the feature fusion module and then multiplies the four paths of features by a classification matrix to give a final classification result.
The processing system for endoscopic ultrasound images in the above embodiment of the present invention uses the image data of three modes in the CP-EBUS imaging technology, and processes these images through the multi-modal neural network, so as to identify the lymph nodes of the ultrasound images more accurately, thereby facilitating the improvement of the accuracy of lymph node diagnosis.
In another embodiment of the present invention, there is also provided a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the method for processing an endoscopic ultrasound image in any one of the above embodiments.
In another embodiment of the present invention, there is also provided a computer-readable storage medium having stored thereon a computer program for executing the processing method of endoscopic ultrasound images in any one of the above embodiments when the program is executed by a processor.
To verify the effectiveness of the above-described embodiments of the present invention, experiments were performed on authentic CP-EBUS datasets and performance comparisons were performed with existing methods used by human experts. In the experiment, the model was cross-validated five-fold on 2205 images collected from 245 lymph nodes, and the model selected from the five validations was tested on 441 images collected from 49 additional lymph nodes, and the results were recorded as the mean and confidence interval of the five model tests. And comparing the expert groups on the same test set, testing the test sets by the three experts respectively, and recording the average value and the confidence interval of the test results of the three experts as results.
The results of the experiment are shown in table 1:
as can be seen from table 1, the lymph node identification result obtained by the above embodiment of the present invention has significantly improved accuracy compared to the result of the existing method implemented by the present human experts, and the superiority and effectiveness of the present invention are well verified. In addition, consistency analysis is performed on the lymph node identification result of the embodiment of the invention and the expert group result, the kappa value of the result of the embodiment of the invention is 0.7605, and the kappa value of the human expert group as comparison is 0.5800, which shows that the lymph node identification result of the embodiment of the invention has better diagnosis consistency.
It should be noted that, the steps in the method provided by the present invention may be implemented by using corresponding modules, devices, units, and the like in the system, and those skilled in the art may refer to the technical solution of the system to implement the step flow of the method, that is, the embodiment in the system may be understood as a preferred example for implementing the method, and details are not described herein.
Those skilled in the art will appreciate that, in addition to implementing the system and its various means provided by the present invention in purely computer readable program code means, the system and its various means provided by the present invention can be implemented with the same functionality in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like by entirely programming the method steps logically. Therefore, the system and various devices thereof provided by the present invention can be regarded as a hardware component, and the devices included in the system and various devices thereof for realizing various functions can also be regarded as structures in the hardware component; means for performing the functions may also be regarded as structures within both software modules and hardware components for performing the methods.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The above-described preferred features may be used in any combination without conflict with each other.
Claims (12)
1. An endoscopic ultrasound image processing method, comprising:
acquiring ultrasonic images of three modes in a CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;
classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes;
the classifying the ultrasonic images in three modes by adopting the multi-mode neural network comprises the following steps:
extracting the image characteristics of the ultrasonic image of each mode;
performing feature fusion on the image features of each mode to obtain image fusion features of the three modes;
classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes;
the feature fusion is realized by adopting a plurality of feature fusion modules, and the fusion operation of each feature fusion module is divided into four paths;
for the extracted image characteristics of the three mode ultrasonic images, directly outputting the received original signals by using three paths respectively, and slightly outputting the original signals by using a fourth path according to the relative position of the characteristic fusion module;
for the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths; for the subsequent fusion operation of the feature fusion module, firstly, the three features are subjected to weighted connection, and then, the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion.
2. The method for processing endoscopic ultrasound images as defined in claim 1, wherein said extracting image features of ultrasound images of each mode comprises:
inputting the ultrasonic image of each mode into a neural network;
the neural network performs feature extraction on the ultrasonic image of each mode by using convolution operation, attention mechanism and cross-layer connection operation to obtain corresponding image features.
3. The method for processing endoscopic ultrasound images according to claim 1, wherein said feature fusion of image features of each mode comprises:
and fusing the image features of different modes by using learnable weighted connection operation to obtain a comprehensive expression of the multi-mode image, namely the image fusion features of three modes.
4. The method for processing endoscopic ultrasound images as claimed in claim 1, further comprising an image preprocessing performed before said extracting image features of ultrasound images of each mode, so that the preprocessed ultrasound images of each modality are suitable as inputs of a neural network.
5. The method for processing endoscopic ultrasound images according to claim 4, wherein the image preprocessing comprises:
and automatically positioning the scanning frames in the elastic E mode image and the Doppler blood flow F mode image, and selecting a key analysis area by using a minimum rectangular frame according to the positioning.
6. The method for processing endoscopic ultrasound images according to claim 5, wherein said automatically positioning the scan frames in the elastic E-mode and Doppler blood flow F-mode images, and selecting the key analysis area using the minimum rectangular frame according to the positioning comprises:
giving an image of elastic E-mode or Doppler flow F-mode
And obtaining a mask matrix m set according to the corresponding modal imaging rule
By removing the irrelevant information in this step, a pretreatment is obtainedIntermediate signal of theory
Then, the minimum rectangle containing the detection frame is determined through a frame detection operator G,
wherein
Is the original signal contained by the smallest rectangle.
7. The method for processing endoscopic ultrasound images according to claim 4, wherein the image preprocessing comprises:
and evaluating the quality of the image according to the characteristics of each mode, and sending out a prompt aiming at the image with unqualified quality to avoid useless analysis.
8. The method for processing endoscopic ultrasound images according to claim 7, wherein said evaluating the quality of the images according to the characteristics of each mode comprises:
-for the elastic E-mode image, determining colored pixels according to the variance of the three channel values at each position, if the proportion of colored pixels is less than a threshold T e Considering the image quality as unqualified, otherwise, continuously calculating the average intensity of the image, and if the intensity is greater than the threshold value T I If the image quality is qualified, otherwise, the image quality is unqualified;
for images of the Doppler flow F-mode, determining colored pixels from the variance of the three channel values at each location, if the proportion of colored pixels is less than a threshold T f The image quality is considered to be qualified, otherwise, the image quality is not qualified;
for images in grayscale B mode, the quality is qualified by default.
9. An endoscopic ultrasound image processing system, comprising:
the image acquisition module is used for acquiring ultrasonic images of three modes in the CP-EBUS imaging technology, wherein the three modes are an elastic mode E, a gray scale mode B and a Doppler blood flow mode F;
the image processing module is used for classifying the ultrasonic images in the three modes by adopting a multi-mode neural network to obtain a processing result for identifying the lymph nodes;
the image processing module adopts a multi-modal neural network to classify the ultrasonic images in three modes, and comprises the following steps:
extracting the image characteristics of the ultrasonic image of each mode;
performing feature fusion on the image features of each mode to obtain image fusion features of the three modes;
classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes;
the feature fusion is realized by adopting a plurality of feature fusion modules, and the fusion operation of each feature fusion module is divided into four paths;
for the extracted image characteristics of the three mode ultrasonic images, directly outputting the received original signals by using three paths respectively, and slightly outputting the original signals by using a fourth path according to the relative position of the characteristic fusion module;
for the fusion operation of the first feature fusion module, the fourth path performs weighted splicing on the three features of the other three paths; for the subsequent fusion operation of the feature fusion module, firstly, the three features are subjected to weighted connection, and then, the fusion feature of the previous layer and the current fusion feature are subjected to cross-layer feature fusion.
10. The endoscopic ultrasound image processing system according to claim 9, wherein said image processing module comprises:
the characteristic extraction module is used for extracting the image characteristics of the ultrasonic image of each mode;
the characteristic fusion module is used for carrying out characteristic fusion on the image characteristics of each mode to obtain the image fusion characteristics of the three modes;
and the classifier is used for classifying the image fusion characteristics to obtain a processing result for identifying the lymph nodes.
11. A terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor when executing the program being adapted to perform the method of processing endoscopic ultrasound images as claimed in any one of claims 1 to 8.
12. A computer-readable storage medium on which a computer program is stored, characterized in that the program, when executed by a processor, is for executing the method of processing endoscopic ultrasound images as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011585174.0A CN112741651B (en) | 2020-12-25 | 2020-12-25 | Method and system for processing ultrasonic image of endoscope |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011585174.0A CN112741651B (en) | 2020-12-25 | 2020-12-25 | Method and system for processing ultrasonic image of endoscope |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112741651A CN112741651A (en) | 2021-05-04 |
| CN112741651B true CN112741651B (en) | 2022-11-25 |
Family
ID=75646448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011585174.0A Active CN112741651B (en) | 2020-12-25 | 2020-12-25 | Method and system for processing ultrasonic image of endoscope |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112741651B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113255781B (en) * | 2021-05-28 | 2022-04-01 | 上海市胸科医院 | Representative picture selecting method and device for CP-EBUS and diagnosis system |
| CN113496489B (en) * | 2021-09-06 | 2021-12-24 | 北京字节跳动网络技术有限公司 | Training method of endoscope image classification model, image classification method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109544517A (en) * | 2018-11-06 | 2019-03-29 | 中山大学附属第医院 | Method and system are analysed in multi-modal ultrasound group credit based on deep learning |
| CN110428410A (en) * | 2019-07-31 | 2019-11-08 | 腾讯医疗健康(深圳)有限公司 | Eyeground medical image processing method, device, equipment and storage medium |
| CN110930367A (en) * | 2019-10-31 | 2020-03-27 | 上海交通大学 | Multi-modal ultrasound image classification method and breast cancer diagnosis device |
| CN111096767A (en) * | 2020-01-08 | 2020-05-05 | 南京市第一医院 | Deep learning-based mediastinal lymph node ultrasound elastic image segmentation and classification method |
| CN111432730A (en) * | 2018-10-24 | 2020-07-17 | 中国医学科学院北京协和医院 | Imaging method and imaging system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10441250B2 (en) * | 2015-10-08 | 2019-10-15 | Zmk Medical Technologies Inc. | 3D multi-parametric ultrasound imaging |
| WO2017092615A1 (en) * | 2015-11-30 | 2017-06-08 | 上海联影医疗科技有限公司 | Computer aided diagnosis system and method |
| US11364013B2 (en) * | 2017-01-05 | 2022-06-21 | Koninklijke Philips N.V. | Ultrasound imaging system with a neural network for image formation and tissue characterization |
| US11026634B2 (en) * | 2017-04-05 | 2021-06-08 | doc.ai incorporated | Image-based system and method for predicting physiological parameters |
-
2020
- 2020-12-25 CN CN202011585174.0A patent/CN112741651B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111432730A (en) * | 2018-10-24 | 2020-07-17 | 中国医学科学院北京协和医院 | Imaging method and imaging system |
| CN109544517A (en) * | 2018-11-06 | 2019-03-29 | 中山大学附属第医院 | Method and system are analysed in multi-modal ultrasound group credit based on deep learning |
| CN110428410A (en) * | 2019-07-31 | 2019-11-08 | 腾讯医疗健康(深圳)有限公司 | Eyeground medical image processing method, device, equipment and storage medium |
| CN110930367A (en) * | 2019-10-31 | 2020-03-27 | 上海交通大学 | Multi-modal ultrasound image classification method and breast cancer diagnosis device |
| CN111096767A (en) * | 2020-01-08 | 2020-05-05 | 南京市第一医院 | Deep learning-based mediastinal lymph node ultrasound elastic image segmentation and classification method |
Non-Patent Citations (1)
| Title |
|---|
| 基于超声的影像组学在乳腺癌诊断中的研究进展;崔文举等;《肿瘤影像学》;20200828;第29卷(第4期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112741651A (en) | 2021-05-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109886273B (en) | CMR image segmentation and classification system | |
| CN110060774B (en) | Thyroid nodule identification method based on generative confrontation network | |
| Carrer et al. | Automatic pleural line extraction and COVID-19 scoring from lung ultrasound data | |
| KR102125127B1 (en) | Method of brain disorder diagnosis via deep learning | |
| CN110930367B (en) | Multi-modal ultrasound image classification method and breast cancer diagnosis device | |
| CN109543719B (en) | Cervical atypical lesion diagnosis model and device based on multi-modal attention model | |
| CN111028206A (en) | Prostate cancer automatic detection and classification system based on deep learning | |
| CN107133638B (en) | Multi-parameter MRI prostate cancer CAD method and system based on two classifiers | |
| CN112741651B (en) | Method and system for processing ultrasonic image of endoscope | |
| Bourbakis | Detecting abnormal patterns in WCE images | |
| Zhao et al. | Attention residual convolution neural network based on U-net (AttentionResU-Net) for retina vessel segmentation | |
| CN116758336A (en) | Medical image intelligent analysis system based on artificial intelligence | |
| Gopalakrishnan et al. | Itl-cnn: Integrated transfer learning-based convolution neural network for ultrasound pcos image classification | |
| Nurmaini et al. | An improved semantic segmentation with region proposal network for cardiac defect interpretation | |
| CN117218127B (en) | Ultrasonic endoscope auxiliary monitoring system and method | |
| CN114332910A (en) | Human body part segmentation method for similar feature calculation of far infrared image | |
| CN113506274A (en) | Detection system for human cognitive condition based on visual saliency difference map | |
| CN111640127A (en) | Accurate clinical diagnosis navigation method for orthopedics department | |
| CN115089112B (en) | Post-stroke cognitive impairment risk assessment model building method and device and electronic equipment | |
| Delmoral et al. | Segmentation of pathological liver tissue with dilated fully convolutional networks: A preliminary study | |
| Pradhan | A Novel Threshold based Method for Vessel Intensity Detection and Extraction from Retinal Images | |
| CN112242193B (en) | Automatic blood vessel puncture method based on deep learning | |
| KR102384083B1 (en) | Apparatus and Method for Diagnosing Sacroiliac Arthritis and Evaluating the Degree of Inflammation using Magnetic Resonance Imaging | |
| CN115170492A (en) | Intelligent prediction and evaluation system for postoperative vision of cataract patient based on AI (artificial intelligence) technology | |
| KR102360615B1 (en) | Medical image diagnosis assistance apparatus and method using a plurality of medical image diagnosis algorithm for endoscope images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |