CN108319977B - Cervical biopsy region identification method and device based on channel information multi-mode network - Google Patents
Cervical biopsy region identification method and device based on channel information multi-mode network Download PDFInfo
- Publication number
- CN108319977B CN108319977B CN201810092566.XA CN201810092566A CN108319977B CN 108319977 B CN108319977 B CN 108319977B CN 201810092566 A CN201810092566 A CN 201810092566A CN 108319977 B CN108319977 B CN 108319977B
- Authority
- CN
- China
- Prior art keywords
- image
- cervical
- network
- cervix
- acetic acid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/032—Recognition of patterns in medical or anatomical images of protuberances, polyps nodules, etc.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a cervical biopsy area identification method and a device based on a channel information multi-mode network, wherein the device comprises the following steps: the image acquisition unit is used for acquiring a physiological saline image, an acetic acid image and an iodine image of the cervix; the data processing unit comprises a trained cervical biopsy region identification model, analyzes and processes the physiological saline image, the acetic acid image and the iodine image, and outputs a probability label of the cervical biopsy region; the cervical biopsy region identification model comprises: the detection network layer is used for extracting characteristic graphs and position information of cervical surface areas in the physiological saline image, the acetic acid image and the iodine image; combining the characteristics with a prediction network layer, splicing the 3 characteristic graphs and the position information on the channel dimension, and outputting a probability label of a biopsy area of the cervix through characteristic fusion and identification; and the display unit acquires and displays the probability label. The cervical biopsy region identification device can assist a doctor in accurately judging whether the cervix of a patient needs to be subjected to further biopsy.
Description
Technical Field
The invention relates to the field of medical image processing, in particular to a cervical biopsy region identification method and device based on a channel information multi-modal network.
Background
Cervical cancer is the most common gynecological malignancy. The age of the primary cancer is 30-35 years old, the invasive cancer is 45-55 years old, and the incidence of the primary cancer tends to be younger in recent years.
The examination steps of cervical lesions mainly comprise three steps: (1) cervical cytology, most commonly pap smear; (2) colposcopy, if the cytological result is abnormal, the colposcopy is required to be carried out, and the changes of cervical epithelial color, blood vessels and the like are observed; (3) and (4) carrying out cervical tissue biopsy, wherein if the colposcopy is questioned, a doctor can take a few cervical tissues to carry out biopsy on a suspicious lesion under the positioning of the colposcope, and the biopsy result also becomes the final conclusion of the cervical lesion.
The colposcopy comprises the specific steps of directly exposing the cervix, sequentially smearing 0.9% physiological saline, 3% -5% acetic acid solution and compound iodine solution on the surface of the cervix, carefully observing whether abnormal regions (regions needing biopsy) exist in the cervical squame junction and the columnar epithelial region through shot cervical images, and guiding and selecting accurate parts for biopsy according to the information, so that blind biopsy is avoided, and the biopsy positive rate and the diagnosis accuracy are improved.
The colposcopy is a detection method based on subjective experience, and the judgment of abnormal regions and abnormal ranges needs to depend on experience accumulation and intuitive judgment capability of doctors, and whether the judgment is accurate directly relates to the positive rate and the diagnosis accuracy rate of biopsy. With the development of medical informatization and big data, a large number of colposcopy results are accumulated and stored in the form of image data. At present, many machine learning and deep learning methods are applied to colposcopic detection images, including cervical orifice detection, vinegar area detection, cervical lesion degree prediction, and the like, and these methods can play a certain auxiliary role, but cannot fundamentally assist doctors in making more accurate judgments. Moreover, most of the methods only use the colposcopic cervical images acted by 3% -5% acetic acid solution, which is inconsistent with the common medical means that doctors judge whether there is a biopsy area through the characteristic change of the combination of the images of 0.9% physiological saline, 3% -5% acetic acid solution, compound iodine solution and the like. Therefore, how to reasonably utilize medical images and medical experiences to design an auxiliary device for cervical biopsy region identification which combines the above problems and fundamentally assists doctors to make more accurate judgment is a problem to be solved.
Disclosure of Invention
The invention provides a cervical biopsy area recognition device based on a channel information multi-modal network, which is used for collecting a physiological saline image, an acetic acid image and an iodine image of the cervix of a patient, respectively extracting the characteristics of the two images, fusing the two images and outputting a probability label that the cervix has a biopsy area, thereby fundamentally assisting a doctor to make more accurate judgment on whether the cervix of the patient needs biopsy.
The invention provides the following technical scheme:
a cervical biopsy region identification apparatus based on a channel information multi-modal network, comprising:
the image acquisition unit acquires a physiological saline image, an acetic acid image and an iodine image of the cervix and sends the images to the data processing unit;
the data processing unit comprises a trained cervical biopsy region identification model, the cervical biopsy region identification model analyzes and processes a saline physiological image, an acetic acid image and an iodine image, and outputs a probability label of the existence of a biopsy region of the cervix;
the cervical biopsy region identification model comprises:
the detection network layer comprises 3 independent detection sub-networks which are respectively used for extracting a characteristic diagram and position information of a cervical surface area in a physiological saline image, an acetic acid image and an iodine image;
combining the characteristics with a prediction network layer, splicing 3 characteristic graphs and position information extracted by the detection network layer on a channel dimension, and outputting a probability label of a biopsy area of the cervix through characteristic fusion and identification;
and the display unit acquires and displays the probability label.
The cervical biopsy area identification device provided by the invention acquires a physiological saline image, an acetic acid image and an iodine image of the cervix through the image acquisition unit, extracts the position information and the characteristic diagram of the cervical surface area in each image through the detection network layer of the cervical biopsy area identification model, combines and identifies the position information and the characteristic diagram of 3 cervical surface areas through the characteristic combination prediction network layer to obtain a probability label of the existence of the biopsy area of the cervix, and can assist a doctor in accurately judging whether the biopsy area exists in the cervix of a patient through the display unit for display processing.
In the cervical biopsy area identification device, a detection network is adopted to extract a characteristic diagram of a cervical surface area, and then the characteristic diagram of the cervical surface area is identified, so that the identification is more targeted, and the output identification result is more accurate.
The normal saline image of the cervix is the cervix image coated with the normal saline, the acetic acid image is the cervix image coated with the normal saline and 3% -5% acetic acid solution in sequence, and the iodine image is the cervix image coated with the normal saline, 3% -5% acetic acid solution and compound iodine solution in sequence.
If the cervix has a region needing biopsy, abnormal blood vessels are easier to observe under the action of physiological saline; under the action of 3-5% acetic acid solution, the product can present the characteristics of thick vinegar white and mosaic blood vessel; under the action of the compound iodine solution, the characteristics of bright orange, yellow mustard, spot coloring and the like can be presented, but the existence of the characteristics can not determine that the cervix can have pathological changes, and a doctor needs to carry out further biopsy.
The detection subnetwork comprises a feature extraction network, a cervical surface nomination network and a cervical surface detection network which are sequentially connected, and a channel selection module is added to the feature extraction network.
The feature extraction network is used for extracting a feature map of a physiological saline image, an acetic acid image or an iodine image and screening channels of the feature map.
The feature extraction network is a ResNet50 network model added with a channel selection module; the characteristic extraction network comprises a convolution layer, a maximum pooling layer and a plurality of convolution groups which are sequentially connected; the convolution groups are composed of residual error units; each residual unit comprises a plurality of convolutional layers, and in each residual unit, the characteristic diagram before entering the first convolutional layer also directly flows to the last convolutional layer and is added with the characteristic diagram output by the last convolutional layer to be used as the output of the residual unit.
The channel selection module is added after the last residual error unit of each convolution group.
The channel selection module performs channel selection operation on the input feature map to obtain the weight of each channel of the feature map, multiplies the weight by the feature map, adds the multiplication result to the feature map, and outputs the feature map after channel screening; the weight value range is 0-1.
The purpose of the construction is to perform channel screening once on the feature map to be output of each convolution group, so as to prevent redundant channel features from being excessive. The operation of the channel selection module is equivalent to multiplying the activation value of the input feature map by (1+ weight), wherein the value of the weight is between 0 and 1.
The cervical surface nomination network is an RPN network; consists of 1 convolutional layer with a convolutional filter size of 3 × 3 and a convolution step of 1, and 2 convolutional layers with a convolutional filter size of 1 × 1 and a convolution step of 1 in parallel.
The cervical surface nomination network is used for obtaining the position information of the cervical surface area in the characteristic diagram.
The cervical surface detection network module consists of a ROIPooling layer and two parallel full-connection layers.
And the cervical surface detection network performs Crop operation on the characteristic diagram according to the position information of the cervical surface area to obtain the characteristic diagram and the position information of the cervical surface area.
Preferably, the cervical surface detection network is followed by the channel selection module.
And the channel selection module is used for respectively screening the channels of the feature maps of the cervical surface regions output by the 3 cervical surface detection networks, and the feature maps of the cervical surface regions after the channels are screened enter the feature combination prediction network layer.
The training method of the cervical biopsy region identification model comprises the following steps:
(1) acquiring a physiological saline image, an acetic acid image and an iodine image of a cervix, preprocessing the physiological saline image, the acetic acid image and the iodine image, marking a cervix surface area, identifying whether a biopsy area exists in the cervix or not, and constructing a training set;
the pretreatment method comprises the following steps: carrying out z-score standardization processing and data augmentation operation on the physiological saline image, the acetic acid image and the iodine image;
forming a training sample by taking a physiological saline image, an acetic acid image and an iodine image of the same cervix as a group of data, and marking whether a biopsy area exists in the group of images;
specifically, identification and labeling refers to: identifying whether the acetic acid image has characteristics of thick vinegar white and mosaic blood vessel, and marking; and identifying whether the characteristics of bright orange, mustard yellow and spot coloring exist in the iodine image and marking.
Preferably, in the training set, the ratio of the number of samples of the biopsy area to the number of samples of the normal cervix uteri is 0.8-1.2: 1;
(2) training a cervical biopsy region recognition model by using a training set, comprising:
(2-1) pre-training the detection network layer:
respectively inputting the physiological saline image, the acetic acid image and the iodine image in the training set into respective detection sub-networks, training until a loss function is converged, and storing model parameters of the detection sub-networks;
(2-2) training the cervical biopsy region identification model:
loading the model parameters of the detection sub-network obtained in the step (2-1) into a cervical biopsy region identification model;
respectively inputting the physiological saline image, the acetic acid image and the iodine image in the training set into respective detection sub-networks, outputting a probability label with a biopsy area after passing through a feature combination prediction network layer, and training until a cervical biopsy area identification model converges;
and saving the trained cervical biopsy region identification model parameters.
The invention also provides a method for identifying the cervical biopsy region by adopting the cervical biopsy region identification device, which comprises the following steps:
(1) collecting a physiological saline image, an acetic acid image and an iodine image of the cervix by an image collecting unit, and inputting the images into a cervix biopsy area identification model in a data processing unit;
(2) and analyzing and processing the physiological saline image, the acetic acid image and the iodine image through the cervical biopsy area identification model, outputting a probability label of the existence of the biopsy area of the cervix, and displaying the probability label on a display unit.
Compared with the prior art, the invention has the beneficial effects that:
the cervical biopsy area identification device disclosed by the invention is based on the medical experience that a doctor judges whether the cervix needs to be further biopsied or not through the cervical image characteristic change after the action of physiological saline, 3% -5% acetic acid solution and compound iodine solution, performs learning modeling according to a large number of cervical images of colposcopy examination, and identifies the cervical biopsy area according to the established model, so that the doctor can be fundamentally assisted to make more accurate judgment on whether the cervix needs to be further biopsied or not.
The cervical biopsy region identification device provided by the invention utilizes the detection idea to perform classification tasks, accurately positions the ROI of the cervical surface by the aid of a detection network, integrates the image characteristics of three stages in colposcopy, removes redundant channel characteristics by the aid of the channel selection module, retains the most effective and outstanding channel characteristics, and provides great help for the accuracy of the final classification result.
Drawings
Fig. 1 is a schematic workflow diagram of a cervical biopsy area identification device of the present invention;
FIG. 2 is a schematic flow chart of pre-training a detector sub-network;
FIG. 3 is a schematic structural diagram of a channel selection module;
fig. 4 is a schematic diagram of a feature-combined prediction network layer structure.
Detailed Description
The invention will be described in further detail below with reference to the drawings and examples, which are intended to facilitate the understanding of the invention without limiting it in any way.
The cervical biopsy region identification device of the present invention includes:
the image acquisition unit acquires a physiological saline image, an acetic acid image and an iodine image of the cervix and sends the images to the data processing unit;
the data processing unit comprises a trained cervical biopsy region identification model, the cervical biopsy region identification model analyzes and processes a saline physiological image, an acetic acid image and an iodine image, and outputs a probability label of the existence of a biopsy region of the cervix;
and the display unit acquires and displays the probability label.
The doctor comprehensively judges whether the cervix of the patient needs to be further biopsied or not according to the probability label output by the cervix biopsy area identification device and by combining the physiological saline image, the acetic acid image and the iodine image of the patient, and further judges whether the cervix has pathological changes or not.
The work flow of the cervical biopsy region identification apparatus of the present invention is shown in fig. 1.
The image acquisition unit is a colposcope. When a doctor clinically inspects a patient by a colposcope, the doctor can sequentially use physiological saline, 3% -5% acetic acid solution and compound iodine solution to smear the cervix, and judge whether a biopsy area exists by observing the change of cervical squamous column junction and columnar intraepithelial features.
Therefore, the features of each stage play a very important role in the final lesion level judgment, and how to extract effective and important feature information in the images of the three stages is important. For example, under the action of 3% -5% acetic acid solution, the characteristics of 'thick vinegar white' and 'mosaic' and the characteristics of 'bright orange yellow' and 'spot-like coloring' are important basis for doctors to judge the cervical biopsy area. After extracting the features from the colposcopy images at each stage, the features at each stage are combined to predict the final cervical biopsy region. Therefore, how to utilize the multi-image features is the key to the accuracy of predicting whether a biopsy region exists in the cervix.
In order to accurately extract effective characteristics of each stage, the cervical biopsy area identification model in the data processing unit preliminarily extracts the characteristics of images of each stage by using a residual convolutional neural network, then extracts the characteristics of a cervical surface area in each image by using a detection method, and then further extracts the characteristics by using the residual convolutional neural network based on the extracted cervical surface area characteristics. By using a detection method to selectively extract image data, namely only paying Attention to the most important cervical surface region characteristics, and selectively ignoring cervical surface surrounding regions, the introduction of an Attention mechanism based on strong supervision can lead the network to focus on the most important region characteristics, thereby leading the network to selectively learn the important region characteristics.
Essentially, the feature extraction network maps 3 × M × M images of three RGB channels to a three-dimensional tensor representation of C × M × M of more channels through a large amount of learning, where C > 3 and M > M, and a large number of channels with redundant feature information must be generated in this process due to the multiplied increase of the number of channels.
The cervical biopsy area identification model respectively maps high-resolution images to multi-channel three-dimensional tensors through three independent feature extraction networks with channel selection modules, and before the combination of three stages of image features, the channel selection module is added to the last residual error unit for generating each type of feature map, a weight is distributed to each channel of the finally generated feature map, and the convolutional neural network realizes the self-selection of a large number of channels by learning the weight, namely, effective feature information is reserved, and redundant or ineffective feature information is inhibited. Finally, the feature maps of the three stages are effectively combined together, and then a final prediction result of whether the cervix has the biopsy region or not is obtained through a classification network.
All the convolution layers described in the text of the present invention refer to convolution layers followed by batch regularization layers and ReLU activation function layers, and all the fully-connected layers refer to fully-connected layers followed by ReLU activation function layers, and will not be described in detail hereinafter.
The construction and training of the cervical biopsy region identification model comprises the following steps:
the method comprises the following steps: construction of training sets
The physiological saline image, the acetic acid image and the iodine image which are collected during the colposcopy are extracted, the physiological saline image, the acetic acid image and the iodine image of the same individual case are used as a group of data to form a training sample, and a doctor marks the cervical surface position information of each group of images and whether a biopsy region exists in the cervix.
Specifically, whether the characteristics of thick vinegar white and mosaic blood vessel exist in the acetic acid image is identified and marked; and identifying whether the characteristics of bright orange, mustard yellow and spot coloring exist in the iodine image and marking.
The ratio of the number of samples of the biopsy area with the existence of the cervix to the number of samples of the normal cervix is 1: 1. all samples are divided into three data sets, a training set (1373), a testing set (394), and a validation set (192).
To make the image data easier to train, we have all image data processed by z-score normalization (zero-mean normalization) with the mean value subtracted and divided by the standard deviation, while to prevent the network from overfitting and enrich the training samples to some extent, we also have data augmentation operations on the images at random before they are input into the network.
Step two: detection of network layer construction and joint training
The detection network layer comprises 3 independent detection sub-networks. As shown in fig. 2, the detection subnetwork includes a feature extraction network, a cervical surface nomination network and a cervical surface detection network, which are connected in sequence, and a channel selection module is added to the feature extraction network.
The feature extraction network is a ResNet50 network model with a channel selection module added. The feature extraction network comprises convolution layers with convolution filter size of 7 multiplied by 7 and convolution step length of 2, a maximum pooling layer with pooling filter size of 3 multiplied by 3 and pooling step length of 2 and 4 convolution groups which are connected in sequence.
The 4 convolution groups are respectively composed of 3 residual error units, 4 residual error units, 6 residual error units and 3 residual error units.
Each residual unit consists of 3 convolutional layers with convolutional filter sizes of 3 × 3, 1 × 1, and 3 × 3, respectively, and with a convolution step size of 1 (with the exception of the convolution step size of the first convolutional layer in the first residual unit of each convolutional group, which is 2). In each residual unit, the feature map before entering the first convolutional layer will also flow directly after the third convolutional layer and be added with the feature map output by the third convolutional layer to be used as the output of the residual unit.
The channel selection module is added after the last residual unit of each convolution group. The structure of the channel selection module is shown in fig. 3, the feature map output by the last residual unit of each convolution group sequentially passes through the global Pooling layer, the 1 × 1 convolution layer, the ReLU activation layer, the 1 × 1 convolution layer and the Sigmoid activation layer to obtain the weight of each channel of the feature map, the weight is multiplied by the feature map, the multiplication result is added to the feature map, and the feature map after channel screening is output.
The purpose of the construction is to perform channel screening on the feature map to be output of each convolution group once, so as to prevent redundant channel features from being excessive; the reason why the channel selection module is adopted by the user is to design a branch circuit to be added with the feature diagram on the main path, so that the activation value on the feature diagram is multiplied by a value (1+ weight) (the weight is between 0 and 1), because the activation value on the feature diagram is multiplied by a weight value every time if the branch circuit is not adopted, although the function of screening channel information can be achieved, the activation value becomes small after the channel selection module passes through a plurality of channel selection modules, and the inference on the final result is influenced.
The cervical surface nomination network mainly comprises 1 convolutional layer with the size of 3 multiplied by 3 and the convolution step length of 1 and 2 convolutional layers with the size of 1 multiplied by 1 and the convolution and step length of 1 which are parallel.
And the cervical surface detection network mainly comprises 1 ROIPooling layer and two parallel full-connection layers.
The cervical surface detection network is followed by a channel selection module for carrying out channel screening on the characteristic diagram of the cervical surface area output by the cervical surface detection network.
The 3 detector sub-networks are trained by respectively using a physiological saline image, an acetic acid image and an iodine image in a training set, and here, we only take the training of the acetic acid image as an example.
Firstly, inputting an acetic acid image into a feature extraction network to obtain a high-dimensional feature map, and then respectively inputting the feature map into a cervical surface nomination network and a ROIPooling layer. In the area nomination network, two parallel convolution layers respectively output position information possibly existing on the cervical surface and the possibility of the cervical surface existing on the position to the ROIPooling layer, and the error obtained by comparing the two prediction information with the real label can optimize the feature extraction network and the cervical surface nomination network.
And the ROIPooling layer performs Crop operation on the feature map output by the feature extraction network according to the position information output by the cervical surface nomination network to obtain the feature map possibly containing the cervical surface and the position information (collectively called ROI), and the ROIPooling layer is noted to be divided into two paths, wherein one path leads to the feature combination prediction network, and the other path continues to lead to a full-connection layer of the cervical surface detection network.
The probability of biopsy areas existing in the ROI and the position offset information between the ROI and the real cervical surface position are respectively obtained after the ROI is led to a full connection layer of the cervical surface detection network, and the cervical surface detection network and the feature extraction network can be optimized by comparing the two output information with real values to obtain errors (note that the probability of biopsy areas existing obtained here is not a final result and is only used for optimizing a detection sub-network).
The ROI may be channel screened by a channel selection module before it leads to the feature-binding prediction network layer.
During training, the training set is used for training the model, and the training completion can be judged after the loss curve and the accuracy curve are stabilized. In the training process, the validation set is used to test the model effect in the training process.
And finishing the training of the detection network layer and storing the model parameters of the detection network layer.
Step three: construction and training of feature-combined predictive network layers
The structure of the feature combination prediction network layer is shown in fig. 4, and includes a feature combination network and a feature fusion network connected in sequence.
The feature binding network performs a stacking operation on the channel-screened ROIs obtained from the 3 detector subnetworks.
The feature fusion network comprises 3 full connection layers and 1 cross entropy activation function layer.
When the characteristics are trained in combination with the prediction network layer, the stored model parameters of the detection network layer are loaded into the cervical biopsy region identification model. And respectively inputting the physiological saline image, the acetic acid image and the iodine image in the training set into respective detection sub-network layers, respectively inputting the output ROI screened by the channels into a feature set feature combination prediction network layer, respectively outputting the final probability of the existence of a biopsy region after feature combination and feature fusion, and comparing the result with a true value to obtain an error for training the feature combination prediction network layer.
And finishing the training of the characteristic combined prediction network layer and storing the model parameters of the detection network layer.
So far, training of the cervical biopsy region identification model is completed.
When a new patient exists, a colposcope is adopted to collect images of physiological saline, 3% -5% acetic acid solution and compound iodine solution of the cervix of the patient respectively, the data processing unit acquires the image information and inputs the image information into a cervical biopsy area identification model, so that a probability label of the existence of a biopsy area of the cervix of the patient can be output and displayed on a display unit, and a doctor judges whether the patient needs further biopsy by combining the physiological saline, the 3% -5% acetic acid solution and the compound iodine solution image of the patient according to the output probability label so as to judge whether the cervix of the patient has lesion.
The above-mentioned embodiments are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only specific embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions, equivalents, etc. made within the scope of the principles of the present invention should be included in the scope of the present invention.
Claims (6)
1. A cervical biopsy region identification device based on a channel information multi-modal network is characterized by comprising:
the image acquisition unit acquires a physiological saline image, an acetic acid image and an iodine image of the cervix and sends the images to the data processing unit;
the data processing unit comprises a trained cervical biopsy region identification model, the cervical biopsy region identification model analyzes and processes a saline physiological image, an acetic acid image and an iodine image, and outputs a probability label of the existence of a biopsy region of the cervix;
the display unit is used for acquiring and displaying the probability label;
the cervical biopsy region identification model comprises:
the detection network layer comprises 3 independent detection sub-networks which are respectively used for extracting a characteristic diagram and position information of a cervical surface area in a physiological saline image, an acetic acid image and an iodine image;
combining the characteristics with a prediction network layer, splicing 3 characteristic graphs and position information extracted by the detection network layer on a channel dimension, and outputting a probability label of a biopsy area of the cervix through characteristic fusion and identification;
the detection sub-network comprises the following components connected in sequence:
the characteristic extraction network is additionally provided with a channel selection module and is used for extracting a characteristic diagram of a physiological saline image, an acetic acid image or an iodine image and screening channels of the characteristic diagram; the characteristic extraction network comprises a convolution layer, a maximum pooling layer and a plurality of convolution groups which are sequentially connected; the convolution group consists of a plurality of residual error units; the channel selection module is added behind the last residual error unit of each convolution group; the channel selection module performs channel selection operation on the input feature map to obtain the weight of each channel of the feature map, multiplies the weight by the feature map, adds the multiplication result to the feature map, and outputs the feature map after channel screening; the weight value range is 0-1;
the cervical surface nomination network is used for acquiring the position information of a cervical surface area in the characteristic diagram;
and the cervical surface detection network performs Crop operation on the characteristic diagram according to the position information of the cervical surface area to obtain the characteristic diagram and the position information of the cervical surface area.
2. The cervical biopsy region identification apparatus based on channel information multi-modal network as claimed in claim 1, wherein the cervical surface detection network module is composed of a ROIPooling layer and two parallel fully connected layers.
3. The cervical biopsy region identification apparatus based on channel information multi-modal network as claimed in claim 2, wherein the cervical surface detection network is followed by the channel selection module.
4. The cervical biopsy region recognition device based on the multi-modal network of channel information according to any one of claims 1 to 3, wherein the training method of the cervical biopsy region recognition model comprises:
(1) acquiring a physiological saline image, an acetic acid image and an iodine image of a cervix, preprocessing the physiological saline image, the acetic acid image and the iodine image, marking a cervix surface area, identifying whether a biopsy area exists in the cervix or not, and constructing a training set;
(2) training a cervical biopsy region recognition model by using a training set, comprising:
(2-1) training a detection network layer:
respectively inputting the physiological saline image, the acetic acid image and the iodine image in the training set into respective detection sub-networks, training until the detection sub-networks converge, and storing model parameters of the detection sub-networks;
(2-2) training a cervical lesion recognition model:
loading the model parameters of the detection sub-network obtained in the step (2-1) into a cervical biopsy region identification model;
respectively inputting the physiological saline image, the acetic acid image and the iodine image in the training set into respective detection sub-networks, outputting a probability label with a biopsy area after passing through a feature combination prediction network layer, and training until a cervical biopsy area identification model converges;
and saving the trained cervical biopsy region identification model parameters.
5. The cervical biopsy region identification apparatus based on the multi-modal network of channel information as claimed in claim 4, wherein in step (1), the preprocessing method is: the z-score normalization process and data augmentation procedure were performed on the saline, acetic and iodine images.
6. A cervical biopsy region identification method based on the cervical biopsy region identification device according to any one of claims 1 to 5, characterized by comprising the steps of:
(1) collecting a physiological saline image, an acetic acid image and an iodine image of the cervix by an image collecting unit, and inputting the images into a cervix biopsy area identification model in a data processing unit;
(2) and analyzing and processing the physiological saline image, the acetic acid image and the iodine image through the cervical biopsy area identification model, outputting a probability label of the existence of the biopsy area of the cervix, and displaying the probability label on a display unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810092566.XA CN108319977B (en) | 2018-01-30 | 2018-01-30 | Cervical biopsy region identification method and device based on channel information multi-mode network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810092566.XA CN108319977B (en) | 2018-01-30 | 2018-01-30 | Cervical biopsy region identification method and device based on channel information multi-mode network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108319977A CN108319977A (en) | 2018-07-24 |
| CN108319977B true CN108319977B (en) | 2020-11-10 |
Family
ID=62887779
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810092566.XA Active CN108319977B (en) | 2018-01-30 | 2018-01-30 | Cervical biopsy region identification method and device based on channel information multi-mode network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108319977B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112602097A (en) * | 2018-08-31 | 2021-04-02 | 奥林巴斯株式会社 | Data processing system and data processing method |
| CN109376594A (en) | 2018-09-11 | 2019-02-22 | 百度在线网络技术（北京）有限公司 | Visual perception method, apparatus, equipment and medium based on automatic driving vehicle |
| CN109363632A (en) * | 2018-09-26 | 2019-02-22 | 北京三医智慧科技有限公司 | The deciphering method of pulse profile data and the solution read apparatus of pulse profile data |
| CN109544512B (en) * | 2018-10-26 | 2020-09-18 | 浙江大学 | Multi-mode-based embryo pregnancy result prediction device |
| CN109543719B (en) * | 2018-10-30 | 2020-09-08 | 浙江大学 | Cervical atypical lesion diagnosis model and device based on multi-modal attention model |
| CN109859159B (en) * | 2018-11-28 | 2020-10-13 | 浙江大学 | Cervical lesion region segmentation method and device based on multi-mode segmentation network |
| CN111369567B (en) * | 2018-12-26 | 2022-12-16 | 腾讯科技（深圳）有限公司 | Method and device for segmenting target object in three-dimensional image and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102282569A (en) * | 2008-10-10 | 2011-12-14 | 国际科学技术医疗系统有限责任公司 | Methods for tissue classification in cervical imagery |
| CN103750810A (en) * | 2013-12-30 | 2014-04-30 | 深圳市理邦精密仪器股份有限公司 | Method and device for performing characteristic analysis for images acquired by electronic colposcope |
| CN104881631A (en) * | 2015-04-16 | 2015-09-02 | 广西师范大学 | Multi-characteristic integrated cervical cell image characteristic extraction and identification method, and cervical cell characteristic identification device |
| CN106529446A (en) * | 2016-10-27 | 2017-03-22 | 桂林电子科技大学 | Vehicle type identification method and system based on multi-block deep convolutional neural network |
| CN106780475A (en) * | 2016-12-27 | 2017-05-31 | 北京市计算中心 | A kind of image processing method and device based on histopathologic slide's image organizational region |
| CN107066583A (en) * | 2017-04-14 | 2017-08-18 | 华侨大学 | A kind of picture and text cross-module state sensibility classification method merged based on compact bilinearity |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9922272B2 (en) * | 2014-09-25 | 2018-03-20 | Siemens Healthcare Gmbh | Deep similarity learning for multimodal medical images |
-
2018
- 2018-01-30 CN CN201810092566.XA patent/CN108319977B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102282569A (en) * | 2008-10-10 | 2011-12-14 | 国际科学技术医疗系统有限责任公司 | Methods for tissue classification in cervical imagery |
| CN103750810A (en) * | 2013-12-30 | 2014-04-30 | 深圳市理邦精密仪器股份有限公司 | Method and device for performing characteristic analysis for images acquired by electronic colposcope |
| CN104881631A (en) * | 2015-04-16 | 2015-09-02 | 广西师范大学 | Multi-characteristic integrated cervical cell image characteristic extraction and identification method, and cervical cell characteristic identification device |
| CN106529446A (en) * | 2016-10-27 | 2017-03-22 | 桂林电子科技大学 | Vehicle type identification method and system based on multi-block deep convolutional neural network |
| CN106780475A (en) * | 2016-12-27 | 2017-05-31 | 北京市计算中心 | A kind of image processing method and device based on histopathologic slide's image organizational region |
| CN107066583A (en) * | 2017-04-14 | 2017-08-18 | 华侨大学 | A kind of picture and text cross-module state sensibility classification method merged based on compact bilinearity |
Non-Patent Citations (1)
| Title |
|---|
| Medical Image Segmentation Based on Multi-Modal;Zhe Guo et al;《arXiv》;20171103;第1-10页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108319977A (en) | 2018-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108319977B (en) | Cervical biopsy region identification method and device based on channel information multi-mode network | |
| CN108388841B (en) | Cervical biopsy region identification method and device based on multi-feature deep neural network | |
| US20240135544A1 (en) | Method and system for digital staining of label-free fluorescence images using deep learning | |
| RU2765619C1 (en) | Computer classification of biological tissue | |
| CN109543719B (en) | Cervical atypical lesion diagnosis model and device based on multi-modal attention model | |
| CN108257129B (en) | Cervical biopsy region auxiliary identification method and device based on multi-mode detection network | |
| EP3705025A1 (en) | Image diagnosis assistance apparatus, data collection method, image diagnosis assistance method, and image diagnosis assistance program | |
| EP4187486A1 (en) | Method for co-expression analysis | |
| JP2023544466A (en) | Training method and device for diagnostic model of lung adenocarcinoma and squamous cell carcinoma based on PET/CT | |
| CN109948671B (en) | Image classification method, device, storage medium and endoscopic imaging equipment | |
| CN110189293A (en) | Cell image processing method, device, storage medium and computer equipment | |
| US20230251199A1 (en) | Identifying auto-fluorescent artifacts in a multiplexed immunofluorescent image | |
| CN112741651A (en) | Method and system for processing ultrasonic image of endoscope | |
| CN116740435A (en) | Breast cancer ultrasonic image classifying method based on multi-mode deep learning image group science | |
| CN114372951A (en) | Nasopharyngeal carcinoma positioning and segmenting method and system based on image segmentation convolutional neural network | |
| CN115049666A (en) | Endoscope virtual biopsy device based on color wavelet covariance depth map model | |
| CN109859159B (en) | Cervical lesion region segmentation method and device based on multi-mode segmentation network | |
| CN115063592A (en) | Multi-scale-based full-scanning pathological feature fusion extraction method and system | |
| CN113450305B (en) | Medical image processing method, system, equipment and readable storage medium | |
| Sobrevilla et al. | Fuzzy-based analysis of microscopic color cervical pap smear images: nuclei detection | |
| CN115294434A (en) | Implementation method of multi-modal learning convolutional neural network model based on molecular spectral image feature acquisition and fusion | |
| CN113222061B (en) | MRI image classification method based on two-way small sample learning | |
| CN114898862A (en) | Cervical cancer computer-aided diagnosis method based on convolutional neural network and pathological section image | |
| CN111798427B (en) | System for detecting karyokiness in gastrointestinal stromal tumor based on migration learning | |
| CN114330484A (en) | Method and system for classification and focus identification of diabetic retinopathy through weak supervision learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |