CN116091427A - Dermatosis classification device, classification method and storage medium - Google Patents
Dermatosis classification device, classification method and storage medium Download PDFInfo
- Publication number
- CN116091427A CN116091427A CN202211711452.1A CN202211711452A CN116091427A CN 116091427 A CN116091427 A CN 116091427A CN 202211711452 A CN202211711452 A CN 202211711452A CN 116091427 A CN116091427 A CN 116091427A
- Authority
- CN
- China
- Prior art keywords
- feature
- local
- attention
- fusion
- clinical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30088—Skin; Dermal
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Ultra Sonic Daignosis Equipment (AREA)
Abstract
The invention discloses a skin disease classification device, a classification method and a storage medium, which are used for respectively carrying out global feature extraction and local feature extraction on clinical images and ultrasonic images of the same skin disease focus of a skin disease patient to obtain global clinical image features and local clinical image features of the clinical images, global ultrasonic image features and local ultrasonic image features of the ultrasonic images; inputting a first attention characteristic of the local clinical image characteristic and a second attention characteristic of the local ultrasonic image characteristic into a preset multi-mode characteristic fusion module to obtain a first local characteristic and a second local characteristic; splicing the first local features with the global clinical image features, and splicing the second local features with the global ultrasonic image features to obtain clinical fusion features and ultrasonic fusion features; the clinical fusion features and the ultrasonic fusion features are input into a preset classifier to predict the skin disease type of the skin patient to be more relevant to the region of interest, thereby improving the accuracy of the skin disease classification.
Description
Technical Field
The present invention relates to the field of deep learning technology, and in particular, to a dermatological classification device, a classification method, and a storage medium
Background
Skin disease is one of the most common diseases in the world, and is used to express abnormal skin tissue, and can be used as an important index for diagnosing cancer, for example: basal cell tumor, lipoma, cyst, nevus, wart, etc. In clinical practice, the diagnosis of skin diseases is mainly performed by an expert. But for dermatologists who are not trained, large-scale dermatological screening and diagnosis cannot be performed quickly.
With the continuous development of artificial neural network technology in recent years, the skin disease images of skin disease patients can be classified through the artificial neural network to determine whether the skin disease of the skin disease patients belongs to benign or malignant. The dermatological image may refer to a clinical image, an ultrasound image, etc., the clinical image is photographed by a camera, and the ultrasound image is photographed by an ultrasound imaging device. However, the visual characteristics between benign and malignant of each skin disease are highly similar, so that the existing medical image recognition method based on the artificial neural network has the problem of low recognition accuracy when recognizing the region of interest.
Disclosure of Invention
The invention mainly aims to provide a skin disease classification method, a terminal and a storage medium, and aims to solve the problem that in the prior art, when a medical image identification method based on an artificial neural network is used for identifying a region of interest, identification accuracy is low.
In order to achieve the above object, the present invention provides a dermatological disorder classification apparatus, the apparatus comprising:
the image acquisition module is used for acquiring clinical images and ultrasonic images of the same skin disease focus of a skin disease patient;
the feature extraction module is used for carrying out global feature extraction and local feature extraction on the clinical image and the ultrasonic image respectively to obtain global clinical image features and local clinical image features of the clinical image and global ultrasonic image features and local ultrasonic image features of the ultrasonic image; wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area;
the attention characteristic extraction module is used for extracting attention characteristics of the local clinical image characteristics and the local ultrasonic image characteristics to obtain a first attention characteristic and a second attention characteristic;
the feature interaction fusion module is used for inputting the first attention feature and the second attention feature into the preset multi-mode feature fusion module to perform feature interaction fusion to obtain a first local feature and a second local feature;
the feature stitching module is used for stitching the first local feature with the global clinical image feature to obtain a clinical fusion feature; the second local feature is spliced with the global ultrasonic image feature to obtain an ultrasonic fusion feature;
The prediction module is used for predicting the skin disease type of the dermatological patient through a preset classifier based on the clinical fusion characteristics and the ultrasonic fusion characteristics; wherein the dermatological type comprises: benign, malignant.
Optionally, the feature extraction module includes:
the cutting unit is used for cutting the clinical image and the ultrasonic image to obtain a local clinical image containing a focus area in the clinical image and a local ultrasonic image containing a focus area in the ultrasonic image; wherein the focus areas corresponding to the local clinical image and the local ultrasonic image are the same;
the feature extraction unit is used for inputting the clinical image and the ultrasonic image into a corresponding first preset convolutional neural network to perform feature extraction so as to obtain the global clinical image feature and the global ultrasonic image feature; and
and the local clinical image and the local ultrasonic image are input into a second preset convolution nerve for feature extraction, so that the local clinical image features and the local ultrasonic image features are obtained.
Optionally, the second preset convolutional neural network is composed of a plurality of basic blocks, and residual connection is performed among the basic blocks;
Each basic block is formed by sequentially connecting a first convolution layer, a first normalization layer and a first ReLU activation function.
Optionally, the attention feature extraction module includes:
the first splicing unit is used for inputting the local clinical image characteristics and the local ultrasonic image characteristics into a preset collaborative learning module, and splicing the local clinical image characteristics and the local ultrasonic image characteristics through a connecting layer of the preset collaborative learning module to obtain first fusion characteristics;
and the attention feature extraction unit is used for respectively inputting the first fusion feature into a first attention learning branch and a second attention learning branch of the preset collaborative learning module to obtain the first attention feature and the second attention feature.
Optionally, the attention feature extraction unit is specifically configured to:
inputting the first fusion features into a second convolution layer of the preset collaborative learning module to perform feature extraction to obtain first fusion features after feature extraction;
and respectively inputting the first fusion characteristic after the characteristic extraction into the first attention mechanical learning branch network and the second attention mechanical learning branch network to obtain the first attention characteristic and the second attention characteristic.
Optionally, the first attention mechanical learning branch network is formed by sequentially connecting a third convolution layer, a second ReLU activation function, a second normalization layer, a maximum pooling layer, a fourth convolution layer, a second ReLU activation function, a third normalization layer and a first sigmoid activation function;
the second attention learning branch network is formed by sequentially connecting a fifth convolution layer, a third ReLU activation function, a fourth normalization layer, a maximum pooling layer, a sixth convolution layer, a fifth ReLU activation function, a fifth normalization layer and a second sigmoid activation function.
Optionally, the feature interaction fusion module includes:
the second splicing unit is used for splicing the first attention characteristic and the second attention characteristic to obtain an attention fusion characteristic;
the feature extraction unit is used for inputting the attention fusion features into a first channel attention module and a second channel attention module of the preset multi-mode feature fusion module respectively to obtain a first output feature of the clinical image and a second output feature of the ultrasonic image; and
the method comprises the steps of inputting the attention fusion features into a seventh convolution layer of the preset multi-mode feature fusion module respectively for feature extraction, and inputting the attention fusion features after feature extraction into an activation function of the preset multi-mode feature fusion module to obtain an attention feature map;
The first feature interaction fusion unit is used for generating a fusion public feature according to the attention feature map, the first attention feature and the second attention feature;
the second feature fusion interaction unit is used for inputting the fusion public feature and the first output feature into an eighth convolution layer of the preset multi-mode feature fusion module after splicing to obtain the first local feature; and
and the ninth convolution layer is used for inputting the fusion public feature and the second output feature into the preset multi-mode feature fusion module after the fusion public feature and the second output feature are spliced, so that the second local feature is obtained.
Optionally, the prediction module is specifically configured to:
inputting the clinical fusion characteristics into a first CBS module to obtain a first output value; and
inputting the ultrasonic fusion characteristic into a second CBS module to obtain a second output value;
inputting the first output value and the second output value into a preset classifier, and determining the skin disease type of the skin disease patient through the preset classifier;
the first CBS module sequentially comprises a tenth convolution layer, a sixth normalization layer and a first SiLU activation function; the second CBS module is composed of an eleventh convolution layer, a seventh normalization layer and a second SiLU activation function in sequence.
In order to achieve the above object, the present invention also provides a skin disease classifying method of a skin disease classifying device, the method comprising:
acquiring clinical images and ultrasonic images of the same dermatological focus of a dermatological patient;
global feature extraction and local feature extraction are respectively carried out on the clinical image and the ultrasonic image, so that global clinical image features and local clinical image features of the clinical image, and global ultrasonic image features and local ultrasonic image features of the ultrasonic image are obtained; wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area;
extracting attention features of the local clinical image features and the local ultrasonic image features to obtain a first attention feature and a second attention feature;
inputting the first attention characteristic and the second attention characteristic into a preset multi-mode characteristic fusion module to perform characteristic interaction fusion to obtain a first local characteristic and a second local characteristic;
splicing the first local features and the global clinical image features to obtain clinical fusion features; the second local feature is spliced with the global ultrasonic image feature to obtain an ultrasonic fusion feature;
Predicting the skin disease type of the dermatological patient by a preset classifier based on the clinical fusion feature and the ultrasonic fusion feature; wherein the dermatological type comprises: benign, malignant.
To achieve the above object, the present invention also provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps in the dermatological classification method as described above.
According to the invention, two modal information of a skin disease patient, namely a clinical image and an ultrasonic image aiming at the same skin disease focus, are used for obtaining corresponding global clinical image characteristics and local clinical image characteristics, global ultrasonic image characteristics and local ultrasonic image characteristics, a preset multi-modal characteristic fusion module is input according to the first attention characteristics of the local clinical image characteristics and the second attention characteristics of the local ultrasonic image for carrying out characteristic interaction fusion, a first local characteristic and a second local characteristic which are mutually fused in two modes are obtained, and then the first local characteristic and the global clinical image characteristics are used for carrying out fusion, and the second local characteristic and the global ultrasonic image characteristics are used for carrying out fusion to obtain clinical fusion characteristics and ultrasonic fusion characteristics respectively, so that the characteristics are more relevant to the focus area, the skin disease classification is carried out by the clinical fusion characteristics and the ultrasonic fusion characteristics on the basis again, the skin disease classification is more focused on the area of interest, the accuracy of the identification of the area of interest is improved, the classification result is more accurate, and the misdiagnosis condition is avoided.
Drawings
Fig. 1 is a schematic structural diagram of a dermatological classification apparatus according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for classifying skin diseases according to an embodiment of the present invention;
fig. 3 is a flowchart of step S202 provided in an embodiment of the present invention;
FIG. 4 is another flow chart of a method for classifying skin disorders according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a basic block structure according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a preset collaborative learning module according to an embodiment of the present invention;
fig. 7 is a flowchart of step S203 provided in the embodiment of the present invention;
FIG. 8 is a block diagram of a preset multi-modal feature fusion module according to an embodiment of the present invention;
fig. 9 is a flowchart of step S204 provided in the embodiment of the present invention;
fig. 10 is a flowchart of step S206 provided in an embodiment of the present invention;
fig. 11 is another flowchart of a method for classifying skin diseases according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more clear and clear, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
An embodiment of the present invention provides a dermatological disease classification apparatus, as shown in fig. 1, including at least: the system comprises an image acquisition module 110, a feature extraction module 120, an attention feature extraction module 130, a feature interaction fusion module 140, a feature stitching module 150 and a prediction module 160.
Wherein the image acquisition module 110 is used for acquiring clinical images and ultrasonic images of the same dermatological focus of a dermatological patient.
The feature extraction module 120 is configured to perform global feature extraction and local feature extraction on the clinical image and the ultrasound image, respectively, to obtain global clinical image features and local clinical image features of the clinical image, and global ultrasound image features and local ultrasound image features of the ultrasound image; wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area.
The attention feature extraction module 130 is configured to perform attention feature extraction on the local clinical image feature and the local ultrasound image feature, so as to obtain a first attention feature and a second attention feature.
The feature interaction fusion module 140 is configured to input the first attention feature and the second attention feature into a preset multi-mode feature fusion module to perform feature interaction fusion, so as to obtain a first local feature and a second local feature.
The feature stitching module 150 is configured to stitch the first local feature and the global clinical image feature to obtain a clinical fusion feature; and the second local feature is spliced with the global ultrasonic image feature to obtain an ultrasonic fusion feature.
The prediction module 160 is configured to predict a dermatological type of a dermatological patient by a preset classifier based on the clinical fusion feature and the ultrasound fusion feature. Among the skin disease types include: benign, malignant.
Further, as shown in fig. 1, the feature extraction module 120 includes: a clipping unit 121 and a feature extraction unit 122.
The clipping unit 121 is used for clipping the clinical image and the ultrasound image to obtain a local clinical image including the lesion area in the clinical image and a local ultrasound image including the lesion area in the ultrasound image.
Wherein, the focus areas corresponding to the local clinical image and the local ultrasonic image are the same.
The feature extraction unit 122 is configured to input the clinical image and the ultrasound image into a corresponding first preset convolutional neural network to perform feature extraction, so as to obtain global clinical image features and global ultrasound image features; and the local clinical image and the local ultrasonic image are input into a second preset convolution nerve for feature extraction, so that local clinical image features and local ultrasonic image features are obtained.
In some embodiments of the present invention, the second preset convolutional neural network is composed of a plurality of basic blocks, and residual connection is performed between the basic blocks. And each basic block is formed by sequentially connecting a first convolution layer, a first normalization layer and a first ReLU activation function.
Further, the attention feature extraction module 130 includes: a first stitching unit 131, and an attention feature extraction unit 132.
Specifically, the first stitching unit 131 is configured to input the local clinical image feature and the local ultrasound image feature into a preset collaborative learning module, stitch the local clinical image feature and the local ultrasound image feature through a connection layer that obtains the preset collaborative learning module, and obtain a first fusion feature;
the attention feature extraction unit 132 is configured to input the first fusion feature into a first attention learning branch and a second attention learning branch of a preset collaborative learning module, respectively, to obtain a first attention feature and a second attention feature.
Further, the attention feature extraction unit 132 is specifically configured to:
inputting the first fusion features into a second convolution layer of a preset collaborative learning module to perform feature extraction to obtain first fusion features after feature extraction;
And respectively inputting the first fusion characteristic after the characteristic extraction into a first attention mechanical learning branch network and a second attention learning branch network to obtain a first attention characteristic and a second attention characteristic.
In some embodiments of the present invention, the first attention mechanical learning branch network is formed by sequentially connecting a third convolution layer, a second ReLU activation function, a second normalization layer, a maximum pooling layer, a fourth convolution layer, a second ReLU activation function, a third normalization layer, and a first sigmoid activation function.
And the second attention learning branch network is formed by sequentially connecting a fifth convolution layer, a third ReLU activation function, a fourth normalization layer, a maximum pooling layer, a sixth convolution layer, a fifth ReLU activation function, a fifth normalization layer and a second sigmoid activation function.
As shown in fig. 1, the feature interaction fusion module 140 includes: the device comprises a second splicing unit 141, a feature extraction unit 142, a first feature interaction fusion unit 143 and a second feature interaction fusion unit 144.
Specifically, the second stitching unit 141 is configured to stitch the first attention feature and the second attention feature to obtain an attention fusion feature.
The feature extraction unit 142 is configured to input the attention fusion feature into a first channel attention module and a second channel attention module of a preset multi-mode feature fusion module, respectively, to obtain a first output feature of the clinical image and a second output feature of the ultrasound image; and the seventh convolution layer is used for respectively inputting the attention fusion features into the preset multi-mode feature fusion module to extract the features, and inputting the attention fusion features after the feature extraction into an activation function of the preset multi-mode feature fusion module to obtain an attention feature map.
The first feature interaction fusion unit 143 is configured to generate a fused common feature according to the attention feature map, the first attention feature and the second attention feature.
The second feature fusion interaction unit 144 is configured to splice the fused common feature and the first output feature, and input the spliced common feature and the first output feature into an eighth convolution layer of the preset multi-mode feature fusion module to obtain a first local feature; and a ninth convolution layer used for inputting the spliced common features and the second output features into a preset multi-mode feature fusion module after splicing the common features and the second output features to obtain second local features.
In some embodiments of the invention, the prediction module is specifically configured to:
inputting the clinical fusion characteristics into a first CBS module to obtain a first output value; and
inputting the ultrasonic fusion characteristic into a second CBS module to obtain a second output value;
inputting the first output value and the second output value into a preset classifier, and predicting the skin disease type of the skin disease patient through the preset classifier;
the first CBS module sequentially comprises a tenth convolution layer, a sixth normalization layer and a first SiLU activation function; the second CBS module consists of an eleventh convolutional layer, a seventh normalizing layer, and a second SiLU activation function in sequence.
Based on this, an embodiment of the present invention provides a skin disease classification method based on the skin disease classification device, as shown in fig. 2, where the skin disease classification method may at least include the following steps:
S201, acquiring clinical images and ultrasonic images of the same dermatological focus of a dermatological patient.
In the embodiment of the invention, aiming at the same skin disease focus of a skin disease patient, two corresponding mode data, namely a clinical image corresponding to a clinical mode and an ultrasonic image corresponding to an ultrasonic mode, are acquired.
Specifically, the same skin disease focus of the skin disease patient is photographed by a camera and an ultrasonic imaging device respectively, and a clinical image and an ultrasonic image of the same skin disease focus are obtained.
In addition, whether the photographed clinical image and the ultrasonic image meet corresponding preset image standards can be detected firstly, and if not, the clinical image and the ultrasonic image can be subjected to image preprocessing firstly, so that the clinical image and the ultrasonic image meeting the corresponding preset image standards are obtained. The image preprocessing may include: scaling, rotation, cropping, horizontal flipping, normalization processing, etc., are not particularly limited in embodiments of the present invention.
S202, global feature extraction and local feature extraction are respectively carried out on the clinical image and the ultrasonic image, so that global clinical image features and local clinical image features of the clinical image, and global ultrasonic image features and local ultrasonic image features of the ultrasonic image are obtained.
Wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area.
The clinical image and the ultrasonic image include a background area around the lesion area in addition to the lesion area. Thus, the local clinical image features are image features of a lesion area in a clinical image and the local ultrasound image features are image features of a lesion area in an ultrasound image.
Further, as shown in fig. 3, step S202 may include at least the following steps:
s301, cutting the clinical image and the ultrasonic image to obtain a local clinical image containing a focus area in the clinical image and a local ultrasonic image containing the focus area in the ultrasonic image.
Wherein, the focus areas corresponding to the local clinical image and the local ultrasonic image are the same.
Further, the clinical image and the ultrasonic image can be respectively subjected to image recognition to determine the clinical focus area in the clinical image and the ultrasonic focus area in the ultrasonic image. Then, cutting the clinical image according to the clinical focus area to obtain a local clinical image; and cutting the ultrasonic image according to the ultrasonic focus area to obtain a local ultrasonic image.
S302, inputting the clinical image and the ultrasonic image into a corresponding first preset convolutional neural network to perform feature extraction, and obtaining global clinical image features and global ultrasonic image features.
In an embodiment of the present invention, as shown in fig. 4, two first preset convolutional neural networks 410 may be provided, and the two first preset convolutional neural networks are trained convolutional neural networks, for example: an efficiency network.
The method comprises the steps that a first preset convolutional neural network is used for extracting features of clinical images to obtain global clinical image features; the other first preset convolutional neural network is used for extracting features of the ultrasonic image to obtain global ultrasonic image features.
S303, inputting the local clinical image and the local ultrasonic image into a second preset convolution nerve for feature extraction to obtain local clinical image features and local ultrasonic image features.
In an embodiment of the present invention, as shown in fig. 4, the second preset convolutional neural network 450 may be composed of a plurality of basic blocks, and residual connection is performed between the basic blocks. As shown in fig. 5, each basic block is composed of a first convolution layer 510, a first normalization layer 520, and a first ReLU activation function 530 connected in sequence.
And, the second preset convolutional neural network can share network parameters when extracting features of the local clinical image and the local ultrasonic image, and the process can be described as:
f 1 ＝G(x 1 )；
f 2 ＝G(x 2 )；
x i+1 ＝G i+1 (x i )＝ReLu i+1 (BN i+1 (Conv i+1 (x i )))；
wherein f 1 Representing local clinical image features after feature extraction through a second preset convolution network; f (f) 2 Representing feature extraction via a second predetermined convolutional networkTaking the characteristics of the local ultrasonic image; x is x 1 Representing an input local clinical image; x is x 2 Representing an input local ultrasound image; g (·) represents a second preset convolutional network; g i+1 (. Cndot.) represents the corresponding basic block structure. i=1 represents a clinical modality, and i=2 represents an ultrasound modality.
And S203, attention characteristic extraction is carried out on the local clinical image characteristics and the local ultrasonic image characteristics, and a first attention characteristic of the local clinical image and a second attention characteristic of the local ultrasonic image are obtained.
In an embodiment of the present invention, as shown in fig. 4, the local clinical image features and the local ultrasound image features may be input into a trained preset collaborative learning module 420 to obtain a first attention feature of the local clinical image and a second attention feature of the local ultrasound image.
As shown in fig. 6, the preset collaborative learning module sequentially includes: a connection layer 610, a second convolution layer 620 (whose convolution kernel may be 1*1), a first attention learning branch network 630, and a second attention learning branch network 640. The first attention learning branch network 630 is formed by sequentially connecting a third convolution layer 631, a second ReLU activation function 632, a second normalization layer 632, a maximum pooling layer 633, a fourth convolution layer 634, a second ReLU activation function 635, a third normalization layer 635, and a first sigmoid activation function 636; the second attention learning branch network 640 is composed of a fifth convolution layer 641, a third ReLU activation function 642, a fourth normalization layer 642, a maximum pooling layer 643, a sixth convolution layer 644, a fifth ReLU activation function 645, a fifth normalization layer 645, and a second sigmoid activation function 646 connected in this order.
Further, as shown in fig. 7, step S203 may be implemented at least by:
s701, inputting the local clinical image features and the local ultrasonic image features into a preset collaborative learning module, and splicing the local clinical image features and the local ultrasonic image features through a connecting layer of the preset collaborative learning module to obtain a first fusion feature.
It will be appreciated that the stitching of the local clinical image features and the local ultrasound image features may be performed with the same feature dimensions, and if the feature dimensions are different, the two may be preprocessed to make the feature dimensions the same.
S702, inputting the first fusion features into a second convolution layer of a preset collaborative learning module to perform feature extraction, and obtaining the first fusion features after feature extraction.
In the embodiment of the invention, the first fusion characteristic is extracted through the second convolution layer, so that the two mode information of the ultrasonic mode and the clinical mode can be fused better, and the accuracy of the skin disease classification is further improved.
S703, inputting the first fusion features after feature extraction into a first attention mechanical learning branch network and a second attention mechanical learning branch network of a preset collaborative learning module respectively to obtain a first attention feature and a second attention feature.
The first attention mechanical training branch network and the second attention mechanical training branch network do not share weights.
Further, the preset collaborative learning model may be expressed as:
F＝Conv(Cat(f 1 ,f 2 ))
F 1 ＝sigmoid 1 (CRB 1 (F))
F 2 ＝sigmoid 2 (CRB 2 (F))
wherein Cat (f) 1 ,f 2 ) As the first fusion feature, F is the first fusion feature after feature extraction, sigmoid 1 、sigmoid 2 To activate the function F 1 For the first attentional feature, F 2 For the second attentional feature, CRB 1 Representing first attention learning network branches, CRB 2 Representing a second attention to the learning network branch.
Through the steps S701-S703, the characteristic information between the two modes can be learned, and the influence of different modes on the classification of the skin diseases is fully considered, so that the accuracy of the classification of the skin diseases is further improved.
S204, inputting the first attention characteristic and the second attention characteristic into a preset multi-mode characteristic fusion module to perform characteristic interaction fusion, and obtaining a first local characteristic of the clinical image and a second local characteristic of the ultrasonic image.
In the embodiment of the present invention, as shown in fig. 4, the preset multi-modal feature fusion module 460 is also trained, and as shown in fig. 8, the preset multi-modal feature fusion module includes: a first channel attention module 810, a second channel attention module 820, a seventh convolution layer 830, a third sigmoid activation function 840, an eighth convolution layer 850, and a ninth convolution layer 860.
Further, as shown in fig. 9, step S204 may include at least the following steps:
and S901, splicing the first attention characteristic and the second attention characteristic to obtain an attention fusion characteristic.
S902, respectively inputting the attention fusion features into a first channel attention module and a second channel attention module of a preset multi-mode feature fusion module to obtain a first output feature and a super second output feature.
As shown in fig. 8, attention fusion feature F 1 + 2 And respectively inputting the first channel attention module and the second channel attention module to obtain a first output characteristic and a second output characteristic.
S903, respectively inputting the attention fusion features into a seventh convolution layer of a preset multi-mode feature fusion module to extract the features, and inputting the attention fusion features after the feature extraction into an activation function of the preset multi-mode feature fusion module to obtain an attention feature map.
In particular, the attention profile can be expressed as:
U＝Sigmoid(Conv(Cat(F 1 ,F 2 )))；
wherein Cat (F) 1 ,F 2 ) And (5) representing the attention fusion characteristics obtained after the splicing, and U representing an attention characteristic diagram.
S904, generating a fusion public feature according to the attention characteristic diagram, the first attention characteristic and the second attention characteristic.
In particular, the attention profile may be multiplied by a first attention profile, a second attention profile,and adding the multiplied characteristic information to obtain a fused common characteristic F ′ The following is shown:
F ′ ＝*(F 1 + 2 )。
further, the multiplication of the attention profile with the first attention profile, respectively, means that each pixel point in the attention profile is multiplied with the first attention profile. Similarly, the attention profile is multiplied by the second attention profile, respectively, meaning that each pixel in the attention profile is multiplied by the second attention profile.
It should be noted that, step S902 may be performed first, and then steps S903 to S904 may be performed. Or, steps S903 to S904 are performed first, and then step S902 is performed; alternatively, step S902 and steps S903-S904 are performed simultaneously, which is not specifically limited in the embodiment of the present invention.
S905, splicing the fusion public feature and the first output feature, and inputting the spliced fusion public feature and the first output feature into an eighth convolution layer of a preset multi-mode feature fusion module to obtain the first local feature.
As shown in fig. 8, common features F are fused ′ Splicing the first output characteristic and the first output characteristic on the channel, and performing convolution processing through a sixth convolution layer to obtain a second mode interaction
′
Local feature F 1 The following is shown:
F 1 ′ ＝Conv(Cat(F ′ ,ECA(F 1 )))；
wherein ECA (·) represents the first channel attention module.
S906, the fusion public feature and the second output feature are spliced and then input into a ninth convolution layer of a preset multi-mode feature fusion module, and a second local feature is obtained.
As shown in fig. 8, common features F are fused ′ Splicing the first output characteristic and the second output characteristic on the channel, and carrying out convolution processing through a seventh convolution layer to obtain a first mode interaction after the interaction of the two modes
′
Local feature F 2 The following is shown:
F 2 ′ ＝Conv(Cat(F ′ ,ECA(F 2 )))；
wherein ECA (·) represents the second channel attention module.
It should be noted that step S905 may be performed first, and then step S906 may be performed; or, step S906 is performed first, and then step S905 is performed; alternatively, step S905 is performed simultaneously with step S906, which is not specifically limited in the embodiment of the present invention.
And S205, splicing the first local features and the global clinical image features to obtain clinical fusion features.
And S206, splicing the second local feature and the global ultrasonic image feature to obtain an ultrasonic fusion feature.
′
As shown in FIG. 4, a first local feature F 1 And global clinical image feature H 1 Performing splicing
′
Obtaining clinical fusion characteristic M 1 . Second local feature F 2 And global ultrasound image feature H 2 Splicing to obtain ultrasonic fusion characteristics M 2 。
It should be noted that step S205 may be performed first, and then step S206 may be performed; or, step S206 is executed first, and step S205 is executed next; alternatively, step S205 and step S206 are performed simultaneously, which is not particularly limited in the embodiment of the present invention.
S207, predicting the skin disease type of the skin disease patient through a preset classifier based on the clinical fusion characteristics and the ultrasonic fusion characteristics.
Among the skin disease types include: benign, malignant.
As shown in fig. 10, step S1006 of the embodiment of the present invention may at least include the following steps:
s1001, inputting the clinical fusion characteristic into a first CBS module to obtain a first output value.
S1002, inputting the ultrasonic fusion characteristic into a second CBS module to obtain a second output value.
As shown in fig. 4, the first CBS module 430 is composed of a tenth convolution layer 431, a sixth normalization layer 432, and a first SiLU activation function 433 in this order; the second CBS module 440 consists of, in order, an eleventh convolution layer 441, a seventh normalization layer 442, and a second SiLU activation function 443.
It should be noted that, step S1001 may be executed first, and then step S1002 may be executed; or, step S1002 is executed first, and then step S1001 is executed; alternatively, the step S1001 and the step S1002 are performed simultaneously, which is not specifically limited in the embodiment of the present invention.
S1003, inputting the first output value and the second output value into a preset classifier, and predicting the skin disease type of the skin disease patient through the preset classifier.
In the embodiment of the invention, the preset classifier is a trained classifier and is used for classifying the types of the skin diseases and determining whether the skin diseases of the skin patients are benign or malignant.
As shown in fig. 11, the clinical image and the ultrasonic image obtained for the same dermatological focus are respectively subjected to image preprocessing, the clinical image and the ultrasonic image after the image preprocessing are input into a dermatological classification model as input data for prediction, and the dermatological type corresponding to the dermatological focus is determined according to the predicted label. Wherein the skin disease classification model is a trained model, and the structure of the skin disease classification model is shown as 400 in fig. 4. And, to different skin diseases, for example basal cell carcinoma, lipoma use its corresponding skin disease classification model, can be according to clinical image, ultrasonic image and corresponding classification label (including benign, malignant) of different skin diseases as the training sample, carry on model training, get with the classified model of skin disease trained.
The invention provides a skin classification method, which is characterized in that two types of modal information of a skin patient, namely clinical images and ultrasonic images aiming at the same skin disease focus, are used for obtaining corresponding global clinical image features and local clinical image features, global ultrasonic image features and local ultrasonic image features, a preset multi-modal feature fusion module is input according to the first attention features of the local clinical image features and the second attention features of the local ultrasonic images for carrying out feature interaction fusion, so that the first local features and the second local features of the two types of modal fusion are obtained, and then the first local features and the global clinical image features are spliced, and the second local features and the global ultrasonic image features are spliced to obtain clinical fusion features and ultrasonic fusion features respectively, so that the characteristics of focus areas are more relevant, the distinguishing characteristic information of a diaphragm is enhanced, the skin disease classification is carried out on the basis of the clinical fusion features and the ultrasonic fusion features again, the skin disease classification is carried out on the basis of the clinical fusion features and the ultrasonic fusion features, the accuracy of identifying the region of interest is improved, the classification result is more accurate, and the situation of misdiagnosis is avoided.
Based on the above-described dermatological classification method, the present invention also provides a computer-readable storage medium storing one or more programs executable by one or more processors to implement the steps in the dermatological classification method described in the above-described embodiments.
All embodiments in the application are described in a progressive manner, and identical and similar parts of all embodiments are mutually referred, so that each embodiment mainly describes differences from other embodiments. In particular, for apparatus and storage medium embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the description of method embodiments in part.
The apparatus and the storage medium provided in the embodiments of the present application are in one-to-one correspondence with the methods, so that the apparatus and the storage medium also have similar beneficial technical effects as the corresponding methods, and since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the apparatus and the storage medium are not described again here.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Of course, those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by a computer program for instructing relevant hardware (e.g., processor, controller, etc.), the program may be stored on a computer readable storage medium, and the program may include the above described methods when executed. The computer readable storage medium may be a memory, a magnetic disk, an optical disk, etc.
It is to be understood that the invention is not limited in its application to the examples described above, but is capable of modification and variation in light of the above teachings by those skilled in the art, and that all such modifications and variations are intended to be included within the scope of the appended claims.
Claims (10)
1. A dermatological disorder classification apparatus, the apparatus comprising:
the image acquisition module is used for acquiring clinical images and ultrasonic images of the same skin disease focus of a skin disease patient;
the feature extraction module is used for carrying out global feature extraction and local feature extraction on the clinical image and the ultrasonic image respectively to obtain global clinical image features and local clinical image features of the clinical image and global ultrasonic image features and local ultrasonic image features of the ultrasonic image; wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area;
The attention characteristic extraction module is used for extracting attention characteristics of the local clinical image characteristics and the local ultrasonic image characteristics to obtain a first attention characteristic and a second attention characteristic;
the feature interaction fusion module is used for inputting the first attention feature and the second attention feature into the preset multi-mode feature fusion module to perform feature interaction fusion to obtain a first local feature and a second local feature;
the feature stitching module is used for stitching the first local clinical feature with the global clinical image feature to obtain a clinical fusion feature; the second local feature is spliced with the global ultrasonic image feature to obtain an ultrasonic fusion feature;
the prediction module is used for predicting the skin disease type of the dermatological patient through a preset classifier based on the clinical fusion characteristics and the ultrasonic fusion characteristics; wherein the dermatological type comprises: benign, malignant.
2. The dermatological classification device of claim 1, wherein the feature extraction module comprises:
the cutting unit is used for cutting the clinical image and the ultrasonic image to obtain a local clinical image containing a focus area in the clinical image and a local ultrasonic image containing a focus area in the ultrasonic image; wherein the focus areas corresponding to the local clinical image and the local ultrasonic image are the same;
The feature extraction unit is used for inputting the clinical image and the ultrasonic image into a corresponding first preset convolutional neural network to perform feature extraction so as to obtain the global clinical image feature and the global ultrasonic image feature; and
and the local clinical image and the local ultrasonic image are input into a second preset convolution nerve for feature extraction, so that the local clinical image features and the local ultrasonic image features are obtained.
3. The dermatological classification device of claim 2, wherein the second predetermined convolutional neural network is comprised of a plurality of basic blocks, each basic block being connected by a residual error;
each basic block is formed by sequentially connecting a first convolution layer, a first normalization layer and a first ReLU activation function.
4. The dermatological classification device of claim 1, wherein the attention feature extraction module includes:
the first splicing unit is used for inputting the local clinical image characteristics and the local ultrasonic image characteristics into a preset collaborative learning module, and splicing the local clinical image characteristics and the local ultrasonic image characteristics through a connecting layer of the preset collaborative learning module to obtain first fusion characteristics;
And the attention feature extraction unit is used for respectively inputting the first fusion feature into a first attention learning branch and a second attention learning branch of the preset collaborative learning module to obtain the first attention feature and the second attention feature.
5. The dermatological classification device of claim 4, wherein the attention feature extraction unit is specifically configured to:
inputting the first fusion features into a second convolution layer of the preset collaborative learning module to perform feature extraction to obtain first fusion features after feature extraction;
and respectively inputting the first fusion characteristic after the characteristic extraction into the first attention mechanical learning branch network and the second attention mechanical learning branch network to obtain the first attention characteristic and the second attention characteristic.
6. The dermatological classification device of claim 4, wherein the first attention mechanical branch network is comprised of a third convolution layer, a second ReLU activation function, a second normalization layer, a max pooling layer, a fourth convolution layer, a second ReLU activation function, a third normalization layer, a first sigmoid activation function, connected in sequence;
the second attention learning branch network is formed by sequentially connecting a fifth convolution layer, a third ReLU activation function, a fourth normalization layer, a maximum pooling layer, a sixth convolution layer, a fifth ReLU activation function, a fifth normalization layer and a second sigmoid activation function.
7. The dermatological classification device of claim 1, wherein the feature interaction fusion module comprises:
the second splicing unit is used for splicing the first attention characteristic and the second attention characteristic to obtain an attention fusion characteristic;
the feature extraction unit is used for inputting the attention fusion features into a first channel attention module and a second channel attention module of the preset multi-mode feature fusion module respectively to obtain a first output feature of the clinical image and a second output feature of the ultrasonic image; and
the method comprises the steps of inputting the attention fusion features into a seventh convolution layer of the preset multi-mode feature fusion module respectively for feature extraction, and inputting the attention fusion features after feature extraction into an activation function of the preset multi-mode feature fusion module to obtain an attention feature map;
the first feature interaction fusion unit is used for generating a fusion public feature according to the attention feature map, the first attention feature and the second attention feature;
the second feature fusion interaction unit is used for inputting the fusion public feature and the first output feature into an eighth convolution layer of the preset multi-mode feature fusion module after splicing to obtain the first local feature; and
And the ninth convolution layer is used for inputting the fusion public feature and the second output feature into the preset multi-mode feature fusion module after the fusion public feature and the second output feature are spliced, so that the second local feature is obtained.
8. The dermatological classification device of claim 1, wherein the prediction module is specifically configured to:
inputting the clinical fusion characteristics into a first CBS module to obtain a first output value; and
inputting the ultrasonic fusion characteristic into a second CBS module to obtain a second output value;
inputting the first output value and the second output value into a preset classifier, and determining the skin disease type of the skin disease patient through the preset classifier;
the first CBS module sequentially comprises a tenth convolution layer, a sixth normalization layer and a first SiLU activation function; the second CBS module is composed of an eleventh convolution layer, a seventh normalization layer and a second SiLU activation function in sequence.
9. A method of classifying skin disorders of a skin disorder classifying device, the method comprising:
acquiring clinical images and ultrasonic images of the same dermatological focus of a dermatological patient;
global feature extraction and local feature extraction are respectively carried out on the clinical image and the ultrasonic image, so that global clinical image features and local clinical image features of the clinical image, and global ultrasonic image features and local ultrasonic image features of the ultrasonic image are obtained; wherein the local clinical image features and the local ultrasound image features are local features for the same lesion area;
Extracting attention features of the local clinical image features and the local ultrasonic image features to obtain a first attention feature and a second attention feature;
inputting the first attention characteristic and the second attention characteristic into a preset multi-mode characteristic fusion module to perform characteristic interaction fusion to obtain a first local characteristic and a second local characteristic;
splicing the first local features and the global clinical image features to obtain clinical fusion features; the second local feature is spliced with the global ultrasonic image feature to obtain an ultrasonic fusion feature;
predicting the skin disease type of the dermatological patient by a preset classifier based on the clinical fusion feature and the ultrasonic fusion feature; wherein the dermatological type comprises: benign, malignant.
10. A computer-readable storage medium storing one or more programs executable by one or more processors to perform the steps in the dermatological classification method of any of claims 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211711452.1A CN116091427A (en) | 2022-12-29 | 2022-12-29 | Dermatosis classification device, classification method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211711452.1A CN116091427A (en) | 2022-12-29 | 2022-12-29 | Dermatosis classification device, classification method and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116091427A true CN116091427A (en) | 2023-05-09 |
Family
ID=86205642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211711452.1A Pending CN116091427A (en) | 2022-12-29 | 2022-12-29 | Dermatosis classification device, classification method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116091427A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118196102A (en) * | 2024-05-17 | 2024-06-14 | 华侨大学 | Method and device for detecting breast ultrasonic tumor lesion area based on double-network shadow removal |
-
2022
- 2022-12-29 CN CN202211711452.1A patent/CN116091427A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118196102A (en) * | 2024-05-17 | 2024-06-14 | 华侨大学 | Method and device for detecting breast ultrasonic tumor lesion area based on double-network shadow removal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110689025B (en) | Image recognition method, device and system and endoscope image recognition method and device | |
| EP4006831A1 (en) | Image processing method and apparatus, server, medical image processing device and storage medium | |
| CN112102237A (en) | Brain tumor recognition model training method and device based on semi-supervised learning | |
| CN110796199B (en) | Image processing method and device and electronic medical equipment | |
| CN111209916B (en) | Focus identification method and system and identification equipment | |
| Muhammad et al. | Visual saliency models for summarization of diagnostic hysteroscopy videos in healthcare systems | |
| CN110827236B (en) | Brain tissue layering method, device and computer equipment based on neural network | |
| CN114565761A (en) | Deep learning-based method for segmenting tumor region of renal clear cell carcinoma pathological image | |
| Yue et al. | Retinal vessel segmentation using dense U-net with multiscale inputs | |
| CN112330624A (en) | Medical image processing method and device | |
| CN112561937A (en) | Context-aware medical image segmentation method based on lesion boundary information response | |
| CN116091427A (en) | Dermatosis classification device, classification method and storage medium | |
| WO2024074921A1 (en) | Distinguishing a disease state from a non-disease state in an image | |
| Dubey et al. | Deconstructive human face recognition using deep neural network | |
| Hu et al. | Automatic detection of melanins and sebums from skin images using a generative adversarial network | |
| CN116703837B (en) | MRI image-based rotator cuff injury intelligent identification method and device | |
| CN114283406A (en) | Cell image recognition method, device, equipment, medium and computer program product | |
| CN113705595A (en) | Method, device and storage medium for predicting degree of abnormal cell metastasis | |
| CN117237351A (en) | Ultrasonic image analysis method and related device | |
| Xian et al. | Automatic tongue image quality assessment using a multi-task deep learning model | |
| CN115116117A (en) | Learning input data acquisition method based on multi-mode fusion network | |
| CN112907525B (en) | Mammary gland magnetic resonance image intelligent classification method, device, equipment and medium | |
| CN117726822B (en) | Three-dimensional medical image classification segmentation system and method based on double-branch feature fusion | |
| CN117496323B (en) | Multi-scale second-order pathological image classification method and system based on transducer | |
| Naidu et al. | Dermato: A Deep Learning based Application for Acne Subtype and Severity Detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |