WO2023108968A1 - 基于知识驱动的深度学习图像分类方法和系统 - Google Patents

基于知识驱动的深度学习图像分类方法和系统 Download PDF

Info

Publication number
WO2023108968A1
WO2023108968A1 PCT/CN2022/087216 CN2022087216W WO2023108968A1 WO 2023108968 A1 WO2023108968 A1 WO 2023108968A1 CN 2022087216 W CN2022087216 W CN 2022087216W WO 2023108968 A1 WO2023108968 A1 WO 2023108968A1
Authority
WO
WIPO (PCT)
Prior art keywords
training
image classification
model
module
knowledge
Prior art date
Application number
PCT/CN2022/087216
Other languages
English (en)
French (fr)
Inventor
鄂海红
宋美娜
何佳雯
胡天翼
张如如
李国英
王莉菲
袁立飞
Original Assignee
北京邮电大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京邮电大学 filed Critical 北京邮电大学
Publication of WO2023108968A1 publication Critical patent/WO2023108968A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present application relates to the technical field of image classification, in particular to a knowledge-driven deep learning image classification method and system.
  • the size of the data volume is an important factor for the classification effect of the deep learning model.
  • a large amount of labeled data is required for self-learning. If the data volume is too small, the model cannot effectively learn the key features in the image, thus affecting image classification. the accuracy rate.
  • the acquisition and labeling of datasets are relatively simple, so there are currently abundant datasets for researchers to train models.
  • the acquisition and labeling of training data sets are very difficult and expensive, and the amount of data in related data sets is not enough, which brings certain challenges to researchers. .
  • the main research methods in the field of image classification are: data-driven, which directly uses large data sets for model training; data enhancement, which uses image processing operations such as image flipping, rotation, scaling, and contrast enhancement to artificially expand data.
  • the expanded data set is used for model training; transfer learning, model training is performed on large batches of natural image data sets, and model parameters are transferred to professional fields with less data.
  • data-driven provides sufficient training data for the deep learning model, so that the model can effectively learn key image features, but this solution requires a large amount of labeled data, which cannot be applied to professional fields with insufficient data.
  • Data enhancement the use of data enhancement methods can compensate for the impact of insufficient data to a certain extent, but the image obtained by the expansion has a high similarity with the original image, and the improvement of the accuracy of the model is limited, and if the data enhancement method is used improperly, there may even be Wrong labeled data may be introduced, which will adversely affect the learning of the model.
  • Migration learning reduces the dependence of the model on the amount of data by migrating the trained feature extraction model to the learning of new tasks, but the data distribution of the transferred source domain and the target domain are often different. Migration learning introduces model weights and also provides The target domain task introduces certain errors.
  • This application aims to solve one of the technical problems in the related art at least to a certain extent.
  • the first purpose of this application is to propose a knowledge-driven deep learning image classification method.
  • the second purpose of this application is to propose a knowledge-driven deep learning image classification system.
  • the embodiment of the first aspect of the present application proposes a knowledge-driven deep learning image classification method, including: constructing a knowledge-driven deep learning image classification model, and training the constructed model, wherein, based on The knowledge-driven deep learning image classification model includes a feature extraction module, a priori knowledge pre-training module, and an image classification module; obtain the image to be classified, use the feature extraction module to perform feature extraction on the image to be classified, and obtain the feature vector; divide the feature vector Input the prior knowledge pre-training module and the image classification module to obtain the prior knowledge multi-label classification results and image classification results, wherein the training of the constructed model includes prior knowledge pre-training and image classification task training, and the data used for training
  • the labeling of the set includes prior knowledge labeling and classification labeling, prior knowledge pre-training, including: Step S1: use the feature extraction module and prior knowledge pre-training module, and use the dataset labeled with prior knowledge for training, and extract features The weight of the model is fine-tuned; Step S2: If the pre
  • the training of the image classification task includes: Step 1: Using the feature extraction module and the image classification module, and using the classification labeled data set, the knowledge-driven depth Learn the image classification model to train the classification task; Step 2: If the training result of the model classification task does not reach the expected accuracy, adjust the hyperparameters, and repeat Step 1 until the training result of the model classification task reaches the expected accuracy, and complete the classification task training.
  • the feature extraction module is used to perform feature extraction on the image to be classified, and the feature vector is obtained, expressed as:
  • Model baseline is the feature extraction model
  • F is the feature vector
  • x is the input image.
  • the feature vector is input into the prior knowledge pre-training module, specifically, the prior knowledge score is obtained by using the fully connected layer, and the input image is obtained by selecting a category with a score greater than a preset threshold in the prior knowledge score.
  • Prior knowledge multi-label classification results where the prior knowledge score is expressed as:
  • F is the feature vector
  • W k is the weight matrix of the fully connected layer.
  • the feature vector is input into the image classification module, specifically using the fully connected layer to obtain the image classification result score, and the image classification result of the input image is obtained by selecting a category with a score greater than a preset threshold in the image classification result score , where the image classification result score is expressed as:
  • F is the feature vector
  • W c is the weight matrix of the image classification fully connected layer.
  • the knowledge-driven deep learning image classification model also includes a heat map visualization module, and uses the heat map visualization module to visualize the heat map of the image classification results to obtain a heat map, including the following steps:
  • the reflow gradient is globally averaged and pooled in the width and height dimensions to obtain the importance weight of the feature map;
  • the feature map importance weight is expressed as:
  • h is the height of the feature map of the last layer
  • w is the width of the feature map of the last layer
  • Z h ⁇ w
  • A represents the last convolutional layer of the feature extraction model
  • a k ij represents the value of the last convolutional layer at channel k, height i, and width j
  • the heat map is expressed as:
  • RELU() represents the RELU activation function
  • a k represents the matrix of the last convolutional layer of the feature extraction model with channel k
  • the embodiment of the second aspect of the present application proposes a knowledge-driven deep learning image classification system, including: an acquisition module and a knowledge-driven deep learning image classification model, and a knowledge-driven deep learning image classification model Including feature extraction module, prior knowledge pre-training module, image classification module, wherein,
  • the obtaining module is used to obtain the image to be classified and input it into the knowledge-driven deep learning image classification model;
  • the feature extraction module is used to use the feature extraction model to perform feature extraction on the image to be classified to obtain a feature vector
  • the prior knowledge pre-training module is used to apply the feature vector to the prior knowledge pre-training to obtain the prior knowledge multi-label classification result;
  • the image classification module is used for applying the feature vector to the image classification to obtain the image classification result.
  • the knowledge-driven deep learning image classification model further includes a heat map visualization module, which is used to visualize the heat map of the image classification result to obtain a heat map.
  • it also includes training a knowledge-driven deep learning image classification model, including:
  • Step S1 Select an appropriate feature extraction model according to the data set and task characteristics
  • Step S2 Perform prior knowledge pre-training on the feature extraction model using the prior knowledge annotation results
  • Step S3 If the pre-training result does not reach the expected accuracy, adjust the hyperparameters or the feature extraction model, repeat step S2 until the pre-training result reaches the expected accuracy, and complete the prior knowledge pre-training;
  • Step S4 using the classification and labeling results to perform classification task training on the feature extraction model
  • Step S5 If the training result of the model classification task does not reach the expected accuracy, adjust the hyperparameters, and repeat step S4 until the training result of the model classification task reaches the expected accuracy, and the classification task training is completed.
  • FIG. 1 is a flowchart of a knowledge-driven deep learning image classification method provided in some embodiments of the present application
  • FIG. 2 is a schematic framework diagram of a knowledge-driven deep learning image classification model in some embodiments of the present application
  • FIG. 3 is a structural diagram of a knowledge-driven deep learning image classification model in some embodiments of the present application.
  • FIG. 4 is an example diagram of a knowledge-driven deep learning image classification model in a dual-modal scenario in some embodiments of the present application
  • Fig. 5 is the training flowchart of the knowledge-driven deep learning image classification model in some embodiments of the present application.
  • FIG. 6 is a flow chart of using a knowledge-driven deep learning image classification model in some embodiments of the present application.
  • Fig. 7 is a schematic structural diagram of a knowledge-driven deep learning image classification system provided in some embodiments of the present application.
  • Knowledge-driven does not depend on the amount of training data, and improves the learning efficiency of the model by artificially adding prior knowledge.
  • prior knowledge By introducing prior knowledge to reduce the model's dependence on data, it provides a good solution for areas where data set acquisition or labeling is difficult, but there are currently few studies in this field, and most researchers will a priori Knowledge is designed as the segmentation of image features, while reducing the amount of data, it also further brings the pressure of data segmentation and labeling.
  • a knowledge-driven deep learning image classification method proposed in this application reduces the amount of training data required by the deep learning model, and does not need to introduce complex segmentation and labeling, reducing the pressure of data labeling.
  • FIG. 1 is a flow chart of a knowledge-driven deep learning image classification method provided in Embodiment 1 of the present application.
  • the knowledge-driven deep learning image classification method includes the following steps 101 to 103 .
  • Step 101 constructing a knowledge-driven deep learning image classification model, and training the constructed model, wherein the knowledge-driven deep learning image classification model includes a feature extraction module, a priori knowledge pre-training module, and an image classification module;
  • Step 102 obtain the image to be classified, and use the feature extraction module to perform feature extraction on the image to be classified to obtain a feature vector;
  • step 103 the feature vectors are respectively input into the prior knowledge pre-training module and the image classification module to obtain the prior knowledge multi-label classification result and the image classification result.
  • the training of the built model includes prior knowledge pre-training and image classification task training
  • the annotation of the dataset used for training includes prior knowledge annotation and classification annotation
  • prior knowledge pre-training includes: Step S1: Using feature Extraction module and prior knowledge pre-training module, and use the data set marked by prior knowledge for training, and fine-tune the weight of the feature extraction model; Step S2: If the pre-training result does not meet the expected accuracy, adjust the hyperparameters or feature extraction model , repeat step S1 until the pre-training result reaches the expected accuracy, and complete the prior knowledge pre-training.
  • the knowledge-driven deep learning image classification method in the embodiment of the present application constructs a knowledge-driven deep learning image classification model and trains the constructed model, wherein the knowledge-driven deep learning image classification model includes a feature extraction module , prior knowledge pre-training module, image classification module; obtain the image to be classified, use the feature extraction module to perform feature extraction on the image to be classified, and obtain the feature vector; input the feature vector into the prior knowledge pre-training module and the image classification module respectively, Obtain prior knowledge multi-label classification results and image classification results, wherein the training of the constructed model includes prior knowledge pre-training and image classification task training, and the annotation of the data set used for training includes prior knowledge annotation and classification annotation, Prior knowledge pre-training, including: Step S1: use the feature extraction module and prior knowledge pre-training module, and use the data set marked by prior knowledge for training, and fine-tune the weight of the feature extraction model; Step S2: if the pre-training If the result does not reach the expected accuracy, adjust the hyperparameters or feature extraction model, repeat step S1 until the pre-
  • This application uses a multi-label classification method to integrate prior knowledge into the learning of the deep learning model, which effectively relieves the pressure brought by segmentation and labeling, and improves the usability of the system in some professional image classification fields.
  • the feature extraction module of the knowledge-driven deep learning image classification model of the embodiment of the present application highlights the representative features in the image, and the prior knowledge pre-training module allows the model to fully learn the prior knowledge and image classification in the form of pre-training.
  • the module completes image classification tasks, and the heat map visualization module provides interpretability.
  • this application innovatively introduces a two-stage knowledge-driven method. The first stage is to train and learn prior knowledge, that is, the key image features used for classification task decision-making, and the second stage is to perform image classification task training.
  • the training of the image classification task includes:
  • Step 1 Use the feature extraction module and the image classification module, and use the classified and labeled data set to train the classification task on the knowledge-driven deep learning image classification model that has been pre-trained with prior knowledge;
  • Step 2 If the training result of the model classification task does not reach the expected accuracy, adjust the hyperparameters, and repeat step 1 until the training result of the model classification task reaches the expected accuracy, and the classification task training is completed.
  • the feature extraction module transforms the input image to highlight representative features in the image, such as edges, corners, colors, and so on.
  • the mainstream feature extraction models in computer vision can be used in Knowledge_Model (a knowledge-driven deep learning image classification model), such as VGGNet, GoogleNet, ResNet, etc.
  • Knowledge_Model can choose different feature extraction models.
  • Model baseline is the feature extraction model
  • F is the feature vector
  • x is the input image.
  • the knowledge-driven deep learning image classification model Before performing the image classification task, pre-train the knowledge-driven deep learning image classification model with the help of the multi-label classification labeling result y k of prior knowledge. In this way, the knowledge-driven deep learning image classification model can learn Prior knowledge reduces the dependence of the knowledge-driven deep learning image classification model on the amount of training data, and improves the learning efficiency and accuracy of the knowledge-driven deep learning image classification model.
  • prior knowledge is generally marked as the characteristic performance in the image that plays a decisive role in the classification task.
  • prior knowledge can be marked as retinal hemorrhage and drusen in fundus images. , or retinal fluid in optical coherence tomography, macular pigment epithelial detachment and other pathological signs;
  • prior knowledge annotations can be marked as frequency band information such as center frequency and bandwidth, or Quadrature phase shift keying, quadrature amplitude modulation and other modulation methods.
  • the feature vector is input into the prior knowledge pre-training module, specifically using the fully connected layer to obtain the prior knowledge score
  • the prior knowledge multi-label classification result of the input image is obtained by selecting the category with a score greater than the preset threshold in the prior knowledge score, for example, by selecting the prior knowledge score
  • the category with a score greater than 0.5 is used to obtain the prior knowledge multi-label classification result of the input image, where the prior knowledge score is expressed as:
  • F is the feature vector
  • W k is the weight matrix of the fully connected layer.
  • the prior knowledge is marked in the form of multi-label classification, and the feature vector F is processed using a fully connected layer.
  • the fully connected layers of the prior knowledge pre-training module are different from the fully connected layers of the image classification module.
  • the feature extraction model Model baseline will fine-tune the weights according to the multi-label classification results.
  • the knowledge-driven deep learning image classification model is forced to learn the features and prior knowledge in the image.
  • the corresponding relationship between labels helps the knowledge-driven deep learning image classification model to better pay attention to the image features that play a decisive role in the classification task.
  • multi-label classification labeling is much less difficult than segmentation labeling, which effectively reduces the cost of dataset labeling compared to existing knowledge-driven methods.
  • the image classification task is trained using the label label y of the image classification task.
  • the fully connected layer of the prior knowledge multi-label classification module is discarded, the feature vector F output by the feature extraction module is used, and the image classification is realized through the fully connected layer of the image classification module.
  • the feature vector is input into the image classification module, specifically using the fully connected layer to obtain the image classification result score
  • the image classification result of the input image is obtained by selecting the category whose score is greater than the preset threshold in the image classification result score.
  • the image classification result score can be selected by The category with a score greater than 0.5 is used to obtain the classification result of the input image, where the score of the image classification result is expressed as:
  • F is the feature vector
  • W c is the weight matrix of the image classification fully connected layer.
  • the image classification module participates in training after the prior knowledge multi-label classification module, and reuses the feature extraction model Model baseline .
  • Model baseline has effectively learned the prior knowledge in the pre-training process. In the image classification task, it can better pay attention to the image features that play a decisive role in the classification results.
  • the training of image classification can converge faster, and It can achieve better classification accuracy than directly training the image classification model.
  • this application visualizes the image classification results as a heat map.
  • the knowledge-driven deep learning image classification model also includes a heat map visualization module, and uses the heat map visualization module to visualize the heat map of the image classification results to obtain the heat map, including the following steps:
  • h is the height of the feature map of the last layer
  • w is the width of the feature map of the last layer
  • Z h ⁇ w
  • A represents the last convolutional layer of the feature extraction model
  • a k ij represents the value of the last convolutional layer at channel k, height i, and width j
  • the feature map importance weight captures the degree of influence of the channel k of the feature map of the last convolutional layer on the target category c, and combines the obtained feature map importance weight with the feature map activation value weighted combination, and then obtains the heat map through the RELU activation function ,Expressed as:
  • RELU() represents the RELU activation function
  • a k represents the matrix of the last convolutional layer of the feature extraction model with channel k
  • FIG. 2 is a schematic framework diagram of a knowledge-driven deep learning image classification model according to an embodiment of the present application.
  • the knowledge-driven deep learning image classification model includes a feature extraction module, a prior knowledge pre-training module, and an image classification module, wherein the feature extraction module is used to use the feature extraction model to perform classification on images to be classified. Feature extraction to obtain feature vectors; prior knowledge pre-training module for applying feature vectors to prior knowledge pre-training to obtain prior knowledge multi-label classification results; image classification module for applying feature vectors to image classification, Get the image classification result.
  • the knowledge-driven deep learning image classification model also includes a heat map visualization module, which is used to visualize the heat map of the image classification results to obtain a heat map.
  • FIG. 3 is a structural diagram of a knowledge-driven deep learning image classification model according to an embodiment of the present application.
  • Model baseline is a feature extraction model.
  • the knowledge-driven deep learning image classification model receives the input x and outputs the classification result of the image
  • the key feature multi-label classification results referred to for classification Can be expressed as: Among them, the knowledge-driven deep learning image classification model is represented by "Knowledge_Model".
  • the input of the image feature extraction module is the image data x
  • the output is the feature vector F extracted by the feature extraction model.
  • the input of the prior knowledge pre-training module is the feature vector F
  • the output is the prior knowledge score by choosing The category with a score greater than the preset threshold is used to obtain the prior knowledge multi-label classification result of the input image
  • the image classification task is trained using the label label y of the image classification task.
  • the fully connected layer of the prior knowledge multi-label classification module is discarded, the feature vector F output by the feature extraction module is used, and the image classification is realized through the fully connected layer of the image classification module.
  • the input of the image classification module is the feature vector F
  • the output is the classification result score by choosing Classification results of the input image are obtained from the category with a score greater than the preset threshold Heat map visualization module, using image classification results to score Backpropagation to the last convolutional layer of the feature extraction model, the reflow gradient is globally averaged and pooled in the width and height dimensions, and the importance weight of the feature map is obtained, which is weighted and combined with the activation value of the feature map, and then through the RELU activation function. Get a heatmap.
  • the knowledge-driven deep learning image classification model is also applicable. For example: the scene of synthesizing multiple medical images for disease diagnosis.
  • n feature extraction modules A total of n feature vectors ⁇ F 1 , F 2 ,...,F n ⁇ are obtained, where any F i ⁇ R N , i ⁇ 1,2,...,n ⁇ .
  • the feature extraction models are independent of each other and the weights are not shared.
  • prior knowledge pre-training modules the prior knowledge of different modal images is different, and a prior knowledge pre-training module needs to be constructed according to the specific annotation content to realize the multi-label classification task of this modality image. In this way, each feature extraction model can better pay attention to the image features that the input of the corresponding modality plays a decisive role in the classification task.
  • image classification module concatenate n feature vectors to form a fusion multimodal feature vector F con ⁇ R n ⁇ N , the image classification module takes the fusion feature vector F con as input, and outputs the image classification result score
  • the n input modalities are comprehensively analyzed by means of feature fusion, and the final result of the image classification task is given.
  • FIG. 4 is an example diagram of a knowledge-driven deep learning image classification model in a dual-modal scenario according to an embodiment of the present application.
  • FIG. 5 is a flow chart of training a knowledge-driven deep learning image classification model according to an embodiment of the present application.
  • the training result of the classification task of the knowledge-driven deep learning image classification model does not meet the expected accuracy, adjust the hyperparameters, and use the classification annotation data set again to classify the knowledge-driven deep learning image classification
  • the model is trained on the classification task until the training result of the classification task of the knowledge-driven deep learning image classification model reaches the expected accuracy; if the training result of the classification task of the knowledge-driven deep learning image classification model reaches the expected accuracy, the process ends.
  • FIG. 6 is a flow chart of using the knowledge-driven deep learning image classification model according to the embodiment of the present application.
  • the image to be classified is obtained and uploaded to the knowledge-driven deep learning image classification model; the feature extraction module of the knowledge-driven deep learning image classification model extracts features from the image to obtain a feature vector; The vector is applied to the prior knowledge pre-training module and the image classification module; the basis for model classification (multi-label classification results of prior knowledge) and model classification results (classification task results) are obtained, and the heat map visualization of the image classification results is performed to obtain the model classification A heat map of the region of interest of the input image.
  • FIG. 7 is a schematic structural diagram of a knowledge-driven deep learning image classification system provided in Embodiment 2 of the present application.
  • the knowledge-driven deep learning image classification system includes an acquisition module and a knowledge-driven deep learning image classification model
  • the knowledge-driven deep learning image classification model includes a feature extraction module, prior knowledge pre-training module, image classification module, where,
  • Obtaining module 10 is used for obtaining the image to be classified and input in the knowledge-driven deep learning image classification model;
  • Feature extraction module 20 is used for using feature extraction model to carry out feature extraction to the image to be classified, obtains feature vector;
  • the prior knowledge pre-training module 30 is used to apply the feature vector to the prior knowledge pre-training to obtain the prior knowledge multi-label classification result;
  • Image classification module 40 is used for applying feature vector to image classification, obtains image classification result,
  • the knowledge-driven deep learning image classification model also includes a thermal map visualization module 50 for visualizing the image classification result to obtain a thermal map.
  • training the knowledge-driven deep learning image classification model including:
  • Step S1 Select an appropriate feature extraction model according to the data set and task characteristics
  • Step S2 Perform prior knowledge pre-training on the feature extraction model using the prior knowledge annotation results
  • Step S3 If the pre-training result does not reach the expected accuracy, adjust the hyperparameters or the feature extraction model, repeat step S2 until the pre-training result reaches the expected accuracy, and complete the prior knowledge pre-training;
  • Step S4 using the classification and labeling results to perform classification task training on the feature extraction model
  • Step S5 If the training result of the model classification task does not reach the expected accuracy, adjust the hyperparameters, and repeat step S4 until the training result of the model classification task reaches the expected accuracy, and the classification task training is completed.
  • the knowledge-driven deep learning image classification system of the embodiment of the present application includes an acquisition module and a knowledge-driven deep learning image classification model, and the knowledge-driven deep learning image classification model includes a feature extraction module, a priori knowledge pre-training module, The image classification module, wherein, the acquisition module is used to obtain the image to be classified and input it into the knowledge-driven deep learning image classification model; the feature extraction module is used to use the feature extraction model to perform feature extraction on the image to be classified to obtain the feature vector ;
  • the prior knowledge pre-training module is used to apply the feature vector to the prior knowledge pre-training to obtain the prior knowledge multi-label classification result; the image classification module is used to apply the feature vector to the image classification to obtain the image classification result.
  • this application adopts the multi-label classification method to integrate prior knowledge into the learning of the deep learning model, which effectively relieves the pressure brought by segmentation and labeling, and improves the usability of the system in some professional image classification fields.
  • the feature extraction module of the knowledge-driven deep learning image classification model of the embodiment of the present application highlights the representative features in the image, and the prior knowledge pre-training module allows the model to fully learn the prior knowledge and image classification in the form of pre-training.
  • the module completes image classification tasks, and the heat map visualization module provides interpretability.
  • this application innovatively introduces a two-stage knowledge-driven method. The first stage is to train and learn prior knowledge, that is, the key image features used for classification task decision-making, and the second stage is to perform image classification task training.
  • the knowledge-driven deep learning image classification method and system of the embodiments of the present application can enhance the accuracy of image feature extraction and classification tasks, reduce the dependence of deep learning models on data volume, and relieve the pressure of data collection and labeling in some professional fields .
  • this application adopts the multi-label classification method to integrate prior knowledge into the learning of the deep learning model, which effectively relieves the pressure brought by segmentation and labeling, and improves the usability of the system in some professional image classification fields.
  • first and second are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features.
  • the features defined as “first” and “second” may explicitly or implicitly include at least one of these features.
  • “plurality” means at least two, such as two, three, etc., unless otherwise specifically defined.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device.
  • computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program can be printed, as it may be possible, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or other suitable processing if necessary.
  • the program is processed electronically and stored in computer memory.
  • each part of the present application may be realized by hardware, software, firmware or a combination thereof.
  • various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware as in another embodiment, it can be implemented by any one or a combination of the following techniques known in the art: a discrete Logic circuits, ASICs with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.
  • each functional unit in each embodiment of the present application may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module.
  • the above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.
  • the storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提出了一种基于知识驱动的深度学习图像分类方法和系统，涉及图像分类技术领域，该方法包括：构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练；获取待分类的图像，使用特征提取模块对待分类的图像进行特征提取，得到特征向量；将特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果。

Description

基于知识驱动的深度学习图像分类方法和系统
相关申请的交叉引用
本申请基于申请号为202111531480.0、申请日为2021年12月14日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及图像分类技术领域，尤其涉及一种基于知识驱动的深度学习图像分类方法和系统。
背景技术
当前深度学习技术在图像分类任务的研究已经达到了相对成熟的阶段，卷积神经网络被认为是一种强大的用于视觉图像分析的深度学习模型，它可以很好地实现图像中复杂的特征提取和识别，例如在ImageNet图像分类任务中提出的AlexNet、Inception-v3、EfficientNet等模型，已实现了最高97.7％的Top-5准确率。
数据量的大小是深度学习模型分类效果的重要因素，模型在训练过程中，需要大量的标注数据进行自主学习，数据量过小会导致模型无法有效学习到图像中的关键特征，从而影响图像分类的准确率。在自然图像的分类任务中，如人脸识别、猫狗分类等，数据集的获取和标注较简单，因此当前已存在数据量丰富的数据集供研究者进行模型的训练。然而在一些专业领域中，如医学影像、电磁信号图像的智能识别等，训练数据集的获取和标注都非常困难且价格昂贵，相关数据集的数据量不够，为研究者带来了一定的挑战。
目前图像分类领域的主要研究方法有：数据驱动，直接使用大数据量的数据集进行模型的训练；数据增强，使用图像翻转、旋转、缩放、对比度增强等图像处理操作对数据进行人为扩充，使用扩充后的数据集进行模型的训练；迁移学习，在大批量的自然图像数据集上进行模型训练，并将模型参数迁移到数据量较少的专业领域上。
现有技术中，数据驱动为深度学习模型提供了充足的训练数据，使模型可以有效学习到关键的图像特征，但该方案需要大批量的标注数据，无法适用于数据量不足的专业领域。数据增强，使用数据增强方式一定程度上弥补了数据量不足带来的影响，但扩充得到的图像与原图相似性较高，模型准确率的提升有限，并且如果数据增强方法使用不当，甚至有可能引入错误的标注数据，为模型的学习带来不利影响。迁移学习通过将已训练好的特征提取模型迁移至新任务的学习中，减少模型对数据量的依赖，但迁移的源域和目标域的数据分布往往不同，迁移学习引入模型权重的同时也为目标域任务引入了一定的误差。
综上所述，在现阶段的深度学习图像分类研究中，大部分的研究工作还未能提出有效的方法缓解某些专业领域的数据收集、数据标注压力，使深度学习技术在相关领域的 发展受到限制。
发明内容
本申请旨在至少在一定程度上解决相关技术中的技术问题之一。
为此，本申请的第一个目的在于提出一种基于知识驱动的深度学习图像分类方法。
本申请的第二个目的在于提出一种基于知识驱动的深度学习图像分类系统。
为达上述目的，本申请第一方面实施例提出了一种基于知识驱动的深度学习图像分类方法，包括：构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；获取待分类的图像，使用特征提取模块对待分类的图像进行特征提取，得到特征向量；将特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，其中，对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，先验知识预训练，包括：步骤S1：使用特征提取模块和先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
在本申请的一个实施例中，图像分类任务的训练，包括：步骤一：使用特征提取模块和图像分类模块，并使用分类标注的数据集，对经过先验知识预训练的基于知识驱动的深度学习图像分类模型进行分类任务的训练；步骤二：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤一，直到模型分类任务训练结果达到预期精度，完成分类任务训练。
在本申请的一个实施例中，使用特征提取模块对待分类的图像进行特征提取，得到特征向量，表示为：
F＝Model baseline(x)
其中，Model baseline为特征提取模型，F为特征向量，x为输入图像。
在本申请的一个实施例中，将特征向量输入先验知识预训练模块，具体为使用全连接层得到先验知识得分，通过选择先验知识得分中得分大于预设阈值的类别得到输入图像的先验知识多标签分类结果，其中，先验知识得分表示为：
Figure PCTCN2022087216-appb-000001
其中，F为特征向量，W k为全连接层的权重矩阵。
在本申请的一个实施例中，将特征向量输入图像分类模块，具体为使用全连接层得到图像分类结果得分，通过选择图像分类结果得分中得分大于预设阈值的类别得到输入图像的图像分类结果，其中，图像分类结果得分表示为：
Figure PCTCN2022087216-appb-000002
其中，F为特征向量，W c为图像分类全连接层的权重矩阵。
在本申请的一个实施例中，基于知识驱动的深度学习图像分类模型还包括热力图可视化 模块，使用热力图可视化模块对图像分类结果进行热力图可视化，得到热力图，包括以下步骤：
使用图像分类结果得分，反向传播至特征提取模型的最后一层卷积层，回流的梯度在宽度和高度维度上全局平均池化，获得特征图重要性权重；
将得到的特征图重要性权重与特征图激活值加权组合，然后通过RELU激活函数来获得热力图，
其中，特征图重要性权重表示为：
Figure PCTCN2022087216-appb-000003
其中，h为最后一层特征图的高度，w为最后一层特征图的宽度，Z＝h×w，
Figure PCTCN2022087216-appb-000004
表示图像分类结果得分，A表示特征提取模型的最后一层卷积层，A k ij表示最后一层卷积层在通道为k、高为i、宽为j处的值，
热力图表示为：
Figure PCTCN2022087216-appb-000005
其中，RELU()表示RELU激活函数，A k表示特征提取模型的最后一层卷积层在通道为k的矩阵，
Figure PCTCN2022087216-appb-000006
表示特征图重要性权重。
为达上述目的，本申请第二方面实施例提出了一种基于知识驱动的深度学习图像分类系统，包括：获取模块和基于知识驱动的深度学习图像分类模型，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块，其中，
获取模块，用于获取待分类的图像并输入基于知识驱动的深度学习图像分类模型中；
特征提取模块，用于使用特征提取模型对待分类的图像进行特征提取，得到特征向量；
先验知识预训练模块，用于将特征向量应用于先验知识预训练，得到先验知识多标签分类结果；
图像分类模块，用于将特征向量应用于图像分类，得到图像分类结果。
在本申请的一个实施例中，基于知识驱动的深度学习图像分类模型还包括热力图可视化模块，用于对图像分类结果进行热力图可视化，得到热力图。
在本申请的一个实施例中，还包括，对基于知识驱动的深度学习图像分类模型进行训练，包括：
步骤S1：根据数据集及任务特点选择合适的特征提取模型；
步骤S2：使用先验知识标注结果对特征提取模型进行先验知识预训练；
步骤S3：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S2，直到预训练结果达到预期精度，完成先验知识预训练；
步骤S4：使用分类标注结果对特征提取模型进行分类任务的训练；
步骤S5：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤S4，直到模型分类任务训练结果达到预期精度，完成分类任务训练。
本申请附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显， 或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1为本申请一些实施例中所提供的一种基于知识驱动的深度学习图像分类方法的流程图；
图2为本申请一些实施例中的基于知识驱动的深度学习图像分类模型的框架示意图；
图3为本申请一些实施例中的基于知识驱动的深度学习图像分类模型的结构图；
图4为本申请一些实施例中的双模态场景下基于知识驱动的深度学习图像分类模型的示例图；
图5为本申请一些实施例中的基于知识驱动的深度学习图像分类模型的训练流程图；
图6为本申请一些实施例中的基于知识驱动的深度学习图像分类模型的使用流程图；
图7为本申请一些实施例中提供的一种基于知识驱动的深度学习图像分类系统的结构示意图。
具体实施方式
下面详细描述本申请的实施例，实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
知识驱动不依赖于训练数据的数据量，通过人为添加先验知识提高模型的学习效率。通过引入先验知识的方式降低模型对数据的依赖程度，为数据集获取或标注困难的领域提供了很好的解决方法，但当前在该领域的研究较少，且大多数研究者将先验知识设计为图像特征的分割，在减少数据量的同时也进一步带来了数据的分割标注压力。
本申请提出的一种基于知识驱动的深度学习图像分类方法，减少了深度学习模型需要的训练数据量，并且不需要引入复杂的分割标注，降低了数据标注的压力。
下面参考附图描述本申请实施例的基于知识驱动的深度学习图像分类方法和系统。
图1为本申请实施例一所提供的一种基于知识驱动的深度学习图像分类方法的流程图。
如图1所示，该基于知识驱动的深度学习图像分类方法包括以下步骤101至步骤103。
步骤101，构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；
步骤102，获取待分类的图像，使用特征提取模块对待分类的图像进行特征提取，得到特征向量；
步骤103，将特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果。
其中，对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，先验知识预训练，包括：步骤S1：使用特征提取模块和先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
本申请实施例的基于知识驱动的深度学习图像分类方法，通过构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；获取待分类的图像，使用特征提取模块对待分类的图像进行特征提取，得到特征向量；将特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，其中，对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，先验知识预训练，包括：步骤S1：使用特征提取模块和先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。由此，能够增强图像特征提取及分类任务的准确度，降低深度学习模型对数据量的依赖程度，缓解部分专业领域的数据收集和标注压力。本申请采用多标签分类方式将先验知识融入到深度学习模型的学习中，有效缓解了分割标注带来的压力，提升了系统在部分专业图像分类领域的可用性。
本申请实施例的基于知识驱动的深度学习图像分类模型的特征提取模块突出图像中具有代表性的特征、先验知识预训练模块以预训练的形式让模型对先验知识进行充分学习、图像分类模块完成图像分类任务、热力图可视化模块提供可解释性。并且本申请创新性地引入了两阶段知识驱动方法，第一阶段对先验知识，即用于分类任务决策的关键图像特征进行训练学习，第二阶段进行图像分类任务的训练。
进一步地，在本申请实施例中，图像分类任务的训练，包括：
步骤一：使用特征提取模块和图像分类模块，并使用分类标注的数据集，对经过先验知识预训练的基于知识驱动的深度学习图像分类模型进行分类任务的训练；
步骤二：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤一，直到模型分类任务训练结果达到预期精度，完成分类任务训练。
进一步地，在本申请实施例中，特征提取模块对输入图像进行变换，以突出图像中具有代表性的特征，如：边缘、角、颜色等。计算机视觉中主流的特征提取模型皆可用于Knowledge_Model(基于知识驱动的深度学习图像分类模型)，如VGGNet、GoogleNet、ResNet等。针对不同的图像分类任务，Knowledge_Model可选择不同的特征提取模型。
使用特征提取模块对待分类的图像进行特征提取，得到特征向量，表示为：
F＝Model baseline(x)
其中，Model baseline为特征提取模型，F为特征向量，x为输入图像。
在执行图像分类任务之前，先借助于先验知识的多标签分类标注结果y k进行基于知识驱 动的深度学习图像分类模型的预训练，通过这种方式让基于知识驱动的深度学习图像分类模型学习先验知识，降低基于知识驱动的深度学习图像分类模型对训练数据量的依赖，提高基于知识驱动的深度学习图像分类模型的学习效率和准确率。
其中，先验知识一般标注为图像中对分类任务起决定性作用的特征表现，例如，使用眼科影像进行眼底疾病辅助诊断的任务中，先验知识可标注为眼底影像中的视网膜出血、玻璃膜疣，或光学相干断层扫描中的视网膜内积液、黄斑区色素上皮脱离等病变体征；使用电磁信号图像进行信号类别分类的任务中，先验知识标注可标注为中心频率、带宽等频带信息，或正交相移键控、正交幅度调制等调制方式。
进一步地，在本申请实施例中，将特征向量输入先验知识预训练模块，具体为使用全连接层获得先验知识
Figure PCTCN2022087216-appb-000007
的得分
Figure PCTCN2022087216-appb-000008
通过选择先验知识得分中得分大于预设阈值的类别得到输入图像的先验知识多标签分类结果，例如，通过选择先验知识得分
Figure PCTCN2022087216-appb-000009
中得分大于0.5的类别来得到输入图像的先验知识多标签分类结果，其中，先验知识得分表示为：
Figure PCTCN2022087216-appb-000010
其中，F为特征向量，W k为全连接层的权重矩阵。
在本申请实施例中，先验知识以多标签分类的形式进行标注，并使用全连接层对特征向量F进行处理。先验知识预训练模块的全连接层与图像分类模块的全连接层不相同。
在进行先验知识预训练的过程中，特征提取模型Model baseline会根据多标签分类结果对权重进行微调，通过这种方式强制基于知识驱动的深度学习图像分类模型学习图像中的特征与先验知识标签的对应关系，帮助基于知识驱动的深度学习图像分类模型更好地关注到对分类任务起决定性作用的图像特征。与此同时，多标签分类标注的难度远小于分割标注，相比于现有的知识驱动方法，有效降低了数据集标注的成本。
在先验知识多标签分类任务训练完成后，使用图像分类任务的标注标签y进行图像分类任务的训练。在这个阶段，舍弃掉先验知识多标签分类模块的全连接层，使用特征提取模块输出的特征向量F，并通过图像分类模块的全连接层实现图像的分类。
进一步地，在本申请实施例中，将特征向量输入图像分类模块，具体为使用全连接层获得图像分类结果
Figure PCTCN2022087216-appb-000011
的得分
Figure PCTCN2022087216-appb-000012
通过选择图像分类结果得分中得分大于预设阈值的类别得到输入图像的图像分类结果，例如，可以通过选择图像分类结果得分
Figure PCTCN2022087216-appb-000013
中得分大于0.5的类别来得到输入图像的分类结果，其中，图像分类结果得分表示为：
Figure PCTCN2022087216-appb-000014
其中，F为特征向量，W c为图像分类全连接层的权重矩阵。
图像分类模块在先验知识多标签分类模块之后参与训练，并对特征提取模型Model baseline进行复用。Model baseline在预训练过程中已经对先验知识进行了有效的学习，在图像分类任务中可以更好地关注到对分类结果起决定性作用的图像特征，图像分类的训练可以更快地收敛，并能取得比直接训练图像分类模型更好的分类准确率。
为了更好地理解Knowledge_Model对于输入图像的关注区域，对图像分类任务在视觉上进行可视化解释，本申请对图像分类结果进行热力图可视化。
进一步地，在本申请实施例中，基于知识驱动的深度学习图像分类模型还包括热力图可视化模块，使用热力图可视化模块对图像分类结果进行热力图可视化，得到热力图，包括以下步骤：
令
Figure PCTCN2022087216-appb-000015
为Knowledge_Model判断输入图像x是否为分类种类c时重点关注的图像区域热力图，首先使用图像分类结果类c对应分数
Figure PCTCN2022087216-appb-000016
反向传播至特征提取模型Model baseline的最后一层卷积层A，回流的梯度在宽度和高度维度上全局平均池化，获得特征图重要性权重w c，表示为：
Figure PCTCN2022087216-appb-000017
其中，h为最后一层特征图的高度，w为最后一层特征图的宽度，Z＝h×w，
Figure PCTCN2022087216-appb-000018
表示图像分类结果得分，A表示特征提取模型的最后一层卷积层，A k ij表示最后一层卷积层在通道为k、高为i、宽为j处的值，
特征图重要性权重捕获了最后一个卷积层特征图的通道k对于目标类别c的影响程度，将得到的特征图重要性权重与特征图激活值加权组合，然后通过RELU激活函数来获得热力图，表示为：
Figure PCTCN2022087216-appb-000019
其中，RELU()表示RELU激活函数，A k表示特征提取模型的最后一层卷积层在通道为k的矩阵，
Figure PCTCN2022087216-appb-000020
表示特征图重要性权重。
图2为本申请实施例的基于知识驱动的深度学习图像分类模型的框架示意图。
如图2所示，该基于知识驱动的深度学习图像分类模型，包括特征提取模块、先验知识预训练模块、图像分类模块，其中，特征提取模块，用于使用特征提取模型对待分类的图像进行特征提取，得到特征向量；先验知识预训练模块，用于将特征向量应用于先验知识预训练，得到先验知识多标签分类结果；图像分类模块，用于将特征向量应用于图像分类，得到图像分类结果。该基于知识驱动的深度学习图像分类模型还包括热力图可视化模块，用于对图像分类结果进行热力图可视化，得到热力图。
图3为本申请实施例的基于知识驱动的深度学习图像分类模型的结构图。
如图3所示，定义数据集D＝{x|y k,y}，其中x为输入模型的图像数据，y k为x的先验知识标签，且以多标签形式标注，y为x的图像分类标签，Model baseline为特征提取模型。基于知识驱动的深度学习图像分类模型接收输入x，输出对图像的分类结果
Figure PCTCN2022087216-appb-000021
以及做出分类所参考的关键特征多标签分类结果
Figure PCTCN2022087216-appb-000022
可表示为：
Figure PCTCN2022087216-appb-000023
其中，基于知识驱动的深度学习图像分类模型以“Knowledge_Model”表示。图像特征提取模块的输入为图像数据x，输出为特征提取模型提取得到的特征向量F。先验知识预训练模块的输入为特征向量F，输出为先验知识
Figure PCTCN2022087216-appb-000024
的得分
Figure PCTCN2022087216-appb-000025
通过选择
Figure PCTCN2022087216-appb-000026
中得分大于预设阈值的类别来得到输入图像的先验知识多标签分类结果
Figure PCTCN2022087216-appb-000027
在先验知识多标签分类任务训练完成后，使用图像分类任务的标注标签y进行图像分类任务的训练。在这个阶段，舍弃掉先验知识多标签分类模块的全连接层，使用特征提取模块输出的特征向量F，并通过图像分类模块的全连接层实现图像的 分类。图像分类模块的输入是特征向量F，输出是分类结果
Figure PCTCN2022087216-appb-000028
的得分
Figure PCTCN2022087216-appb-000029
通过选择
Figure PCTCN2022087216-appb-000030
中得分大于预设阈值的类别来得到输入图像的分类结果
Figure PCTCN2022087216-appb-000031
热力图可视化模块，利用图像分类结果得分
Figure PCTCN2022087216-appb-000032
反向传播至特征提取模型的最后一层卷积层，回流的梯度在宽度和高度维度上全局平均池化，获得特征图重要性权重，与特征图激活值加权组合，然后通过RELU激活函数来获得热力图。
在一些需要多模态图像进行分类的场景下，基于知识驱动的深度学习图像分类模型也同样适用。例如：综合多种医学影像进行疾病诊断的场景。
假设存在n种模态的图像，n≥1，则需要为每一种模态分别构建特征提取模块和先验知识预训练模块，整个模型只需要构建1个图像分类模块：
n个特征提取模块：共获取到n个特征向量{F 1,F 2,…,F n}，其中任意F i∈R N，i∈{1,2,…,n}。特征提取模型相互独立，权重不共享。
n个先验知识预训练模块：不同模态图像的先验知识不相同，需根据具体的标注内容构建先验知识预训练模块，实现该模态图像的多标签分类任务。通过这种方式使每个特征提取模型更好地关注到对应模态的输入对分类任务起决定性作用的图像特征。
1个图像分类模块：将n个特征向量拼接形成融合多模态的特征向量F con∈R n×N，图像分类模块以融合特征向量F con作为输入，并输出对图像分类结果
Figure PCTCN2022087216-appb-000033
的得分
Figure PCTCN2022087216-appb-000034
通过特征融合的方式对n个输入模态进行综合分析，并给出图像分类任务的最终结果。
图4为本申请实施例的双模态场景下基于知识驱动的深度学习图像分类模型的示例图。
如图4所示，在该双模态场景下基于知识驱动的深度学习图像分类模型的示例中，眼底影像和OCT影像分别进行图像特征提取和先验知识预训练，在先验知识被充分学习后，融合两个模态图像的特征向量F 1和F 2得到F con，并执行图像分类任务。
图5为本申请实施例的基于知识驱动的深度学习图像分类模型的训练流程图。
如图5所示，首先根据数据集及任务特点选择合适的特征提取模型Model baseline；之后使用特征提取模块和先验知识预训练模块，并使用先验知识标注的数据集进行先验知识预训练；判断模型预训练结果是否达到预期精度，若模型预训练结果未达到预期精度，则调整超参数或者特征提取模型Model baseline，再次使用先验知识标注数据集进行预训练，直到预训练结果达到预期精度；若预训练结果达到预期精度，则使用特征提取模块和图像分类模块，并使用分类标注数据集对基于知识驱动的深度学习图像分类模型进行分类任务的训练；判断基于知识驱动的深度学习图像分类模型分类任务训练结果是否达到预期精度，若基于知识驱动的深度学习图像分类模型分类任务训练结果未达到预期精度，则调整超参数，再次使用分类标注数据集对基于知识驱动的深度学习图像分类模型进行分类任务的训练，直到基于知识驱动的深度学习图像分类模型分类任务训练结果达到预期精度；若基于知识驱动的深度学习图像分类模型分类任务训练结果达到预期精度，则流程结束。
图6为本申请实施例的基于知识驱动的深度学习图像分类模型的使用流程图。
如图6所示，获取待分类的图像并上传至基于知识驱动的深度学习图像分类模型；基于知识驱动的深度学习图像分类模型的特征提取模块对图像进行特征提取，得到特征向量；分别将特征向量应用于先验知识预训练模块和图像分类模块；得到模型分类的依据(先验知识多标签分类结果)和模型分类结果(分类任务结果)，对图像分类结果进行热力图可视化，得到模型分类时对输入图像的关注区域热力图。
图7为本申请实施例二所提供的一种基于知识驱动的深度学习图像分类系统的结构示意图。
如图7所示，该基于知识驱动的深度学习图像分类系统，包括获取模块和基于知识驱动的深度学习图像分类模型，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块，其中，
获取模块10，用于获取待分类的图像并输入基于知识驱动的深度学习图像分类模型中；
特征提取模块20，用于使用特征提取模型对待分类的图像进行特征提取，得到特征向量；
先验知识预训练模块30，用于将特征向量应用于先验知识预训练，得到先验知识多标签分类结果；
图像分类模块40，用于将特征向量应用于图像分类，得到图像分类结果，
该基于知识驱动的深度学习图像分类模型还包括热力图可视化模块50，用于对图像分类结果进行热力图可视化，得到热力图。
进一步地，在本申请实施例中，还包括，对基于知识驱动的深度学习图像分类模型进行训练，包括：
步骤S1：根据数据集及任务特点选择合适的特征提取模型；
步骤S2：使用先验知识标注结果对特征提取模型进行先验知识预训练；
步骤S3：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S2，直到预训练结果达到预期精度，完成先验知识预训练；
步骤S4：使用分类标注结果对特征提取模型进行分类任务的训练；
步骤S5：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤S4，直到模型分类任务训练结果达到预期精度，完成分类任务训练。
本申请实施例的基于知识驱动的深度学习图像分类系统，包括获取模块和基于知识驱动的深度学习图像分类模型，基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块，其中，获取模块，用于获取待分类的图像并输入基于知识驱动的深度学习图像分类模型中；特征提取模块，用于使用特征提取模型对待分类的图像进行特征提取，得到特征向量；先验知识预训练模块，用于将特征向量应用于先验知识预训练，得到先验知识多标签分类结果；图像分类模块，用于将特征向量应用于图像分类，得到图像分类结果。
由此，能够增强图像特征提取及分类任务的准确度，降低深度学习模型对数据量的依赖 程度，缓解部分专业领域的数据收集和标注压力。并且本申请采用多标签分类方式将先验知识融入到深度学习模型的学习中，有效缓解了分割标注带来的压力，提升了系统在部分专业图像分类领域的可用性。
本申请实施例的基于知识驱动的深度学习图像分类模型的特征提取模块突出图像中具有代表性的特征、先验知识预训练模块以预训练的形式让模型对先验知识进行充分学习、图像分类模块完成图像分类任务、热力图可视化模块提供可解释性。并且本申请创新性地引入了两阶段知识驱动方法，第一阶段对先验知识，即用于分类任务决策的关键图像特征进行训练学习，第二阶段进行图像分类任务的训练。
本申请实施例的基于知识驱动的深度学习图像分类方法和系统，能够增强图像特征提取及分类任务的准确度，降低深度学习模型对数据量的依赖程度，缓解部分专业领域的数据收集和标注压力。并且本申请采用多标签分类方式将先验知识融入到深度学习模型的学习中，有效缓解了分割标注带来的压力，提升了系统在部分专业图像分类领域的可用性。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现定制逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本申请的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本申请的实施例所属技术领域的技术人员所理解。
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(电子装置)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)， 可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。如，如果用硬件来实现和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
此外，在本申请各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。
上述提到的存储介质可以是只读存储器，磁盘或光盘等。尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (12)

  1. 一种基于知识驱动的深度学习图像分类方法，其特征在于，包括以下步骤：
    构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，所述基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；
    获取待分类的图像，使用所述特征提取模块对所述待分类的图像进行特征提取，得到特征向量；
    将所述特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，
    其中，所述对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，
    所述先验知识预训练，包括：
    步骤S1：使用所述特征提取模块和所述先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；
    步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
  2. 如权利要求1所述的方法，其特征在于，所述图像分类任务的训练，包括：
    步骤一：使用所述特征提取模块和所述图像分类模块，并使用分类标注的数据集，对经过先验知识预训练的基于知识驱动的深度学习图像分类模型进行分类任务的训练；
    步骤二：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤一，直到模型分类任务训练结果达到预期精度，完成分类任务训练。
  3. 如权利要求1所述的方法，其特征在于，所述使用所述特征提取模块对所述待分类的图像进行特征提取，得到特征向量，表示为：
    F＝Model baseline(x)
    其中，Model baseline为特征提取模型，F为特征向量，x为输入图像。
  4. 如权利要求1所述的方法，其特征在于，将所述特征向量输入先验知识预训练模块，具体为使用全连接层得到先验知识得分，通过选择所述先验知识得分中得分大于预设阈值的类别得到输入图像的先验知识多标签分类结果，其中，所述先验知识得分表示为：
    Figure PCTCN2022087216-appb-100001
    其中，F为特征向量，W k为全连接层的权重矩阵。
  5. 如权利要求1所述得方法，其特征在于，将所述特征向量输入图像分类模块，具体为使用全连接层得到图像分类结果得分，通过选择所述图像分类结果得分中得分大于预设阈值的类别得到输入图像的图像分类结果，其中，所述图像分类结果得分表示为：
    Figure PCTCN2022087216-appb-100002
    其中，F为特征向量，W c为图像分类全连接层的权重矩阵。
  6. 如权利要求3或5所述的方法，其特征在于，所述基于知识驱动的深度学习图像分 类模型还包括热力图可视化模块，使用所述热力图可视化模块对图像分类结果进行热力图可视化，得到热力图，包括以下步骤：
    使用所述图像分类结果得分，反向传播至所述特征提取模型的最后一层卷积层，回流的梯度在宽度和高度维度上全局平均池化，获得特征图重要性权重；
    将得到的特征图重要性权重与特征图激活值加权组合，然后通过RELU激活函数来获得热力图，
    其中，所述特征图重要性权重表示为：
    Figure PCTCN2022087216-appb-100003
    其中，h为最后一层特征图的高度，w为最后一层特征图的宽度，Z＝h×w，
    Figure PCTCN2022087216-appb-100004
    表示所述图像分类结果得分，A表示特征提取模型的最后一层卷积层，A k ij表示最后一层卷积层在通道为k、高为i、宽为j处的值，
    所述热力图表示为：
    Figure PCTCN2022087216-appb-100005
    其中，RELU()表示RELU激活函数，A k表示特征提取模型的最后一层卷积层在通道为k的矩阵，
    Figure PCTCN2022087216-appb-100006
    表示所述特征图重要性权重。
  7. 一种基于知识驱动的深度学习图像分类系统，其特征在于，包括获取模块和基于知识驱动的深度学习图像分类模型，所述基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块，其中，
    所述获取模块，用于获取待分类的图像并输入所述基于知识驱动的深度学习图像分类模型中；
    所述特征提取模块，用于使用特征提取模型对待分类的图像进行特征提取，得到特征向量；
    所述先验知识预训练模块，用于将所述特征向量应用于先验知识预训练，得到先验知识多标签分类结果；
    所述图像分类模块，用于将所述特征向量应用于图像分类，得到图像分类结果。
  8. 如权利要求7所述的系统，其特征在于，所述基于知识驱动的深度学习图像分类模型还包括热力图可视化模块，用于对图像分类结果进行热力图可视化，得到热力图。
  9. 如权利要求7所述的系统，其特征在于，还包括，对所述基于知识驱动的深度学习图像分类模型进行训练，包括：
    步骤S1：根据数据集及任务特点选择合适的特征提取模型；
    步骤S2：使用先验知识标注结果对特征提取模型进行先验知识预训练；
    步骤S3：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S2，直到预训练结果达到预期精度，完成先验知识预训练；
    步骤S4：使用分类标注结果对特征提取模型进行分类任务的训练；
    步骤S5：若模型分类任务训练结果未达到预期精度，调整超参数，重复进行步骤S4， 直到模型分类任务训练结果达到预期精度，完成分类任务训练。
  10. 一种电子设备，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行以下步骤：
    构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，所述基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；
    获取待分类的图像，使用所述特征提取模块对所述待分类的图像进行特征提取，得到特征向量；
    将所述特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，
    其中，所述对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，
    所述先验知识预训练，包括：
    步骤S1：使用所述特征提取模块和所述先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；
    步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
  11. 一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行以下步骤：
    构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，所述基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；
    获取待分类的图像，使用所述特征提取模块对所述待分类的图像进行特征提取，得到特征向量；
    将所述特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，
    其中，所述对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，
    所述先验知识预训练，包括：
    步骤S1：使用所述特征提取模块和所述先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；
    步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
  12. 一种计算机程序产品，包括计算机程序，所述计算机程序在被处理器执行时实现以下步骤：
    构建基于知识驱动的深度学习图像分类模型，并对构建的模型进行训练，其中，所述基于知识驱动的深度学习图像分类模型包括特征提取模块、先验知识预训练模块、图像分类模块；
    获取待分类的图像，使用所述特征提取模块对所述待分类的图像进行特征提取，得到特征向量；
    将所述特征向量分别输入先验知识预训练模块和图像分类模块，得到先验知识多标签分类结果和图像分类结果，
    其中，所述对构建的模型进行训练包括先验知识预训练和图像分类任务的训练，训练使用的数据集的标注包括先验知识标注和分类标注，
    所述先验知识预训练，包括：
    步骤S1：使用所述特征提取模块和所述先验知识预训练模块，并使用先验知识标注的数据集进行训练，对特征提取模型的权重进行微调；
    步骤S2：若预训练结果未达到预期精度，调整超参数或者特征提取模型，重复进行步骤S1，直到预训练结果达到预期精度，完成先验知识预训练。
PCT/CN2022/087216 2021-12-14 2022-04-15 基于知识驱动的深度学习图像分类方法和系统 WO2023108968A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111531480.0 2021-12-14
CN202111531480.0A CN114266920A (zh) 2021-12-14 2021-12-14 基于知识驱动的深度学习图像分类方法和系统

Publications (1)

Publication Number Publication Date
WO2023108968A1 true WO2023108968A1 (zh) 2023-06-22

Family

ID=80827252

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/087216 WO2023108968A1 (zh) 2021-12-14 2022-04-15 基于知识驱动的深度学习图像分类方法和系统

Country Status (2)

Country Link
CN (1) CN114266920A (zh)
WO (1) WO2023108968A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114266920A (zh) * 2021-12-14 2022-04-01 北京邮电大学 基于知识驱动的深度学习图像分类方法和系统
CN117272134A (zh) * 2023-09-01 2023-12-22 中国地质大学（武汉） 深度学习模型、海底地貌分类模型构建方法及分类方法

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040052328A1 (en) * 2002-09-13 2004-03-18 Sabol John M. Computer assisted analysis of tomographic mammography data
CN108665901A (zh) * 2018-05-04 2018-10-16 广州国音科技有限公司 一种音素/音节提取方法及装置
CN109934261A (zh) * 2019-01-31 2019-06-25 中山大学 一种知识驱动参数传播模型及其少样本学习方法
CN111028153A (zh) * 2019-12-09 2020-04-17 南京理工大学 图像处理和神经网络训练方法、装置及计算机设备
CN111429421A (zh) * 2020-03-19 2020-07-17 北京推想科技有限公司 模型生成方法、医学图像分割方法、装置、设备及介质
CN112560668A (zh) * 2020-12-14 2021-03-26 南京航空航天大学 一种基于场景先验知识的人体行为识别方法
CN113781465A (zh) * 2021-09-18 2021-12-10 长春理工大学 基于Grad-CAM的医学图像分割模型可视化方法
CN114266920A (zh) * 2021-12-14 2022-04-01 北京邮电大学 基于知识驱动的深度学习图像分类方法和系统

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040052328A1 (en) * 2002-09-13 2004-03-18 Sabol John M. Computer assisted analysis of tomographic mammography data
CN108665901A (zh) * 2018-05-04 2018-10-16 广州国音科技有限公司 一种音素/音节提取方法及装置
CN109934261A (zh) * 2019-01-31 2019-06-25 中山大学 一种知识驱动参数传播模型及其少样本学习方法
CN111028153A (zh) * 2019-12-09 2020-04-17 南京理工大学 图像处理和神经网络训练方法、装置及计算机设备
CN111429421A (zh) * 2020-03-19 2020-07-17 北京推想科技有限公司 模型生成方法、医学图像分割方法、装置、设备及介质
CN112560668A (zh) * 2020-12-14 2021-03-26 南京航空航天大学 一种基于场景先验知识的人体行为识别方法
CN113781465A (zh) * 2021-09-18 2021-12-10 长春理工大学 基于Grad-CAM的医学图像分割模型可视化方法
CN114266920A (zh) * 2021-12-14 2022-04-01 北京邮电大学 基于知识驱动的深度学习图像分类方法和系统

Also Published As

Publication number Publication date
CN114266920A (zh) 2022-04-01

Similar Documents

Publication Publication Date Title
Fan et al. Adversarial learning for mono-or multi-modal registration
JP7391846B2 (ja) ディープニューラルネットワークを使用したコンピュータ支援診断
Zhu et al. Adversarial deep structural networks for mammographic mass segmentation
Hu et al. Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets
Tareef et al. Optimizing the cervix cytological examination based on deep learning and dynamic shape modeling
WO2023108968A1 (zh) 基于知识驱动的深度学习图像分类方法和系统
CN110674866A (zh) 迁移学习特征金字塔网络对X-ray乳腺病灶图像检测方法
Shi et al. Bayesian voxdrn: A probabilistic deep voxelwise dilated residual network for whole heart segmentation from 3d mr images
Ogiela et al. Natural user interfaces in medical image analysis
Wen et al. Review of research on the instance segmentation of cell images
Barrowclough et al. Binary segmentation of medical images using implicit spline representations and deep learning
Tian et al. Interactive prostate MR image segmentation based on ConvLSTMs and GGNN
Kondratenko et al. Artificial neural networks for recognition of brain tumors on MRI images
Abid et al. Multi-modal medical image classification using deep residual network and genetic algorithm
Khattar et al. Computer assisted diagnosis of skin cancer: a survey and future recommendations
Tang et al. Lesion segmentation and RECIST diameter prediction via click-driven attention and dual-path connection
Chen et al. Residual block based nested U-type architecture for multi-modal brain tumor image segmentation
Gopikha et al. Regularised Layerwise Weight Norm Based Skin Lesion Features Extraction and Classification.
Chang Knowledge-guided data-centric AI in healthcare: progress, shortcomings, and future directions
Chatterjee et al. A survey on techniques used in medical imaging processing
Ullah et al. DSFMA: Deeply supervised fully convolutional neural networks based on multi-level aggregation for saliency detection
Kaur et al. Deep CNN-based method for segmenting lung fields in digital chest radiographs
Shin et al. Three aspects on using convolutional neural networks for computer-aided detection in medical imaging
Sun et al. MetaSeg: A survey of meta-learning for image segmentation
Li et al. Segmentation evaluation with sparse ground truth data: Simulating true segmentations as perfect/imperfect as those generated by humans

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22905736

Country of ref document: EP

Kind code of ref document: A1