WO2025007941A1 - 图像分类方法、训练图像分类模型的方法及装置 - Google Patents

图像分类方法、训练图像分类模型的方法及装置 Download PDF

Info

Publication number
WO2025007941A1
WO2025007941A1 PCT/CN2024/103726 CN2024103726W WO2025007941A1 WO 2025007941 A1 WO2025007941 A1 WO 2025007941A1 CN 2024103726 W CN2024103726 W CN 2024103726W WO 2025007941 A1 WO2025007941 A1 WO 2025007941A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
representations
classification
multiple cluster
feature representation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/103726
Other languages
English (en)
French (fr)
Inventor
夏英达
袁铭泽
张灵
姚佳文
裘茗烟
董和鑫
周靖人
吕乐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Publication of WO2025007941A1 publication Critical patent/WO2025007941A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the present application relates to the field of computer vision technology, and in particular to a method for image classification, a method for training an image classification model, and a device.
  • the present application provides a method for image classification, a method and a device for training an image classification model, so as to improve the accuracy of image classification results.
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.
  • the method further includes:
  • the image to be classified is segmented using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category.
  • feature extraction is performed on the image to be classified to obtain a feature representation of the image, which includes:
  • Feature extraction is performed on the image to be classified to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution is used as the feature representation of the image.
  • performing cross-attention processing on initial representations of multiple cluster centers obtained by pre-training using feature representations of the image includes:
  • the query matrix is obtained by using the initial representation of multiple cluster centers, and the query matrix is input into a multi-layer serialized transformer network.
  • Each layer of the transformer network corresponds to each resolution in order from low to high resolution.
  • Each layer of the transformer network uses the feature representation of each token at the corresponding resolution to obtain the key matrix and value matrix, and performs cross-attention processing on the query matrix input to the transformer network of this layer to obtain the query matrix output by the transformer network of this layer.
  • the query matrix output by the last layer of the Transformer network is used to obtain the multiple cluster center representations.
  • using the multiple cluster center representations and the category assignment matrix to perform classification includes:
  • the cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image to be classified belongs to the target category.
  • a method for training an image classification model comprising:
  • Acquire training data including a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category;
  • the training data is used to train an image classification model
  • the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.
  • the training sample further includes a region mask of a preset category annotated on the image sample;
  • the image classification model further includes a segmentation network;
  • the segmentation network segments the image sample using the category assignment matrix to obtain image regions of preset categories, where the preset categories include the target category;
  • the training goal also includes: minimizing the difference between the image region of the preset category and the corresponding region mask.
  • the feature extraction network performs feature extraction on the image sample.
  • the step of obtaining the feature representation of the image sample includes: extracting features from the image sample, sequentially obtaining feature representations of each element Token at multiple resolutions, and using the feature representation of each Token at the highest resolution as the feature representation of the image sample;
  • the first decoding network includes multiple layers of serially connected Transformer networks.
  • the initial representations of the multiple cluster centers are input into the first layer of Transformer networks as query matrices.
  • Each layer of Transformer networks corresponds to each resolution one by one in order from low to high resolution.
  • Each layer of Transformer networks uses the feature representations of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the current layer of Transformer networks to obtain the query matrix output by the current layer of Transformer networks.
  • the query matrix output by the last layer of Transformer networks is used to obtain the multiple cluster center representations.
  • the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification, including:
  • the classification network averages the multiple cluster center representations to obtain a cluster average representation
  • the cluster average representation and the cluster pooling representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image sample belongs to the target category.
  • an image classification method which is executed by a cloud server, and the method includes:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • the classification result is returned to the user terminal.
  • a method for computer-aided diagnosis of cancer comprising:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.
  • a fifth aspect provides a computer-aided diagnosis method for gastric cancer, the method comprising:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to gastric cancer.
  • a computer-aided diagnosis system for cancer comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform a computer-aided diagnosis method for cancer, the method comprising:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.
  • an image classification device comprising:
  • An image acquisition unit configured to acquire an image to be classified
  • a feature extraction unit configured to extract features from the image to be classified to obtain a feature representation of the image
  • a first decoding unit is configured to perform cross-attention processing on initial representations of multiple cluster centers obtained by pre-training using feature representations of the image to obtain multiple cluster center representations;
  • a second decoding unit is configured to perform decoding processing using the feature representation of the image and the plurality of cluster center representations to obtain a category assignment matrix
  • the image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the image to be classified belongs to a target category.
  • a device for training an image classification model comprising:
  • a sample acquisition unit is configured to acquire training data including a plurality of training samples, wherein the training samples include image samples and labels indicating whether the image samples belong to a target category;
  • a model training unit is configured to train an image classification model using the training data, wherein the image classification model includes: a feature extraction network, a first decoding network, a second decoding network, and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on initial representations of multiple cluster centers to obtain multiple cluster center representations; The second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.
  • an image classification device comprising:
  • An image acquisition unit configured to acquire a medical image for a target organ
  • a feature extraction unit configured to extract features from the medical image to obtain a feature representation of the medical image
  • a first decoding unit is configured to perform cross-attention processing on initial representations of multiple cluster centers obtained by pre-training using the feature representation of the medical image to obtain multiple cluster center representations;
  • a second decoding unit is configured to perform decoding processing using the feature representation of the medical image and the plurality of cluster center representations to obtain a category assignment matrix
  • the image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.
  • a computer readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first to fifth aspects are implemented.
  • an electronic device comprising:
  • a memory associated with the one or more processors the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first to fifth aspects above.
  • a computer program is provided.
  • the computer program is executed in a computer, the computer is caused to execute the steps of any one of the methods described in the first to fifth aspects above.
  • a computer program product comprising a computer program, which, when executed in a computer, enables the computer to execute the steps of any one of the methods described in the first to fifth aspects above.
  • the present application decodes the high-level semantics contained in the multiple cluster centers obtained through pre-training together with the feature representation of the image to obtain a category assignment matrix, so that the category assignment matrix can reflect the image features associated with the cluster centers, that is, match the image features to the corresponding cluster centers, thereby making the image classification results obtained based on the cluster center representation and the category assignment matrix have higher accuracy and recall rate.
  • the present application can further utilize the category assignment matrix to segment the image to be classified to obtain image regions of preset categories, thereby providing an interpretable reference for image classification.
  • the image texture can be perceived from multiple scales in the process of obtaining multiple cluster center representations, and the feature representations of multiple cluster centers are redistributed with the feature representation of the image, so that the classification process and segmentation
  • the process is both sensitive to local semantics and globally aware, further improving the accuracy and recall of image classification and image segmentation.
  • the difference between the image area of the preset category obtained by segmenting the image sample using the segmentation network and the area mask of the preset category annotated for the image sample can be used to assist in the training of the image classification model, thereby further improving the effect and performance of the image classification model.
  • FIG1 is a diagram of a system architecture applicable to an embodiment of the present application.
  • FIG2 is a flow chart of an image classification method provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of the principle of an image classification model provided in an embodiment of the present application.
  • FIG4 is a flow chart of a method for training an image classification model provided in an embodiment of the present application.
  • FIG5 is a schematic diagram of the principle of a training image classification model provided in an embodiment of the present application.
  • FIG6 is a flow chart of an image classification method applied to the medical field provided by an embodiment of the present application.
  • FIG7 is a schematic block diagram of an image classification device provided in an embodiment of the present application.
  • FIG8 is a schematic block diagram of a device for training an image classification model provided in an embodiment of the present application.
  • FIG. 9 is a schematic block diagram of an electronic device provided in an embodiment of the present application.
  • the word “if” as used herein may be interpreted as “when” or “when"
  • the phrases “if it is determined” or “if (stated condition or event) is detected” may be interpreted as “when it is determined” or “in response to the determination” or “when detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.
  • FIG1 shows an exemplary system architecture to which the present application embodiment can be applied.
  • the system architecture includes a model training device and an image classification device on the server side, and may also include a user terminal.
  • the model training device is used to perform model training in an offline stage. That is, after obtaining training data, the method provided in the embodiment of the present application can be used to perform model training to obtain an image classification model.
  • the image classification device is used to classify the image to be classified online using the trained image classification model to obtain a classification result of whether the image belongs to the target category.
  • the model training device and the image classification device can be set as independent servers, or can be set on the same server or server group, or can be set on independent or the same cloud server.
  • Cloud server also known as cloud computing server or cloud host, is a host product in the cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical host and virtual private server (VPs, Virtual Private Server) services.
  • the model training device and the image segmentation device can also be set on a computer terminal with strong computing power.
  • the user can send the image to be classified to the image classification device at the server through the network through the user terminal.
  • the image classification device classifies the image to be classified using the method provided in the embodiment of the present application, the classification result is returned to the user terminal.
  • the user terminals mentioned above may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, smart medical devices, PCs (Personal Computers), etc.
  • smart mobile devices may include, for example, mobile phones, tablet computers, laptop computers, PDAs (Personal Digital Assistants), Internet cars, etc.
  • Smart home devices may include, for example, smart TVs, smart refrigerators, etc.
  • Wearable devices may include, for example, smart watches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality), mixed reality devices (i.e., devices that can support both virtual reality and augmented reality), etc.
  • the above-mentioned image classification device may also perform image classification in an offline manner, for example, performing image classification on batches of images to be classified respectively.
  • model training devices image classification devices, image classification models and user terminals in Figure 1 is only illustrative. According to implementation requirements, there may be any number of model training devices, image classification devices, image classification models and user terminals.
  • first and second in the present disclosure do not have limitations on size, order, quantity, etc., and are only used to distinguish them in name.
  • first image segmentation model and “second image segmentation model” It is used to distinguish two speech sounds in terms of name.
  • first segmentation result and “second segmentation result” are used to distinguish two segmentation results in terms of name.
  • first target and “second target” are used to distinguish two targets in terms of name. And so on.
  • FIG2 is a flow chart of an image classification method provided in an embodiment of the present application, which can be performed by an image classification device in the system shown in FIG1. As shown in FIG2, the method can include the following steps:
  • Step 202 Obtain an image to be classified.
  • Step 204 extract features from the image to be classified to obtain feature representation of the image.
  • Step 208 Perform decoding processing using the feature representation of the image and multiple cluster center representations to obtain a category assignment matrix.
  • Step 210 Classification is performed using multiple cluster center representations and category assignment matrices to obtain a classification result of whether the image to be classified belongs to the target category.
  • the high-level semantics contained in the multiple cluster centers obtained through pre-training in this application are decoded together with the feature representation of the image to obtain a category assignment matrix, so that the category assignment matrix can reflect the image features associated with the cluster centers, that is, match the image features to the corresponding cluster centers, thereby making the image classification results obtained based on the cluster center representation and the category assignment matrix have a higher accuracy.
  • step 202 ie, "obtaining an image to be classified" is described in detail.
  • the image to be classified in this application may be a two-dimensional image or a three-dimensional image, a grayscale image or a color image.
  • the images to be classified may contain different contents.
  • the images to be classified are usually images containing traffic elements such as vehicles, pedestrians, roads, and traffic facilities.
  • the purpose of image classification may be to determine whether the image belongs to a specific traffic scene.
  • the images to be classified are usually images containing product information.
  • the purpose of image classification may be to determine whether the image belongs to a specific product category.
  • images are usually medical images, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound examination images, etc., containing organs such as lungs, liver, pancreas, and colon.
  • CT Computer Tomography
  • MRI Magnetic Resonance Imaging
  • ultrasound examination images etc.
  • organs such as lungs, liver, pancreas, and colon.
  • the purpose of image classification may be to determine whether the image belongs to a specific category of organ area, or whether there is a certain specific category of abnormality.
  • Steps 204 to 210 in the above process can be implemented by a pre-trained image classification model.
  • the image classification model provided by the embodiment of the present application can include: a feature extraction network, a first decoding network, a second decoding network and a classification network.
  • the above step 204 namely "extracting features from the image to be classified to obtain feature representation of the image", can be performed by the above feature extraction network.
  • the feature extraction network can be implemented based on the Transformer network to extract image features and obtain The feature representation of each token (element) in the classified image.
  • the feature extraction network can adopt VIT (Vision Transformer), RAN (Residual Attention Network), U-Net (U-Net is a variant of FCN (Fully Convolutional Network), which was proposed to solve the problem of biomedical imaging. Due to its good effect, it was widely used in various fields of image segmentation).
  • Each token of an image refers to the elements that make up the image.
  • the blocks in the image and the start of the sequence are all tokens.
  • the blocks can be composed of one or more pixels.
  • the blocks can be composed of one or more voxels.
  • a network such as U-Net can be used as a feature extraction network to obtain feature representations of each Token at multiple resolutions, and the accuracy of image classification can be improved through such multi-scale features.
  • U-Net adopts an Encoder-Decoder architecture, where the Encoder performs downsampling, and the Decoder performs upsampling and feature splicing to obtain feature representations of multiple different resolutions, that is, the feature representation of each Token at a low resolution is first obtained, and then the feature representation of each Token at a higher resolution is obtained in turn. The feature representation of each Token at the highest resolution is then used as the feature representation of the image.
  • the feature representation F of the image is obtained through the feature extraction network, F ⁇ RA ⁇ HWD .
  • A is the feature dimension
  • H, W and D represent the height, width and length of the three-dimensional image (taking the input image to be classified as a three-dimensional image as an example).
  • the above step 206 i.e., "using the feature representation of the image to perform cross-attention processing on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations" can be executed by the first decoding network in the image classification model.
  • the first decoding network actually transforms the initial representation of a set of cluster centers into a cluster center representation that integrates the image semantics through the cross-attention of the feature representation of the image.
  • the first decoding network may include a multi-layer series of Transformer networks.
  • the query matrix can be obtained using the initial representations of multiple cluster centers, and the query matrix is input into the multi-layer series of transformation Transformer networks.
  • Each layer of Transformer networks corresponds to each resolution in order from low to high resolution.
  • Each layer of Transformer networks uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input to the current layer of Transformer network to obtain the query matrix output by the current layer of Transformer network; the query matrix output by the last layer of Transformer network is used to obtain multiple cluster center representations.
  • Cn represents the matrix corresponding to the multiple cluster centers output by the nth layer Transformer network
  • Cn -1 represents the matrix corresponding to the multiple cluster centers output by the n-1th layer Transformer network
  • its input Cn -1 is the matrix C initial corresponding to the initial representation of multiple cluster centers.
  • Q is obtained from Cn -1
  • K and V are obtained from the feature representation of each token at the corresponding resolution of the Transformer network of this layer.
  • T represents the transposition process.
  • N is the number of cluster centers.
  • the first decoder is equivalent to grouping the tokens in the image according to the initial cluster centers.
  • the cross-attention process is similar to the K-means clustering algorithm, and the argmax process is used to replace the softmax process in the traditional Transformer network.
  • the matrix corresponding to the multiple cluster center representations obtained in the end is represented as C ⁇ R N ⁇ A .
  • the above step 208 namely "using the feature representation of the image and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix" can be executed by the above second decoding network.
  • This step actually redistributes the feature representation of each token based on N cluster centers and assigns them to different clusters. Therefore, the result of this step is called the category assignment matrix.
  • Softmax N represents the Softmax (normalized exponential function) processing performed on the N cluster center dimensions.
  • the above step 210 namely "using multiple cluster center representations and category assignment matrices to perform classification to obtain a classification result of whether the image to be classified belongs to the target category", can be performed by the classification network in the image classification model shown in FIG. 3 .
  • the learned multiple cluster center representations have high-level semantics and reflect the differences between clusters and the similarities within clusters, combining multiple cluster center representations with the category assignment matrix for classification can significantly improve the classification accuracy and recall rate compared to directly using the feature representation of the image for classification.
  • multiple cluster center representations can be averaged to obtain a cluster average representation; and the category assignment matrix can be pooled to obtain a cluster pooled representation; then the cluster average representation and the cluster pooled representation are integrated, and the integrated feature representation is used for classification to obtain a classification result of whether the image to be classified belongs to the target category.
  • the cluster average representation can be obtained by averaging multiple cluster center representations in the channel dimension.
  • other processing methods can also be used to convert the matrix C into a vector.
  • the cluster average representation is
  • cluster average representation and cluster pooling representation can be spliced, as shown in Figure 3, and After splicing, it passes through several layers of MLP (Multilayer Perceptron) to obtain the classification result of whether the image to be classified belongs to the target category.
  • MLP Multilayer Perceptron
  • the classification result may be binary classification or multi-classification. Taking binary classification as an example, the output classification result may belong to the target category or not belong to the target category.
  • the segmentation network in the image classification model can also use the category assignment matrix to segment the image to be classified to obtain image regions of preset categories.
  • the category assignment matrix redistributes the feature representation of each Token based on N cluster centers to assign Therefore, the category assignment matrix can be projected onto K channels to obtain K categories of image regions, i.e., the segmentation results.
  • K is a preset positive integer greater than 1.
  • the above K categories include the target category corresponding to the classification network.
  • the segmentation result of the image to be classified can provide interpretability for the classification result, so that users can refer to and understand the area of the target category in the image to be classified.
  • FIG4 is a flow chart of a method for training an image classification model provided in an embodiment of the present application, and the method flow can be executed by the model training device in the system shown in FIG1. As shown in FIG4, the method may include the following steps:
  • Step 402 Acquire training data including a plurality of training samples, where the training samples include image samples and labels indicating whether the image samples belong to a target category.
  • some images known to belong to the target category or not belong to the target category can be obtained as image samples, and labels of whether the image samples belong to the target category can be annotated.
  • some images are obtained as image samples, and labels of whether the image samples belong to the target category are manually annotated.
  • the image sample may be annotated with a region mask of a preset category, that is, the region of the preset category in the image sample is annotated to form a region mask of a specific category.
  • Manual annotation may be adopted, or image samples of regions of known preset categories may be obtained, and the regions of the preset categories therein are annotated to form a region mask.
  • the above preset categories include target categories.
  • the training data set S can be expressed as: ⁇ (X i ,Y i ,P i )
  • i 1,2,...,m ⁇ , where Xi is an image sample, Pi can be expressed as a label indicating whether Xi belongs to the target category, Yi is a region mask of a preset category marked on Xi , and m is the number of training samples.
  • Step 404 Use the training data to train the image classification model, the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from the image sample to obtain a feature representation of the image sample; the first decoding network uses the feature representation of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representation of the image sample and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.
  • the feature extraction network extracts features from the image sample Xi to obtain a feature representation F i of the image sample.
  • the feature extraction network can be implemented based on a Transformer network to extract image features and obtain a feature representation of each Token (element) in the image sample.
  • the feature extraction network can extract features from image samples, and obtain the feature representations of each element token at multiple resolutions in turn, and use the feature representations of each token at the highest resolution as the feature representation of the image sample.
  • U-Net can be used as a feature extraction network to obtain the feature representations of each token at multiple resolutions, and the accuracy of image classification can be improved through this multi-scale feature.
  • U-Net uses It is an Encoder-Decoder architecture. The Encoder performs downsampling, and the Decoder performs upsampling and feature concatenation to obtain feature representations of multiple different resolutions. That is, the feature representation of each Token at a low resolution is first obtained, and then the feature representation of each Token at a higher resolution is obtained in turn. The feature representation of each Token at the highest resolution is then used as the feature representation of the image.
  • the first decoding network uses the feature representation Fi of the image sample to perform cross-attention processing on the initial representations of multiple cluster centers (the corresponding matrix representation is C initial ) to obtain multiple cluster center representations (the corresponding matrix representation is C i ).
  • the first decoding network may include a multi-layer series of Transformer networks, and the initial representations of multiple cluster centers are input into the first layer of Transformer network as a query matrix.
  • Each layer of Transformer network corresponds to each resolution in order from low to high.
  • Each layer of Transformer network obtains a key matrix and a value matrix using the feature representation of each Token at the corresponding resolution, and performs cross-attention processing on the query matrix input into the Transformer network of this layer to obtain the query matrix output by the Transformer network of this layer; and multiple cluster center representations are obtained using the query matrix output by the last layer of Transformer network.
  • the second decoding network uses the feature representation Fi of the image sample and multiple cluster center representations Ci for decoding to obtain the category assignment matrix Mi.
  • the feature representation of each token is redistributed to different clusters based on multiple cluster centers, so the result obtained by the second decoding network is called the category assignment matrix.
  • the classification network uses multiple cluster center representations Ci and category assignment matrices Mi for classification to obtain the classification result of whether the image sample belongs to the target category.
  • the classification network can average multiple cluster center representations to obtain a cluster average representation; perform pooling on the category assignment matrix to obtain a cluster pooling representation; integrate the cluster average representation and the cluster pooling representation, and use the integrated feature representation for classification to obtain a classification result of whether the image sample belongs to the target category.
  • the cluster average representation can be obtained by averaging multiple cluster center representations in the channel dimension.
  • other processing methods can also be used to convert the matrix Ci into a vector.
  • the cluster average representation is
  • cluster average representation and cluster pooling representation can be spliced, as shown in Figure 5. and After splicing, it passes through several layers of MLP (Multilayer Perceptron) to obtain the classification result of whether the image to be classified belongs to the target category.
  • MLP Multilayer Perceptron
  • the classification result may be binary classification or multi-classification. Taking binary classification as an example, the output classification result may belong to the target category or not belong to the target category.
  • the training objectives in the above training process include minimizing the classification results The difference between the label Pi for Xi in the training sample.
  • the segmentation network in the image classification model can also use the category assignment The matrix segments the image samples to obtain image regions of preset categories.
  • the category assignment matrix is obtained by redistributing the feature representation of each token based on multiple cluster centers to assign it to different clusters, so that each token belonging to the same cluster can be treated as a whole. Therefore, the category assignment matrix can be projected to K channels to obtain K categories of image regions, i.e., the segmentation results.
  • K is a preset positive integer greater than 1.
  • the K categories include the target category corresponding to the classification network.
  • the training objectives in the above training process may further include: minimizing the image area of a preset category The difference between the corresponding region mask Yi .
  • the goal of this training can be used to assist in learning the classification results during the model training process.
  • a loss function can be constructed according to the above training objectives, and the value of the loss function is used in each round of iteration to update the model parameters by a method such as gradient descent until the preset training end condition is met.
  • the training end condition may include, for example, the value of the loss function is less than or equal to a preset loss function threshold, the number of iterations reaches a preset number threshold, etc.
  • the total loss function L can be constructed, for example:
  • remote sensing images can be used as images to be classified for mineral crystal detection, oil exploration, etc., that is, the remote sensing images can be classified using the method provided in the embodiments of the present application to determine whether the remote sensing images belong to a specific mineral crystal category, and the remote sensing images can be segmented to obtain various categories of regions, including specific mineral crystal regions, thereby providing an interpretability reference.
  • equipment images can be collected using equipment such as drones and inspection robots, and the equipment images can be used as images to be classified, and the equipment images can be classified using the method provided in the embodiments of the present application to determine whether they belong to an abnormal category, that is, whether there is an abnormality.
  • the equipment image can be segmented to obtain background areas, equipment areas, abnormal areas, etc. to provide an interpretability reference.
  • FIG6 is a flow chart of an image classification method applied to a medical scenario provided by an embodiment of the present application. As shown in FIG6 , the method may specifically include the following steps:
  • Step 602 Acquire medical images collected for the target organ.
  • the medical images collected here are collected in a non-invasive manner, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound examination, etc.
  • CT Computer Planar Tomography
  • MRI Magnetic Resonance Imaging
  • ultrasound examination etc.
  • the target organ can be the stomach, lung, liver, pancreas, colon, etc.
  • it can be a non-contrast CT image including the stomach.
  • a plain scan CT device usually generates a multi-layer image after scanning a circle. Therefore, the plain scan CT image X ⁇ R H ⁇ W ⁇ D , where H, W and D represent three-dimensional images (the input image to be classified is a three-dimensional image For example, the height, width and number of layers.
  • Step 604 extract features from the medical image to obtain feature representation of the medical image.
  • Step 606 Using the feature representation of the medical image, cross-attention processing is performed on the initial representations of the multiple cluster centers obtained by pre-training to obtain multiple cluster center representations.
  • step 604 and step 606 can refer to the description of step 204 and step 206 in the previous embodiment, and will not be repeated here.
  • the number of cluster centers N is a hyperparameter.
  • the value of N is usually greater than the number of categories K of the subsequent segmented regions, because lesions such as gastric cancer may have multiple subtypes. For example, the characteristics of gastric cancer at different stages of development are different, so more cluster centers will be selected.
  • N is generally an empirical value or an experimental value, such as 8.
  • Step 608 Decoding is performed using the feature representation of the medical image and the multiple cluster center representations to obtain a category assignment matrix.
  • Step 610 Classify using multiple cluster center representations and category assignment matrices to obtain a classification result of whether the medical image belongs to the target lesion.
  • step 608 and step 610 can refer to the description of step 208 and step 210 in the previous embodiment, and will not be repeated here.
  • the classification result obtained in step 610 can be a 2-classification, i.e. normal or gastric cancer.
  • the result of detecting whether there is a suspicion of gastric cancer based on the plain scan CT image of the stomach is provided to the physician for reminder and reference.
  • the initial feature representation of multiple cluster centers is cross-attention processed based on the feature representation of each Token of multi-resolution, and the feature representation of multiple cluster centers is redistributed with the feature representation of the image, so that the classification process can be sensitive to local tissues and have a global awareness of organ physiology.
  • the segmentation network in the image classification model can also segment the medical image using the category assignment matrix to obtain image regions of preset categories.
  • the preset categories may include background, organs, and lesions. Still taking the plain scan CT image of the stomach as an example, the background region, the stomach region, and the gastric cancer region can be segmented, thereby providing the physician with a reference for interpretability.
  • feature extraction may also be performed on the medical image to obtain feature representations of each element Token at multiple resolutions, and the feature representation of each Token at the highest resolution may be used as the feature representation of the medical image.
  • the query matrix can be obtained using the initial representations of multiple cluster centers, and the query matrix is input into a multi-layer serial transformation Transformer network.
  • Each layer of the Transformer network corresponds to each resolution in order from low to high resolution.
  • Each layer of the Transformer network obtains a key matrix and a value matrix using the feature representation of each Token at the corresponding resolution, and cross-attention processing is performed on the query matrix input to the Transformer network of this layer to obtain the query matrix output by the Transformer network of this layer; the query matrix output by the last layer of the Transformer network is used to obtain multiple cluster center representations.
  • multiple cluster center representations can be averaged to obtain a cluster average representation; and the category assignment matrix can be pooled to obtain a cluster pool representation; then the cluster average representation and the cluster pool representation are integrated, and the integrated feature representation is used for classification to obtain whether the medical image belongs to a class.
  • the present application also provides a computer-aided diagnosis method for gastric cancer, the method comprising:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to gastric cancer.
  • the image samples can be plain scan CT images of the stomach that are known to be normal or gastric cancer as each Xi , and each plain scan CT image is labeled with a label of normal or gastric cancer as Pi . Since the region labeling of plain scan CT is relatively difficult, the physician can label the background region, the stomach region, and the gastric cancer region of the contrast CT image corresponding to each Xi , and then align the labeled contrast CT image with the plain scan CT image, thereby forming a relatively rough but highly reliable mask of each region.
  • Gastric cancer is the third leading cause of cancer-related death worldwide, with a 5-year survival rate of about 33%. If relevant symptoms can be detected at an early stage, the 5-year survival rate can be significantly improved. Since early gastric tumors may only invade the mucosa and muscular layer, it is difficult to identify without gastric contrast agent injection.
  • existing detection methods such as barium meal gastography, endoscopic examination, and serum pepsinogen level examination are invasive, costly, and have large side effects, and are difficult to be well applied to early gastric cancer detection.
  • a computer device performs image classification to achieve gastric cancer detection, that is, the final output classification result is whether it belongs to gastric cancer, for example, a binary classification result of normal and gastric cancer is output, thereby providing a reference basis or reminder to doctors or patients as intermediate data, so as to facilitate further examination and diagnosis in the future.
  • the stomach plain scan CT image it is possible to segment the stomach plain scan CT image to obtain the background area, stomach area and lesion area (i.e. the area where the gastric cancer is located), thereby providing an interpretable reference for doctors or patients as intermediate data.
  • this is a new detection method that is non-invasive, low-cost and easy to promote, and it also has a good effect in terms of accuracy.
  • the method provided in the embodiment of the present application obtained a classification result of gastric cancer with a sensitivity of 85.0% and a specificity of 92.6% on a test set of stomach plain scan CT images of 100 gastric cancer patients and 148 normal patients.
  • the average sensitivity of radiologists in manually distinguishing gastric cancer from stomach plain scan CT images is 73.5%, and the specificity is 84.3%.
  • the image classification method based on computer vision technology provided in the embodiment of the present application is more effective.
  • sensitivity is also called true positive rate, which refers to the probability that the classification result of a test on a gastric cancer sample is gastric cancer.
  • Specificity is also called true negative rate, which refers to the probability that the classification result of a test on a normal sample is normal.
  • the third baseline (denoted as "nnUNet-Joint”) integrates the CNN classification head into UNet and performs end-to-end training.
  • the 95% confidence intervals of the AUC, sensitivity, and specificity values can be obtained from 1000 bootstrap copies of the test data set for statistical analysis.
  • a DeLong test can be performed between two AUCs (the image classification method of the present disclosure and the comparison method), and a permutation test can be performed between two sensitivities or specificities (the image classification method of the present disclosure and the comparison method and radiologists).
  • the image classification method of the present invention outperforms the other three baselines (Table 1) in all indicators, especially in terms of AUC and sensitivity.
  • Table 1 The advantage of the method of the present invention is that it captures local and global information at the same time by virtue of the unique architecture of the mask transformer. It also extracts high-level semantics from the cluster representation, making it suitable for classification and facilitating the overall decision-making process.
  • the method of the present invention achieves a considerable specificity of 97.7% on the external test set, which is crucial to reducing false positives and unnecessary human workload in terms of opportunistic screening.
  • the ROC curve of the model proposed in this disclosure is better than the results of detection by two experienced radiologists.
  • the model achieved a sensitivity of 85.0% in detecting gastric cancer, significantly exceeding the average performance of doctors (73.5%) and the best performing doctor (75.0%), while maintaining high specificity.
  • Table 2 compares the performance of the disclosed model with that of two radiologists. The results show that the disclosed model performs better in detecting early tumors (T1, T2) and provides more accurate tumor localization. Specifically, the disclosed model detected 60.0% (6/10) T1 cancers and 77.8% (7/9) T2 cancers, surpassing the best performing expert (50% T1, 55.6% T2). At the same time, the disclosed model maintained a reliable detection rate and confidence for T3 and T4 tumors. The localization accuracy was 2 out of 34 T3 tumors (missed).
  • FIG7 shows a schematic block diagram of an image classification device according to an embodiment.
  • the device 700 includes: an image acquisition unit 701, a feature extraction unit 702, a first decoding unit 703, a second decoding unit 704, and an image classification unit 705, and may further include an image segmentation unit 706.
  • the main functions of each component unit are as follows:
  • the image acquisition unit 701 is configured to acquire an image to be classified.
  • the feature extraction unit 702 is configured to extract features from the image to be classified to obtain a feature representation of the image.
  • the first decoding unit 703 is configured to perform cross-attention processing on the initial representations of multiple cluster centers obtained by pre-training using the feature representation of the image to obtain multiple cluster center representations.
  • the second decoding unit 704 is configured to perform decoding processing using the feature representation of the image and the multiple cluster center representations, Get the class assignment matrix.
  • the image classification unit 705 is configured to perform classification using multiple cluster center representations and category assignment matrices to obtain a classification result of whether the image to be classified belongs to a target category.
  • the image segmentation unit 706 is configured to segment the image to be classified using the category assignment matrix to obtain image regions of preset categories, where the preset categories include target categories.
  • the feature extraction unit 702 may be specifically configured to: perform feature extraction on the image to be classified, obtain feature representations of each element Token at multiple resolutions, and use the feature representations of each Token at the highest resolution as feature representations of the image.
  • the first decoding unit 703 can use the initial representations of multiple cluster centers to obtain a query matrix, and input the query matrix into a multi-layer serially connected Transformer network.
  • Each layer of the Transformer network corresponds to each resolution in order from low to high resolution.
  • Each layer of the Transformer network uses the feature representation of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input to the Transformer network of this layer to obtain the query matrix output by the Transformer network of this layer; use the query matrix output by the last layer of the Transformer network to obtain multiple cluster center representations.
  • the image classification unit 705 can be specifically configured to average multiple cluster center representations to obtain a cluster average representation; perform pooling processing on the category assignment matrix to obtain cluster pooling features; integrate the cluster average representation and the cluster pooling representation, and perform classification using the integrated feature representation to obtain a classification result of whether the image to be classified belongs to the target category.
  • the above-mentioned feature extraction unit 702, first decoding unit 703, second decoding unit 704, image classification unit 705 and image segmentation unit 706 respectively correspond to the feature extraction network, first decoding network, second decoding network, classification network and segmentation network in the image classification model shown in Figure 3.
  • image classification unit 705 and image segmentation unit 706 respectively correspond to the feature extraction network, first decoding network, second decoding network, classification network and segmentation network in the image classification model shown in Figure 3.
  • FIG8 is a schematic block diagram of a device for training an image classification model provided in an embodiment of the present application.
  • the device may include: a sample acquisition unit 801 and a model training unit 802.
  • the main functions of each component unit are as follows:
  • the sample acquisition unit 801 is configured to acquire training data including a plurality of training samples, where the training samples include image samples and labels indicating whether the image samples belong to a target category.
  • the model training unit 802 is configured to train an image classification model using training data, and the image classification model includes: a feature extraction network, a first decoding network, a second decoding network and a classification network; the feature extraction network extracts features from image samples to obtain feature representations of the image samples; the first decoding network uses the feature representations of the image samples to perform cross-attention processing on the initial representations of multiple cluster centers to obtain multiple cluster center representations; the second decoding network uses the feature representations of the image samples and the multiple cluster center representations to perform decoding processing to obtain a category assignment matrix; the classification network uses the multiple cluster center representations and the category assignment matrix to perform classification to obtain a classification result of whether the image sample belongs to a target category; the training objectives include: minimizing the difference between the classification result and the corresponding label.
  • the training samples may also include preset categories of image samples labeled. Region Mask.
  • Image classification models also include segmentation networks.
  • the segmentation network segments the image sample using the category assignment matrix to obtain image regions of preset categories, where the preset categories include target categories.
  • the training objective may also include: minimizing the difference between image regions of preset categories and corresponding region masks.
  • the feature extraction network can extract features from image samples, obtain feature representations of each element Token at multiple resolutions in turn, and use the feature representations of each Token at the highest resolution as the feature representation of the image sample.
  • the first decoding network may include multiple layers of serially connected Transformer networks, and the initial representations of multiple cluster centers are input into the first layer of Transformer networks as query matrices.
  • Each layer of Transformer networks corresponds to each resolution one by one in order from low to high resolution.
  • Each layer of Transformer networks uses the feature representations of each Token at the corresponding resolution to obtain a key matrix and a value matrix, and performs cross-attention processing on the query matrix input into the current layer of Transformer networks to obtain the query matrix output by the current layer of Transformer networks; and uses the query matrix output by the last layer of Transformer networks to obtain multiple cluster center representations.
  • the classification network can average multiple cluster center representations to obtain a cluster average representation; perform pooling on the category assignment matrix to obtain a cluster pooling representation; integrate the cluster average representation and the cluster pooling representation, and use the integrated feature representation for classification to obtain a classification result of whether the image sample belongs to the target category.
  • an image classification device for medical diagnosis.
  • the device includes: an image acquisition unit, a feature extraction unit, a first decoding unit, a second decoding unit and an image classification unit, and may further include an image segmentation unit.
  • the main functions of each component unit are as follows:
  • An image acquisition unit configured to acquire a medical image for a target organ
  • a feature extraction unit configured to extract features from the medical image to obtain a feature representation of the medical image
  • a first decoding unit is configured to perform cross-attention processing on initial representations of multiple cluster centers obtained by pre-training using the feature representation of the medical image to obtain multiple cluster center representations;
  • a second decoding unit is configured to perform decoding processing using the feature representation of the medical image and the plurality of cluster center representations to obtain a category assignment matrix
  • the image classification unit is configured to perform classification using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.
  • each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.
  • the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
  • the device embodiment described above is only exemplary, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. It can be based on actual needs. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment. A person skilled in the art may understand and implement the solution without creative work.
  • user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • an embodiment of the present application further provides a cancer computer-aided diagnosis system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform an image classification method, the method comprising:
  • cross-attention processing is performed on the initial representations of multiple cluster centers obtained by pre-training to obtain multiple cluster center representations
  • Classification is performed using the multiple cluster center representations and the category assignment matrix to obtain a classification result of whether the medical image belongs to a target lesion.
  • an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.
  • an electronic device comprising:
  • a memory associated with the one or more processors the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.
  • the present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.
  • the present application also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of any one of the methods described in the aforementioned method embodiments.
  • FIG. 9 exemplarily shows the architecture of the electronic device, which may include a processor 910, a video display adapter 911, a disk drive 912, an input/output interface 913, a network interface 914, and a memory 920.
  • the processor 910, the video display adapter 911, the disk drive 912, the input/output interface 913, the network interface 914, and the memory 920 may be communicatively connected via a communication bus 930.
  • the processor 910 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.
  • a general-purpose CPU a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.
  • ASIC application-specific integrated circuit
  • the memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc.
  • the memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, and a basic input and output system (BIOS) 922 for controlling the low-level operation of the electronic device 900.
  • BIOS basic input and output system
  • a web browser 923, a data storage management system 924, and an image classification device/model training device 925, etc. can also be stored.
  • the above-mentioned image classification device/model training device 925 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application.
  • the relevant program code is stored in the memory 920 and is called and executed by the processor 910.
  • the input/output interface 913 is used to connect the input/output module to realize information input and output.
  • the input/output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions.
  • the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc.
  • the output device may include a display, a speaker, a vibrator, an indicator light, etc.
  • the network interface 914 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices.
  • the communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
  • the bus 930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 910, the video display adapter 911, the disk drive 912, the input/output interface 913, the network interface 914, and the memory 920).
  • the various components of the device eg, the processor 910, the video display adapter 911, the disk drive 912, the input/output interface 913, the network interface 914, and the memory 920).
  • the above device only shows a processor 910, a video display adapter 911, a disk drive 912, an input/output interface 913, a network interface 914, a memory 920, a bus 930, etc., in the specific implementation process, the device may also include other components necessary for normal operation.
  • the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请实施例公开了一种图像分类方法、训练图像分类模型的方法及装置。主要技术方案包括：获取待分类的图像；对所述待分类的图像进行特征提取，得到所述图像的特征表示；利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。通过本申请能够使得图像分类结果具有更高的准确率。

Description

图像分类方法、训练图像分类模型的方法及装置
本申请要求于2023年07月04日提交中国专利局、申请号为202310822213.1、申请名称为“图像分类方法、训练图像分类模型的方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机视觉技术领域，特别是涉及一种图像分类的方法、训练图像分类模型的方法及装置。
背景技术
图像分类是根据图像的语义信息对不同类别的图像进行区分，是计算机视觉中重要的基础问题。图像分类在很多领域都有着广泛的应用，例如：交通领域的交通场景识别、电商领域的商品图像自动分类，医学领域的图像识别等等。
在一些特殊领域中对于图像分类的准确率和召回率具有较高的要求。虽然目前已经存在利用深度学习模型来对图像进行分类的相关技术，但分类结果的准确率仍有待提高。
发明内容
有鉴于此，本申请提供了一种图像分类的方法、训练图像分类模型的方法及装置，以便于提高图像分类结果的准确率。
本申请提供了如下方案：
第一方面，提供了一种图像分类方法，所述方法包括：
获取待分类的图像；
对所述待分类的图像进行特征提取，得到所述图像的特征表示；
利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
根据本申请实施例中一可实现的方式，所述方法还包括：
利用所述类别分配矩阵对所述待分类的图像进行分割，得到预设类别的图像区域，所述预设类别包括所述目标类别。
根据本申请实施例中一可实现的方式，对所述待分类的图像进行特征提取，得到所述图像的特征表示包括：
对所述待分类的图像进行特征提取，得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为所述图像的特征表示。
根据本申请实施例中一可实现的方式，利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理包括：
利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；
利用最后一层Transformer网络输出的查询矩阵得到所述多个聚类中心表示。
根据本申请实施例中一可实现的方式，利用所述多个聚类中心表示和所述类别分配矩阵进行分类包括：
对所述多个聚类中心表示进行求平均处理，得到聚类平均表示；
对所述类别分配矩阵进行池化处理，得到聚类池化特征；
将所述聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
第二方面，提供了一种训练图像分类模型的方法，所述方法包括：
获取包括多个训练样本的训练数据，所述训练样本包括图像样本以及所述图像样本是否属于目标类别的标签；
利用所述训练数据训练图像分类模型，所述图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；所述特征提取网络对所述图像样本进行特征提取，得到所述图像样本的特征表示；所述第一解码网络利用所述图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；所述第二解码网络利用所述图像样本的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述图像样本是否属于目标类别的分类结果；所述训练的目标包括：最小化所述分类结果与对应标签之间的差异。
根据本申请实施例中一可实现的方式，所述训练样本还包括对所述图像样本标注的预设类别的区域掩码；所述图像分类模型还包括分割网络；
所述分割网络利用所述类别分配矩阵对所述图像样本进行分割，得到预设类别的图像区域，所述预设类别包括所述目标类别；
所述训练的目标还包括：最小化所述预设类别的图像区域与对应区域掩码之间的差异。
根据本申请实施例中一可实现的方式，所述特征提取网络对所述图像样本进行特征提 取，得到所述图像样本的特征表示包括：对所述图像样本进行特征提取，依次得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为所述图像样本的特征表示；
所述第一解码网络包括多层串连的Transformer网络，所述多个聚类中心的初始表示作为查询矩阵输入第一层Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到所述多个聚类中心表示。
根据本申请实施例中一可实现的方式，所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分类包括：
所述分类网络对所述多个聚类中心表示进行求平均处理，得到聚类平均表示；
对所述类别分配矩阵进行池化处理，得到聚类池化表示；
将所述聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到所述图像样本是否属于目标类别的分类结果。
第三方面，提供了一种图像分类方法，由云端服务器执行，所述方法包括：
获取来自用户终端的待分类的图像；
对所述待分类的图像进行特征提取，得到所述图像的特征表示；
利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果；
将所述分类结果返回给所述用户终端。
第四方面，提供了一种癌症计算机辅助诊断方法，其特征在于，所述方法包括：
获取针对目标器官采集的医学影像；
对所述医学影像进行特征提取，得到所述医学影像的特征表示；
利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
第五方面提供了一种胃癌计算机辅助诊断方法，所述方法包括：
获取针对胃部采集的医学影像；
对所述医学影像进行特征提取，得到所述医学影像的特征表示；
利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于胃癌的分类结果。
第六方面，提供了一种癌症的计算机辅助诊断系统，包括存储器、处理器以及存储在所述存储器上并在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序可用于执行一种癌症的计算机辅助诊断方法，所述方法包括：
获取针对目标器官采集的医学影像；
对所述医学影像进行特征提取，得到所述医学影像的特征表示；
利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
第七方面，提供了一种图像分类装置，所述装置包括：
图像获取单元，被配置为获取待分类的图像；
特征提取单元，被配置为对所述待分类的图像进行特征提取，得到所述图像的特征表示；
第一解码单元，被配置为利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
第二解码单元，被配置为利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
图像分类单元，被配置为利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
第八方面，提供了一种训练图像分类模型的装置，所述装置包括：
样本获取单元，被配置为获取包括多个训练样本的训练数据，所述训练样本包括图像样本以及所述图像样本是否属于目标类别的标签；
模型训练单元，被配置为利用所述训练数据训练图像分类模型，所述图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；所述特征提取网络对所述图像样本进行特征提取，得到所述图像样本的特征表示；所述第一解码网络利用所述图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示； 所述第二解码网络利用所述图像样本的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述图像样本是否属于目标类别的分类结果；所述训练的目标包括：最小化所述分类结果与对应标签之间的差异。
第九方面，提供了一种图像分类装置，所述装置包括：
图像获取单元，被配置为获取针对目标器官采集的医学影像；
特征提取单元，被配置为对所述医学影像进行特征提取，得到所述医学影像的特征表示；
第一解码单元，被配置为利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
第二解码单元，被配置为利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
图像分类单元，被配置为利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。第十方面，提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述第一方面至第五方面中任一项所述的方法的步骤。
第十一方面，提供了一种电子设备，包括：
一个或多个处理器；以及
与所述一个或多个处理器关联的存储器，所述存储器用于存储程序指令,所述程序指令在被所述一个或多个处理器读取执行时，执行上述第一方面至第五方面中任一项所述的方法的步骤。
第十二方面，提供了一种计算机程序，所述计算机程序在计算机中执行时，令计算机执行上述第一方面至第五方面中任一项所述的方法的步骤。
第十三方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序在计算机中执行时，令计算机执行上述第一方面至第五方面中任一项所述的方法的步骤。
根据本申请提供的具体实施例，本申请公开了以下技术效果：
1)本申请通过预先训练得到的多个聚类中心所包含的高水平的语义，与图像的特征表示共同进行解码处理，得到类别分配矩阵，使得类别分配矩阵能够体现与聚类中心关联的图像特征，即将图像特征匹配到相应的聚类中心，进而使得依据聚类中心表示和类别分配矩阵得到的图像分类结果具有更高的准确率和召回率。
2)本申请能够进一步利用类别分配矩阵对待分类的图像进行分割，得到预设类别的图像区域，从而为图像分类提供可解释性的参考。
3)本申请中通过提取多分辨率的特征，并利用多分辨率的特征多个聚类中心的初始表示进行交叉注意力处理，使得在得到多个聚类中心表示的过程中能够从多尺度感知图像纹理，并将多个聚类中心的特征表示与图像的特征表示进行重新分配，使得分类过程和分割 过程既对局部语义敏感，又具有全局意识，更进一步提高图像分类和图像分割的准确率和召回率。
4)在训练图像分类模型的过程中，能够利用分割网络对图像样本进行分割得到的预设类别的图像区域与针对图像样本标注的预设类别的区域掩码之间的差异，对图像分类模型进行辅助训练，从而进一步提高图像分类模型的效果和性能。
当然，实施本申请的任一产品并不一定需要同时达到以上所述的所有优点。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为是本申请实施例所适用的系统架构图；
图2为本申请实施例提供的图像分类方法的流程图；
图3为本申请实施例提供的图像分类模型的原理性示意图；
图4为本申请实施例提供的训练图像分类模型的方法流程图；
图5为本申请实施例提供的训练图像分类模型的原理性示意图；
图6为本申请实施例提供的应用于医学领域的图像分类方法的流程图；
图7为本申请实施例提供的图像分类装置的示意性框图；
图8为本申请实施例提供的训练图像分类模型的装置示意性框图；
图9为本申请实施例提供的电子设备的示意性框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员所获得的所有其他实施例，都属于本申请保护的范围。
在本发明实施例中使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本发明。在本发明实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。
应当理解，本文中使用的术语“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当…… 时”或“响应于确定”或“响应于检测”。类似地，取决于语境，短语“如果确定”或“如果检测(陈述的条件或事件)”可以被解释成为“当确定时”或“响应于确定”或“当检测(陈述的条件或事件)时”或“响应于检测(陈述的条件或事件)”。
传统基于深度神经网络的图像分类方法大多是从图像中提取图像的特征表示后，利用提取的图像的特征表示直接进行分类。但这种方式在一些复杂场景下得到的图像分类结果的准确率和召回率较低，无法满足实际的场景需求。
有鉴于此，本申请提供了一种全新的图像分类思路。为了方便对本申请的理解，首先对本申请所适用的系统架构进行简单描述。图1示出了可以应用本申请实施例的示例性系统架构，如图1中所示，该系统架构包括服务端的模型训练装置和图像分类装置，还可以包括用户终端。
其中，模型训练装置用以在离线阶段进行模型训练。即在获取训练数据后，可以采用本申请实施例提供的方法进行模型训练，得到图像分类模型。
图像分类装置用以在线上利用已经训练得到的图像分类模型，对待分类的图像进行分类，得到图像是否属于目标类别的分类结果。
模型训练装置和图像分类装置可以分别设置为独立的服务器，也可以设置于同一个服务器或服务器群组，还可以设置于独立的或者同一云服务器。云服务器又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决传统物理主机与虚拟专用服务器(VPs，Virtual Private Server)服务中存在的管理难度大，服务扩展性弱的缺陷。模型训练装置和图像分割装置还可以设置于具有较强计算能力的计算机终端。
作为其中一种可实现的方式，用户可以通过用户终端将待分类的图像通过网络发送给服务端的图像分类装置。由图像分类装置采用本申请实施例提供的方法对待分类的图像进行分类后，将分类结果返回给用户终端。
上述用户终端可以包括但不限于智能移动终端、智能家居设备、可穿戴式设备、智能医疗设备、PC(Personal Computer，个人计算机)等。其中智能移动设备可以包括诸如手机、平板电脑、笔记本电脑、PDA(Personal Digital Assistant，个人数字助理)、互联网汽车等。智能家居设备可以包括诸如智能电视、智能冰箱等等。可穿戴式设备可以包括诸如智能手表、智能眼镜、智能手环、VR(Virtual Reality，虚拟现实)设备、AR(Augmented Reality，增强现实设备)、混合现实设备(即可以支持虚拟现实和增强现实的设备)等等。
需要说明的是，上述图像分类装置除了在线上进行图像分类之外，也可以采用离线的方式进行图像分类，例如针对批量的待分类图像分别进行图像分类。
应该理解，图1中的模型训练装置、图像分类装置、图像分类模型和用户终端的数目仅仅是示意性的。根据实现需要，可以具有任意数目的模型训练装置、图像分类装置、图像分类模型和用户终端。
需要说明的是，本公开中涉及的“第一”、“第二”等限定并不具备大小、顺序和数量等方面的限制，仅仅用以在名称上加以区分。例如“第一图像分割模型”和“第二图像分割模型” 用以在名称上区分两条语音。再例如“第一分割结果”和“第二分割结果”用以在名称上区分两个分割结果。再例如“第一目标”和“第二目标”用以在名称上区分两种目标。等等。
图2为本申请实施例提供的图像分类方法的流程图，该方法可以由图1所示系统中的图像分类装置执行。如图2中所示，该方法可以包括以下步骤：
步骤202：获取待分类图像。
步骤204：对待分类的图像进行特征提取，得到图像的特征表示。
步骤206：利用图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示。
步骤208：利用图像的特征表示和多个聚类中心表示进行解码处理，得到类别分配矩阵。
步骤210：利用多个聚类中心表示和类别分配矩阵进行分类，得到待分类的图像是否属于目标类别的分类结果。
由上述流程可以看出，本申请通过预先训练得到的多个聚类中心所包含的高水平的语义，与图像的特征表示共同进行解码处理，得到类别分配矩阵，使得类别分配矩阵能够体现与聚类中心关联的图像特征，即将图像特征匹配到相应的聚类中心，进而使得依据聚类中心表示和类别分配矩阵得到的图像分类结果具有更高的准确率。
下面对上述各步骤进行详细描述。首先对上述步骤202即“获取待分类的图像”进行详细描述。
在本申请中涉及的待分类的图像可以是二维图像，也可以是三维图像。可以是灰度图像，也可以是彩色图像。
在不同的应用领域待分类的图像可以包含不同的内容。例如，在交通领域，待分类的图像通常为包含诸如车辆、行人、道路、交通设施等交通元素的图像，图像分类的目的可以是确定图像是否属于特定的交通场景。再例如，在电商领域，待分类的图像通常为包含商品信息的图像，图像分类的目的可以是确定图像是否属于特定的商品类目。再例如，在医学领域，图像通常为医学影像，例如CT(Computed Tomography，断层扫描图像)、MRI(Magnetic Resonance Imaging，核磁共振图像)、超音波检查图像等，包含肺、肝脏、胰腺、结肠等器官，图像分类的目的可以是确定图像是否属于特定类别的器官区域，或者是否存在某种特定类别的异常。
上述流程中的步骤204～210可以由预先训练得到的图像分类模型实现。如图3中所示，本申请实施例提供的图像分类模型可以包括：特征提取网络、第一解码网络、第二解码网络和分类网络。
其中，上述步骤204即“对待分类的图像进行特征提取，得到图像的特征表示”可以由上述的特征提取网络执行。
特征提取网络可以基于Transformer(转换)网络实现，实现图像特征的提取，得到待 分类的图像中各Token(元素)的特征表示。例如特征提取网络可以采用诸如VIT(Vision Transformer，视觉转换器)、RAN(Residual Attention Network，残差注意力网络)、U-Net(U-Net是FCN(Fully Convolutional Network，全卷积神经网络)的一种变体，其提出是为了解决生物医学影像的问题，由于效果很好后来被广泛地应用于图像分割的各个领域)等。
图像的各Token指的是构成图像的元素。对于图像而言，将图像切分成不重叠的区块序列，则图像中的区块以及序列的起始符均为Token。对于二维图像而言，其中的区块可以由一个或多个像素构成。对于三维图像而言，其中的区块可以由一个或多个体素构成。
作为其中一种优选的实施方式，可以采用诸如U-Net作为特征提取网络，得到多种分辨率下的各Token的特征表示，通过这种多尺度特征来提高图像分类的准确性。具体地，U-Net采用的是Encoder(编码器)-Decoder(解码器)架构，Encoder进行下采样，Decoder进行上采样和特征拼接，得到多个不同分辨率的特征表示，即先得到低分辨率下各Token的特征表示，进而依次得到更高分辨率下各Token的特征表示。然后将最高分辨率下的各Token的特征表示作为图像的特征表示。
如图3中所示，假设输入的待分类图像表示为X，通过特征提取网络得到图像的特征表示F，F∈RA×HWD。其中，A为特征维度，H、W和D分别表示三维图像(以输入的待分类图像为三维图像为例)的高、宽和长。
上述步骤206即“利用图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示”可以由图像分类模型中的第一解码网络执行。
本步骤中，第一解码网络实际上是将一组聚类中心的初始表示，通过图像的特征表示的交叉注意力，转化为融合了图像语义的聚类中心表示。
若特征提取网络得到多个不同分辨率的特征表示，则作为其中一种优选的实施方式，第一解码网络可以包括多层串连的Transformer网络。可以利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到多个聚类中心表示。
每一层Transformer网络执行的处理可以表示如下：
其中，Cn表示第n层Transformer网络输出的多个聚类中心表示对应的矩阵，Cn-1为第n-1层Transformer网络输出的多个聚类中心表示对应的矩阵，对于第一层Transformer网络，其输入的Cn-1为多个聚类中心的初始表示对应的矩阵Cinitial。Q由Cn-1得到，K和V均由该层Transformer网络对应分辨率下的各Token的特征表示得到。T表示转置处理。N为聚类中心的数量。
从上述过程中可以看出，第一解码器相当于依据初始聚类中心对图像中的各Token进行分组，其中交叉注意力的处理与K-means聚类算法类似，使用argmax处理替代传统Transformer网络中的softmax处理。最终得到的多个聚类中心表示对应的矩阵表示为C∈RN×A。
上述步骤208即“利用图像的特征表示和多个聚类中心表示进行解码处理，得到类别分配矩阵”可以由上述的第二解码网络执行。
本步骤实际上是基于N个聚类中心对各Token的特征表示进行重新分配，分配到不同的聚类中，因此将本步骤得到的结果称为类别分配矩阵。
第二解码网络可以将多个聚类中心表示对应的矩阵C和图像的特征表示F进行矩阵相乘，然后进行Softmax得到类别分配矩阵对应的矩阵M，可以表示为：
M＝SoftmaxN(CF)               (2)
其中，SoftmaxN()表示在N聚类中心维度上进行的Softmax(归一化指数函数)的处理。
上述步骤210即“利用多个聚类中心表示和类别分配矩阵进行分类，得到待分类的图像是否属于目标类别的分类结果”可以由图3所示的图像分类模型中的分类网络执行。
由于学习到的多个聚类中心表示具有高水平的语义，同时体现了聚类之间的差异和聚类内部的相似性，因此，将多个聚类中心表示和类别分配矩阵进行结合考虑进行分类，相比较直接利用图像的特征表示进行分类而言，能够显著提高分类的精确度和召回率。
作为其中一种可实现的方式，可以对多个聚类中心表示进行求平均处理，得到聚类平均表示；以及对类别分配矩阵进行池化处理，得到聚类池化表示；然后将聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到待分类的图像是否属于目标类别的分类结果。
例如，聚类平均表示可以在通道维度上对多个聚类中心表示进行求平均处理后得到的。除了求平均处理之外，也可以采用其他处理方式将矩阵C转化成向量。在图3中，聚类平均表示为
对类别分配矩阵进行池化处理时，可以采用诸如全局最大池化等。在图3中，池化处理后得到的聚类池化表示为
将聚类平均表示和聚类池化表示进行的整合可以是进行拼接，如图3中所示，可以将和进行拼接后，经过若干层MLP(Multilayer Perceptron，多层感知机)，得到待分类的图像是否属于目标类别的分类结果
在本申请实施例中，分类结果可以是二分类，也可以是多分类。以二分类为例，输出的分类结果可以是属于目标类别，或者不属于目标类别。
更进一步地，在本申请实施例中，还可以由图像分类模型中的分割网络利用类别分配矩阵对待分类的图像进行分割，得到预设类别的图像区域。
类别分配矩阵是基于N个聚类中心对各Token的特征表示进行重新分配，以分配到不 同的聚类中得到的，因此可以将属于同一聚类中的各Token作为一个整体来对待。因此，可以将类别分配矩阵投影到K个通道，从而得到K个类别的图像区域即分割结果K为预设的大于1的正整数。其中，上述K个类别中包括分类网络对应的目标类别。待分类图像的分割结果可以为分类结果提供可解释性，供用户参考并了解目标类别在待分类的图像中的区域。
下面对上述实施例采用的图像分类模型的训练过程进行详细描述。图4为本申请实施例提供的训练图像分类模型的方法流程图，该方法流程可以由图1所示系统中的模型训练装置执行。如图4中所示，该方法可以包括以下步骤：
步骤402：获取包括多个训练样本的训练数据，训练样本包括图像样本以及图像样本是否属于目标类别的标签。
在本申请实施例中，可以获取一些已知属于目标类别或者不属于目标类别的图像作为图像样本，并对图像样本标注是否属于目标类别的标签。或者，获取一些图像作为图像样本，人工对图像样本标注是否属于目标类别的标签。这些图像样本可以根据实际需求从具体应用领域中获取。
更进一步地，还可以对图像样本标注预设类别的区域掩码(Mask)，即对图像样本中预设类别的区域进行标注，形成特定类别的区域掩码。可以采用人工标注的方式，也可以获取已知预设类别的区域的图像样本，将其中预设类别的区域进行标注形成区域掩码。上述预设类别包括目标类别。
例如训练数据集S可以表示为：{(Xi,Yi,Pi)|i＝1,2,…,m}，其中，Xi为图像样本，Pi可以表示为对Xi标注的是否属于目标类别的标签，Yi为对Xi标注的预设类别的区域掩码，m为训练样本的数量。
步骤404：利用训练数据训练图像分类模型，图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；特征提取网络对图像样本进行特征提取，得到图像样本的特征表示；第一解码网络利用图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；第二解码网络利用图像样本的特征表示和多个聚类中心表示进行解码处理，得到类别分配矩阵；分类网络利用多个聚类中心表示和类别分配矩阵进行分类，得到图像样本是否属于目标类别的分类结果；训练的目标包括：最小化分类结果与对应标签之间的差异。
如图5中所示，特征提取网络对图像样本Xi进行特征提取，得到图像样本的特征表示Fi。特征提取网络可以基于Transformer(转换)网络实现，实现图像特征的提取，得到图像样本中各Token(元素)的特征表示。
作为其中一种可实现的方式特征提取网络可以对图像样本进行特征提取，依次得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为图像样本的特征表示。例如可以采用诸如U-Net作为特征提取网络，得到多种分辨率下的各Token的特征表示，通过这种多尺度特征来提高图像分类的准确性。具体地，U-Net采用的 是Encoder(编码器)-Decoder(解码器)架构，Encoder进行下采样，Decoder进行上采样和特征拼接，得到多个不同分辨率的特征表示，即先得到低分辨率下各Token的特征表示，进而依次得到更高分辨率下各Token的特征表示。然后将最高分辨率下的各Token的特征表示作为图像的特征表示。
第一解码网络利用图像样本的特征表示Fi对多个聚类中心的初始表示(对应矩阵表示为Cinitial)进行交叉注意力处理，得到多个聚类中心表示(对应的矩阵表示为Ci)。
若特征提取网络得到多个不同分辨率的特征表示，则作为其中一种优选的实施方式，第一解码网络可以包括多层串连的Transformer网络，多个聚类中心的初始表示作为查询矩阵输入第一层Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到多个聚类中心表示。
第二解码网络利用图像样本的特征表示Fi和多个聚类中心表示Ci进行解码处理，得到类别分配矩阵Mi。实际上是基于多个聚类中心对各Token的特征表示进行重新分配，分配到不同的聚类中，因此将第二解码网络得到的结果称为类别分配矩阵。
分类网络利用多个聚类中心表示Ci和类别分配矩阵Mi进行分类，得到图像样本是否属于目标类别的分类结果。
作为其中一种可实现的方式，分类网络可以对多个聚类中心表示进行求平均处理，得到聚类平均表示；对类别分配矩阵进行池化处理，得到聚类池化表示；将聚类平均表示和聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到图像样本是否属于目标类别的分类结果。
例如，聚类平均表示可以在通道维度上对多个聚类中心表示进行求平均处理后得到的。除了求平均处理之外，也可以采用其他处理方式将矩阵Ci转化成向量。在图5中，聚类平均表示为
对类别分配矩阵进行池化处理时，可以采用诸如全局最大池化等。在图5中，池化处理后得到的聚类池化表示为
将聚类平均表示和聚类池化表示进行的整合可以是进行拼接，如图5中所示，可以将和进行拼接后，经过若干层MLP(Multilayer Perceptron，多层感知机)，得到待分类的图像是否属于目标类别的分类结果
在本申请实施例中，分类结果可以是二分类，也可以是多分类。以二分类为例，输出的分类结果可以是属于目标类别，或者不属于目标类别。
上述训练过程中训练的目标包括最小化分类结果与训练样本中针对Xi标注的标签Pi之间的差异。
更进一步地，在本申请实施例中，还可以由图像分类模型中的分割网络利用类别分配 矩阵对图像样本进行分割，得到预设类别的图像区域。
类别分配矩阵是基于多个聚类中心对各Token的特征表示进行重新分配，以分配到不同的聚类中得到的，因此可以将属于同一聚类中的各Token作为一个整体来对待。因此，可以将类别分配矩阵投影到K个通道，从而得到K个类别的图像区域即分割结果K为预设的大于1的正整数。其中，上述K个类别中包括分类网络对应的目标类别。
上述训练过程中训练的目标可以进一步包括：最小化预设类别的图像区域与对应区域掩码Yi之间的差异。该训练的目标可以用以在模型训练过程中辅助对分类结果的学习。
作为其中一种可实现的方式，在本说明书实施例中，可以依据上述训练目标构造损失函数，在每一轮迭代中利用损失函数的取值，采用诸如梯度下降等方式更新模型参数，直至满足预设的训练结束条件。其中训练结束条件可以包括诸如损失函数的取值小于或等于预设的损失函数阈值，迭代次数达到预设的次数阈值等。
若采用上述两个训练的目标，则可以构造总损失函数L，例如：
其中，b为一个批(batch)的训练样本的数量，体现和Yi之间的差异，体现和Pi之间的差异。
本申请上述实施例提供的图像分类方法可以应用于多种场景。例如，可以将遥感图像作为待分类的图像进行矿物晶体检测、石油探测等，即采用本申请实施例提供的方法对遥感图像进行分类，以确定遥感图像是否属于特定矿物晶体类别，同时可以对遥感图像进行分割，得到各种类别的区域，其中包括特定矿物晶体区域，从而提供可解释性参考。再例如，可以用于一些恶劣环境下的设备异常检测，即可以采用无人机、巡检机器人等设备采集设备图像，将设备图像作为待分类图像，采用本申请实施例提供的方法对设备图像进行分类，以确定是否属于异常类别，即就是否存在异常。同时可以对设备图像进行分割，得到背景区域、设备区域、异常区域等来提供可解释性参考。
还可以应用于其他场景，但其中一个较为重要的场景是医学场景，可以利用本申请实施例提供的方法对医学影像进行分类，以确定是否存在特定病灶。下面以此应用场景进行举例描述。
图6为本申请实施例提供的应用于医学场景的图像分类方法流程图，如图6中所示，该方法可以具体包括以下步骤：
步骤602：获取针对目标器官采集的医学影像。
在此采集的医学影像是采用非侵入式的采集方式，例如CT(Computed Tomography，断层扫描)、MRI(Magnetic Resonance Imaging，核磁共振)、超音波检查等。目标器官可以是胃、肺、肝脏、胰腺、结肠等器官。例如，可以是包含胃部的平扫CT(Non-Contrast CT)图像等。
以胃部的平扫CT图像为例，通常平扫CT设备扫描一圈会产生多层图像，因此，平扫CT图像X∈RH×W×D，其中H、W和D分别表示三维图像(以输入的待分类图像为三维图 像为例)的高、宽和层数。
步骤604：对医学影像进行特征提取，得到医学影像的特征表示。
步骤606：利用医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示。
上述步骤604和步骤606可以参见之前实施例中针对步骤204和步骤206的记载，在此不做赘述。
聚类中心的数量N为超参数，N的取值通常大于后续分割区域的类别数量K，因为诸如胃癌等病变可能存在多个亚种，例如胃癌的不同发展期体现出的特征是不同的，因此会取更多数量的聚类中心。N一般取经验值或实验值，例如取8。
步骤608：利用医学影像的特征表示和多个聚类中心表示进行解码处理，得到类别分配矩阵。
步骤610：利用多个聚类中心表示和类别分配矩阵进行分类，得到医学影像是否属于目标病变的分类结果。
上述步骤608和步骤610可以参见之前实施例中针对步骤208和步骤210的记载，在此不做赘述。
以胃部的平扫CT图像为例，步骤610得到的分类结果可以是一个2分类，即正常或者胃癌。从而实现了基于胃部的平扫CT图像检测是否有胃癌嫌疑的结果以提供给医师进行提醒和参考。其中，基于多分辨率的各Token的特征表示对多个聚类中心的初始特征表示进行交叉注意力处理，并将多个聚类中心的特征表示与图像的特征表示进行重新分配，使得分类过程既能够对局部组织敏感，又对器官生理学具有全局意识。
更进一步地，在本申请实施例中，还可以由图像分类模型中的分割网络利用类别分配矩阵对医学影像进行分割，得到预设类别的图像区域。其中预设类别可以包括背景、器官和病灶。仍以胃部的平扫CT图像为例，可以分割得到背景区域、胃部区域和胃癌区域，从而提供给医师作为可解释性的参考。
在本申请实施例中，还可以对医学影像进行特征提取，得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为医学影像的特征表示。
若特征提取网络得到多个不同分辨率的特征表示，则作为其中一种优选的实施方式，可以利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到多个聚类中心表示。
作为其中一种可实现的方式，可以对多个聚类中心表示进行求平均处理，得到聚类平均表示；以及对类别分配矩阵进行池化处理，得到聚类池化表示；然后将聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到医学图像是否属于 目标病变的分类结果。本申请实施例还提供一种胃癌计算机辅助诊断方法，所述方法包括：
获取针对胃部采集的医学影像；
对所述医学影像进行特征提取，得到所述医学影像的特征表示；
利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于胃癌的分类结果。
在采用图4所示流程对检测胃癌的图像分类模型进行训练时，图像样本可以是已知正常或胃癌的胃部的平扫CT图像作为各Xi，对各平扫CT图像标注正常或胃癌的标签作为Pi。由于平扫CT的区域标注较为困难，因此，可以由医师对各Xi对应的增强CT(Contrast CT)图像进行背景区域、胃部区域和胃癌区域的标注后，将标注后的增强CT图像与平扫CT图像对齐，从而形相对粗糙但可靠性高的各区域掩码。
胃癌是全球范围内癌症相关死亡的第三大原因，5年生存率约为33％，如果能够在早期检测出相关症状则可以显著提高5年生存率。由于早期胃肿瘤可能只侵犯黏膜和肌层，如果没有进行胃部造影剂注射很难识别。目前诸如钡餐胃部造影、内镜检查、血清胃蛋白酶原水平检查等已有的检测方式都是侵入式的，成本高且副作用大，难以很好地应用于早期胃癌检测。采用本申请实施例提供的上述方式，能够利用诸如胃部平扫CT图像等无创、成本低的方式来获取图像，并利用本申请实施例提供的方式，由计算机设备进行图像分类来实现胃癌检测，即最终输出的分类结果为是否属于胃癌，例如输出正常、胃癌的二分类结果，从而作为中间数据给医师或患者提供参考依据或提醒，以方便后续可以采用更进一步地检查和诊断。同时，能够对胃部平扫CT图像进行图像分割，分割得到背景区域、胃部区域以及病灶区域(即胃癌所在区域)，从而作为中间数据给医师或患者提供可解释性地参考。显然，这是一种无创、低成本且易于推广的新型检测方法，并且在精度上也有很好的效果。
经过实验，本申请实施例提供的方法在100例胃癌患者和148例正常患者的胃部平扫CT图像的测试集上，分类得到胃癌的分类结果的灵敏度为85.0％，特异性为92.6％。而放射科医生人为对胃部平扫CT图像进行胃癌判别的平均灵敏度为73.5％，特异性为84.3％，显然通过本申请实施例提供的基于计算机视觉技术的图像分类方法效果更优。其中灵敏度又称为真阳性率，是指对于胃癌样本进行测试的分类结果为胃癌的概率。特异性又称为真阴性率，是指对于正常个样本进行测试的分类结果为正常的概率。
上述对本说明书特定实施例进行了描述。其它实施例在所附权利要求书的范围内。在一些情况下，在权利要求书中记载的动作或步骤可以按照不同于实施例中的顺序来执行并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要求示出的特定顺序或者 连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的或者可能是有利的。
参见下述表1，表1输出了本公开的图像分类方法与三个基准的比较分析。前两个基准使用的两种方法属于“分类分割”(S4C)，使用nnUNet和TransUNet。其中，表示DeLong测试(本公开与nnUNet-S4C)的p<0.05；*表示排列检验(本公开与nnUNet-S4C和放射科医生)的p<0.05。如果分割的肿瘤体积超过使验证集的灵敏度和特异性之和最大化的阈值，则病例被分类为阳性。第三条基线(表示为“nnUNet-Joint”)将CNN分类头集成到UNet，并进行端到端训练。可以从测试数据集的1000个bootstrap副本中获得了AUC、灵敏度和特异性值的95％置信区间，以进行统计分析。为了获得统计显着性，可以在两个AUC(本公开的图像分类方法与比较方法)之间进行DeLong检验，并在两个灵敏度或特异性(本公开的图像分类方法与比较方法和放射科医生)之间进行排列检验。
表1
参照表1所示，本公开的图像分类方法在所有指标上都优于其他三个基准(表1)，特别是在AUC和灵敏度方面。本公开的方法的优点是，它凭借掩模Transformer的独特架构同时捕获局部和全局信息。它还从集群表示中提取高级语义，使其适合分类并促进整体决策过程。此外，本公开的方法在外部测试集上达到了97.7％的相当大的特异性，这对于减少在机会性筛选方面的误报和不必要的人力工作量的至关重要。
本公开提出的模型的ROC曲线优于由两名经验丰富的放射科医生进行检测的结果。该模型在检测胃癌方面达到了85.0％的灵敏度，显着超过了医生的平均表现(73.5％)，也超过了表现最好的医生(75.0％)，同时保持高特异性。
参照表2所示，按肿瘤(T)分期分层的患者水平检测和肿瘤水平定位的表现。表2将本公开的模型的性能与两位放射科医生的性能进行比较。结果表明，本公开的模型在检测早期肿瘤(T1、T2)方面表现更好，并提供更精确的肿瘤定位。具体来说，本公开的模型检测出60.0％(6/10)T1癌症和77.8％(7/9)T2癌症，超过了表现最好的专家(50％ T1，55.6％ T2)。同时，本公开的模型对T3和T4肿瘤保持了可靠的检测率和可信 的定位精度(34个T3肿瘤中有2个漏检)。
表2不同T阶段胃癌的患者层面检测和肿瘤层面定位结果(％)
如表3所示，在相对较大的测试患者规模(通过整合内部和外部测试集，n＝1151)的情况下，本公开的方法在胃癌检测的敏感性方面超越或与现有的筛查工具相当，具有相似的特异性水平，如表3所示。其中，*表示在测试集中将两个原位肿瘤移除。表示该比较中只考虑早期胃癌病例，包括原位肿瘤、T1和T2期，其中19例中本公开的方法成功检出17例。
表3在大规模人群中将最先进的血液检测应用在胃癌检测方面、UGIS和内窥镜筛查的表现，以及高级放射科医生在窄带成像上使用放大胃镜(magnifying endoscopy，ME-NBI)的早期胃癌检出率比较
根据另一方面的实施例，提供了一种图像分类装置。图7示出根据一个实施例的图像分类装置的示意性框图。如图7所示，该装置700包括：图像获取单元701、特征提取单元702、第一解码单元703、第二解码单元704和图像分类单元705，还可以进一步图像分割单元706。其中各组成单元的主要功能如下：
图像获取单元701，被配置为获取待分类的图像。
特征提取单元702，被配置为对待分类的图像进行特征提取，得到图像的特征表示。
第一解码单元703，被配置为利用图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示。
第二解码单元704，被配置为利用图像的特征表示和多个聚类中心表示进行解码处理， 得到类别分配矩阵。
图像分类单元705，被配置为利用多个聚类中心表示和类别分配矩阵进行分类，得到待分类的图像是否属于目标类别的分类结果。
更进一步地，图像分割单元706，被配置为利用类别分配矩阵对待分类的图像进行分割，得到预设类别的图像区域，预设类别包括目标类别。
作为其中一种可实现的方式，上述特征提取单元702可以具体被配置为：对待分类的图像进行特征提取，得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为图像的特征表示。
相应地，第一解码单元703可以利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到多个聚类中心表示。
作为其中一种可实现的方式，图像分类单元705，可以具体被配置为对多个聚类中心表示进行求平均处理，得到聚类平均表示；对类别分配矩阵进行池化处理，得到聚类池化特征；将聚类平均表示和聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到待分类的图像是否属于目标类别的分类结果。
上述特征提取单元702、第一解码单元703、第二解码单元704、图像分类单元705和图像分割单元706分别对应于图3所示图像分类模型中的特征提取网络、第一解码网络、第二解码网络、分类网络和分割网络，具体细节可以参见上述方法实施例中针对图3的相关记载，在此不做赘述。
图8为本申请实施例提供的训练图像分类模型的装置的示意性框图，如图8中所示，该装置可以包括：样本获取单元801和模型训练单元802。其中各组成单元的主要功能如下：
样本获取单元801，被配置为获取包括多个训练样本的训练数据，训练样本包括图像样本以及图像样本是否属于目标类别的标签。
模型训练单元802，被配置为利用训练数据训练图像分类模型，图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；特征提取网络对图像样本进行特征提取，得到图像样本的特征表示；第一解码网络利用图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；第二解码网络利用图像样本的特征表示和多个聚类中心表示进行解码处理，得到类别分配矩阵；分类网络利用多个聚类中心表示和类别分配矩阵进行分类，得到图像样本是否属于目标类别的分类结果；训练的目标包括：最小化分类结果与对应标签之间的差异。
作为其中一种可实现的方式，上述训练样本还可以包括对图像样本标注的预设类别的 区域掩码。图像分类模型还包括可以分割网络。
分割网络利用类别分配矩阵对图像样本进行分割，得到预设类别的图像区域，预设类别包括目标类别。相应地，上述训练的目标还可以包括：最小化预设类别的图像区域与对应区域掩码之间的差异。
作为其中一种可实现的方式，特征提取网络可以对图像样本进行特征提取，依次得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为图像样本的特征表示。
相应地，第一解码网络可以包括多层串连的Transformer网络，多个聚类中心的初始表示作为查询矩阵输入第一层Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到多个聚类中心表示。
作为其中一种可实现的方式，分类网络可以对多个聚类中心表示进行求平均处理，得到聚类平均表示；对类别分配矩阵进行池化处理，得到聚类池化表示；将聚类平均表示和聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到图像样本是否属于目标类别的分类结果。
根据另一方面的实施例，提供了一种图像分类装置，应用于医学诊断。该装置包括：图像获取单元、特征提取单元、第一解码单元、第二解码单元和图像分类单元，还可以进一步图像分割单元。其中各组成单元的主要功能如下：
图像获取单元，被配置为获取针对目标器官采集的医学影像；
特征提取单元，被配置为对所述医学影像进行特征提取，得到所述医学影像的特征表示；
第一解码单元，被配置为利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
第二解码单元，被配置为利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
图像分类单元，被配置为利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于装置实施例而言，由于其基本相似于方法实施例，所以描述得比较简单，相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需 要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
需要说明的是，本申请所涉及的用户信息(包括但不限于用户设备信息、用户个人信息等)和数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)，均为经用户授权或者经过各方充分授权的信息和数据，并且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准，并提供有相应的操作入口，供用户选择授权或者拒绝。
另外，本申请实施例还提供了一种癌症计算机辅助诊断系统，包括存储器、处理器以及存储在所述存储器上并在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序可用于执行一种图像分类方法，所述方法包括：
获取针对目标器官采集的医学影像；
对所述医学影像进行特征提取，得到所述医学影像的特征表示；
利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
另外，本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现前述方法实施例中任一项所述的方法的步骤。
以及一种电子设备，包括：
一个或多个处理器；以及
与所述一个或多个处理器关联的存储器，所述存储器用于存储程序指令,所述程序指令在被所述一个或多个处理器读取执行时，执行前述方法实施例中任一项所述的方法的步骤。
本申请还提供了一种计算机程序产品，包括计算机程序，该计算机程序在被处理器执行时实现前述方法实施例中任一项所述的方法的步骤。
本申请还提供了一种计算机程序，其中，当所述计算机程序在计算机中执行时，令计算机执行前述方法实施例中任一项所述的方法的步骤。
其中，图9示例性的展示出了电子设备的架构，具体可以包括处理器910，视频显示适配器911，磁盘驱动器912，输入/输出接口913，网络接口914，以及存储器920。上述处理器910、视频显示适配器911、磁盘驱动器912、输入/输出接口913、网络接口914，与存储器920之间可以通过通信总线930进行通信连接。
其中，处理器910可以采用通用的CPU、微处理器、应用专用集成电路(Application Specific Integrated Circuit，ASIC)、或者一个或多个集成电路等方式实现，用于执行相关程序，以实现本申请所提供的技术方案。
存储器920可以采用ROM(Read Only Memory，只读存储器)、RAM(Random Access Memory，随机存取存储器)、静态存储设备，动态存储设备等形式实现。存储器920可以存储用于控制电子设备900运行的操作系统921，用于控制电子设备900的低级别操作的基本输入输出系统(BIOS)922。另外，还可以存储网页浏览器923，数据存储管理系统924，以及图像分类装置/模型训练装置925等等。上述图像分类装置/模型训练装置925就可以是本申请实施例中具体实现前述各步骤操作的应用程序。总之，在通过软件或者固件来实现本申请所提供的技术方案时，相关的程序代码保存在存储器920中，并由处理器910来调用执行。
输入/输出接口913用于连接输入/输出模块，以实现信息输入及输出。输入输出/模块可以作为组件配置在设备中(图中未示出)，也可以外接于设备以提供相应功能。其中输入设备可以包括键盘、鼠标、触摸屏、麦克风、各类传感器等，输出设备可以包括显示器、扬声器、振动器、指示灯等。
网络接口914用于连接通信模块(图中未示出)，以实现本设备与其他设备的通信交互。其中通信模块可以通过有线方式(例如USB、网线等)实现通信，也可以通过无线方式(例如移动网络、WIFI、蓝牙等)实现通信。
总线930包括一通路，在设备的各个组件(例如处理器910、视频显示适配器911、磁盘驱动器912、输入/输出接口913、网络接口914，与存储器920)之间传输信息。
需要说明的是，尽管上述设备仅示出了处理器910、视频显示适配器911、磁盘驱动器912、输入/输出接口913、网络接口914，存储器920，总线930等，但是在具体实施过程中，该设备还可以包括实现正常运行所必需的其他组件。此外，本领域的技术人员可以理解的是，上述设备中也可以仅包含实现本申请方案所必需的组件，而不必包含图中所示的全部组件。
通过以上的实施方式的描述可知，本领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件平台的方式来实现。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机程序产品的形式体现出来，该计算机程序产品可以存储在存储介质中，如ROM/RAM、磁碟、光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例或者实施例的某些部分所述的方法。
以上对本申请所提供的技术方案进行了详细介绍，本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处。综上所述，本说明书内容不应理解为对本申请的限制。

Claims (24)

  1. 一种图像分类方法，其特征在于，所述方法包括：
    获取待分类的图像；
    对所述待分类的图像进行特征提取，得到所述图像的特征表示；
    利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    利用所述类别分配矩阵对所述待分类的图像进行分割，得到预设类别的图像区域，所述预设类别包括所述目标类别。
  3. 根据权利要求1或2所述的方法，其特征在于，对所述待分类的图像进行特征提取，得到所述图像的特征表示包括：
    对所述待分类的图像进行特征提取，得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为所述图像的特征表示。
  4. 根据权利要求3所述的方法，其特征在于，利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理包括：
    利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；
    利用最后一层Transformer网络输出的查询矩阵得到所述多个聚类中心表示。
  5. 根据权利要求1至4中任一项所述的方法，其特征在于，利用所述多个聚类中心表示和所述类别分配矩阵进行分类包括：
    对所述多个聚类中心表示进行求平均处理，得到聚类平均表示；
    对所述类别分配矩阵进行池化处理，得到聚类池化特征；
    将所述聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
  6. 一种训练图像分类模型的方法，其特征在于，所述方法包括：
    获取包括多个训练样本的训练数据，所述训练样本包括图像样本以及所述图像样本是否属于目标类别的标签；
    利用所述训练数据训练图像分类模型，所述图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；所述特征提取网络对所述图像样本进行特征提取， 得到所述图像样本的特征表示；所述第一解码网络利用所述图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；所述第二解码网络利用所述图像样本的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述图像样本是否属于目标类别的分类结果；所述训练的目标包括：最小化所述分类结果与对应标签之间的差异。
  7. 根据权利要求6所述的方法，其特征在于，所述训练样本还包括对所述图像样本标注的预设类别的区域掩码；所述图像分类模型还包括分割网络；
    所述分割网络利用所述类别分配矩阵对所述图像样本进行分割，得到预设类别的图像区域，所述预设类别包括所述目标类别；
    所述训练的目标还包括：最小化所述预设类别的图像区域与对应区域掩码之间的差异。
  8. 根据权利要求6或7所述的方法，其特征在于，所述特征提取网络对所述图像样本进行特征提取，得到所述图像样本的特征表示包括：对所述图像样本进行特征提取，依次得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为所述图像样本的特征表示；
    所述第一解码网络包括多层串连的Transformer网络，所述多个聚类中心的初始表示作为查询矩阵输入第一层Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；利用最后一层Transformer网络输出的查询矩阵得到所述多个聚类中心表示。
  9. 根据权利要求6至8中任一项所述的方法，其特征在于，所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分类包括：
    所述分类网络对所述多个聚类中心表示进行求平均处理，得到聚类平均表示；
    对所述类别分配矩阵进行池化处理，得到聚类池化表示；
    将所述聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到所述图像样本是否属于目标类别的分类结果。
  10. 一种图像分类方法，由云端服务器执行，其特征在于，所述方法包括：
    获取来自用户终端的待分类的图像；
    对所述待分类的图像进行特征提取，得到所述图像的特征表示；
    利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果；
    将所述分类结果返回给所述用户终端。
  11. 一种癌症计算机辅助诊断方法，其特征在于，所述方法包括：
    获取针对目标器官采集的医学影像；
    对所述医学影像进行特征提取，得到所述医学影像的特征表示；
    利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
  12. 根据权利要求11所述的方法，其特征在于，所述方法还包括：
    利用所述类别分配矩阵对所述医学影像进行分割，得到预设类别的图像区域，所述预设类别包括所述目标病变。
  13. 根据权利要求11或12所述的方法，其特征在于，对所述医学影像进行特征提取，得到所述医学影像的特征表示包括：
    对所述医学影像进行特征提取，得到多种分辨率下的各元素Token的特征表示，将最高分辨率下的各Token的特征表示作为所述医学影像的特征表示。
  14. 根据权利要求13所述的方法，其特征在于，利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理包括：
    利用多个聚类中心的初始表示得到查询矩阵，将查询矩阵输入多层串连的转换Transformer网络，各层Transformer网络按照分辨率从低到高的顺序与各分辨率一一对应，各层Transformer网络利用对应分辨率下的各Token的特征表示得到键矩阵和值矩阵，对输入本层Transformer网络的查询矩阵进行交叉注意力处理，得到本层Transformer网络输出的查询矩阵；
    利用最后一层Transformer网络输出的查询矩阵得到所述多个聚类中心表示。
  15. 根据权利要求11至14中任一项所述的方法，其特征在于，利用所述多个聚类中心表示和所述类别分配矩阵进行分类包括：
    对所述多个聚类中心表示进行求平均处理，得到聚类平均表示；
    对所述类别分配矩阵进行池化处理，得到聚类池化特征；
    将所述聚类平均表示和所述聚类池化表示进行整合，利用整合得到的特征表示进行分类，得到所述医学影像是否属于目标病变的分类结果。
  16. 一种胃癌计算机辅助诊断方法，所述方法包括：
    获取针对胃部采集的医学影像；
    对所述医学影像进行特征提取，得到所述医学影像的特征表示；
    利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉 注意力处理，得到多个聚类中心表示；
    利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于胃癌的分类结果。
  17. 一种癌症计算机辅助诊断系统，包括存储器、处理器以及存储在所述存储器上并在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序可用于执行一种癌症计算机方法，所述方法包括：
    获取针对目标器官采集的医学影像；
    对所述医学影像进行特征提取，得到所述医学影像的特征表示；
    利用所述医学影像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    利用所述医学影像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述医学影像是否属于目标病变的分类结果。
  18. 一种图像分类装置，其特征在于，所述装置包括：
    图像获取单元，被配置为获取待分类的图像；
    特征提取单元，被配置为对所述待分类的图像进行特征提取，得到所述图像的特征表示；
    第一解码单元，被配置为利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    第二解码单元，被配置为利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    图像分类单元，被配置为利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
  19. 一种训练图像分类模型的装置，其特征在于，所述装置包括：
    样本获取单元，被配置为获取包括多个训练样本的训练数据，所述训练样本包括图像样本以及所述图像样本是否属于目标类别的标签；
    模型训练单元，被配置为利用所述训练数据训练图像分类模型，所述图像分类模型包括：特征提取网络、第一解码网络、第二解码网络和分类网络；所述特征提取网络对所述图像样本进行特征提取，得到所述图像样本的特征表示；所述第一解码网络利用所述图像样本的特征表示对多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；所述第二解码网络利用所述图像样本的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；所述分类网络利用所述多个聚类中心表示和所述类别分配矩阵进行分 类，得到所述图像样本是否属于目标类别的分类结果；所述训练的目标包括：最小化所述分类结果与对应标签之间的差异。
  20. 一种图像分类系统，包括存储器、处理器以及存储在所述存储器上并在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序可用于执行一种图像分类方法，所述方法包括：
    获取待分类的图像；
    对所述待分类的图像进行特征提取，得到所述图像的特征表示；
    利用所述图像的特征表示对预先训练得到的多个聚类中心的初始表示进行交叉注意力处理，得到多个聚类中心表示；
    利用所述图像的特征表示和所述多个聚类中心表示进行解码处理，得到类别分配矩阵；
    利用所述多个聚类中心表示和所述类别分配矩阵进行分类，得到所述待分类的图像是否属于目标类别的分类结果。
  21. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现权利要求1至16中任一项所述的方法的步骤。
  22. 一种电子设备，其特征在于，包括：
    一个或多个处理器；以及
    与所述一个或多个处理器关联的存储器，所述存储器用于存储程序指令,所述程序指令在被所述一个或多个处理器读取执行时，执行权利要求1至16中任一项所述的方法的步骤。
  23. 一种计算机程序产品，包括计算机可执行指令，该计算机可执行指令被处理器执行时实现权利要求1至16任一项所述方法的步骤。
  24. 一种计算机程序，该计算机程序被处理器执行时实现权利要求1至16任一项所述方法的步骤。
PCT/CN2024/103726 2023-07-04 2024-07-04 图像分类方法、训练图像分类模型的方法及装置 Ceased WO2025007941A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310822213.1 2023-07-04
CN202310822213.1A CN116958659B (zh) 2023-07-04 2023-07-04 图像分类方法、训练图像分类模型的方法及装置

Publications (1)

Publication Number Publication Date
WO2025007941A1 true WO2025007941A1 (zh) 2025-01-09

Family

ID=88452204

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/103726 Ceased WO2025007941A1 (zh) 2023-07-04 2024-07-04 图像分类方法、训练图像分类模型的方法及装置

Country Status (3)

Country Link
CN (1) CN116958659B (zh)
TW (1) TW202503689A (zh)
WO (1) WO2025007941A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119942245A (zh) * 2025-04-03 2025-05-06 温州大学 一种基于功能性核磁共振成像进行图像分类的方法
CN120088567A (zh) * 2025-03-03 2025-06-03 浙江大学 多任务自身免疫性胃炎胃镜图像分类方法、设备、介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116958659B (zh) * 2023-07-04 2024-04-02 阿里巴巴达摩院(杭州)科技有限公司 图像分类方法、训练图像分类模型的方法及装置
CN119648712B (zh) * 2024-11-27 2025-09-16 海南省航天技术创新中心 聚类驱动的光学遥感影像全景分割方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113971735A (zh) * 2021-09-16 2022-01-25 西安电子科技大学 一种深度图像聚类方法、系统、设备、介质及终端
WO2022127227A1 (zh) * 2020-12-15 2022-06-23 西安交通大学 一种基于多视图半监督的淋巴结的分类方法、系统及设备
CN115239675A (zh) * 2022-07-29 2022-10-25 平安科技（深圳）有限公司 分类模型的训练方法、图像分类方法和装置、设备及介质
CN115631183A (zh) * 2022-11-07 2023-01-20 华东理工大学 基于双路解码器实现针对x光片图像的分类识别的方法、系统、装置、处理器及存储介质
US20230186600A1 (en) * 2021-12-09 2023-06-15 Vinai Artificial Intelligence Application And Research Joint Stock Company Method of clustering using encoder-decoder model based on attention mechanism and storage medium for image recognition
CN116958659A (zh) * 2023-07-04 2023-10-27 阿里巴巴达摩院(杭州)科技有限公司 图像分类方法、训练图像分类模型的方法及装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114283289A (zh) * 2021-07-07 2022-04-05 中国人民解放军火箭军工程大学 一种基于多模型融合的图像分类方法
CN113807412B (zh) * 2021-08-30 2023-09-26 广州大学 一种多标签图像分类方法、装置、设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022127227A1 (zh) * 2020-12-15 2022-06-23 西安交通大学 一种基于多视图半监督的淋巴结的分类方法、系统及设备
CN113971735A (zh) * 2021-09-16 2022-01-25 西安电子科技大学 一种深度图像聚类方法、系统、设备、介质及终端
US20230186600A1 (en) * 2021-12-09 2023-06-15 Vinai Artificial Intelligence Application And Research Joint Stock Company Method of clustering using encoder-decoder model based on attention mechanism and storage medium for image recognition
CN115239675A (zh) * 2022-07-29 2022-10-25 平安科技（深圳）有限公司 分类模型的训练方法、图像分类方法和装置、设备及介质
CN115631183A (zh) * 2022-11-07 2023-01-20 华东理工大学 基于双路解码器实现针对x光片图像的分类识别的方法、系统、装置、处理器及存储介质
CN116958659A (zh) * 2023-07-04 2023-10-27 阿里巴巴达摩院(杭州)科技有限公司 图像分类方法、训练图像分类模型的方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120088567A (zh) * 2025-03-03 2025-06-03 浙江大学 多任务自身免疫性胃炎胃镜图像分类方法、设备、介质
CN119942245A (zh) * 2025-04-03 2025-05-06 温州大学 一种基于功能性核磁共振成像进行图像分类的方法

Also Published As

Publication number Publication date
CN116958659A (zh) 2023-10-27
TW202503689A (zh) 2025-01-16
CN116958659B (zh) 2024-04-02

Similar Documents

Publication Publication Date Title
Zhou et al. Automatic COVID‐19 CT segmentation using U‐Net integrated spatial and channel attention mechanism
US20260004933A1 (en) Systems and methods for analysis of medical images for scoring of inflammatory bowel disease
Yogapriya et al. Gastrointestinal tract disease classification from wireless endoscopy images using pretrained deep learning model
Debelee et al. Deep learning in selected cancers’ image analysis—a survey
WO2025007941A1 (zh) 图像分类方法、训练图像分类模型的方法及装置
Pogorelov et al. Efficient disease detection in gastrointestinal videos–global features versus neural networks
Ghosh et al. Deep transfer learning for automated intestinal bleeding detection in capsule endoscopy imaging
Sheu et al. Interpretable classification of pneumonia infection using eXplainable AI (XAI-ICP)
Liu et al. Detection and diagnosis of colitis on computed tomography using deep convolutional neural networks
CN111986189B (zh) 一种基于ct影像的多类别肺炎筛查深度学习装置
Dash et al. Cervical transformation zone segmentation and classification based on improved Inception-ResNet-V2 using colposcopy images
Aria et al. ADA‐COVID: Adversarial Deep Domain Adaptation‐Based Diagnosis of COVID‐19 From Lung CT Scans Using Triplet Embeddings
Hu et al. Content‐based gastric image retrieval using convolutional neural networks
Liu et al. An xception model based on residual attention mechanism for the classification of benign and malignant gastric ulcers
Liu et al. TN‐USMA Net: Triple normalization‐based gastrointestinal stromal tumors classification on multicenter EUS images with ultrasound‐specific pretraining and meta attention
Sun et al. COVID-19 CT image segmentation method based on swin transformer
Du et al. Improving the classification performance of esophageal disease on small dataset by semi-supervised efficient contrastive learning
Choi et al. Mask R-CNN based multiclass segmentation model for endotracheal intubation using video laryngoscope
Chen et al. Identifying cardiomegaly in chest x-rays using dual attention network
Li et al. Developing an image-based deep learning framework for automatic scoring of the pentagon drawing test
Gao et al. Application of deep learning for early screening of colorectal precancerous lesions under white light endoscopy
Xue et al. Using deep learning for detecting gender in adult chest radiographs
CN115994212A (zh) 视觉问答处理方法、视觉问答模型的训练方法及装置
Uma Maheswari et al. Chaotic satin bowerbird optimizer based advanced AI techniques for detection of COVID-19 diseases from CT scans images
Li et al. Anatomical partition‐based deep learning: an automatic nasopharyngeal MRI recognition scheme

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24835419

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE