CN113255766B - Image classification method, device, equipment and storage medium - Google Patents
Image classification method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113255766B CN113255766B CN202110570158.2A CN202110570158A CN113255766B CN 113255766 B CN113255766 B CN 113255766B CN 202110570158 A CN202110570158 A CN 202110570158A CN 113255766 B CN113255766 B CN 113255766B
- Authority
- CN
- China
- Prior art keywords
- class
- decoupling
- feature map
- image
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000000605 extraction Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 10
- 238000010586 diagram Methods 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 19
- 230000004913 activation Effects 0.000 claims description 18
- 230000004927 fusion Effects 0.000 claims description 11
- 238000005096 rolling process Methods 0.000 claims description 5
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 21
- 206010003694 Atrophy Diseases 0.000 description 10
- 230000037444 atrophy Effects 0.000 description 10
- 238000012549 training Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 238000005286 illumination Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 208000001491 myopia Diseases 0.000 description 2
- 230000004379 myopia Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The application is applicable to the technical field of image processing and provides an image classification method, an image classification system and a storage medium. The image classification method comprises the following steps: extracting a multi-channel feature map of an image to be classified; decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps; obtaining the relationship among classes according to the single-class decoupling feature graphs obtained by decoupling; aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map; and determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map. According to the method and the device, through feature decoupling and inter-class relation extraction, the existence probability of the features which possibly coexist is calculated independently, whether the feature labels exist or not is judged respectively, and the accuracy of classifying the image features is improved.
Description
Technical Field
The application belongs to the technical field of image processing, and particularly relates to an image classification method, an image classification device, image classification equipment and a storage medium.
Background
In an image recognition scene, several similar features often appear on one image at a time. When the existing machine learning model learns the samples, the coexistence phenomenon among the features can be learned. The machine learning model is used for identifying the image, and a plurality of features with more coexistence phenomena in the sample are identified, so that the model usually considers that when one feature appears, other features also appear, and different features are difficult to identify independently, and the result confusion, namely the classification accuracy is influenced.
Taking the recognition of myopia fundus color photographs as an example, at present, the recognition of myopia fundus color photographs can be realized by training a multi-label classified convolutional neural network. A single sample in the training set may contain one or more classification tags in the leopard-like fundus, diffuse atrophy, plaque atrophy, and macular atrophy simultaneously; however, in the model training process, the dependency relationship among the classification labels is not considered, and when a large number of categories coexist in the training set, the convolutional neural network is caused to learn a large number of coupling features, so that different feature labels are difficult to distinguish independently in reasoning prediction, and label categories with high coexistence frequency are easy to be confused. For example, the training set contains many images in which leopard-shaped eyeground and plaque atrophy coexist, and when the network model estimates an image with leopard-shaped eyeground, the network model tends to consider that plaque atrophy exists simultaneously, which leads to deviation of the result of classification prognosis.
Disclosure of Invention
The embodiment of the application provides an image classification method, an image classification device, image classification equipment and a storage medium, which can improve the accuracy of image classification.
In a first aspect, an embodiment of the present application provides an image classification method, including:
extracting a multi-channel feature map of an image to be classified;
decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps;
obtaining the relationship among classes according to the single-class decoupling feature graphs obtained by decoupling;
aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map;
and determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map.
The decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps includes:
performing convolution operation on the multi-channel feature map to obtain a convolution feature map;
processing the convolution feature images based on a preset activation function to obtain a plurality of single-class feature images;
and multiplying each single-class feature map with the multi-channel feature map to obtain a single-class decoupling feature map.
Specifically, the multi-channel feature map includes a shallow feature map and a deep feature map, and the single-class decoupling feature map includes a single-class shallow decoupling feature map and a single-class deep decoupling feature map;
the decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps includes:
and respectively decoupling the shallow layer feature map and the deep layer feature map to obtain a single type shallow layer decoupling feature map and a single type deep layer decoupling feature map.
Illustratively, the inter-class relationships include an inter-class relationship matrix;
the obtaining the inter-class relationship according to the single-class decoupling feature graphs obtained by decoupling includes:
fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relationship matrix;
the calculating, for each single-class decoupling feature map, a classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map, includes:
inputting the single-class deep decoupling feature images and the relation matrix between classes into a preset image convolution network to obtain a first classification probability of each single-class decoupling feature image;
the determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map comprises the following steps:
and determining the category of the image to be classified according to the first classification probability of each single-category decoupling feature map.
As a possible implementation manner, after extracting the multi-channel feature map of the image to be classified, the method further includes:
calculating a second classification probability of the multi-channel feature map through a full connection layer;
fusing the first classification probability and the second classification probability to obtain a third classification probability;
the determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map comprises the following steps:
and determining the category of the image to be classified according to the third classification probability of each single-category decoupling feature map.
In a second aspect, an embodiment of the present application provides an image classification apparatus, including:
the feature extraction module is used for extracting a multi-channel feature map of the image to be classified;
the characteristic decoupling module is used for decoupling the multi-channel characteristic diagrams to obtain a plurality of single-class decoupling characteristic diagrams;
the inter-class relation extraction module is used for obtaining the inter-class relation according to each single-class decoupling characteristic diagram obtained by decoupling; aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map;
and the classification module is used for determining the class of the image to be classified according to the classification probability of each single-class decoupling feature map.
Wherein, the characteristic decoupling module includes:
the convolution unit comprises a convolution network with a 1*1 convolution kernel and is used for carrying out convolution operation on the multichannel characteristic map to obtain a convolution characteristic map;
the activation unit comprises an activation function and is used for processing the convolution feature images based on a preset activation function to obtain a plurality of single-class feature images;
and the decoupling unit is used for multiplying each single-class feature map with the multi-channel feature map respectively to obtain single-class decoupling feature maps.
The single-class decoupling feature map comprises a single-class shallow decoupling feature map and a single-class deep decoupling feature map;
correspondingly, the inter-class relation extraction module comprises:
the fusion unit is used for fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relation matrix;
and the graph rolling unit is used for inputting the single-class deep decoupling feature graphs and the inter-class relation matrix into a preset graph rolling network to obtain a first classification probability of each single-class decoupling feature graph.
In a third aspect, an embodiment of the present application provides an image classification apparatus, including: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the image classification method according to any one of the first aspects above when the computer program is executed.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, comprising: the computer readable storage medium stores a computer program which, when executed by a processor, implements the image classification method as described in the first aspect above.
In a fifth aspect, embodiments of the present application provide a computer program product, which, when run on a terminal device, causes the terminal device to perform the image classification method according to any one of the first aspects above.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Compared with the prior art, the embodiment of the application has the beneficial effects that: the method comprises the steps of decoupling a multi-channel feature map of an image to obtain a plurality of single-class feature maps, extracting the inter-class relation of the single-class feature maps, and determining the possibility of each feature label of the image according to the inter-class relation so as to classify the labels of the image. Through feature decoupling and inter-class relation extraction, the probability of existence of the features which possibly coexist is calculated independently, whether the feature labels exist or not is judged respectively, and accuracy of classifying the image features is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly introduce the drawings that are needed in the embodiments or the description of the prior art, it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an image classification method according to an embodiment of the present application;
FIG. 2 is a flow chart of an image classification method according to another embodiment of the present application;
FIG. 3 is a schematic diagram of an image classification apparatus according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a convolutional network model provided in an embodiment of the present application;
fig. 5 is a schematic diagram of an image classification apparatus according to another embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
The embodiment provides an image classification method which is suitable for scenes in which feature recognition is performed on images and label classification is performed according to image features. For example, classification of trees in scenery, classification of gender or age of photographs of people, classification of food in refrigerators, etc., are particularly useful for classifying image representations of bottom-of-eye photographs. The category of the image is represented by category labels, which correspond to different features of the image, and in fundus illumination classification, category labels include leopard-like, diffuse atrophy, plaque atrophy, macular atrophy, and the like, and in different fundus illumination, a plurality of category labels may coexist.
The images to be classified are acquired by an image acquisition device, which can be a digital camera, a mobile phone, a fundus camera and the like according to specific application scenes. The image classification method is executed by a corresponding image classification device, the image acquisition equipment and the image classification device can be independent equipment, communication connection is established between the image acquisition equipment and the image classification device, the image classification device can be equipment with image processing capability such as a PC (personal computer), a tablet personal computer, a server, a cloud computing terminal and the like, and a memory and a processor are integrated on the image classification device; in other implementations of the present application, the image classification device may be integrated with the image acquisition apparatus. In the fundus color illumination classification scene, fundus color illumination collection is carried out through a fundus camera, and fundus color illumination is classified through a PC.
When the image acquisition equipment and the image classification device are independently arranged, the image acquisition equipment and the image classification device are also respectively provided with a communication unit, the communication unit provides wired or wireless communication between the image acquisition equipment and the image classification device, and the image acquisition equipment sends the acquired image to be classified to the image classification device.
Next, an exemplary description will be given taking classification for fundus illumination as an example. Fig. 1 is a flowchart of an image classification method according to the present embodiment. As shown in fig. 1, the image classification method is performed by an image classification apparatus, and includes the steps of:
s11, extracting a multi-channel feature map of the image to be classified.
And inputting the images to be classified into a backbone neural network (backbone) to extract space semantic features, and obtaining a multi-channel feature map. The input image to be classified is typically a three-channel color image, and the multi-channel feature map includes a shallow feature map and a deep feature map.
The backup adopts a network structure of Resnet50, resnet50 comprises four groups of residual blocks, an image to be classified is taken as the input of a 1 st group of residual blocks, the output of the 1 st group of residual blocks is taken as the input of a 2 nd group of residual blocks, the output of the 2 nd group of residual blocks is taken as the input of a 3 rd group of residual blocks, and the like, shallow characteristic diagrams are respectively output from the convolution calculation results of the 2 nd group of residual blocks, and deep characteristic diagrams are output from the convolution calculation results of the 4 th group of residual blocks.
S12, decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps.
Performing convolution operation on the multi-channel feature map through a convolution network with a 1*1 convolution kernel to extract features, so as to obtain a convolution feature map; 1*1 convolution kernels can reduce the number of channels of a feature map to match the number of class labels without changing the size of the feature map. And inputting the feature graphs with the same number of channels and labels into a preset activation function, and outputting a plurality of single-class feature graphs corresponding to the classified labels by the activation function. And multiplying each single-class feature map with the multi-channel feature map to obtain a single-class decoupling feature map.
Correspondingly, the shallow feature map and the deep feature map are subjected to feature decoupling respectively, and the obtained single-class decoupling feature map comprises a single-class shallow decoupling feature map and a single-class deep decoupling feature map corresponding to each class label.
S13, obtaining the relationship among the classes according to the decoupling characteristic diagrams of the single classes obtained by decoupling.
And fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map of each class label through concat fusion operation to generate an inter-class relation matrix. The inter-class relationship of the class labels is represented by an inter-class relationship matrix.
S14, aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map.
Inputting the single-class deep decoupling feature images and the relation matrix between classes into a preset image convolution network, and calculating to obtain a first classification probability of each single-class decoupling feature image. The first classification probability is a classification probability based on a relationship between classes.
And S15, determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map.
And setting probability thresholds for different class labels respectively, and considering that the image features in the image to be classified have the class labels when the classification probability of the class labels reaches the probability thresholds. For example, when the classification probability of the single feature map of the plaque atrophy reaches a probability threshold, the fundus color photograph is considered to have a label of the plaque atrophy.
According to the embodiment, the multi-channel feature images of the images are decoupled, a plurality of single-class decoupling feature images corresponding to class labels are obtained, the inter-class relations of the single-class decoupling feature images are extracted, and the label coexistence situation of the images is determined according to the inter-class relations so as to classify the labels of the images. The classification mode considers objective relations among the class labels, avoids misjudgment of the trained model on coexistence conditions of the class labels caused by sample imbalance in the traditional convolutional network model training, and improves accuracy of classifying the image features.
Fig. 2 is a flow chart of an image classification method according to another embodiment. On the basis of the embodiment, for the situation that the image features in the images to be classified are not obvious in distinction and are mixed and coexist more, the relationship between the classes and the image features need to be comprehensively considered for more accurate classification.
As shown in fig. 2, the image classification method includes the steps of:
s21, extracting a multi-channel feature map of the image to be classified.
The multi-channel feature map includes a shallow feature map and a deep feature map.
S22, calculating the second classification probability of the multi-channel feature map through the full connection layer.
The full connection layer is used for performing classification tasks based on features and calculating second classification probabilities of the multi-channel feature graphs. The second classification probability is a classification probability based on the image features.
S23, decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps.
The decoupling shallow feature map obtains a single-class shallow decoupling feature map, and the decoupling deep feature map obtains a single-class deep decoupling feature map.
S24, obtaining the relationship among the classes according to the decoupling characteristic diagrams of the single classes obtained by decoupling.
And fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relationship matrix for representing the inter-class relationship.
S25, calculating first classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map aiming at each single-class decoupling feature map.
And inputting the single-class deep decoupling feature images and the relation matrix between classes into a preset image convolution network to obtain the first classification probability of each single-class decoupling feature image.
S26, fusing the first classification probability and the second classification probability to obtain a third classification probability.
For the class labels corresponding to each single class decoupling feature map, the simplest fusion mode is to take the average value of the first class probability and the second class probability as the third class probability of the class labels.
Weights can be set for the first classification probability and the second classification probability respectively for fusion according to the complexity degree of the image in the actual application scene.
And S27, determining the category of the image to be classified according to the third classification probability of each single-category decoupling feature map.
And setting probability thresholds for different class labels respectively, and considering that the image features in the image to be classified have the class labels when the classification probability of the class labels reaches the probability thresholds.
According to the embodiment, on the basis of the method embodiment, the first classification probability is obtained based on the inter-class relation, the second classification probability is obtained based on the image features, and the values of the first classification probability and the second classification probability are comprehensively considered according to the complexity of the images to be classified, so that the image classification result is not influenced by feature coexistence, and features which often coexist are not missed.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way. The steps with the same content are not repeated to describe the working process.
Furthermore, the present embodiment also provides an image classification device, which is composed of software and/or hardware and is used for executing the image classification method. Fig. 3 is a schematic structural diagram of an image classification apparatus according to the present embodiment. As shown in fig. 3, the image classification apparatus includes:
the characteristic extraction module is used for extracting a multi-channel characteristic diagram of the image to be classified, wherein the multi-channel characteristic diagram comprises a shallow characteristic diagram and a deep characteristic diagram;
the characteristic decoupling module is used for decoupling the multi-channel characteristic diagrams to obtain a plurality of single-class decoupling characteristic diagrams, wherein the single-class decoupling characteristic diagrams comprise single-class shallow layer decoupling characteristic diagrams and single-class deep layer decoupling characteristic diagrams;
the inter-class relation extraction module is used for obtaining the inter-class relation according to each single-class decoupling characteristic diagram obtained by decoupling; aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map;
and the classification module is used for determining the class of the image to be classified according to the classification probability of each single-class decoupling feature map.
Wherein, the characteristic decoupling module includes: a convolution unit, an activation unit and a decoupling unit.
The convolution unit comprises a convolution network with a 1*1 convolution kernel and is used for carrying out convolution operation on the multichannel characteristic map to obtain a convolution characteristic map;
the activation unit comprises a preset activation function, and the convolution feature images are processed based on the preset activation function to obtain a plurality of single-class feature images;
and the decoupling unit is used for multiplying each single-class feature map with the multi-channel feature map respectively to obtain single-class decoupling feature maps.
Correspondingly, the inter-class relation extraction module comprises: a fusion unit and a graph convolution unit.
The fusion unit is used for fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relation matrix;
the graph convolution unit is used for inputting the single-class deep decoupling feature graphs and the relation matrix between classes into a preset graph convolution network to obtain a first classification probability of each single-class decoupling feature graph.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.
Specifically, the feature extraction module, the feature decoupling module and the inter-class relation extraction module of the image classification device form a convolution network model. Fig. 4 is a schematic structural diagram of a convolutional network model provided in the present embodiment.
The feature extraction module extracts spatial semantic features through a backbone neural network (backbone) of a convolution layer, wherein the backbone neural network comprises a plurality of groups of residual blocks, and each group of residual blocks comprises a plurality of residual modules. In some embodiments, the feature extraction module may employ a Residual Network (Resnet), such as a Resnet50 Network architecture. The residual network is more sensitive to fluctuation of data, and is more beneficial to identifying and extracting approximate image features.
As a non-limiting example, as shown in fig. 4, the Resnet50 network includes four sets of Residual blocks, which in turn include 3, 4, 6, 3 Residual blocks (Residual blocks). And the residual blocks are used for sequentially extracting multi-channel feature images of the images to be classified to respectively obtain a shallow feature image and a deep feature image. Specifically, the image to be classified is taken as the input of the 1 st group of residual blocks, the output of the 1 st group of residual blocks is taken as the input of the 2 nd group of residual blocks, the output of the 2 nd group of residual blocks is taken as the input of the 3 rd group of residual blocks, and the like, and the shallow characteristic map and the deep characteristic map are respectively output from the convolution calculation results of the 2 nd group of residual blocks and the 4 th group of residual blocks.
And the feature decoupling module is used for respectively performing feature decoupling on the shallow feature map and the deep feature map to obtain a single-class shallow decoupling feature map and a single-class deep decoupling feature map of each single-class decoupling feature map.
The characteristic decoupling module comprises a convolution network with a 1*1 convolution kernel, an activation function and a decoupling calculation unit; the convolution network is used for carrying out convolution operation on the multi-channel feature map to extract features and obtain a convolution feature map; the convolution kernel of 1*1 can reduce the number of channels of the feature map without changing the size of the feature map, and obtain the number of feature maps consistent with the number of category labels. And inputting each convolution feature graph into the activation function, calculating and outputting a single-class feature graph corresponding to the class label by the activation function, wherein the activation function can be a sigmoid function, and can be a Tanh function or a ReLU function in other application scenes. And the decoupling calculation unit multiplies each single-class feature map with the multi-channel feature map respectively to obtain single-class decoupling feature maps.
The inter-class relationship extraction module includes a fusion calculation unit and a two-layer graph rolling network (Graph Convolutional Network, GCN). The fusion calculation unit fuses the single-class shallow decoupling feature map and the single-class deep decoupling feature map through fusion (concat) operation to generate an inter-class relationship matrix (Correlation Matrix). And taking the single-class deep decoupling feature map and the inter-class relation matrix as the input of a map convolution network, and calculating to obtain the first classification probability of the single-class decoupling feature map. The first classification probability is a classification probability based on a relationship between classes.
In other embodiments, the feature extraction module further includes a full connection layer, where the full connection layer is configured to perform a classification task, and calculate a second classification probability of the multi-channel feature map. The second classification probability is a classification probability based on the image features.
The inter-class relation extraction module is further used for fusing the first classification probability and the second classification probability to obtain a third classification probability. For the class labels corresponding to each single class decoupling feature map, the simplest fusion mode is to take the average value of the class labels as the classification probability of the class labels. In other embodiments, the first classification probability and the second classification probability may also be fused by respectively weighting the two.
The inter-class relation extraction module is specifically configured to determine a class of the image to be classified according to the first classification probability or the third classification probability. For the situation that the difference between the image features in the images to be classified is obvious, the images to be classified can be subjected to label classification by adopting the first classification probability, and otherwise, the images to be classified are subjected to label classification by adopting the third classification probability.
And setting probability thresholds for different class labels respectively, and considering that the image features in the image to be classified have the class labels when the classification probability of the class labels reaches the probability thresholds.
Before practical application, the convolutional network model needs to be trained, the convolutional network model performs feature extraction and feature decoupling on sample images to obtain a plurality of single-class decoupling feature images, class labels of the single-class decoupling feature images are marked on each single-class decoupling feature image respectively for the convolutional network model to learn, and the convolutional network model can obtain inter-class relations among the class labels through learning a large number of sample images to further learn to obtain classification probability.
In the application, the convolution network model performs feature extraction and feature decoupling on the image to be classified, calculates and outputs the classification probability of each class label according to the single class decoupling feature graph based on the learned class relation, and further determines whether the image to be classified has the corresponding class label or not according to the classification probability and the probability threshold.
The existing machine learning model only performs image feature extraction, the output classification probability ignores the relation before the class label, the influence of a training sample is easy to be affected, and when the training sample is not comprehensive and balanced, the learned classification probability is not accurate enough. The convolutional network model provided by the embodiment decouples the features, extracts the relation between the coexisting features, weakens the influence of the training sample deviation on the coexisting features, and enables the classification to be more objective.
Fig. 5 is a schematic structural diagram of an image classification apparatus according to another embodiment of the present application. As shown in fig. 5, the present embodiment further provides an image classification apparatus, including: the image classification system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the image classification method according to the embodiment of the method when executing the computer program.
Specifically, the trained convolutional network model is stored in the memory, and the processor is used for determining the category of the image to be classified by calling the convolutional network model to implement the image classification method according to the embodiment of the method.
The processor may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may be an internal storage unit of the image capturing device or the image sorting apparatus in some embodiments, such as a hard disk or a memory, or may be an external storage device of the image capturing device or the image sorting apparatus in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), etc. Further, the memory may also include both internal storage units and external storage devices. The memory is used to store an operating system, application programs, boot loader (BootLoader), data, and other programs, etc., such as program code for a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output, such as acquired images to be classified, various types of feature maps, etc.
The processor may determine the class of the image to be classified, for example, classifying the fundus color illumination, by the trained convolutional network model described above.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps that may implement the various method embodiments described above.
Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that may be performed in the various method embodiments described above.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/terminal apparatus, recording medium, computer Memory, read-Only Memory (ROM), random access Memory (RAM, random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (5)
1. An image classification method, comprising:
extracting a multi-channel feature map of an image to be classified;
decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps;
obtaining the relationship among classes according to the single-class decoupling feature graphs obtained by decoupling;
aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map;
determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map;
the decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps includes:
performing convolution operation on the multi-channel feature map to obtain a convolution feature map;
processing the convolution feature images based on a preset activation function to obtain a plurality of single-class feature images;
multiplying each single-class feature map with the multi-channel feature map respectively to obtain a single-class decoupling feature map;
the multi-channel feature map comprises a shallow feature map and a deep feature map, and the single-class decoupling feature map comprises a single-class shallow decoupling feature map and a single-class deep decoupling feature map;
the decoupling the multi-channel feature map to obtain a plurality of single-class decoupling feature maps includes:
respectively decoupling the shallow feature map and the deep feature map to obtain a single-class shallow decoupling feature map and a single-class deep decoupling feature map;
the inter-class relationship comprises an inter-class relationship matrix;
the obtaining the inter-class relationship according to the single-class decoupling feature graphs obtained by decoupling includes:
fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relationship matrix;
the calculating, for each single-class decoupling feature map, a classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map, includes:
inputting the single-class deep decoupling feature images and the relation matrix between classes into a preset image convolution network to obtain a first classification probability of each single-class decoupling feature image;
the determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map comprises the following steps:
and determining the category of the image to be classified according to the first classification probability of each single-category decoupling feature map.
2. The image classification method of claim 1, further comprising, after extracting the multi-channel feature map of the image to be classified:
calculating a second classification probability of the multi-channel feature map through a full connection layer;
fusing the first classification probability and the second classification probability to obtain a third classification probability;
the determining the category of the image to be classified according to the classification probability of each single-category decoupling feature map comprises the following steps:
and determining the category of the image to be classified according to the third classification probability of each single-category decoupling feature map.
3. An image classification apparatus, comprising:
the feature extraction module is used for extracting a multi-channel feature map of the image to be classified;
the characteristic decoupling module is used for decoupling the multi-channel characteristic diagrams to obtain a plurality of single-class decoupling characteristic diagrams;
the inter-class relation extraction module is used for obtaining the inter-class relation according to each single-class decoupling characteristic diagram obtained by decoupling; aiming at each single-class decoupling feature map, calculating the classification probability of the corresponding single-class decoupling feature map according to the inter-class relationship and the single-class decoupling feature map;
the classification module is used for determining the class of the image to be classified according to the classification probability of each single-class decoupling feature map;
wherein, the characteristic decoupling module includes:
the convolution unit comprises a convolution network with a 1*1 convolution kernel and is used for carrying out convolution operation on the multichannel characteristic map to obtain a convolution characteristic map;
the activation unit comprises an activation function and is used for processing the convolution feature images based on a preset activation function to obtain a plurality of single-class feature images;
the decoupling unit is used for multiplying each single-class feature map with the multi-channel feature map respectively to obtain single-class decoupling feature maps;
the single-class decoupling feature map comprises a single-class shallow decoupling feature map and a single-class deep decoupling feature map;
the inter-class relation extraction module comprises:
the fusion unit is used for fusing the single-class shallow decoupling feature map and the single-class deep decoupling feature map to generate an inter-class relation matrix;
and the graph rolling unit is used for inputting the single-class deep decoupling feature graphs and the inter-class relation matrix into a preset graph rolling network to obtain a first classification probability of each single-class decoupling feature graph.
4. An image classification apparatus, characterized by comprising: memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to claim 1 or 2 when executing the computer program.
5. A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the image classification method according to claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110570158.2A CN113255766B (en) | 2021-05-25 | 2021-05-25 | Image classification method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110570158.2A CN113255766B (en) | 2021-05-25 | 2021-05-25 | Image classification method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113255766A CN113255766A (en) | 2021-08-13 |
| CN113255766B true CN113255766B (en) | 2023-12-22 |
Family
ID=77184151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110570158.2A Active CN113255766B (en) | 2021-05-25 | 2021-05-25 | Image classification method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113255766B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023068330A1 (en) * | 2021-10-20 | 2023-04-27 | ＤｅｅｐＥｙｅＶｉｓｉｏｎ株式会社 | Information processing device, information processing method, and computer-readable recording medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145743A (en) * | 2018-07-19 | 2019-01-04 | 叶涵 | A kind of image-recognizing method and device based on deep learning |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
| CN111488945A (en) * | 2020-04-17 | 2020-08-04 | 上海眼控科技股份有限公司 | Image processing method, image processing device, computer equipment and computer readable storage medium |
| CN111507403A (en) * | 2020-04-17 | 2020-08-07 | 腾讯科技（深圳）有限公司 | Image classification method and device, computer equipment and storage medium |
| CN111553419A (en) * | 2020-04-28 | 2020-08-18 | 腾讯科技（深圳）有限公司 | Image identification method, device, equipment and readable storage medium |
| CN111914775A (en) * | 2020-08-06 | 2020-11-10 | 平安科技（深圳）有限公司 | Living body detection method, living body detection device, electronic apparatus, and storage medium |
| CN112749653A (en) * | 2020-12-31 | 2021-05-04 | 平安科技（深圳）有限公司 | Pedestrian detection method, device, electronic equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10482633B2 (en) * | 2016-09-12 | 2019-11-19 | Zebra Medical Vision Ltd. | Systems and methods for automated detection of an indication of malignancy in a mammographic image |
| US11042586B2 (en) * | 2016-12-29 | 2021-06-22 | Shutterstock, Inc. | Clustering search results based on image composition |
| JP7165752B2 (en) * | 2019-10-22 | 2022-11-04 | センスタイム インターナショナル ピーティーイー．リミテッド | Image processing method and apparatus, processor, storage medium |
-
2021
- 2021-05-25 CN CN202110570158.2A patent/CN113255766B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145743A (en) * | 2018-07-19 | 2019-01-04 | 叶涵 | A kind of image-recognizing method and device based on deep learning |
| CN109886273A (en) * | 2019-02-26 | 2019-06-14 | 四川大学华西医院 | A kind of CMR classification of image segmentation system |
| CN111488945A (en) * | 2020-04-17 | 2020-08-04 | 上海眼控科技股份有限公司 | Image processing method, image processing device, computer equipment and computer readable storage medium |
| CN111507403A (en) * | 2020-04-17 | 2020-08-07 | 腾讯科技（深圳）有限公司 | Image classification method and device, computer equipment and storage medium |
| CN111553419A (en) * | 2020-04-28 | 2020-08-18 | 腾讯科技（深圳）有限公司 | Image identification method, device, equipment and readable storage medium |
| CN111914775A (en) * | 2020-08-06 | 2020-11-10 | 平安科技（深圳）有限公司 | Living body detection method, living body detection device, electronic apparatus, and storage medium |
| CN112749653A (en) * | 2020-12-31 | 2021-05-04 | 平安科技（深圳）有限公司 | Pedestrian detection method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113255766A (en) | 2021-08-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3493101B1 (en) | Image recognition method, terminal, and nonvolatile storage medium | |
| CN110889312B (en) | Living body detection method and apparatus, electronic device, computer-readable storage medium | |
| CN108776819A (en) | A kind of target identification method, mobile terminal and computer readable storage medium | |
| CN110188829B (en) | Neural network training method, target recognition method and related products | |
| CN111881707B (en) | Image reproduction detection method, identity verification method, model training method and device | |
| KR20170026222A (en) | Method and device for classifying an object of an image and corresponding computer program product and computer-readable medium | |
| US7643674B2 (en) | Classification methods, classifier determination methods, classifiers, classifier determination devices, and articles of manufacture | |
| CN107133629B (en) | Picture classification method and device and mobile terminal | |
| CN110688883A (en) | Vehicle and pedestrian detection method and device | |
| CN111881741A (en) | License plate recognition method and device, computer equipment and computer-readable storage medium | |
| CN114419363A (en) | Target classification model training method and device based on label-free sample data | |
| CN112949459A (en) | Smoking image recognition method and device, storage medium and electronic equipment | |
| CN112052907A (en) | Target detection method and device based on image edge information and storage medium | |
| CN112884147A (en) | Neural network training method, image processing method, device and electronic equipment | |
| CN113255766B (en) | Image classification method, device, equipment and storage medium | |
| CN112257628A (en) | Method, device and equipment for identifying identities of outdoor competition athletes | |
| CN116206334A (en) | Wild animal identification method and device | |
| CN117523218A (en) | Label generation, training of image classification model and image classification method and device | |
| CN112989869B (en) | Optimization method, device, equipment and storage medium of face quality detection model | |
| CN113537248A (en) | Image recognition method and device, electronic equipment and storage medium | |
| CN113537253A (en) | Infrared image target detection method and device, computing equipment and storage medium | |
| CN113837236A (en) | Method and device for identifying target object in image, terminal equipment and storage medium | |
| CN111881187A (en) | Method for automatically establishing data processing model and related product | |
| CN113762382B (en) | Model training and scene recognition method, device, equipment and medium | |
| CN116597527B (en) | Living body detection method, living body detection device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221031 Address after: 518000 Guangdong, Shenzhen, Futian District Futian street Fu'an community Yitian road 5033, Ping An financial center, 23 floor. Applicant after: PING AN TECHNOLOGY (SHENZHEN) Co.,Ltd. Applicant after: SHANDONG EYE INSTITUTE Address before: 518000 Guangdong, Shenzhen, Futian District Futian street Fu'an community Yitian road 5033, Ping An financial center, 23 floor. Applicant before: PING AN TECHNOLOGY (SHENZHEN) Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |