CN111797936A - Image emotion classification method and device based on saliency detection and multi-level feature fusion - Google Patents

Abstract

The invention discloses an image emotion classification method based on saliency detection and multi-level feature fusion. Firstly, the saliency detection network is used to extract the saliency map of the emotion image; The modulation of the feature map, so that the Inception‑v4 network can pay more attention to the emotional expression area of the emotional image, thereby effectively improving the accuracy of image emotion classification; finally, the Inception‑v4 network is used to modulate the modulation feature map of the twin neural network. Classification is performed to accurately obtain the emotion category corresponding to the emotion image. The image emotion classification method provided by the present invention can accurately locate the emotion expression area in the emotion image, and realizes that the classification network pays higher attention to the emotion expression area of the emotion image by means of feature modulation, thereby effectively improving the image emotion classification. accuracy of the method.

Description

Image sentiment classification method based on saliency detection and multi-level feature fusion and device

technical field

The invention relates to the technical field of image emotion classification, in particular to an image emotion classification method and device based on saliency detection and multi-level feature fusion.

Background technique

With the development and popularization of photography technology and social networks, people have become accustomed to sharing experiences and expressing opinions online through images or videos. This creates an urgent need for the processing and understanding of image and video content. Humans are better able to perceptually understand high-level semantics and emotions than low-level visual representations. In recent years, sentiment hierarchy analysis of image content in psychology, affective computing, and multimedia communities has received extensive attention. The analysis of images at the emotional level is also the top priority of image content analysis, which can be widely used in human-computer interaction, public opinion analysis, and image retrieval.

The expression of image emotion is mainly determined by the emotional area in the image, such as the target in the image, and the existing methods do not pay more attention to the emotional area of the image, so it is impossible to obtain more discriminative emotional features.

SUMMARY OF THE INVENTION

The present invention provides an image emotion classification method and device based on saliency detection and multi-level feature fusion, which are used to overcome the defects of the prior art such as less attention to emotional regions.

In order to achieve the above object, the present invention proposes an image emotion classification method based on saliency detection and multi-level feature fusion, and the image emotion classification method includes:

constructing an emotional image set; the emotional image set includes marked emotional images;

Establish a training set and a verification set according to the emotional image set, and use a saliency detection network to extract the saliency maps of the emotional images in the training set and the verification set;

The emotional images and saliency maps in the training set are input into a pre-built image emotion classification model; the image emotion classification model includes a twin neural network and an Inception-v4 network;

The Siamese neural network is trained by using the emotional images and saliency maps in the training set, and the saliency map in the training set is used to perform feature modulation on the corresponding emotional images through the trained Siamese neural network to obtain modulation feature map;

Use the modulation feature map to train the Inception-v4 network to obtain a trained image emotion classification model;

Use the emotional images and saliency maps in the verification set to verify the trained image sentiment classification model;

The image to be classified and the saliency map of the to-be-classified image are input into the verified image emotion classification model for classification, and the image emotion category is obtained.

In order to achieve the above object, the present invention also proposes an image emotion classification device based on saliency detection and multi-level feature fusion, including:

an image set building module for constructing an emotional image set; the emotional image set includes marked emotional images;

A saliency map acquisition module, used for establishing a training set and a verification set according to the emotional image set, and extracting the saliency map of the emotional images in the training set and the verification set by using a saliency detection network;

A model training module for inputting the emotional images and saliency maps in the training set into a pre-built image emotion classification model; the image emotion classification model includes a twin neural network and an Inception-v4 network; using the emotion in the training set The image and the saliency map are used to train the twin neural network, and the saliency map in the training set is used to perform feature modulation on the corresponding emotional image through the trained twin neural network to obtain a modulation feature map; using the modulation The feature map is used to train the Inception-v4 network to obtain a trained image emotion classification model;

A model verification module for verifying the trained image sentiment classification model using the emotional images and saliency maps in the verification set;

The classification module is used for inputting the image to be classified and the saliency map of the image to be classified into the verified image emotion classification model for classification to obtain the image emotion category.

To achieve the above object, the present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

In order to achieve the above object, the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method are implemented.

Compared with the prior art, the beneficial effects of the present invention are:

The image emotion classification method based on saliency detection and multi-level feature fusion provided by the present invention firstly uses the saliency detection network to extract the saliency map of the emotion image; The modulation of the image, so that the Inception-v4 network can pay more attention to the emotional expression area of the emotional image, thereby effectively improving the accuracy of image emotion classification; finally, the Inception-v4 network is used to modulate the modulation feature map modulated by the twin neural network. classification to accurately obtain the emotion category corresponding to the emotion image. The image emotion classification method provided by the present invention can accurately locate the emotion expression area in the emotion image, and realizes that the classification network pays higher attention to the emotion expression area of the emotion image by means of feature modulation, thereby effectively improving the image emotion classification. accuracy of the method.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained according to the structures shown in these drawings without creative efforts.

1 is a flowchart of an image emotion classification method based on saliency detection and multi-level feature fusion provided by the present invention;

Fig. 2 is the overall structure diagram of the image emotion classification method based on saliency detection and multi-level feature fusion provided by the present invention;

3 is a structural diagram of an image emotion classification model provided by the present invention;

Figure 4 is a graph of emotional wheel and emotional distance.

The realization, functional characteristics and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

The present invention proposes an image emotion classification method based on saliency detection and multi-level feature fusion. As shown in Figure 1 and Figure 2, the image emotion classification method includes:

101: Construct an emotional image set; the emotional image set includes marked emotional images;

The emotional images in the emotional image set come from the International Affective Image System Subset (IAPSa), the Abstract Painting Dataset (Abstract), the Art Image Collection (ArtPhoto), and several weakly labeled emotional images collected from Flicker and Instagram, etc.

Labeling means that the emotional images in the emotional image set are pre-marked with emotional categories.

The emotion categories include anger, disgust, fear, sadness, amusement, awe, contentment, and excitement.

102: Establish a training set and a verification set according to the emotional image set, and use a saliency detection network to extract the saliency maps of the emotional images in the training set and the verification set;

Saliency detection network, see Liu J J, Hou Q, Cheng M M, et al. A Simple Pooling-Based Design for Real-Time Salient Object Detection [J]. 2019.

103: Input the emotional images and saliency maps in the training set into a pre-built image emotion classification model; the image emotion classification model includes the Siamese neural network and the Inception-v4 network;

Siamese neural network consists of two identical neural networks with shared weights.

The Inception-v4 network is a classification network with excellent performance.

104: Use the emotional image and saliency map in the training set to train the twin neural network, and use the saliency map in the training set to perform feature modulation on the corresponding emotional image through the trained twin neural network to obtain a modulation feature map;

The modulation of the feature map of the saliency map to the feature map of the corresponding emotional image is realized by the Siamese neural network, so that the Inception-v4 network can pay more attention to the emotional expression area of the emotional image, thereby effectively improving the accuracy of image emotion classification.

105: Use the modulation feature map to train the Inception-v4 network to obtain a trained image emotion classification model;

106: Validate the trained image emotion classification model by using the emotion images and saliency maps in the validation set;

107 : Input the image to be classified and the saliency map of the image to be classified into the verified image emotion classification model for classification, and obtain an image emotion category.

The image emotion classification method based on saliency detection and multi-level feature fusion provided by the present invention firstly uses the saliency detection network to extract the saliency map of the emotion image; The modulation of the image, so that the Inception-v4 network can pay more attention to the emotional expression area of the emotional image, thereby effectively improving the accuracy of image emotion classification; finally, the Inception-v4 network is used to modulate the modulation feature map modulated by the twin neural network. classification to accurately obtain the emotion category corresponding to the emotion image. The image emotion classification method provided by the present invention can accurately locate the emotion expression area in the emotion image, and realizes that the classification network pays higher attention to the emotion expression area of the emotion image by means of feature modulation, thereby effectively improving the image emotion classification. accuracy of the method.

In one embodiment, for step 103, the Siamese neural network is shown in FIG. 3, including a first branch and a second branch, the first branch is used for feature extraction on emotional images, and the second branch is used for saliency map Perform feature extraction; both the first branch and the second branch are composed of four convolutional layers (BasicConv2d) with the same convolution kernel size; the outputs of the second and fourth convolutional layers in the second branch connected to the outputs of the second and fourth convolutional layers in the first branch.

In order to ensure that the features of emotional images and the features of saliency maps maintain a similar feature space in the network propagation, both branches of this Siamese neural network are composed of four convolutional layers with the same convolution kernel size. For the first branch, the input and output channels of the four convolutional layers are all 3; while the input and input channels of the four convolutional layers of the second branch are all 1.

After obtaining the saliency map of the emotional image through the saliency detection network, in order to allow the saliency map to successfully constrain the classification model to give higher attention to the emotional region of the emotional image, a Siamese neural network is introduced. In the Siamese neural network, the saliency map in the second branch is used to modulate the emotional images in the first branch, and the attention to the emotional regions in the emotional images is enhanced.

In a certain embodiment, for step 104, the trained twin neural network uses the saliency map in the training set to perform feature modulation on the corresponding emotional image to obtain a modulation feature map, including:

401: Input the emotional image a(x, y, z) in the training set into the first branch of the trained Siamese neural network, and the saliency map b(x, y) into the second branch; where x, y are the coordinate systems , z is the three color channels of the emotional image.

402: Obtain the feature map S output by the second convolutional layer in the second branch (S∈R ^w×h , where w is the width of the feature map, and h is the height of the feature map), and the feature map S and the The feature map T (T ∈ R ^{c ×w × h} , c is the number of channels of the feature map) output by the second convolutional layer in the first branch is multiplied by the corresponding elements to obtain the feature map H (H ∈ R ^{c ×w×h} ), and the feature map H and the feature map T are added with corresponding elements to obtain the feature map G;

The multiplication operation is to use the feature map S to modulate the feature map T;

The addition operation is to avoid the problem that the features of the T part of the feature map are completely ignored after the multiplication operation, and the addition operation re-emphasizes the features of the original image (input emotional image).

403: Input the feature map G into the third convolutional layer of the first branch, and input the feature map S into the third convolutional layer of the second branch;

404: Obtain the feature map S' output by the fourth convolutional layer in the second branch, and perform the corresponding element comparison between the feature map S' and the feature map T' output by the fourth convolutional layer in the first branch. A multiplication operation is performed to obtain a feature map H', and an addition operation of corresponding elements is performed on the feature map H' and the feature map T' to obtain a modulation feature map F.

The feature modulation is to make the salient region in the feature map T obtain a larger response value, that is, to let the Inception-v4 network give more attention to the emotional expression region in the emotional image. At the same time, to ensure that the effect is obvious, two consecutive modulations are carried out.

In another embodiment, since the number of feature maps S and feature maps T is irrelevant, only a single feature map S is considered. The calculation formula of the feature map G is:

G=f(T(w,h,c)*[S(w,h)+1]) (1)

In the formula, f represents the sigmoid activation function, so that 0<T∈Rw ^×h <1, so as to ensure a suitable range for feature modulation; T(w,h,c) represents the output of the second convolutional layer in the first branch The feature map of ; S(w, h) represents the feature map output by the second convolutional layer in the second branch; w, h, c represent the width, height and number of channels of the feature map, respectively;

The calculation formula of the modulation feature map F is:

F=f(T'(w,h,c)*[S'(w,h)+1]) (2)

In the formula, T'(w,h,c) represents the feature map output by the fourth convolutional layer in the first branch; S'(w,h) represents the feature map output by the fourth convolutional layer in the second branch .

In the next embodiment, the Inception-v4 network is shown in Fig. 3, which is a multi-branch structure, including three convolutional layers (BasicConv2d), one Mixed_3a module, one Mixed_4a module, one Mixed_5a module, four Inception_A modules, one Reduction_A module, seven Inception_B modules, one Reduction_B module, three Inception_C modules, one average pooling layer (Avarage Pooling) and one fully connected layer (FullyConnection); after the Mixed_5a module, Reduction-A module and Reduction-B module The side branch structure (BasicConv2d) is respectively introduced, and the side branch structure consists of a convolutional layer; the outputs of the three side branch structures are connected to a fully connected layer (Fully Connection), and the fully connected layer uses It fuses the side branch features output by the three side branch structures and outputs the fused feature, and outputs the fused feature to the output end of the average pooling layer.

The three side branch structures are respectively composed of a convolutional layer with an output channel of 256, the size of the convolution kernel is 1, and the convolution stride is 1. In order to fuse the features output by the three side branch structures, in this embodiment, a fully connected layer (Fully Connection) with 256 layer neurons is defined behind the three side branch structures. Finally, set the number of neurons in the last fully connected layer of the Inception-v4 network to 8, corresponding to 8 emotion categories, as the final classifier. Through the above network structure, three different levels of features L ₁ , L ₂ , L ₃ in the deep network can be obtained from the three side branch structures, and the top-level feature L ₄ of the deep network can be obtained. Inception-v4 network Four levels of feature maps.

After obtaining different levels of features, how to integrate these different levels of features in the model method is extremely important. Through observation, it is found that in image emotion recognition, the semantic features of images play a greater role in emotion recognition than the style features of images. Therefore, in this embodiment, when the feature is fused, a higher degree of attention is given to the semantic feature.

The feature fusion of this embodiment is specifically:

Step 1: First perform the concat operation on the features L ₁ , L ₂ , and L ₃ , and then input them into the fully connected layer to obtain the feature L.

Step 2: Concat the feature L and the semantic feature L ₄ to obtain the final classification feature F.

This embodiment fully considers the influence of different types of features on emotional arousal, so as to formulate more expressive features to further solve the problem of the gap between emotional features and emotional expression, thereby improving the accuracy of emotion classification.

In this embodiment, a multi-branch structure is introduced on the basis of the Inception-v4 network, so as to obtain the style and semantic features of the image at the low-level and high-level of the network respectively, and then fuse the multi-level features through the Softmax method. Get the sentiment type prediction distribution of the image sentiment classification model.

In a certain embodiment, the last fully connected layer in the Inception-v4 network obtains the probability that the emotional image belongs to the i-th emotional category by Softmax:

In the formula, _yi represents the probability that the emotional image belongs to the i-th emotional category; z _i represents the activation value of the emotional image belonging to the i-th category; z _j represents the activation value of the emotional image belonging to the j-th category; C represents the emotional category.

In another embodiment, for step 105, because the data volume of the emotion data set is relatively small, the strategy of transfer learning is adopted first, and the Inception-v4 network is pre-trained on the ImageNet data set. Then, the Inception-v4 network is trained using the modulation feature map.

In the next embodiment, the image emotion classification model adopts a multi-task loss function based on emotion diversity constraints, and the multi-task loss function is:

L _multi =L _cls +λL _ed (4)

表示主导情绪的概率；p_j表示主导情绪j的类别概率；dis_ij表示Mikels’wheel中定义的情绪i和主导情绪j之间的距离；i,j∈B表示情感类别i和j在同一个极性中。In the formula, L _multi represents the multi-task loss function; L _cls represents the traditional classification loss; L _ed represents the loss of other accompanying _emotions ; _λ represents the weight; represents the probability that the model predicts that the emotional image belongs to the i-th emotional category; f(i) represents the probability of other accompanying emotions; j represents the dominant emotion; i represents other accompanying emotions;

represents the probability of the dominant emotion; p _j represents the category probability of the dominant emotion j; dis _ij represents the distance between the emotion i and the dominant emotion j defined in Mikels'wheel; i,j∈B represents the emotion category i and j in the same in polarity.

In the collection of most image sentiment data, the majority voting strategy is widely adopted to obtain the sentiment labels of images. Starting from the diversity of emotional expressions, we consider the distribution of emotions based on the probability of labels. Inspired by research in emotion theory, the relationship between two emotions determines how similar they are. And two emotions from similar to completely opposite can be expressed through Mikels'Wheel (see Figure 4, Figure 4 (a) is the emotional wheel, (b) is the emotional distance), through the distance defined by Mikels'Wheel, can calculate the two The distance between the emotions, and the distance represents the similarity of the two emotions, that is, the smaller the distance d _ij between the emotion i and the emotion j, the more similar the two emotions are. Therefore, through the definition of Mikels'Wheel, the probability label of emotional image can be obtained.

In order to obtain a more reasonable probability label, the present invention further combines the research of emotion theory. The emotion of emotional images is divided into two polarities: negative (anger, disgust, fear, sadness) N and positive (amusement, awe, contentment, excitement) P , through the study of emotion theory, the diversity expression of emotion is actually more of a dominant emotion accompanied by multiple other emotions of the same polarity, so when generating label probability, the relationship of emotion polarity is introduced.

According to the dominant emotion and the definition of distance in Mikels'Wheel, the probability distribution of other emotions with the same polarity as the dominant emotion is calculated, and the probability distribution of emotions with opposite polarity is set to 0. The calculation formulas are expressed as formulas (7) and (8).

Through the category labels and probability labels of emotional images, a multi-task loss function is introduced, formula (5).

The distributed labels of emotional images are generated according to Mikels'Wheel, and the emotional distribution loss of the predicted distribution and distributed labels is calculated, and a new loss function is formed by introducing the weight λ and the classification loss to achieve the constraints on the diversity of image expression. .

In this embodiment, stochastic gradient descent is used to optimize the multi-task loss function L _multi , and {a _i |i=1,2,...,C} is defined to represent the activation value of the ith emotion category in the last fully connected layer.

The gradient can be calculated by equation (9):

The representation of image emotion is subjective and diverse, while most emotion datasets are collected from single emotion labels generated by voting. But in fact, different areas of an image may express different emotions, and it is difficult to classify an image into a single emotion type. The emotional expression of an image is often a dominant emotion accompanied by one or more other emotions of the same polarity. Therefore, the use of a single emotion label will lead to inaccurate labeling of image emotion datasets, which greatly affects the accuracy of image emotion classification. Therefore, the relationship of emotion polarity is introduced in this embodiment, and the dominant emotion and other emotions with the same polarity as the dominant emotion are comprehensively considered by introducing a multi-task loss function, so as to improve the accuracy of image emotion classification.

In this embodiment, starting from the diversity of image emotion expression, a new loss function calculated by combining emotion distribution loss is proposed to complete the constraint on the diversity of image emotion expression.

The present invention also proposes an image emotion classification device based on saliency detection and multi-level feature fusion, including:

an image set building module for constructing an emotional image set; the emotional image set includes marked emotional images;

A saliency map acquisition module, used for establishing a training set and a verification set according to the emotional image set, and extracting the saliency map of the emotional images in the training set and the verification set by using a saliency detection network;

A model training module is used to input the emotional images and saliency maps in the training set into a pre-built image emotion classification model; the image emotion classification model includes a twin neural network and an Inception-v4 network; using the The emotion image and the saliency map are used to train the twin neural network, and the saliency map in the training set is used to perform feature modulation on the corresponding emotion image through the trained twin neural network to obtain a modulation feature map; Modulate the feature map to train the Inception-v4 network to obtain a trained image emotion classification model;

A model verification module for verifying the trained image sentiment classification model using the emotional images and saliency maps in the verification set;

The classification module is used for inputting the image to be classified and the saliency map of the image to be classified into the verified image emotion classification model for classification to obtain the image emotion category.

In one embodiment, for the model training module, the Siamese neural network is shown in FIG. 3, including a first branch and a second branch, the first branch is used for feature extraction on emotional images, and the second branch is used for saliency Feature extraction is performed on the graph; the first branch and the second branch are both composed of four convolutional layers (BasicConv2d) with the same convolution kernel size; the outputs of the second and fourth convolutional layers in the second branch The terminal is connected to the output terminal of the second and fourth convolutional layers in the first branch.

In order to ensure that the features of emotional images and the features of saliency maps maintain a similar feature space in the network propagation, both branches of this Siamese neural network are composed of four convolutional layers with the same convolution kernel size. For the first branch, the input and output channels of the four convolutional layers are all 3; while the input and input channels of the four convolutional layers of the second branch are all 1.

After obtaining the saliency map of the emotional image through the saliency detection network, in order to allow the saliency map to successfully constrain the classification model to give higher attention to the emotional region of the emotional image, a Siamese neural network is introduced. In the Siamese neural network, the saliency map in the second branch is used to modulate the emotional images in the first branch, and the attention to the emotional regions in the emotional images is enhanced.

In a certain embodiment, the model training module further includes:

401: Input the emotional image a(x, y, z) in the training set into the first branch of the trained Siamese neural network, and the saliency map b(x, y) into the second branch; where x, y are the coordinate systems , z is the three color channels of the emotional image.

402: Obtain the feature map S output by the second convolutional layer in the second branch (S∈R ^w×h , where w is the width of the feature map, and h is the height of the feature map), and the feature map S and the The feature map T (T ∈ R ^{c ×w × h} , c is the number of channels of the feature map) output by the second convolutional layer in the first branch is multiplied by the corresponding elements to obtain the feature map H (H ∈ R ^{c ×w×h} ), and the feature map H and the feature map T are added with corresponding elements to obtain the feature map G;

The multiplication operation is to use the feature map S to modulate the feature map T;

The addition operation is to avoid the problem that the features of the T part of the feature map are completely ignored after the multiplication operation, and the addition operation re-emphasizes the features of the original image (input emotional image).

403: Input the feature map G into the third convolutional layer of the first branch, and input the feature map S into the third convolutional layer of the second branch;

404: Obtain the feature map S' output by the fourth convolutional layer in the second branch, and perform the corresponding element comparison between the feature map S' and the feature map T' output by the fourth convolutional layer in the first branch. A multiplication operation is performed to obtain a feature map H', and an addition operation of corresponding elements is performed on the feature map H' and the feature map T' to obtain a modulation feature map F.

The feature modulation is to make the salient region in the feature map T obtain a larger response value, that is, to let the Inception-v4 network give more attention to the emotional expression region in the emotional image. At the same time, to ensure that the effect is obvious, two consecutive modulations are carried out.

In another embodiment, since the number of feature maps S and feature maps T is irrelevant, only a single feature map S is considered. The calculation formula of the feature map G is:

G=f(T(w,h,c)*[S(w,h)+1]) (1)

In the formula, f represents the sigmoid activation function, so that 0<T∈Rw ^×h <1, so as to ensure a suitable range for feature modulation; T(w,h,c) represents the output of the second convolutional layer in the first branch The feature map of ; S(w, h) represents the feature map output by the second convolutional layer in the second branch; w, h, c represent the width, height and number of channels of the feature map, respectively;

The calculation formula of the modulation feature map F is:

F=f(T'(w,h,c)*[S'(w,h)+1]) (2)

In the formula, T'(w,h,c) represents the feature map output by the fourth convolutional layer in the first branch; S'(w,h) represents the feature map output by the fourth convolutional layer in the second branch .

In the next embodiment, the Inception-v4 network is shown in Fig. 3, which is a multi-branch structure, including three convolutional layers (BasicConv2d), one Mixed_3a module, one Mixed_4a module, one Mixed_5a module, four Inception_A modules, one Reduction_A module, seven Inception_B modules, one Reduction_B module, three Inception_C modules, one average pooling layer (Avarage Pooling) and one fully connected layer (FullyConnection); after the Mixed_5a module, Reduction-A module and Reduction-B module The side branch structure (BasicConv2d) is respectively introduced, and the side branch structure consists of a convolutional layer; the outputs of the three side branch structures are connected to a fully connected layer (Fully Connection), and the fully connected layer uses It fuses the side branch features output by the three side branch structures and outputs the fused feature, and outputs the fused feature to the output end of the average pooling layer.

The three side branch structures are respectively composed of a convolutional layer with an output channel of 256, the size of the convolution kernel is 1, and the convolution stride is 1. In order to fuse the features output by the three side branch structures, in this embodiment, a fully connected layer (Fully Connection) with 256 layer neurons is defined behind the three side branch structures. Finally, set the number of neurons in the last fully connected layer of the Inception-v4 network to 8, corresponding to 8 emotion categories, as the final classifier. Through the above network structure, three different levels of features L ₁ , L ₂ , L ₃ in the deep network can be obtained from the three side branch structures, and the top-level feature L ₄ of the deep network can be obtained. Inception-v4 network Four levels of feature maps.

After obtaining different levels of features, how to integrate these different levels of features in the model method is extremely important. Through observation, it is found that in image emotion recognition, the semantic features of images play a greater role in emotion recognition than the style features of images. Therefore, in this embodiment, when the feature is fused, a higher degree of attention is given to the semantic feature.

The feature fusion of this embodiment is specifically:

Step 1: First perform the concat operation on the features L ₁ , L ₂ , and L ₃ , and then input them into the fully connected layer to obtain the feature L.

Step 2: Concat the feature L and the semantic feature L ₄ to obtain the final classification feature F.

This embodiment fully considers the influence of different types of features on emotional arousal, so as to formulate more expressive features to further solve the problem of the gap between emotional features and emotional expression, thereby improving the accuracy of emotion classification.

In this embodiment, a multi-branch structure is introduced on the basis of the Inception-v4 network, so as to obtain the style and semantic features of the image at the low-level and high-level of the network respectively, and then fuse the multi-level features through the Softmax method. Get the sentiment type prediction distribution of the image sentiment classification model.

In a certain embodiment, the last fully connected layer in the Inception-v4 network obtains the probability that the emotional image belongs to the i-th emotional category by Softmax:

In the formula, _yi represents the probability that the emotional image belongs to the i-th emotional category; z _i represents the activation value of the emotional image belonging to the i-th category; z _j represents the activation value of the emotional image belonging to the j-th category; C represents the emotional category.

In another embodiment, in the model training module, because the data volume of the emotion data set is relatively small, the strategy of transfer learning is adopted first, and the Inception-v4 network is pre-trained on the ImageNet data set. Then, the Inception-v4 network is trained using the modulation feature map.

In the next embodiment, in the model training module, the image emotion classification model adopts a multi-task loss function based on emotion diversity constraints, and the multi-task loss function is:

L _multi =L _cls +λL _ed (4)

表示主导情绪的概率；p_j表示主导情绪j的类别概率；dis_ij表示Mikels’wheel中定义的情绪i和主导情绪j之间的距离；i,j∈B表示情感类别i和j在同一个极性中。In the formula, L _multi represents the multi-task loss function; L _cls represents the traditional classification loss; L _ed represents the loss of other accompanying _emotions ; _λ represents the weight; represents the probability that the model predicts that the emotional image belongs to the i-th emotional category; f(i) represents the probability of other accompanying emotions; j represents the dominant emotion; i represents other accompanying emotions;

represents the probability of the dominant emotion; p _j represents the category probability of the dominant emotion j; dis _ij represents the distance between the emotion i and the dominant emotion j defined in Mikels'wheel; i,j∈B represents the emotion category i and j in the same in polarity.

In the collection of most image sentiment data, the majority voting strategy is widely adopted to obtain the sentiment labels of images. Starting from the diversity of emotional expressions, we consider the distribution of emotions based on the probability of labels. Inspired by research in emotion theory, the relationship between two emotions determines how similar they are. And two emotions from similar to completely opposite can be expressed through Mikels'Wheel (see Figure 4, Figure 4 (a) is the emotional wheel, (b) is the emotional distance), through the distance defined by Mikels'Wheel, can calculate the two The distance between the emotions, and the distance represents the similarity of the two emotions, that is, the smaller the distance d _ij between the emotion i and the emotion j, the more similar the two emotions are. Therefore, through the definition of Mikels'Wheel, the probability label of emotional image can be obtained.

In order to obtain a more reasonable probability label, the present invention further combines the research of emotion theory. The emotion of emotional images is divided into two polarities: negative (anger, disgust, fear, sadness) N and positive (amusement, awe, contentment, excitement) P , through the study of emotion theory, the diversity expression of emotion is actually more of a dominant emotion accompanied by multiple other emotions of the same polarity, so when generating label probability, the relationship of emotion polarity is introduced.

According to the dominant emotion and the definition of distance in Mikels'Wheel, the probability distribution of other emotions with the same polarity as the dominant emotion is calculated, and the probability distribution of emotions with opposite polarity is set to 0. The calculation formulas are expressed as formulas (7) and (8).

Through the category labels and probability labels of emotional images, a multi-task loss function is introduced, formula (5).

The distributed labels of emotional images are generated according to Mikels'Wheel, and the emotional distribution loss of the predicted distribution and distributed labels is calculated, and a new loss function is formed by introducing the weight λ and the classification loss to achieve the constraints on the diversity of image expression. .

In this embodiment, stochastic gradient descent is used to optimize the multi-task loss function L _multi , and {a _i |i=1,2,...,C} is defined to represent the activation value of the ith emotion category in the last fully connected layer.

The gradient can be calculated by equation (9):

The representation of image emotion is subjective and diverse, while most emotion datasets are collected from single emotion labels generated by voting. However, in fact, different regions of an image may express different emotions, and it is difficult to classify an image into a single emotion type. The emotional expression of an image is often a dominant emotion accompanied by one or more other emotions of the same polarity. Therefore, the use of a single emotion label will lead to inaccurate labeling of image emotion datasets, which greatly affects the accuracy of image emotion classification. Therefore, the relationship of emotion polarity is introduced in this embodiment, and the dominant emotion and other emotions with the same polarity as the dominant emotion are comprehensively considered by introducing a multi-task loss function, so as to improve the accuracy of image emotion classification.

In this embodiment, starting from the diversity of image emotion expression, a new loss function calculated by combining emotion distribution loss is proposed to complete the constraint on the diversity of image emotion expression.

The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when the processor executes the computer program.

The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method are implemented.

The above descriptions are only the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Under the inventive concept of the present invention, the equivalent structural transformations made by the contents of the description and drawings of the present invention, or the direct/indirect application Other related technical fields are included in the scope of patent protection of the present invention.

Claims (10)

1. an image emotion classification method based on saliency detection and multi-level feature fusion, it is characterized in that, described image emotion classification method comprises: constructing an emotional image set; the emotional image set includes marked emotional images; Establish a training set and a verification set according to the emotional image set, and use a saliency detection network to extract the saliency maps of the emotional images in the training set and the verification set; The emotional images and saliency maps in the training set are input into a pre-built image emotion classification model; the image emotion classification model includes a twin neural network and an Inception-v4 network; The Siamese neural network is trained by using the emotional images and saliency maps in the training set, and the saliency map in the training set is used to perform feature modulation on the corresponding emotional images through the trained Siamese neural network to obtain modulation feature map; Use the modulation feature map to train the Inception-v4 network to obtain a trained image emotion classification model; Use the emotional images and saliency maps in the verification set to verify the trained image sentiment classification model; The image to be classified and the saliency map of the to-be-classified image are input into the verified image emotion classification model for classification, and the image emotion category is obtained. 2. The image emotion classification method according to claim 1, wherein the Siamese neural network comprises a first branch and a second branch, the first branch is used to perform feature extraction on emotional images, and the second branch is used to perform feature extraction on emotional images. The saliency map is used for feature extraction; the first branch and the second branch are both composed of four convolutional layers with the same convolution kernel size; the outputs of the second and fourth convolutional layers in the second branch connected to the outputs of the second and fourth convolutional layers in the first branch. 3. image emotion classification method as claimed in claim 2, is characterized in that, utilizes the saliency map in described training set to carry out feature modulation to corresponding emotional image by trained twin neural network, obtains modulation feature map, comprises: The emotional images in the training set are input into the first branch of the trained twin neural network, and the saliency map is input into the second branch; Obtain the feature map S output by the second convolution layer in the second branch, and perform multiplication of the corresponding elements on the feature map S and the feature map T output by the second convolution layer in the first branch to obtain features Figure H, and the feature map H and the feature map T are added with corresponding elements to obtain the feature map G; Input the feature map G into the third convolutional layer of the first branch, and input the feature map S into the third convolutional layer of the second branch; Obtain the feature map S' output by the fourth convolution layer in the second branch, and perform the multiplication operation of the corresponding elements on the feature map S' and the feature map T' output by the fourth convolution layer in the first branch , obtain the feature map H', and perform the addition operation of the corresponding elements on the feature map H' and the feature map T' to obtain the modulation feature map F. 4. image emotion classification method as claimed in claim 3, is characterized in that, the calculation formula of described feature map G is: G=f(T(w,h,c)*[S(w,h)+1]) (1) In the formula, f represents the sigmoid activation function; T(w,h,c) represents the feature map output by the second convolutional layer in the first branch; S(w,h) represents the second convolutional layer in the second branch The output feature map; w, h, c represent the width, height and number of channels of the feature map, respectively; The calculation formula of the modulation characteristic map F is: F=f(T'(w,h,c)*[S'(w,h)+1]) (2) In the formula, T'(w,h,c) represents the feature map output by the fourth convolutional layer in the first branch; S'(w,h) represents the feature map output by the fourth convolutional layer in the second branch . 5. The image emotion classification method as claimed in claim 1, wherein the Inception-v4 network is a multi-branch structure, comprising successively three convolutional layers, a Mixed_3a module, a Mixed_4a module, a Mixed_5a module, four Inception_A modules, one Reduction_A module, seven Inception_B modules, one Reduction_B module, three Inception_C modules, one average pooling layer and one fully connected layer; the Mixed_5a module, Reduction-A module and Reduction-B module are introduced respectively after a side branch structure, the side branch structure is composed of a convolution layer; the output ends of the three side branch structures are connected to a fully connected layer, and the fully connected layer is used to output the three side branch structures The side branch features are fused and output the fused features, and the fused features are output to the output of the average pooling layer. 6. The image emotion classification method as claimed in claim 5, wherein the last fully connected layer in the Inception-v4 network obtains the probability that the emotional image belongs to the i-th emotional category by Softmax mode:

In the formula, _yi represents the probability that the emotional image belongs to the i-th emotional category; z _i represents the activation value of the emotional image belonging to the i-th category; z _j represents the activation value of the emotional image belonging to the j-th category; C represents the emotional category. 7. The image emotion classification method according to claim 1, wherein the image emotion classification model adopts a multi-task loss function based on emotion diversity constraints, and the multi-task loss function is: L _multi =L _cls +λL _ed (4)

In the formula, L _multi represents the multi-task loss function; L _cls represents the traditional classification loss; L _ed represents the loss of other accompanying _emotions ; _λ represents the weight; represents the probability that the model predicts that the emotional image belongs to the i-th emotional category; j represents the dominant emotion; i represents other accompanying emotions;

represents the probability of the dominant emotion; p _j represents the class probability of the dominant emotion j; f(i) represents the probability of other accompanying emotions; dis _ij represents the distance between the emotion i defined in Mikels'wheel and the dominant emotion j; i,j ∈B indicates that sentiment categories i and j are in the same polarity. 8. An image emotion classification device based on saliency detection and multi-level feature fusion, characterized in that, comprising: an image set building module for constructing an emotional image set; the emotional image set includes marked emotional images; A saliency map acquisition module, used for establishing a training set and a verification set according to the emotional image set, and extracting the saliency map of the emotional images in the training set and the verification set by using a saliency detection network; A model training module for inputting the emotional images and saliency maps in the training set into a pre-built image emotion classification model; the image emotion classification model includes a twin neural network and an Inception-v4 network; using the emotion in the training set The image and the saliency map are used to train the twin neural network, and the saliency map in the training set is used to perform feature modulation on the corresponding emotional image through the trained twin neural network to obtain a modulation feature map; using the modulation The feature map is used to train the Inception-v4 network to obtain a trained image emotion classification model; A model verification module for verifying the trained image sentiment classification model using the emotional images and saliency maps in the verification set; The classification module is used for inputting the image to be classified and the saliency map of the image to be classified into the verified image emotion classification model for classification to obtain the image emotion category. 9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when the processor executes the computer program . 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images