CN113673567B - Panoramic image emotion recognition method and system based on multi-angle sub-region self-adaptation - Google Patents

Abstract

The invention discloses a panorama emotion recognition method and system based on multi-angle sub-region self-adaptation, which includes a multi-angle rotation module, a feature extraction module, a sub-region self-adaptation module, a multi-scale fusion module and an emotion classification module for predicting user emotion recognition in an immersive virtual environment. Using the spherical multi-angle rotation algorithm to generate a series of equidistant cylindrical projection panoramas, input the convolutional neural network to obtain different levels of feature advantages. Guide local features through global features, adaptively establish the correlation between current scale context features, and capture the global and local context dependencies of feature maps at different levels. Sampling feature maps at different levels, splicing in the channel dimension to realize feature fusion, and obtaining user's emotional classification labels. The present invention can correctly predict user's emotional preference and distribution in various scenarios, and improve user experience under VR.

Description

Panoramic image emotion recognition method and system based on multi-angle sub-region self-adaptation

technical field

The invention relates to the field of emotion recognition, in particular to a panorama emotion recognition method and system based on multi-angle sub-region self-adaptation.

Background technique

Emotion is a psychological and physiological state, accompanied by the process of cognition and consciousness. The research on human emotion and cognition is the advanced stage of artificial intelligence. With the vigorous development of artificial intelligence and deep learning, it is possible to build emotional models capable of perceiving, recognizing and understanding human emotions. By endowing machines with the ability to make intelligent, sensitive, and friendly feedback on user emotions, and ultimately create a natural environment where humans and humans, and humans and machines coexist harmoniously, this beautiful vision guides a new direction for the future application of computers.

Traditional emotion induction methods include pictures, text, voice, video, etc., but the actual prediction effect of the corresponding emotion recognition data set is not satisfactory. Virtual reality technology achieves the purpose of emotional induction through immersive, realistic and three-dimensional experience, and is a better element of emotional induction. In recent years, deep learning has made revolutionary progress in practice, but in terms of emotional interaction, emotion label data based on virtual reality-induced states is scarce, and effective emotion research methods and models are lacking. Panorama is a storage form of all-round and real spatial information on a two-dimensional plane, which can be used as an effective material for analyzing the emotion of VR immersive virtual environment.

Contents of the invention

In order to overcome the shortcomings and deficiencies of the prior art, the present invention proposes a panorama emotion recognition system and method based on multi-angle sub-region self-adaptation.

The present invention uses the display characteristics of the panoramic content in the head-mounted display and the equidistant columnar projection method to design a spherical multi-angle rotation algorithm to obtain panoramic images of different angles, and combines it with a context-adaptive convolutional neural network to effectively improve the accuracy of emotional classification labels.

The present invention adopts following technical scheme:

A panorama emotion recognition method based on multi-angle sub-region adaptation, comprising:

Multi-angle rotation step: using spherical multi-angle rotation and equidistant cylindrical projection to realize the transformation from three-dimensional omnidirectional stereoscopic view to two-dimensional plane panorama;

Feature extraction step: use the pre-trained convolutional neural network model to perform feature extraction on the two-dimensional plane panorama, and obtain feature maps of different levels;

Sub-region adaptation step: input feature maps of different levels, find global and local correlations, adaptively establish context features of the current scale, and capture global and local context dependencies of feature maps of different levels;

Multi-scale fusion step: Unify the size of feature maps at different levels through the up-sampling step, and stitch them together in the channel dimension to achieve multi-scale feature fusion;

Emotion classification step: According to the advantages of different levels of features, determine the target emotion, and output the corresponding emotion label.

Further, the multi-angle rotation of the spherical surface is specifically:

Establish a three-dimensional spherical coordinate system with the user's head as the center of the sphere, and project the 360-degree panorama presented by the user under the head-mounted display onto the surface of the sphere;

Rotate the projection image according to the content distribution characteristics of the panorama;

The rotation includes horizontal rotation and vertical rotation. Horizontal rotation realizes the rotation of the cut edge content on both sides to the middle main viewing area; vertical rotation realizes the rotation of severely distorted and distorted content at the two poles to near the equator.

Further, the equidistant cylindrical projection is to map the meridians to vertical lines with constant intervals, map the latitude lines to horizontal lines with constant intervals, and project the equidistant cylindrical projection of the three-dimensional stereoscopic view to the two-dimensional panorama.

Furthermore, the three-dimensional spherical coordinates are the right-hand coordinate system, the field of view angle is 90 degrees, and the user's binocular direct viewing direction is taken as the horizontal axis, then the center coordinates of the front viewport are [0,0,0]; the center coordinates of the right viewport are [90,0,0]; the center coordinates of the rear viewport are [180,0,0]; ; corresponding to the six faces of the cube that are tangent to the sphere.

Further, the feature extraction step is specifically:

Input the two-dimensional panoramic image into the pre-trained convolutional neural network, extract the hierarchical structure of different feature spaces common in the visual world, and form a feature vector set [X ¹ ,X ² ,...,X ^l ], each element in the set represents the feature map of the current level.

Further, the sub-area adaptation step includes two branches: a sub-area content characterization branch and an emotional contribution characterization branch;

The sub-area content representation branch uses an input feature map with an input size of h×w×c through an adaptive average pooling operation to obtain a sub-region content representation y ^s , where h, w, c, and s respectively represent the height, width, number of channels and preset size of the feature map;

The emotional contribution degree characterization branch specifically includes:

Perform global pooling on each element in the feature vector set [X ¹ ,X ² ,...,X ^l ] to obtain a global information representation g(X ^l ) with a size of 1×1×c;

Use the broadcast mechanism to add the global information representation g(X ^l ) element by element to the input feature map to realize the residual connection, and convert the number of channels into s ² through the 1x1 convolution operation, so as to construct an adaptive emotional contribution matrix a ^s of size hw×s ² ;

Multiply the adaptive emotion contribution matrix a ^s with the sub-region content representation y ^s to obtain the context feature representation vector Z ^l , which represents the degree of association between each pixel i and each sub-region y ^s×s .

Further, the adaptive average pooling divides the input feature map into s×s sub-regions, obtains a group of sub-region representations Y ^s×s =[y ¹ ,y ² ,...,y ^s×s ], and transforms the feature map with a size of s×s×c into a sub-region content representation y ^s of s ² ×c.

Further, the specific steps of constructing the emotional contribution matrix a ^s are as follows: set the contribution degree of the sub-region y ^s×s to the emotional classification label at point i of the feature map as a _i , then any point i of the feature map corresponds to s×s emotional contribution vectors a _i , forming a set Transformation obtains the emotion contribution degree matrix a ^s , whose size is hw×s ² .

Further, the multi-scale fusion step is specifically: using upsampling operations, such as deconvolution or interpolation operations, to realize multi-scale feature maps at different levels, with uniform sizes, and splicing in the channel dimension to complete feature fusion, and finally obtain the total information representation of the combination of the underlying geometric information representation and the high-level semantic information representation with a size of H×W×(C ₁ +C ₂ +...+C _l ).

A system for implementing a panorama emotion recognition method based on multi-angle sub-region adaptation, comprising:

Multi-angle rotation module: used for multi-angle rotation and equidistant cylindrical projection to realize the conversion from 3D panoramic view to 2D panoramic view;

Feature extraction module: used to perform feature extraction on two-dimensional panoramas to obtain feature maps of different levels;

Sub-region adaptive module: used to correlate regions with consistent emotional classification labels, global features guide local features to adaptively establish the relevance of contextual features at the current scale, and capture long-distance dependencies;

Multi-scale fusion module: used to unify the size of feature maps of different levels and splicing them in the channel dimension to realize multi-scale feature fusion;

Sentiment classification module: According to the advantages of different levels of features, determine the target emotion, and output the corresponding emotion label.

The present invention has following beneficial effect:

1. Aiming at the scarcity of emotional label data in the induced state of virtual reality, a spherical multi-angle rotation algorithm is proposed to achieve data enhancement. Establish a three-dimensional spherical coordinate system for the 360-degree view of the user's virtual environment, rotate the sphere at multiple angles along different coordinate axes, and then perform equidistant cylindrical projection to obtain expanded data samples, which can effectively improve the generalization ability of the model.

2. Equidistant columnar projection projects longitude and latitude equidistantly onto a rectangular plane, which will cause serious distortion and distortion of the panoramic content at the upper and lower poles. The data samples expanded by the spherical multi-angle rotation algorithm can maintain the invariance of rotation and alleviate the distortion. At the same time, the edge information on both sides is rotated to the central main viewing area, so that the content features can be better captured and extracted by the emotional model, and the accuracy of model recognition is improved.

3. Use the pre-trained convolutional neural network to extract features at different levels of the panorama, and give full play to the complementary advantages of the underlying detailed information and high-level semantic information. Guide local features through global features, adaptively establish the correlation between different regions or objects in feature maps, and capture long-distance dependencies. In this way, the prediction performance of the model on the emotion-induced region of the panorama is effectively improved.

4. The present invention fills the gap in the field of emotion recognition of panoramas, and helps to interpret user emotions and collect feedback in an immersive virtual environment, which is crucial for the development of VR application scenarios such as user behavior prediction and VR scene modeling.

Description of drawings

Fig. 1 is a flow chart of the general implementation method of the present invention.

Fig. 2 is a schematic diagram of a user wearing a head-mounted display in a virtual environment.

Fig. 3(a) and Fig. 3(b) are three-dimensional spherical coordinates and two-dimensional schematic diagrams after projection, respectively.

Fig. 4 is a schematic diagram of the effect of the multi-angle rotation algorithm rotating 180 degrees along the x-axis.

Fig. 5 is a schematic diagram of the sub-region adaptive module of the present invention.

Fig. 6 is a schematic diagram of a model framework of the overall implementation method of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Example

As shown in Figure 1, a panorama emotion recognition method based on multi-angle sub-region adaptation is used to identify and predict user emotions in an immersive virtual environment, including the following:

The multi-angle rotation module presents an interactive 360-degree view to the user in the immersive virtual environment, as shown in Figure 2, and uses a spherical multi-angle rotation algorithm to obtain a series of data expansion samples. And use the equidistant cylindrical projection to map the meridians into vertical lines with constant spacing, and map the latitude lines into horizontal lines with constant spacing, so as to complete the conversion from three-dimensional omnidirectional stereoscopic view to two-dimensional flat panorama.

HMD in Fig. 2 represents a head-mounted display.

The spherical multi-angle rotation algorithm is specifically as follows: establish a three-dimensional Cartesian coordinate system with the user's head as the center of the sphere. The sphere is rotated by a certain angle along the horizontal axis, so that the object that is seriously distorted at the two poles is rotated at multiple angles to near the equator to improve the distortion. At the same time, the sphere is rotated by a certain angle along the vertical axis, and the cut edge content on both sides is rotated to the central main viewing area.

The purpose of using the multi-angle rotation algorithm is to rotate the emotion-inducing area of the panorama to a position close to the equator of the main view according to the content distribution characteristics of the panorama, so as to reduce the adverse effects caused by distortion and distortion projection, and facilitate the model to capture relevant features.

The rotation includes horizontal rotation and vertical rotation. Horizontal rotation realizes the rotation of the cut edge content on both sides to the middle main viewing area; vertical rotation realizes the rotation of severely distorted and distorted content at the two poles to near the equator.

Further, the spherical multi-angle rotation algorithm specifically includes the following steps:

Construct a three-dimensional spherical coordinate system with the user's head as the origin o, which conforms to the right-handed coordinate system, as shown in Figure 3(a). Using the spherical multi-angle rotation algorithm, the sphere is rotated 90 degrees in the horizontal direction and repeated twice to realize the rotation of the cut edge content on both sides to the middle main viewing area, as shown in Figure 4. Then rotate the sphere 45 degrees in the vertical direction, repeat 4 times, and rotate the object that was severely distorted at the poles to near the equator to improve the distortion. Each panorama gets 2x4=8 kinds of data augmentation results.

Suppose the height of the panorama is H, the width is W, the coordinates of any point on the plane are (u, v), the corresponding three-dimensional ball coordinates are (x, y, z), and the latitude and longitude are Then the relationship between latitude and longitude and spherical coordinates is as follows:

The conversion formula of the same point in three-dimensional space and two-dimensional plane is as follows:

The meridian is mapped to vertical lines with constant spacing, and the latitude is mapped to horizontal lines with constant spacing, as shown in Figure 3(b).

In the field of emotion recognition, due to the limitation of content distortion in the panorama ERP storage format, in order to facilitate the model to capture relevant features, the multi-angle algorithm needs to rotate the object or area that induces emotion to the position close to the equator of the main view, so that it is projected to the center of the two-dimensional plane through an equidistant rectangle. However, different panoramas require different rotation angles, and it is impractical to manually customize each panorama. The invention facilitates batch preprocessing by setting a unified rotation angle and times. Generally speaking, the sphere is rotated horizontally by 90 degrees, repeated twice, and then rotated by 45 degrees along the x-axis, repeated four times, and each panorama can obtain 2x4=8 kinds of results, which can basically meet the above requirements.

The feature extraction module uses a convolutional neural network pre-trained on large-scale image classification tasks to achieve feature extraction. For the input image I, use the formula X ^l = f(Σk ^l X ^l-1 +b ^l ) to extract the hierarchical structure of different feature spaces common to the visual world, and form a set of feature map vectors [X ¹ ,X ² ,...,X ^l ]. Among them, k ^l is the convolution kernel of layer l, X ^l-1 is the feature map output by layer l-1, and b ^l is the bias item. Each element in the set represents the feature map of the current level, which is used as the input of the sub-region adaptation module to play the complementary advantages of different levels of information.

The sub-region adaptive module, as shown in Figure 5, adaptively establishes the context features of the current scale by looking for global and local correlations, and captures the global and local context dependencies of feature maps at different levels. This module is composed of sub-area content characterization branch and emotional contribution characterization branch, specifically:

The sub-area content representation branch performs adaptive average pooling on each element in the feature vector set [X ¹ ,X ² ,...,X ^l ]. The adaptive average pooling function is defined as follows:

kernel_size=(input_size+2×padding)-(output_size-1)×stride

That is, the input size, output size, boundary padding and moving step determine the size of the current convolution kernel. Convert the feature map ^Xl with size h×w×c to s×s×c, where h, w, c, s represent the height, width, number of channels and preset size of the feature map respectively. Then adaptive average pooling divides the input feature map into s×s sub-regions, and obtains a set of sub-regions representing Y ^s×s =[y ¹ ,y ² ,...,y ^s×s ]. Transform a feature map of size s×s×c into a sub-region content representation y ^s of s ² ×c.

The emotional contribution representation branch performs global average pooling on each element in the feature vector set [X ¹ ,X ² ,...,X ^l ] to obtain a global information representation g(X ^l ) with a size of 1×1×c. The broadcast mechanism is used to add the 1×1×c global information representation and the input feature map pixel by pixel to realize the residual connection, and obtain the feature map with the size of h×w×c.

Suppose the contribution degree of the sub-region y ^s×s to the emotional classification label at point i of the feature map is a _i , and the number of channels is converted to s ² through a 1x1 convolution operation, then any point i of the feature map corresponds to s×s emotional contribution vectors a _i , forming a set The transformation results in a hw×s ² adaptive emotion contribution matrix a ^s .

Multiply the emotional contribution degree matrix a ^s output by the emotional contribution degree representation branch with the sub-region content representation y ^s output by the sub-region content representation branch, the function is defined as follows:

The contextual feature representation vector Z ^l is obtained, which represents the degree of association between each pixel i and each sub-region y ^s×s , and its internal emotional contribution vector A _i represents the global and local connection weights, which are automatically optimized with the continuous iteration of the network.

Further, the dependence refers to the correlation between two or more emotional subjects. The feature extraction module uses the global and local features of the panorama to realize the recognition of different regions or objects, such as emotional subjects such as people and cats, but this is not enough as a standard for emotional prediction. It is also necessary to adaptively establish the correlation between the human and the cat through the sub-region adaptive module, and the human is teasing or stroking the kitten, so as to give the correct positive emotional label.

The multi-scale fusion module realizes feature fusion of feature maps at different levels. The upsampling operation is used to unify the size of the feature maps of different levels, and then the feature maps of the uniform size are spliced in the channel dimension, and finally the combination of the underlying geometric information representation and the high-level semantic information representation with a size of H×W×(C ₁ +C ₂ +...+C _l ) is obtained.

The emotion classification module can achieve high emotional classification effect for panoramas with salient subjects and panoramas without salient subjects. Due to the parameter redundancy of the fully connected layer, the global average pooling is used to replace the fully connected layer to play the role of a "classifier". Emotion recognition for panoramas with salient subjects using deep features that pay more attention to abstract semantic information. Emotion recognition for panoramas without salient subjects using shallow features that provide perceptual information about details such as edges, stripes, and colors. The emotional classification labels with higher accuracy are obtained. The overall framework of the model is shown in Figure 6.

The features extracted by convolution operations at different levels of the feature extraction module are different. The low-level convolutions such as conv layer_1 and 2 extract visual layer features, such as color, texture, outline, etc., and the high-level convolutions such as conv layer 4 and 5 extract object layer and concept layer features, that is, abstract semantic information. Predicting the emotional regions of different/same panoramas needs to combine the feature advantages of different levels. If the content of the panorama is a single and straightforward natural scene, the underlying color and texture information is the key to correct classification; if the content of the panorama is a complex multi-object interaction scene, then the high-level semantic information is very important. The sub-region adaptation module helps to better capture the emotion-induced regions by establishing the correlation between different regions and objects in the feature map, so as to give the correct emotion labels.

In this embodiment, the feature extraction module extracts the 4-layer feature maps of conv layer_2, 3, 4, and 5. At the same time, the feature maps of each layer are sent to the sub-region adaptive module, and the correlation of different regions is established under different scales S=1, 2, 4, n (the number of s is not limited, generally the combination of 1, 2, and 4 has the best effect). Because the size of feature maps at different levels is different, it is necessary to use a multi-scale fusion module. First, unify the scale, and then stitch all the above-mentioned feature maps in the channel dimension. The total features after stitching are used as the basis for emotion classification, and finally the emotional polarity of the input panorama is obtained, that is, positive or negative.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the embodiment, and any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods, and are all included within the protection scope of the present invention.

Claims (6)

1. A panorama emotion recognition method based on multi-angle sub-region self-adaptation, characterized in that, comprising: Multi-angle rotation step: using spherical multi-angle rotation and equidistant cylindrical projection to realize the transformation from three-dimensional omnidirectional stereoscopic view to two-dimensional plane panorama; Feature extraction step: use the pre-trained model to perform feature extraction on the two-dimensional panorama, and obtain feature maps of different levels; Sub-region adaptation step: input feature maps of different levels, find global and local correlations, adaptively establish context features of the current scale, and capture global and local context dependencies of feature maps of different levels; Multi-scale fusion step: splicing feature maps of different levels in the channel dimension to realize multi-scale feature fusion; Emotion classification step: according to the advantages of different levels of features, determine the target emotion, and output the corresponding emotion label; The multi-angle rotation of the spherical surface is specifically: Establish a three-dimensional spherical coordinate system with the user's head as the center of the sphere, and project the 360-degree panorama presented by the user under the head-mounted display onto the surface of the sphere; Rotate the projection image according to the content distribution characteristics of the panorama; The rotation includes horizontal rotation and vertical rotation. Horizontal rotation realizes the rotation of the edge content cut on both sides to the middle main viewing area; vertical rotation realizes the rotation of the severely distorted content at the two poles to the vicinity of the equator; The equidistant cylindrical projection is to map the meridians into vertical lines with constant spacing, map the latitude lines into horizontal lines with constant spacing, and project the equidistant cylindrical projection of the three-dimensional stereoscopic view to the two-dimensional panorama; The sub-area self-adaptation step includes two branches: a sub-area content characterization branch and an emotional contribution characterization branch; The sub-region content representation branch uses the feature map with an input size of h×w×c through an adaptive average pooling operation to obtain a sub-region content representation y ^s , where h, w, c, and s represent the height, width, channel number and preset size of the feature map; The emotional contribution degree characterization branch specifically includes: Perform global pooling on each element in the feature vector set [X ¹ ,X ² ,...,X ^l ] to obtain a global information representation g(X ^l ) with a size of 1×1×c; Use the broadcast mechanism to add the global information representation g(X ^l ) element by element to the input feature map to realize the residual connection, and convert the number of channels into s ² through the 1x1 convolution operation, so as to construct an adaptive emotional contribution matrix a ^s of size hw×s ² ; Multiply the adaptive emotion contribution matrix a ^s with the sub-region content representation y ^s to obtain the context feature representation vector Z ^l , which represents the degree of association between each pixel i and each sub-region y ^s×s . 2. panorama emotion recognition method according to claim 1, it is characterized in that, three-dimensional spherical coordinates is the right-hand coordinate system, and angle of view is 90 degrees, and user binocular direct-looking direction is used as horizontal axis, then front viewport center coordinates are [0,0,0]; Right viewport center coordinates are [90,0,0]; Back viewport center coordinates are [180,0,0]; Left viewport center coordinates are [-90,0,0]; Upper viewport center coordinates are [0,90,0] ;The coordinates of the center of the lower viewport are [0,-90,0]; corresponding to the six faces of the cube that are tangent to the sphere. 3. panorama picture emotion recognition method according to claim 1, is characterized in that, described feature extraction step is specifically: Input the two-dimensional panoramic image into the pre-trained convolutional neural network, extract the hierarchical structure of different feature spaces common in the visual world, and form a feature vector set [X ¹ ,X ² ,...,X ^l ], each element in the set represents the feature map of the current level. 4. The panorama emotion recognition method according to claim 1, characterized in that constructing an adaptive emotion contribution degree matrix a ^s with a size of hw×s ² , specifically: setting the contribution degree of the sub-region y ^s×s to the emotion classification label at point i of the feature map as a _i , converting the number of channels into s ² through a 1×1 convolution operation, then any point i of the feature map corresponds to s×s emotion contribution degree vectors a _i , forming a set The transformation results in a hw×s ² adaptive emotion contribution matrix a ^s . 5. The panorama emotion recognition method according to claim 1, wherein the adaptive average pooling divides the input feature map into s×s sub-regions, obtains a group of sub-region representations Y ^s×s =[y ¹ ,y ² ,...,y ^s×s ], and transforms the feature map whose size is s×s×c into a sub-region content representation y ^s of s ² ×c. 6. A system that realizes the panorama emotion recognition method based on the multi-angle sub-region adaptation described in any one of claims 1-5, is characterized in that, comprising: Multi-angle rotation module: used for multi-angle rotation and equidistant cylindrical projection to realize the conversion from 3D panoramic view to 2D panoramic view; Feature extraction module: perform feature extraction on two-dimensional panoramic images, obtain feature maps of different levels, and capture global and local context dependencies of feature maps; Sub-area adaptive module: correlate areas with consistent emotional classification labels, and adaptively establish contextual features of the current scale by looking for global and local correlations; Multi-scale fusion module: splicing feature maps in the channel dimension for multi-scale feature fusion; Sentiment classification module: According to the advantages of different levels of features, determine the target emotion, and output the corresponding emotion label.