CN110046656A - Multi-modal scene recognition method based on deep learning - Google Patents

Abstract

The invention discloses a multi-modal scene recognition method based on deep learning, which includes the following steps: S1, performing word segmentation processing on short text; S2, inputting a group of pictures and short text word segmentation and corresponding labels into respective convolutional neural networks Train in the network; S3, train the short text classification model; S4, train the image classification model; S5, calculate the cross entropy of the fully connected layer output in S3 and S4 and the standard classification result respectively, calculate the average Euclidean distance and use this as the loss value, and then feed it back to the respective convolutional neural networks, and finally obtain a complete multimodal scene recognition model; S6, add the text and image prediction result vectors to obtain the final classification result; S7, add the short text to be recognized and images are respectively input into the trained multi-modal scene recognition model for scene recognition. The present invention proposes a multi-modal scene search method, which provides users with more accurate and convenient scene recognition.

Description

Multimodal scene recognition method based on deep learning

technical field

The invention relates to a multimodal scene recognition method, in particular to a multimodal scene recognition method based on deep learning, and belongs to the fields of artificial intelligence and pattern recognition.

Background technique

Deep learning is a new field of machine learning. Its purpose is to make machine learning closer to human intelligence. Convolutional neural network is a representative algorithm of deep learning. It has the characteristics of simple structure, strong adaptability, few training parameters and many connections. Therefore, this network has been widely used in image processing and pattern recognition for many years.

Specifically, a convolutional neural network is a hierarchical model whose input is raw data. Through a series of operations such as convolution operations, pooling operations, and nonlinear activation functions, the high-level semantic information is transformed from the original data layer by layer. The data input layer is extracted and abstracted layer by layer. This process is called "feedforward operation". Finally, the last layer of the convolutional neural network outputs the objective function. By designing the loss function, the error loss between the predicted value and the real value is calculated, and then the error is fed back layer by layer from the last layer through the back propagation algorithm to update parameters at each layer and feed forward again after updating the parameters. Repeat this until the network model converges, so as to achieve the purpose of model training.

At present, the commonly used modal fusion methods mainly include decision fusion and feature fusion.

Decision fusion refers to the weighted synthesis of the two types of results on the basis of obtaining the two modal classification results to obtain the final result. Meng-Ju Han et al. proposed a decision fusion strategy in their research, which normalized the average Euclidean distance between the training samples and the decision plane as the fusion weight, and achieved about 5% higher than single-modality. Recognition rate. Although the process of decision fusion is relatively simple, the results obtained are not objective enough.

Feature fusion refers to classifying again after the features extracted from the two modalities are fused. In their research, S.Emerich et al. performed feature fusion on the extracted facial expression features and speech features, and the fusion feature recognition rate and robustness were both improved compared to single-modality features. The results obtained by the method of feature fusion are relatively objective, but the implementation method is too complicated.

To sum up, how to propose a new multi-modal scene recognition method on the basis of the existing technology, keep the respective advantages of decision fusion and feature fusion as much as possible, and overcome their respective shortcomings, which becomes the It solves the problem to be solved urgently by those skilled in the art.

SUMMARY OF THE INVENTION

In view of the above-mentioned defects in the prior art, the purpose of the present invention is to propose a multi-modal scene recognition method based on deep learning, comprising the following steps:

S1. Perform word segmentation on short texts;

S2. Input a set of pictures and short text word segmentation and corresponding labels into their respective convolutional neural networks for training;

S3. Train a short text classification model;

S4, train the image classification model;

S5. Calculate the cross-entropy of the fully connected layer output in S3 and S4 and the standard classification result respectively, calculate the average Euclidean distance and use this as the loss value, and then feed back to the respective convolutional neural network, and repeat the training until the model converges , and finally obtain a complete multimodal scene recognition model;

S6. Add the trained text and image prediction result vectors to obtain the final classification result;

S7. Input the short text and image to be recognized into the trained multimodal scene recognition model, respectively, to perform scene recognition.

Preferably, S1 specifically includes the following steps: using a stuttering word segmentation tool to perform word segmentation processing on the short text.

Preferably, S3 specifically includes the following steps:

S31, quantify the word segmentation result of the input short text, and input it into three parallel convolutional layers;

S32, sending the outputs of the three parallel convolutional layers to the linear correction unit layer and the pooling layer in turn to obtain multiple pooled output results;

S33. Connect a plurality of the pooled output results together, discard them at random, and use them as the input of the fully connected layer, and finally calculate the fully connected layer to obtain a text classification prediction result vector output.

Preferably, the three parallel convolutional layers include a first convolutional layer, a second convolutional layer and a third convolutional layer, and the first convolutional layer has 384 convolution kernels with a size of 3*128, so The second convolution layer has 256 convolution kernels of size 4*128, and the third convolution layer has 128 convolution kernels of size 5*128.

Preferably, S4 specifically includes the following steps:

S41. Send the input image to the first-layer convolutional network, extract the corresponding number of features in the image through the number of designed convolution kernels, and output the result of the convolution layer;

S42. Pool the output of the convolution layer, compress the amount of data kernel parameters through the convolution kernel, reduce overfitting, and then input the pooling result into the next layer of convolution, and repeat the convolution pooling for 4 times to make The weights in the convolution kernel are initialized to random values, and the model parameters are obtained through continuous training;

S43: Input the pooling result of the last layer into the fully connected layer, and after random discarding, the image classification prediction result vector output is obtained by calculation.

Preferably, calculating the average Euclidean distance as described in S5 and using it as the loss value specifically includes the following steps: using the loss function S to calculate the loss value, and the calculation formula of the loss function S is:

Among them, h ₁ =H(p ₁ ,q ₁ ),h ₂ =H(p ₂ ,q ₂ ),h ₃ =H(p ₁ ,p ₂ ), p ₁ is the text classification prediction result vector output in S3 , p ₂ is the image classification prediction result vector output in S4, q ₁ is the text classification standard result vector, q ₂ is the image classification standard result vector, and H( ) is the cross entropy function.

Preferably, S6 specifically includes the following steps: using the Softmax function to add the trained text and image prediction result vectors to obtain the final classification result.

Compared with the prior art, the advantages of the present invention are mainly reflected in the following aspects:

The deep learning-based multi-modal scene recognition method provided by the present invention proposes a brand-new multi-modal scene search method, and provides users with a more accurate and convenient scene recognition method. The method of the invention comprehensively extracts the features of text and images, designs a new loss function, and utilizes the information of various modalities to improve the accuracy of scene recognition.

The present invention also provides reference for other related problems in the same field, and can be expanded and extended based on this, and has a very broad application prospect when applied to other technical solutions related to the scene recognition method.

The specific embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings of the embodiments, so as to make the technical solutions of the present invention easier to understand and grasp.

Description of drawings

FIG. 1 is a schematic structural diagram of a multimodal scene recognition model constructed by the present invention.

Detailed ways

Aiming at the problems of inaccurate results and high complexity of the existing scene recognition methods, the present invention provides a new multi-modal scene recognition method based on deep learning. feature information of text modalities, and fuse multi-modal feature information to improve the accuracy of scene recognition.

Further, the deep learning-based multimodal scene recognition method of the present invention includes the following steps.

S1. Use the stuttering word segmentation tool to segment the short text.

S2. Input a set of pictures and short text word segmentation and corresponding labels into the respective convolutional neural networks for training.

S3. Train a short text classification model. Specifically include the following steps:

S31. In the training process of the short text classification model, quantify the input short text word segmentation result and input it into three parallel convolutional layers.

The three parallel convolutional layers include a first convolutional layer, a second convolutional layer, and a third convolutional layer, the first convolutional layer has 384 convolution kernels of size 3*128, the second convolutional layer The convolution layer has 256 convolution kernels of size 4*128, and the third convolution layer has 128 convolution kernels of size 5*128.

S32. Send the outputs of the three parallel convolutional layers to the linear correction unit (relu) layer and the pooling layer in turn to obtain multiple pooled output results.

S33. Connect a plurality of the pooled output results together, discard them at random, and use them as the input of the fully connected layer, and finally calculate the fully connected layer to obtain a text classification prediction result vector output.

S4. Train the image classification model. Specifically include the following steps:

S41. Send the input image to the first-layer convolutional network, extract the corresponding number of features in the image through the designed number of convolution kernels, and output the result of the convolution layer.

S42. Pool the output of the convolution layer, compress the amount of data kernel parameters through the convolution kernel, reduce overfitting, and then input the pooling result into the next layer of convolution, and repeat the convolution pooling for 4 times to make The weights in the convolution kernel are initialized to random values, and are continuously trained to obtain model parameters suitable for the method of the present invention.

S43: Input the pooling result of the last layer into the fully connected layer, and after random discarding, the image classification prediction result vector output is obtained by calculation.

S5. Calculate the cross-entropy of the fully connected layer output in S3 and S4 and the standard classification result respectively, calculate the average Euclidean distance and use this as the loss value, and then feed back to the respective convolutional neural network, and repeat the training until the model converges , and finally a complete multimodal scene recognition model is obtained. The model structure is shown in Figure 1.

The calculating the average Euclidean distance as the loss value specifically includes the following steps: using the loss function S to calculate the loss value, and the calculation formula of the loss function S is:

Among them, h ₁ =H(p ₁ ,q ₁ ),h ₂ =H(p ₂ ,q ₂ ),h ₃ =H(p ₁ ,p ₂ ), p ₁ is the text classification prediction result vector output in S3 , p ₂ is the image classification prediction result vector output in S4, q ₁ is the text classification standard result vector, q ₂ is the image classification standard result vector, and H( ) is the cross entropy function.

S6. Use the Softmax function to add the trained text and image prediction result vectors to obtain the final classification result.

S7. Input the short text and image to be recognized into the trained multimodal scene recognition model, respectively, to perform scene recognition.

In general, the present invention integrates the image convolutional neural network and the short text convolutional neural network, adopts a new decision fusion method, first obtains two types of classification results through training, and then calculates the cross-entropy between the standard results and the results. Then calculate the cross entropy between the classification results obtained from the training between the two modalities, and finally calculate the average Euclidean distance of the three as the loss value to return to the feedforward network to update the parameters, which has a higher recognition rate than the prior art. .

The deep learning-based multi-modal scene recognition method provided by the present invention proposes a brand-new multi-modal scene search method, and provides users with a more accurate and convenient scene recognition method. The method of the invention comprehensively extracts the features of text and images, designs a new loss function, and utilizes the information of various modalities to improve the accuracy of scene recognition.

The present invention also provides reference for other related problems in the same field, and can be expanded and extended based on this, and has a very broad application prospect when applied to other technical solutions related to the scene recognition method.

The present invention also provides reference for other related problems in the same field, and can be expanded and extended on the basis of this, and has a very broad application prospect when applied to other technical solutions related to the emotion recognition and analysis method.

It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, but that the present invention may be embodied in other specific forms without departing from the spirit and essential characteristics of the present invention. Therefore, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the invention is to be defined by the appended claims rather than the foregoing description, which are therefore intended to fall within the scope of the claims. All changes that come within the meaning and range of equivalents of , are intended to be embraced within the invention, and any reference signs in the claims shall not be construed as limiting the involved claim.

In addition, it should be understood that although this specification is described in terms of embodiments, not each embodiment only includes an independent technical solution, and this description in the specification is only for the sake of clarity, and those skilled in the art should take the specification as a whole , the technical solutions in each embodiment can also be appropriately combined to form other implementations that can be understood by those skilled in the art.

Claims (7)

1. a kind of multi-modal scene recognition method based on deep learning, which comprises the steps of:

S1, word segmentation processing is carried out to short text；

S2, one group of picture and short text participle and corresponding label are inputted in respective convolutional neural networks and is trained；

S3, training short text disaggregated model；

S4, training picture classification model；

S5, the full articulamentum output in S3 and S4 is calculated into cross entropy with criteria classification result respectively, calculates average Euclidean distance And in this, as penalty values, respective convolutional neural networks are then fed back to again, training is repeated, until model is restrained, most Complete multi-modal scene Recognition model is obtained eventually；

S6, trained text is added with image prediction result vector, obtains final classification results；

S7, short text to be identified and image are inputted to the multi-modal scene Recognition model trained respectively, carry out field Scape identification.

2. the multi-modal scene recognition method according to claim 1 based on deep learning, which is characterized in that S1 is specifically wrapped It includes following steps: word segmentation processing being carried out to short text using stammerer participle tool.

3. the multi-modal scene recognition method according to claim 1 based on deep learning, which is characterized in that S3 is specifically wrapped Include following steps:

S31, the short text word segmentation result of input is quantified, is inputted in three parallel-convolution layers；

S32, the output of three parallel-convolution layers is sequentially sent in linear amending unit layer and pond layer, obtains multiple ponds Change output result；

S33, multiple pondization output results are linked together, by random drop, as the input of full articulamentum, finally Full articulamentum is calculated, the output of text classification prediction result vector is obtained.

4. the multi-modal scene recognition method according to claim 3 based on deep learning, it is characterised in that: described three Parallel-convolution layer includes the first convolutional layer, the second convolutional layer and third convolutional layer, and first convolutional layer has 384 3* The convolution kernel of 128 sizes, second convolutional layer have the convolution kernel of 256 4*128 sizes, and the third convolutional layer has The convolution kernel of 128 5*128 sizes.

5. the multi-modal scene recognition method according to claim 3 based on deep learning, which is characterized in that S4 is specifically wrapped Include following steps:

S41, the picture of input is sent into first layer convolutional network, is extracted by the convolution kernel number of design corresponding special in picture Number is levied, convolutional layer result is exported；

S42, the output of convolutional layer is subjected to pond, by the amount of convolution kernel compressed data nuclear parameter, reduces over-fitting, then by pond Change result and input next layer of convolution, passes through 4 convolution ponds repeatedly, make the weight initialization random value in convolution kernel, not Disconnected training obtains model parameter；

S43, image classification prediction result is calculated by random drop in the full articulamentum of the last layer pond result input Vector output.

6. the multi-modal scene recognition method according to claim 5 based on deep learning, which is characterized in that described in S5 Calculate average Euclidean distance and in this, as penalty values, specifically comprise the following steps: using loss function S calculate penalty values, institute The calculation formula for stating loss function S is

Wherein, h₁=H (p₁,q₁),h₂=H (p₂,q₂),h₃=H (p₁,p₂), p₁For the text classification prediction result that is exported in S3 to Amount, p₂For the image classification prediction result vector exported in S4, q₁For text classification standard results vector, q₂For image classification mark Quasi- result vector, H () are to intersect entropy function.

7. the multi-modal scene recognition method according to claim 1 based on deep learning, which is characterized in that S6 is specifically wrapped It includes following steps: trained text being added with image prediction result vector using Softmax function, obtains final classification As a result.