CN109472232A - Video semantic representation method, system and medium based on multimodal fusion mechanism - Google Patents

Abstract

The present disclosure discloses a video semantic representation method, system and medium based on a multi-modal fusion mechanism. Feature extraction: extracts the visual features, voice features, motion features, text features and domain features of the video itself; Speech, motion, text features and domain features are fused through the constructed multi-level implicit Dirichlet distribution topic model; feature mapping: the fused features are mapped to a high-level semantic space, and the fused feature representation sequence is obtained . The model takes advantage of the unique advantages of topic models in the field of semantic analysis, and the video representations obtained by the training of the proposed model have ideal distinction in the semantic space.

Description

Video semantic representation method, system and medium based on multimodal fusion mechanism

technical field

The present disclosure relates to a video semantic representation method, system and medium based on a multimodal fusion mechanism.

Background technique

With the explosive growth of the amount of data in the Internet age, the arrival of the era of media big data has been accelerated. Among them, video, as an important carrier of multimedia information, is closely related to people's lives. The evolution of massive data not only requires great changes in data processing methods, but also brings great challenges to video storage, processing and application. An urgent problem to be solved is how to organize and manage data effectively. With the continuous generation of data, due to the limitation of hardware conditions, data can only be stored in segments or time-sharing, which will inevitably result in different degrees of information loss. Therefore, providing a concise and efficient data representation method for video is meaningful for video analysis and improving data management efficiency.

Video data has the following characteristics: 1) In terms of data form, video data has a multi-modal and complex structure, which is an incompletely structured data stream. Each video is a streaming structure composed of a series of image frames distributed along the time axis, showing various characteristics such as vision and motion in the multi-dimensional space of space and time, and at the same time incorporating audio characteristics in the time span. It has strong expressiveness and large amount of information, and the content it contains is rich, massive, and unstructured. The multimodality contained in videos brings great challenges to video representation; 2) In terms of content composition, videos have strong logic. It is composed of a series of logical units and contains rich semantic information. Through several consecutive frames, events that occur in a specific time and space environment can be depicted to express specific semantic content. The diversity of video content and the difference and ambiguity in understanding video content make it difficult to extract features that characterize video data, which in turn makes video understanding based on semantic information more challenging.

Traditional data representation methods, such as vision-based video feature learning methods, can obtain concise representations of videos. However, in order to reasonably construct good features, certain experience and professional domain characteristics are required. The application of deep learning methods has made significant progress in vision tasks, but there are still problems such as "semantic gap" and "multimodal heterogeneity gap". At present, establishing an effective representation of video by adopting multimodal fusion technology is an effective way to cross the "multimodal heterogeneity" gap. The most natural way to understand video is to express the content of the video based on the multimodal information in the video and use the high-level concepts in human thinking, which is also the best way to cross the "semantic gap". However, for video analysis in a specific domain, it is necessary to comprehensively use the corresponding domain features and existing multimodal fusion techniques to mine effective representation patterns to complete specific tasks. Despite the continuous development of computer technology, how to make computers accurately understand the semantic concepts in videos is still a difficult problem.

SUMMARY OF THE INVENTION

In order to solve the deficiencies of the prior art, the present disclosure provides a video semantic representation method, system and medium based on a multimodal fusion mechanism, which is an extensible general representation model. In the process of model training and overall optimization, not only for single The number of modal information is scalable, and domain features contained in any type of video can be incorporated into the model. The model fully considers the relationship between the modalities, and integrates the multi-modal interaction process into the joint training and overall optimization process of the entire model. The model takes advantage of the unique advantages of topic models in the field of semantic analysis, and the video representations obtained by the training of the proposed model have ideal distinction in the semantic space.

In order to solve the above-mentioned technical problems, the present disclosure adopts the following technical solutions:

As a first aspect of the present disclosure, a video semantic representation method based on a multimodal fusion mechanism is provided;

Video semantic representation methods based on multimodal fusion mechanism, including:

Feature extraction: extract the visual features, voice features, motion features, text features and domain features of the video itself;

Feature fusion: The extracted visual, speech, motion, text features and domain features are used for feature fusion through the constructed multi-level implicit Dirichlet distribution topic model;

Feature mapping: Map the fused features to a high-level semantic space to obtain the fused feature representation sequence.

As some possible implementations, the specific steps for extracting visual features of a video are:

Preprocessing: video segmentation, that is, dividing the video into several shots; the image frames in each shot form a sequence of image frames in chronological order;

Step (a1): establish a deep learning neural network model;

The deep learning neural network model includes: an input layer, a first convolution layer C1, a first pooling layer S2, a second convolution layer C3, a second pooling layer S4, and a third convolution layer C5 connected in sequence , the fully connected layer F6 and the output layer;

Step (a2): input the image frame sequence of each shot of the video into the input layer of the deep learning neural network model, and the input layer transfers the image frame to the first convolution layer C1;

The first convolution layer C1 uses a set of trainable convolution kernels to convolve each frame of the image frame sequence of the video, and averages the feature maps of each layer obtained by the convolution to obtain the average feature map, and then obtains the average feature map. The average feature map of , together with the biased input activation function, outputs a set of feature maps;

The first pooling layer S2 performs an overlapping pooling operation on the pixel value of each pixel of the feature map obtained by the first convolution layer C1 to reduce the length and width of the output feature map matrix of the first convolution layer; Then pass the operation result to the second convolutional layer C3;

The second convolution layer C3 is used to perform a convolution operation on the operation result of the first pooling layer S2; the number of convolution kernels of the second convolution layer C3 is the number of convolution kernels of the first convolution layer C1 twice;

The second pooling layer S4 is used to perform an overlapping pooling operation on the feature map output by the second convolution layer C3 to reduce the size of the feature map matrix;

The third convolution layer C5 uses a convolution kernel of the same size as the second pooling layer S4 to perform convolution operations on the results of the second pooling layer S4, and finally obtains several 1×1 pixel feature maps;

The fully connected layer F6 fully connects each neuron in this layer with each neuron in the third convolution layer C5, and expresses the result obtained by the third convolution layer C5 as a feature vector;

The output layer is used to input the feature vector output by the fully connected layer F6 into the classifier for classification, and calculate the classification accuracy. When the classification accuracy is lower than the set threshold, adjust the parameters through back propagation, and repeat step (a2 ) until the classification accuracy is higher than the set threshold; when the classification accuracy is higher than the set threshold, the feature vector corresponding to the classification accuracy higher than the set threshold is used as the learning result of the final video visual feature.

As some possible implementations, the specific steps for extracting the voice features of the video are:

Extract the speech signal in the video, convert the audio data into spectrogram, use the spectrogram as the input of the deep learning neural network model, and then perform unsupervised learning on the audio information through the deep learning neural network model, and through the fully connected layer, get Vector representation of video speech features.

As some possible implementations, the specific steps for extracting the motion features of the video are:

The optical flow field in the video is extracted, and the optical flow direction is weighted and counted to obtain the histogram feature of the optical flow direction information (HOF Histogram of Oriented Optical Flow), which is used as the vector representation of the motion feature.

As some possible implementations, the specific steps for extracting the text features of the video are as follows:

Collect the text in the video frame and the surrounding text information of the video (such as video title, annotation, etc.), and use the bag-of-words model to extract text features from the text information.

The domain feature refers to the rule feature set by the domain to which the video belongs. For example, a football video will do some scene specifications (such as frontcourt, backcourt and penalty area, etc.) and event definitions (such as shooting, corner kick, free kick, etc.) according to the rules and broadcast specifications of football games. News videos have basically the same temporal structure and scene semantics, that is, news footage switches chronologically between announcers and news reports. Advertising videos usually contain logo information associated with the advertised goods or services.

As some possible implementations, the specific steps of the multimodal feature fusion are:

Step (a1): Use LDA Latent Dirichlet Allocation to map the visual feature vector of the video from the visual feature space to the semantic feature space Γ; the input of the LDA latent Dirichlet distribution topic model is the visual feature vector of the video, and the output of the LDA implicit Dirichlet distribution topic model is the semantic representation on the feature space Γ;

Step (a2): Use the LDA implicit Dirichlet distribution topic model to map the speech feature vector of the video from the speech feature space to the semantic feature space Γ; the input of the LDA implicit Dirichlet distribution topic model is the speech feature of the video vector, the output is the semantic representation on the feature space Γ;

Step (a3): Use the LDA implicit Dirichlet distribution topic model to map the optical flow direction information histogram feature of the video from the motion feature space to the semantic feature space Γ; the input of LDA is the video optical flow direction information histogram feature, the output is the semantic representation on the feature space Γ;

Step (a4): LDA implicit Dirichlet distribution topic model, which maps the text features of the video from the text feature space to the semantic feature space Γ; the input of LDA is the text of the video, and the output is the semantic representation on the feature space Γ ;

Step (a5): Convert the video domain features into prior knowledge Ω;

Step (a6): Using the LDA implicit Dirichlet distribution topic model, the semantic representation of each modal feature obtained from steps (a1) to (a4) on the semantic feature space Γ is used to set the weight of each modal feature. , the video representation after modal fusion is obtained by weighted fusion.

The calculation process of the weight of each modal feature is as follows:

Step (a61): Select a topic distribution θ|α～Dir(α), where α is the prior parameter of Dirichlet prior distribution;

Step (a62): For each word in the training sample video, select a top-level topic assignment The topic obeys a multinomial distribution;

Step (a63): For each modal feature weight ρ∈NV={NV modal feature dictionaries}, select an underlying topic assignment The topic obeys a multinomial distribution;

Step (a64): Under each modal feature weight ρ, based on the selected topic, combined with domain knowledge Ω, from the distribution generate a word.

For a single video d, given α, β and ρ, topic θ and top topic In joint mapping from NV unimodal space to high-level semantic space, topic θ and top-level topic The joint distribution probability p(θ,z ^top ,d|α,Ω,ρ,β)p(β) is:

Among them, the parameters θ and z ^top are hidden variables; the parameters θ and z ^top are eliminated by finding the marginal distribution.

where p(β ^η ) represents the prior relationship between dictionary elements in η modal space,

The Gauss-Markov random field prior model is used, namely:

Among them, Π ^j represents the set of words with a priori relation in n modal space, σ _i is the smoothing coefficient of the model, used to adjust the prior model; exp represents the exponential function with the natural constant e as the base;

For a video corpus D containing M videos, its generation probability is obtained by multiplying the edge probabilities of the M videos:

The objective function is set to the likelihood function of D, namely

When the likelihood function of D is maximized, the corresponding parameter ρ is the weight corresponding to each unimodal feature, log represents the logarithm with the base a, represents the likelihood function.

As a second aspect of the present disclosure, a video semantic representation system based on a multimodal fusion mechanism is provided;

A video semantic representation system based on a multi-modal fusion mechanism includes: a memory, a processor, and computer instructions stored in the memory and running on the processor, and when the computer instructions are run by the processor, complete any of the methods described above. A step of.

As a third aspect of the present disclosure, a computer-readable storage medium is provided;

A computer-readable storage medium having computer instructions stored thereon, when the computer instructions are executed by a processor, the steps described in any of the above methods are completed.

Compared with the prior art, the beneficial effects of the present disclosure are:

(1) This disclosure focuses on a video semantic representation method based on a multimodal fusion mechanism, which comprehensively utilizes algorithms in related fields such as image processing, pattern recognition, and machine learning to process sequence information in videos. It will provide new research perspectives and theoretical references for video representation analysis in different fields.

(2) Combining traditional methods and deep learning methods to study the effective representation of videos at the semantic level, effectively shortening the "multimodal gap" and "semantic gap" that are common in video understanding.

(3) Propose a deep visual feature learning model based on an adaptive learning mechanism. The adaptability of the automatic learning mechanism is mainly manifested in two aspects: First, the use of shot detection technology makes the input of the deep model a set of variable-length frame sequences , the number of frames can be adaptively adjusted according to the length of the shot; the second is in the C2 pooling layer, the size and step size of the pooling window are dynamically calculated according to the scale of the feature map, so as to ensure that the data representation dimensions of all shot videos are consistent.

(4) Design video lens adaptive 3D deep learning neural network, study the visual feature automatic learning algorithm of video features, improve the performance of the classifier and optimize the parameters of the whole system, and characterize the video in the most effective way. visual information.

(5) A multi-modal and multi-level topic representation model is proposed. The main features of this model are as follows: First, it is an extensible general representation model, and the number of single-modal information is extensible. The domain features contained in any type of video can also be incorporated into the model, which improves the pertinence of video representation capabilities; second, the model fully considers the relationship between various modalities, and integrates the multi-modal interaction process into the whole process. In the joint training and overall optimization of the model; the third is to use the unique advantages of the topic model in the field of semantic analysis, the video representation obtained by training has an ideal distinction in the semantic space, and can effectively obtain the concise representation of the video.

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application.

Figure 1. Video lens adaptive 3D deep learning architecture;

Figure 2 Adaptive 3D convolution process;

Figure 3 The process of using convolution kernel for convolution calculation;

Figure 4. The overall framework of the video multimodal fusion mechanism;

Figure 5. Multimodal and multi-level topic generation model.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

The present disclosure first proposes a spatiotemporal feature learning model of an adaptive frame selection mechanism to obtain visual features of a video. Then, on this basis, we further propose a model that can effectively integrate visual features and other modal features in combination with domain features to achieve semantic representation of videos.

In order to achieve the above purpose, the video representation model described in this disclosure combines traditional methods and deep learning methods, and comprehensively uses the advantages of traditional feature selection technology, deep learning mechanism and topic model theory to conduct research on the multi-modal fusion mechanism of video. Efficient representation of videos is further investigated at the semantic level.

The specific research technical plans are as follows:

Firstly, an in-depth analysis of the learning mechanism of video spatiotemporal information representation is carried out, and an effective representation of video visual information is obtained on the basis of ensuring the continuity and integrity of spatiotemporal information; then, the fusion mechanism of multimodal information is studied, and the The domain features of the video are integrated into the multi-modal information fusion process of the video, and finally a set of semantic representation models of the domain video are established.

(1) Automatic learning of spatiotemporal deep features of video features

Designing a video shot feature learning model with strong data fitting ability and learning ability can give full play to the advantages of layer-by-layer feature extraction. Using shot detection technology, the length of shot is used as an adaptive learning unit, and the spatiotemporal sequence information contained in the video shot is mined by layer-by-layer extraction. Therefore, a 3D deep learning network model for video lens adaptation is designed (see Figure 1).

The process is as follows:

Step 1: Use video shot detection technology to segment the video.

Step 2: Take a set of video frames of the shot as the input of the model, and then pass the information to different layers in turn, and each layer passes a set of filters to obtain the most salient feature information of the observation data in different categories.

Step 3: Finally all the pixel values of the shot frames are rasterized and concatenated into a vector.

The above adaptive 3D convolution process is embodied in the C1 convolution layer. Figure 2 shows the process of convolving a shot frame containing a length of L, that is, taking L frame sequences as input, through a set of learnable filters The device convolves the corresponding positions of different frames, then fuses and averages the obtained neurons, and finally outputs a set of feature maps through an activation function. For video frames, we consider the spatial connections within the frame to be local, so each neuron is set to perceive only local regions.

During the convolution process, the weights of neurons in the same plane layer are shared by weights. Figure 3 shows the process of nonlinear transformation using convolution kernels.

In Figure 3, W=(w ₁ ,w ₂ ,...,w _K ) represents the weight of the convolution kernel in the convolution layer, which is a set of learnable parameters; a=(a ₁ ,a ₂ ,...,a _L ) is the local receptive field of the corresponding position in L frames, where a _i =(a _i1 ,a _i2 ,...,a _iK ). When performing a convolution operation on an image with a certain convolution kernel, a certain area of the input image is involved in the convolution operation, and the size of this area is the receptive field.

In the S2 pooling layer, the pixel unit of the feature map obtained by the C1 convolution layer is firstly weighted and calculated, and then the non-linear function is run to pass the operation result to the next layer. For the pooling operation, overlapping pooling is used in the implementation process, that is, the set step size is smaller than the pooling window size.

In view of the differences in the size of video frames from different data sources, after the C1 convolution operation, the size of the obtained feature maps may be inconsistent, which will cause the feature dimension of each video shot in the fully connected layer to be different, which will lead to inconsistent data representation. The problem. The strategy adopted in this disclosure is to dynamically calculate the size and step size of the pooling window according to the scale of the feature map, thereby ensuring the dimensional consistency of the representation vectors for all video shots.

In the C3 convolution layer, the convolution kernels with twice the number of the C1 convolution layer are designed to act on the S2 layer, in order to detect more feature information as the spatial resolution decreases between layers. The S4 pooling layer adopts a similar operation to that of S2, and the purpose is to make the operation result continue to be passed to the next layer through the downsampling technique. In the C5 layer, the convolution kernel of the same size as the S4 layer is used to perform the convolution operation on the feature map obtained by the S4 layer, and finally a series of feature maps with a size of 1 × 1 are obtained. F6 is a fully connected graph. By connecting each neuron with all neurons in C5, the representation purpose of finally expressing the input as a feature vector of a certain length is achieved. Then, the features obtained by training are sent to the classifier to further improve the performance of the classifier to optimize the parameters of the whole system, and to represent the visual information of the video in the most effective way.

(2) Multimodal Information Fusion for Video Representation

In the preprocessing stage of multi-modal feature fusion of the video, it is necessary to extract and characterize the modal features of the video separately. Generally speaking, there are two categories of video features: one is general features, including:

1) Visual features with time series, including time dimension and space dimension information;

2) Text features, including the text in the video frame and the surrounding text of the video, using the bag-of-words model to convert the text information into a modelable digital description;

3) Motion features, first extract the optical flow information in the video, and then use the optical flow histogram HOF (Histogram of Oriented Optical Flow) to describe;

4) Audio features, first convert the audio information in the video into a spectrogram, and then use the spectrogram as an input to perform unsupervised learning by fine-tuning the existing network model to obtain a vector representation of the speech information.

The other category is domain features, which are related to video types and specific application domains.

The process of processing the modal features in the video is not a simple combination of various features, but the interaction and fusion of several different modal features. The present disclosure takes the popular latent Dirichlet distribution in the subject model as an entry point, and integrates subject theories such as machine learning, image processing, and speech recognition to fuse the above-mentioned video modal information. By constructing a multi-level topic model, the video data is organically mapped from each modal space and domain features to a high-level space, and the video-level representation sequence on the high-level space is obtained. Figure 4 presents the overall framework of the multimodal fusion mechanism.

For the above (2), the multi-modal information fusion of video representation needs to extract each modal feature of the video separately. Then a multi-level topic model is constructed for multi-feature fusion. The process is as follows:

(1) Extract each modal feature of the video separately, that is, extract the visual information, voice information, motion information, and text information of the video respectively.

For visual information, the unit of shot (a set of video frames) is used as the input of the model. By fine-tuning existing network models such as AlexNet, GoogleNet, unsupervised learning of visual information is performed. Finally, the pixel values of all shot frames are rasterized. For speech information, convert the audio data into a spectrogram, use the spectrogram information as the input of the model, and then perform unsupervised learning on the audio information by fine-tuning the existing network model, and through the full connection layer to obtain a vector; for motion information, it is proposed to first extract the optical flow information in the video, and then use the optical flow histogram HOF (Histogram of Oriented Optical Flow) to characterize; for text information, including the text in the video frame and the video surrounding Text, a bag-of-words model is proposed to convert textual information into a modelable digital description.

(2) Build a multi-level topic model to achieve multi-feature fusion

By constructing a multi-level topic model (Fig. 5), the video data is organically mapped from each modal feature space and domain features to a high-level semantic space to achieve multi-modal fusion. The specific implementation process is as follows:

The model assumes that the corpus contains M videos, denoted as D={d ₁ ,d ₂ ,...,d _M }, and each video d _i (1≤i≤M) contains a set of potential topic information, we consider these Topic information is generated by the organic mapping of dictionary elements in each modal space to a high-level semantic space according to a certain distribution under certain prior conditions. The model uses video as the processing unit, involving a total of two levels of topic models, realizing multi-modal information fusion with domain features as prior knowledge, and finally obtaining topic representations in the form of vectors. The model contains two levels of topics, represented by Z ^top and Z ^low respectively, Z ^top represents the topic after video fusion, Z ^low represents the topic before fusion, the former is composed of the latter according to the polynomial distribution with ρ as the parameter, The parameter Ω corresponds to the domain feature of the video. The model considers that words in different modal spaces are independent and identically distributed. The construction of the graph feature dictionary adopts the K-means clustering technology and constructs it in the way of a bag of words model.

The parameter θ in the model obeys the Dirichlet distribution with α as the prior parameter, which represents the distribution contained in the currently processed video. The parameter NV is the number of multimodalities, and β represents dictionaries in different modal spaces. The model realizes the problem of weight setting of different modalities when converting from multimodal space to semantic space by solving the parameter ρ.

The generation process of each video corpus is shown in the following table:

The above generation process:

Step 1: Select a topic distribution θ|α～Dir(α), where α is the prior parameter of Dirichlet prior distribution;

Step 2: For each word in the video, select a top-level topic assignment The topic obeys a multinomial distribution;

Step 3: For NV modal spaces, under modal space p, select an underlying topic assignment The topic obeys a multinomial distribution;

Step 4: According to the selected topic, from the distribution generate a word.

For a single video d, given parameters α, β, ρ, combine domain knowledge Ω, topic θ in the model, top topic When jointly mapping from the NV modal space to the high-level space, their joint distribution probability is:

The parameters θ and z ^top are hidden variables. It can be eliminated by finding the marginal distribution.

The above p(β ^η ) represents the prior relationship between dictionary elements in η modal space,

A typical Gauss-Markov random field prior model is used, namely:

In the above formula, ^Πj represents the set of words with a priori relationship in the _n -modal space, and σi is the smoothing coefficient of the model, which is used to adjust the priori model.

For a video corpus D containing M videos, its likelihood value can be obtained by multiplying the edge probabilities of the M videos:

It is expected that finding suitable parameters α, ρ, β can maximize the following likelihood function of the corpus, that is, the objective function is expressed as:

By solving the above model, the organic fusion of multimodal features and video domain features can be achieved, and finally the semantic representation of the video can be obtained.

The general idea of the above process is shown in Figure 4. Summarizing the above process, the multi-modal and multi-level topic model proposed by the present disclosure has the following characteristics compared with the existing methods:

1) It is an extensible general representation model. In the process of model training and overall optimization, it is not only extensible for the number of single-modal information, but also can incorporate the domain features contained in any type of video into the model. , which improves the pertinence of video representation capabilities;

2) The model fully considers the relationship between the modalities, and integrates the multi-modal interaction process into the joint training and overall optimization process of the entire model;

3) The topic model itself has its unique advantages in the field of semantic analysis. On this basis, the video representation obtained by the model training has an ideal distinction in the semantic space, which is also one of the effective ways to obtain concise representation of video. one.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

Claims (9)

1. The video semantic representation method based on the multi-mode fusion mechanism is characterized by comprising the following steps:

feature extraction: extracting visual features, voice features, motion features, text features and field features of the video;

feature fusion: performing feature fusion on the extracted visual, voice, motion and text features and the field features through a constructed multilayer hidden Dirichlet distributed topic model;

characteristic mapping: and mapping the fused features to a high-level semantic space to obtain a fused feature representation sequence.

2. The method for video semantic representation based on the multi-modal fusion mechanism as claimed in claim 1, wherein the specific steps for extracting the visual features of the video are as follows:

pretreatment: video segmentation, namely segmenting a video into a plurality of shots; the image frames in each lens form an image frame sequence according to the time sequence;

step (a 1): establishing a deep learning neural network model;

the deep learning neural network model comprises: the input layer, the first scrolling layer C1, the first pooling layer S2, the second scrolling layer C3, the second pooling layer S4, the third scrolling layer C5, the full-connection layer F6 and the output layer are connected in sequence;

step (a 2): inputting the image frame sequence of each lens of the video into an input layer of the deep learning neural network model, and transmitting the image frames to a first convolution layer C1 by the input layer;

the first convolution layer C1 is used for performing convolution on each frame image in the image frame sequence of the video by using a group of trainable convolution cores, averaging all layers of feature maps obtained by the convolution to obtain an average feature map, and outputting a group of feature mapping maps by using the obtained average feature map and a bias input activation function;

the first pooling layer S2 is used for performing overlapping pooling operation on the pixel value of each pixel point of the feature map obtained by the first convolution layer C1, so that the length and the width of the first convolution layer output feature map matrix are reduced; then the operation result is transmitted to a second convolution layer C3;

a second convolution layer C3 for performing convolution operation on the operation result of the first pooling layer S2; the number of convolution kernels of the second convolution layer C3 is twice the number of convolution kernels of the first convolution layer C1;

a second pooling layer S4 for performing an overlap pooling operation on the feature map output by the second convolutional layer C3 to reduce the size of the feature map matrix;

a third convolution layer C5, which performs convolution operation on the result of the second pooling layer S4 by using a convolution kernel with the same size as the second pooling layer S4 to finally obtain a plurality of characteristic graphs of 1 × 1 pixels;

a full-connection layer F6 for fully connecting each neuron of the layer with each neuron in the third convolutional layer C5, and expressing the result obtained by the third convolutional layer C5 as a feature vector;

the output layer is used for inputting the feature vectors output by the full connection layer F6 into a classifier for classification, calculating the classification accuracy, and when the classification accuracy is lower than a set threshold, adjusting parameters through back propagation to repeatedly execute the step (a2) until the classification accuracy is higher than the set threshold; and when the classification accuracy is higher than the set threshold, the feature vector corresponding to the classification accuracy higher than the set threshold is used as the final learning result of the video visual features.

3. The method for video semantic representation based on the multi-modal fusion mechanism as claimed in claim 1, wherein the specific steps for extracting the voice features of the video are as follows:

extracting voice signals in the video, converting audio data into a spectrogram, inputting the spectrogram serving as a deep learning neural network model, performing unsupervised learning on audio information through the deep learning neural network model, and obtaining vector representation of video voice characteristics through a full connection layer.

4. The method for semantic representation of video based on multi-modal fusion mechanism as claimed in claim 1, wherein the specific steps for extracting the motion features of the video are as follows:

and extracting an optical flow field in the video, and carrying out weighted statistics on the optical flow direction to obtain optical flow direction information histogram features as vector representation of the motion features.

5. The method for video semantic representation based on the multi-modal fusion mechanism as claimed in claim 1, wherein the specific steps for extracting the text features of the video are as follows:

the method comprises the steps of collecting characters in a video frame and peripheral text information of a video, and extracting text features from the text information by adopting a word bag model.

6. The method as claimed in claim 1, wherein the domain feature is a rule feature set by a domain to which the video belongs.

7. The method for video semantic representation based on multi-modal fusion mechanism as claimed in claim 1, wherein,

the specific steps of the multi-modal feature fusion are as follows:

step (a 1): mapping visual feature vectors of the video from a visual feature space to a semantic feature space gamma by using an LDA hidden Dirichlet distributed topic model; the input of the LDA hidden Dirichlet distribution topic model is a visual feature vector of the video, and the output of the LDA hidden Dirichlet distribution topic model is a semantic representation on a feature space gamma;

step (a 2): mapping the voice feature vector of the video from a voice feature space to a semantic feature space gamma by using an LDA hidden Dirichlet distributed topic model; the input of an LDA hidden Dirichlet distributed topic model is a voice feature vector of a video, and the output is a semantic representation on a feature space gamma;

step (a 3): mapping the optical flow direction information histogram feature of the video from a motion feature space to a semantic feature space gamma by using an LDA hidden Dirichlet distribution topic model; the input of the LDA is the optical flow direction information histogram feature of the video, and the output is the semantic representation on the feature space gamma;

step (a 4): the LDA implies a Dirichlet distribution topic model, and the text features of the video are mapped to a semantic feature space gamma from a text feature space; the input of the LDA is a text of a video, and the output is a semantic representation on a feature space gamma;

step (a 5): converting the video domain features into a priori knowledge omega;

step (a 6): and (3) by using an LDA hidden Dirichlet distributed topic model, setting the weight of each modal feature on the semantic feature space gamma of each modal feature obtained in the steps (a1) and (a4), and obtaining the video representation after modal fusion through weighted fusion.

8. The video semantic representation system based on the multi-mode fusion mechanism is characterized by comprising the following steps: a memory, a processor, and computer instructions stored on the memory and executed on the processor, the computer instructions, when executed by the processor, performing the steps of the method of any of claims 1-7.

9. A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, perform the steps of the method of any one of claims 1 to 7.