CN106934352A - A kind of video presentation method based on two-way fractal net work and LSTM - Google Patents

Abstract

The invention discloses a video description method based on two-way fractal network and LSTM. The method first samples the key frames of the video to be described, and extracts the optical flow features between two adjacent frames of the original video, then learns and obtains the high-level feature expressions of the video frames and optical flow features through two fractal networks, and then separately It is input to two recursive neural network models based on LSTM units, and finally the output values of the two independent models at each moment are weighted and averaged to obtain a description sentence corresponding to the video. The present invention uses the information of the original video frame and optical flow for the video to be described, and the added optical flow feature compensates the dynamic information that will inevitably be lost in the sampling frame, and takes into account the change of the video in the space dimension and time dimension. Furthermore, the abstract visual feature representation of the underlying features is performed through a novel fractal network, so as to more accurately analyze and mine the connections between people, objects, behaviors, and spatial location relationships involved in the video.

Description

A video description method based on two-way fractal network and LSTM

technical field

The invention belongs to the technical field of video description and deep learning, and in particular relates to a video description method based on a two-way fractal network and LSTM.

Background technique

With the advancement of technology and the development of society, all kinds of video camera terminals, especially smart phones, have become very popular, and the price of hardware storage has become increasingly low, which makes the flow of multimedia information grow exponentially. In the face of a large number of video information streams, how to efficiently and automatically analyze, recognize and understand massive video information with minimal human intervention, and thus describe it semantically, has become a hot topic in the current image processing and computer vision research fields. topic. For most people, it may be a simple matter to watch a short video and then describe the video in words. However, for a machine, it is a challenging task to generate a natural language description by extracting the pixel information of each frame image in the video, analyzing and processing it.

Enabling machines to efficiently and automatically describe videos also has broad application prospects in computer vision fields such as video retrieval, human-computer interaction, and traffic security, which will further promote people's research on semantic descriptions of videos.

Contents of the invention

The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a video description method based on two-way fractal network and LSTM.

In order to achieve the above object, the present invention adopts the following technical solutions:

A video description method based on two-way fractal network and LSTM, which is characterized in that firstly, the key frame sampling of the video to be described is carried out, and the optical flow characteristics between two adjacent frames of the original video are extracted, and then the two fractal networks are respectively Learn and obtain the high-level feature expression of the key frame and optical flow features, and then input them to two recurrent neural network models based on LSTM units, and finally perform a weighted average of the output values of the two independent recurrent neural network models at each moment, so as to obtain The description sentence corresponding to the video. Specifically include the following steps:

S1. Sampling the key frames of the video to be described, and extracting the optical flow features between two adjacent frames of the original video;

S2. Learning and obtaining high-level feature expressions of video frames and optical flow features through two fractal networks;

S3. Input the high-level feature vectors obtained in the previous step to two recurrent neural networks based on LSTM units;

S4. Perform a weighted average of the output values of the two independent models at each moment to obtain a description sentence corresponding to the video.

Preferably, the optical flow feature extraction of the video to be described in step S1 is specifically:

S1.1. Calculate the optical flow feature values in the x-direction and y-direction of each adjacent two frames of the video, and normalize to the pixel range of [0,255];

S1.2. Calculate the amplitude value of the optical flow, and combine the characteristic values of the optical flow obtained in the previous step to form an optical flow graph.

Preferably, the specific steps for obtaining high-level feature representations of key frames and optical flow features in step S2 are:

S2.1, the key frames of the video obtained in step S1 are sequentially input to the first fractal network that processes the spatial dimension relationship in the order of time points, and the corresponding visual feature vectors are sequentially generated through the nonlinear mapping relationship of the network;

S2.2. The optical flow diagram obtained in step S1 is sequentially input into the second fractal network for processing the time-dimensional relationship in the order of time points, and the corresponding motion feature vectors are sequentially generated through the nonlinear mapping relationship of the network.

Preferably, the repeated application by a single extension rule in steps S2.1 and S2.2 generates an extremely deep network whose structural layout is a truncated fractal; the network contains interacting subpaths of different lengths, but Does not contain any straight-through connections; at the same time, in order to achieve the ability to extract high-performance fixed-depth sub-networks, a method of path discarding is used to regularize the rules of co-adaptation of sub-paths in the fractal architecture; for fractal networks, the simplicity of training is comparable to that of Corresponding to the simplicity of the design, a single loss function connected to the last layer is sufficient to drive the internal behavior to mimic deep supervision; the adopted fractal network is a deep convolutional neural network based on fractal structure.

Preferably, repeated application of a single extension rule in steps S2.1 and S2.2 generates an extremely deep network, and its structural layout is a truncated fractal specifically:

The base case f ₁ (z) contains a single layer of selected type between the input and output; let _C denote the index of the truncated fractal f _C (·), which defines the network architecture, connections, and layer types. Among them, the basic situation is that the network representation containing a single convolutional layer is represented by formula (1-1):

f ₁ (z)=conv(z) (1-1)

Recursively define the next fractal as formula (1-2):

In formula (1-2), means composite, and Represents the connection operation, C corresponds to the number of columns, or the width of the network f _C (·); depth is defined as the number of conv layers on the longest path from input to output, proportional to 2 ^C-1 ; used for classification The convolutional network of usually dispersely arranges the pooling layer; in order to achieve the same purpose, use f _C (·) as the building unit, and stack it with the next pooling layer B times to obtain the total depth B 2 ^C-1 ; the connection operation Merge two feature blocks into one; a feature block is the result of a conv layer: maintain active tensors for a fixed number of channels in a spatial region; the number of channels corresponds to the number of filters in the previous conv layer; when divided The shape is expanded to combine adjacent connections into a single connection layer; a connection layer combines all its input feature blocks into a single output block.

Preferably, a method of path discarding in steps S2.1 and S2.2 regularizes the co-adaptation rules of sub-paths in the fractal architecture as follows: since the fractal network contains additional large-scale structures, use a method similar to dropout and drop- Connect's coarse-grained regularization strategy, path discarding prohibits the co-adaptation of parallel paths by randomly discarding the operands of the connection layer. This method effectively prevents the network from using one path as an anchor and another path as a correction. Fitting behavior; two sampling strategies are used:

For local, the connection layer discards each input with a fixed probability, but guarantees to keep at least one input;

Globally, each path is selected for the entire network, by restricting this path to a single column, incentivizing each column to be a strong predictor.

Preferably, the input of high-level feature vectors to two LSTM unit-based recursive neural network models described in step S3 is specifically:

The recurrent neural network based on the LSTM unit contains two layers of LSTM units, the first layer and the second layer contain 1000 neurons respectively, and the forward propagation process of each LSTM neuron unit can be expressed as:

i _t = σ(W _xi x _t +W _hi h _t-1 +b _i ) (1-3)

f _t = σ(W _xf x _t +W _hf h _t-1 +b _f ) (1-4)

o _t = σ(W _xo x _t +W _ho h _t-1 +b _o ) (1-5)

c _t =f _t *c _t-1 +i _t *g _t (1-7)

Among them, σ(x _t )=(1+e ^-xt ) ^-1 is the sigmoid nonlinear activation function, is the hyperbolic tangent nonlinear activation function; it , f _t , o _t , c _t respectively represent the state quantities corresponding to the input gate, memory gate, output gate and core gate at time _t ; for each logic gate, W _xi , W _xf , W _xo , W _xg represent the weight transfer matrix corresponding to the input gate, memory gate, output gate and core gate respectively, W _hi , W _hf , _{Who ho} , W _hg represent the input gate, memory gate, output gate and core gate respectively The weight transfer matrix corresponding to the hidden layer variable h _t -1 at time t-1, b _i , b _f , b _o , b _g represent the bias vectors corresponding to the input gate, memory gate, output gate and core gate, respectively.

Preferably, the structure of the neural network model in step S3 is:

Based on the recurrent neural network structure diagram of the two-layer LSTM unit, the recurrent neural network of the two-layer stacked LSTM unit is used to encode and decode the input feature vector, thereby realizing the conversion of natural language text; among them, the first layer of LSTM The neuron completes the encoding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the input of the second layer of LSTM neurons; when the feature vectors of all video frames are input to the first layer of LSTM neurons After the unit, the LSTM neuron in the second layer will receive an indicator and start the task of decoding; in the decoding stage, the network will lose information, so the goal of model parameter training and learning is to express in a given hidden layer Under the premise of the output prediction at the previous moment, maximize the logarithmic likelihood function of the entire output sentence prediction; for the model represented by the parameter θ and the output sentence Y=(y ₁ ,y ₂ ,…,y _m ), the parameter The optimization objective can be expressed as:

Here, θ is a parameter, Y represents the output prediction sentence, h is the expression of the hidden layer, and the stochastic gradient descent method is used to optimize the objective function, and the error of the entire network is accumulated and transferred in the time dimension through the back propagation algorithm.

Preferably, in step S4, the output values of the two independent neural network models at each moment are weighted and averaged to obtain the description sentence corresponding to the video. The specific operation is as follows:

S4.1, weighted average the output values of the second layer LSTM neurons at each moment of the two independent recurrent neural network models;

S4.2, using the softmax function to calculate the probability of occurrence of each word in the vocabulary V, expressed as:

Among them, y represents the predicted word, z _t represents the output value of the recurrent neural network at time t, and W _y represents the weight value of the word in the vocabulary.

S4.3. In the decoding stage at each moment, take the word with the highest probability in the output value of the softmax function, so as to obtain the corresponding video description sentence.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. The optical flow feature added by the present invention compensates the dynamic information that will inevitably be lost in the sampling frame, and takes into account the changes in the spatial and temporal dimensions of the video.

2. A video description method based on a two-way fractal network and LSTM provided by the present invention can automatically generate a sentence of descriptive language about video content end-to-end by processing any input video, which can be applied to video retrieval, Applications such as video surveillance and human-computer interaction.

3. The present invention expresses the abstract visual features of the underlying features through a novel fractal network, thereby more accurately analyzing and mining the connections of people, objects, behaviors, and spatial position relationships involved in the video.

Description of drawings

Fig. 1 is the flow chart of the video description method based on two-way fractal network and LSTM provided by the present invention;

Fig. 2 is the schematic diagram of the fractal sub-network adopted by the embodiment of the present invention;

FIG. 3 is a schematic diagram of a recurrent neural network based on LSTM units used in an embodiment of the present invention.

detailed description

The present invention will be further described in detail below in conjunction with the embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

Sampling the key frames of the video to be described, and extracting the optical flow features between two adjacent frames of the original video, and then learning and obtaining the high-level feature expressions of the key frames and optical flow features through two fractal networks, and then input them into two Based on the recurrent neural network model of the LSTM unit, the output values of the two independent recurrent neural network models at each moment are weighted and averaged to obtain the description sentence corresponding to the video.

Fig. 1 is the overall flowchart of the present invention, comprises the following steps:

(1) Sampling the key frame of the video to be described, and extracting the optical flow features between two adjacent frames of the original video; wherein, the specific operation of extracting the optical flow features of the video to be described is:

1. Calculate the optical flow values in the x-direction and y-direction of each adjacent two frames of the video, and normalize to the pixel range of [0,255];

2. Calculate the amplitude value of the optical flow, and combine the optical flow eigenvalues obtained in the previous step to form an optical flow graph.

(2) Learning and obtaining high-level feature representations of video frames and optical flow features through two fractal networks respectively. The sample frames of the video obtained in the first step are sequentially input to the first fractal network that processes the spatial dimension relationship in the order of time points, and the corresponding visual feature vectors are sequentially generated through the nonlinear mapping relationship of the network; for the obtained optical flow graph The order of time points is sequentially input to the second fractal network that deals with the time dimension relationship, and the corresponding motion feature vectors are sequentially generated through the nonlinear mapping relationship of the network.

The fractal network mainly introduces a design strategy based on self-similarity on the macro-architecture of the neural network. An extremely deep network is generated through the repeated application of a single expansion rule, and its structural layout is a truncated fractal. The network contains interacting subpaths of varying lengths, but does not contain any through-connections. At the same time, in order to achieve the ability to extract high-performance fixed-depth sub-networks, a method of path discarding is used to regularize the co-adaptation of sub-paths in the fractal architecture. For fractal networks, the simplicity of training corresponds to the simplicity of design, a single loss function connected to the last layer is sufficient to drive the internal behavior to mimic deep supervision. The fractal network used in the present invention is a deep convolutional neural network based on a fractal structure.

A schematic diagram of the fractal structure is given in Fig. 2. The base case f ₁ (z) contains a single layer of selected type between the input and the output; let C denote the index of the truncated fractal f _C ( ), f _C ( ) defines Network architecture, connections, and layer types are covered. Among them, the basic situation is that the network representation containing a single convolutional layer is represented by formula (1-1):

f ₁ (z)=conv(z) (1-1)

Then recursively define the next fractal structure as formula (1-2):

In formula (1-2), means composite, and Represents the connection operation, C corresponds to the number of columns, or the width of the network f _C (·); depth is defined as the number of conv layers on the longest path from input to output, proportional to 2 ^C-1 ; used for classification The convolutional network of usually dispersely arranges the pooling layer; in order to achieve the same purpose, use f _C (·) as the building unit, and stack it with the next pooling layer B times to obtain the total depth B 2 ^C-1 ; the connection operation Combines two feature blocks into one, where one feature block is the result of a convolutional layer: a tensor of activations maintained for a fixed number of channels in a spatial region. The number of channels corresponds to the number of filters in the previous convolutional layer. When the fractal is expanded, adjacent connections are merged into a single connected layer. As shown on the right side of Figure 2, this concatenated layer spans multiple columns, combining all its input feature blocks into a single output block.

Since fractal networks contain additional large-scale structures, a coarse-grained regularization strategy similar to dropout and drop-connect is proposed. Path discarding prohibits the co-adaptation of parallel paths by randomly discarding the operands of the connection layer, which effectively prevents the network from using one path as an anchor and the other path as a correction that may cause overfitting behavior. There are mainly two sampling strategies used here:

For local, the connection layer discards each input with a fixed probability, but guarantees to keep at least one input;

Globally, each path is selected for the entire network, by restricting this path to a single column, incentivizing each column to be a strong predictor.

(3) Input the high-level feature vectors obtained in the previous step into two recurrent neural networks based on LSTM units. The recurrent neural network based on the LSTM unit contains two layers of LSTM units, the first layer and the second layer contain 1000 neurons respectively, and the forward propagation process of each LSTM neuron unit can be expressed as:

i _t = σ(W _xi x _t +W _hi h _t-1 +b _i ) (1-3)

f _t = σ(W _xf x _t +W _hf h _t-1 +b _f ) (1-4)

o _t = σ(W _xo x _t +W _ho h _t-1 +b _o ) (1-5)

c _t =f _t *c _t-1 +i _t *g _t (1-7)

Among them, σ(x _t )=(1+e ^-xt ) ^-1 is the sigmoid nonlinear activation function, is the hyperbolic tangent nonlinear activation function; it , f _t , o _t , c _t respectively represent the state quantities corresponding to the input gate, memory gate, output gate and core gate at time _t ; for each logic gate, W _xi , W _xf , W _xo , W _xg represent the weight transfer matrix corresponding to the input gate, memory gate, output gate and core gate respectively, W _hi , W _hf , _{Who ho} , W _hg represent the input gate, memory gate, output gate and core gate respectively The weight transfer matrix corresponding to the hidden layer variable h _t -1 at time t-1, b _i , b _f , b _o , b _g represent the bias vectors corresponding to the input gate, memory gate, output gate and core gate, respectively.

The recurrent neural network structure diagram based on the two-layer LSTM unit is given in Figure 3. We use the recurrent neural network of the two-layer stacked LSTM unit to encode and decode the input feature vector, thereby realizing the natural language text. convert. Among them, the first layer of LSTM neurons completes the encoding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the input of the second layer of LSTM neurons; when the feature vectors of all video frames are input After reaching the first layer of LSTM neurons, the second layer of LSTM neurons will receive an indicator and start the task of decoding. In the decoding stage, the network will lose information, so the goal of model parameter training and learning is to maximize the logarithmic likelihood function of the entire output sentence prediction given the hidden layer expression and the output prediction at the previous moment . For a model represented by parameter θ and output sentence Y=(y ₁ ,y ₂ ,…,y _m ), the parameter optimization objective can be expressed as:

Here, θ is a parameter, Y represents the output prediction sentence, h is the expression of the hidden layer, and the stochastic gradient descent method is used to optimize the objective function, and the error of the entire network is accumulated and transferred in the time dimension through the back propagation algorithm.

(4) Weighted average the output values of the two independent models at each moment and obtain the description sentence corresponding to the video. The specific operation is as follows:

1. Weighted average the output values of the second layer LSTM neurons at each moment of the two independent models;

2. Use the softmax function to calculate the occurrence probability of each word in the vocabulary V, expressed as:

Among them, y represents the predicted word, z _t represents the output value of the recurrent neural network at time t, and W _y represents the weight value of the word in the vocabulary.

3. In the decoding stage at each moment, take the word with the highest probability in the output value of the softmax function, so as to obtain the corresponding video description sentence.

The above-mentioned embodiment is a preferred embodiment of the present invention, but the embodiment of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, Simplifications should be equivalent replacement methods, and all are included in the protection scope of the present invention.

Claims (9)

1. A video description method based on two-way fractal network and LSTM, characterized in that, at first the video to be described is subjected to key frame sampling, and the optical flow feature between two adjacent frames of the original video is extracted, and then the two fractal The network learns and obtains high-level feature expressions of key frames and optical flow features, and then inputs them to two recurrent neural network models based on LSTM units, and finally weights and averages the output values of the two independent recurrent neural network models at each moment, so that Obtaining a descriptive sentence corresponding to the video; specifically including the following steps: S1. Sampling the key frames of the video to be described, and extracting the optical flow features between two adjacent frames of the original video; S2. Learning and obtaining high-level feature expressions of key frames and optical flow features through two fractal networks; wherein the fractal network is generated by repeated application of a single extension rule; S3. Input the high-level feature vectors obtained in the previous step into two recurrent neural network models based on LSTM units; S4. Perform a weighted average of the output values of the two independent recursive neural network models at each moment to obtain a description sentence corresponding to the video. 2. A kind of video description method based on two-way fractal network and LSTM according to claim 1, it is characterized in that, described in step S1, the optical flow feature of video to be described is specifically: S1.1. Calculate the optical flow feature values in the x-direction and y-direction of each adjacent two frames of the video, and normalize to the pixel range of [0,255]; S1.2. Calculate the amplitude value of the optical flow, and combine the characteristic values of the optical flow obtained in the previous step to form an optical flow graph. 3. a kind of video description method based on two-way fractal network and LSTM according to claim 1, it is characterized in that, in the step S2, the concrete steps of obtaining the high-level feature expression of key frame and optical flow feature are: S2.1, the key frames of the video obtained in step S1 are sequentially input to the first fractal network that processes the spatial dimension relationship in the order of time points, and the corresponding visual feature vectors are sequentially generated through the nonlinear mapping relationship of the network; S2.2. The optical flow diagram obtained in step S1 is sequentially input into the second fractal network for processing the time-dimensional relationship in the order of time points, and the corresponding motion feature vectors are sequentially generated through the nonlinear mapping relationship of the network. 4. A kind of video description method based on two-way fractal network and LSTM according to claim 3, it is characterized in that, in steps S2.1 and S2.2, the repeated application of a single expansion rule generates an extremely deep network , its structural layout is a truncated fractal; the network contains interacting sub-paths of different lengths, but does not contain any through-connection; meanwhile, in order to achieve the ability to extract high-performance fixed-depth sub-networks, a path discarding method is adopted The method regularizes the rules for the co-adaptation of subpaths in fractal architectures; for fractal networks, the simplicity of training corresponds to the simplicity of design, and a single loss function connected to the last layer is sufficient to drive the internal behavior to mimic deep supervision; the adopted Fractal networks are deep convolutional neural networks based on fractal structures. 5. A kind of video description method based on two-way fractal network and LSTM according to claim 4, it is characterized in that, in steps S2.1 and S2.2, a very deep network is generated by repeated application of a single extension rule , its structural layout is a truncated fractal, specifically: The base case f ₁ (z) contains a single layer of selected type between the input and output; let C denote the index of the truncated fractal f _C (·), f _C (·) defines the network architecture, connection and layer type; where, the base The situation is that a network containing a single convolutional layer is expressed as formula (1-1): f ₁ (z)=conv(z) (1-1) Recursively define the next fractal as formula (1-2): In formula (1-2), means composite, and Represents the connection operation, C corresponds to the number of columns, or the width of the network f _C (·); depth is defined as the number of ^conv layers on the longest path from input to output, proportional to 2 ^C-1 ; used for classification The convolutional network of usually dispersely arranges the pooling layer; in order to achieve the same purpose, use f _C (·) as the building unit, and stack it with the next pooling layer B times to obtain the total depth B 2 ^C-1 ; the connection operation Merge two feature blocks into one; a feature block is the result of a conv layer: maintain active tensors for a fixed number of channels in a spatial region; the number of channels corresponds to the number of filters in the previous conv layer; when divided The shape is expanded to combine adjacent connections into a single connection layer; a connection layer combines all its input feature blocks into a single output block. 6. A kind of video description method based on two-way fractal network and LSTM according to claim 4, it is characterized in that, in steps S2.1 and S2.2, a kind of method that path discards is the co-adaptation of sub-path in regularization fractal architecture The specific rules are: Since the fractal network contains additional large-scale structures, a coarse-grained regularization strategy similar to dropout and drop-connect is used, and path discarding prohibits the co-adaptation of parallel paths by randomly discarding the operands of the connection layer. This method effectively prevents the network from using one path as an anchor and another path as a correction that may cause over-fitting behavior; two sampling strategies are used: For local, the connection layer discards each input with a fixed probability, but guarantees to keep at least one input; Globally, each path is selected for the entire network, by restricting this path to a single column, incentivizing each column to be a strong predictor. 7. A kind of video description method based on two-way fractal network and LSTM according to claim 1, it is characterized in that, described in step S3, high-level feature vector is input to two recursive neural network models based on LSTM unit specifically: The recurrent neural network based on the LSTM unit contains two layers of LSTM units, the first layer and the second layer contain 1000 neurons respectively, and the forward propagation process of each LSTM neuron unit can be expressed as: i _t = σ(W _xi x _t +W _hi h _t-1 +b _i ) (1-3) f _t = σ(W _xf x _t +W _hf h _t-1 +b _f ) (1-4) o _t = σ(W _xo x _t +W _ho h _t-1 +b _o ) (1-5) c _t =f _t *c _t-1 +i _t *g _t (1-7) in, is the sigmoid nonlinear activation function, is the hyperbolic tangent nonlinear activation function; it , f _t , o _t , c _t respectively represent the state quantities corresponding to the input gate, memory gate, output gate and core gate at time _t ; for each logic gate, W _xi , W _xf , W _xo , W _xg represent the weight transfer matrix corresponding to the input gate, memory gate, output gate and core gate respectively, W _hi , W _hf , _{Who ho} , W _hg represent the input gate, memory gate, output gate and core gate respectively The weight transfer matrix corresponding to the hidden layer variable h _t -1 at time t-1, b _i , b _f , b _o , b _g represent the bias vectors corresponding to the input gate, memory gate, output gate and core gate, respectively. 8. A kind of video description method based on two-way fractal network and LSTM according to claim 7, it is characterized in that, in the step S3, neural network model structure is: A recurrent neural network based on two layers of LSTM units realizes the conversion of natural language text; among them, the first layer of LSTM neurons completes the encoding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the first The input of the two-layer LSTM neuron; when the feature vectors of all video frames are input to the first-layer LSTM neuron, the second-layer LSTM neuron will receive an indicator and start the task of decoding; in the decoding stage , the network will lose information, so the goal of model parameter training and learning is to maximize the log-likelihood function of the entire output sentence prediction under the premise of the given hidden layer expression and the output prediction at the previous moment; For the model represented by θ and the output sentence Y=(y ₁ ,y ₂ ,…,y _m ), the parameter optimization objective can be expressed as: ${\mathrm{<span\; class="notranslate">\; \&theta;</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}\underset{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$ Here, θ is a parameter, Y represents the output prediction sentence, h is the expression of the hidden layer, and the stochastic gradient descent method is used to optimize the objective function, and the error of the entire network is accumulated and transferred in the time dimension through the back propagation algorithm. 9. A kind of video description method based on two-way fractal network and LSTM according to claim 1, it is characterized in that, step S4 carries out weighted average with the output value of two neural network independent models at each moment and obtains the descriptive sentence corresponding to video The operation is: S4.1, weighted average the output values of the second layer LSTM neurons at each moment of the two independent recurrent neural network models; S4.2, using the softmax function to calculate the probability of occurrence of each word in the vocabulary V, expressed as: $\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; V</span>}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$ Among them, y represents the predicted word, z _t represents the output value of the recurrent neural network at time t, and W _y represents the weight value of the word in the vocabulary; S4.3. In the decoding stage at each moment, take the word with the highest probability in the output value of the softmax function, so as to obtain the corresponding video description sentence.