CN106934352A - A kind of video presentation method based on two-way fractal net work and LSTM - Google Patents

Abstract

The invention discloses a kind of video presentation method based on two-way fractal net work and LSTM.Methods described carries out the sampling of key frame to video to be described, first, and extract the Optical-flow Feature between adjacent two frame of former video, then learnt respectively by two fractal net works and obtain frame of video and the expression of the high-level characteristic of Optical-flow Feature, two recurrent neural networks models based on LSTM units are separately input to again, finally the output valve at two each moment of independent model is weighted averagely, so as to obtain descriptive statement corresponding with the video.The present invention has been utilized respectively the information of former frame of video and light stream to video to be described, the Optical-flow Feature of addition compensate for the multidate information that sample frame is inevitably lost, it is contemplated that change of the video on Spatial Dimension and time dimension.Furthermore, abstract visual signature is carried out to low-level image feature by novel fractal net work and is expressed, so that people, thing, behavior and spatial relation for being more accurately related in analysis mining video etc. are contacted.

Description

Video description method based on two-way fractal network and LSTM

Technical Field

The invention belongs to the technical field of video description and deep learning, and particularly relates to a video description method based on a two-way fractal network and an LSTM.

Background

With the progress of science and technology and the development of society, various video camera terminals, especially smart phones, are very popular, and the price of hardware storage is increasingly low, so that the multimedia information flow is exponentially increased. In the presence of a large amount of video information streams, how to perform efficient and automatic analysis, recognition and understanding on massive video information with the human intervention minimized, so as to describe semantically, has become a popular topic in the current image processing and computer vision research fields. It may be a simple matter for most people to describe a video in language after watching a brief video. However, it is a challenging task for a machine to generate a natural language description by extracting pixel information of each frame of image in a video, analyzing and processing the pixel information.

The machine can efficiently and automatically describe the video, and has wide application prospects in the field of computer vision such as video retrieval, man-machine interaction, traffic security and the like, so that the research of people on semantic description of the video is further promoted.

Disclosure of Invention

The invention mainly aims to overcome the defects of the prior art and provide a video description method based on a two-way fractal network and an LSTM.

In order to achieve the purpose, the invention adopts the following technical scheme:

a video description method based on two-way fractal network and LSTM is characterized in that firstly, sampling key frames of a video to be described is carried out, optical flow characteristics between two adjacent frames of an original video are extracted, then, high-level characteristic expressions of the key frames and the optical flow characteristics are obtained through learning of the two fractal networks respectively, then, the high-level characteristic expressions are input into two recurrent neural network models based on an LSTM unit respectively, and finally, weighted average is carried out on output values of the two independent recurrent neural network models at each moment, so that description sentences corresponding to the video are obtained. The method specifically comprises the following steps:

s1, sampling key frames of a video to be described, and extracting optical flow characteristics between two adjacent frames of an original video;

s2, learning and obtaining high-level feature expressions of video frames and optical flow features through two fractal networks respectively;

s3, respectively inputting the high-level feature vectors obtained in the previous step into two recurrent neural networks based on an LSTM unit;

and S4, carrying out weighted average on the output values of the two independent models at each moment and obtaining the description sentences corresponding to the video.

Preferably, in step S1, the extracting optical flow features of the video to be described specifically includes:

s1.1, respectively calculating optical flow characteristic values in the x direction and the y direction of every two adjacent frames of the video, and normalizing to a pixel range of [0,255 ];

and S1.2, calculating the amplitude value of the optical flow, and combining the optical flow characteristic values obtained in the last step to form an optical flow graph.

Preferably, the specific steps of obtaining the high-level feature expression of the key frame and the optical flow feature in step S2 are as follows:

s2.1, sequentially inputting the key frames of the video obtained in the step S1 to a fractal network for processing the spatial dimension relation in a time point sequence, and sequentially generating corresponding visual feature vectors through the nonlinear mapping relation of the network;

and S2.2, sequentially inputting the optical flow graph obtained in the step S1 to a second fractal network for processing the time dimension relation in the time point sequence, and sequentially generating corresponding motion characteristic vectors through the nonlinear mapping relation of the network.

Preferably, for repeated application in steps S2.1 and S2.2 by a single extension rule, an extremely deep network is generated whose structural layout is a truncated fractal; the network comprises interacting sub-paths of different lengths but does not comprise any through connections; meanwhile, in order to realize the capability of extracting the high-performance fixed-depth sub-network, a path abandoning method is adopted to regularize the rule of cooperative adaptation of the sub-paths in the fractal architecture; for fractal networks, the simplicity of training corresponds to the simplicity of design, with a single loss function connected to the last layer sufficient to drive internal behavior to mimic deep supervision; the adopted fractal network is a deep convolutional neural network based on a fractal structure.

Preferably, the repeated application of the single extension rule in steps S2.1 and S2.2 generates an extremely deep network whose structural layout is a truncated fractal specifically:

basic situation f₁(z) layers comprising a single selected type between input and output; let C denote a cutoff scoreShape f_CIndex of (c), f_C(. cndot.) defines the network architecture, connections, and layer types. Wherein the basic case is a network representation comprising a single convolutional layer as in equation (1-1):

f₁(z)＝conv(z) (1-1)

the following fractal is recursively defined as in equation (1-2):

in the formula (1-2), the first and second,represents compounding ofIndicating a connection operation, C corresponding to the number of columns, or network f_CWidth of (·); depth is defined as the number of conv layers on the longest path from input to output, proportional to 2^C-1(ii) a Convolutional networks for classification typically have a distributed arrangement of layers of aggregation; for the same purpose, use is made of f_CAs a building unit, stack it with the next collection layer B times, resulting in a total depth B.2^C-1(ii) a Connecting operationCombining the two feature blocks into one; a feature block is the result of a conv layer: maintaining an activated tensor for fixed channels in a spatial region; the number of channels corresponds to the number of filters of the preceding conv layer; when the parts are spread, combining the adjacent connections into a single connection layer; the connection layer merges all its input feature blocks into a single output block.

Preferably, the rule for regularizing the cooperative adaptation of the sub-paths in the fractal architecture by using the path discarding method in steps S2.1 and S2.2 specifically includes: because the fractal network comprises an additional large-scale structure, a coarse-grained regularization strategy similar to dropout and drop-connect is used, and paths abandon that the common adaptation of parallel paths is forbidden by randomly discarding operands of a connection layer, so that the mode effectively prevents the network from using one path as an anchor and the other path as a correction to possibly cause over-fitting behavior; two sampling strategies were employed:

for local, the connectivity layer discards each input with a fixed probability, but guarantees that at least one input is retained;

for global, each path is chosen for the entire network, and by restricting this path to being single-column, each column is motivated to be a powerful predictor.

Preferably, the step S3 of inputting the high-level feature vector into two LSTM unit-based recurrent neural network models is specifically:

the recurrent neural network based on the LSTM units includes two layers of LSTM units, the first layer and the second layer respectively include 1000 neurons, wherein the forward propagation process of each LSTM neural unit can be expressed as:

i_t＝σ(W_xix_t+W_hih_t-1+b_i) (1-3)

f_t＝σ(W_xfx_t+W_hfh_t-1+b_f) (1-4)

o_t＝σ(W_xox_t+W_hoh_t-1+b_o) (1-5)

c_t＝f_t*c_t-1+i_t*g_t(1-7)

wherein, σ (x)_t)＝(1+e^-xt)^-1Is a sigmoid non-linear activation function,is a hyperbolic tangent nonlinear activation function; i.e. i_t，f_t，o_t，c_tRespectively representing the state quantities corresponding to the input gate, the memory gate, the output gate and the core gate at the time t; for each logic gate, W_xi，W_xf，W_xo，W_xgRespectively representing the weight transfer matrices, W, corresponding to the input gate, memory gate, output gate and core gate_hi，W_hf，W_ho，W_hgRespectively representing hidden layer variable h of input gate, memory gate, output gate and core gate at time t-1_t-1Corresponding weight transfer matrix, b_i，b_f，b_o，b_gRespectively representing the corresponding offset vectors of the input gate, the memory gate, the output gate and the core gate.

Preferably, the neural network model structure in step S3 is:

based on the recurrent neural network structure diagram of the two layers of LSTM units, the recurrent neural network of the LSTM unit stacked by two layers is utilized to carry out the operation of coding and decoding the input feature vector, thereby realizing the conversion of the natural language text; the first layer of LSTM neurons finish the coding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the input of the second layer of LSTM neurons; when the feature vectors of all video frames are input into the first layer of LSTM neurons, the second layer of LSTM neurons can receive an indicator and start a decoding task; in the decoding stage, the network has information loss, so the goal of model parameter training and learning is to maximize the log-likelihood function of the whole output statement prediction on the premise of giving hidden layer expression and output prediction at the previous moment; for output sentence Y with parameter θ₁,y₂,…,y_m) The model, the parametric optimization objective, can be expressed as:

here, θ is a parameter, Y represents an output prediction statement, h is a hidden layer expression, the objective function is optimized by using a random gradient descent method, and errors of the whole network are cumulatively transferred in a time dimension by a back propagation algorithm.

Preferably, in step S4, the specific operation of weighted averaging the output values of the two neural network independent models at each time point and obtaining the description sentence corresponding to the video is:

s4.1, carrying out weighted average on output values of second-layer LSTM neurons at each moment of the two independent recurrent neural network models;

s4.2, calculating the occurrence probability of each word in the vocabulary V by adopting a softmax function, wherein the probability is expressed as:

where y denotes the predicted word, z_tRepresenting the output value, W, of the recurrent neural network at time t_yRepresenting the weight value of the word in the vocabulary.

And S4.3, in the decoding stage of each moment, taking the word with the maximum probability in the output value of the softmax function, thereby obtaining the corresponding video description sentence.

Compared with the prior art, the invention has the following advantages and beneficial effects:

1. the optical flow characteristics added by the invention compensate the dynamic information which is inevitably lost by the sampling frames, and the change of the video in the space dimension and the time dimension is considered.

2. The video description method based on the two-way fractal network and the LSTM can automatically generate a descriptive language about video contents from end to end by processing any input video, and can be applied to the application fields of video retrieval, video monitoring, human-computer interaction and the like.

3. The invention carries out abstract visual feature expression on the bottom layer features through a novel fractal network, thereby more accurately analyzing and mining the relation of people, objects, behaviors, spatial position relation and the like in the video.

Drawings

Fig. 1 is a flow framework diagram of a two-way fractal network and LSTM based video description method provided by the present invention;

FIG. 2 is a schematic diagram of a fractal subnetwork used in an embodiment of the present invention;

FIG. 3 is a schematic diagram of an LSTM unit based recurrent neural network employed by an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.

The method comprises the steps of sampling key frames of a video to be described, extracting optical flow characteristics between two adjacent frames of an original video, learning and obtaining high-level characteristic expressions of the key frames and the optical flow characteristics through two fractal networks, inputting the high-level characteristic expressions into two recurrent neural network models based on an LSTM unit, and performing weighted average on output values of the two independent recurrent neural network models at each moment to obtain a description sentence corresponding to the video.

FIG. 1 is an overall flow chart of the present invention, comprising the steps of:

(1) sampling key frames of a video to be described, and extracting optical flow characteristics between two adjacent frames of an original video; wherein, the specific operation of extracting the optical flow characteristic of the video to be described is as follows:

1. respectively calculating the light flow values in the x direction and the y direction of every two adjacent frames of the video, and normalizing to the pixel range of [0,255 ];

2. and calculating the magnitude value of the optical flow, and combining the optical flow characteristic values obtained in the last step to form an optical flow graph.

(2) And respectively learning and obtaining high-level feature expressions of video frames and optical flow features through two fractal networks. Sequentially inputting the sampling frames of the video obtained in the first step to a fractal network for processing the spatial dimension relation in the order of time points, and sequentially generating corresponding visual feature vectors through the nonlinear mapping relation of the network; and sequentially inputting the obtained light flow diagrams to a second fractal network for processing the time dimension relation in the time point sequence, and sequentially generating corresponding motion characteristic vectors through the nonlinear mapping relation of the network.

The fractal network is mainly characterized in that a self-similarity-based design strategy is introduced on a macroscopic framework of a neural network, an extremely deep network is generated through repeated application of a single expansion rule, and the structural layout of the fractal network is a truncated fractal. The network comprises interacting sub-paths of different lengths but does not comprise any through connections. Meanwhile, in order to realize the capability of extracting high-performance fixed-depth sub-networks, a path rejection method is adopted to regularize the cooperative adaptation of sub-paths in the fractal architecture. For fractal networks, the simplicity of training corresponds to the simplicity of the design, with a single loss function connected to the last layer sufficient to drive internal behavior to mimic deep supervision. The fractal network adopted in the invention is a deep convolutional neural network based on a fractal structure.

As shown in FIG. 2, a schematic diagram of a fractal structure, a basic scenario f₁(z) layers comprising a single selected type between input and output; let C denote the truncated fractal f_CIndex of (c), f_C(. cndot.) defines the network architecture, connections, and layer types. It is composed ofThe basic case is a network representation containing a single convolutional layer as in equation (1-1):

f₁(z)＝conv(z) (1-1)

the following fractal structure is then defined by recursion as in equation (1-2):

in the formula (1-2), the first and second,represents compounding ofIndicating a connection operation, C corresponding to the number of columns, or network f_CWidth of (·); depth is defined as the number of conv layers on the longest path from input to output, proportional to 2^C-1(ii) a Convolutional networks for classification typically have a distributed arrangement of layers of aggregation; for the same purpose, use is made of f_CAs a building unit, stack it with the next collection layer B times, resulting in a total depth B.2^C-1(ii) a Connecting operationCombining two feature blocks into one, wherein one feature block is the result of one convolutional layer: the activated tensor is maintained for fixed channels in a spatial region. The number of channels corresponds to the number of preceding convolutional layer filters. When the portions are extended, adjacent connections merge into a single connection layer. As shown on the right side of fig. 2, this connectivity layer spans multiple columns, merging all of its input feature blocks into a single output block.

Since fractal networks contain additional large scale structures, it is proposed to use a coarse-grained regularization strategy like dropout and drop-connect. Path dropping inhibits the common adaptation of parallel paths by randomly dropping operands at the link layer, which effectively prevents the network from using one path as an anchor and another path as a correction that may cause overfitting behavior. Here, two sampling strategies are mainly used:

for local, the connectivity layer discards each input with a fixed probability, but guarantees that at least one input is retained;

for global, each path is chosen for the entire network, and by restricting this path to being single-column, each column is motivated to be a powerful predictor.

(3) And respectively inputting the high-level feature vectors obtained in the last step into two recurrent neural networks based on the LSTM unit. The recurrent neural network based on the LSTM units includes two layers of LSTM units, the first layer and the second layer respectively include 1000 neurons, wherein the forward propagation process of each LSTM neural unit can be expressed as:

i_t＝σ(W_xix_t+W_hih_t-1+b_i) (1-3)

f_t＝σ(W_xfx_t+W_hfh_t-1+b_f) (1-4)

o_t＝σ(W_xox_t+W_hoh_t-1+b_o) (1-5)

c_t＝f_t*c_t-1+i_t*g_t(1-7)

wherein, σ (x)_t)＝(1+e^-xt)^-1Is a sigmoid non-linear activation function,is a hyperbolic tangent nonlinear activation function; i.e. i_t，f_t，o_t，c_tRespectively representing the state quantities corresponding to the input gate, the memory gate, the output gate and the core gate at the time t; for each logic gate, W_xi，W_xf，W_xo，W_xgRespectively representing the weight transfer matrices, W, corresponding to the input gate, memory gate, output gate and core gate_hi，W_hf，W_ho，W_hgRespectively representing hidden layer variable h of input gate, memory gate, output gate and core gate at time t-1_t-1Corresponding weight transfer matrix, b_i，b_f，b_o，b_gRespectively representing the corresponding offset vectors of the input gate, the memory gate, the output gate and the core gate.

As shown in fig. 3, the recurrent neural network structure based on two layers of LSTM units is used to perform the operations of encoding and decoding the input feature vector, so as to realize the conversion of the natural language text. The first layer of LSTM neurons finish the coding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the input of the second layer of LSTM neurons; when all the feature vectors of the video frames are input into the first layer of LSTM neurons, the second layer of LSTM neurons receive an indicator and begin the decoding task. In the decoding stage, the network has information loss, so the goal of model parameter training and learning is to maximize the log-likelihood function of the whole output statement prediction on the premise of giving hidden layer expression and output prediction at the last moment. For output sentence Y with parameter θ₁,y₂,…,y_m) The model, the parametric optimization objective, can be expressed as:

here, θ is a parameter, Y represents an output prediction statement, h is a hidden layer expression, the objective function is optimized by using a random gradient descent method, and errors of the whole network are cumulatively transferred in a time dimension by a back propagation algorithm.

(4) Carrying out weighted average on output values of two independent models at each moment and obtaining a description sentence corresponding to a video, wherein the specific operation is as follows:

1. carrying out weighted average on output values of the second layer of LSTM neurons at each moment of the two independent models;

2. the probability of occurrence of each word in the vocabulary V is calculated using the softmax function, and is expressed as:

where y denotes the predicted word, z_tRepresenting the output value, W, of the recurrent neural network at time t_yRepresenting the weight value of the word in the vocabulary.

3. And in the decoding stage at each moment, the word with the maximum probability in the output value of the softmax function is taken, so that the corresponding video description sentence is obtained.

The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.

Claims (9)

1. A video description method based on two-way fractal network and LSTM is characterized in that firstly, sampling key frames of a video to be described is carried out, optical flow characteristics between two adjacent frames of an original video are extracted, then the two fractal networks are used for learning respectively and obtaining high-level characteristic expressions of the key frames and the optical flow characteristics, then the high-level characteristic expressions are respectively input into two recurrent neural network models based on an LSTM unit, and finally, weighted average is carried out on output values of the two independent recurrent neural network models at each moment, so that description sentences corresponding to the video are obtained; the method specifically comprises the following steps:

s1, sampling key frames of a video to be described, and extracting optical flow characteristics between two adjacent frames of an original video;

s2, learning and obtaining high-level feature expressions of key frames and optical flow features through two fractal networks respectively; wherein the fractal network is generated by repeated application of a single expansion rule;

s3, respectively inputting the high-level feature vectors obtained in the previous step into two recurrent neural network models based on an LSTM unit;

and S4, carrying out weighted average on the output values of the two independent recurrent neural network models at each moment and obtaining the description sentences corresponding to the video.

2. The method for describing the video based on the two-way fractal network and the LSTM according to claim 1, wherein the step S1 of extracting the optical flow features from the video to be described specifically includes:

s1.1, respectively calculating optical flow characteristic values in the x direction and the y direction of every two adjacent frames of the video, and normalizing to a pixel range of [0,255 ];

and S1.2, calculating the amplitude value of the optical flow, and combining the optical flow characteristic values obtained in the last step to form an optical flow graph.

3. The video description method based on two-way fractal network and LSTM as claimed in claim 1, wherein the specific steps of obtaining the high-level feature expression of the key frame and optical flow features in step S2 are as follows:

s2.1, sequentially inputting the key frames of the video obtained in the step S1 to a fractal network for processing the spatial dimension relation in a time point sequence, and sequentially generating corresponding visual feature vectors through the nonlinear mapping relation of the network;

and S2.2, sequentially inputting the optical flow graph obtained in the step S1 to a second fractal network for processing the time dimension relation in the time point sequence, and sequentially generating corresponding motion characteristic vectors through the nonlinear mapping relation of the network.

4. A video description method based on two-way fractal network and LSTM according to claim 3, characterized in that the repeated application of the single extension rule in steps S2.1 and S2.2 generates an extremely deep network whose structural layout is a truncated fractal; the network comprises interacting sub-paths of different lengths but does not comprise any through connections; meanwhile, in order to realize the capability of extracting the high-performance fixed-depth sub-network, a path abandoning method is adopted to regularize the rule of cooperative adaptation of the sub-paths in the fractal architecture; for fractal networks, the simplicity of training corresponds to the simplicity of design, with a single loss function connected to the last layer sufficient to drive internal behavior to mimic deep supervision; the adopted fractal network is a deep convolutional neural network based on a fractal structure.

5. The method for describing videos based on two-way fractal network and LSTM according to claim 4, wherein the repeated application of the single extension rule in steps S2.1 and S2.2 generates an extremely deep network whose structural layout is a truncated fractal, specifically:

basic situation f₁(z) layers comprising a single selected type between input and output; let C denote the truncated fractal f_CIndex of (c), f_C() defines network architecture, connections, and layer types; wherein the basic case is a network representation comprising a single convolutional layer as in equation (1-1):

f₁(z)＝conv(z) (1-1)

the following fractal is recursively defined as in equation (1-2):

in the formula (1-2), the first and second,represents compounding ofIndicating a connection operation, C corresponding to the number of columns, or network f_CWidth of (·); depth is defined as the longest path from input to output^convNumber of layers, proportional to 2^C-1(ii) a Convolutional networks for classification typically have a distributed arrangement of layers of aggregation; for the same purpose, use is made of f_CAs a building unit, stack it with the next collection layer B times, resulting in a total depth B.2^C-1(ii) a Connecting operationCombining the two feature blocks into one; a feature block is the result of a conv layer: maintaining an activated tensor for fixed channels in a spatial region; the number of channels corresponds to the number of filters of the preceding conv layer; when the parts are spread, combining the adjacent connections into a single connection layer; the connection layer merges all its input feature blocks into a single output block.

6. The video description method according to claim 4, wherein the regularized adaptation rule of the sub-paths in the fractal architecture by using one of the path discarding methods in steps S2.1 and S2.2 is specifically: because the fractal network comprises an additional large-scale structure, a coarse-grained regularization strategy similar to dropout and drop-connect is used, and paths abandon that the common adaptation of parallel paths is forbidden by randomly discarding operands of a connection layer, so that the mode effectively prevents the network from using one path as an anchor and the other path as a correction to possibly cause over-fitting behavior; two sampling strategies were employed:

for local, the connectivity layer discards each input with a fixed probability, but guarantees that at least one input is retained;

for global, each path is chosen for the entire network, and by restricting this path to being single-column, each column is motivated to be a powerful predictor.

7. The method for describing a video based on a two-way fractal network and an LSTM according to claim 1, wherein the step S3 of inputting the high-level feature vector to two recurrent neural network models based on LSTM units specifically includes: the recurrent neural network based on the LSTM units includes two layers of LSTM units, the first layer and the second layer respectively include 1000 neurons, wherein the forward propagation process of each LSTM neural unit can be expressed as:

i_t＝σ(W_xix_t+W_hih_t-1+b_i) (1-3)

f_t＝σ(W_xfx_t+W_hfh_t-1+b_f) (1-4)

o_t＝σ(W_xox_t+W_hoh_t-1+b_o) (1-5)

c_t＝f_t*c_t-1+i_t*g_t(1-7)

wherein,is a sigmoid non-linear activation function,is a hyperbolic tangent nonlinear activation function; i.e. i_t，f_t，o_t，c_tRespectively representing the state quantities corresponding to the input gate, the memory gate, the output gate and the core gate at the time t; for each logic gate, W_xi，W_xf，W_xo，W_xgRespectively representing the weight transfer matrices, W, corresponding to the input gate, memory gate, output gate and core gate_hi，W_hf，W_ho，W_hgRespectively represent an input gate and a memory gateThe output gate and the core gate hide the layer variable h at the time t-1_t-1Corresponding weight transfer matrix, b_i，b_f，b_o，b_gRespectively representing the corresponding offset vectors of the input gate, the memory gate, the output gate and the core gate.

8. The video description method based on two-way fractal network and LSTM of claim 7, wherein the neural network model structure in step S3 is:

realizing the conversion of natural language texts based on a recurrent neural network of two layers of LSTM units; the first layer of LSTM neurons finish the coding process of the input visual feature vector at each moment, and then the hidden layer expression output at each moment is used as the input of the second layer of LSTM neurons; when the feature vectors of all video frames are input into the first layer of LSTM neurons, the second layer of LSTM neurons can receive an indicator and start a decoding task; in the decoding stage, the network has information loss, so the goal of model parameter training and learning is to maximize the log-likelihood function of the whole output statement prediction on the premise of giving hidden layer expression and output prediction at the previous moment; for output sentence Y with parameter θ₁,y₂,…,y_m) The model, the parametric optimization objective, can be expressed as:

${\mathrm{\θ}}^{*}=\arg \max \underset{(h,Y)}{\Σ}\log p\left(Y\right|h;\mathrm{\θ})---(1-9)$

here, θ is a parameter, Y represents an output prediction statement, h is a hidden layer expression, the objective function is optimized by using a random gradient descent method, and errors of the whole network are cumulatively transferred in a time dimension by a back propagation algorithm.

9. The method for describing the video based on the two-way fractal network and the LSTM according to claim 1, wherein the step S4 is specifically operated to perform weighted average on the output values of the two neural network independent models at each moment and obtain the description sentence corresponding to the video:

s4.1, carrying out weighted average on output values of second-layer LSTM neurons at each moment of the two independent recurrent neural network models;

s4.2, calculating the occurrence probability of each word in the vocabulary V by adopting a softmax function, wherein the probability is expressed as:

$P\left(y\right|z_t)=\frac{\exp \left(W_y{z}_t\right)}{{\mathrm{\Σ}}_{y^{\′}\∈V}\exp \left(W_{y^{\′}}{z}_t\right)}---(1-10)$

where y denotes the predicted word, z_tRepresenting the output value, W, of the recurrent neural network at time t_yRepresenting the weight value of the word in the vocabulary;

and S4.3, in the decoding stage of each moment, taking the word with the maximum probability in the output value of the softmax function, thereby obtaining the corresponding video description sentence.