CN113573076A - Method and apparatus for video coding - Google Patents

Abstract

The present application relates to video coding technology in the field of artificial intelligence, and provides a video coding method and apparatus, which can generate more accurate prediction frames, thereby improving video coding efficiency. The video coding method includes: acquiring reconstructed frames of a fixed number of second video frames before a first video frame to be coded in a video sequence; according to the reconstructed frames of the second video frame and the global long-term memory of the video sequence , generating a synthetic reference frame of the first video frame, wherein the global long-term memory is based on the reconstructed frame sum of each video frame in a plurality of video frames preceding the first video frame in the video sequence The synthetic reference frame of each video frame is determined; and the first video frame is encoded according to the synthetic reference frame of the first video frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

Description

Method and apparatus for video coding

technical field

The present application relates to video coding technology in the field of artificial intelligence, and more particularly, to a video coding method and apparatus.

Background technique

As users' demands for high-quality video continue to increase, new-generation video coding standards such as high-efficiency video coding (HEVC) have been proposed. Video is composed of consecutive video frames, and there is a strong correlation between adjacent video frames. Inter-frame prediction in mainstream video coding standards utilizes this temporal correlation to achieve data compression. By putting the encoded reconstructed frame into the reference frame list as a reference for the next frame encoding, block-level motion estimation is used to eliminate temporal redundancy between adjacent frames. However, the traditional reference mode also has shortcomings. For example, limited by the block-level linear motion search mechanism, it is difficult for traditional inter-frame prediction to describe irregular motion patterns such as rotational motion, which hinders the further improvement of video compression efficiency. Therefore, how to improve the inter-frame prediction mechanism so that it can better handle the complex motion between frame sequences is of great significance for the further development of the encoder.

In recent years, deep learning technology has developed rapidly and has shown good performance in the field of computer vision. Therefore, there are also attempts to use deep learning technology to improve the effect of inter-frame prediction. In video encoders, there is a low delay encoding configuration. In this configuration, the coding sequence of frames is consistent with the numbering of each frame in the video sequence, that is, the coding of video frames is performed sequentially from front to back. This encoding configuration can be applied to usage scenarios such as live streaming. Also because of this coding structure, there is a strong temporal continuity between the frame sequences to be coded. Therefore, there is a method to use deep learning technology to output the predicted frame of the next frame by using several encoded frames as the input of the neural network, and use the predicted result as an additional reference frame for the encoding process of the next frame, Thereby, the video coding efficiency is improved.

However, in this method, in the process of generating reference frames, only a fixed number of frames are input to the neural network to generate predicted frames, and the information contained in the remaining encoded frames is not utilized. Therefore, how to further improve the video coding efficiency is an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides a video coding method and apparatus, which can generate more accurate prediction frames, thereby improving video coding efficiency.

In a first aspect, a video encoding method is provided, the method comprising:

Obtaining reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence;

A synthetic reference frame of the first video frame is generated according to the reconstructed frame of the second video frame and the global long-term memory of the video sequence, wherein the global long-term memory is based on the first video sequence in the video sequence. Determined by the reconstructed frame of each video frame in a plurality of video frames preceding a video frame and the composite reference frame of each video frame;

The first video frame is encoded according to a composite reference frame of the first video frame.

Therefore, the embodiment of the present application generates a synthetic reference frame of the first video frame according to the reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence, and the global long-term memory of the video sequence, The first video frame is then encoded according to the composite reference frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

The fixed number of second video frames before the first video frame may be, for example, the first two video frames before the first video frame, or the first three video frames before the first video frame, or the first video frame The previous previous video frame is not limited in this embodiment of the present application. Here, the reconstructed frames of a fixed number of second video frames before the first video frame include short-term temporal information before encoding the first video frame.

The global long-term memory of the video sequence may be determined according to a plurality of encoded video frames in the video sequence, for example, determined according to the reconstructed frame of each video frame in the plurality of video frames and the synthesized reference frame of each video of. Exemplarily, the multiple video frames may be all or part of the encoded video frames in the video sequence, which is not limited in this embodiment of the present application.

Since the reconstructed frames of the fixed number of second video frames contain strong temporal correlations between adjacent video frames, and the global long-term memory contains long-term temporal information of the video sequence, according to the fixed number of previous video frames of the first video frame The reconstructed frame of the second video frame and the global long-term memory of the video sequence are used to generate a synthetic reference frame, which can be used as a reference frame when the first video frame is encoded. The ability of motion (eg nonlinear motion, rotational motion).

It should be noted that, the video coding method in this embodiment of the present application may be applied to a low delay coding configuration. In the low delay encoding configuration, the encoding order of video frames is the same as the order of their frame numbers, that is, sequential encoding. For example, before encoding the t-th (t>2) video frame, the t-1-th and t-2-th video frames have been encoded. Therefore, before encoding the t-th video frame, the reconstructed frames of the t-1-th video frame and the t-2-th video frame may be obtained first. In this way, the embodiments of the present application can realize the simultaneous advancement of the global long-term memory and the encoding process.

In conjunction with the first aspect, in some implementations of the first aspect, further includes:

obtaining a reconstructed frame of the first video frame;

The global long-term memory is updated according to the difference between the reconstructed frame of the first video frame and the synthetic reference frame of the first video frame. Here, the difference can be regarded as an error generated during the encoding process of the first video frame.

Therefore, in the embodiments of the present application, the global long-term memory is updated in real time according to the difference between the reconstructed frame of the video frame and the synthetic reference frame of the video frame, so that the video frame sequence can be iteratively encoded, thereby realizing long-term time The continuous transfer of domain information enables the synthetic reference frame of each video frame in the video sequence to have the ability to describe auxiliary motion between video sequences, thereby further improving coding performance. For example, the present application can dynamically update the global long-term memory from the encoding of the first frame to the end of the encoding of the last frame, so that the temporal information can be fully utilized to generate more accurate prediction frames and improve the video encoding efficiency.

In some embodiments, the difference value may be acquired by a module or unit provided in the reference frame generation module, and the long-term memory is updated according to the difference value. In other embodiments, a special module or unit, such as a memory updating module, may also acquire the difference value, and update the long-term memory according to the difference value, which is not limited in this embodiment of the present application.

In conjunction with the first aspect, in some implementations of the first aspect, further includes:

extracting feature information of the reconstructed frame of the second video frame;

Wherein, generating the synthetic reference frame of the first video frame according to the reconstructed frame of the second video frame and the global long-term memory of the video sequence includes:

The feature information of the reconstructed frame of the second video frame and the global long-term memory are input into a reference frame generation network model, and a synthetic reference frame of the first video frame is obtained, wherein the reference frame generation network model is based on training The training data sample set includes a plurality of video sequence samples, and each video sequence sample includes a lossless video frame and a video frame obtained by lossy compression of the lossless video frame.

Therefore, in this embodiment of the present application, deep learning can be used to use the reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence and the global long-term memory of the video sequence as the input of the neural network model, Thus, the synthetic reference frame of the first video frame is output, and then the first video frame is encoded according to the synthetic reference frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

With reference to the first aspect, in some implementations of the first aspect, the feature information of the reconstructed frame of the second video frame and the global long-term memory are input into the reference frame to generate a network model, and the first video is obtained. A composite reference frame of frames, including:

Obtain the synthetic reference frame of the first video frame according to the following formula:

表示所述第一视频帧的合成参考帧，t表示所述第一视频帧的帧号，i表示所述第一视频帧之前的第二视频帧的帧号，I_i表示第i帧视频的重建帧，

表示深度学习中的局部卷积操作，K_i表示所述第一视频帧中所有像素的卷积核系数的集合，⊙表示像素级别的点乘操作，用于将对于输入帧进行局部卷积得到的结果的像素级别加权相加，其权值矩阵为M_i，ε_t表示对所述第一视频帧进行编码时的所述全局长期记忆，γ_t表示所述第二视频帧的重建帧的特征信息，t，i为正整数，且t＞2。in,

Represents the synthetic reference frame of the first video frame, t represents the frame number of the first video frame, i represents the frame number of the second video frame before the first video frame, I _i represents the ith video frame number. reconstructed frame,

represents the local convolution operation in deep learning, K _i represents the set of convolution kernel coefficients of all pixels in the first video frame, and ⊙ represents the pixel-level dot product operation, which is used to obtain the input frame by local convolution The _pixel -level _{weighted addition} of the results of Feature information, t, i are positive integers, and t>2.

Therefore, in the embodiment of the present application, the method of generating convolution kernel coefficients for each pixel, that is, the method of local convolution, has stronger expressive ability than using the same regression method for all pixels, so it can A better regression effect can be achieved, thereby helping to generate more accurate predicted frames, thereby improving video coding efficiency. Here, regression refers to the process of generating synthetic reference frames.

With reference to the first aspect, in some implementation manners of the first aspect, the second video frame includes the first two video frames of the first video frame, wherein the first video frame is in the video sequence The third video frame of the video frame, or the video frame following the third video frame.

In some embodiments, since the number of reconstructed frames input to the reference frame generation module is insufficient when encoding the first frame and the second video frame of the entire video sequence, the reference frame generation module may not generate the first frame at this time. A composite reference frame of a video frame, or a composite video frame of a second video frame.

After that, starting from the fourth video frame, for example, the reference frame generation module and the encoder module can enter the normal working mode, that is, the reference frame generation module continuously generates a synthetic reference frame according to the long-term memory and the reconstructed frames of the first two video frames. The module exclusively authorizes the synthesized reference frame and produces a reconstructed frame of the currently encoded video frame. Optionally, the encoder module can obtain the difference between the reconstructed frame of the video frame and the reference synthetic frame, so that the reference frame generation module can complete the update of the long-term memory according to the difference. The above process can be repeated continuously until all the video frames in the video frame sequence are completely encoded.

With reference to the first aspect, in some implementations of the first aspect, when the first video frame is a third video frame in the video sequence, the global long-term memory is 0. Additionally, the global long-term memory can be set to 0 before encoding the third video frame.

With reference to the first aspect, in some implementations of the first aspect, the encoding the first video frame according to the synthesized reference frame of the first video frame includes:

obtaining a reference frame list of the first video frame, where the reference frame list includes reconstructed frames of at least two video frames that have been encoded;

Remove the reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the first video frame in the reference frame list, and add the synthesized reference frame of the first video frame to the reference frame in the reference frame list. the position of the removed reconstructed frame;

The first video frame is encoded according to the reference frame list.

Therefore, since the reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the first video frame and the first video frame in the reference frame list has the weakest temporal correlation with the first video frame, by comparing the reference frame list with the first video frame The reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the first video frame is removed, and the synthetic reference frame of the first video frame is added to the reference frame list, which is used as a reference in the current encoding process, thereby Utilize the time domain connection between different frames to achieve better compression effect.

In a second aspect, a video encoding apparatus is provided, which is used to execute the method in the first aspect or any possible implementation manner of the first aspect. Specifically, the device includes a device for executing the first aspect or the first aspect A module for a method in any possible implementation of an aspect.

In a third aspect, a video encoding apparatus is provided, including: a memory and a processor. Wherein, the memory is used to store instructions, the processor is used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the execution causes the video encoding apparatus to perform the first aspect or the first aspect. A method in any possible implementation.

In a fourth aspect, a computer-readable medium is provided for storing a computer program, the computer program comprising instructions for performing the method of the first aspect or any possible implementation of the first aspect.

In a fifth aspect, there is provided a computer program product comprising instructions, when the computer program product runs on a computer, the computer program product causes the computer to execute the first aspect or any possible implementation manner of the first aspect.

It should be understood that the beneficial effects obtained by the second to fifth aspects of the present application and the corresponding implementation manners can be referred to the beneficial effects obtained by the first aspect of the present application and the corresponding implementation manners, which will not be repeated.

Description of drawings

FIG. 1 shows a schematic block diagram of a video coding system provided by an embodiment of the present application.

FIG. 2 shows a schematic flowchart of a video coding method provided by an embodiment of the present application.

FIG. 3 shows a specific example of encoding the t-th video frame.

FIG. 4 shows a specific example of video coding provided by an embodiment of the present application.

FIG. 5 shows an example of a PSNR curve of the video coding solution according to the embodiment of the present application.

FIG. 6 shows a schematic block diagram of an apparatus for video encoding provided by an embodiment of the present application.

FIG. 7 shows a schematic block diagram of another apparatus for video coding provided by an embodiment of the present application.

Detailed ways

The technical solutions in the present application will be described below with reference to the accompanying drawings.

FIG. 1 shows a schematic block diagram of a system 100 for video coding provided by an embodiment of the present application. Exemplarily, the system 100 may be set on an intelligent video storage and playback device, such as an electronic product (mobile phone, TV, computer, etc.) with a camera function, a video playback function, or a video storage function. As shown in FIG. 1 , the system 100 includes a reference frame generation module 110 and an encoder module 120 . Exemplarily, the reference frame generation module 110 may be a long short-term memory (LSTM) network, and the encoder module 120 may be an HEVC video encoder.

The reference frame generation module 110 is configured to generate a synthetic reference frame of the first video frame to be encoded according to the encoded video frames in the video sequence. Here, the encoded video frame includes all or part of the encoded video frames before the first video frame. For example, in addition to a fixed number of encoded video frames before the first video frame, these encoded video frames may also include encoded video frames before the fixed number of video frames. This application implements The example does not limit this.

In some embodiments, the above-mentioned encoded video frames may include reconstructed frames of a fixed number of second video frames before the current first video frame to be encoded, and the global long-term memory of the video sequence (also referred to as long-term memory, global memory, etc., not limited). Wherein, the reconstructed frame refers to the encoding end device (for example, the encoder module 120 in the above-mentioned system 100 ) to simulate the decoding end device to restore the video frame to obtain the video frame.

The fixed number of second video frames before the first video frame may be, for example, the first two video frames before the first video frame, or the first three video frames before the first video frame, or the first video frame The previous previous video frame is not limited in this embodiment of the present application. Here, the reconstructed frames of a fixed number of second video frames before the first video frame include short-term temporal information before encoding the first video frame.

The global long-term memory of the video sequence may be determined according to a plurality of encoded video frames in the video sequence, for example, determined according to the reconstructed frame of each video frame in the plurality of video frames and the synthesized reference frame of each video of. Exemplarily, the multiple video frames may be all or part of the encoded video frames in the video sequence, which is not limited in this embodiment of the present application.

For example, in the case where the video frame currently to be encoded is the first video frame, the global long-term memory input to the reference frame generation module 110 may be based on multiple (eg all) videos in the video sequence preceding the first video frame A reconstructed frame for each video frame in the frame and a composite reference frame for that each video frame are determined.

Since the reconstructed frames of the fixed number of second video frames contain strong temporal correlations between adjacent video frames, and the global long-term memory contains long-term temporal information of the video sequence, according to the fixed number of previous video frames of the first video frame The reconstructed frame of the second video frame and the global long-term memory of the video sequence are used to generate a synthetic reference frame, which can be used as a reference frame when the first video frame is encoded. The ability of motion (eg nonlinear motion, rotational motion).

In this embodiment of the present application, the synthetic reference frame may also be referred to as a reference frame, a prediction frame, or the like, which is not limited in this embodiment of the present application.

The encoder module 120 is configured to use the synthesized reference frame of the first video frame as an additional reference frame for the encoding process of the first video frame to encode the first video frame.

Therefore, the embodiment of the present application generates a synthetic reference frame of the first video frame according to the reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence, and the global long-term memory of the video sequence, The first video frame is then encoded according to the composite reference frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

Optionally, the encoder module 120 may also obtain the reconstructed frame of the first video frame, for example, by simulating a decoder device to restore the first video frame, to obtain the reconstructed frame of the first video frame.

In some embodiments, system 100 may maintain this global long-term memory. For example, after the encoding of a video frame is completed, the global long-term memory is updated according to the reconstructed frame of the video frame. Exemplarily, after acquiring the reconstructed frame of the first video frame, the difference between the reconstructed frame of the first video frame and the synthetic reference frame of the first video frame can be determined, and the global long-term value can be updated according to the difference. memory. The updated global long-term memory can be used in the generation process of the synthetic reference frame of the next video frame of the first video frame. Specifically, the generation process and encoding process of the composite reference frame of the next frame are similar to the generation process and encoding process of the composite reference frame of the first video frame, and are not described again.

Therefore, in the embodiment of the present application, by updating the global long-term memory in real time according to the difference between the reconstructed frame of the video frame and the synthetic reference frame, iterative encoding of the video frame sequence can be implemented, thereby realizing long-term temporal information The continuous transfer of the video sequence enables the synthetic reference frame of each video frame in the video sequence to have the ability to describe the auxiliary motion between the video sequences, thereby further improving the coding performance. For example, the present application can dynamically update the global long-term memory from the encoding of the first frame to the end of the encoding of the last frame, so that the temporal information can be fully utilized to generate more accurate prediction frames and improve the video encoding efficiency.

FIG. 2 shows a schematic flowchart of a video coding method 200 provided by an embodiment of the present application. The method 200 may be applied to the system 100 of video coding shown in FIG. 1 . As shown in FIG. 2 , method 200 includes steps 210 to 240 .

210. Acquire reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence. Exemplarily, reference may be made to the above description for the second video frame of the fixed data, and details are not repeated here.

FIG. 3 shows a specific example of encoding the t-th video frame. Here, the t-th video frame is an example of the above-mentioned first video frame. As shown in FIG. 3 , the fixed number of second video frames before the t-th video frame may be the first two video frames of the t-th video frame, that is, the t-1 th video frame and the t-2 th video frame frame. Here, t is a positive integer and t>2. At this time, the reconstructed frame of the second video frame is the reconstructed frame of the t-1 th video frame (ie, the t-1 reconstructed frame in FIG. 3 ) and the reconstructed frame of the t-2 th video frame (ie, FIG. 3 ). t-2 reconstruction frame in ).

It should be noted that, the video coding method in this embodiment of the present application may be applied to a low delay coding configuration. In the low delay encoding configuration, the encoding order of video frames is the same as the order of their frame numbers, that is, sequential encoding. For example, before encoding the t-th (t>2) video frame, the t-1-th and t-2-th video frames have been encoded. Therefore, before encoding the t-th video frame, the reconstructed frames of the t-1-th video frame and the t-2-th video frame may be obtained first.

Therefore, the embodiment of the present application can be designed for a low delay encoding configuration, and in this configuration, the encoding of frames is performed in sequence, that is, the encoding sequence is consistent with the frame sequence number. Therefore, the embodiment of the present application can implement global long-term memory and Synchronous advancement of the encoding process.

220. Generate a synthetic reference frame of the first video frame according to the reconstructed frame of the second video frame and the global long-term memory of the video sequence, wherein the global long-term memory is based on all the data in the video sequence. It is determined by the reconstructed frame of each video frame in the plurality of video frames before the first video frame and the synthesized reference frame of each video frame. The global long-term memory and the synthetic reference frame may refer to the above description, and will not be repeated here.

Exemplarily, can continue to refer to FIG. 3, after obtaining the reconstruction frame of the t-1th frame and the t-2th frame of the video frame, the reference frame generation module can be used to generate the t-th frame. The synthetic reference frame of the video frame (that is, t synthetic reference frame in Fig. 3). Exemplarily, at this time, the input of the reference frame generation module may be the reconstructed frames of the t-1th frame and the t-2th frame of the video frame, and the long-term memory of the video sequence. Here, the long-term memory is used together with the reconstructed frames of the t-1th frame and the t-2th frame video frame as the input of the reference frame generation module, which can ensure the input of more sufficient time domain information.

230. Encode the first video frame according to the synthesized reference frame of the first video frame.

Exemplarily, continuing to refer to FIG. 3 , after the reference frame generation module generates the t-synthesized reference frame, the encoder module may read the t-synthesized reference frame. In some optional embodiments, the encoder module may put the t synthetic reference frame into a reference frame list in the encoder module, and complete the encoding of the t-th video frame according to the reference frame list. The reference frame list may include at least two video frames, which are used as references in the encoding process of the current frame.

In some embodiments, in the process of encoding each frame of video, the encoder module may maintain a list of reference frames corresponding to the frame of video. Exemplarily, before the encoder module obtains the synthetic reference frame of the t-th frame of video, the reference frame list corresponding to the t-th frame may store reconstructed frames of several video frames that have been encoded. After the encoder module obtains the synthesized reference frame of the t-th video frame, the frame number with the largest difference from the frame number of the currently encoded video frame (for example, the first video frame, or the t-th video frame) in the reference frame list can be determined. The reconstructed frame corresponding to the frame number is removed, and the synthesized reference frame of the current video frame (such as the first video frame, or the t-th video frame) obtained from the reference frame generation module is added to the reference frame list, for example, it can be added to The position of the removed reconstructed frame in the reference frame list is not limited in this embodiment of the present application.

Since the reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the first video frame and the first video frame in the reference frame list has the weakest temporal correlation with the first video frame, by comparing the first video frame in the reference frame list with the first video frame The reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the video frames is removed, and the synthesized reference frame of the first video frame is added to the reference frame list, which is used as a reference in the current encoding process, so as to use The time domain connection between different frames achieves better compression effect.

Therefore, the embodiment of the present application generates a synthetic reference frame of the first video frame according to the reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence, and the global long-term memory of the video sequence, The first video frame is then encoded according to the composite reference frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

In some optional embodiments, the method 200 may further include step 240, obtaining a reconstructed frame (t reconstructed frame) of the first video frame, and obtaining the reconstructed frame of the first video frame and the first video according to the reconstructed frame of the first video frame The difference value of the synthetic reference frame of the frame, the global long-term memory is updated.

Exemplarily, continuing to refer to FIG. 3 , the encoding end device (eg, the encoder module) may simulate the decoding end to restore the t video frame, so as to obtain the above-mentioned t reconstructed frame. After acquiring the t reconstructed frame, the difference between the t reconstructed frame and the t synthetic reference frame can be calculated. Here, the difference value can be considered as an error generated in the process of encoding the t-th video frame. After the difference value is obtained, the long-term memory maintained by the difference value can be dynamically updated according to the difference value, so that the future prediction process of the reference frame generation module (that is, the process of generating the next frame, such as the synthetic reference frame of the t+1 frame) can be realized. ) can perceive the error generated in the encoding process of the t-th video frame, thereby helping to generate a more accurate synthetic reference frame in the future and improving the video encoding efficiency.

In some embodiments, the difference value may be acquired by a module or unit provided in the reference frame generation module, and the long-term memory is updated according to the difference value. In other embodiments, a special module or unit, such as a memory updating module, may also acquire the difference value, and update the long-term memory according to the difference value, which is not limited in this embodiment of the present application.

It should be noted that, for the video encoding process in FIG. 3, since the number of reconstructed frames input to the reference frame generation module is insufficient when encoding the first frame and the second video frame of the entire video sequence, therefore At this time, the reference frame generation module may not generate a synthetic reference frame of the first frame of video frame, or a synthetic video frame of the second frame of video frame.

Before encoding the third frame of video frame, the reconstructed frame of the first frame of video frame and the reconstructed frame of the second frame of video frame have been generated, so at this time, the reference frame generation module can be used to generate the composite of the third frame of video frame reference frame. This is also in the process of encoding the entire video frame. The reference frame generation module generates a synthetic reference frame for the first time, so there is no long-term memory of the video sequence at this time. Before that, the reference frame generation module can set the long-term memory to 0. At this time, the long-term memory set to 0 and the reconstructed frames of the first and second video frames can be used as the input of the synthesis process, and the synthesized reference frame of the third video frame can be output. After that, starting from the fourth video frame, the reference frame generation module and the encoder module can enter the normal working mode, that is, the reference frame generation module continuously generates synthetic reference frames according to the long-term memory and the reconstructed frames of the first two video frames, and the encoder The module exclusively authorizes the synthesized reference frame and produces a reconstructed frame of the currently encoded video frame. Optionally, the encoder module can obtain the difference between the reconstructed frame of the video frame and the reference synthetic frame, so that the reference frame generation module can complete the update of the long-term memory according to the difference. The above process can be repeated continuously until all the video frames in the video frame sequence are completely encoded.

Hereinafter, with reference to FIG. 4 , a specific example of video coding provided by this embodiment of the present application will be described in detail. It should be noted that the following examples are only for helping those skilled in the art to understand and implement the embodiments of the present invention, rather than limiting the scope of the embodiments of the present invention. Those skilled in the art can perform equivalent transformations or modifications according to the examples given herein, and such transformations or modifications should still fall within the scope of the embodiments of the present invention.

Exemplarily, the system architecture of FIG. 4 can be implemented on a platform equipped with i7-9700K, 32G memory, GTX1080Ti based on Ubuntu 18.04. Please refer to FIG. 4 , the system includes a feature extraction module 410 , a reference frame generation module 420 , an encoder module 430 and a memory update module 440 . The feature extraction module 410, the reference frame generation module 420 and the memory update module 440 can be implemented by a neural network module. At this time, the system in FIG. 4 may also be referred to as a neural network model, or a neural network system.

然后，编码器模块430可以根据该合成参考帧

对第一视频帧进行编码。之后，编码器模块还可以获取并输出该第一视频帧的重建帧I_t，记忆更新模块440可以获取第一视频帧的重建帧I_t，并获取合成参考帧

与重建帧I_t的差值R_t，再由记忆更新模块440中的记忆更新网络430根据该差值R_t对全局长期记忆ε_t进行更新。In the example shown in FIG. 4 , the feature extraction module 410 can be used to extract a fixed number of reconstructed frames of the second video frame before the first video frame (eg t-1 reconstructed frame I _t-1 and t-2 reconstructed frame I _t-2 ), and then the extracted feature information of the reconstructed frame of the second video frame and the global long-term memory _εt are input to the reference frame to generate a network model, and the synthetic reference frame of the first video frame is obtained.

The encoder module 430 can then synthesize the reference frame from the

The first video frame is encoded. After that, the encoder module can also acquire and output the reconstructed frame It of the first video frame, and the memory updating module ₄₄₀ can acquire the reconstructed frame It of the first video _frame , and acquire the synthetic reference frame

The global long-term memory ε _{t is updated by the memory updating network 430 in the memory updating module 440 according to the difference R t with} the reconstructed frame It _.

Wherein, each neural network module in FIG. 4 may be obtained by training using a training data sample set, the training data sample set includes a plurality of video sequence samples, each video sequence sample includes a lossless video frame, and the lossless video frame Video frames obtained by lossy compression.

The specific parameters of each neural network module in the system of FIG. 4 are described below. Taking the encoding of the t-th video frame as an example, it is assumed that the length and width of the two frames of images input to the neural network model are h and w respectively.

Feature extraction module 410:

Input: Reconstructed frames of the previous two frames (ie, t-2 reconstructed frame It _-2 and t-1 reconstructed frame It- ₁ in Figure 4). The number of input channels of the feature extraction module 410 may be 6, that is, the reconstructed frames of the previous two frames are spliced on the number of channels (the number of input channels of each reconstructed frame is 3, that is, three input channels of RGB).

Network parameters: as shown in Table 1 below:

Table 1

It should be noted that the serial numbers in Table 1 are the names obtained after the feature extraction module 410 is numbered from left to right. For example, the leftmost convolution module in the feature extraction module 410 is called convolution 1, and so on. . The dashed line represents summing the outputs of the two parts, such as summing the outputs of convolution 1 and upsampling 5. This is a skipconnection structure common in autoencoders to pass low level information.

The reference frame generation module 420 includes a local convolution parameter generation network 421 and a weight generation network 422 .

Local convolution parameter generation network 421:

enter:

(1) The sum of the outputs of the convolution 2 and the upsampling 2 of the feature extraction module 410 is marked as γ _t ;

(2) Long-term memory, labeled ε _t .

Input integration method: splicing according to the number of channels in (1) and (2) above.

Output: The number of channels is 51, and the length and width are the same as the local convolution coefficients of the input image.

Network parameters: as shown in Table 2 below:

Table 2

Weight generation network 422:

Input: The sum of upsampling 5 and convolution 1.

Output: The number of channels is 1, and the weight matrix with the same length and width as the input image, that is, M _t-1 marked in Figure 4 . In addition, M _t-2 in Figure 4 can be obtained by subtracting M _t-1 from the all-ones matrix.

Network parameters: as shown in Table 3 below:

table 3

Wherein, the generation method of the synthetic reference frame (that is, the operation method of the dashed box 423 in FIG. 4 ) is shown in the following formula (1):

表示深度学习中的局部卷积操作，K_i表示所述第一视频帧中所有像素的卷积核系数的集合，即包括为每一个像素点单独生成卷积核系数，⊙表示像素级别的点乘操作，用于将对于两个输入帧进行局部卷积得到的结果的像素级别加权相加，其权值矩阵为M_i。Among them, I _i represents the reconstructed frame of the input two frames of video,

Represents the local convolution operation in deep learning, K _i represents the set of convolution kernel coefficients of all pixels in the first video frame, that is, including generating convolution kernel coefficients for each pixel point separately, ⊙ represents the pixel-level point The multiplication operation is used to add the pixel-level weighted sum of the results obtained by local convolution of the two input frames, and its weight matrix is M _i .

Therefore, in the embodiment of the present application, the method of generating convolution kernel coefficients for each pixel, that is, the method of local convolution, has stronger expressive ability than using the same regression method for all pixels, so it can A better regression effect can be achieved, thereby helping to generate more accurate predicted frames, thereby improving video coding efficiency. Here, regression refers to the process of generating synthetic reference frames.

Memory update network 440:

enter:

的差值R_t，通道数为3，长宽和输入图片相同；(1) _t reconstructed frame It and synthetic reference frame

The difference R _t of , the number of channels is 3, and the length and width are the same as the input picture;

(2) The states h _i-1 , c _i-1 of the previous cycle of the LSTM.

Output: ConvLSTM state at the next moment, i.e. h _i , c _i , the number of channels is 32.

Network parameters: as shown in Table 4 below:

Table 4

Next, the training method of the neural network model in FIG. 4 is described.

用于模拟真实编码过程中产生的图像质量损失现象。同时，该神经网络模型还设置有一个全局长期记忆ε，其中在第t帧视频帧的预测过程中，输入的全局长期记忆可以被标记为ε_t。Exemplarily, the Vimeo 90K data set can be used to train the neural network model in this embodiment of the present application. In this data set, 89,800 video sequences can be included, and each video sequence can have 7 consecutive lossless video frames, which can be marked. is {I ₁ ,I ₂ ,...,I ₁ }. First, the dataset can be processed, such as lossy compression of each frame of each video sequence in it, resulting in a degraded sequence of video frames

It is used to simulate the image quality loss phenomenon produced in the real encoding process. At the same time, the neural network model is also set with a global long-term memory ε, in which the input global long-term memory can be marked as ε _t during the prediction process of the t-th video frame.

Here is an example of the training process:

中，选取前两帧有损帧，即

和

并将初始记忆ε₃设置成0，对第三帧进行预测。将这三者输入参考帧生成模块，计算得到第三帧的预测帧

接下来，计算预测帧

与无损第三帧I₃的损失函数，并将误差反向传播，更新网络参数。然后，计算预测帧

与有损帧

之间的预测误差，并将初始为0的长期记忆ε₃更新为ε₄，用于第四帧的预测。Step 1: Sequence of downgraded video frames

, select the first two lossy frames, namely

and

And set the initial memory ε3 to 0 to make predictions for the _third frame. Input these three into the reference frame generation module, and calculate the predicted frame of the third frame

Next, compute the predicted frame

With the loss function of the lossless third frame I ₃ , and back-propagating the error, update the network parameters. Then, calculate the predicted frame

with lossy frames

and update the long-term memory ε ₃ from 0 to ε ₄ for the prediction of the fourth frame.

与

那么这一次的输入是有损帧

与

同时，在上一次前向传播之后，长期记忆也已经被更新成了

因此将

以及

一并输入网络，进行前向传播，得到预测帧

并计算预测帧

与无损帧I_t+3的损失函数，并将误差反向传播，更新网络参数。然后，计算预测帧

与有损帧

之间的预测误差，将长期记忆更新为

Step 2: Select the next two frames of the input frame of the last network forward pass and the long-term memory after the last update as input. For example, if the input during the last loop was

and

So this time the input is a lossy frame

and

At the same time, after the last forward pass, the long-term memory has also been updated to

Therefore will

as well as

Enter the network together, carry out forward propagation, and get the predicted frame

and calculate the predicted frame

Loss function with lossless frame It ₊₃ , and back-propagates the error to update the network parameters. Then, calculate the predicted frame

with lossy frames

The prediction error between , updating the long-term memory as

Step 3: Repeat Step 2 until all frames in this video sequence are used. Since each video sequence has 7 consecutive frames in the training data, step 2 is repeated 4 times for each video sequence.

Step 4: Select other video sequences in the training data, and repeat the above three steps until the neural network is fitted.

FIG. 5 shows an example of a peak signal-to-noise ratio (PSNR) curve of the video coding solution according to the embodiment of the present application. Exemplarily, the encoding test can be performed on the FourPeople test sequence under the low delay encoding configuration of the encoder of the present application. As shown in FIG. 5 , compared with the traditional HEVC solution, the memory-augmented auto-regressive network (MAAR-Net) solution of the embodiment of the present application can achieve a BD-RATE (Bjontegaard rate) of 10.6 % performance boost. Wherein, in FIG. 5 , the abscissa is the bitrate, and the unit is Kbps, and the ordinate is the luminance Y-peak signal to noise ratio (Y-peak signal to noise ratio, YPSNR), and the unit is dB.

Therefore, in this embodiment of the present application, deep learning can be used to use the reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence and the global long-term memory of the video sequence as the input of the neural network model, Thus, the synthetic reference frame of the first video frame is output, and then the first video frame is encoded according to the synthetic reference frame. Since the synthetic reference frame has the ability to describe complex motion between video sequences, the embodiments of the present application can generate more accurate prediction frames and improve video coding efficiency.

In a video coding solution in the prior art, in the process of using the LSTM network to generate the reference frame, only a fixed number of reconstructed frames before the currently coded video frame are input into the network to generate the reference frame. Taking the input of the four frames before the currently encoded video frame as an example, each time a frame is input, the LSTM state is updated once until all the input of the four reconstructed frames is completed, that is, the state of the LSTM is updated four times, and the generated current state is finally output. The reference frame of the encoded video frame. However, in the process of generating the reference frame, only the information of the input four reconstructed frames is included, and the information contained in the remaining encoded reconstructed frames is not used. In addition, the generation process of each reference frame is independent, that is, before each reference frame is generated, the memory of the network will be reset, which hinders the transmission of long-term temporal information, so that accurate prediction frames cannot be generated.

In the embodiment of the present application, the global long-term memory is maintained since the encoding process of the video sequence, and the time-domain span of the global long-term memory may be as long as hundreds of frames. For example, from the encoding process of the first frame to the encoding process of the last frame, the global long-term memory is always dynamically updated, so as to ensure that the input of the network contains a long enough time domain span, so that the time domain information can be fully utilized to generate more Accurately predict frames and improve the efficiency of video coding.

It should be noted that, on a video playback device, the solutions provided by the embodiments of the present application may be implemented in the form of hardware chips or software codes, which are not limited in the embodiments of the present application.

An embodiment of the present application further provides an apparatus for video encoding, please refer to FIG. 6 . Exemplarily, the video encoding apparatus 600 may be a video storage and playback device. In this embodiment of the present application, the apparatus 600 may include an acquiring unit 610 , a generating unit 620 and an encoding unit 630 .

The obtaining unit 610 is configured to obtain reconstructed frames of a fixed number of second video frames before the first video frame to be encoded in the video sequence.

A generating unit 620, configured to generate a synthetic reference frame of the first video frame according to the reconstructed frame of the second video frame and the global long-term memory of the video sequence, wherein the global long-term memory is based on the video It is determined by a reconstructed frame of each video frame in a plurality of video frames preceding the first video frame in the sequence and a composite reference frame of each video frame.

The encoding unit 630 is configured to encode the first video frame according to the synthesized reference frame of the first video frame.

Among some possible implementations, it also includes:

An update unit, configured to acquire the reconstructed frame of the first video frame, and update the global long-term memory according to the difference between the reconstructed frame of the first video frame and the synthetic reference frame of the first video frame.

In some possible implementations, the obtaining unit 610 is further configured to extract feature information of the reconstructed frame of the second video frame;

The generating unit 620 is specifically configured to generate a network model from the feature information of the reconstructed frame of the second video frame and the global long-term memory input reference frame, and obtain a synthetic reference frame of the first video frame, wherein, The reference frame generation network model is obtained by training with a training data sample set, the training data sample set includes a plurality of video sequence samples, each video sequence sample includes a lossless video frame, and the lossless video frame is subjected to Video frames obtained by lossy compression.

In some possible implementations, the generating unit 620 is specifically configured to obtain the synthetic reference frame of the first video frame according to the following formula:

表示所述第一视频帧的合成参考帧，t表示所述第一视频帧的帧号，i表示所述第一视频帧之前的第二视频帧的帧号，I_i表示第i帧视频的重建帧，

表示深度学习中的局部卷积操作，K_i表示所述第一视频帧中所有像素的卷积核系数的集合，⊙表示像素级别的点乘操作，用于将对于输入帧进行局部卷积得到的结果的像素级别加权相加，其权值矩阵为M_i，ε_t表示对所述第一视频帧进行编码时的所述全局长期记忆，γ_t表示所述第二视频帧的重建帧的特征信息，t，i为正整数，且t＞2。in,

Represents the synthetic reference frame of the first video frame, t represents the frame number of the first video frame, i represents the frame number of the second video frame before the first video frame, I _i represents the ith video frame number. reconstructed frame,

represents the local convolution operation in deep learning, K _i represents the set of convolution kernel coefficients of all pixels in the first video frame, and ⊙ represents the pixel-level dot product operation, which is used to obtain the result obtained by performing local convolution on the input frame. _pixel -level _{weighted addition} of the results of Feature information, t, i are positive integers, and t>2.

In some possible implementations, the second video frame includes the first two video frames of the first video frame, where the first video frame is the third video frame in the video sequence, or The video frame after the third video frame.

In some possible implementations, when the first video frame is the third video frame in the video sequence, the global long-term memory is 0.

In some possible implementations, the encoding unit 630 is specifically configured to obtain a reference frame list of the first video frame, where the reference frame list includes reconstructed frames of at least two video frames that have been encoded; refer to The reconstructed frame corresponding to the frame number with the largest difference between the frame numbers of the first video frame in the frame list is removed, and the synthesized reference frame of the first video frame is added to the removed frame number in the reference frame list. the position of the reconstructed frame; encode the first video frame according to the reference frame list.

FIG. 7 is a schematic diagram of a hardware structure of an apparatus 700 for video encoding according to an embodiment of the present application. The apparatus 700 shown in FIG. 7 can be regarded as a kind of computer equipment, and the apparatus 700 can be used as an implementation manner of the apparatus for video coding according to the embodiment of the present application, and can also be used as a kind of the video coding method according to the embodiment of the present application. In an implementation manner, the apparatus 700 includes a processor 701 , a memory 702 , an input/output interface 703 and a bus 705 , and may also include a communication interface 704 . The processor 701 , the memory 702 , the input/output interface 703 and the communication interface 704 are connected to each other through the bus 705 for communication.

The processor 701 may use a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to Implement the functions required to be performed by the modules in the apparatus for processing media data according to the embodiment of the present application, or execute the method for processing media data according to the method embodiment of the present application. The processor 701 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 701 or an instruction in the form of software. The above-mentioned processor 701 may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and, in combination with its hardware, completes the functions required to be performed by the modules included in the apparatus for processing media data of the embodiments of the present application, or executes the functions of the method embodiments of the present application. A method for processing media data.

The memory 702 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 702 may store operating systems as well as other applications. When the functions to be performed by the modules included in the apparatus for processing media data according to the embodiment of the present application are implemented through software or firmware, or when the method for processing media data according to the method embodiment of the present application is executed, the method provided by the embodiment of the present application is used to realize the The program code of the technical solution is stored in the memory 702, and the processor 701 executes the operations required by the modules included in the apparatus for processing media data, or executes the method for processing media data provided by the method embodiments of the present application.

The input/output interface 703 is used for receiving input data and information, and outputting data such as operation results.

The communication interface 704 uses a transceiving device such as, but not limited to, a transceiver to enable communication between the device 700 and other devices or a communication network. It can be used as an acquisition module or a sending module in the processing device.

Bus 705 may include a pathway for communicating information between various components of device 700 (eg, processor 701 , memory 702 , input/output interface 703 , and communication interface 704 ).

It should be noted that although the apparatus 700 shown in FIG. 7 only shows the processor 701, the memory 702, the input/output interface 703, the communication interface 704 and the bus 705, in the specific implementation process, those skilled in the art should understand that, The apparatus 700 also includes other devices necessary for normal operation, for example, a display for displaying video data to be played. Meanwhile, according to specific needs, those skilled in the art should understand that the apparatus 700 may further include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the apparatus 700 may only include the necessary devices for implementing the embodiments of the present application, and does not necessarily include all the devices shown in FIG. 7 .

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium includes a computer program, which, when executed on a computer, causes the computer to execute the method provided by the above method embodiments.

The embodiments of the present application also provide a computer program product containing instructions, when the computer program product runs on a computer, the computer is made to execute the method provided by the above method embodiments.

It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.

It should be understood that the descriptions of the first, second, etc. appearing in the embodiments of the present application are only for the purpose of illustrating and distinguishing the description objects, and have no order, nor do they represent a special limitation on the number of devices in the embodiments of the present application. It constitutes any limitation to the embodiments of the present application.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (15)

1. A method of video encoding, comprising:

acquiring a reconstruction frame of a fixed number of second video frames before a first video frame to be coded in a video sequence;

generating a composite reference frame of the first video frame from a reconstructed frame of the second video frame and a global long-term memory of the video sequence, wherein the global long-term memory is determined from a reconstructed frame of each video frame and a composite reference frame of each video frame in a plurality of video frames preceding the first video frame in the video sequence;

encoding the first video frame according to the synthesized reference frame of the first video frame.

2. The method of claim 1, further comprising:

acquiring a reconstructed frame of the first video frame;

and updating the global long-term memory according to the difference value of the reconstructed frame of the first video frame and the synthesized reference frame of the first video frame.

3. The method of claim 1 or 2, further comprising:

extracting characteristic information of a reconstructed frame of the second video frame;

wherein the generating a synthesized reference frame for the first video frame from the reconstructed frame for the second video frame and the global long-term memory for the video sequence comprises:

and generating a network model by using the characteristic information of the reconstructed frame of the second video frame and the global long-term memory input reference frame, and acquiring a synthesized reference frame of the first video frame, wherein the network model for generating the reference frame is obtained by training by using a training data sample set, the training data sample set comprises a plurality of video sequence samples, each video sequence sample comprises a lossless video frame, and the video frame is acquired by performing lossy compression on the lossless video frame.

4. The method of claim 3, wherein generating a network model from the feature information of the reconstructed frame of the second video frame and the global long-term memory input reference frame, and obtaining the synthesized reference frame of the first video frame comprises:

obtaining a synthesized reference frame of the first video frame according to the following formula:

wherein,

a composite reference frame representing the first video frame, t representing a frame number of the first video frame, I representing a frame number of a second video frame preceding the first video frame, I_iA reconstructed frame representing the video of the ith frame,

representing local convolution operations in deep learning, K_iA set of convolution kernel coefficients indicating all pixels in the first video frame, a dot product operation indicating a pixel level for performing a weighted addition of pixel levels obtained by performing a partial convolution on the input frame, and a weight matrix of M_i，ε_tRepresenting said global long term memory, γ, when encoding said first video frame_tAnd the characteristic information of the reconstructed frame of the second video frame is represented, t, i is a positive integer, and t is greater than 2.

5. The method according to any of claims 1-4, wherein the second video frame comprises the first two video frames of the first video frame, and wherein the first video frame is a third video frame in the video sequence or a video frame subsequent to the third video frame.

6. The method of claim 5, wherein the global long term memory is 0 if the first video frame is a third video frame in the video sequence.

7. The method according to any of claims 1-6, wherein said encoding said first video frame based on a synthesized reference frame of said first video frame comprises:

acquiring a reference frame list of the first video frame, wherein the reference frame list comprises reconstructed frames of at least two video frames which are encoded;

removing a reconstructed frame corresponding to a frame number with the largest difference between the frame numbers of the first video frame in a reference frame list, and adding a synthesized reference frame of the first video frame to the position of the removed reconstructed frame in the reference frame list;

encoding the first video frame according to the reference frame list.

8. An apparatus for video encoding, comprising:

the device comprises an acquisition unit, a coding unit and a decoding unit, wherein the acquisition unit is used for acquiring the reconstructed frames of a fixed number of second video frames before a first video frame to be coded in a video sequence;

a generating unit, configured to generate a synthesized reference frame of the first video frame according to a reconstructed frame of the second video frame and a global long-term memory of the video sequence, wherein the global long-term memory is determined according to a reconstructed frame of each video frame and a synthesized reference frame of each video frame in a plurality of video frames before the first video frame in the video sequence;

and the coding unit is used for coding the first video frame according to the synthesized reference frame of the first video frame.

9. The apparatus of claim 8, further comprising:

and the updating unit is used for acquiring the reconstructed frame of the first video frame and updating the global long-term memory according to the difference value between the reconstructed frame of the first video frame and the synthesized reference frame of the first video frame.

10. The apparatus according to claim 8 or 9, wherein the obtaining unit is further configured to:

extracting characteristic information of a reconstructed frame of the second video frame;

wherein the generating unit is specifically configured to:

and generating a network model by using the characteristic information of the reconstructed frame of the second video frame and the global long-term memory input reference frame, and acquiring a synthesized reference frame of the first video frame, wherein the network model for generating the reference frame is obtained by training by using a training data sample set, the training data sample set comprises a plurality of video sequence samples, each video sequence sample comprises a lossless video frame, and the video frame is acquired by performing lossy compression on the lossless video frame.

11. The apparatus according to claim 10, wherein the generating unit is specifically configured to:

obtaining a synthesized reference frame of the first video frame according to the following formula:

wherein,

a composite reference frame representing the first video frame, t representing a frame number of the first video frame, I representing a frame number of a second video frame preceding the first video frame, I_iA reconstructed frame representing the video of the ith frame,

representing local convolution operations in deep learning, K_iA set of convolution kernel coefficients representing all pixels in the first video frame, a point multiplication operation indicating a pixel level for a pairPerforming pixel-level weighted addition of the results obtained by partial convolution on the input frame with a weight matrix of M_i，ε_tRepresenting said global long term memory, γ, when encoding said first video frame_tAnd the characteristic information of the reconstructed frame of the second video frame is represented, t, i is a positive integer, and t is greater than 2.

12. The apparatus according to any of claims 8-11, wherein the second video frame comprises the first two video frames of the first video frame, and wherein the first video frame is a third video frame in the video sequence or a video frame subsequent to the third video frame.

13. The apparatus of claim 12, wherein the global long term memory is 0 if the first video frame is a third video frame in the video sequence.

14. The apparatus according to any of claims 8-13, wherein the encoding unit is specifically configured to:

acquiring a reference frame list of the first video frame, wherein the reference frame list comprises reconstructed frames of at least two video frames which are encoded;

removing a reconstructed frame corresponding to a frame number with the largest difference between the frame numbers of the first video frame in a reference frame list, and adding a synthesized reference frame of the first video frame to the position of the removed reconstructed frame in the reference frame list;

encoding the first video frame according to the reference frame list.

15. An apparatus of video encoding, comprising: a processor and a memory, wherein the memory is configured to store instructions and the processor is configured to execute the instructions, and when the processor executes the instructions stored by the memory, the apparatus for video encoding is configured to perform the method of any of claims 1-7.

Images