WO2020177108A1 - 一种视频帧插值方法、装置及设备 - Google Patents

一种视频帧插值方法、装置及设备 Download PDF

Info

Publication number
WO2020177108A1
WO2020177108A1 PCT/CN2019/077245 CN2019077245W WO2020177108A1 WO 2020177108 A1 WO2020177108 A1 WO 2020177108A1 CN 2019077245 W CN2019077245 W CN 2019077245W WO 2020177108 A1 WO2020177108 A1 WO 2020177108A1
Authority
WO
WIPO (PCT)
Prior art keywords
pyramid
layer
frame
deep learning
learning model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/077245
Other languages
English (en)
French (fr)
Inventor
王荣刚
张浩贤
王振宇
高文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University Shenzhen Graduate School
Original Assignee
Peking University Shenzhen Graduate School
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University Shenzhen Graduate School filed Critical Peking University Shenzhen Graduate School
Priority to US16/626,409 priority Critical patent/US11354541B2/en
Publication of WO2020177108A1 publication Critical patent/WO2020177108A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G06F18/2137—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/18—Image warping, e.g. rearranging pixels individually
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of computer technology, and in particular to a video frame interpolation method, device and equipment.
  • Video frame interpolation technology (super frame rate technology) and related applications are developing rapidly, with a wide range of application scenarios, such as virtual viewpoint synthesis, video frame rate up-conversion, 4K video conversion, and slow-motion video conversion. Since these applications all need to generate video intermediate frames that do not exist before, how to make the intermediate frames more realistic and reasonable is a key technology in real applications.
  • the embodiments of the present application provide a video frame interpolation method, device, and equipment for improvement.
  • the video frame training set including an even number of continuous video frames and a first key frame, wherein the first key frame is an intermediate frame of the even number of continuous video frames;
  • the pyramid deep learning model including a plurality of convolutional neural network layers, each of the convolutional neural network layers is used to generate intermediate frames with different resolutions;
  • the multiple video frames to be processed are input to the modified pyramid deep learning model to generate intermediate frames of the multiple video frames.
  • the video frame training set acquisition module is used to acquire a video frame training set.
  • the video frame training set includes an even number of continuous video frames and a first key frame, wherein the first key frame is the image of the even number of continuous video frames Intermediate frame
  • a pyramid deep learning model building module configured to build a pyramid deep learning model, the pyramid deep learning model including a plurality of convolutional neural network layers, and each of the convolutional neural network layers is used to generate intermediate frames with different resolutions;
  • the second key frame generation module is configured to input the even number of continuous video frames into the pyramid deep learning model to generate a second key frame
  • a pyramid deep learning model modification module configured to modify the pyramid deep learning model according to the second key frame and the first key frame to generate a modified pyramid deep learning model
  • the intermediate frame generation module is used to input multiple to-be-processed video frames into the modified pyramid deep learning model to generate intermediate frames of the multiple video frames.
  • At least one processor and,
  • a memory communicatively connected with the at least one processor; wherein,
  • the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can:
  • the video frame training set including an even number of continuous video frames and a first key frame, wherein the first key frame is an intermediate frame of the even number of continuous video frames;
  • the pyramid deep learning model including a plurality of convolutional neural network layers, each of the convolutional neural network layers is used to generate intermediate frames with different resolutions;
  • the multiple video frames to be processed are input to the modified pyramid deep learning model to generate intermediate frames of the multiple video frames.
  • the present invention adopts a pyramid deep learning model to synthesize low-resolution video intermediate frames from low-resolution video frames. Then, each layer of the pyramid increases the resolution by estimating the residual motion information between the video frames. Correct the motion information obtained last time in the pyramid, and finally synthesize the corresponding resolution intermediate frame.
  • the present invention adopts the pyramid refinement strategy to effectively estimate the motion information and the occlusion area from coarse to fine, while fully mining the space-time domain between multiple frames Information to improve the quality of the intermediate frame.
  • FIG. 1 is a schematic flowchart of a video frame interpolation method provided by an embodiment of this specification
  • FIG. 2 is a schematic diagram of the structure of the pyramid deep learning model in Embodiment 1 of this specification;
  • Figure 3 is a schematic diagram of the optical flow from the intermediate frames of each layer of the pyramid to each video frame
  • FIG. 4 is a schematic diagram of the structure of the pyramid deep learning model in the second embodiment of this specification.
  • FIG. 5 is a schematic structural diagram of a video frame interpolation device corresponding to FIG. 1 provided by an embodiment of this specification;
  • FIG. 6 is a schematic structural diagram of a video frame interpolation device corresponding to FIG. 1 provided by an embodiment of the specification.
  • FIG. 1 is a schematic flowchart of a video frame interpolation method provided by an embodiment of this specification. From a program perspective, the execution body of the process can be a program or an application client loaded on an application server.
  • the process can include the following steps:
  • Step 101 Obtain a video frame training set.
  • the video frame training set includes an even number of continuous video frames and a first key frame, wherein the first key frame is an intermediate frame of the even number of continuous video frames.
  • the key frame is the middle frame of multiple video frames.
  • first”, “second” and so on in this article are only used to distinguish, and have no practical meaning.
  • Each set of video frame data is a video frame training set.
  • the video frame training set includes an even number of video frames, at least 2 frames, and 4 frames or more are better.
  • Continuous refers to continuous in time. There are 4 video frames in the video frame set. From the time sequence, they are marked as video frame 1, video frame 2, video frame 3 and video frame 4, and the first key frame is located in the video Between frame 2 and video frame 3.
  • the purpose of training is to use the training model to interpolate video frame 1, video frame 2, video frame 3, and video frame 4 to obtain the intermediate frame (ie, the second key frame) of video frame 2 and video frame 3, and then compare it with the first key Frames are compared to modify the training model.
  • Step 102 Construct a pyramid deep learning model, the pyramid deep learning model includes a plurality of convolutional neural network layers, and each convolutional neural network layer is used to generate intermediate frames of different resolutions.
  • the pyramid deep learning model includes multiple layers, and each layer can generate intermediate frames based on multiple input video frames.
  • the difference is that the resolution of the input video frames of each layer is different, and the generated intermediate frames It also corresponds to the resolution of the input video frame, and from the low layer to the high layer, the resolution is gradually increased.
  • the resolution of the first layer is 1/4
  • the second resolution is 1/2
  • the resolution of the third layer is the original resolution.
  • each layer of the pyramid does not exist independently.
  • the low-resolution video frame parameters will be used in the calculation of the high-resolution intermediate frame.
  • the video frame data can be motion information, occlusion areas, and so on.
  • Convolutional neural network is a deep neural network with a convolutional structure.
  • the convolutional structure can reduce the amount of memory occupied by the deep network. Its three key operations, one is the local receptive field, and the other is weight sharing. The third is the pooling layer, which effectively reduces the number of network parameters and alleviates the over-fitting problem of the model.
  • Convolutional neural network is a multi-layer supervised learning neural network.
  • the convolutional layer and pool sampling layer of the hidden layer are the core modules to realize the feature extraction function of the convolutional neural network.
  • the network model uses the gradient descent method to minimize the loss function to reversely adjust the weight parameters in the network layer by layer, and improves the accuracy of the network through frequent iterative training.
  • the low hidden layer of the convolutional neural network is composed of a convolutional layer and a maximum pool sampling layer alternately, and the high-level is the hidden layer and logistic regression classifier of the fully connected layer corresponding to the traditional multi-layer perceptron.
  • the input of the first fully connected layer is the feature image obtained by feature extraction of the convolutional layer and the sub-sampling layer.
  • the last output layer is a classifier, which can use logistic regression, Softmax regression or even support vector machine to classify the input image.
  • the convolutional neural network structure includes: convolutional layer, sampling layer, full link layer. Each layer has multiple feature maps, each feature map extracts a feature of the input through a convolution filter, and each feature map has multiple neurons.
  • each neuron is extracted. Once the local feature is extracted, its positional relationship with other features is also determined.
  • the input of each neuron and the previous layer The local receptive fields are connected, and each feature extraction layer is followed by a calculation layer for local average and secondary extraction, also called feature mapping layer.
  • Each calculation layer of the network consists of multiple feature mapping planes, all of which are The weights of the neurons are equal.
  • the mapping from the input layer to the hidden layer is usually called a feature mapping, that is, the feature extraction layer is obtained through the convolutional layer, and the feature mapping layer is obtained after pooling.
  • the network structure can better adapt to the image structure
  • Weight sharing can reduce the training parameters of the network, making the neural network structure simpler and more adaptable.
  • the key point of the present invention is to use the refined frames from coarse to fine to make full use of the correlation between multiple consecutive frames, thereby improving the quality of the intermediate frames produced.
  • Step 103 Input the even number of continuous video frames into the pyramid deep learning model to generate a second key frame.
  • the parameters in the pyramid deep learning model are preset. Input an even number of continuous video frames according to the different requirements of each layer, and each layer will output an intermediate frame with a different resolution. This is The result of calculation by the pyramid deep learning model.
  • the second key frame may be one video frame or multiple video frames.
  • Step 104 Correct the pyramid depth learning model according to the second key frame and the first key frame to generate a corrected pyramid depth learning model.
  • the video intermediate frame generated by the pyramid deep learning model that is, the second key frame
  • the real video intermediate frame ie, the first key frame
  • the pyramid depth is corrected according to the difference between the two Learning the model parameters in the model until the difference between the calculated intermediate frame and the real intermediate frame generated by the modified pyramid deep learning model is within the set range. Then, it can be considered that the pyramid deep learning model has been trained.
  • Step 105 Input multiple video frames to be processed into the modified pyramid deep learning model, and generate intermediate frames of the multiple video frames.
  • the modified pyramid deep learning model is used to perform video frame interpolation to obtain video intermediate frames.
  • the method in Figure 1 adopts a pyramid deep learning model to synthesize low-resolution video intermediate frames from low-resolution video frames. Then, each layer of the pyramid increases the resolution by estimating the difference between the video frames. The residual motion information is used to correct the motion information obtained last time by the pyramid, and finally synthesize the corresponding resolution intermediate frame.
  • the present invention adopts the pyramid refinement strategy to effectively estimate the motion information and the occlusion area from coarse to fine, while fully mining multiple frames The temporal and spatial information in between improves the quality of the intermediate frame.
  • the inputting the even-numbered continuous video frames into the pyramid deep learning model specifically includes:
  • Input the processed even number of continuous video frames into the first layer of the pyramid deep learning model, and generate an optical flow set and a set of occlusion masks from intermediate frames to each video frame;
  • the pyramid deep learning model has multiple layers.
  • the first layer has a different framework from other layers.
  • the first layer is defined as the topmost layer of the pyramid.
  • the resolution of the video frame input to the first layer of the pyramid is determined according to the number of layers of the pyramid. If the pyramid is two layers, the resolution of the video frame input to the first layer is 1/2. The layer is the original resolution. If the pyramid is three layers, the resolution of the video frame input by the first layer is 1/4, and the resolution of the video frame input by the second layer is 1/2, then the input of the third layer The original resolution of the video frame dimension. Then, the preset rule may be: the resolution of the first layer is: 1/2 (a-1) , where a is the total number of layers of the pyramid, and the resolution of the k-th layer is: 1/2 (ak) . According to the above formula, it can be inferred that for the 8-layer pyramid, the resolution of the input video frame at the sixth layer is 1/4.
  • the resolution of the video frame is determined, and only the even-numbered continuous video frames need to be down-sampled to obtain the video frame of the corresponding resolution. Then obtain the collinear optical flow set and the occlusion mask set from the intermediate frame to each video frame according to the input video frame. If there are multiple video frames, then the collinear optical flow set and the occlusion mask set correspond to how many optical flows there are Matrix and occlusion mask matrix, and finally obtain the intermediate frame of the corresponding resolution according to the above content, that is, calculate the intermediate frame.
  • the optical flow diagram represents the displacement information of the pixel points between two frames, (assuming it is the optical flow diagram from the first frame to the second frame).
  • Each pixel position (x, y) of the optical flow diagram corresponds to ( ⁇ x, ⁇ y), which means the first
  • the pixel point at the pixel position in one frame passes through ( ⁇ x, ⁇ y) and moves to the pixel position (x+ ⁇ x, y+ ⁇ y) in the second frame.
  • the optical flow deformation here refers to the use of optical flow information to transform the second frame into the first frame.
  • the real intermediate frame corresponding to the resolution of the first layer can be understood as the intermediate frame of the first key frame (original resolution) reduced to the resolution corresponding to the first layer, for example, the resolution corresponding to the first layer is 1/8, then, the real intermediate frame corresponding to the resolution of the first layer is the video frame with the first key frame reduced to 1/8 resolution. Then, according to the difference between the calculated intermediate frame and the real intermediate frame, the parameters of the convolutional neural network layer in the first layer of the pyramid are modified, so as to minimize the difference between the calculated intermediate frame and the real intermediate frame.
  • the second resolution of the video frame input by the Kth layer in the pyramid deep learning model is determined according to a preset rule, and the resolution of the video frame input by the Kth layer is higher than that of the (K-1)th layer.
  • the resolution of the video frame, the resolution of the input video frame of the last layer of the pyramid deep learning model is the original resolution, and K is a natural number greater than or equal to 2;
  • each optical flow in the first optical flow set to process the video frame input by the Kth layer to generate a first deformation map set
  • the above content specifically introduces the process of generating intermediate frames by the pyramid calculation at the second layer and above.
  • the only difference is the calculation method of the optical flow set.
  • the calculation of the optical flow of the K layer uses the optical flow of the (K-1) layer to obtain the optical flow set of the K resolution through the up-sampling 2 times the resolution of the optical flow, and then the optical flow of the K resolution
  • the deformation map set and the occlusion mask set obtained by the deformation of the video frame input in the K layer, according to the optical flow set and the deformation map set, output the residual flow set, and the residual flow set is added to the optical flow set of the K resolution
  • the parameters of the first layer to the Kth layer in the pyramid deep learning model need to be corrected at the same time. This is because the K-th layer directly or indirectly uses the parameters from the first to the K-th pyramid in the process of calculating the intermediate frame, so it needs to be corrected at the same time.
  • the method provided by the present invention synthesizes low-resolution video intermediate frames by estimating the motion information and occlusion regions between down-sampled low-resolution video frames through the top layer of the pyramid. Then, while the resolution of each subsequent layer of the pyramid increases, the motion information obtained from the upper layer of the pyramid is corrected by estimating the residual motion information between the video frames, and the corresponding resolution video intermediate frame is synthesized. By analogy, accurate motion information is finally obtained, and the intermediate frames of the original resolution video are synthesized.
  • the generating the calculation intermediate frame of the Kth layer according to the optical flow set of the Kth layer and the occlusion mask set of the Kth layer specifically includes:
  • Generating the calculation intermediate frame of the Kth layer according to the second deformation map set and the occlusion mask set of the Kth layer specifically includes:
  • the calculation intermediate frame of the K-th layer is calculated using the following formula:
  • I t,k represents the calculated intermediate frame of the k-th layer
  • M k,i represents the occlusion mask relative to the i-th deformed image in the first deformed image set
  • w(I k,i ,F k,i ) Represents the i-th deformed map in the first deformed map set
  • processing the even-numbered continuous video frames according to the second resolution specifically includes:
  • the feature map of the corresponding resolution of the Kth layer and the set of input video frames of the corresponding resolution of the Kth layer are input to the Kth layer of the pyramid deep learning model.
  • a feature extractor based on three-dimensional convolution is also provided to mine the context information in the spatio-temporal domain between multiple frames.
  • the feature extractor takes multiple original resolution video frames as input and outputs a 4-dimensional feature map (multi-frame image input, output a feature map ,
  • the 4 dimensions are depth, height, width, channel).
  • the 4-dimensional feature map is split in the depth dimension, and then combined in the channel dimension to generate a 3-dimensional feature map (height, width, channel) (3D multi-frame context features).
  • This 3-dimensional feature map is transformed into feature maps of different resolutions through averaging pooling, and passed into the corresponding level in the multi-frame pyramid deep learning refinement framework to help it make better use of the temporal and spatial information between multiple frames.
  • the 3D U-net feature extractor consists of a 4-layer encoder and a 3-layer decoder.
  • the kernel size of the first aggregation layer is 1 ⁇ 2 ⁇ 2, and the purpose is not to merge the time signal too early.
  • each layer is convolved by 3 ⁇ 3 ⁇ 3 (upconvolution of the last layer of the decoder, only two steps in the spatial dimension) of two strides in each dimension, And 3 ⁇ 3 ⁇ 3 convolution composition.
  • processing the even-numbered continuous video frames according to the second resolution specifically includes:
  • the four-dimensional feature map is processed by convolution with a step size of 2 to generate a four-dimensional feature map with a resolution corresponding to the Kth layer;
  • another feature extractor based on three-dimensional convolution is also provided to mine the context information in the spatio-temporal domain between multiple frames.
  • This feature extractor uses a feature pyramid architecture (3D pyramid feature extractor), and each layer of the pyramid uses two layers of three-dimensional convolution to generate a three-dimensional feature map (3D multi-frame context feature), and then passes through a volume with a step size of 2.
  • the product reduces the resolution of the feature layer by 1/2.
  • Feature maps of different resolutions are passed into the corresponding levels in the multi-frame pyramid deep learning refinement framework to help them make better use of the spatio-temporal information between multiple frames.
  • the correcting the pyramid deep learning model according to the second key frame and the first key frame specifically includes:
  • the correction of the parameters from the first layer to the Kth layer in the pyramid deep learning model according to the calculated intermediate frame of the Kth layer and the real intermediate frame of the corresponding resolution of the Kth layer specifically includes :
  • the embodiment of this specification also provides a multi-step perception loss function to train the pyramid deep learning model.
  • the low-resolution pyramid level use the low-level convolutional layer of the pre-trained network (the low-level convolutional layer obtains low-level semantic information such as edges, lines, and corners, while the high-level convolutional layer can get more Advanced semantic information, retaining more details) to extract the features of the output result and the true value respectively, and train the network by comparing the difference between the two features through the L2 paradigm; as the resolution of the pyramid level gradually increases, the prediction is gradually used Train the higher-level convolutional layer of the network to extract the features of the output result and the true value respectively, and compare the differences between the two features through the L2 paradigm.
  • the present invention provides two feature extractors based on three-dimensional convolution (3D U-net feature extractor, 3D pyramid feature extractor) to mine the context information in the spatio-temporal domain between multiple frames.
  • 3D U-net feature extractor 3D pyramid feature extractor
  • These feature information will be embedded in the depth of the multi-frame pyramid Learning the level corresponding to the refinement framework helps it make better use of the spatio-temporal domain information between multiple frames; the present invention provides a multi-step perception loss function to train a pyramid structure deep learning network.
  • the low-level convolutional layer of the pre-trained network is used to extract the features of the output result and the true value respectively, and the difference between the two features is compared through the L2 paradigm to train the network; with the pyramid level resolution With the gradual increase of, the higher-level convolution layer of the pre-trained network is gradually used to extract the features of the output result and the true value respectively.
  • the present invention can also take multiple frames (2 or more) video frames as input, make full use of the information between multiple frames, effectively solve the problems of occlusion, motion blur, etc., and improve the robustness of synthetic intermediate frames .
  • FIG 2 is a schematic diagram of the structure of the pyramid deep learning model in the first embodiment of this specification.
  • P0, P1, and P2 are the U-net convolutional neural network corresponding to each layer of the three-layer pyramid, and conv represents convolution Floor.
  • four frames are used as input to generate an intermediate frame between the second frame and the third frame.
  • the collinear optical flow is shown in Figure 3 (left).
  • the deformed image is multiplied by the corresponding occlusion mask and then added to obtain the 1/4 resolution intermediate frame.
  • This embodiment uses a 3D U-net feature extractor, which takes 4 original resolution video frames as input and outputs a 4-dimensional feature map.
  • the 4-dimensional feature map is split in the depth dimension, and then combined in the channel dimension to generate a 3-dimensional feature map (3D multi-frame context feature).
  • This 3-dimensional feature map is transformed into feature maps of different resolutions through averaging pooling, and passed into the corresponding level in the above-mentioned 4-frame three-layer pyramid deep learning refinement framework to help it better the spatio-temporal information between multiple frames .
  • the feature extraction process is embedded in the solution process of the 4-frame three-layer pyramid deep learning refinement framework.
  • the real intermediate frames of 1/4, 1/2 and the original resolution will be used as the supervision information for each layer of the pyramid.
  • the 0th layer of the pyramid uses the conv2_2 convolutional layer of the pre-trained network VGG19 to extract the 1/4 resolution results and the true value features, and compares the difference between the two features through the L2 paradigm; the first layer of pyramid The conv3_2 convolutional layer of the pre-trained network VGG19 is used to extract the features of the 1/2 resolution result and the true value, and the difference between the two features is compared through the L2 paradigm; the second layer of the pyramid uses the conv4_3 volume of the pre-trained network VGG19 Layers are used to extract the features of the final result and the true value, and the differences between the two features are compared through the L2 paradigm.
  • a deep learning network that takes 4 frames as input and outputs the intermediate frame between the second and third frames can be finally obtained.
  • FIG 4 is a schematic diagram of the structure of the pyramid deep learning model in the second embodiment of this specification.
  • this example uses a 3D pyramid feature extractor, which uses 4
  • the original resolution video frame is used as input, and each layer of the pyramid uses two layers of three-dimensional convolution to generate a three-dimensional feature map (3D multi-frame context feature), and then the feature layer resolution is reduced by 1 through convolution with a step size of 2. /2.
  • the feature maps of different resolutions are passed into the corresponding levels in the 4-frame three-layer pyramid deep learning refinement framework to help them better the spatio-temporal information between multiple frames, as shown in Figure 4.
  • the feature extraction process is embedded in the solution process of the 4-frame three-layer pyramid deep learning refinement framework.
  • FIG. 5 is a schematic structural diagram of a video frame interpolation device corresponding to FIG. 1 provided by an embodiment of this specification. As shown in Figure 5, the device may include:
  • the video frame training set acquisition module 501 is configured to acquire a video frame training set.
  • the video frame training set includes an even number of continuous video frames and a first key frame, where the first key frame is the even number of continuous video frames Middle frame
  • the pyramid deep learning model building module 502 is used to build a pyramid deep learning model, the pyramid deep learning model includes a plurality of convolutional neural network layers, and each of the convolutional neural network layers is used to generate intermediate frames with different resolutions;
  • the second key frame generation module 503 is configured to input the even number of continuous video frames into the pyramid deep learning model to generate a second key frame;
  • a pyramid deep learning model modification module 504 configured to modify the pyramid deep learning model according to the second key frame and the first key frame, and generate a modified pyramid deep learning model
  • the intermediate frame generating module 505 is configured to input multiple video frames to be processed into the modified pyramid deep learning model to generate intermediate frames of the multiple video frames.
  • the second key frame generating module 503 may specifically include:
  • the first resolution determining unit is configured to determine the first resolution of the video frame input by the first layer in the pyramid deep learning model according to a preset rule
  • a first resolution processing unit configured to process the even-numbered continuous video frames according to the first resolution
  • the optical flow set and occlusion mask set generating unit is used to input the processed even-numbered continuous video frames to the first layer of the pyramid deep learning model to generate optical flow sets and occlusion masks from intermediate frames to each video frame Model set
  • the calculation intermediate frame generating unit of the first layer is configured to generate the calculation intermediate frame of the first layer according to the optical flow set and the occlusion mask set;
  • the first parameter correction unit is configured to correct the parameters of the first layer in the pyramid deep learning model according to the calculated intermediate frame of the first layer and the real intermediate frame of the corresponding resolution of the first layer.
  • the second key frame generating module 503 may specifically include:
  • the second resolution determining unit is configured to determine the second resolution of the video frame input by the Kth layer in the pyramid deep learning model according to a preset rule, and the resolution of the video frame input by the Kth layer is higher than that of the (K-th) 1)
  • the resolution of the video frame input by the layer, the resolution of the video frame input by the last layer of the pyramid deep learning model is the original resolution, and K is a natural number greater than or equal to 2;
  • a second resolution processing unit configured to process the even-numbered continuous video frames according to the second resolution to generate a K-th layer input video frame
  • the first optical flow set generating unit is configured to perform up-sampling of each optical flow in the optical flow set generated by the (K-1)th layer by interpolation with 2 times the resolution to generate the first optical flow set;
  • a first deformed map set generating unit configured to use each optical flow in the first optical flow set to process the video frames input by the Kth layer to generate a first deformed map set
  • a residual flow set and a K-th layer occlusion mask set generating unit configured to generate a residual flow set and a K-th layer occlusion mask set according to the first optical flow set and the first deformation map set;
  • the optical flow set generating unit of the Kth layer is configured to generate the optical flow set of the Kth layer according to the first optical flow set and the residual flow set;
  • a calculation intermediate frame generating unit configured to generate the calculation intermediate frame of the K layer according to the optical flow set of the K layer and the occlusion mask set of the K layer;
  • the second parameter correction unit is configured to correct the parameters of the first layer to the Kth layer in the pyramid deep learning model according to the calculated intermediate frame of the Kth layer and the real intermediate frame of the corresponding resolution of the Kth layer.
  • the calculation intermediate frame generating unit may specifically include:
  • the second deformation map set generation subunit is configured to generate a second deformation map set according to the corresponding optical flow deformations in the video frame input by the Kth layer and the optical flow set of the Kth layer;
  • the calculation intermediate frame generation subunit is configured to generate the calculation intermediate frame of the Kth layer according to the second deformation map set and the occlusion mask set of the Kth layer.
  • calculation intermediate frame generation subunit may be specifically used for:
  • the calculation intermediate frame of the K-th layer is calculated using the following formula:
  • I t,k represents the calculated intermediate frame of the k-th layer
  • M k,i represents the occlusion mask relative to the i-th deformed image in the first deformed image set
  • w(I k,i ,F k,i ) Represents the i-th deformed map in the first deformed map set
  • the second resolution processing unit may specifically include:
  • the first feature extraction subunit is configured to use a feature extractor to generate a four-dimensional feature map according to the even-numbered continuous video frames;
  • the three-dimensional feature map generating subunit is used to split the four-dimensional feature map in the depth dimension, and then combine it in the channel dimension to generate a three-dimensional feature map;
  • the average pooling processing subunit is used to perform average pooling processing on the three-dimensional feature map to generate a feature map with a corresponding resolution of the Kth layer;
  • the input video frame generation subunit of the resolution corresponding to the Kth layer is used to downsample the even-numbered continuous video frames to generate the input video frame of the corresponding resolution of the Kth layer;
  • the first input subunit is configured to input the feature map corresponding to the resolution of the Kth layer and the input video frame set of the corresponding resolution of the Kth layer to the Kth layer of the pyramid deep learning model.
  • the second resolution processing unit may specifically include:
  • the second feature extraction unit is configured to generate and input the even number of continuous video frames into the 3D pyramid feature extractor to generate a four-dimensional feature map;
  • a four-dimensional feature map processing unit configured to process the three-dimensional feature map using convolution with a step size of 2 to generate a four-dimensional feature map with a resolution corresponding to the Kth layer;
  • the three-dimensional feature map processing unit splits the four-dimensional feature map corresponding to the resolution of the Kth layer in the depth dimension, and then combines it in the channel dimension to generate a three-dimensional feature map of the corresponding resolution of the Kth layer;
  • a down-sampling processing unit configured to down-sample the even-numbered continuous video frames to generate a set of input video frames with a resolution corresponding to the Kth layer;
  • the second input subunit is configured to input the three-dimensional feature map corresponding to the resolution of the Kth layer and the input video frame set of the corresponding resolution of the Kth layer to the Kth layer of the pyramid deep learning model.
  • the pyramid deep learning model correction module 504 may specifically include:
  • the first characteristic parameter extraction unit is configured to extract the first characteristic parameter of the first key frame
  • the second characteristic parameter extraction unit is configured to extract the second characteristic parameter of the second key frame
  • a difference result generating unit configured to generate a difference result between the first key frame and the second key frame according to the first characteristic parameter and the second characteristic parameter;
  • the weight parameter adjustment unit is configured to adjust the weight parameter of the pyramid deep learning model according to the difference result.
  • the second parameter correction unit may specifically include:
  • the third feature parameter extraction subunit is used to extract the third feature parameter of the calculated intermediate frame by using the multi-level convolutional layer of the pre-trained network; wherein, the larger the value of K, the number of layers of the multi-level convolutional layer more;
  • a fourth feature parameter extraction subunit configured to use the multi-level convolutional layer of the pre-training network to extract the fourth feature parameter of the real intermediate frame
  • the difference result generation subunit is used to compare the difference between the third characteristic parameter and the fourth characteristic parameter using the L2 paradigm, and generate a difference result;
  • the parameter modification subunit is used to modify the parameters of the first layer to the Kth layer in the pyramid deep learning model according to the difference result.
  • the embodiment of this specification also provides a device corresponding to the above method.
  • FIG. 6 is a schematic structural diagram of a video frame interpolation device corresponding to FIG. 1 provided by an embodiment of the specification. As shown in FIG. 6, the device 600 may include:
  • At least one processor 610 and,
  • a memory 630 communicatively connected with the at least one processor; wherein,
  • the memory 630 stores instructions 620 that can be executed by the at least one processor 610, and the instructions are executed by the at least one processor 610, so that the at least one processor 610 can:
  • the video frame training set including an even number of continuous video frames and a first key frame, wherein the first key frame is an intermediate frame of the even number of continuous video frames;
  • the pyramid deep learning model including a plurality of convolutional neural network layers, each of the convolutional neural network layers is used to generate intermediate frames with different resolutions;
  • the multiple video frames to be processed are input to the modified pyramid deep learning model to generate intermediate frames of the multiple video frames.
  • a typical implementation device is a computer.
  • the computer may be, for example, a personal computer, a laptop computer, a cell phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or Any combination of these devices.
  • the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.
  • the computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.
  • processors CPU
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-permanent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types.
  • This application can also be practiced in distributed computing environments. In these distributed computing environments, remote processing devices connected through a communication network perform tasks.
  • program modules can be located in local and remote computer storage media including storage devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Signal Processing (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Television Systems (AREA)

Abstract

本说明书实施例公开了一种视频帧插值方法、装置及设备。方案包括：获取视频帧训练集合，视频帧训练集合包括偶数张连续视频帧和第一关键帧，第一关键帧为偶数张连续视频帧的中间帧；构建金字塔深度学习模型，金字塔深度学习模型包括多个卷积神经网络层，各卷积神经网络层用于生成不同分辨率的中间帧；将偶数张连续视频帧输入至金字塔深度学习模型中，生成第二关键帧；根据第二关键帧和第一关键帧修正金字塔深度学习模型；然后根据修正后的金字塔深度学习模型进行视频帧插值。本发明充分地挖掘了多帧之间的时空域信息，而且采取金字塔精进策略，有效地对运动信息，遮挡区域进行估计，极大提高了中间帧的质量。

Description

一种视频帧插值方法、装置及设备
本申请要求享有2019年3月1日提交的名称为“一种视频帧插值方法、装置及设备”的中国专利申请CN201910156565.1的优先权，其全部内容通过引用并入本文中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种视频帧插值方法、装置及设备。
背景技术
视频帧插值技术(超帧率技术)和相关应用正在快速发展，有着广泛的应用场景，诸如虚拟视点合成、视频帧率上转换、4K视频的转换以及慢动作视频转换等。由于这些应用均需要生成原来并不存在的视频中间帧，因此如何使得中间帧更真实、更合理是现实应用中的一个关键技术。
现有技术中，大部分视频中间帧的生成方法，首先计算相邻两帧之间的光流，然后根据光流对中间帧进行插值。这种方法极大地依赖于光流的质量，不然所生成的中间帧会产生明显人工痕迹。而且光流的计算需要复杂的优化过程，非常的耗时。新兴的基于深度学习的方法主要有两类：第一类方法直接利用卷积神经网络生成中间帧，以真实中间帧作为监督信息训练网络，但这类方法通常会导致模糊的结果；第二类方法利用卷积神经网络无监督地得到两帧之间的运动信息，然后根据这些运动信息对中间帧进行插值，同样以真实中间帧作为监督信息训练网络，这类方法虽然可以有效避免模糊结果，但往往依赖于精确的运动信息估计，遮挡区域估计等，但是现有技术并不存在精确估计运动信息和遮挡区域的方法。
发明内容
有鉴于此，本申请实施例提供了一种视频帧插值方法、装置及设备，用于 提高。
为解决上述技术问题，本说明书实施例是这样实现的：
本说明书实施例提供的一种视频帧插值方法，包括：
获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
本说明书实施例提供的一种视频帧插值装置，包括：
视频帧训练集合获取模块，用于获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
金字塔深度学习模型构建模块，用于构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
第二关键帧生成模块，用于将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
金字塔深度学习模型修正模块，用于根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
中间帧生成模块，用于将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
本说明书实施例提供的一种视频帧插值设备，包括：
至少一个处理器；以及，
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够：
获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
本说明书实施例采用的上述至少一个技术方案能够达到以下有益效果：
本发明采取金字塔深度学习模型，由低分辨率的视频帧合成低分辨率的视频中间帧，然后，金字塔的每一层在分辨率增加的同时，通过估计视频帧之间的残差运动信息来修正金字塔上一次得到的运动信息，最终合成对应分辨率中间帧，本发明采取金字塔精进策略，由粗到细，有效地对运动信息，遮挡区域进行估计，同时充分挖掘多帧之间的时空域信息，提高了中间帧的质量。
附图说明
此处所说明的附图用来提供对本申请的进一步理解，构成本申请的一部分，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的不当限定。在附图中：
图1为本说明书实施例提供的一种视频帧插值方法的流程示意图；
图2为本说明书实施例一的金字塔深度学习模型的结构示意图；
图3为金字塔各层中间帧到各视频帧的光流示意图；
图4为本说明书实施例二的金字塔深度学习模型的结构示意图；
图5为本说明书实施例提供的对应于图1的一种视频帧插值装置的结构示意图；
图6为本说明书实施例提供的对应于图1的一种视频帧插值设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请具体实施例及相应的附图对本申请技术方案进行清楚、完整地描述。显然，所描述的实施例仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
以下结合附图，详细说明本申请各实施例提供的技术方案。
图1为本说明书实施例提供的一种视频帧插值方法的流程示意图。从程序角度而言，流程的执行主体可以为搭载于应用服务器的程序或应用客户端。
如图1所示，该流程可以包括以下步骤：
步骤101：获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧。
在本说明书实施例中，关键帧即为多个视频帧的中间帧。另外，本文中的“第一”“第二”等只是用来进行区分，并没有实际的意义。
在进行训练的时候，需要有大量的视频帧数据。每一组视频帧数据即为视频帧训练集合，视频帧训练集合中包括偶数张视频帧，最少为2张，4张及以上更佳。连续是指时间上连续，视频帧集合中有4张视频帧，则从时间顺序上， 分别标记为视频帧1、视频帧2、视频帧3和视频帧4，而第一关键帧则位于视频帧2和视频帧3中间。训练的目的就是采用训练模型对视频帧1、视频帧2、视频帧3和视频帧4进行插值，得到视频帧2和视频帧3的中间帧(即第二关键帧)，然后与第一关键帧进行比较，从而来修正训练模型。
在进行训练视频帧数据选择时，可以在原来的视频帧数据中，选择连续的5帧视频帧，而将中间的一帧作为第一关键帧。
步骤102：构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧。
在本说明书实施例中，金字塔深度学习模型包括多层，每层均可以根据输入多个视频帧生成中间帧，不同之处在于，每层输入的视频帧的分辨率不同，而生成的中间帧也是对应于输入的视频帧的分辨率的，而且，由低层到高层，分辨率是逐步升高的，以3层金字塔为例，第一层的分辨率为1/4，第二次的分辨率为1/2，第三层的分辨率为原分辨率。另外，金字塔的每层之间也不是独立存在的，低层分辨率的视频帧参数会用于高层分辨率的中间帧的计算。视频帧数据可以是运动信息、遮挡区域等。
卷积神经网络是一种带有卷积结构的深度神经网络，卷积结构可以减少深层网络占用的内存量，其三个关键的操作，其一是局部感受野，其二是权值共享，其三是pooling层，有效的减少了网络的参数个数，缓解了模型的过拟合问题。
卷积神经网络整体架构：卷积神经网络是一种多层的监督学习神经网络，隐含层的卷积层和池采样层是实现卷积神经网络特征提取功能的核心模块。该网络模型通过采用梯度下降法最小化损失函数对网络中的权重参数逐层反向调节，通过频繁的迭代训练提高网络的精度。卷积神经网络的低隐层是由卷积层和最大池采样层交替组成，高层是全连接层对应传统多层感知器的隐含层和逻辑回归分类器。第一个全连接层的输入是由卷积层和子采样层进行特征提取得到的特征图像。最后一层输出层是一个分类器，可以采用逻辑回归，Softmax 回归甚至是支持向量机对输入图像进行分类。
卷积神经网络结构包括：卷积层，采样层，全链接层。每一层有多个特征图，每个特征图通过一种卷积滤波器提取输入的一种特征，每个特征图有多个神经元。
输入图像统计和滤波器进行卷积之后，提取该局部特征，一旦该局部特征被提取出来之后，它与其他特征的位置关系也随之确定下来了，每个神经元的输入和前一层的局部感受野相连，每个特征提取层都紧跟一个用来求局部平均与二次提取的计算层，也叫特征映射层，网络的每个计算层由多个特征映射平面组成，平面上所有的神经元的权重相等。
通常将输入层到隐藏层的映射称为一个特征映射，也就是通过卷积层得到特征提取层，经过pooling之后得到特征映射层。
卷积神经网络相比一般神经网络在图像理解中的优点在于：
1)网络结构能够较好的适应图像的结构；
2)同时进行特征提取和分类，使得特征提取有助于特征分类；
3)权值共享可以减少网络的训练参数，使得神经网络结构变得简单，适应性更强。
本发明的重点就在于，利用从粗到细的细化框架来充分利用多个连续框架之间的相关性，从而提高生产的中间帧的质量。
步骤103：将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧。
在本说明书实施例中，金字塔深度学习模型中的参数是预设好的，将偶数张连续视频帧按照每层的不同要求输入进去，每层都会对应输出一个不同分辨率的中间帧，这是经过金字塔深度学习模型计算的结果。
在本说明书实施例中，第二关键帧可以是一个视频帧，还可以是多个视频帧。
步骤104：根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学 习模型，生成修正后的金字塔深度学习模型。
在本说明书实施例中，将通过金字塔深度学习模型生成的视频中间帧，即第二关键帧，与真实的视频中间帧(即第一关键帧)进行对比，根据两者的差异去修正金字塔深度学习模型中的模型参数，直至通过修正后的金字塔深度学习模型生成的计算中间帧与真实中间帧的差异在设定的范围内。那么，则可以认为，金字塔深度学习模型已经训练完成。
步骤105：将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
在本说明书实施例中，采用修正后的金字塔深度学习模型来进行视频帧插值，得到视频中间帧。
图1中的方法，通过采取金字塔深度学习模型，由低分辨率的视频帧合成低分辨率的视频中间帧，然后，金字塔的每一层在分辨率增加的同时，通过估计视频帧之间的残差运动信息来修正金字塔上一次得到的运动信息，最终合成对应分辨率中间帧，本发明采取金字塔精进策略，由粗到细，有效地对运动信息，遮挡区域进行估计，同时充分挖掘多帧之间的时空域信息，提高了中间帧的质量。
基于图1的方法，本说明书实施例还提供了该方法的一些具体实施方式，下面进行说明。
可选的，所述将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，具体包括：
根据预设规则确定所述金字塔深度学习模型中第一层输入的视频帧的第一分辨率；
根据所述第一分辨率对所述偶数张连续视频帧进行处理；
将处理后的偶数张连续视频帧输入至所述金字塔深度学习模型中的第一层，生成中间帧到各视频帧的光流集合和遮挡掩模集合；
根据所述光流集合和所述遮挡掩模集合生成所述第一层的计算中间帧；
根据所述第一层的计算中间帧和所述第一层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层的参数。
在本说明书实施例中，金字塔深度学习模型是有多层的，第一层与其他层的框架有所不同，定义第一层为金字塔的最顶层。这里着重介绍了金字塔第一层计算中间帧的方法。
首先，需要确定金字塔第一层输入的视频帧的分辨率，这可以根据金字塔的层数来确定，如果金字塔是两层，则第一层输入的视频帧的分辨率为1/2，第二层是原分辨率，如果金字塔是三层，则第一层输入的视频帧的分辨率为1/4，则第二层输入的视频帧的分辨率为1/2，则第三层输入的视频帧维原分辨率。那么，预设规则可以是：第一层分辨率为：1/2 (a-1)，其中，a为金字塔的总层数，第k层分辨率为：1/2 (a-k)。根据上述公式可以推知，8层的金字塔，第6层输入视频帧的分辨率为1/4。
确定了视频帧的分辨率，只需将偶数张连续视频帧经过下采样处理，得到对应分辨率的视频帧即可。然后根据输入的视频帧得到中间帧到各视频帧的共线光流集合以及遮挡掩膜集合，有多个视频帧，则共线光流集合以及遮挡掩膜集合中就对应有多少个光流矩阵和遮挡掩膜矩阵，最后根据上述内容得到对应分辨率的中间帧，即计算中间帧。
光流图代表了两帧间像素点的位移信息，(假设是第一帧到第二帧的光流图)光流图每一个像素位置(x，y)对应(Δx，Δy)，表示第一帧该像素位置的像素点经过(Δx，Δy)，移动至第二帧像素位置(x+Δx，y+Δy)。这里的光流变形就是指利用光流信息，将第二帧形变成第一帧。
第一层对应分辨率的真实中间帧，可以理解为将第一关键帧(原分辨率)经过降分辨率为第一层对应的分辨率的中间帧，如，第一层对应的分辨率为1/8，则，第一层对应分辨率的真实中间帧即为，将第一关键帧降为1/8分辨率的视频帧。然后根据计算中间帧和真实中间帧的差异，来修正第一层金字塔中的卷积神经网络层的参数，从而令计算中间帧和真实中间帧的差异降低最小。
可选的，根据预设规则确定所述金字塔深度学习模型中第K层输入的视频帧的第二分辨率，第K层输入的视频帧的分辨率高于第(K-1)层输入的视频帧的分辨率，所述金字塔深度学习模型的最后一层输入的视频帧的分辨率为原分辨率，K为大于等于2的自然数；
根据所述第二分辨率对所述偶数张连续视频帧进行处理，生成第K层输入的视频帧；
将第(K-1)层生成的光流集合中的各光流进行上采样2倍分辨率的插值，生成第一光流集合；
采用所述第一光流集合中的各光流对所述第K层输入的视频帧进行处理，生成第一变形图集合；
根据所述第一光流集合和所述第一变形图集合生成残差流集合和第K层的遮挡掩膜集合；
根据所述第一光流集合和所述残差流集合生成第K层的光流集合；
根据所述第K层的光流集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧；
根据所述第K层的计算中间帧和所述第K层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层至第K层的参数。
上述内容，具体介绍了第2层及以上的金字塔计算生成中间帧的过程。与第一层金字塔的大部分相同，唯一的不同在于，光流集合的计算方式。
第K层的光流的计算采用第(K-1)层的光流经过上采样2倍分辨率的插值得到第K层分辨率的光流集合，然后通过第K层分辨率的光流将第K层输入的视频帧变形得到的变形图集合和遮挡掩膜集合，根据光流集合和变形图集合，输出残差流集合，残差流集合与第K层分辨率的光流集合相加得到第K层的光流集合，然后再根据第K层的光流集合和遮挡掩膜集合生成第K层的中间帧。
另外，需要说明的是，当第K层的计算中间帧和第K层对应分辨率的真 实中间帧存在差异时，需要同时修正金字塔深度学习模型中的第一层至第K层的参数。这是因为，第K层在计算中间帧的过程中直接或间接用到了第一层至第K层金字塔的参数，所以需要同时进行修正。
本发明提供的方法通过金字塔顶层通过估计经下采样的小分辨率视频帧之间的运动信息、遮挡区域，合成小分辨率视频中间帧。然后，金字塔接下来的每一层在分辨率增加的同时，通过估计视频帧之间的残差运动信息来修正金字塔上一层得到的运动信息，合成对应分辨率视频中间帧。以此类推，最终得到精确的运动信息，合成原始分辨率视频中间帧。
可选的，所述根据所述第K层的光流集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧，具体包括：
根据所述第K层输入的视频帧与所述第K层的光流集合中对应的光流变形，生成第二变形图集合；
根据所述第二变形图集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧，具体包括：
采用以下公式计算所述第K层的计算中间帧：
Figure PCTCN2019077245-appb-000001
其中，I t,k表示第k层的计算中间帧，M k，i表示第一变形图集合中相对于第i个变形图的遮挡掩模，w(I k，i,F k，i)表示第一变形图集合中的第i个变形图，
Figure PCTCN2019077245-appb-000002
表示矩阵点乘运算。
可选的，所述根据所述第二分辨率对所述偶数张连续视频帧进行处理，具体包括：
采用特征提取器根据所述偶数张连续视频帧生成一张四维特征图；
将所述四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成三维特征图；
对所述三维特征图进行平均池化处理，生成第K层对应分辨率的特征图；
将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧集合；
将所述第K层对应分辨率的特征图和所述第K层对应分辨率的输入视频帧集合输入至所述金字塔深度学习模型中的第K层。
本说明书实施例中，还提供了一种基于三维卷积的特征提取器来挖掘多帧间时空域的上下文信息。具体为：采用3D U-net架构(3D U-net特征提取器)，该特征提取器以多张原分辨率的视频帧作为输入，输出4维特征图(多帧图片输入，输出一个特征图，4个维度是depth，height，width，channel)。4维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成3维特征图(height，width，channel)(3D多帧上下文特征)。该3维特征图通过平均池化，变换为不同分辨率的特征图，并传入多帧金字塔深度学习精进框架中对应的层次，帮助其更好的利用多帧间的时空域信息。
3D U-net特征提取器由4层编码器和3层解码器组成。在编码器部分中，每层包含3×3×3卷积，接着是LeakyReLu(α＝0.1)和2×2×2最大池(除了第一个和最后一个层)。第一个汇集层的内核大小为1×2×2，目的是不过早合并时间信号。
在解码器部分中，每层由每个维度中两个步幅的3×3×3(解码器的最后一层的上卷积，仅在空间维度上有两个步幅)上卷积，以及3×3×3卷积组成。
上面的上卷积和卷积后面分别是LeakyReLu。因此，我们的提取器需要4×h×w×3(深度×高度×宽度×信道)体积作为输入和输出4×h×w×8特征图。通过在深度维度中分割输出并将它们堆叠在通道维度中，将输出转换为h×w×323D多帧上下文特征。如图1所示，3D多帧上下文特征被下采样通过平均汇集并在MPR框架的每个级别与第一层U-net堆叠。
可选的，所述根据所述第二分辨率对所述偶数张连续视频帧进行处理，具体包括：
将所述偶数张连续视频帧生成输入3D金字塔特征提取器中，生成一张四 维特征图；；
采用步长为2的卷积对所述四维特征图进行处理，生成第K层对应分辨率的四维特征图；
对所述第K层对应分辨率的四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成第K层对应分辨率的三维特征图；
将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧集合；
将所述第K层对应分辨率的三维特征图和所述第K层对应分辨率的输入视频帧集合输入至所述金字塔深度学习模型中的第K层。
本说明书实施例中，还提供了另一种基于三维卷积的特征提取器来挖掘多帧间时空域的上下文信息。这种特征提取器采用特征金字塔架构(3D金字塔特征提取器)，其金字塔每层均采用两层三维卷积来生成3维特征图(3D多帧上下文特征)，然后通过步长为2的卷积将特征层分辨率下降1/2。不同分辨率的特征图传入多帧金字塔深度学习精进框架中对应的层次，帮助其更好的利用多帧间的时空域信息。
可选的，所述根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，具体包括：
提取所述第一关键帧的第一特征参数；
提取所述第二关键帧的第二特征参数；
根据所述第一特征参数和所述第二特征参数生成所述第一关键帧和所述第二关键帧的差异结果；
根据所述差异结果调整所述金字塔深度学习模型的权重参数。
在本说明书实施例中，
可选的，所述根据所述第K层的计算中间帧和所述第K层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层至第K层的参数，具体包括：
采用预训练网络的多层次卷积层提取所述计算中间帧的第三特征参数；其中，K的数值越大，所述多层次卷积层的层数越多；
采用所述预训练网络的多层次卷积层提取所述真实中间帧的第四特征参数；
采用L2范式比较所述第三特征参数和所述第四特征参数的差异，生成差异结果；
根据所述差异结果修正所述金字塔深度学习模型中的第一层至第K层的参数。
本说明书实施例还提供了一种多步感知损失函数来训练金字塔深度学习模型。具体为：对于低分辨率的金字塔层次，利用预训练网络的低层次卷积层(低层次卷积层得到的是边、线、角等低级的语义信息，而高层次卷积层可以得到更高级的语义信息，保留更多细节)来分别提取输出结果和真实值的特征，并通过L2范式比较两个特征之间的差异训练网络；随着金字塔层次分辨率的逐渐提高，则逐渐用预训练网络的更高层次卷积层来分别提取输出结果和真实值的特征，并通过L2范式比较两个特征之间的差异。通过大量数据训练(通过损失函数比较生成中间帧和真实中间帧的差异，其差异反向传播至网络中，修改网络的权重参数，使得生成中间帧和真实中间帧越来越逼近)。最终可得到以多帧作为输入，输出多帧之间中间帧的深度学习网络。
需要指出的是，低分辨率的中间帧更关注运动预测的准确性，因此，采用低层次的卷积层提取局部特征有效提高运动预测的精度；对于高分辨率的中间帧，逐步关注生成中间帧的质量和细节，因此，采用高层次的卷积层进行特征参数提取。
本发明提供了两种基于三维卷积的特征提取器(3D U-net特征提取器、3D金字塔特征提取器)来挖掘多帧间时空域的上下文信息，这些特征信息将会嵌入多帧金字塔深度学习精进框架对应的层次，帮助其更好的利用多帧间的时空域信息；本发明提供了一种多步感知损失函数来训练金字塔架构深度学习网 络。对于低分辨率的金字塔层次，利用预训练网络的低层次卷积层来分别提取输出结果和真实值的特征，并通过L2范式比较两个特征之间的差异训练网络；随着金字塔层次分辨率的逐渐提高，则逐渐用预训练网络的更高层次卷积层来分别提取输出结果和真实值的特征。
本发明除了采取金字塔精进策略，还能以多帧(2帧或以上)视频帧作为输入，充分地利用多帧之间信息，有效解决遮挡、运动模糊等问题，提高合成中间帧的鲁棒性。
图2为本说明书实施例一的金字塔深度学习模型的结构示意图，如图2所示，P0、P1、P2分别是三层金字塔每一层对应得U-net卷积神经网络，conv代表卷积层。该实施例中，以四帧作为输入，生成第二帧与第三帧之间的中间帧。首先，定义四帧三层金字塔的深度学习精进框架。第0层金字塔以1/4下采样的四帧I k＝0作为输入，输出共线光流集合F k＝0以及遮挡掩模集合M k＝0。共线光流如图3(左)所示。4张1/4下采样的视频帧经光流变形，生成4张1/4分辨率的变形图。变形图与对应的遮挡掩模相乘再相加，便得到1/4分辨率中间帧。第1层金字塔以1/2下采样的四帧I k＝1、上一层的光流上采样后的结果u(F k＝0)、以及通过光流u(F k＝0)将视频帧I k＝1变形得到的变形图集合w(I k＝1,u(F k＝0))，输出残差流集合f k＝1和遮挡掩模集合M k＝1。第1层的光流集合F k＝1＝u(F k＝0)+f k＝1，经修正后的光流如图3(右)所示。该层的的1/2分辨率中间帧通过F k＝1和M k＝1同理可得。最后一层金字塔以原始的四帧I k＝2、上一层的光流上采样后的结果u(F k＝1)、以及通过光流u(F k＝1)将视频帧I k＝2变形得到的变形图集合w(I k＝2,u(F k＝1))，输出残差流集合f k＝2和遮挡掩模集合M k＝2。第一层的光流集合F k＝2＝u(F k＝1)+f k＝2，最终的中间帧通过F k＝2和M k＝2求解得到。
该实施例采用3D U-net特征提取器，该特征提取器以4张原分辨率的视频帧作为输入，输出4维特征图。4维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成3维特征图(3D多帧上下文特征)。该3维特征图通过平均池化，分别变换为不同分辨率的特征图，并传入上述4帧三层金字塔深度 学习精进框架中对应的层次，帮助其更好的多帧间的时空域信息。特征提取过程嵌入于4帧三层金字塔深度学习精进框架的求解过程中。
真实的1/4、1/2以及原始分辨率的中间帧将作为金字塔每一层的监督信息。第0层金字塔，利用预训练网络VGG19的conv2_2卷积层来分别提取1/4分辨率的结果和真实值的特征，并通过L2范式比较两个特征之间的差异；第1层金字塔，利用预训练网络VGG19的conv3_2卷积层来分别提取1/2分辨率结果和真实值的特征，并通过L2范式比较两个特征之间的差异；第2层金字塔，利用预训练网络VGG19的conv4_3卷积层来分别提取最终结果和真实值的特征，并通过L2范式比较两个特征之间的差异。通过大量数据训练，最终可得到以4帧作为输入，输出第二帧和第三帧之间中间帧的深度学习网络。
图4为本说明书实施例二的金字塔深度学习模型的结构示意图，如图4所示，该实施例与实施例一不同的是：该实例采用3D金字塔特征提取器，该特征提取器以4张原分辨率的视频帧作为输入，其金字塔每层均采用两层三维卷积来生成3维特征图(3D多帧上下文特征)，然后通过步长为2的卷积将特征层分辨率下降1/2。不同分辨率的特征图传入4帧三层金字塔深度学习精进框架中对应的层次，帮助其更好的多帧间的时空域信息，如图4所示。特征提取过程嵌入于4帧三层金字塔深度学习精进框架的求解过程中。
基于同样的思路，本说明书实施例还提供了上述方法对应的装置。图5为本说明书实施例提供的对应于图1的一种视频帧插值装置的结构示意图。如图5所示，该装置可以包括：
视频帧训练集合获取模块501，用于获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
金字塔深度学习模型构建模块502，用于构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
第二关键帧生成模块503，用于将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
金字塔深度学习模型修正模块504，用于根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
中间帧生成模块505，用于将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
可选的，第二关键帧生成模块503，具体可以包括：
第一分辨率确定单元，用于根据预设规则确定所述金字塔深度学习模型中第一层输入的视频帧的第一分辨率；
第一分辨率处理单元，用于根据所述第一分辨率对所述偶数张连续视频帧进行处理；
光流集合和遮挡掩模集合生成单元，用于将处理后的偶数张连续视频帧输入至所述金字塔深度学习模型中的第一层，生成中间帧到各视频帧的光流集合和遮挡掩模集合；
第一层的计算中间帧生成单元，用于根据所述光流集合和所述遮挡掩模集合生成所述第一层的计算中间帧；
第一参数修正单元，用于根据所述第一层的计算中间帧和所述第一层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层的参数。
可选的，第二关键帧生成模块503，具体还可以包括：
第二分辨率确定单元，用于根据预设规则确定所述金字塔深度学习模型中第K层输入的视频帧的第二分辨率，第K层输入的视频帧的分辨率高于第(K-1)层输入的视频帧的分辨率，所述金字塔深度学习模型的最后一层输入的视频帧的分辨率为原分辨率，K为大于等于2的自然数；
第二分辨率处理单元，用于根据所述第二分辨率对所述偶数张连续视频帧进行处理，生成第K层输入的视频帧；
第一光流集合生成单元，用于将第(K-1)层生成的光流集合中的各光流 进行上采样2倍分辨率的插值，生成第一光流集合；
第一变形图集合生成单元，用于采用所述第一光流集合中的各光流对所述第K层输入的视频帧进行处理，生成第一变形图集合；
残差流集合和第K层的遮挡掩膜集合生成单元，用于根据所述第一光流集合和所述第一变形图集合生成残差流集合和第K层的遮挡掩膜集合；
第K层的光流集合生成单元，用于根据所述第一光流集合和所述残差流集合生成第K层的光流集合；
计算中间帧生成单元，用于根据所述第K层的光流集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧；
第二参数修正单元，用于根据所述第K层的计算中间帧和所述第K层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层至第K层的参数。
可选的，所述计算中间帧生成单元，具体可以包括：
第二变形图集合生成子单元，用于根据所述第K层输入的视频帧与所述第K层的光流集合中对应的光流变形，生成第二变形图集合；
计算中间帧生成子单元，用于根据所述第二变形图集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧。
可选的，所述计算中间帧生成子单元，具体可以用于：
采用以下公式计算所述第K层的计算中间帧：
Figure PCTCN2019077245-appb-000003
其中，I t,k表示第k层的计算中间帧，M k，i表示第一变形图集合中相对于第i个变形图的遮挡掩模，w(I k，i,F k，i)表示第一变形图集合中的第i个变形图，
Figure PCTCN2019077245-appb-000004
表示矩阵点乘运算。
可选的，所述第二分辨率处理单元，具体可以包括：
第一特征提取子单元，用于采用特征提取器根据所述偶数张连续视频帧生 成一张四维特征图；
三维特征图生成子单元，用于将所述四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成三维特征图；
平均池化处理子单元，用于对所述三维特征图进行平均池化处理，生成第K层对应分辨率的特征图；
第K层对应分辨率的输入视频帧生成子单元，用于将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧；
第一输入子单元，用于将所述第K层对应分辨率的特征图和所述第K层对应分辨率的输入视频帧集合输入至所述金字塔深度学习模型中的第K层。
可选的，所述第二分辨率处理单元，具体可以包括：
第二特征提取单元，用于将所述偶数张连续视频帧生成输入3D金字塔特征提取器中，生成一张四维特征图；
四维特征图处理单元，用于采用步长为2的卷积对所述三维特征图进行处理，生成第K层对应分辨率的四维特征图；
三维特征图处理单元，对所述第K层对应分辨率的四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成第K层对应分辨率的三维特征图；
下采样处理单元，用于将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧集合；
第二输入子单元，用于将所述第K层对应分辨率的三维特征图和所述第K层对应分辨率的输入视频帧集合输入至所述金字塔深度学习模型中的第K层。
可选的，所述金字塔深度学习模型修正模块504，具体可以包括：
第一特征参数提取单元，用于提取所述第一关键帧的第一特征参数；
第二特征参数提取单元，用于提取所述第二关键帧的第二特征参数；
差异结果生成单元，用于根据所述第一特征参数和所述第二特征参数生成所述第一关键帧和所述第二关键帧的差异结果；
权重参数调整单元，用于根据所述差异结果调整所述金字塔深度学习模型 的权重参数。
可选的，所述第二参数修正单元，具体可以包括：
第三特征参数提取子单元，用于采用预训练网络的多层次卷积层提取所述计算中间帧的第三特征参数；其中，K的数值越大，所述多层次卷积层的层数越多；
第四特征参数提取子单元，用于采用所述预训练网络的多层次卷积层提取所述真实中间帧的第四特征参数；
差异结果生成子单元，用于采用L2范式比较所述第三特征参数和所述第四特征参数的差异，生成差异结果；
参数修正子单元，用于根据所述差异结果修正所述金字塔深度学习模型中的第一层至第K层的参数。
基于同样的思路，本说明书实施例还提供了上述方法对应的设备。
图6为本说明书实施例提供的对应于图1的一种视频帧插值设备的结构示意图。如图6所示，设备600可以包括：
至少一个处理器610；以及，
与所述至少一个处理器通信连接的存储器630；其中，
所述存储器630存储有可被所述至少一个处理器610执行的指令620，所述指令被所述至少一个处理器610执行，以使所述至少一个处理器610能够：
获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机。具体的，计算机例如可以为个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字助理、媒体播放器、导航设备、电子邮件设备、游戏控制台、平板计算机、可穿戴设备或者这些设备中的任何设备的组合。
为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本申请时可以把各单元的功能在同一个或多个软件和/或硬件中实现。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使 得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。
本申请可以在由计算机执行的计算机可执行指令的一般上下文中描述，例如程序模块。一般地，程序模块包括执行特定任务或实现特定抽象数据类型的例程、程序、对象、组件、数据结构等等。也可以在分布式计算环境中实践本申请，在这些分布式计算环境中，由通过通信网络而被连接的远程处理设备来执行任务。在分布式计算环境中，程序模块可以位于包括存储设备在内的本地和远程计算机存储介质中。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
以上所述仅为本申请的实施例而已，并不用于限制本申请。对于本领域技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原理之内所 作的任何修改、等同替换、改进等，均应包含在本申请的权利要求范围之内。

Claims (11)

  1. 一种视频帧插值方法，其特征在于，包括：
    获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
    构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
    将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
    根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
    将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
  2. 如权利要求1所述的方法，其特征在于，所述将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，具体包括：
    根据预设规则确定所述金字塔深度学习模型中第一层输入的视频帧的第一分辨率；
    根据所述第一分辨率对所述偶数张连续视频帧进行处理；
    将处理后的偶数张连续视频帧输入至所述金字塔深度学习模型中的第一层，生成中间帧到各视频帧的光流集合和遮挡掩模集合；
    根据所述光流集合和所述遮挡掩模集合生成所述第一层的计算中间帧；
    根据所述第一层的计算中间帧和所述第一层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层的参数。
  3. 如权利要求1所述的方法，其特征在于，所述将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，具体包括：
    根据预设规则确定所述金字塔深度学习模型中第K层输入的视频帧的第二分辨率，其中，第K层输入的视频帧的分辨率高于第(K-1)层输入的视频帧的分辨率，所述金字塔深度学习模型的最后一层输入的视频帧的分辨率为原 分辨率，第K层输入的视频帧的分辨率高于第(K-1)层输入的视频帧的分辨率，所述金字塔深度学习模型的最后一层输入的视频帧的分辨率为原分辨率，K为大于等于2的自然数；
    根据所述第二分辨率对所述偶数张连续视频帧进行处理，生成第K层输入的视频帧；
    将第(K-1)层生成的光流集合中的各光流进行上采样2倍分辨率的插值，生成第一光流集合；
    采用所述第一光流集合中的各光流对所述第K层输入的视频帧进行处理，生成第一变形图集合；
    根据所述第一光流集合和所述第一变形图集合生成残差流集合和第K层的遮挡掩膜集合；
    根据所述第一光流集合和所述残差流集合生成第K层的光流集合；
    根据所述第K层的光流集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧；
    根据所述第K层的计算中间帧和所述第K层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层至第K层的参数。
  4. 如权利要求3所述的方法，其特征在于，所述根据所述第K层的光流集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧，具体包括：
    根据所述第K层输入的视频帧与所述第K层的光流集合中对应的光流变形，生成第二变形图集合；
    根据所述第二变形图集合和所述第K层的遮挡掩模集合生成所述第K层的计算中间帧。
  5. 如权利要求4所述的方法，其特征在于，所述根据所述第二变形图集合和所述遮挡掩模集合生成所述第K层的计算中间帧，具体包括：
    采用以下公式计算所述第K层的计算中间帧：
    Figure PCTCN2019077245-appb-100001
    其中，I t,k表示第k层的计算中间帧，M k，i表示第一变形图集合中相对于第i个变形图的遮挡掩模，w(I k，i,F k，i)表示第一变形图集合中的第i个变形图，
    Figure PCTCN2019077245-appb-100002
    表示矩阵点乘运算。
  6. 如权利要求3所述的方法，其特征在于，所述根据所述第二分辨率对所述偶数张连续视频帧进行处理，具体包括：
    采用特征提取器根据所述偶数张连续视频帧生成一张四维特征图；
    将所述四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成三维特征图；
    对所述三维特征图进行平均池化处理，生成第K层对应分辨率的特征图；
    将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧集合；
    将所述第K层对应分辨率的特征图和所述第K层对应分辨率的输入视频帧集合输入至所述金字塔深度学习模型中的第K层。
  7. 如权利要求3所述的方法，其特征在于，所述根据所述第二分辨率对所述偶数张连续视频帧进行处理，具体包括：
    将所述偶数张连续视频帧输入3D金字塔特征提取器中，生成一张四维特征图；
    采用步长为2的卷积对所述四维特征图进行处理，生成第K层对应分辨率的四维特征图；
    对所述第K层对应分辨率的四维特征图在深度维度上进行拆分，再在通道维度上进行结合，生成第K层对应分辨率的三维特征图；
    将所述偶数张连续视频帧进行下采样，生成第K层对应分辨率的输入视频帧集合；
    将所述第K层对应分辨率的三维特征图和所述第K层对应分辨率的输入 视频帧集合输入至所述金字塔深度学习模型中的第K层。
  8. 如权利要求1所述的方法，其特征在于，所述根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，具体包括：
    提取所述第一关键帧的第一特征参数；
    提取所述第二关键帧的第二特征参数；
    根据所述第一特征参数和所述第二特征参数生成所述第一关键帧和所述第二关键帧的差异结果；
    根据所述差异结果调整所述金字塔深度学习模型的权重参数。
  9. 如权利要求3所述的方法，其特征在于，所述根据所述第K层的计算中间帧和所述第K层对应分辨率的真实中间帧修正所述金字塔深度学习模型中的第一层至第K层的参数，具体包括：
    采用预训练网络的多层次卷积层提取所述计算中间帧的第三特征参数；其中，K的数值越大，所述多层次卷积层的层数越多；
    采用所述预训练网络的多层次卷积层提取所述真实中间帧的第四特征参数；
    采用L2范式比较所述第三特征参数和所述第四特征参数的差异，生成差异结果；
    根据所述差异结果修正所述金字塔深度学习模型中的第一层至第K层的参数。
  10. 一种视频帧插值装置，其特征在于，包括：
    视频帧训练集合获取模块，用于获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
    金字塔深度学习模型构建模块，用于构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
    第二关键帧生成模块，用于将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
    金字塔深度学习模型修正模块，用于根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
    中间帧生成模块，用于将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
  11. 一种视频帧插值设备，包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够：
    获取视频帧训练集合，所述视频帧训练集合包括偶数张连续视频帧和第一关键帧，其中，所述第一关键帧为所述偶数张连续视频帧的中间帧；
    构建金字塔深度学习模型，所述金字塔深度学习模型包括多个卷积神经网络层，各所述卷积神经网络层用于生成不同分辨率的中间帧；
    将所述偶数张连续视频帧输入至所述金字塔深度学习模型中，生成第二关键帧；
    根据所述第二关键帧和所述第一关键帧修正所述金字塔深度学习模型，生成修正后的金字塔深度学习模型；
    将待处理的多张视频帧输入至所述修正后的金字塔深度学习模型，生成所述多张视频帧的中间帧。
PCT/CN2019/077245 2019-03-01 2019-03-07 一种视频帧插值方法、装置及设备 Ceased WO2020177108A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/626,409 US11354541B2 (en) 2019-03-01 2019-03-07 Method, apparatus, and device for video frame interpolation

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910156565.1 2019-03-01
CN201910156565.1A CN109905624B (zh) 2019-03-01 2019-03-01 一种视频帧插值方法、装置及设备

Publications (1)

Publication Number Publication Date
WO2020177108A1 true WO2020177108A1 (zh) 2020-09-10

Family

ID=66945996

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/077245 Ceased WO2020177108A1 (zh) 2019-03-01 2019-03-07 一种视频帧插值方法、装置及设备

Country Status (3)

Country Link
US (1) US11354541B2 (zh)
CN (1) CN109905624B (zh)
WO (1) WO2020177108A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113051430A (zh) * 2021-03-26 2021-06-29 北京达佳互联信息技术有限公司 模型训练方法、装置、电子设备、介质及产品
CN114255164A (zh) * 2020-09-22 2022-03-29 阿里巴巴集团控股有限公司 图像处理方法以及装置
US11354541B2 (en) 2019-03-01 2022-06-07 Peking University Shenzhen Graduate School Method, apparatus, and device for video frame interpolation
CN115002379A (zh) * 2022-04-25 2022-09-02 武汉大学 视频插帧方法、训练方法、装置、电子设备和存储介质
CN116193161A (zh) * 2023-02-24 2023-05-30 招联消费金融有限公司 一种视频插帧方法、装置和存储介质
US11689693B2 (en) 2020-04-30 2023-06-27 Boe Technology Group Co., Ltd. Video frame interpolation method and device, computer readable storage medium

Families Citing this family (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12425605B2 (en) * 2018-03-21 2025-09-23 Nvidia Corporation Image in-painting for irregular holes using partial convolutions
WO2020253947A1 (en) * 2019-06-18 2020-12-24 Xzimg Limited Recurrent multi-task convolutional neural network architecture
US11527005B2 (en) * 2019-07-22 2022-12-13 Samsung Electronics Co., Ltd. Video depth estimation based on temporal attention
CN110399842B (zh) * 2019-07-26 2021-09-28 北京奇艺世纪科技有限公司 视频处理方法、装置、电子设备及计算机可读存储介质
US11594006B2 (en) * 2019-08-27 2023-02-28 Nvidia Corporation Self-supervised hierarchical motion learning for video action recognition
CN112699714B (zh) * 2019-10-23 2022-06-10 魔门塔(苏州)科技有限公司 一种针对图像的致盲场景检测方法及车载终端
CN110830848B (zh) * 2019-11-04 2021-12-07 上海眼控科技股份有限公司 图像插值方法、装置、计算机设备和存储介质
US11430138B2 (en) * 2020-03-05 2022-08-30 Huawei Technologies Co., Ltd. Systems and methods for multi-frame video frame interpolation
CN111476868B (zh) * 2020-04-07 2023-06-23 哈尔滨工业大学 基于深度学习的动画生成模型训练、动画生成方法及装置
CN113556496B (zh) * 2020-04-23 2022-08-09 京东方科技集团股份有限公司 视频分辨率提升方法及装置、存储介质及电子设备
TWI791979B (zh) * 2020-04-28 2023-02-11 長庚醫療財團法人林口長庚紀念醫院 三維醫學影像的建立方法
CN111586412B (zh) * 2020-05-06 2022-07-26 华为技术有限公司 高清视频处理方法、主设备、从设备和芯片系统
US11367165B2 (en) * 2020-05-19 2022-06-21 Facebook Technologies, Llc. Neural super-sampling for real-time rendering
CN113727141B (zh) 2020-05-20 2023-05-12 富士通株式会社 视频帧的插值装置以及方法
CN112001479B (zh) * 2020-07-18 2024-04-16 北京达佳互联信息技术有限公司 基于深度学习模型的处理方法、系统及电子设备
CN112104830B (zh) * 2020-08-13 2022-09-27 北京迈格威科技有限公司 视频插帧方法、模型训练方法及对应装置
CN112215174A (zh) * 2020-10-19 2021-01-12 江苏中讯通物联网技术有限公司 一种基于计算机视觉的环卫车辆状态分析方法
CN112364911B (zh) * 2020-11-06 2022-11-11 东北石油大学 电阻率成像的反演方法、装置及存储介质
CN112422870B (zh) * 2020-11-12 2021-09-17 复旦大学 一种基于知识蒸馏的深度学习视频插帧方法
CN112584076B (zh) * 2020-12-11 2022-12-06 北京百度网讯科技有限公司 视频的插帧方法、装置及电子设备
US12327176B1 (en) * 2020-12-16 2025-06-10 Ambarella International Lp Convolutional neural network with multiple output frames
CN113014936B (zh) * 2021-02-24 2022-12-13 北京百度网讯科技有限公司 视频插帧方法、装置、设备以及存储介质
CN113112577B (zh) * 2021-04-20 2022-06-14 网易（杭州）网络有限公司 过渡帧预测模型的训练方法以及过渡帧预测方法
CN113382247B (zh) * 2021-06-09 2022-10-18 西安电子科技大学 基于间隔观测的视频压缩感知系统及方法、设备及存储介质
US12003885B2 (en) * 2021-06-14 2024-06-04 Microsoft Technology Licensing, Llc Video frame interpolation via feature pyramid flows
CN113365110B (zh) * 2021-07-14 2023-01-31 北京百度网讯科技有限公司 模型训练、视频插帧方法，装置，设备以及存储介质
CN114013334B (zh) * 2021-11-09 2023-09-15 成都工贸职业技术学院 一种新能源汽车电池物态检测处理装置
CN114170558B (zh) * 2021-12-14 2024-08-13 北京有竹居网络技术有限公司 用于视频处理的方法、系统、设备、介质和产品
CN114255169B (zh) * 2021-12-24 2026-03-06 脸萌有限公司 视频生成方法及设备
CN114387312B (zh) * 2021-12-27 2025-08-15 苏州智加科技有限公司 图像处理方法、装置、电子设备及存储介质
CN114519839A (zh) * 2022-02-21 2022-05-20 首都体育学院 一种用于运动场地活动人员的关键镜头捕捉方法
CN114581493B (zh) * 2022-03-04 2024-12-03 三星电子（中国）研发中心 双向光流估计方法和装置
US12493990B2 (en) 2022-03-17 2025-12-09 Advanced Micro Devices, Inc. Locking mechanism for image classification
US12293485B2 (en) 2022-03-17 2025-05-06 Advanced Micro Devices, Inc. Super resolution upscaling
CN114598833B (zh) * 2022-03-25 2023-02-10 西安电子科技大学 基于时空联合注意力的视频插帧方法
CN114842400B (zh) * 2022-05-23 2025-10-17 山东海量信息技术研究院 基于残差块和特征金字塔的视频帧生成方法及系统
CN115065796B (zh) * 2022-06-14 2025-07-25 三星电子（中国）研发中心 视频中间帧的生成方法和装置
CN115588153B (zh) * 2022-10-10 2024-02-02 山东财经大学 一种基于3D-DoubleU-Net的视频帧生成方法
CN115909160B (zh) * 2022-11-24 2025-10-24 湖南科技大学 深度视频插帧检测方法、装置及计算机可读存储介质
CN118118676A (zh) * 2022-11-29 2024-05-31 北京三星通信技术研究有限公司 由电子设备执行的方法、电子设备和存储介质
CN115941872B (zh) * 2023-02-08 2023-06-16 吉林大学 一种水下高速移动目标的视频插帧方法及其系统
CN116546237B (zh) * 2023-03-20 2026-03-24 南开大学 一种基于全对多场变换的视频插帧方法及系统
CN118828051A (zh) * 2023-04-18 2024-10-22 北京小米移动软件有限公司 一种视频插帧方法及装置、终端设备
CN116681631B (zh) * 2023-05-06 2026-03-24 中国科学院声学研究所南海研究站 基于对偶网络的低质量胶片图像修复增强方法及系统
US20250069319A1 (en) * 2023-08-21 2025-02-27 Advanced Micro Devices, Inc. Multi-channel disocclusion mask for interpolated frame recertification
KR20250031866A (ko) * 2023-08-29 2025-03-07 포항공과대학교 산학협력단 이미지 변화 데이터 생성 방법 및 장치

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107578377A (zh) * 2017-08-31 2018-01-12 北京飞搜科技有限公司 一种基于深度学习的超分辨率图像重建方法及系统
WO2018170393A2 (en) * 2017-03-17 2018-09-20 Portland State University Frame interpolation via adaptive convolution and adaptive separable convolution
CN108596258A (zh) * 2018-04-27 2018-09-28 南京邮电大学 一种基于卷积神经网络随机池化的图像分类方法
CN108734660A (zh) * 2018-05-25 2018-11-02 上海通途半导体科技有限公司 一种基于深度学习的图像超分辨率重建方法及装置
CN109151474A (zh) * 2018-08-23 2019-01-04 复旦大学 一种生成新视频帧的方法

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010026838A1 (ja) * 2008-09-04 2010-03-11 独立行政法人科学技術振興機構 フレームレート変換装置、並びに、その対応点推定装置、対応点推定方法及び対応点推定プログラム
US10531093B2 (en) 2015-05-25 2020-01-07 Peking University Shenzhen Graduate School Method and system for video frame interpolation based on optical flow method
US10776688B2 (en) * 2017-11-06 2020-09-15 Nvidia Corporation Multi-frame video interpolation using optical flow
EP3496100A1 (en) * 2017-12-08 2019-06-12 Nokia Technologies Oy Method and apparatus for applying video viewing behavior
CN108492326B (zh) * 2018-01-31 2021-11-23 北京大学深圳研究生院 分辨率从低到高的逐渐精细化的立体匹配方法及系统
WO2019168765A1 (en) * 2018-02-27 2019-09-06 Portland State University Context-aware synthesis for video frame interpolation
CN108805814B (zh) * 2018-06-07 2020-05-19 西安电子科技大学 基于多频段深度卷积神经网络的图像超分辨重建方法
US10977802B2 (en) * 2018-08-29 2021-04-13 Qualcomm Incorporated Motion assisted image segmentation
CN109379550B (zh) * 2018-09-12 2020-04-17 上海交通大学 基于卷积神经网络的视频帧率上变换方法及系统
CN109905624B (zh) 2019-03-01 2020-10-16 北京大学深圳研究生院 一种视频帧插值方法、装置及设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018170393A2 (en) * 2017-03-17 2018-09-20 Portland State University Frame interpolation via adaptive convolution and adaptive separable convolution
CN107578377A (zh) * 2017-08-31 2018-01-12 北京飞搜科技有限公司 一种基于深度学习的超分辨率图像重建方法及系统
CN108596258A (zh) * 2018-04-27 2018-09-28 南京邮电大学 一种基于卷积神经网络随机池化的图像分类方法
CN108734660A (zh) * 2018-05-25 2018-11-02 上海通途半导体科技有限公司 一种基于深度学习的图像超分辨率重建方法及装置
CN109151474A (zh) * 2018-08-23 2019-01-04 复旦大学 一种生成新视频帧的方法

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11354541B2 (en) 2019-03-01 2022-06-07 Peking University Shenzhen Graduate School Method, apparatus, and device for video frame interpolation
US11689693B2 (en) 2020-04-30 2023-06-27 Boe Technology Group Co., Ltd. Video frame interpolation method and device, computer readable storage medium
CN114255164A (zh) * 2020-09-22 2022-03-29 阿里巴巴集团控股有限公司 图像处理方法以及装置
CN113051430A (zh) * 2021-03-26 2021-06-29 北京达佳互联信息技术有限公司 模型训练方法、装置、电子设备、介质及产品
CN113051430B (zh) * 2021-03-26 2024-03-26 北京达佳互联信息技术有限公司 模型训练方法、装置、电子设备、介质及产品
CN115002379A (zh) * 2022-04-25 2022-09-02 武汉大学 视频插帧方法、训练方法、装置、电子设备和存储介质
CN115002379B (zh) * 2022-04-25 2023-09-26 武汉大学 视频插帧方法、训练方法、装置、电子设备和存储介质
CN116193161A (zh) * 2023-02-24 2023-05-30 招联消费金融有限公司 一种视频插帧方法、装置和存储介质

Also Published As

Publication number Publication date
US20210383169A1 (en) 2021-12-09
CN109905624A (zh) 2019-06-18
CN109905624B (zh) 2020-10-16
US11354541B2 (en) 2022-06-07

Similar Documents

Publication Publication Date Title
CN109905624B (zh) 一种视频帧插值方法、装置及设备
US20240135490A1 (en) Image processing method and device, training method of neural network, image processing method based on combined neural network model, constructing method of combined neural network model, neural network processor, and storage medium
CN109377530B (zh) 一种基于深度神经网络的双目深度估计方法
CN107154023B (zh) 基于生成对抗网络和亚像素卷积的人脸超分辨率重建方法
US11900567B2 (en) Image processing method and apparatus, computer device, and storage medium
CN110378838B (zh) 变视角图像生成方法，装置，存储介质及电子设备
WO2019120110A1 (zh) 图像重建方法及设备
CN112446835B (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质
US12561755B2 (en) System and method for image super-resolution
CN114049381A (zh) 一种融合多层语义信息的孪生交叉目标跟踪方法
CN113870335A (zh) 一种基于多尺度特征融合的单目深度估计方法
CN115187638A (zh) 一种基于光流遮罩的无监督单目深度估计方法
CN113393457B (zh) 一种结合残差密集块与位置注意力的无锚框目标检测方法
CN106157307A (zh) 一种基于多尺度cnn和连续crf的单目图像深度估计方法
CN113705358B (zh) 一种基于特征映射的多角度侧脸正面化方法
WO2023279936A1 (en) Methods and systems for high definition image manipulation with neural networks
US11823349B2 (en) Image generators with conditionally-independent pixel synthesis
CN120238709B (zh) 基于深度信息与时频优化的轨迹控制视频生成方法及装置
US20230281829A1 (en) Bidirectional optical flow estimation method and apparatus
US12506842B2 (en) Uncertainty-guided frame interpolation for video rendering
CN114694119A (zh) 基于重参数化和特征加权的交通标志检测方法及相关装置
KR20240096225A (ko) 보간 모델 학습 방법 및 보간 프레임 생성 모듈 학습 장치
CN112396554A (zh) 一种基于生成对抗网络的图像超分辨率算法
CN119516603A (zh) 一种面向边缘计算设备的改进多人姿态估计方法
CN116266336A (zh) 视频超分辨率重建方法、装置、计算设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19918469

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19918469

Country of ref document: EP

Kind code of ref document: A1