WO2022166245A1 - 一种视频帧的超分辨率重构方法 - Google Patents

一种视频帧的超分辨率重构方法 Download PDF

Info

Publication number
WO2022166245A1
WO2022166245A1 PCT/CN2021/123605 CN2021123605W WO2022166245A1 WO 2022166245 A1 WO2022166245 A1 WO 2022166245A1 CN 2021123605 W CN2021123605 W CN 2021123605W WO 2022166245 A1 WO2022166245 A1 WO 2022166245A1
Authority
WO
WIPO (PCT)
Prior art keywords
features
video frames
video
resolution
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/123605
Other languages
English (en)
French (fr)
Inventor
张登银
周超
陈灿
李俊江
刘子捷
程义
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing University of Posts and Telecommunications
Original Assignee
Nanjing University of Posts and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Posts and Telecommunications filed Critical Nanjing University of Posts and Telecommunications
Priority to US17/529,203 priority Critical patent/US11995796B2/en
Publication of WO2022166245A1 publication Critical patent/WO2022166245A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the invention relates to a super-resolution reconstruction method of video frames, which belongs to the neighborhood of video processing technology.
  • Video super-resolution technology which converts low-resolution video into high-resolution video, has also emerged and is widely used in many neighborhoods such as communications, surveillance, remote sensing, entertainment, and medical care.
  • deep neural network has become the mainstream method in the field of super-resolution due to its strong nonlinear representation ability.
  • the video sequence Compared with a single image, in addition to the spatial correlation, the video sequence also has a higher temporal correlation. How to utilize the temporal correlation of video sequences is the main research focus of video super-resolution reconstruction algorithms. However, taking advantage of the temporal correlation of video sequences often needs to address frame-to-frame occlusion, fast motion, and motion blur.
  • Existing video super-resolution algorithms are mainly solved by two steps of alignment and fusion. In the alignment step, existing methods are mainly based on motion estimation/motion compensation methods, and then motion estimation/motion compensation often leads to poor reconstruction results due to insufficient accuracy. In the fusion step, existing methods lack the penalty for low-quality features and the incentive for high-quality features in a progressive framework. Therefore, the existing video super-resolution reconstruction algorithms are still insufficient in reconstruction fidelity and robustness, which often leads to blurring and distortion of reconstructed videos, resulting in poor user experience.
  • the purpose of the present invention is to provide a super-resolution reconstruction method of video frames, so as to solve the deficiencies in the fidelity and robustness of reconstruction in the video super-resolution reconstruction algorithm in the prior art, resulting in blurred, distorted, distorted reconstructed video.
  • the present invention adopts the following technical solutions:
  • a method for super-resolution reconstruction of video frames comprising:
  • the first video frame of the first resolution and its multiple adjacent frames are input into the pre-trained super-resolution reconstruction network, and the super-resolution reconstruction network outputs the second video frame of the second resolution corresponding to the first video frame.
  • the super-resolution reconstruction network includes a feature extraction sub-network, a spatiotemporal non-local alignment sub-network, an attention progressive fusion sub-network, and an up-sampling sub-network which are sequentially connected.
  • the training method of the super-resolution reconstruction network includes the following steps:
  • the training sample set includes multiple high-resolution-low-resolution video sequence pairs; each high-resolution-low-resolution video sequence pair includes adjacent 2N+1 third video frames and 1 four fourth video frames, where N is a positive integer; the resolution of the third video frame is low resolution, and the resolution of the fourth video frame is high resolution; wherein, the fourth video frame is all A high-resolution version of the intermediate frame of the 2N+1 third video frames, where the intermediate frame of the 2N+1 third video frames is the N+1th frame;
  • the 2N+1 third video frames are input into the feature extraction sub-network to obtain the extraction features of the 2N+1 third video frames output by them;
  • the aligned extraction features of the 2N+1 third video frames and the extraction features of the intermediate frames of the 2N+1 third video frames are input into the attention progressive fusion sub-network to obtain the first fusion intermediate frame feature;
  • the first fused intermediate frame feature is dimensionally transformed by the upsampling sub-network to obtain the second fused intermediate frame feature; the dimension of the second fused intermediate frame feature is equal to the dimension of the fourth video frame;
  • the super-resolution reconstruction network is updated with the goal that the first output result converges to the fourth video frame.
  • the feature extraction sub-network is an N_fe layer convolutional network.
  • the extracted features of the 2N+1 third video frames are input into the spatiotemporal non-local alignment sub-network to obtain the aligned extracted features of the 2N+1 third video frames, including:
  • the spatiotemporal non-local alignment sub-network obtains the weighted sum features according to the extracted features of the 2N+1 third video frames, and obtains the weighted sum features according to the extracted features of the intermediate frames of the 2N+1 third video frames and the weighted features. and features to obtain the aligned extracted features of the 2N+1 third video frames.
  • the spatiotemporal non-local alignment sub-network obtains its weighted sum feature according to the extracted features of the 2N+1 third video frames, it also includes, extracting the 2N+1 third video frames.
  • the feature is dimensionally transformed to change its dimension from the first dimension to the second dimension;
  • the method further includes performing dimension transformation on the weighted sum feature to change its dimension from the second dimension to the first dimension.
  • the attention progressive fusion sub-network is formed by successively connecting N_apf attention progressive fusion blocks;
  • the N_apf attention progressive fusion blocks include a first attention progressive fusion block;
  • the first attention progressive fusion block Including 2N+2 first convolutional layers and 2N+2 second convolutional layers, and a temporal attention module;
  • the obtaining of the first fusion intermediate frame feature includes:
  • the operation for the first attention progressive fusion block includes inputting the features corresponding to the 2N+1 third video frames and the features of the intermediate frames of the 2N+1 third video frames into 2N+2 first convolutions respectively. layer, input each output result into the temporal attention module to obtain the integrated intermediate frame features; the integrated intermediate frame features are respectively combined with the features corresponding to 2N+1 third video frames and the features corresponding to 2N+1.
  • the features of the intermediate frames of the third video frames are cascaded, and the results of each cascade are input into 2N+2 second convolution layers respectively, and the obtained convolution results are respectively described as corresponding to 2N+1 third video frames
  • the features of the 2N+1 third video frames are superimposed with the features of the intermediate frames of the third video frames, and each superposition result is output;
  • the features corresponding to the 2N+1 third video frames are the 2N+1 features
  • the aligned extraction features of the third video frame, the features of the intermediate frames of the 2N+1 third video frames are the extracted features of the intermediate frames of the 2N+1 third video frames;
  • the features corresponding to 2N+1 third video frames and 2N+1 third The feature of the intermediate frame of the video frame is the each superposition result output by the last attention progressive fusion block of the first attention progressive fusion block;
  • the first attention progressive fusion block When the first attention progressive fusion block is the last of the N_apf attention progressive fusion blocks in connection order, the first attention progressive fusion block further includes a third convolution layer, the After the first attention progressive fusion block cascades the superposition results, the obtained concatenated results are input into the third convolutional layer, and the output result of the third convolutional layer is used as the attention progressive fusion sub-network output result.
  • the time domain attention module calculates and obtains the respective outputs according to the output results corresponding to the 2N+1 third video frames and the output results corresponding to the intermediate frames of the 2N+1 third video frames in the output results.
  • the similarity results of each element included in the output results corresponding to the 2N+1 third video frames and the output results corresponding to the intermediate frames of the 2N+1 third video frames are compared with the obtained similarity results respectively.
  • the output results corresponding to the 2N+1 third video frames are multiplied, and the obtained results are cascaded to obtain the integrated intermediate frame feature.
  • the training method of the super-resolution reconstruction network is divided into two stages.
  • the temporal attention module is shielded, and in the second stage, the temporal attention module is Unmasking; the switching between the first stage and the second stage depends on whether a predetermined convergence threshold is reached during the training process of the first stage.
  • the upsampling sub-network adopts a convolutional network.
  • the present invention has the following beneficial technical effects:
  • the video super-resolution reconstruction method involved in the present invention is different from the traditional optimization-based method. It has a faster reconstruction speed and is suitable for application scenarios with high real-time requirements.
  • This method has a spatiotemporal non-local alignment sub-network. Compared with other video super-resolution networks, it can integrate the global information of adjacent frames and align to each point of the intermediate frame, making full use of the information of each point in adjacent frames. information. At the same time, this sub-network not only utilizes the temporal correlation of adjacent frames, but also performs non-local operations on the intermediate frame itself, making full use of the spatial correlation. Therefore, this method can better utilize the spatial and temporal correlation to integrate and align the information of the adjacent frames, namely the intermediate frames themselves, to improve the alignment effect.
  • the present invention adopts a sub-network based on progressive fusion of attention. Compared with other video super-resolution networks, it can gradually and repeatedly fuse multiple frames, and in each fusion, the attention mechanism is used to effectively punish Low-quality features to improve the final effect of progressive fusion.
  • FIG. 1 is a schematic diagram of the principle of a method for super-resolution reconstruction of a video frame according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a feature extraction sub-network according to an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of an attention progressive fusion block according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a temporal attention module according to an embodiment of the present invention.
  • the existing video super-resolution reconstruction algorithms suffer from insufficient reconstruction fidelity and robustness, so the reconstructed videos are blurred, distorted, and have poor user experience.
  • an embodiment of the present invention provides a super-resolution reconstruction method of a video frame, and the method at least includes the following steps:
  • the first video frame of the first resolution and its multiple adjacent frames are input into the pre-trained super-resolution reconstruction network, and the super-resolution reconstruction network outputs the second video frame of the second resolution corresponding to the first video frame.
  • the super-resolution reconstruction network includes a feature extraction sub-network, a spatiotemporal non-local alignment sub-network, an attention progressive fusion sub-network, and an up-sampling sub-network which are sequentially connected.
  • the training method of the super-resolution reconstruction network may include the following steps:
  • Step A Obtain a training sample set, the training sample set includes multiple high-resolution-low-resolution video sequence pairs; each high-resolution-low-resolution video sequence pair includes adjacent 2N+1 third videos frame and a fourth video frame, wherein N is a positive integer; the resolution of the third video frame is low resolution, and the resolution of the fourth video frame is high resolution; wherein, the fourth video frame The frame is a high-resolution version of the intermediate frame of the 2N+1 third video frames, and the intermediate frame of the 2N+1 third video frames is the N+1 th frame.
  • Step B Input the 2N+1 third video frames into the feature extraction sub-network, and obtain the extraction features of the output 2N+1 third video frames;
  • the feature extraction sub-network may be: N_fe layer convolutional network. That is to say, this sub-network performs feature extraction for each input low-resolution frame separately, that is, an N_fe layer convolutional network is input, and the output is the extracted features of 2N+1 frames.
  • Step C Input the extracted features of the 2N+1 third video frames into the spatiotemporal non-local alignment sub-network to obtain the aligned extracted features of the 2N+1 third video frames.
  • the extracted features of the 2N+1 third video frames may be input into the spatiotemporal non-local alignment sub-network.
  • the spatiotemporal non-local alignment sub-network obtains its weighted sum feature according to the extracted features of the 2N+1 third video frames, and obtains its weighted sum feature according to the extracted features of the intermediate frames of the 2N+1 third video frames and the weighted feature. and features to obtain the aligned extracted features of the 2N+1 third video frames.
  • the extracted features of the 2N+1 third video frames may be extracted before the spatiotemporal non-local alignment sub-network obtains the weighted sum features of the 2N+1 third video frames according to the extracted features of the 2N+1 third video frames.
  • the extracted features of the 2N+1 third video frames may be extracted. Perform dimension transformation to change its dimension from the first dimension to the second dimension; and after obtaining its weighted sum feature, perform dimension transformation on the weighted sum feature to change its dimension from the second dimension to the first dimension dimension.
  • This sub-network takes as input the intermediate frame feature F t and each adjacent frame feature F t+n (
  • ⁇ N) are dimensionally deformed, and the dimension is changed from H ⁇ W ⁇ C to (H/r) ⁇ (W/r) ⁇ Cr 2 , where, H, W and C are the height, width and channel number of the first dimension, respectively, and r is an integer greater than 1.
  • the purpose of deformation is to reduce the memory required for network computing. Then, for each point in the deformed F t+n , the following calculation is performed:
  • i is the subscript of the point
  • ri and x j represent the points in F t and F t+n
  • f( ⁇ , ⁇ ) is a two-element function, which can be set as where ⁇ ( ⁇ ) and ⁇ ( ⁇ ) are two unary linear functions implemented by convolutional layers, which are used to calculate the correlation between two points.
  • g( ⁇ ) is a linear mapping function
  • C( ⁇ ) is a normalization function, which can be set as The softmax implementation is available in the network.
  • y i represents the aligned feature at this point. This feature is the weighted sum of all points in adjacent frames, and the weight is determined by the correlation between f(ri , x j ) and the points in the intermediate frame.
  • the w( ⁇ ) function contains a convolutional layer and deformation operations.
  • Step D Input the aligned extracted features of the 2N+1 third video frames and the extracted features of the intermediate frames of the 2N+1 third video frames into the attention progressive fusion sub-network to obtain the first fusion intermediate frame features.
  • the attention progressive fusion sub-network may be formed by sequentially connecting N_apf attention progressive fusion blocks; the N_apf attention progressive fusion blocks include a first attention progressive fusion block; the first attention The force progressive fusion block includes 2N+2 first convolutional layers and 2N+2 second convolutional layers, and a temporal attention module;
  • the operation for the first attention progressive fusion block may include inputting the features corresponding to the 2N+1 third video frames and the features of the intermediate frames of the 2N+1 third video frames into 2N+2 first
  • the convolutional layer input each output result into the time domain attention module, and obtain the integrated intermediate frame features; respectively combine the integrated intermediate frame features with the features corresponding to 2N+1 third video frames and the features corresponding to 2N
  • the features of the middle frame of +1 third video frame are cascaded, and the results of each cascade are input into 2N+2 second convolution layers respectively, and the obtained convolution results are respectively described as corresponding to 2N+1 third
  • the features of the video frame and the features of the intermediate frames of the 2N+1 third video frames are superimposed, and each superposition result is output;
  • the features corresponding to the 2N+1 third video frames are the 2N+1 features
  • the aligned extraction features of the third video frame, the features of the intermediate frames of the 2N+1 third video frames are the extracted features of the intermediate frames of the 2N+1 third video frames;
  • the features corresponding to 2N+1 third video frames and 2N+1 third The feature of the intermediate frame of the video frame is the each superposition result output by the last attention progressive fusion block of the first attention progressive fusion block;
  • the first attention progressive fusion block When the first attention progressive fusion block is the last of the N_apf attention progressive fusion blocks in connection order, the first attention progressive fusion block further includes a third convolution layer, the After the first attention progressive fusion block cascades the superposition results, the obtained concatenated results are input into the third convolutional layer, and the output result of the third convolutional layer is used as the attention progressive fusion sub-network output result.
  • the temporal attention module may, according to the output results corresponding to the 2N+1 third video frames and the output results corresponding to the intermediate frames of the 2N+1 third video frames, respectively, Calculate the similarity results of each element included in the output results corresponding to the 2N+1 third video frames and the output results corresponding to the intermediate frames of the 2N+1 third video frames in the output results, and the obtained The similarity results are respectively multiplied by the output results corresponding to the 2N+1 third video frames, and the obtained results are cascaded to obtain the integrated intermediate frame feature.
  • the similarity result can be calculated based on the sigmoid function.
  • the input of the attention progressive fusion sub-network is the aligned adjacent frame features (2N+1) and the intermediate frame features output by the spatiotemporal non-local sub-network, and the output is the fused intermediate frame features .
  • This sub-network is composed of N_apf attention progressive fusion blocks cascaded, each block is shown in Figure 3.
  • each feature (including intermediate frame features ) first separately input a convolutional layer C 1 to get and Then it is input into the temporal attention module, namely FA(), and the output of the temporal attention module is the integrated intermediate frame feature I a .
  • the temporal attention module FA() is characterized by 2N+2 adjacent frames (including the intermediate frame itself) and intermediate frame features As input, the integrated intermediate frame feature I a is output, as shown in Figure 4. Specifically, the adjacent frames are first calculated each point in the The similarity of the corresponding points:
  • the output of the last attention progressive fusion block (including 2N+2 features) is cascaded and then fed into a convolutional layer for integration as the output of this sub-network.
  • Step E performing dimension transformation on the first fused intermediate frame feature through the upsampling sub-network to obtain the second fused intermediate frame feature; the dimension of the second fused intermediate frame feature is equal to the dimension of the fourth video frame.
  • the upsampling sub-network may be based on a convolutional network.
  • the upsampling sub-network can use the convolutional layer to perform dimension transformation on the number of channels on the output of the attention progressive fusion sub-network, and the dimension increases from H ⁇ W ⁇ C to H ⁇ W ⁇ (c ⁇ k 2 ), and deformed to obtain (k ⁇ H) ⁇ (k ⁇ H) ⁇ c.
  • H, W, C are the output dimensions of the attention progressive fusion sub-network
  • k is the resolution magnification
  • c is the number of channels of the output frame.
  • Step F performing interpolation calculation on the intermediate frames of the 2N+1 third video frames to obtain an interpolation intermediate frame, the dimension of which is equal to the dimension of the fourth video frame;
  • Step G superimposing the second fusion intermediate frame feature and the interpolation intermediate frame to obtain a first output result
  • the low-resolution intermediate frame is subjected to bicubic interpolation to a desired size, and then added to the output of the upsampling sub-network to obtain the output of the network.
  • Step H The super-resolution reconstruction network is updated with the aim that the first output result converges to the fourth video frame.
  • the training method of the super-resolution reconstruction network can be divided into two stages.
  • the temporal attention module is shielded
  • the temporal attention module is The force module is unshielded; the switching between the first stage and the second stage depends on whether a predetermined convergence threshold is reached during the training process of the first stage.
  • the method for super-resolution reconstruction of video frames provided by the embodiment of the present invention has the following advantages. Compared with the traditional optimization-based method, the super-resolution reconstruction method has a faster reconstruction speed and is suitable for application scenarios with high real-time requirements.
  • this method has a spatiotemporal non-local alignment sub-network. Compared with other video super-resolution networks, it can integrate the global information of adjacent frames, align to each point of the intermediate frame, and make full use of each point of adjacent frames. point information.
  • this sub-network not only utilizes the temporal correlation of adjacent frames, but also performs non-local operations on the intermediate frame itself, making full use of the spatial correlation. Therefore, this method can better utilize the spatial and temporal correlation to integrate and align the information of the adjacent frames, namely the intermediate frames themselves, to improve the alignment effect.
  • the method adopts a sub-network based on progressive fusion of attention, which, compared with other video super-resolution networks, can progressively and repeatedly fuse multiple frames, and in each fusion, utilize the attention mechanism Effectively penalize low-quality features and improve the final effect of progressive fusion.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Multimedia (AREA)
  • Television Systems (AREA)

Abstract

一种视频帧的超分辨率重构方法，用于将低分辨率的视频恢复成高分辨率的视频。该方法包括：将第一分辨率的第一视频帧及其多个相邻帧输入预先训练的超分辨率重构网络，所述超分辨率重构网络输出第一视频帧对应的第二分辨率的第二视频帧；所述第二分辨率高于第一分辨率；其中，所述超分辨率重构网络包括依次连接的特征提取子网络、时空非局部对齐子网络、注意力渐进融合子网络、上采样子网络。该方法能够充分利用视频序列中的全局的时域与空域相关性，并在时域特征的不断融合中通过注意力机制来减少低质量特征的负面影响，提高高质量特征的贡献权重，因此能够有效提高输出的高分辨率视频的保真性和鲁棒性。

Description

一种视频帧的超分辨率重构方法 技术领域
本发明涉及一种视频帧的超分辨率重构方法，属于视频处理技术邻域。
背景技术
随着信息社会的高速发展，人们对于高分辨率视频的需求越来越高。将低分辨率视频转化为高分辨视频的视频超分辨率技术也应运而生，被广泛应用于通信、监控、遥感、娱乐和医疗等众多邻域。随着近几年来深度学习的高速发展，深度神经网络由于其强大的非线性表示能力，已成为超分辨研究领域的主流方法。
对于单幅图像的超分辨率，目前的研究主要集中于如何设计有效的网络结构来充分利用图像的内部特征，即空域相关性，以提高重构质量，例如残差网络，浓密网络，注意力机制等，都为重构质量的提升带来了可观的增益。
与单幅图像相比，视频序列除了空域相关性，还拥有较高的时域相关性。如何利用视频序列的时域相关性是视频超分辨率重构算法的主要研究热点。然而，利用视频序列的时域相关性，往往需要解决帧与帧之间的遮挡、快速运动、以及运动模糊等问题。已有的视频超分辨率算法主要通过对齐和融合两个步骤来解决。在对齐步骤中，已有的方法主要基于运动估计/运动补偿的方法，然后运动估计/运动补偿 往往由于精度不足导致较差的重构结果。在融合步骤中，已有的方法在渐进的框架中，缺少了对低质量的特征的惩罚和对高质量特征的激励。因此，已有的视频超分辨重构算法在重构的保真性、鲁棒性仍有不足，往往导致重构视频的模糊、畸变，因而用户体验较差。
发明内容
本发明的目的在于提供一种视频帧的超分辨率重构方法，以解决现有技术中视频超分辨重构算法存在重构的保真性、鲁棒性不足，导致重构视频模糊、畸变、用户体验较差的问题。
为解决上述技术问题，本发明采用如下技术方案：
一种视频帧的超分辨率重构方法，包括：
将第一分辨率的第一视频帧及其多个相邻帧输入预先训练的超分辨率重构网络，所述超分辨率重构网络输出第一视频帧对应的第二分辨率的第二视频帧；所述第二分辨率高于第一分辨率；
其中，所述超分辨率重构网络包括依次连接的特征提取子网络、时空非局部对齐子网络、注意力渐进融合子网络、上采样子网络。
进一步的，所述超分辨率重构网络的训练方法，包括以下步骤：
获取训练样本集，所述训练样本集包括多个高分辨率-低分辨率视频序列对；每一个高分辨率-低分辨率视频序列对包括相邻的2N+1个第三视频帧以及1个第四视频帧，其中N为正整数；所述第三视频帧的分辨率为低分辨率，所述第四视频帧的分辨率为高分辨率；其中，所述第四视频帧为所述2N+1个第三视频帧的中间帧的高分辨率版本，所述2N+1个第三视频帧的中间帧为第N+1帧；
将所述2N+1个第三视频帧输入特征提取子网络，得到其输出的2N+1个第三视频帧的提取特征；
将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络，获取所述2N+1个第三视频帧的已对齐的提取特征；
将所述2N+1个第三视频帧的已对齐的提取特征和所述2N+1个第三视频帧的中间帧的提取特征输入注意力渐进融合子网络，得到第一融合中间帧特征；
通过上采样子网络对第一融合中间帧特征进行维度变换，获取第二融合中间帧特征；所述第二融合中间帧特征的维度等于第四视频帧的维度；
对所述2N+1个第三视频帧的中间帧进行插值计算，获得插值中间帧，所述插值中间帧的维度等于第四视频帧的维度；
叠加所述第二融合中间帧特征和所述插值中间帧，获取第一输出结果；
以第一输出结果趋同于第四视频帧为目标，更新所述超分辨率重构网络。
进一步的，所述特征提取子网络为N_fe层卷积网络。
进一步的，将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络，获取所述2N+1个第三视频帧的已对齐的提取特征，包括：
将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络；
所述时空非局部对齐子网络根据所述2N+1个第三视频帧的提取特征，获取其加权和特征，根据所述2N+1个第三视频帧的中间帧的提取特征和所述加权和特征，获取所述2N+1个第三视频帧的已对齐的提取特征。
进一步的，在所述时空非局部对齐子网络根据所述2N+1个第三视频帧的提取特征，获取其加权和特征之前，还包括，对所述2N+1个第三视频帧的提取特征进行维度变换，将其维度由第一维度变为第二维度；
在所述获取其加权和特征之后，还包括，对所述加权和特征进行维度变换，将其维度由第二维度变为第一维度。
进一步的，所述注意力渐进融合子网络由N_apf个注意力渐进融合块依次连接构成；所述N_apf个注意力渐进融合块包括第一注意力渐进融合块；所述第一注意力渐进融合块包括2N+2个第一卷积层和2N+2个第二卷积层，以及一个时域注意力模块；
所述得到第一融合中间帧特征，包括：
针对第一注意力渐进融合块的操作包括，将对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征，分别输入2N+2个第一卷积层，将各输出结果输入时域注意力模块，获取整合后的中间帧特征；将整合后的中间帧特征分别与所述对应于2N+1个第三视频帧的特征和对应于2N+1个第三视频帧的中间帧的特征进行级联，将各级联结果分别输入2N+2个第二卷积层，将获得的卷积结果分别所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间 帧的特征进行叠加，输出各叠加结果；
当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征为所述2N+1个第三视频帧的已对齐的提取特征，2N+1个第三视频帧的中间帧的特征为所述2N+1个第三视频帧的中间帧的提取特征；
当所述第一注意力渐进融合块，按连接次序不是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征为所述第一注意力渐进融合块的上一个注意力渐进融合块输出的所述各叠加结果；
当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意力渐进融合块的最后一个时，所述第一注意力渐进融合块还包括，一个第三卷积层，所述第一注意力渐进融合块将所述各叠加结果进行级联后，将获得的级联结果输入第三卷积层，并将第三卷积层的输出结果作为所述注意力渐进融合子网络的输出结果。
进一步的，所述将各输出结果输入时域注意力模块，获取整合后的中间帧特征，包括：
时域注意力模块根据所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果，分别计算得到所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果包含的各元素的相似性结果，将得到的各相似性结果分别与所述对应于2N+1个第三视频帧的输出结果相乘，并将各个获得结果级联，得到所述整合后的中 间帧特征。
进一步的，所述相似性结果基于sigmoid函数计算获得。
进一步的，所述超分辨率重构网络的训练方法分为两阶段，在其中的第一阶段，屏蔽所述时域注意力模块，在其中的第二阶段，对所述时域注意力模块解除屏蔽；所述第一阶段和第二阶段的切换依据第一阶段的训练过程中是否达到预定的收敛阈值。
进一步的，所述上采样子网络采用卷积网络。
与现有技术相比，本发明具有如下有益技术效果：
本发明实施例提供的一种视频帧的超分辨率重构方法，由于深度神经网络具有极快的前馈速度，使得本发明所涉及的视频超分辨率重构方法与传统基于优化的方法相比有更快的重构速度，适用于实时性要求高的应用场景。
该方法中具有时空非局部的对齐子网络，与其他视频超分辨率网络相比，能够整合相邻帧的全局信息，对齐至中间帧的每一个点，充分利用了相邻帧每个点的信息。同时，此子网络不仅利用了相邻帧时域相关性，还对中间帧本身进行了非局部操作，充分利用了空域的相关性。因此，该方法能够较好的利用时空相关性，将相邻帧即中间帧本身的信息进行整合对齐，提升对齐的效果。
本发明采用了一种基于注意力渐进融合的子网络，与其他视频超分辨率网络相比，能够渐进的、多次的融合多个帧，并在每一次融合中，利用注意力机制有效惩罚低质量的特征，提高渐进融合的最终效果。
附图说明
图1为本发明实施例的一种视频帧的超分辨率重构方法的原理示意图；
图2为本发明实施例的特征提取子网络的示意图；
图3为本发明实施例的注意力渐进融合块的示意图；
图4为本发明实施例的时域注意力模块的示意图。
具体实施方式
下面结合具体实施例对本发明作进一步描述。以下实施例仅用于更加清楚地说明本发明的技术方案，而不能以此来限制本发明的保护范围。
如前所述，已有的视频超分辨重构算法村存在重构的保真性、鲁棒性不足，因而重构视频的模糊、畸变、用户体验较差的问题。
为解决上述技术问题，本发明实施例提供了一种视频帧的超分辨率重构方法，该方法至少包括如下步骤：
将第一分辨率的第一视频帧及其多个相邻帧输入预先训练的超分辨率重构网络，所述超分辨率重构网络输出第一视频帧对应的第二分辨率的第二视频帧；所述第二分辨率高于第一分辨率；
其中，所述超分辨率重构网络包括依次连接的特征提取子网络、时空非局部对齐子网络、注意力渐进融合子网络、上采样子网络。
结合图1所示，超分辨率重构网络的训练方法，可以包括以下步骤：
步骤A：获取训练样本集，所述训练样本集包括多个高分辨率- 低分辨率视频序列对；每一个高分辨率-低分辨率视频序列对包括相邻的2N+1个第三视频帧以及1个第四视频帧，其中N为正整数；所述第三视频帧的分辨率为低分辨率，所述第四视频帧的分辨率为高分辨率；其中，所述第四视频帧为所述2N+1个第三视频帧的中间帧的高分辨率版本，所述2N+1个第三视频帧的中间帧为第N+1帧。
步骤B：将所述2N+1个第三视频帧输入特征提取子网络，得到其输出的2N+1个第三视频帧的提取特征；在一个具体的实施例中，特征提取子网络可以为N_fe层卷积网络。也就是说，此子网络对每一个输入的低分辨率帧单独的进行特征提取，即输入一个N_fe层卷积网络，输出为2N+1个帧的提取特征。
步骤C：将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络，获取所述2N+1个第三视频帧的已对齐的提取特征。
在一个具体的实施例中，可以将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络。例如，在一个例子中，相邻帧个数为2N＝6，总的输入帧数为2N+1＝7，用于特征提取的卷积层个数为N_fe＝5，注意力渐进融合块的个数N_apf＝20。
所述时空非局部对齐子网络根据所述2N+1个第三视频帧的提取特，获取其加权和特征，根据所述2N+1个第三视频帧的中间帧的提取特征和所述加权和特征，获取所述2N+1个第三视频帧的已对齐的提取特征。
在一个具体的实施例中，可以在时空非局部对齐子网络根据所述2N+1个第三视频帧的提取特征，获取其加权和特征之前，对2N+1 个第三视频帧的提取特征进行维度变换，将其维度由第一维度变为第二维度；还可以在所述获取其加权和特征之后，对所述加权和特征进行维度变换，将其维度由第二维度变为第一维度。
在一个更具体的实施例中，如图2所示。此子网络将中间帧特征F t与每一个相邻帧特征F t+n(|n|≤N)(包括中间帧F t自身)作为输入，输出其已对齐的相邻帧特征。具体的，首先将所有特征张量F t+n(|n|≤N)进行维度变形，维度从H×W×C变为(H/r)×(W/r)×Cr 2，其中，H，W和C分别为第一维度的高，宽和信道数，r为一个大于1的整数。变形的目的是为了减少网络计算所需内存。然后，对变形过的F t+n中每一个点，进行如下计算：
Figure PCTCN2021123605-appb-000001
其中，i为此点的下标，r i和x j表示F t和F t+n中的点，f(·,·)为一个两元函数，可设为
Figure PCTCN2021123605-appb-000002
其中θ(·)与φ(·)为两个通过卷积层实现的一元线性函数，用于计算两点之间的关联性。g(·)为一个线性映射函数，C(·)为归一化函数，可设为
Figure PCTCN2021123605-appb-000003
在网络中可用softmax实现。y i表示此点对齐过的特征。此特征为相邻帧中所有的点的加权和，权重由f(r i,x j)计算与中间帧的点的关联性决定。最后，将y i输入一个卷积层，变形回原来的尺寸，再与x j相加进行输出，如下式所示：z i＝w(y i)+x j，
其中，w(·)函数包含了一个卷积层与变形的操作。
步骤D：将所述2N+1个第三视频帧的已对齐的提取特征和所述2N+1个第三视频帧的中间帧的提取特征输入注意力渐进融合子网络， 得到第一融合中间帧特征。
在一个具体的实施例中，注意力渐进融合子网络可以由N_apf个注意力渐进融合块依次连接构成；所述N_apf个注意力渐进融合块包括第一注意力渐进融合块；所述第一注意力渐进融合块包括2N+2个第一卷积层和2N+2个第二卷积层，以及一个时域注意力模块；
针对第一注意力渐进融合块的操作，可以包括，将对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征，分别输入2N+2个第一卷积层，将各输出结果输入时域注意力模块，获取整合后的中间帧特征；将整合后的中间帧特征分别与所述对应于2N+1个第三视频帧的特征和对应于2N+1个第三视频帧的中间帧的特征进行级联，将各级联结果分别输入2N+2个第二卷积层，将获得的卷积结果分别所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征进行叠加，输出各叠加结果；
当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征为所述2N+1个第三视频帧的已对齐的提取特征，2N+1个第三视频帧的中间帧的特征为所述2N+1个第三视频帧的中间帧的提取特征；
当所述第一注意力渐进融合块，按连接次序不是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征为所述第一注意力渐进融合块的上一个注意力渐进融合块输出的所述各叠加结果；
当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意 力渐进融合块的最后一个时，所述第一注意力渐进融合块还包括，一个第三卷积层，所述第一注意力渐进融合块将所述各叠加结果进行级联后，将获得的级联结果输入第三卷积层，并将第三卷积层的输出结果作为所述注意力渐进融合子网络的输出结果。
在一个例子中，时域注意力模块可以根据所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果，分别计算得到所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果包含的各元素的相似性结果，将得到的各相似性结果分别与所述对应于2N+1个第三视频帧的输出结果相乘，并将各个获得结果级联，得到所述整合后的中间帧特征。在一个例子中，相似性结果可以基于sigmoid函数计算获得。
在一个更具体的实施例中，注意力渐进融合子网络的输入为时空非局部子网络输出的已对齐的相邻帧特征(2N+1个)与中间帧特征，输出为融合的中间帧特征。此子网络由N_apf个注意力渐进融合块级联够成，每一个块如图3所示。在此块中，每一个特征
Figure PCTCN2021123605-appb-000004
(包括中间帧特征
Figure PCTCN2021123605-appb-000005
)先单独输入一个卷积层C 1，得到
Figure PCTCN2021123605-appb-000006
与
Figure PCTCN2021123605-appb-000007
然后将其一起输入时域注意力模块，即FA()，时域注意力模块的输出为整合过后的中间帧特征I a。将中间帧特征I a再与原特征
Figure PCTCN2021123605-appb-000008
及
Figure PCTCN2021123605-appb-000009
进行级联，再次融合，输入第二个卷积层C 2，最后与
Figure PCTCN2021123605-appb-000010
及
Figure PCTCN2021123605-appb-000011
相加进行输出得到融合过的相邻帧O t+n及O r，用于输入下一个渐进融合块，进行下一次的融合，整个过程如下式所示：
Figure PCTCN2021123605-appb-000012
Figure PCTCN2021123605-appb-000013
Figure PCTCN2021123605-appb-000014
Figure PCTCN2021123605-appb-000015
Figure PCTCN2021123605-appb-000016
其中，时域注意力模块FA()，以2N+2个相邻帧特征
Figure PCTCN2021123605-appb-000017
(包括中间帧自身)与中间帧特征
Figure PCTCN2021123605-appb-000018
作为输入，输出整合过后的中间帧特征I a，如图4所示。具体的，首先计算相邻帧
Figure PCTCN2021123605-appb-000019
中每一个点与
Figure PCTCN2021123605-appb-000020
对应的点的相似性：
Figure PCTCN2021123605-appb-000021
其中，p(·)与q(·)为两个通过卷积层实现的一元线性函数。然后，将此相似性与原本的特征
Figure PCTCN2021123605-appb-000022
逐像素相乘：
Figure PCTCN2021123605-appb-000023
其中，⊙表示逐像素相乘。最后，将所有帧
Figure PCTCN2021123605-appb-000024
级联，通过一个卷积网络进行输出，得到I a。
在注意力渐进融合子网络中，最后的一个注意力渐进融合块的输出(包括2N+2个特征)通过级联，然后输入一个卷积层进行整合，作为此子网络的输出。
步骤E：通过上采样子网络对第一融合中间帧特征进行维度变换，获取第二融合中间帧特征；所述第二融合中间帧特征的维度等于第四视频帧的维度。在一个具体的实施例中，上采样子网络可以基于卷积网络。
在一个更具体的实施例中，上采样子网络可以利用卷积层对注意力渐进融合子网络的输出在通道数上进行维度变换，维度从H×W×C增至变为H×W×(c·k 2)，进行变形得到(k·H)×(k·H)×c。其中，H，W，C为注意力渐进融合子网络的输出维度，k为分辨率放大倍数，c为输出帧的通道数，通过数个上述过程，上采样至所需的大小。
步骤F：对所述2N+1个第三视频帧的中间帧进行插值计算，获得插值中间帧，所述插值中间帧的维度等于第四视频帧的维度；
步骤G：叠加所述第二融合中间帧特征和所述插值中间帧，获取第一输出结果；
在一个具体的实施例中，对低分辨率中间帧进行双三次插值到所需大小，再与上采样子网络的输出相加，得到网络的输出。
步骤H：以第一输出结果趋同于第四视频帧为目标，更新所述超分辨率重构网络。
根据一种实施方式，超分辨率重构网络的训练方法可以分为两阶段，在其中的第一阶段，屏蔽所述时域注意力模块，在其中的第二阶段，对所述时域注意力模块解除屏蔽；所述第一阶段和第二阶段的切换依据第一阶段的训练过程中是否达到预定的收敛阈值。
下面，结合实验结果对比表，对本发明方法的效果做进一步说明。
表1本发明实施例结果与现有技术重构质量对比表
Figure PCTCN2021123605-appb-000025
Figure PCTCN2021123605-appb-000026
衡量标准：(峰值信噪比)PSNR、(结构相似性)SSIM，测试对象：10个通用测试集视频序列。
由表1可以看出，本发明方法相比于已有方法有着明显的效果提升，对比于已有的最好方法(PFNL)，本发明PSNR平均提升了0.33dB，SSIM提升了0.0019。
与现有技术相比，本发明实施例提供的一种视频帧的超分辨率重构方法具有如下优点，一方面，由于深度神经网络具有极快的前馈速度，使得本发明所涉及的视频超分辨率重构方法与传统基于优化的方法相比有更快的重构速度，适用于实时性要求高的应用场景。
其次，该方法中具有时空非局部的对齐子网络，与其他视频超分辨率网络相比，能够整合相邻帧的全局信息，对齐至中间帧的每一个点，充分利用了相邻帧每个点的信息。同时，此子网络不仅利用了相邻帧时域相关性，还对中间帧本身进行了非局部操作，充分利用了空域的相关性。因此，该方法能够较好的利用时空相关性，将相邻帧即 中间帧本身的信息进行整合对齐，提升对齐的效果。
最后，该方法采用了一种基于注意力渐进融合的子网络，与其他视频超分辨率网络相比，能够渐进的、多次的融合多个帧，并在每一次融合中，利用注意力机制有效惩罚低质量的特征，提高渐进融合的最终效果。
以上已以较佳实施例公布了本发明，然其并非用以限制本发明，凡采取等同替换或等效变换的方案所获得的技术方案，均落在本发明的保护范围内。

Claims (10)

  1. 一种视频帧的超分辨率重构方法，其特征在于，包括：
    将第一分辨率的第一视频帧及其多个相邻帧输入预先训练的超分辨率重构网络，所述超分辨率重构网络输出第一视频帧对应的第二分辨率的第二视频帧；所述第二分辨率高于第一分辨率；
    其中，所述超分辨率重构网络包括依次连接的特征提取子网络、时空非局部对齐子网络、注意力渐进融合子网络、上采样子网络。
  2. 根据权利要求1所述的方法，其特征在于，所述超分辨率重构网络的训练方法，包括以下步骤：
    获取训练样本集，所述训练样本集包括多个高分辨率-低分辨率视频序列对；每一个高分辨率-低分辨率视频序列对包括相邻的2N+1个第三视频帧以及1个第四视频帧，其中N为正整数；所述第三视频帧的分辨率为低分辨率，所述第四视频帧的分辨率为高分辨率；其中，所述第四视频帧为所述2N+1个第三视频帧的中间帧的高分辨率版本，所述2N+1个第三视频帧的中间帧为第N+1帧；
    将所述2N+1个第三视频帧输入特征提取子网络，得到其输出的2N+1个第三视频帧的提取特征；
    将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络，获取所述2N+1个第三视频帧的已对齐的提取特征；
    将所述2N+1个第三视频帧的已对齐的提取特征和所述2N+1个第三视频帧的中间帧的提取特征输入注意力渐进融合子网络，得到第 一融合中间帧特征；
    通过上采样子网络对第一融合中间帧特征进行维度变换，获取第二融合中间帧特征；所述第二融合中间帧特征的维度等于第四视频帧的维度；
    对所述2N+1个第三视频帧的中间帧进行插值计算，获得插值中间帧，所述插值中间帧的维度等于第四视频帧的维度；
    叠加所述第二融合中间帧特征和所述插值中间帧，获取第一输出结果；
    以第一输出结果趋同于第四视频帧为目标，更新所述超分辨率重构网络。
  3. 根据权利要求1所述的方法，其特征在于，所述特征提取子网络为N_fe层卷积网络。
  4. 根据权利要求2所述的方法，其特征在于，将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络，获取所述2N+1个第三视频帧的已对齐的提取特征，包括：
    将所述2N+1个第三视频帧的提取特征输入时空非局部对齐子网络；
    所述时空非局部对齐子网络根据所述2N+1个第三视频帧的提取特征，获取其加权和特征，根据所述2N+1个第三视频帧的中间帧的提取特征和所述加权和特征，获取所述2N+1个第三视频帧的已对齐的提取特征。
  5. 根据权利要求4所述的方法，其特征在于，在所述时空非局 部对齐子网络根据所述2N+1个第三视频帧的提取特征，获取其加权和特征之前，还包括，对所述2N+1个第三视频帧的提取特征进行维度变换，将其维度由第一维度变为第二维度；
    在所述获取其加权和特征之后，还包括，对所述加权和特征进行维度变换，将其维度由第二维度变为第一维度。
  6. 根据权利要求2所述的方法，其特征在于，所述注意力渐进融合子网络由N_apf个注意力渐进融合块依次连接构成；所述N_apf个注意力渐进融合块包括第一注意力渐进融合块；所述第一注意力渐进融合块包括2N+2个第一卷积层和2N+2个第二卷积层，以及一个时域注意力模块；
    所述得到第一融合中间帧特征，包括：
    针对第一注意力渐进融合块的操作包括，将对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征，分别输入2N+2个第一卷积层，将各输出结果输入时域注意力模块，获取整合后的中间帧特征；将整合后的中间帧特征分别与所述对应于2N+1个第三视频帧的特征和对应于2N+1个第三视频帧的中间帧的特征进行级联，将各级联结果分别输入2N+2个第二卷积层，将获得的卷积结果分别所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征进行叠加，输出各叠加结果；
    当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征为所述2N+1个第三视频帧的已对齐的提取特征，2N+1个第三视频帧的 中间帧的特征为所述2N+1个第三视频帧的中间帧的提取特征；
    当所述第一注意力渐进融合块，按连接次序不是所述N_apf个注意力渐进融合块的首个时，所述对应于2N+1个第三视频帧的特征和2N+1个第三视频帧的中间帧的特征为所述第一注意力渐进融合块的上一个注意力渐进融合块输出的所述各叠加结果；
    当所述第一注意力渐进融合块，按连接次序是所述N_apf个注意力渐进融合块的最后一个时，所述第一注意力渐进融合块还包括，一个第三卷积层，所述第一注意力渐进融合块将所述各叠加结果进行级联后，将获得的级联结果输入第三卷积层，并将第三卷积层的输出结果作为所述注意力渐进融合子网络的输出结果。
  7. 根据权利要求6所述的方法，其特征在于，所述将各输出结果输入时域注意力模块，获取整合后的中间帧特征，包括：
    时域注意力模块根据所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果，分别计算得到所述各输出结果中对应于2N+1个第三视频帧的输出结果与对应于2N+1个第三视频帧的中间帧的输出结果包含的各元素的相似性结果，将得到的各相似性结果分别与所述对应于2N+1个第三视频帧的输出结果相乘，并将各个获得结果级联，得到所述整合后的中间帧特征。
  8. 根据权利要求7所述的方法，其特征在于，所述相似性结果基于sigmoid函数计算获得。
  9. 根据权利要求2所述的方法，其特征在于，所述超分辨率重 构网络的训练方法分为两阶段，在其中的第一阶段，屏蔽所述时域注意力模块，在其中的第二阶段，对所述时域注意力模块解除屏蔽；所述第一阶段和第二阶段的切换依据第一阶段的训练过程中是否达到预定的收敛阈值。
  10. 根据权利要求1所述的方法，其特征在于，所述上采样子网络采用卷积网络。
PCT/CN2021/123605 2021-02-08 2021-10-13 一种视频帧的超分辨率重构方法 Ceased WO2022166245A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/529,203 US11995796B2 (en) 2021-02-08 2021-11-17 Method of reconstruction of super-resolution of video frame

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110170643.0A CN112801877B (zh) 2021-02-08 2021-02-08 一种视频帧的超分辨率重构方法
CN202110170643.0 2021-02-08

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/529,203 Continuation-In-Part US11995796B2 (en) 2021-02-08 2021-11-17 Method of reconstruction of super-resolution of video frame

Publications (1)

Publication Number Publication Date
WO2022166245A1 true WO2022166245A1 (zh) 2022-08-11

Family

ID=75814717

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/123605 Ceased WO2022166245A1 (zh) 2021-02-08 2021-10-13 一种视频帧的超分辨率重构方法

Country Status (3)

Country Link
US (1) US11995796B2 (zh)
CN (1) CN112801877B (zh)
WO (1) WO2022166245A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115358932A (zh) * 2022-10-24 2022-11-18 山东大学 一种多尺度特征融合的人脸超分辨率重构方法及系统
CN115496663A (zh) * 2022-10-12 2022-12-20 南京信息工程大学 基于d3d卷积组内融合网络的视频超分辨率重建方法

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112801877B (zh) * 2021-02-08 2022-08-16 南京邮电大学 一种视频帧的超分辨率重构方法
CN112801881B (zh) * 2021-04-13 2021-06-22 湖南大学 一种高分辨率高光谱计算成像方法、系统及介质
WO2023000179A1 (zh) * 2021-07-20 2023-01-26 Oppo广东移动通信有限公司 视频超分辨网络及视频超分辨、编解码处理方法、装置
CN114119371B (zh) * 2021-11-25 2023-01-10 北京百度网讯科技有限公司 视频超分模型训练方法和装置、视频超分处理方法和装置
CN116437093A (zh) * 2021-12-30 2023-07-14 北京字跳网络技术有限公司 视频帧修复方法、装置、设备、存储介质和程序产品
CN114092339B (zh) * 2022-01-24 2022-05-20 南京理工大学 基于跨帧自注意力变换网络的时空视频超分辨率重建方法
CN114648446B (zh) * 2022-03-31 2026-02-27 网银在线(北京)支付科技有限公司 视频处理方法和装置
US12511866B2 (en) * 2022-08-12 2025-12-30 Huawei Technologies Co., Ltd. Methods and systems for temporal action localization of video data
CN115908128B (zh) * 2022-10-27 2026-04-24 电信科学技术第十研究所有限公司 一种基于视频数据的轻量化超分辨率方法
CN115797177B (zh) * 2022-12-02 2025-06-03 西安电子科技大学 一种基于双重度量特征融合的视频超分辨率重建方法
CN116012230B (zh) * 2023-01-17 2023-09-29 深圳大学 一种时空视频超分辨率方法、装置、设备及存储介质
CN116188274B (zh) * 2023-03-21 2026-04-17 广东工业大学 一种图像超分辨率重建方法
CN116596779B (zh) * 2023-04-24 2023-12-01 天津大学 基于Transformer的Raw视频去噪方法
CN116668738B (zh) * 2023-04-26 2026-04-07 七腾机器人有限公司 一种视频时空超分辨率重构方法、装置及存储介质
CN117061790B (zh) * 2023-10-12 2024-01-30 深圳云天畅想信息科技有限公司 流媒体视频帧渲染方法、装置及存储介质
CN117575915B (zh) * 2024-01-16 2024-07-23 闽南师范大学 一种图像超分辨率重建方法、终端设备及存储介质
CN118446890B (zh) * 2024-04-28 2024-11-22 广东博华超高清创新中心有限公司 一种无时空编码的视频超分辨率方法
CN118154430B (zh) * 2024-05-10 2024-07-16 清华大学 时空角融合动态光场智能成像方法
CN121459139A (zh) * 2024-07-31 2026-02-03 阿里巴巴（中国）有限公司 视频任务处理、视频处理模型的训练、基于视频处理模型的信息处理方法及任务平台
CN118587096B (zh) * 2024-08-06 2024-12-06 中国科学院长春光学精密机械与物理研究所 帧间像素级信息补偿的卫星视频盲超分辨率系统及方法
CN119444570B (zh) * 2024-09-23 2025-11-28 西北大学 基于特征自适应注意力的图像超分辨率重建方法
CN119762344B (zh) * 2024-11-19 2025-09-23 西北大学 视频超分辨率模型构建、视频超分辨率成像方法及系统
CN120070184B (zh) * 2025-02-26 2025-10-24 烟台大学 任意尺度心脏cine MIR超分辨率重建方法及系统
CN120543429A (zh) * 2025-05-15 2025-08-26 北京华科嘉泰科技有限公司 一种基于去雾算法的实验图像增强方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111553861A (zh) * 2020-04-29 2020-08-18 苏州大学 一种图像超分辨率重构方法、装置、设备及可读存储介质
CN111583112A (zh) * 2020-04-29 2020-08-25 华南理工大学 视频超分辨率的方法、系统、装置和存储介质
US20200372609A1 (en) * 2019-05-23 2020-11-26 Beijing Baidu Netcom Science And Technology Co., Ltd. Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium
CN112801877A (zh) * 2021-02-08 2021-05-14 南京邮电大学 一种视频帧的超分辨率重构方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110070511B (zh) * 2019-04-30 2022-01-28 北京市商汤科技开发有限公司 图像处理方法和装置、电子设备及存储介质
CN110706155B (zh) * 2019-09-12 2022-11-29 武汉大学 一种视频超分辨率重建方法
CN111260560B (zh) * 2020-02-18 2020-12-22 中山大学 一种融合注意力机制的多帧视频超分辨率方法
CN111968064B (zh) * 2020-10-22 2021-01-15 成都睿沿科技有限公司 一种图像处理方法、装置、电子设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200372609A1 (en) * 2019-05-23 2020-11-26 Beijing Baidu Netcom Science And Technology Co., Ltd. Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium
CN111553861A (zh) * 2020-04-29 2020-08-18 苏州大学 一种图像超分辨率重构方法、装置、设备及可读存储介质
CN111583112A (zh) * 2020-04-29 2020-08-25 华南理工大学 视频超分辨率的方法、系统、装置和存储介质
CN112801877A (zh) * 2021-02-08 2021-05-14 南京邮电大学 一种视频帧的超分辨率重构方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115496663A (zh) * 2022-10-12 2022-12-20 南京信息工程大学 基于d3d卷积组内融合网络的视频超分辨率重建方法
CN115358932A (zh) * 2022-10-24 2022-11-18 山东大学 一种多尺度特征融合的人脸超分辨率重构方法及系统
CN115358932B (zh) * 2022-10-24 2023-03-24 山东大学 一种多尺度特征融合的人脸超分辨率重构方法及系统

Also Published As

Publication number Publication date
US11995796B2 (en) 2024-05-28
CN112801877B (zh) 2022-08-16
US20220261959A1 (en) 2022-08-18
CN112801877A (zh) 2021-05-14

Similar Documents

Publication Publication Date Title
CN112801877B (zh) 一种视频帧的超分辨率重构方法
Zhao et al. Comprehensive and delicate: An efficient transformer for image restoration
CN109102462B (zh) 一种基于深度学习的视频超分辨率重建方法
Anwar et al. Densely residual laplacian super-resolution
CN110120011B (zh) 一种基于卷积神经网络和混合分辨率的视频超分辨方法
Zhao et al. Invertible image decolorization
CN115222601A (zh) 基于残差混合注意力网络的图像超分辨率重建模型及方法
CN114494050A (zh) 一种基于事件相机的自监督视频去模糊和图像插帧方法
CN114757828B (zh) 基于Transformer的视频时空超分辨率方法
CN111369466B (zh) 基于可变形卷积的卷积神经网络的图像畸变矫正增强方法
CN114972036B (zh) 一种基于融合退化先验的盲图像超分辨率重建方法和系统
CN113379606B (zh) 一种基于预训练生成模型的人脸超分辨方法
CN114841859A (zh) 基于轻量神经网络和Transformer的单图像超分辨率重建方法
CN112150356A (zh) 基于级联框架的单幅压缩图像超分辨率重建方法
CN110706155A (zh) 一种视频超分辨率重建方法
CN115311149A (zh) 图像去噪方法、模型、计算机可读存储介质及终端设备
Liu et al. A densely connected face super-resolution network based on attention mechanism
CN117934278A (zh) 一种基于深度学习的视频超分辨率重建方法
Yue et al. A global appearance and local coding distortion based fusion framework for CNN based filtering in video coding
CN114529456B (zh) 一种视频的超分辨率处理方法、装置、设备及介质
CN116485666A (zh) 基于多尺度动态特征聚合的图像摩尔纹噪声去除方法
CN109949217A (zh) 基于残差学习和隐式运动补偿的视频超分辨率重建方法
CN116208812B (zh) 一种基于立体事件和强度相机的视频插帧方法及系统
Yue et al. A Survey of Deep-Learning-Based Compressed Video Quality Enhancement
Liu et al. Promptsr: Cascade prompting for lightweight image super-resolution

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21924242

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21924242

Country of ref document: EP

Kind code of ref document: A1