WO2023174355A1 - 视频的超分辨率方法及装置 - Google Patents

视频的超分辨率方法及装置 Download PDF

Info

Publication number
WO2023174355A1
WO2023174355A1 PCT/CN2023/081794 CN2023081794W WO2023174355A1 WO 2023174355 A1 WO2023174355 A1 WO 2023174355A1 CN 2023081794 W CN2023081794 W CN 2023081794W WO 2023174355 A1 WO2023174355 A1 WO 2023174355A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image block
backward
pool
super
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/081794
Other languages
English (en)
French (fr)
Inventor
董航
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Priority to US18/725,722 priority Critical patent/US20250061541A1/en
Publication of WO2023174355A1 publication Critical patent/WO2023174355A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present disclosure relates to the field of image processing technology, and in particular, to a video super-resolution method and device.
  • Video super-resolution technology also known as video super-resolution technology, is a technology that restores high-resolution video from low-resolution video. Since video super-resolution business has become a key business in video quality enhancement, video super-resolution technology is one of the current research hotspots in the field of image processing.
  • video super-resolution network models based on deep learning neural networks have achieved many breakthroughs, including better super-resolution effects and better real-time performance.
  • mainstream video super-resolution network models take advantage of the fact that most image frames of the video are in motion. When super-resolving each image frame in the video, its neighboring image frames can provide a large amount of time domain information. , for the video super-resolution network model to perform super-resolution on the current image frame.
  • embodiments of the present disclosure provide a video super-resolution method, including:
  • a super-resolution image frame corresponding to the target image frame is obtained.
  • embodiments of the present disclosure provide a video super-resolution device, including:
  • the image decomposition module is used to decompose the target image frame of the video to be super-resolved into multiple image blocks;
  • a feature acquisition module configured to acquire the super-resolution features of the target image frame based on the plurality of image blocks and image blocks obtained by decomposing other image frames in the video to be super-resolved;
  • An image generation module configured to obtain a super-resolution image frame corresponding to the target image frame according to the super-resolution feature of the target image frame.
  • embodiments of the present disclosure provide an electronic device, including: a memory and a processor, the memory is used to store a computer program; the processor is used to cause the electronic device to implement the first step when calling the computer program.
  • a memory and a processor the memory is used to store a computer program; the processor is used to cause the electronic device to implement the first step when calling the computer program.
  • embodiments of the present disclosure provide a computer-readable storage medium, which when the computer program is executed by a computing device, causes the computing device to implement the first aspect or any optional implementation of the first aspect.
  • embodiments of the present disclosure provide a computer program product.
  • the computer program product When the computer program product is run on a computer, the computer implements the method described in the first aspect or any optional implementation manner of the first aspect. Super-resolution methods for video.
  • Figure 1 is a step flow chart of a video super-resolution method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of image blocks obtained by decomposing image frames according to an embodiment of the present disclosure
  • Figure 3 is a second flowchart of a video super-resolution method provided by an embodiment of the present disclosure
  • Figure 4 is the third flow chart of the video super-resolution method provided by the embodiment of the present disclosure.
  • Figure 5 is one of the structural schematic diagrams of a video super-resolution network provided by an embodiment of the present disclosure
  • Figure 6 is a second flowchart of a video super-resolution method provided by an embodiment of the present disclosure.
  • Figure 7 is one of the schematic diagrams of an image block sequence provided by an embodiment of the present disclosure.
  • Figure 8 is one of the schematic diagrams of the backward feature acquisition module provided by an embodiment of the present disclosure.
  • Figure 9 is a second schematic diagram of the backward feature acquisition module provided by an embodiment of the present disclosure.
  • Figure 10 is the third flowchart of the video super-resolution method provided by the embodiment of the present disclosure.
  • Figure 11 is a second schematic diagram of an image block sequence provided by an embodiment of the present disclosure.
  • Figure 12 is one of the schematic diagrams of the forward feature acquisition module provided by an embodiment of the present disclosure.
  • Figure 13 is a second schematic diagram of the forward feature acquisition module provided by an embodiment of the present disclosure.
  • Figure 14 is a second structural schematic diagram of a video super-resolution network provided by an embodiment of the present disclosure.
  • Figure 15 is a schematic diagram of a video super-resolution device provided by an embodiment of the present disclosure.
  • Figure 16 is a schematic diagram of a video super-resolution device provided by an embodiment of the present disclosure.
  • Figure 17 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure.
  • words such as “first” and “second” are used to describe the same or similar items that have basically the same functions and effects.
  • words such as “first” and “second” do not limit the quantity and execution order.
  • the first feature image set and the second feature image set are only for distinguishing different feature image sets, rather than limiting the order of the feature image sets.
  • words such as “exemplary” or “such as” are used to represent examples, illustrations or explanations. Any embodiment or design described as “exemplary” or “such as” in the present disclosure is not intended to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words “exemplary” or “such as” is intended to present the concept in a concrete manner. Furthermore, in the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of “plurality” means two or more.
  • Embodiments of the present disclosure provide a video super-resolution method for improving the super-resolution effect of videos.
  • the video super-resolution method provided by the embodiment of the present disclosure includes the following steps:
  • the implementation of the above step S11 may include: using a sampling window with a size of one image block and a step size of a preset value. , starting from the first pixel of the target image frame, sampling each position of the target image frame, and treating each sampling area of the sampling window as an image block, thereby decomposing the target image frame into multiple images. piece.
  • the target image frame of the video to be super-resolved includes 1024*512 pixels.
  • the target image frame of the video to be super-resolved can be The image frame is decomposed into 16*8 image blocks. Each image block includes 72*72 pixels, and there is an overlapping area between adjacent image blocks. The width of the overlapping area is 8 pixels.
  • the above step S12 (obtaining the super-resolution features of the target image frame based on the multiple image blocks and image blocks obtained by decomposing other image frames in the video to be super-resolved) includes:
  • the super-resolution features of each image block in the plurality of image blocks are obtained respectively, and the super-resolution characteristics of each image block in the plurality of image blocks are obtained.
  • Super-resolution features are combined to obtain the super-resolution features of the target image frame.
  • the target image frame is the t-th image frame of the video to be super-resolved
  • each image frame of the video to be super-resolved is decomposed into N image blocks
  • decompose the Image blocks obtained from other image frames in the super-resolution video including: image blocks image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block image block
  • the image blocks obtained by decomposing the video target image frame to be super-resolved include: image blocks image block image block Among them, the image block Represents the i-th image block obtained by decomposing the j-th video frame of the video to be super-resolved.
  • the above step S13 (obtaining the super-resolution image frame corresponding to the target image frame according to the super-resolution feature of the target image frame) includes:
  • the super-resolution features of the target image frame and the features of the target image frame are added and fused to obtain a super-resolution image frame corresponding to the target image frame.
  • the video super-resolution method When super-resolving the target image frame of the video to be super-resolved, the video super-resolution method provided by the embodiment of the present disclosure first decomposes the target image frame of the video to be super-resolved into multiple image blocks, and then based on the multiple image blocks. Image blocks and image blocks obtained by decomposing other image frames in the video to be super-resolved are obtained, and the super-resolution features of the target image frame are obtained. Finally, the target image is obtained according to the super-resolution features of the target image frame. The super-resolution image frame corresponding to the frame.
  • the super-resolution method can decompose the target image frame of the video to be super-resolved by decomposing
  • the image blocks obtained from all other image frames in the video to be super-resolved except the target video frame provide temporal information for the image blocks obtained by decomposing the target video frame, thereby obtaining the super-resolution features of the target image frame.
  • embodiments of the present disclosure can use non-adjacent image frames to provide sufficient temporal information for each image block of the target video frame, thereby providing The target video frame provides sufficient temporal information, so embodiments of the present disclosure can improve the super-resolution effect of the video.
  • embodiments of the present disclosure provide another video super-resolution method.
  • the method includes the following steps:
  • the backward feature of any image block is the feature of the image block corresponding to the image block among the image blocks obtained by decomposing the image frame located after the target image frame in the video to be super-resolved.
  • the forward feature of any image block is the feature of the image block corresponding to the image block among the image blocks obtained by decomposing the image frame located before the target image frame in the video to be super-resolved.
  • the backward features of each of the plurality of image blocks are fused to obtain the backward features of the target image frame.
  • the forward features of each of the multiple image blocks are fused to obtain the forward features of the target image frame. feature.
  • step S306 (obtaining the super-resolution features of the target image frame based on the backward features and forward features of the target image frame) includes the following steps a and b:
  • Step a Merge the backward features and forward features of the target image frame to obtain the merged features of the target image frame.
  • the backward features of the target image frame and the forward features of the target image frame can be concatenated in the channel dimension to obtain the merged features of the target image frame.
  • Step b Upsample the merged features of the target image frame to obtain the super-resolution features of the target image frame.
  • the backward features and forward features of the target image frame can be merged first, and then the merged features can be upsampled to obtain, taking the super-resolution features of the target image frame as an example.
  • the backward features and forward features of the target image frame can also be upsampled separately first, and then the upsampling results can be combined to obtain the super-resolution features of the target image frame.
  • S307 Obtain the super-resolution image frame corresponding to the target image frame according to the super-resolution feature of the target image frame.
  • embodiments of the present disclosure provide another video super-resolution method.
  • the method includes the following steps:
  • the backward image block pool includes a backward image block corresponding to each image block in the plurality of image blocks; the backward image block corresponding to any image block is decomposed from the to-be-decomposed image block based on a preset selection rule.
  • the backward image block pool includes N backward image blocks, and the backward feature pool includes N features;
  • the N backward images Blocks correspond to the plurality of image blocks one-to-one, and the N features in the backward feature pool are respectively the features of the N backward image blocks.
  • the target image frame is the t-th image frame of the video to be super-resolved
  • the image block obtained by the image frame includes: first, determining the difference between the image block and the image block obtained by decomposing the t+1th image frame to the last image frame of the video to be super-resolved. Each image block with the same position is obtained to obtain the first image block set. Then from the first image block collection Select the image block The image patch that provides the most effective temporal information, as an image patch The corresponding backward image patch.
  • the forward image block pool includes a forward image block corresponding to each image block in the plurality of image blocks; the forward image block corresponding to any image block is based on a preset selection rule, from decomposing the In the video to be super-resolved, image blocks selected from the image blocks obtained from the image frame before the target image frame; the forward feature pool includes features of each forward image block in the forward image block pool
  • the forward image block pool includes N forward image blocks, and the forward feature pool includes N features;
  • the N forward images Blocks correspond to the plurality of image blocks one-to-one, and the N features in the forward feature pool are respectively the features of the N forward image blocks.
  • select image blocks from the image blocks obtained by decomposing the first image frame to t-1 image frames of the video to be super-resolved may include: first, determining the difference between the image block obtained by decomposing the first image frame to the t-1 image frame of the video to be super-resolved and the image block Each image block with the same position is obtained to obtain the second set of image blocks. Then from the second image block collection Select the image block The image patch that provides the most effective temporal information is used as the image patch The corresponding forward image patch.
  • S405 Obtain the forward feature of each image block in the plurality of image blocks according to the forward image block pool and the forward feature pool.
  • S406 Obtain the backward characteristics of the target image frame according to the backward characteristics of each of the plurality of image blocks.
  • FIG. 5 is a schematic structural diagram of a video super-resolution network used to implement the video super-resolution method in FIG. 4 .
  • the video super-resolution network used to implement the video super-resolution method in Figure 4 includes: decomposition module 51, backward image block pool 52, backward feature pool 53, backward feature transfer module 54, forward image block pool 55, forward feature Pool 56, forward feature transfer module 57, processing module 58 and generation module 59.
  • the decomposition module 51 is used to decompose the target image frame It of the video to be super-resolved into multiple image blocks.
  • the backward image block pool 52 is used to store the backward image block corresponding to each image block in the plurality of image blocks.
  • the backward feature pool 53 is used to store the features of each backward image patch.
  • the backward feature transfer module 54 is used to perform Backward image blocks in the backward image block pool and features in the backward feature pool Obtain the backward features of the target image frame
  • the forward image block pool 55 is used to store the forward image block corresponding to each image block in the plurality of image blocks.
  • Forward feature pool 56 is used to store the features of each forward image patch
  • the forward feature transfer module 57 is used to perform forward image patch in forward image patch pool and the features in the forward feature pool Obtain the forward features of the target image frame
  • the processing module 58 is configured to use backward features of the target image frame and forward features Obtain super-resolution features of the target image frame
  • the generation module 59 is used to generate super-resolution features according to the target image frame. Generate a super-resolution image frame O t corresponding to the target image frame.
  • step S403 obtaining each image block in the plurality of image blocks based on the backward image block pool and the backward feature pool) backward features
  • step S403 obtain each image block in the plurality of image blocks based on the backward image block pool and the backward feature pool) backward features
  • the optical flow of any backward image block is the optical flow between the backward image block and the corresponding image block of the backward image block in the plurality of image blocks.
  • the implementation of the above step S61 may include the following steps 611 and 612:
  • Step 611 Generate a first image block sequence based on the plurality of image blocks, and generate a second image block sequence based on the backward image blocks in the backward image block pool.
  • the ordering of any image block in the first image block sequence is the same as the ordering of the backward image block corresponding to the image block in the second image block sequence.
  • the embodiment of the present disclosure does not limit the arrangement order of the plurality of image blocks in the first image block sequence, nor does it limit the The order in which the backward image blocks in the backward image block pool are arranged in the second image block is such that the order of any image block in the first image block sequence is the same as the order in which the backward image block corresponding to the image block is located.
  • the ordering in the second image block sequence shall be the same.
  • the order in the first image block sequence is: The backward image patch in the backward image patch pool
  • the order in the second image block sequence is: multiple image blocks
  • the ordering of any image block in the first image block sequence is the same as the ordering of the backward image block corresponding to the image block in the second image block sequence.
  • Step 612 Input the first image block sequence and the second image block sequence into the optical flow prediction network model, and obtain each backward image block in the backward image block pool according to the output of the optical flow prediction network model. Optical flow to image patches.
  • S62 Process the characteristics of each backward image block in the backward feature pool according to the optical flow of each backward image block in the backward image block pool, and obtain the characteristics of each backward image block in the backward image block pool. Alignment features for each backward image patch.
  • each backward image block in the backward image block pool the characteristics of each backward image block in the backward feature pool are compared with the corresponding images in the plurality of image blocks.
  • the features of the blocks are aligned to obtain the alignment features of each backward image block in the backward image block pool.
  • the above step S63 (obtaining the plurality of images based on the alignment characteristics of each backward image block in the backward image block pool and the plurality of image blocks)
  • the backward features of each image patch in the patch include:
  • Each image block in the plurality of image blocks and the alignment features of the backward image block corresponding to each image block are processed through a residual block (Residual block) to obtain each image in the plurality of image blocks.
  • Residual block residual block
  • FIG. 6 is a diagram for obtaining the backward features of each image block in the plurality of image blocks according to the backward image block pool and the backward feature pool, and according to the The backward feature of each image block in the plurality of image blocks is a schematic structural diagram of a backward feature acquisition module that obtains the backward feature of the target image frame.
  • the backward feature acquisition module includes: optical flow prediction network model 81 , feature alignment module 82, residual block 83 and feature fusion module 84.
  • the optical flow prediction network model 81 is based on the backward image blocks in the input backward image block pool. and multiple image blocks obtained by decomposing the target image frame of the video to be super-resolved. Output each of the backward image patch pools Optical flow of backward image blocks
  • the feature alignment module 82 is used to calculate the optical flow of each backward image patch in the backward image patch pool. Convert the features of each backward image patch in the backward pooling Aligned with the target image frame, obtain the alignment features of each backward image block in the backward image block pool
  • Residual block 83 is used to align features of each backward image patch according to the backward image patch pool. and the plurality of image blocks Obtain the backward features of each image block in the plurality of image blocks
  • the feature fusion module 84 is used to fuse the backward features of each of the plurality of image blocks. Generate backward features of the target image frame
  • the video super-resolution method provided by the embodiment of the present disclosure also includes:
  • the backward image block pool and the backward feature pool are updated according to the plurality of image blocks and backward features of each of the plurality of image blocks.
  • the backward image block pool and the backward image block pool are configured according to the plurality of image blocks and the backward characteristics of each image block in the plurality of image blocks.
  • An implementation method for updating the feature pool includes the following steps 1) and 2):
  • Step 1) determine whether the absolute value of the optical flow of each backward image block in the backward image block pool is greater than a preset threshold
  • step 1) if the absolute value of the optical flow of the first backward image block in the backward image block pool is greater than the preset threshold, the following step 2) is performed.
  • Step 2) replace the first backward image block in the backward image block pool with the image block corresponding to the first backward image block in the plurality of image blocks, and replace the backward image block with the corresponding image block in the plurality of image blocks.
  • the features of the first backward image block in the backward feature pool are replaced with backward features of corresponding image blocks of the first backward image block in the plurality of image blocks.
  • Figure 9 shows that the backward feature acquisition module is also used to obtain the backward image block according to the plurality of image blocks and the backward characteristics of each image block in the plurality of image blocks.
  • the module used to implement the backward feature acquisition includes: optical flow prediction network model 81, feature alignment module 82, residual block 83, feature fusion module 84 and update module 85.
  • the update module 85 is used to update the image blocks according to the and the backward features of each image patch in the plurality of image patches
  • the backward image block pool and the backward feature pooling Updated to backward image block pooling and backward feature pooling and pool the backward image blocks and backward feature pooling As backward image patch pooling and backward feature pooling when processing the previous image frame of the super-resolution video.
  • step S405 obtaining each image block in the plurality of image blocks based on the forward image block pool and the forward feature pool) forward features
  • step S405 obtain each image block in the plurality of image blocks based on the forward image block pool and the forward feature pool) forward features
  • the optical flow of any forward image block is the optical flow between the forward image block and the corresponding image block of the forward image block in the plurality of image blocks.
  • the implementation of the above step S81 may include the following steps 1011 and 1012:
  • Step 1011 Generate a third image block sequence based on the plurality of image blocks, and generate a fourth image block sequence based on the forward image blocks in the forward image block pool.
  • the ordering of any image block in the third image block sequence is the same as the ordering of the forward image block corresponding to the image block in the fourth image block sequence.
  • the arrangement order of the plurality of image blocks in the third image block sequence is not limited, nor is the arrangement order of the forward image blocks in the forward image block pool in the fourth image block.
  • the ordering of any image block in the third image block sequence shall be the same as the ordering of the forward image block corresponding to the image block in the fourth image block sequence.
  • multiple image blocks The order in the third image block sequence is: Forward image blocks in the forward image block pool
  • the order in the fourth image block sequence is: multiple image blocks
  • the ordering of any image block in the third image block sequence is the same as the ordering of the forward image block corresponding to the image block in the fourth image block sequence.
  • Step 1012 Input the third image block sequence and the fourth image block sequence into the optical flow prediction network model, and obtain each of the forward image block pools according to the output of the optical flow prediction network model. Optical flow for forward image patches.
  • each forward image block in the forward image block pool process the characteristics of each forward image block in the forward feature pool to obtain the features in the forward image block pool. Alignment features of each forward image patch.
  • each forward image block in the forward image block pool the characteristics of each forward image block in the forward feature pool are compared with the corresponding images in the plurality of image blocks.
  • the features of the blocks are aligned, thereby obtaining the alignment features of each forward image block in the forward image block pool.
  • the above step S83 (according to the alignment characteristics of each forward image block in the forward image block pool and the multiple image blocks, obtain the multiple images
  • the forward features of each image patch in the patch include:
  • each image block in the plurality of image blocks and the alignment feature of the forward image block corresponding to each image block are processed to obtain each of the plurality of image blocks. Forward features of image patches.
  • FIG. 12 is used to obtain the forward features of each image block in the plurality of image blocks according to the forward image block pool and the forward feature pool, and according to the A schematic structural diagram of a forward feature acquisition module that acquires forward features of the target image frame from the forward features of each image block in multiple image blocks.
  • the forward feature acquisition module includes: optical flow prediction network model 121, Feature alignment module 122, residual block 123 and feature fusion module 124.
  • the optical flow prediction network model 121 is based on the forward image block in the input forward image block pool. and multiple image blocks obtained by decomposing the target image frame of the video to be super-resolved. Output the optical flow of each forward image block in the forward image block pool
  • the feature alignment module 122 is configured to perform optical flow according to each forward image block in the forward image block pool. Convert the features of each forward image patch in the forward pool to Aligned with the target image frame, obtain the alignment features of each forward image block in the forward image block pool Residual block 123 is used to align features of each forward image patch according to the pool of forward image patches. and the plurality of image blocks Obtain the forward features of each image block in the plurality of image blocks
  • the feature fusion module 124 is configured to fuse forward features of each of the plurality of image blocks. Generate forward features of the target image frame
  • the video super-resolution method provided by the embodiment of the present disclosure also includes:
  • the forward image patch pool and the forward feature pool are updated based on the plurality of image patches and forward features of each of the plurality of image patches.
  • the forward image block pool and the forward An implementation method for updating the feature pool includes the following steps I and II:
  • Step I Determine whether the absolute value of the optical flow of each forward image block in the forward image block pool is greater than the preset value. threshold;
  • step I if the absolute value of the optical flow of the first forward image block in the forward image block pool is greater than the preset threshold, the following step II is performed.
  • Step II Replace the first forward image block in the forward image block pool with the image block corresponding to the first forward image block in the plurality of image blocks, and replace the forward image block with the corresponding image block in the forward image block pool.
  • the features of the first forward image block in the feature pool are replaced with forward features of the image blocks corresponding to the first forward image block in the plurality of image blocks.
  • Figure 13 is the forward feature acquisition module, which is also used to obtain the forward feature according to the multiple image blocks and the forward features of each image block in the multiple image blocks.
  • the forward feature acquisition module includes: optical flow prediction network model 121, feature alignment module 122, residual block 123, feature fusion module 124 and update module 125.
  • the update module 125 is configured to update the image blocks according to the and the forward features of each image patch in the plurality of image patches
  • the forward image block pool and the forward feature pooling Updated to forward image block pooling and forward feature pooling and pool forward image blocks and forward feature pooling As forward image patch pooling and forward feature pooling when processing the next image frame of the super-resolution video.
  • FIG. 14 is a network structure diagram of a video super-resolution network provided by an embodiment of the present disclosure.
  • the target image frame of the video to be super-resolved uses the backward image block pool updated with the t+1th image frame.
  • forward image patch pool updated for the t-1th image frame and forward feature pooling Complete the super score and pair the backward image block pool again and backward feature pooling Update, and the target image frame updated backward image patch pool and backward feature pooling As the backward image patch pool of the next image frame (t+1 image frame) and backward feature pooling Pool the forward image patch updated from the target image frame and forward feature pooling As the forward image patch pool of the previous image frame (t-1th image frame) and forward feature pooling
  • an embodiment of the present disclosure also provides a video super-resolution device.
  • the device embodiment corresponds to the foregoing method embodiment.
  • the device embodiment is no longer The details in the foregoing method embodiments will be described one by one, but it should be clear that the video super-resolution device in this embodiment can correspondingly implement all the contents in the foregoing method embodiments.
  • FIG. 15 is a schematic structural diagram of the video super-resolution device. As shown in Figure 15, the video super-resolution device 1500 includes:
  • the image decomposition module 151 is used to decompose the target image frame of the video to be super-resolved into multiple image blocks;
  • the feature acquisition module 152 is configured to acquire the super-resolution features of the target image frame based on the plurality of image blocks and image blocks obtained by decomposing other image frames in the video to be super-resolved;
  • the image generation module 153 is configured to obtain a super-resolution image frame corresponding to the target image frame according to the super-resolution feature of the target image frame.
  • the feature acquisition module 152 includes:
  • the backward feature acquisition unit 1521 is used to obtain the backward feature of each image block in the plurality of image blocks.
  • the backward feature of any image block is the target image frame located in the decomposed video to be super-resolved.
  • the characteristics of the image block corresponding to the image block in the image block obtained from the subsequent image frame;
  • the forward feature acquisition unit 1522 is used to acquire the forward feature of each image block in the plurality of image blocks.
  • the forward feature of any image block is the target image frame located in the decomposed video to be super-resolved. Characteristics of the image block corresponding to the image block in the image block obtained from the previous image frame;
  • the first feature merging unit 1523 is configured to obtain the backward features of the target image frame according to the backward features of each image block in the plurality of image blocks;
  • the second feature merging unit 1524 is configured to obtain the forward features of the target image frame according to the forward features of each image block in the plurality of image blocks;
  • the feature fusion unit 1525 is configured to obtain the super-resolution features of the target image frame based on the backward features and forward features of the target image frame.
  • the backward feature acquisition unit 1521 is specifically used to acquire a backward image block pool and a backward feature pool.
  • the backward image block pool includes the plurality of images.
  • the backward image block corresponding to each image block in the block; the backward image block corresponding to any image block is obtained from decomposing the image frame located after the target image frame in the video to be super-resolved based on the preset selection rules.
  • image blocks selected from the image blocks; the backward feature pool includes the characteristics of each backward image block in the backward image block pool; according to the backward image block pool and the backward feature pool, Obtain backward features for each of the plurality of image blocks.
  • the backward feature acquisition unit 1521 is specifically used to obtain the optical flow of each backward image block in the backward image block pool, any backward image
  • the optical flow of a block is the optical flow between the backward image block and the corresponding image block of the backward image block in the plurality of image blocks; according to each backward image block in the backward image block pool optical flow, process the features of each backward image block in the backward feature pool, and obtain the alignment features of each backward image block in the backward image block pool; according to the backward image block everyone in the pool Alignment features of one backward image block and the plurality of image blocks, and obtain backward features of each of the plurality of image blocks.
  • the backward feature acquisition unit 1521 is specifically configured to generate a first image block sequence according to the plurality of image blocks, and generate a first image block sequence according to the backward image block pool.
  • the backward image block generates a second image block sequence; the ordering of any image block in the first image block sequence is the same as the ordering of the backward image block corresponding to the image block in the second image block sequence;
  • the first image block sequence and the second image block sequence are input to an optical flow prediction network model, and the value of each backward image block in the backward image block pool is obtained according to the output of the optical flow prediction network model. light flow.
  • the backward feature acquisition unit 1521 is specifically configured to use a residual block to obtain each image block in the plurality of image blocks and the backward feature corresponding to each image block. Alignment features of the forward image blocks are processed to obtain backward features of each of the plurality of image blocks.
  • the backward feature acquisition unit 1521 is also configured to pair the backward features according to the plurality of image blocks and each image block in the plurality of image blocks.
  • the backward image patch pool and the backward feature pool are updated.
  • the backward feature acquisition unit 1521 is specifically configured to determine whether the absolute value of the optical flow of each backward image block in the backward image block pool is greater than a predetermined value. Set a threshold; if the absolute value of the optical flow of the first backward image block in the backward image block pool is greater than the preset threshold, then the first backward image in the backward image block pool is Replace the block with the image block corresponding to the first backward image block in the plurality of image blocks, and replace the feature of the first backward image block in the backward feature pool with the first backward image block. backward features of the corresponding image block among the plurality of image blocks.
  • the forward feature acquisition unit 1522 is specifically used to acquire a forward image block pool and a forward feature pool.
  • the forward image block pool includes the multiple images.
  • the forward image block corresponding to each image block in the block; the forward image block corresponding to any image block is obtained based on the preset selection rules from decomposing the image frame located before the target image frame in the video to be super-resolved.
  • image blocks selected from the image blocks; the forward feature pool includes the characteristics of each forward image block in the forward image block pool; according to the forward image block pool and the forward feature pool, Forward features of each of the plurality of image blocks are obtained.
  • the forward feature acquisition unit 1522 is specifically configured to acquire the optical flow of each forward image block in the forward image block pool, any forward image
  • the optical flow of a block is the optical flow between the forward image block and the corresponding image block of the forward image block in the plurality of image blocks; according to the forward image block pool
  • the optical flow of each forward image block in the forward image block processes the characteristics of each forward image block in the forward feature pool, and obtains the alignment characteristics of each forward image block in the forward image block pool; According to the alignment characteristics of each forward image block in the forward image block pool and the plurality of image blocks, a forward feature of each image block in the plurality of image blocks is obtained.
  • the forward feature acquisition unit 1522 is specifically configured to generate a third image block sequence according to the plurality of image blocks, and generate a third image block sequence according to the forward image block pool.
  • the forward image block generates a fourth image block sequence; the ordering of any image block in the third image block sequence is the same as the ordering of the forward image block corresponding to the image block in the fourth image block sequence;
  • the third image block sequence and the fourth image block sequence are input to an optical flow prediction network model, and the value of each forward image block in the forward image block pool is obtained according to the output of the optical flow prediction network model. light flow.
  • the forward feature acquisition unit 1522 is specifically configured to use a residual block to obtain each image block in the plurality of image blocks and the forward feature corresponding to each image block. Alignment features of the forward image blocks are processed to obtain forward features of each of the plurality of image blocks.
  • the forward feature acquisition unit 1522 is also configured to pair the forward features according to the plurality of image blocks and each image block in the plurality of image blocks.
  • the forward image patch pool and the forward feature pool are updated.
  • the forward feature acquisition unit 1522 is specifically configured to determine whether the absolute value of the optical flow of each forward image block in the forward image block pool is greater than a predetermined value. Set a threshold; if the absolute value of the optical flow of the first forward image block in the forward image block pool is greater than the preset threshold, then the first forward image in the forward image block pool is The blocks are replaced with image blocks corresponding to the first forward image block in the plurality of image blocks, and the features of the first forward image block in the forward feature pool are replaced with the first forward image blocks. Forward features of the corresponding image block among the plurality of image blocks.
  • the feature processing unit 153 is specifically configured to merge the backward features and forward features of the target image frame to obtain the merged features of the target image frame;
  • the merged features of the target image frame are upsampled to obtain the super-resolution features of the target image frame.
  • the video super-resolution device provided in this embodiment can perform the video super-resolution method provided in the above method embodiment. Its implementation principles and technical effects are similar and will not be described again here.
  • FIG 17 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • the electronic device provided by this embodiment includes: a memory 171 and Processor 172, the memory 171 is used to store a computer program; the processor 172 is used to execute the video super-resolution method provided by the above embodiment when the computer program is called.
  • embodiments of the present disclosure also provide a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the computing device implements the above embodiments.
  • embodiments of the present disclosure also provide a computer program product.
  • the computing device implements the video super-resolution method provided by the above embodiments.
  • embodiments of the present disclosure may be provided as methods, systems, or computer program products. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein.
  • the processor can be a Central Processing Unit (CPU), other general-purpose processors, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), or off-the-shelf programmable processors. Gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
  • Memory may include non-volatile memory in computer-readable media, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • Computer-readable media includes permanent and non-permanent, removable and non-removable storage media.
  • Storage media can be implemented by any method or technology to store information, and information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), and read-only memory.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • read-only memory read-only memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other memory technology
  • compact disc read-only memory CD-ROM
  • DVD digital versatile disc
  • Magnetic tape cassettes disk storage or other magnetic storage devices, or any other non-transmission medium, can be used to store information that can be accessed by a computing device.
  • computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

本公开实施例提供了一种视频的超分辨率方法及装置，涉及图像处理技术领域。该方法包括：将待超分视频的目标图像帧分解为多个图像块；根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征；根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。

Description

视频的超分辨率方法及装置
相关申请的交叉引用
本公开是以中国申请号为202210265574.6，申请日为2022年03月17日的申请为基础，并主张其优先权，该中国申请的公开内容在此作为整体引入本公开中。
技术领域
本公开涉及图像处理技术领域，尤其涉及一种视频的超分辨率方法及装置。
背景技术
视频的超分辨率技术又称为视频超分技术，是一种由低分辨率视频恢复出高分辨率视频的技术。由于视频超分辨率业务目前已成为视频画质增强中的重点业务，因此视频超分技术是当前图像处理领域的研究热点之一。
近年来，随着深度学习技术的发展，基于深度学习神经网络的视频超分网络模型实现了许多突破，包括更好的超分效果以及更好的实时性。目前，主流视频超分网络模型均是利用视频的大多数图像帧都处于运动之中，在对视频中的每一个图像帧进行超分时，其邻域图像帧都可以提供大量的时域信息，以供视频超分网络模型对当前图像帧进行超分。
发明内容
第一方面，本公开的实施例提供了一种视频的超分辨率方法，包括：
将待超分视频的目标图像帧分解为多个图像块；
根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征；
根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
第二方面，本公开的实施例提供了一种视频的超分辨率装置，包括：
图像分解模块，用于将待超分视频的目标图像帧分解为多个图像块；
特征获取模块，用于根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征；
图像生成模块，用于根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
第三方面，本公开实施例提供了一种电子设备，包括：存储器和处理器，所述存储器用于存储计算机程序；所述处理器用于在调用计算机程序时，使得所述电子设备实现第一 方面或第一方面任一种可选的实施方式所述的视频的超分辨率方法。
第四方面，本公开实施例提供一种计算机可读存储介质，当所述计算机程序被计算设备执行时，使得所述计算设备实现第一方面或第一方面任一种可选的实施方式所述的视频的超分辨率方法。
第五方面，本公开实施例提供一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机实现第一方面或第一方面任一种可选的实施方式所述的视频的超分辨率方法。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或相关技术中的技术方案，下面将对实施例或相关技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的视频的超分辨率方法的步骤流程图之一；
图2为本公开实施例提供的图像帧分解得到的图像块的示意图；
图3为本公开实施例提供的视频的超分辨率方法的流程图之二；
图4为本公开实施例提供的视频的超分辨率方法的流程图之三；
图5为本公开实施例提供的视频超分网络的结构示意图之一；
图6为本公开实施例提供的视频的超分辨率方法的流程图之二；
图7为本公开实施例提供的图像块序列示意图之一；
图8为本公开实施例提供的后向特征获取模块的示意图之一；
图9为本公开实施例提供的后向特征获取模块的示意图之二；
图10为本公开实施例提供的视频的超分辨率方法的流程图之三；
图11为本公开实施例提供的图像块序列示意图之二；
图12为本公开实施例提供的前向特征获取模块的示意图之一；
图13为本公开实施例提供的前向特征获取模块的示意图之二；
图14为本公开实施例提供的视频超分网络的结构示意图之二；
图15为本公开实施例提供的视频的超分辨率装置的示意图之一；
图16为本公开实施例提供的视频的超分辨率装置的示意图之一
图17为本公开实施例提供的电子设备的硬件结构示意图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
需要说明的是，为了便于清楚描述本公开实施例的技术方案，在本公开的实施例中，采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分，本领域技术人员可以理解“第一”、“第二”等字样并不是在对数量和执行次序进行限定。例如：第一特征图像集合和第二特征图像集合仅仅是为了区分不同的特征图像集合，而不是在对特征图像集合的顺序等进行限定。
在本公开实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本公开实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。此外，在本公开实施例的描述中，除非另有说明，“多个”的含义是指两个或两个以上。
相关技术中，在对视频中的每一个图像帧进行超分时，其邻域图像帧都可以提供大量的时域信息，以供视频超分网络模型对当前图像帧进行超分。然而，在一些视频中，部分区域始终为静止的物体或背景，在对这类视频进行超分时，由于静止的物体或背景会导致运动信息估计误差，且误差会在信息传递过程中累积，进而导致误差逐渐增大；同时，这些物体或背景冗余信息在信息传递中也会导致间隔较远的图像帧的有效时域信息被逐渐替换，进而使得网络无法有效利用间隔较远的图像帧的时域信息。综上，当视频中存在静止的物体或背景时，视频超分网络模型很可能无法获取足够的时域信息对图像帧进行超分，进而导致视频超分效果十分不理想。
本公开实施例提供了一种视频的超分辨率方法，用于提升视频的超分辨率效果。
参照图1所示的步骤流程图，本公开实施例提供的视频的超分辨率方法包括如下步骤：
S11、将待超分视频的目标图像帧分解为多个图像块。
在一些实施例中，上述步骤S11(将待超分视频的目标图像帧分解为多个图像块)的实现方式可以包括：通过尺寸为一个图像块的尺寸、步长为预设值的采样窗口，从所述目标图像帧的第一个像素点开始对所述目标图像帧的各个位置进行采样，并将采样窗口的每一个采样区域作为一个图像块，从而将目标图像帧分解为多个图像块。
示例性的，参照图2所示，待超分视频的目标图像帧包括1024*512个像素点，当采样窗口的尺寸为72*72、步长为64时，可以将待超分视频的目标图像帧分解为16*8个图像块，每一个图像块包括72*72个像素点，且相邻图像块之间具有重叠区域，重叠区域的宽度为8个像素区域。
S12、根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征。
在一些实施例中，上述步骤S12(根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征)包括：
根据分解所述待超分视频中的其它图像帧得到的图像块，分别获取所述多个图像块中每一个图像块的超分特征，以及对所述多个图像块中每一个图像块的超分特征合并，以获取所述目标图像帧的超分特征。
例如：当待超分视频包括n个图像帧，目标图像帧为待超分视频的第t个图像帧，且待超分视频的各个图像帧均被分解为N个图像块时，分解所述待超分视频中的其它图像帧得到的图像块，包括：图像块图像块图像块图像块图像块图像块图像块图像块图像块图像块图像块分解所述待超分视频目标图像帧得到的图像块，包括：图像块图像块图像块其中，图像块表示分解待超分视频的第j个视频帧得到的第i个图像块。因此可以根据图像块图像块图像块图像块图像块图像块图像块图像块图像块图像块图像块分别获取图像块图像块图像块的超分特征，然后对图像块图像块图像块的超分特征合并，获取第t个图像帧的超分特征。
S13、根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
在一些实施例中，上述步骤S13(根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧)包括：
对所述目标图像帧的超分特征和所述目标图像帧的特征进行相加融合，获取所述目标图像帧对应的超分图像帧。
本公开实施例提供的视频的超分辨率方法在对待超分视频的目标图像帧图像进行超分时，首先将待超分视频的目标图像帧分解为多个图像块，然后根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取，获取所述目标图像帧的超分特征，最后根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。相比于相关技术中依赖相邻图像帧提供的时域信息进行图像帧的超分，本公开实施例提供的超分辨率方法可以在对待超分视频的目标图像帧进行超分时，通过分解待超分视频中除目标视频帧以外的其它所有图像帧得到的图像块，为分解目标视频帧得到的图像块提供时域信息，进而获取目标图像帧的超分特征。因此，即使待超分视频中具有在相邻图像帧中静止的物体或背景，本公开实施例也能够利用非相邻图像帧为目标视频帧的各个图像块提供足够的时域信息，进而为目标视频帧提供足够的时域信息，因此本公开实施例能够提升视频的超分辨率效果。
作为对上述实施例的扩展和细化，本公开实施例提供了另一种视频超分辨率方法，参照图3所示，该方法包括如下步骤：
S301、将待超分视频的目标图像帧分解为多个图像块。
S302、获取所述多个图像块中的每一个图像块的后向特征。
其中，任一图像块的后向特征为分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中与该图像块对应的图像块的特征。
S303、获取所述多个图像块中的每一个图像块的前向特征。
其中，任一图像块的前向特征为分解所述待超分视频中位于所述目标图像帧之前的图像帧得到的图像块中与该图像块对应的图像块的特征。
S304、根据所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征。
即，融合所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征。
S305、根据所述多个图像块中的每一个图像块的前向特征，获取所述目标图像帧的前向特征。
即，融合所述多个图像块中的每一个图像块的前向特征，获取所述目标图像帧的前向 特征。
S306、根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征。
作为本公开实施例一种可选的实施方式，上述步骤S306(根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征)，包括如下步骤a和步骤b：
步骤a、合并所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的合并特征。
示例性的，可以将所述目标图像帧的后向特征和所述目标图像帧的前向特征在通道维度上串联，从而获取所述目标图像帧的合并特征。
步骤b、对所述目标图像帧的合并特征进行上采样，获取所述目标图像帧的超分特征。
需要说明的是，上述实施例中能够先合并所述目标图像帧的后向特征和前向特征，然后再对合并后的特征进行上采样获取，以所述目标图像帧的超分特征为例进行说明，但也可以先分别对所述目标图像帧的后向特征和前向特征进行上采样，然后再对上采样结果进行合并，以获取所述目标图像帧的超分特征。
S307、根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
本实施例提供的视频的超分辨率方法的实现原理和技术效果与图1所示视频的超分辨率方法的实现原理和技术效果类似，此处不再赘述。
作为对上述实施例进一步的扩展和细化，本公开实施例提供了另一种视频超分辨率方法，参照图4所示，该方法包括如下步骤：
S401、将待超分视频的目标图像帧分解为多个图像块。
S402、获取后向图像块池和后向特征池。
其中，所述后向图像块池包括所述多个图像块中的每一个图像块对应的后向图像块；任一图像块对应的后向图像块为基于预设选取规则从分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中选取的图像块；所述后向特征池包括所述后向图像块池中的每一个后向图像块的特征。
即，当将目标图像帧分解为N个图像块时，所述后向图像块池中包括N个后向图像块，所述后向特征池中包括N个特征；所述N个后向图像块与所述多个图像块一一对应，所述后向特征池中的N个特征分别为所述N个后向图像块的特征。
在一些实施例中，当目标图像帧为待超分视频的第t个图像帧时，对于所述多个图像块中的图像块基于预设选取规则从分解所述待超分视频的第t+1个图像帧至最后一个 图像帧得到的图像块中选取图像块对应的后向图像块的实现方式包括：首先，确定分解所述待超分视频的第t+1个图像帧至最后一个图像帧得到的图像块中与图像块位置相同的各个图像块，得到第一图像块集合然后从第一图像块集合 中选取能够为图像块提供最有效时域信息的图像块，作为图像块对应的后向图像块。
S403、根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征。
S404、获取前向图像块池和前向特征池。
其中，所述前向图像块池包括所述多个图像块中的每一个图像块对应的前向图像块；任一图像块对应的前向图像块为基于预设选取规则，从分解所述待超分视频中，位于所述目标图像帧之前的图像帧得到的图像块中选取的图像块；所述前向特征池包括所述前向图像块池中的每一个前向图像块的特征
即，当将目标图像帧分解为N个图像块时，所述前向图像块池中包括N个前向图像块，所述前向特征池中包括N个特征；所述N个前向图像块与所述多个图像块一一对应，所述前向特征池中的N个特征分别为所述N个前向图像块的特征。
在一些实施例中，对于所述多个图像块中的图像块基于所述预设选取规则，从分解所述待超分视频的第1个图像帧至t-1个图像帧得到的图像块中，选取图像块对应的前向图像块的实现方式可以包括：首先，确定分解所述待超分视频的第1个图像帧至t-1个图像帧得到的图像块中与图像块位置相同的各个图像块，得到第二图像块集合 然后从第二图像块集合中选取能够为图像块提供最有效时域信息的图像块作为图像块对应的前向图像块。
S405、根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征。
S406、根据所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征。
S407、根据所述多个图像块中的每一个图像块的前向特征，获取所述目标图像帧的前向特征。
S408、根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征。
S409、根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
参照图5所示，图5为用于实现图4视频超分方法的视频超分网络的结构示意图。用于实现图4视频超分方法的视频超分网络包括：分解模块51、后向图像块池52、后向特征池53、后向特征传递模块54、前向图像块池55、前向特征池56、前向特征传递模块57、处理模块58以及生成模块59。
其中，分解模块51用于将待超分视频的目标图像帧It分解为多个图像块后向图像块池52用于存储多个图像块中的每一个图像块对应的后向图像块后向特征池53用于存储每一个后向图像块的特征后向特征传递模块54用于根据所述多个图像块后向图像块池中的后向图像块以及后向特征池中的特征获取所述目标图像帧的后向特征前向图像块池55用于存储多个图像块中的每一个图像块对应的前向图像块前向特征池56用于存储每一个前向图像块的特征前向特征传递模块57用于根据所述多个图像块前向图像块池中的前向图像块以及前向特征池中的特征获取所述目标图像帧的前向特征处理模块58用于根据所述目标图像帧的后向特征和前向特征获取所述目标图像帧的超分特征生成模块59用于根据所述目标图像帧的超分特征生成所述目标图像帧对应的超分图像帧Ot。
本实施例提供的视频的超分辨率方法的实现原理和技术效果与图1所示视频的超分辨率方法的实现原理和技术效果类似，此处不再赘述。
在图4所示实施例的基础上，参照图6所示，上述步骤S403(根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征)包括：
S61、获取所述后向图像块池中的每一个后向图像块的光流。
其中，任一后向图像块的光流为该后向图像块与该后向图像块在所述多个图像块中对应的图像块之间的光流。
作为本公开实施例一种可选的实施方式，上述步骤S61(获取所述后向图像块池中的每一个后向图像块的光流)的实现方式可以包括如下步骤611和步骤612：
步骤611、根据所述多个图像块生成第一图像块序列，并根据所述后向图像块池中的后向图像块生成第二图像块序列。
其中，任一图像块在所述第一图像块序列中的排序与该图像块对应的后向图像块在所述第二图像块序列中的排序相同。
本公开实施例中不限定所述多个图像块在第一图像块序列中的排列顺序，也不限定所 述后向图像块池中的后向图像块在第二图像块中的排列顺序，以任一图像块在所述第一图像块序列中的排序与该图像块对应的后向图像块在所述第二图像块序列中的排序相同为准。
示例性的，参照图7所示，多个图像块在第一图像块序列中的排列顺序依次为：所述后向图像块池中的后向图像块在第二图像块序列中的排列顺序依次为：多个图像块中的任一图像块在所述第一图像块序列中的排序与该图像块对应的后向图像块在所述第二图像块序列中的排序相同。
步骤612、将所述第一图像块序列和所述第二图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出获取所述后向图像块池中的每一个后向图像块的光流。
S62、根据所述后向图像块池中的每一个后向图像块的光流对所述后向特征池中每一个后向图像块的特征进行处理，获取所述后向图像块池中的每一个后向图像块的对齐特征。
即，根据所述后向图像块池中的每一个后向图像块的光流，将所述后向特征池中每一个后向图像块的特征与在所述多个图像块中对应的图像块的特征对齐，从而获取所述后向图像块池中的每一个后向图像块的对齐特征。
S63、根据所述后向图像块池中的每一个后向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的后向特征。
作为本公开实施例一种可选的实施方式，上述步骤S63(根据所述后向图像块池中的每一个后向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的后向特征)包括：
通过残差块(Residual block)对所述多个图像块中的每一个图像块和每一个图像块对应的后向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的后向特征。
参照图8所示，图6为用于实现根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征，以及根据所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征的后向特征获取模块的结构示意图，所述后向特征获取模块包括：光流预测网络模型81、特征对齐模块82、残差块83以及特征融合模块84。
其中，光流预测网络模型81根据输入的所述后向图像块池中后向图像块和分解待超分视频的目标图像帧得到的多个图像块输出所述后向图像块池中的每一 个后向图像块的光流特征对齐模块82用于根据所述后向图像块池中的每一个后向图像块的光流将所述后向池中的每一个后向图像块的特征与所述目标图像帧对齐，获取所述后向图像块池中的每一个后向图像块的对齐特征残差块83用于根据所述后向图像块池中的每一个后向图像块的对齐特征和所述多个图像块获取所述多个图像块中的每一个图像块的后向特征特征融合模块84用于融合所述多个图像块中的每一个图像块的后向特征生成所述目标图像帧的后向特征
作为本公开实施例一种可选的实施方式，本公开实施例提供的视频超分方法还包括：
根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新。
作为本公开实施例一种可选的实施方式，根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新的一种实现方式包括如下步骤1)和步骤2)：
步骤1)、判断所述后向图像块池中的每一个后向图像块的光流的绝对值是否大于预设阈值；
在上述步骤1)中若所述后向图像块池中的第一后向图像块的光流的绝对值大于所述预设阈值，则执行如下步骤2)。
步骤2)、将所述后向图像块池中的所述第一后向图像块替换为所述第一后向图像块在所述多个图像块中对应的图像块，以及将所述后向特征池中所述第一后向图像块的特征替换为所述第一后向图像块在所述多个图像块中对应的图像块的后向特征。
参照图9所示，图9为所述后向特征获取模块还用于根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新时，所述后向特征获取模块的结构示意图。用于实现所述后向特征获取模块包括：光流预测网络模型81、特征对齐模块82、残差块83、特征融合模块84以及更新模块85。
其中，光流预测网络模型81、特征对齐模块82、残差块83以及特征融合模块84的作用与图8中相同，再此不再赘述。更新模块85用于根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征将所述后向图像块池和所述后向特征池更新为后向图像块池和后向特征池并将后向图像块池和后向特征池作为对待超分视频的前一个图像帧进行处理时的后向图像块池和后向特征池。
在图6所示实施例的基础上，参照图10所示，上述步骤S405(根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征)包括：
S101、获取所述前向图像块池中的每一个前向图像块的光流。
其中，任一前向图像块的光流为该前向图像块与该前向图像块在所述多个图像块中对应的图像块之间的光流。
作为本公开实施例一种可选的实施方式，上述步骤S81(获取所述前向图像块池中的每一个前向图像块的光流)的实现方式可以包括如下步骤1011和步骤1012：
步骤1011、根据所述多个图像块生成第三图像块序列，并根据所述前向图像块池中的前向图像块生成第四图像块序列。
其中，任一图像块在所述第三图像块序列中的排序与该图像块对应的前向图像块在所述第四图像块序列中的排序相同。
本公开实施例中不限定所述多个图像块在第三图像块序列中的排列顺序，也不限定所述前向图像块池中的前向图像块在第四图像块中的排列顺序，以任一图像块在所述第三图像块序列中的排序与该图像块对应的前向图像块在所述第四图像块序列中的排序相同为准。
示例性的，参照图11所示，多个图像块在第三图像块序列中的排列顺序依次为：所述前向图像块池中的前向图像块在第四图像块序列中的排列顺序依次为：多个图像块中的任一图像块在所述第三图像块序列中的排序，与该图像块对应的前向图像块在所述第四图像块序列中的排序相同。
步骤1012、将所述第三图像块序列和所述第四图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出，获取所述前向图像块池中的每一个前向图像块的光流。
S102、根据所述前向图像块池中的每一个前向图像块的光流，对所述前向特征池中每一个前向图像块的特征进行处理，获取所述前向图像块池中的每一个前向图像块的对齐特征。
即，根据所述前向图像块池中的每一个前向图像块的光流，将所述前向特征池中每一个前向图像块的特征与在所述多个图像块中对应的图像块的特征对齐，从而获取所述前向图像块池中的每一个前向图像块的对齐特征。
S103、根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的前向特征。
作为本公开实施例一种可选的实施方式，上述步骤S83(根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的前向特征)包括：
通过残差块(Residual block)，对所述多个图像块中的每一个图像块和每一个图像块对应的前向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的前向特征。
参照图12所示，图12为用于实现根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征，以及根据所述多个图像块中的每一个图像块的前向特征获取所述目标图像帧的前向特征的前向特征获取模块的结构示意图，所述前向特征获取模块包括：光流预测网络模型121、特征对齐模块122、残差块123以及特征融合模块124。
其中，光流预测网络模型121根据输入的所述前向图像块池中前向图像块和分解待超分视频的目标图像帧得到的多个图像块输出所述前向图像块池中的每一个前向图像块的光流特征对齐模块122用于根据所述前向图像块池中的每一个前向图像块的光流将所述前向池中的每一个前向图像块的特征与所述目标图像帧对齐，获取所述前向图像块池中的每一个前向图像块的对齐特征残差块123用于根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块获取所述多个图像块中的每一个图像块的前向特征特征融合模块124用于融合所述多个图像块中的每一个图像块的前向特征生成所述目标图像帧的前向特征
作为本公开实施例一种可选的实施方式，本公开实施例提供的视频超分方法还包括：
根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征，对所述前向图像块池和所述前向特征池进行更新。
作为本公开实施例一种可选的实施方式，根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征，对所述前向图像块池和所述前向特征池进行更新的一种实现方式，包括如下步骤Ⅰ和步骤Ⅱ：
步骤Ⅰ、判断所述前向图像块池中的每一个前向图像块的光流的绝对值是否大于预设 阈值；
在上述步骤Ⅰ中，若所述前向图像块池中的第一前向图像块的光流的绝对值大于所述预设阈值，则执行如下步骤Ⅱ。
步骤Ⅱ、将所述前向图像块池中的所述第一前向图像块替换为所述第一前向图像块在所述多个图像块中对应的图像块，以及将所述前向特征池中所述第一前向图像块的特征替换为所述第一前向图像块在所述多个图像块中对应的图像块的前向特征。
参照图13所示，图13为所述前向特征获取模块，还用于根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征，对所述前向图像块池和所述前向特征池进行更新时，所述前向特征获取模块的结构示意图。所述前向特征获取模块包括：光流预测网络模型121、特征对齐模块122、残差块123、特征融合模块124以及更新模块125。
其中，光流预测网络模型121、特征对齐模块122、残差块123以及特征融合模块124的作用与图12中相同，再此不再赘述。更新模块125用于根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征将所述前向图像块池和所述前向特征池更新为前向图像块池和前向特征池并将前向图像块池和前向特征池作为对待超分视频的后一个图像帧进行处理时的前向图像块池和前向特征池。
进一步的，参照图14所示，图14为本公开实施例提供的视频超分网络的网络结构图。参照图14所示，待超分视频的目标图像帧使用第t+1个图像帧更新的后向图像块池和后向特征池以及第t-1个图像帧更新的前向图像块池和前向特征池完成超分，并对再次对后向图像块池和后向特征池进行更新，以及将目标图像帧更新后的后向图像块池和后向特征池作为后一个图像帧(第t+1个图像帧)的后向图像块池和后向特征池将目标图像帧更新后的前向图像块池和前向特征池作为前一个图像帧(第t-1个图像帧)的前向图像块池和前向特征池
基于同一发明构思，作为对上述方法的实现，本公开实施例还提供了一种视频的超分辨率装置，该装置实施例与前述方法实施例对应，为便于阅读，本装置实施例不再对前述方法实施例中的细节内容进行逐一赘述，但应当明确，本实施例中的视频的超分辨率装置能够对应实现前述方法实施例中的全部内容。
本公开实施例提供了一种视频的超分辨率装置，图15为该视频的超分辨率装置的结构示意图，如图15所示，该视频的超分辨率装置1500包括：
图像分解模块151，用于将待超分视频的目标图像帧分解为多个图像块；
特征获取模块152，用于根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征；
图像生成模块153，用于根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
作为本公开实施例一种可选的实施方式，参照图16所示，所述特征获取模块152，包括：
后向特征获取单元1521，用于获取所述多个图像块中的每一个图像块的后向特征，任一图像块的后向特征为分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中与该图像块对应的图像块的特征；
前向特征获取单元1522，用于获取所述多个图像块中的每一个图像块的前向特征，任一图像块的前向特征为分解所述待超分视频中位于所述目标图像帧之前的图像帧得到的图像块中与该图像块对应的图像块的特征；
第一特征合并单元1523，用于根据所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征；
第二特征合并单元1524，用于根据所述多个图像块中的每一个图像块的前向特征，获取所述目标图像帧的前向特征；
特征融合单元1525，用于根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，具体用于获取后向图像块池和后向特征池，所述后向图像块池包括所述多个图像块中的每一个图像块对应的后向图像块；任一图像块对应的后向图像块为基于预设选取规则从分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中选取的图像块；所述后向特征池包括所述后向图像块池中的每一个后向图像块的特征；根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，具体用于获取所述后向图像块池中的每一个后向图像块的光流，任一后向图像块的光流为该后向图像块与该后向图像块在所述多个图像块中对应的图像块之间的光流；根据所述后向图像块池中的每一个后向图像块的光流，对所述后向特征池中每一个后向图像块的特征进行处理，获取所述后向图像块池中的每一个后向图像块的对齐特征；根据所述后向图像块池中的每 一个后向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的后向特征。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，具体用于根据所述多个图像块生成第一图像块序列，并根据所述后向图像块池中的后向图像块生成第二图像块序列；任一图像块在所述第一图像块序列中的排序与该图像块对应的后向图像块在所述第二图像块序列中的排序相同；将所述第一图像块序列和所述第二图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出获取所述后向图像块池中的每一个后向图像块的光流。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，具体用于通过残差块对所述多个图像块中的每一个图像块和每一个图像块对应的后向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的后向特征。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，还用于根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新。
作为本公开实施例一种可选的实施方式，所述后向特征获取单元1521，具体用于判断所述后向图像块池中的每一个后向图像块的光流的绝对值是否大于预设阈值；若所述后向图像块池中的第一后向图像块的光流的绝对值大于所述预设阈值，则将所述后向图像块池中的所述第一后向图像块替换为所述第一后向图像块在所述多个图像块中对应的图像块，以及将所述后向特征池中所述第一后向图像块的特征替换为所述第一后向图像块在所述多个图像块中对应的图像块的后向特征。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，具体用于获取前向图像块池和前向特征池，所述前向图像块池包括所述多个图像块中的每一个图像块对应的前向图像块；任一图像块对应的前向图像块为基于预设选取规则从分解所述待超分视频中位于所述目标图像帧之前的图像帧得到的图像块中选取的图像块；所述前向特征池包括所述前向图像块池中的每一个前向图像块的特征；根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，具体用于获取所述前向图像块池中的每一个前向图像块的光流，任一前向图像块的光流为该前向图像块与该前向图像块在所述多个图像块中对应的图像块之间的光流；根据所述前向图像块池 中的每一个前向图像块的光流对所述前向特征池中每一个前向图像块的特征进行处理，获取所述前向图像块池中的每一个前向图像块的对齐特征；根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的前向特征。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，具体用于根据所述多个图像块生成第三图像块序列，并根据所述前向图像块池中的前向图像块生成第四图像块序列；任一图像块在所述第三图像块序列中的排序与该图像块对应的前向图像块在所述第四图像块序列中的排序相同；将所述第三图像块序列和所述第四图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出获取所述前向图像块池中的每一个前向图像块的光流。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，具体用于通过残差块对所述多个图像块中的每一个图像块和每一个图像块对应的前向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的前向特征。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，还用于根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征对所述前向图像块池和所述前向特征池进行更新。
作为本公开实施例一种可选的实施方式，所述前向特征获取单元1522，具体用于判断所述前向图像块池中的每一个前向图像块的光流的绝对值是否大于预设阈值；若所述前向图像块池中的第一前向图像块的光流的绝对值大于所述预设阈值，则将所述前向图像块池中的所述第一前向图像块替换为所述第一前向图像块在所述多个图像块中对应的图像块，以及将所述前向特征池中所述第一前向图像块的特征替换为所述第一前向图像块在所述多个图像块中对应的图像块的前向特征。
作为本公开实施例一种可选的实施方式，所述特征处理单元153，具体用于合并所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的合并特征；对所述目标图像帧的合并特征进行上采样，获取所述目标图像帧的超分特征。
本实施例提供的视频的超分辨率装置可以执行上述方法实施例提供的视频的超分辨率方法，其实现原理与技术效果类似，此处不再赘述。
基于同一发明构思，本公开实施例还提供了一种电子设备。图17为本公开实施例提供的电子设备的结构示意图，如图17所示，本实施例提供的电子设备包括：存储器171和 处理器172，所述存储器171用于存储计算机程序；所述处理器172用于在调用计算机程序时执行上述实施例提供的视频的超分辨率方法。
基于同一发明构思，本公开实施例还提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，当计算机程序被处理器执行时，使得所述计算设备实现上述实施例提供的视频的超分辨率方法。
基于同一发明构思，本公开实施例还提供了一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算设备实现上述实施例提供的视频的超分辨率方法。
本领域技术人员应明白，本公开的实施例可提供为方法、系统、或计算机程序产品。因此，本公开可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本公开可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质上实施的计算机程序产品的形式。
处理器可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。存储器是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动存储介质。存储介质可以由任何方法或技术来实现信息存储，信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。根据本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
最后应说明的是：以上各实施例仅用以说明本公开的技术方案，而非对其限制；尽管 参照前述各实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本公开各实施例技术方案的范围。

Claims (20)

  1. 一种视频的超分辨率方法，包括：
    将待超分视频的目标图像帧分解为多个图像块；
    根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征；
    根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
  2. 根据权利要求1所述的视频的超分辨率方法，其中，所述根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块，获取所述目标图像帧的超分特征，包括：
    获取所述多个图像块中的每一个图像块的后向特征，其中，任一图像块的后向特征为分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中与该图像块对应的图像块的特征；
    获取所述多个图像块中的每一个图像块的前向特征，其中，任一图像块的前向特征为分解所述待超分视频中位于所述目标图像帧之前的图像帧得到的图像块中与该图像块对应的图像块的特征；
    根据所述多个图像块中的每一个图像块的后向特征，获取所述目标图像帧的后向特征；
    根据所述多个图像块中的每一个图像块的前向特征，获取所述目标图像帧的前向特征；
    根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征。
  3. 根据权利要求2所述的视频的超分辨率方法，其中，所述获取所述多个图像块中的每一个图像块的后向特征，包括：
    获取后向图像块池和后向特征池，其中，所述后向图像块池包括所述多个图像块中的每一个图像块对应的后向图像块，任一图像块对应的后向图像块为基于预设选取规则，从分解所述待超分视频中位于所述目标图像帧之后的图像帧得到的图像块中选取的图像块，所述后向特征池包括所述后向图像块池中的每一个后向图像块的特征；
    根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征。
  4. 根据权利要求3所述的视频的超分辨率方法，其中，所述根据所述后向图像块池和所述后向特征池，获取所述多个图像块中的每一个图像块的后向特征，包括：
    获取所述后向图像块池中的每一个后向图像块的光流，任一后向图像块的光流为该后向图像块与该后向图像块在所述多个图像块中对应的图像块之间的光流；
    根据所述后向图像块池中的每一个后向图像块的光流，对所述后向特征池中每一个后向图像块的特征进行处理，获取所述后向图像块池中的每一个后向图像块的对齐特征；
    根据所述后向图像块池中的每一个后向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的后向特征。
  5. 根据权利要求4所述的视频的超分辨率方法，其中，所述获取所述后向图像块池中的每一个后向图像块的光流，包括：
    根据所述多个图像块生成第一图像块序列，并根据所述后向图像块池中的后向图像块，生成第二图像块序列，其中，任一图像块在所述第一图像块序列中的排序与该图像块对应的后向图像块在所述第二图像块序列中的排序相同；
    将所述第一图像块序列和所述第二图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出，获取所述后向图像块池中的每一个后向图像块的光流。
  6. 根据权利要求4或5所述的视频的超分辨率方法，其中，所述根据所述后向图像块池中的每一个后向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的后向特征，包括：
    通过残差块对所述多个图像块中的每一个图像块和每一个图像块对应的后向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的后向特征。
  7. 根据权利要求3-6任一项所述的视频的超分辨率方法，还包括：
    根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新。
  8. 根据权利要求7所述的视频的超分辨率方法，其中，所述根据所述多个图像块和所述多个图像块中的每一个图像块的后向特征对所述后向图像块池和所述后向特征池进行更新，包括：
    判断所述后向图像块池中的每一个后向图像块的光流的绝对值是否大于预设阈值；
    若所述后向图像块池中的第一后向图像块的光流的绝对值大于所述预设阈值，则将所述后向图像块池中的所述第一后向图像块替换为所述第一后向图像块在所述多个图像块中对应的图像块，以及将所述后向特征池中所述第一后向图像块的特征替换为所述第一后向图像块在所述多个图像块中对应的图像块的后向特征。
  9. 根据权利要求2-8任一项所述的视频的超分辨率方法，其中，所述获取所述多个图像块中的每一个图像块的前向特征，包括：
    获取前向图像块池和前向特征池，其中，所述前向图像块池包括所述多个图像块中的每一个图像块对应的前向图像块，任一图像块对应的前向图像块为基于预设选取规则从分解所述待超分视频中位于所述目标图像帧之前的图像帧得到的图像块中选取的图像块，所述前向特征池包括所述前向图像块池中的每一个前向图像块的特征；
    根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征。
  10. 根据权利要求9所述的视频的超分辨率方法，其中，所述根据所述前向图像块池和所述前向特征池，获取所述多个图像块中的每一个图像块的前向特征，包括：
    获取所述前向图像块池中的每一个前向图像块的光流，其中，任一前向图像块的光流为该前向图像块与该前向图像块在所述多个图像块中对应的图像块之间的光流；
    根据所述前向图像块池中的每一个前向图像块的光流，对所述前向特征池中每一个前向图像块的特征进行处理，获取所述前向图像块池中的每一个前向图像块的对齐特征；
    根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的前向特征。
  11. 根据权利要求10所述的视频的超分辨率方法，其中，所述获取所述前向图像块池中的每一个前向图像块的光流，包括：
    根据所述多个图像块生成第三图像块序列，并根据所述前向图像块池中的前向图像块生成第四图像块序列，其中，任一图像块在所述第三图像块序列中的排序与该图像块对应的前向图像块在所述第四图像块序列中的排序相同；
    将所述第三图像块序列和所述第四图像块序列输入光流预测网络模型，并根据所述光流预测网络模型的输出获取所述前向图像块池中的每一个前向图像块的光流。
  12. 根据权利要求10或11所述的视频的超分辨率方法，其中，所述根据所述前向图像块池中的每一个前向图像块的对齐特征和所述多个图像块，获取所述多个图像块中的每一个图像块的前向特征，包括：
    通过残差块对所述多个图像块中的每一个图像块和每一个图像块对应的前向图像块的对齐特征进行处理，获取所述多个图像块中的每一个图像块的前向特征。
  13. 根据权利要求9-12任一项所述的视频的超分辨率方法，还包括：
    根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征，对所述前向图像块池和所述前向特征池进行更新。
  14. 根据权利要求13所述的视频的超分辨率方法，其中，所述根据所述多个图像块和所述多个图像块中的每一个图像块的前向特征对所述前向图像块池和所述前向特征池进行更新，包括：
    判断所述前向图像块池中的每一个前向图像块的光流的绝对值是否大于预设阈值；
    在所述前向图像块池中的第一前向图像块的光流的绝对值大于所述预设阈值的情况下，将所述前向图像块池中的所述第一前向图像块替换为所述第一前向图像块在所述多个图像块中对应的图像块，以及将所述前向特征池中所述第一前向图像块的特征替换为所述第一前向图像块在所述多个图像块中对应的图像块的前向特征。
  15. 根据权利要求2-14任一项所述的视频的超分辨率方法，其中，所述根据所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的超分特征，包括：
    合并所述目标图像帧的后向特征和前向特征，获取所述目标图像帧的合并特征；
    对所述目标图像帧的合并特征进行上采样，获取所述目标图像帧的超分特征。
  16. 一种视频的超分辨率装置，包括：
    图像分解模块，用于将待超分视频的目标图像帧分解为多个图像块；
    特征获取模块，用于根据所述多个图像块和分解所述待超分视频中的其它图像帧得到的图像块的特征，获取所述多个图像块中的每一个图像块的超分特征；
    特征处理模块，用于根据所述多个图像块中的每一个图像块的超分特征，获取所述目标图像帧的超分特征；
    图像生成模块，用于根据所述目标图像帧的超分特征，获取所述目标图像帧对应的超分图像帧。
  17. 一种电子设备，包括：
    存储器和处理器；
    所述存储器用于存储指令；
    所述处理器用于在执行所述指令时，使得所述电子设备实现权利要求1-15任一项所述的视频的超分辨率方法。
  18. 一种计算机可读存储介质，所述计算机可读存储介质上存储有指令，当所述指令被处理器执行时，实现权利要求1-15任一项所述的视频的超分辨率方法。
  19. 一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机实现如权利要求1-15任一项所述的视频的超分辨率方法。
  20. 一种计算机程序，包括：
    指令，所述指令当由处理器执行时使所述处理器执行如权利要求1-15任一项所述的视频的超分辨率方法。
PCT/CN2023/081794 2022-03-17 2023-03-16 视频的超分辨率方法及装置 Ceased WO2023174355A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/725,722 US20250061541A1 (en) 2022-03-17 2023-03-16 Video super-resolution method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210265574.6A CN116797453A (zh) 2022-03-17 2022-03-17 一种视频的超分辨率方法及装置
CN202210265574.6 2022-03-17

Publications (1)

Publication Number Publication Date
WO2023174355A1 true WO2023174355A1 (zh) 2023-09-21

Family

ID=88022341

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/081794 Ceased WO2023174355A1 (zh) 2022-03-17 2023-03-16 视频的超分辨率方法及装置

Country Status (3)

Country Link
US (1) US20250061541A1 (zh)
CN (1) CN116797453A (zh)
WO (1) WO2023174355A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240068288A (ko) * 2022-11-10 2024-05-17 삼성전자주식회사 적응적 슈퍼 샘플링 방법 및 적응적 슈퍼 샘플링 장치

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110037894A1 (en) * 2009-08-11 2011-02-17 Google Inc. Enhanced image and video super-resolution processing
CN106600536A (zh) * 2016-12-14 2017-04-26 同观科技（深圳）有限公司 一种视频图像超分辨率重建方法及装置
CN111583138A (zh) * 2020-04-27 2020-08-25 Oppo广东移动通信有限公司 视频增强方法及装置、电子设备、存储介质
CN113592709A (zh) * 2021-02-19 2021-11-02 腾讯科技（深圳）有限公司 图像超分处理方法、装置、设备及存储介质
CN113747242A (zh) * 2021-09-06 2021-12-03 海宁奕斯伟集成电路设计有限公司 图像处理方法、装置、电子设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110037894A1 (en) * 2009-08-11 2011-02-17 Google Inc. Enhanced image and video super-resolution processing
CN106600536A (zh) * 2016-12-14 2017-04-26 同观科技（深圳）有限公司 一种视频图像超分辨率重建方法及装置
CN111583138A (zh) * 2020-04-27 2020-08-25 Oppo广东移动通信有限公司 视频增强方法及装置、电子设备、存储介质
CN113592709A (zh) * 2021-02-19 2021-11-02 腾讯科技（深圳）有限公司 图像超分处理方法、装置、设备及存储介质
CN113747242A (zh) * 2021-09-06 2021-12-03 海宁奕斯伟集成电路设计有限公司 图像处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN116797453A (zh) 2023-09-22
US20250061541A1 (en) 2025-02-20

Similar Documents

Publication Publication Date Title
US10861133B1 (en) Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium
CN113628115B (zh) 图像重建的处理方法、装置、电子设备和存储介质
WO2021046953A1 (zh) 一种图像的超分辨率处理方法、系统及设备
US20240265494A1 (en) Realtime video super resolution using motion vectors
US20250191122A1 (en) Image generation method for eliminating splicing seams, computer device and storage medium
WO2021218414A1 (zh) 视频增强方法及装置、电子设备、存储介质
WO2023174355A1 (zh) 视频的超分辨率方法及装置
Hung et al. Image interpolation using convolutional neural networks with deep recursive residual learning
Wang et al. Lightweight interactive feature inference network for single-image super-resolution
CN117808857A (zh) 一种自监督360°深度估计方法、装置、设备及介质
CN110399881B (zh) 一种端到端的基于双目立体图像的质量增强方法及装置
Huang et al. Feature enhanced cascading attention network for lightweight image super-resolution
CN115731313A (zh) Svg格式的图片的处理方法、装置、设备、介质及产品
Hu et al. Adaptive region aggregation for multi‐view stereo matching using deformable convolutional networks
CN119625137A (zh) 图像扩展方法、装置、可读存储介质和程序产品
Wang et al. Enhancing unsupervised shadow removal via multi-intensity shadow generation and diffusion modeling: D. Wang et al.
CN118297979A (zh) 图像处理方法、装置、电子设备、存储介质和程序产品
CN113077481B (zh) 一种图像处理方法、装置、计算机设备和存储介质
WO2024061064A1 (zh) 展示效果处理方法、装置、电子设备及存储介质
CN116801119A (zh) 一种图像处理方法、装置、设备及存储介质
WO2023125522A1 (zh) 一种图像处理方法及装置
CN115423858A (zh) 一种单目深度估计方法、装置、电子设备及存储介质
CN111369425B (zh) 图像处理方法、装置、电子设备和计算机可读介质
CN115147281A (zh) 一种图像参数调整方法、装置、设备及存储介质
WO2023174416A1 (zh) 视频的超分辨率方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23769857

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18725722

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07/01/2025)

122 Ep: pct application non-entry in european phase

Ref document number: 23769857

Country of ref document: EP

Kind code of ref document: A1