CN103402098B - A kind of video frame interpolation method based on image interpolation - Google Patents
A kind of video frame interpolation method based on image interpolation Download PDFInfo
- Publication number
- CN103402098B CN103402098B CN201310361710.2A CN201310361710A CN103402098B CN 103402098 B CN103402098 B CN 103402098B CN 201310361710 A CN201310361710 A CN 201310361710A CN 103402098 B CN103402098 B CN 103402098B
- Authority
- CN
- China
- Prior art keywords
- interpolation
- block
- frame
- motion vector
- motion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Abstract
The invention discloses a kind of video frame interpolation method based on image interpolation, the present invention is directed to the motion estimation result that two-way block search can not obtain, and unidirectional block search can produce " overlapping " and " empty " problem in the interpolation frame of Interpolation compensation, devise a kind of video interleave scheme based on image mending.First, between consecutive frame, carry out one-way movement estimation and the motion vector field smothing filtering of three grades of variable-block sizes；Then, Interpolation compensation obtains initial interpolation frame, and is marked " empty " therein and " overlapping " region, generates area mask to be repaired；Finally, according to mask, initial interpolation frame is carried out image mending, obtain final interpolation frame.The present invention can be effectively improved the blocking effect reduced in interpolation frame, improve picture quality, it is easy to accomplish, treatment effeciency high, it is adaptable to change on the real time frame rate of HD video.
Description
Technical field
The invention belongs to Video processing and transmission field, be specifically related to a kind of video frame interpolation method based on image interpolation.
Background technology
Along with digital TV in high resolution (HDTV) and the development of high-end multimedia information system, people's visual effect to video source
Require more and more higher, it is therefore desirable to improve the frame per second of existing video program source to reach more preferable visual effect；Meanwhile, to the greatest extent
Manage the encoding and decoding technique in current transmission of video and can obtain the highest compression ratio, but in order to adapt to some network bandwidths
Limit, generally the spatial and temporal resolution of video signal being reduced making data volume less, can be come by coding side frame-skipping real in the time domain
Existing.So, the low frame-rate video in decoding end will necessarily cause discontinuous, the degeneration of picture quality of moving, especially quickly
Motion and complex scene become apparent from.To this end, switch technology on video interleave, i.e. frame per second can be used to recover in decoding end
Original frame per second is to improve the subjective vision effect of video image, and this technology can also be used for the conversion between different frame per second video format.
Changing (Frame Rate Up-Conversion, FRUC) technology in video interleave i.e. frame per second, it is by decoding end phase
Insert the mode of intermediate frame between adjacent frame, video is promoted paramount frame per second from low frame per second, it is achieved the conversion between different frame per second.With
People video effect requirement is improved constantly, need bigger video resolution (and display size), faster frame per second,
These demands make the speed to processing module more and more higher with the requirement of storage space, process band in real time to video frequency doubling system
Carry out challenge.
Owing to the repetition of simple frame, frame averagely can produce motion jitter and obscure, in frame per second, conversion is usual the most in actual applications
The method used is motion compensated interpolation based on Block-matching (Motion-Compensated Interpolation, MCI), the method
The interpolation frame quality obtained depends on the precision of estimation of motion vectors.
But, traditional Block-matching carries out being before and after two frame " unidirectional " search during estimation so that of middle interpolation frame
Pixel may have multiple movement locus to pass through, or does not has movement locus to pass through, and produce " overlapping " and " empty " respectively and ask
Topic, asks for an interview Fig. 1.
This technical problem obtains certain solution because of " bi-directional motion estimation " technology, and " bi-directional motion estimation " is with interpolation
Block in frame, as positioning datum, is searched for match block symmetrically, so would not be deposited when Interpolation compensation intermediate frame front and back in frame
The pixel " overlapping " brought in one-way movement method of estimation and " empty "；While it is true, " bi-directional motion estimation " is the most not
Motion vector accurately can be obtained so that after compensating interpolation, blocking effect occurs in interpolation frame.
Grinding of the view interpolation technology can be effectively improved the blocking effect reduced in interpolation frame, improving picture quality, always this area
Study carefully emphasis.
Summary of the invention
For the problems referred to above, the present invention proposes a kind of to be effectively improved the blocking effect reduced in interpolation frame, improve picture quality
View interpolation method based on image interpolation.
The technical solution adopted in the present invention is: a kind of video frame interpolation method based on image interpolation, it is characterised in that include
Following steps:
Step 1: for present frame, with the former frame in adjacent two frames as reference frame, by the unidirectional initial fortune of 16 × 16 block search
Dynamic estimation, obtains the initial motion vectors of present frame；
Step 2: the initial motion vectors of described present frame is carried out the disposal of gentle filter, obtains first motion vector field；
Step 3: judge: whether the block that described initial motion is estimated meets Block-matching search criteria？
If meeting, first with 8 × 8 pieces of these motion vectors of direct succession, then with 4 × 4 pieces of these motion vectors of direct succession,
Finally redirect the step 8 described in execution；
If being unsatisfactory for, then the block being unsatisfactory for Block-matching search criteria is split, carry out second degree of motion estimation, obtain the second time of these blocks
The motion vector estimated；
Step 4: the motion vector estimating described second time carries out the disposal of gentle filter, obtains second level motion vector field；
Step 5: judge: whether the block that described second degree of motion is estimated meets Block-matching search criteria？
If meeting, then with 4 × 4 pieces of these motion vectors of direct succession, and redirect the step 8 described in execution；
If being unsatisfactory for, then the block that will be unsatisfactory for Block-matching search criteria continues to split, and carries out three grades of estimation, obtains the of these blocks
The motion vector that three times are estimated；
Step 6: the motion vector estimating described third time carries out the disposal of gentle filter, obtains third level motion vector field；
Step 7: judge: whether the block of three grades of described estimation meets Block-matching search criteria？
If meeting, then it is final motion vector field by described third level motion vector field, and redirects the step 8 described in execution；
If being unsatisfactory for, then estimating failure, its motion vector directly distributes according to block motion vector about；
Step 8: present frame is carried out Interpolation compensation according to described final motion vector field, obtains initial interpolation frame, and to " empty
Hole " and the pixel in " overlapping " region be marked, obtain area mask to be repaired；
Step 9: according to described area mask to be repaired, initial interpolation frame is carried out image mending, obtain final interpolation frame.
As preferably, described in step 1 based on block search, use three-dimensional recursive search algorithm.
As preferably, the second degree of motion described in step 3 is estimated, uses 8 × 8 pieces and inherits the result that initial motion is estimated,
To the block in previous step again estimation.
As preferably, three described in step 5 grade estimation, use 4 × 4 pieces and inherit the results that second degree of motion are estimated,
To the block in previous step again estimation.
As preferably, described the disposal of gentle filter, the window using 3 × 3 carries out medium filtering process.
As preferably, described in step 9, initial interpolation frame is carried out image mending, use fast marching algorithms.
Relative to prior art, the present invention can be effectively improved blocking effect, the raising picture quality reduced in interpolation frame, it is easy to accomplish,
Treatment effeciency is high, it is adaptable to change on the real time frame rate of HD video.
Accompanying drawing explanation
" empty " and " overlapping " schematic diagram that Interpolation compensation produces is estimated in the one-way movement of Fig. 1: prior art of the present invention.
The algorithm flow chart of Fig. 2: the specific embodiment of the invention.
The 3-D RS spatio-temporal prediction candidate vector position view of Fig. 3: the specific embodiment of the invention.
Three grades of estimation piecemeal schematic diagrams of Fig. 4: the specific embodiment of the invention.
Mask figure is repaired in the cavity of Fig. 5: the specific embodiment of the invention, overlapping region.
Detailed description of the invention
Below with reference to the drawings and specific embodiments, the present invention is further elaborated.
Asking for an interview Fig. 2, the technical solution adopted in the present invention is: a kind of video frame interpolation method based on image interpolation, including following
Step:
Step 1: for present frame, using the former frame in adjacent two frames as reference frame, utilize three-dimensional recursive search (3-D
Recursive Search, 3-D RS) algorithm, carry out one-way movement estimation with 16 × 16 pieces, the initial motion obtaining present frame is vowed
Amount.
Step 2: the initial motion vectors of present frame is carried out medium filtering process, obtains first motion vector field, and to being more than
The block of absolute error and threshold values (threshold_SAD) is marked.
Step 3: judge: whether the block that initial motion is estimated meets Block-matching search criteria？
If meeting, first with 8 × 8 pieces of these motion vectors of direct succession, then vow with these motions of 4 × 4 pieces of direct successions
Amount, finally redirects execution step 8；
If being unsatisfactory for, then the block that will be unsatisfactory for Block-matching search criteria splits, and carries out second degree of motion estimation, obtains the of these blocks
The motion vector of quadratic estimate.
Step 4: the motion vector estimating second time carries out medium filtering process, obtains second level motion vector field, and to greatly
Block labelling again in 1/4th absolute errors and threshold values (threshold_SAD/4).
Step 5: judge: whether the block that second degree of motion is estimated meets Block-matching search criteria？
If meeting, then with 4 × 4 pieces of these motion vectors of direct succession, and redirect execution step 8；
If being unsatisfactory for, then continue to split by the block being unsatisfactory for Block-matching search criteria, carry out three grades of estimation, obtain these blocks
Third time estimate motion vector；
Step 6: the motion vector estimating third time carries out medium filtering process, obtains third level motion vector field, and to greatly
The block labelling again of Yu Shiliu/mono-absolute error and threshold values (threshold_SAD/16).
Step 7: judge: whether the block of three grades of estimation meets Block-matching search criteria？
If meeting, then it is final motion vector field by third level motion vector field, and redirects execution step 8；
If being unsatisfactory for, then estimating failure, it is more than 1/16th absolute error and the block of threshold values (threshold_SAD/16)
Motion vector is directly distributed by the result of block motion vector medium filtering about.
Step 8: present frame is carried out Interpolation compensation according to final motion vector field, obtains initial interpolation frame, and to " empty
Hole " and the pixel in " overlapping " region be marked, obtain area mask to be repaired.
Step 9: utilize Fast marching image mending algorithm, according to area mask to be repaired, initial interpolation frame is repaired,
To final interpolation frame.
Interpolation frame is written between original two frames by the present invention, it is achieved change in video frame rate 2 frequency multiplication.
The block matching algorithms the most ripe in coding and decoding video field.But, in video interleave is applied, require movable information
Can reflect the real motion of object, the motion vector only determined by matching error is likely completely different with real motion, makes
The interpolation frame of motion compensation produces the image information not being inconsistent with subjective vision.Owing to the motion of object video has seriality, therefore
Temporal correlation is there is also between motion vector.
Ask for an interview Fig. 3, three-dimensional recursive search based on Block-matching of the present invention (3-D Recursive Search, 3-D RS)
It is the searching algorithm utilizing sports ground temporal correlation to predict, the most smooth motion vector field can not only be obtained, also reduce simultaneously
Computing cost.If (x, t) is the brightness of pixel x in t frame to F, and T is frame number interval, and B (X) represents in reference frame
Block, X is the position of block, and C is candidate vector, and (X, t) for updating vector, α is constant to U, then in 3-D RS algorithm
Motion estimation error criterion calculating formula is:
Wherein, candidate vector C be current block room and time adjacent block motion vector set in element.Due to currently
The motion vector of block and its time, spatial neighboring blocks motion vector are close, but the most identical, if all of candidate vector
It both is from the original value of neighboring block motion vector, does not then meet practical situation.Therefore, to original motion vector plus a renewal
Vector, the candidate motion vector obtained will be closer to actual, more accurate.
Ask for an interview Fig. 4, when a block comprises multiple different motion object, particularly at movement edge, will be unable to correctly be retouched
State the estimated vector of motion.Therefore the present invention uses multistage Block-matching, the search of " from coarse to fine ", the motion vector of upper level
Next stage can be traveled to, and the estimated value of next stage is the further refinement to upper level estimated result.The present invention uses three grades
Estimation, carries out medium filtering to vector field with the window of 3 × 3 after the estimation of every one-level, is more smoothed
Motion vector field, then useCarry out Interpolation compensation, during wherein s is interpolation frame
Pixel, v is motion vector.
Asking for an interview Fig. 5, in initial interpolation frame, " empty " and " overlapping " region is marked generation mask；Finally, originally
Invention introduces image mending these pixels of technology completion.Owing to the most only need to fill up some narrow and small regions, the present invention adopts
With fast marching algorithms (Fast marching method, FMM), processing speed is fast.
These are only presently preferred embodiments of the present invention, be not intended to limit protection scope of the present invention, therefore, all at this
Any modification, equivalent substitution and improvement etc. made within bright spirit and principle, should be included in protection scope of the present invention it
In.
Claims (6)
1. a video frame interpolation method based on image interpolation, it is characterised in that comprise the following steps:
Step 1: for present frame, with the former frame in adjacent two frames as reference frame, is estimated by the unidirectional initial motion of 16 × 16 block search, obtains the initial motion vectors of present frame；
Step 2: the initial motion vectors of described present frame is carried out the disposal of gentle filter, obtains first motion vector field；
Step 3: judge: whether the block that described initial motion is estimated meets Block-matching search criteria；
If meeting, first with 8 × 8 pieces of these motion vectors of direct succession, then with 4 × 4 pieces of these motion vectors of direct succession, finally redirect the step 8 described in execution；
If being unsatisfactory for, then the block being unsatisfactory for Block-matching search criteria is split, carry out second degree of motion estimation, obtain the motion vector that the second time of these blocks is estimated；
Step 4: the motion vector estimating described second time carries out the disposal of gentle filter, obtains second level motion vector field；
Step 5: judge: whether the block that described second degree of motion is estimated meets Block-matching search criteria；
If meeting, then with 4 × 4 pieces of these motion vectors of direct succession, and redirect the step 8 described in execution；
If being unsatisfactory for, then continue to split by the block being unsatisfactory for Block-matching search criteria, carry out three grades of estimation, obtain the motion vector that the third time of these blocks is estimated；
Step 6: the motion vector estimating described third time carries out the disposal of gentle filter, obtains third level motion vector field；
Step 7: judge: whether the block of three grades of described estimation meets Block-matching search criteria；
If meeting, then it is final motion vector field by described third level motion vector field, and redirects the step 8 described in execution；
If being unsatisfactory for, then estimating failure, its motion vector directly distributes according to block motion vector about；
Step 8: present frame is carried out Interpolation compensation according to described final motion vector field, obtains initial interpolation frame, and is marked the pixel in " empty " and " overlapping " region, obtain area mask to be repaired；
Step 9: according to described area mask to be repaired, initial interpolation frame is carried out image mending, obtain final interpolation frame.
Video frame interpolation method based on image interpolation the most according to claim 1, it is characterised in that: described in step 1 based on block search, use three-dimensional recursive search algorithm.
Video frame interpolation method based on image interpolation the most according to claim 1, it is characterised in that: the second degree of motion described in step 3 is estimated, uses 8 × 8 pieces and inherits the result that initial motion is estimated, to the block in previous step again estimation.
Video frame interpolation method based on image interpolation the most according to claim 1, it is characterised in that: three described in step 5 grade estimation, use 4 × 4 pieces and inherit the result that second degree of motion is estimated, to the block in previous step again estimation.
Video frame interpolation method based on image interpolation the most according to claim 1, it is characterised in that: described the disposal of gentle filter, the window using 3 × 3 carries out medium filtering process.
Video frame interpolation method based on image interpolation the most according to claim 1, it is characterised in that: described in step 9, initial interpolation frame is carried out image mending, use fast marching algorithms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310361710.2A CN103402098B (en) | 2013-08-19 | 2013-08-19 | A kind of video frame interpolation method based on image interpolation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310361710.2A CN103402098B (en) | 2013-08-19 | 2013-08-19 | A kind of video frame interpolation method based on image interpolation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103402098A CN103402098A (en) | 2013-11-20 |
| CN103402098B true CN103402098B (en) | 2016-07-06 |
Family
ID=49565610
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310361710.2A Expired - Fee Related CN103402098B (en) | 2013-08-19 | 2013-08-19 | A kind of video frame interpolation method based on image interpolation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103402098B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016187776A1 (en) * | 2015-05-25 | 2016-12-01 | 北京大学深圳研究生院 | Video frame interpolation method and system based on optical flow method |
| CN106331723B (en) * | 2016-08-18 | 2019-12-13 | 上海交通大学 | Video frame rate up-conversion method and system based on motion region segmentation |
| CN107172426B (en) * | 2017-06-23 | 2019-10-11 | 西安电子科技大学 | Conversion method in the parallel frame per second of OpenCL based on double MIC |
| CN107396111B (en) * | 2017-07-13 | 2020-07-14 | 河北中科恒运软件科技股份有限公司 | Automatic video frame interpolation compensation method and system in mediated reality |
| CN107657591A (en) * | 2017-09-05 | 2018-02-02 | 维沃移动通信有限公司 | A kind of image processing method and mobile terminal |
| CN110896492B (en) * | 2018-09-13 | 2022-01-28 | 阿里巴巴（中国）有限公司 | Image processing method, device and storage medium |
| CN111741304A (en) * | 2019-03-25 | 2020-10-02 | 四川大学 | Method for combining frame rate up-conversion and HEVC (high efficiency video coding) based on motion vector refinement |
| CN110163892B (en) * | 2019-05-07 | 2023-06-20 | 国网江西省电力有限公司检修分公司 | Learning rate progressive updating method based on motion estimation interpolation and dynamic modeling system |
| CN111491204B (en) * | 2020-04-17 | 2022-07-12 | Oppo广东移动通信有限公司 | Video repair method, video repair device, electronic equipment and computer-readable storage medium |
| CN111586321B (en) * | 2020-05-08 | 2023-05-12 | Oppo广东移动通信有限公司 | Video generation method, device, electronic equipment and computer readable storage medium |
| CN113673588A (en) * | 2021-08-12 | 2021-11-19 | 连尚(北京)网络科技有限公司 | Method, apparatus, medium, and program product for video classification |
| CN114372936A (en) * | 2022-01-10 | 2022-04-19 | 中国人民解放军国防科技大学 | Infrared dim target detection method based on image inpainting technology |
| CN114554248B (en) * | 2022-04-27 | 2022-08-02 | 杭州微帧信息科技有限公司 | Video frame interpolation method based on neural network |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103209334A (en) * | 2013-03-18 | 2013-07-17 | 中山大学 | Virtual viewpoint synthesis and void repairing method for 2.5D videos to multi-view (three-dimensional) 3D videos |
-
2013
- 2013-08-19 CN CN201310361710.2A patent/CN103402098B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103209334A (en) * | 2013-03-18 | 2013-07-17 | 中山大学 | Virtual viewpoint synthesis and void repairing method for 2.5D videos to multi-view (three-dimensional) 3D videos |
Non-Patent Citations (3)
| Title |
|---|
| 侯正信.帧率提升中的三维递归搜索块匹配改进算法.《计算机工程与应用》.2011, * |
| 基于小波变换的图像修复算法;肖志云;《内蒙古工业大学学报》;20081231;第27卷(第4期);全文 * |
| 贾茜.基于三维递归搜索的多级运动估计视频帧率上转换算法.《电子与信息学报》.2012,第34卷(第10期), * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103402098A (en) | 2013-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103402098B (en) | A kind of video frame interpolation method based on image interpolation | |
| CN105517671B (en) | Video frame interpolation method and system based on optical flow method | |
| CN106210767B (en) | Video frame rate up-conversion method and system for intelligently improving motion fluency | |
| Wang et al. | Frame rate up-conversion using trilateral filtering | |
| US6625333B1 (en) | Method for temporal interpolation of an image sequence using object-based image analysis | |
| Jeong et al. | Motion-compensated frame interpolation based on multihypothesis motion estimation and texture optimization | |
| Lee et al. | Weighted-adaptive motion-compensated frame rate up-conversion | |
| CN101207707A (en) | System and method for advancing frame frequency based on motion compensation | |
| CN103220488A (en) | Up-conversion device and method of video frame rate | |
| Lee et al. | Frame rate up conversion based on variational image fusion | |
| JP2003163894A (en) | Apparatus and method of converting frame and/or field rate using adaptive motion compensation | |
| KR101519941B1 (en) | Multi-level bidirectional motion estimation method and device | |
| CN102123283B (en) | Interpolated frame acquisition method and device in video frame rate conversion | |
| CN107483960B (en) | Motion compensation frame rate up-conversion method based on spatial prediction | |
| US20150063682A1 (en) | Video disparity estimate space-time refinement method and codec | |
| CN103152566B (en) | A kind of video frame rate method for improving | |
| CN103313059B (en) | The decision method of occlusion areas during a kind of frame per second promotes | |
| Han et al. | Frame rate up-conversion for high-definition video applications | |
| CN101600105A (en) | Frame frequency lifting method | |
| Lee et al. | High frame rate motion compensated frame interpolation in high-definition video processing | |
| KR20150090454A (en) | Method for searching bidirectional motion using multiple frame and image apparatus with the same technique | |
| Rüfenacht et al. | Temporal frame interpolation with motion-divergence-guided occlusion handling | |
| Li et al. | Motion-compensated frame interpolation using patch-based sparseland model | |
| Keller et al. | Temporal super resolution using variational methods | |
| Lu et al. | An artifact information based motion vector processing method for motion compensated frame interpolation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160706 Termination date: 20180819 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |