WO2018170393A3 - Frame interpolation via adaptive convolution and adaptive separable convolution - Google Patents

Frame interpolation via adaptive convolution and adaptive separable convolution Download PDF

Info

Publication number
WO2018170393A3
WO2018170393A3 PCT/US2018/022858 US2018022858W WO2018170393A3 WO 2018170393 A3 WO2018170393 A3 WO 2018170393A3 US 2018022858 W US2018022858 W US 2018022858W WO 2018170393 A3 WO2018170393 A3 WO 2018170393A3
Authority
WO
WIPO (PCT)
Prior art keywords
convolution
pixel
frame
adaptive
patch
Prior art date
Application number
PCT/US2018/022858
Other languages
French (fr)
Other versions
WO2018170393A2 (en
WO2018170393A9 (en
Inventor
Feng Liu
Simon NIKLAUS
Long MAI
Original Assignee
Portland State University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US201762473234P priority Critical
Priority to US62/473,234 priority
Priority to US201762485794P priority
Priority to US62/485,794 priority
Application filed by Portland State University filed Critical Portland State University
Publication of WO2018170393A2 publication Critical patent/WO2018170393A2/en
Publication of WO2018170393A9 publication Critical patent/WO2018170393A9/en
Publication of WO2018170393A3 publication Critical patent/WO2018170393A3/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0127—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computer systems based on biological models
    • G06N3/02—Computer systems based on biological models using neural network models
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4046—Scaling the whole image or part thereof using neural networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Systems, methods, and computer-readable media for context-aware synthesis for video frame interpolation are provided. A convolutional neural network (ConvNet) may, given two input video or image frames, interpolate a frame temporarily in the middle of the two input frames by combining motion estimation and pixel synthesis into a single step and formulating pixel interpolation as a local convolution over patches in the input images. The ConvNet may estimate a convolution kernel based on a first receptive field patch of a first input image frame and a second receptive field patch of a second input image frame. The ConvNet may then convolve the convolutional kernel over a first pixel patch of the first input image frame and a second pixel patch of the second input image frame to obtain color data of an output pixel of the interpolation frame. Other embodiments may be described and/or claimed.
PCT/US2018/022858 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution WO2018170393A2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US201762473234P true 2017-03-17 2017-03-17
US62/473,234 2017-03-17
US201762485794P true 2017-04-14 2017-04-14
US62/485,794 2017-04-14

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020197030137A KR20190132415A (en) 2017-03-17 2018-03-16 Frame Interpolation with Adaptive Convolution and Adaptive Isolated Convolution
US16/495,029 US20200012940A1 (en) 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution

Publications (3)

Publication Number Publication Date
WO2018170393A2 WO2018170393A2 (en) 2018-09-20
WO2018170393A9 WO2018170393A9 (en) 2018-11-15
WO2018170393A3 true WO2018170393A3 (en) 2018-12-20

Family

ID=63522622

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/022858 WO2018170393A2 (en) 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution

Country Status (3)

Country Link
US (1) US20200012940A1 (en)
KR (1) KR20190132415A (en)
WO (1) WO2018170393A2 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10776688B2 (en) 2017-11-06 2020-09-15 Nvidia Corporation Multi-frame video interpolation using optical flow
KR20200071404A (en) * 2018-12-11 2020-06-19 삼성전자주식회사 Image processing apparatus and operating method for the same
CN109905624B (en) * 2019-03-01 2020-10-16 北京大学深圳研究生院 Video frame interpolation method, device and equipment
WO2020216438A1 (en) * 2019-04-23 2020-10-29 Telefonaktiebolaget Lm Ericsson (Publ) A computer software module, a device and a method for accelerating inference for compressed videos
CN110111366A (en) * 2019-05-06 2019-08-09 北京理工大学 A kind of end-to-end light stream estimation method based on multistage loss amount
US10896356B2 (en) * 2019-05-10 2021-01-19 Samsung Electronics Co., Ltd. Efficient CNN-based solution for video frame interpolation
CN110427094A (en) * 2019-07-17 2019-11-08 Oppo广东移动通信有限公司 Display methods, device, electronic equipment and computer-readable medium

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016132146A1 (en) * 2015-02-19 2016-08-25 Magic Pony Technology Limited Visual processing using sub-pixel convolutions

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016132146A1 (en) * 2015-02-19 2016-08-25 Magic Pony Technology Limited Visual processing using sub-pixel convolutions

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GUCAN LONG ET AL.: "Learning Image Matching by Simply Watching Video", ECCV 2016, 2016, pages 434 - 450, XP047355274 *
HAITAM BEN YAHIA: "Frame Interpolation using Convolutional Neural Networks on 2D animation", BACHELOR THESIS, 24 June 2016 (2016-06-24), XP055558906 *

Also Published As

Publication number Publication date
KR20190132415A (en) 2019-11-27
WO2018170393A2 (en) 2018-09-20
US20200012940A1 (en) 2020-01-09
WO2018170393A9 (en) 2018-11-15

Similar Documents

Publication Publication Date Title
US10728474B2 (en) Image signal processor for local motion estimation and video codec
He et al. Fast guided filter
JP6066536B2 (en) Generation of high dynamic range images without ghosting
US8723978B2 (en) Image fusion apparatus and method
CN106504278B (en) High dynamic range tone mapping
US9100589B1 (en) Interleaved capture for high dynamic range image acquisition and synthesis
US9639956B2 (en) Image adjustment using texture mask
JP2013048450A (en) Stereo image and video capturing device with dual digital sensors, and methods of using the same
CN103295194B (en) The controlled tone mapping method with Hemifusus ternatanus of brightness
US10715773B2 (en) Method and system of lens shading color correction using block matching
US20140293074A1 (en) Generating a composite image from video frames
US8760489B1 (en) Method and apparatus for dynamically adjusting aspect ratio of images during a video call
US9031345B2 (en) Optical flow accounting for image haze
WO2014160433A3 (en) Classifying objects in images using mobile devices
US9232199B2 (en) Method, apparatus and computer program product for capturing video content
KR102173786B1 (en) Background modification in video conferencing
WO2016076938A3 (en) High dynamic range image composition using multiple images
US20200020120A1 (en) Reducing textured ir patterns in stereoscopic depth sensor imaging
JP4290193B2 (en) Image processing device
US10885384B2 (en) Local tone mapping to reduce bit depth of input images to high-level computer vision tasks
WO2016019770A1 (en) Method, device and storage medium for picture synthesis
WO2009001510A1 (en) Image processing device, image processing method, and program
MY137026A (en) A system and process for generating high dynamic range images from multiple exposures of a moving scene
MX2009011091A (en) Video detection system and methods.
WO2007097808A3 (en) Camera exposure optimization techniques that take camera and scene motion into account

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18767692

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20197030137

Country of ref document: KR

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 18767692

Country of ref document: EP

Kind code of ref document: A2