WO2018170393A3 - Frame interpolation via adaptive convolution and adaptive separable convolution - Google Patents

Frame interpolation via adaptive convolution and adaptive separable convolution Download PDF

Info

Publication number
WO2018170393A3
WO2018170393A3 PCT/US2018/022858 US2018022858W WO2018170393A3 WO 2018170393 A3 WO2018170393 A3 WO 2018170393A3 US 2018022858 W US2018022858 W US 2018022858W WO 2018170393 A3 WO2018170393 A3 WO 2018170393A3
Authority
WO
WIPO (PCT)
Prior art keywords
convolution
pixel
frame
input
adaptive
Prior art date
Application number
PCT/US2018/022858
Other languages
French (fr)
Other versions
WO2018170393A2 (en
WO2018170393A9 (en
Inventor
Feng Liu
Simon NIKLAUS
Long MAI
Original Assignee
Portland State University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US201762473234P external-priority
Application filed by Portland State University filed Critical Portland State University
Priority to US16/495,029 priority Critical patent/US11468318B2/en
Priority to KR1020197030137A priority patent/KR102474168B1/en
Publication of WO2018170393A2 publication Critical patent/WO2018170393A2/en
Publication of WO2018170393A9 publication Critical patent/WO2018170393A9/en
Publication of WO2018170393A3 publication Critical patent/WO2018170393A3/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/01Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0127Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Computing arrangements based on biological models using neural network models
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00Machine learning
    • G06N20/10Machine learning using kernel methods, e.g. support vector machines [SVM]
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4007Interpolation-based scaling, e.g. bilinear interpolation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4046Scaling the whole image or part thereof using neural networks
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/01Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
    • H04N7/0135Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Systems, methods, and computer-readable media for context-aware synthesis for video frame interpolation are provided. A convolutional neural network (ConvNet) may, given two input video or image frames, interpolate a frame temporarily in the middle of the two input frames by combining motion estimation and pixel synthesis into a single step and formulating pixel interpolation as a local convolution over patches in the input images. The ConvNet may estimate a convolution kernel based on a first receptive field patch of a first input image frame and a second receptive field patch of a second input image frame. The ConvNet may then convolve the convolutional kernel over a first pixel patch of the first input image frame and a second pixel patch of the second input image frame to obtain color data of an output pixel of the interpolation frame. Other embodiments may be described and/or claimed.
PCT/US2018/022858 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution WO2018170393A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/495,029 US11468318B2 (en) 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution
KR1020197030137A KR102474168B1 (en) 2017-03-17 2018-03-16 Frame interpolation with adaptive convolution and adaptive disjoint convolution

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201762473234P 2017-03-17 2017-03-17
US62/473,234 2017-03-17
US201762485794P 2017-04-14 2017-04-14
US62/485,794 2017-04-14

Publications (3)

Publication Number Publication Date
WO2018170393A2 WO2018170393A2 (en) 2018-09-20
WO2018170393A9 WO2018170393A9 (en) 2018-11-15
WO2018170393A3 true WO2018170393A3 (en) 2018-12-20

Family

ID=63522622

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/022858 WO2018170393A2 (en) 2017-03-17 2018-03-16 Frame interpolation via adaptive convolution and adaptive separable convolution

Country Status (3)

Country Link
US (1) US11468318B2 (en)
KR (1) KR102474168B1 (en)
WO (1) WO2018170393A2 (en)

Families Citing this family (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11308361B1 (en) * 2017-07-07 2022-04-19 Twitter, Inc. Checkerboard artifact free sub-pixel convolution
US11475542B2 (en) 2017-07-27 2022-10-18 Nvidia Corporation Neural network system with temporal feedback for adaptive sampling and denoising of rendered sequences
US10706890B2 (en) * 2017-08-24 2020-07-07 Intel Corporation Cinematic space-time view synthesis for enhanced viewing experiences in computing environments
US10776688B2 (en) 2017-11-06 2020-09-15 Nvidia Corporation Multi-frame video interpolation using optical flow
US11122238B1 (en) * 2017-11-07 2021-09-14 Twitter, Inc. Frame interpolation with multi-scale deep loss functions and generative adversarial networks
US11019355B2 (en) * 2018-04-03 2021-05-25 Electronics And Telecommunications Research Institute Inter-prediction method and apparatus using reference frame generated based on deep learning
US10984245B1 (en) * 2018-06-11 2021-04-20 Facebook, Inc. Convolutional neural network based on groupwise convolution for efficient video analysis
JP7135697B2 (en) * 2018-10-11 2022-09-13 株式会社島津製作所 Analysis device, analysis system and analysis method
US11126915B2 (en) * 2018-10-15 2021-09-21 Sony Corporation Information processing apparatus and information processing method for volume data visualization
KR102312337B1 (en) * 2018-10-19 2021-10-14 삼성전자주식회사 AI encoding apparatus and operating method for the same, and AI decoding apparatus and operating method for the same
WO2020080623A1 (en) * 2018-10-19 2020-04-23 삼성전자 주식회사 Method and apparatus for ai encoding and ai decoding of image
US20200137380A1 (en) * 2018-10-31 2020-04-30 Intel Corporation Multi-plane display image synthesis mechanism
CN111178491A (en) * 2018-11-09 2020-05-19 佳能株式会社 Method, device, system and storage medium for training and applying neural network model
CN109472315B (en) * 2018-11-15 2021-09-24 江苏木盟智能科技有限公司 Target detection method and system based on depth separable convolution
KR20200071404A (en) * 2018-12-11 2020-06-19 삼성전자주식회사 Image processing apparatus and operating method for the same
US11080835B2 (en) 2019-01-09 2021-08-03 Disney Enterprises, Inc. Pixel error detection system
GB2580671B (en) * 2019-01-22 2022-05-04 Toshiba Kk A computer vision system and method
US11461653B2 (en) * 2019-01-23 2022-10-04 StradVision, Inc. Learning method and learning device for CNN using 1xK or Kx1 convolution to be used for hardware optimization, and testing method and testing device using the same
US11288818B2 (en) * 2019-02-19 2022-03-29 The Trustees Of The University Of Pennsylvania Methods, systems, and computer readable media for estimation of optical flow, depth, and egomotion using neural network trained using event-based learning
CN109905624B (en) * 2019-03-01 2020-10-16 北京大学深圳研究生院 Video frame interpolation method, device and equipment
EP3959882A1 (en) * 2019-04-23 2022-03-02 Telefonaktiebolaget LM Ericsson (publ) A computer software module, a device and a method for accelerating inference for compressed videos
CN110111366B (en) * 2019-05-06 2021-04-30 北京理工大学 End-to-end optical flow estimation method based on multistage loss
US10896356B2 (en) * 2019-05-10 2021-01-19 Samsung Electronics Co., Ltd. Efficient CNN-based solution for video frame interpolation
US11460851B2 (en) 2019-05-24 2022-10-04 Ford Global Technologies, Llc Eccentricity image fusion
US11521494B2 (en) 2019-06-11 2022-12-06 Ford Global Technologies, Llc Vehicle eccentricity mapping
CN110427094B (en) * 2019-07-17 2021-08-17 Oppo广东移动通信有限公司 Display method, display device, electronic equipment and computer readable medium
WO2021063119A1 (en) * 2019-10-01 2021-04-08 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Method and apparatus for image processing, terminal
US10958869B1 (en) 2019-11-14 2021-03-23 Huawei Technologies Co., Ltd. System, device and method for video frame interpolation using a structured neural network
KR102207736B1 (en) * 2020-01-14 2021-01-26 한국과학기술원 Frame interpolation, apparatus and method using deep neural network
CN111212287A (en) * 2020-01-15 2020-05-29 济南浪潮高新科技投资发展有限公司 Video compression method based on image interpolation method
US11074730B1 (en) 2020-01-23 2021-07-27 Netapp, Inc. Augmented reality diagnostic tool for data center nodes
KR102372046B1 (en) * 2020-03-12 2022-03-11 숭실대학교 산학협력단 Method of vessel structures extraction using artificial intelligence technique, recording medium and device for performing the method
US11508143B2 (en) 2020-04-03 2022-11-22 Disney Enterprises, Inc. Automated salience assessment of pixel anomalies
US20220116567A1 (en) * 2020-04-30 2022-04-14 Boe Technology Group Co., Ltd. Video frame interpolation method and device, computer readable storage medium
CN113727141A (en) * 2020-05-20 2021-11-30 富士通株式会社 Interpolation device and method for video frame
KR20220030084A (en) * 2020-09-02 2022-03-10 삼성전자주식회사 Method and apparatus of processing image
US11216656B1 (en) * 2020-12-16 2022-01-04 Retrocausal, Inc. System and method for management and evaluation of one or more human activities
KR20220092247A (en) * 2020-12-24 2022-07-01 삼성전자주식회사 Electronic apparatus and method for controlling thereof
CN113014936B (en) * 2021-02-24 2022-12-13 北京百度网讯科技有限公司 Video frame insertion method, device, equipment and storage medium
WO2022186620A1 (en) * 2021-03-04 2022-09-09 현대자동차주식회사 Video coding method and apparatus for improving prediction signals of intra prediction
CN113132664B (en) * 2021-04-19 2022-10-04 科大讯飞股份有限公司 Frame interpolation generation model construction method and video frame interpolation method
GB202207295D0 (en) * 2021-05-25 2022-06-29 Samsung Electronics Co Ltd Method and apparatus for video recognition

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016132146A1 (en) * 2015-02-19 2016-08-25 Magic Pony Technology Limited Visual processing using sub-pixel convolutions

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5353119A (en) * 1990-11-15 1994-10-04 Sony United Kingdom Limited Format conversion of digital video signals, integration of digital video signals into photographic film material and the like, associated signal processing, and motion compensated interpolation of images
US20030189579A1 (en) * 2002-04-05 2003-10-09 Pope David R. Adaptive enlarging and/or sharpening of a digital image
US9039621B2 (en) * 2011-07-27 2015-05-26 The United States of America National Institutes of Health (NIH), U.S. Dept. of Health and Human Services (DHHS), NIH Division of Extramural Inventions and Technology Resources (DEITR) Non-invasive ultrasonic gingival tissue diagnosis
US9418458B2 (en) * 2015-01-05 2016-08-16 Superfish Ltd. Graph image representation from convolutional neural networks
AU2016308097B2 (en) * 2015-08-15 2018-08-02 Salesforce.Com, Inc. Three-dimensional (3D) convolution with 3D batch normalization
JP6700712B2 (en) * 2015-10-21 2020-05-27 キヤノン株式会社 Convolution operation device
JP6930428B2 (en) * 2016-01-21 2021-09-01 ソニーグループ株式会社 Information processing equipment, information processing methods, and programs
WO2017200524A1 (en) * 2016-05-16 2017-11-23 United Technologies Corporation Deep convolutional neural networks for crack detection from image data

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016132146A1 (en) * 2015-02-19 2016-08-25 Magic Pony Technology Limited Visual processing using sub-pixel convolutions

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GUCAN LONG ET AL.: "Learning Image Matching by Simply Watching Video", ECCV 2016, 2016, pages 434 - 450, XP047355274 *
HAITAM BEN YAHIA: "Frame Interpolation using Convolutional Neural Networks on 2D animation", BACHELOR THESIS, 24 June 2016 (2016-06-24), XP055558906 *

Also Published As

Publication number Publication date
KR102474168B1 (en) 2022-12-06
WO2018170393A2 (en) 2018-09-20
US11468318B2 (en) 2022-10-11
KR20190132415A (en) 2019-11-27
WO2018170393A9 (en) 2018-11-15
US20200012940A1 (en) 2020-01-09

Similar Documents

Publication Publication Date Title
WO2018170393A3 (en) Frame interpolation via adaptive convolution and adaptive separable convolution
US11196918B2 (en) System, method, and apparatus for determining a high dynamic range image
US9613408B2 (en) High dynamic range image composition using multiple images
WO2020081901A3 (en) Crop yield prediction at field-level and pixel-level
US11276149B2 (en) Double non-local means denoising
US20200084338A1 (en) Image stitching with electronic rolling shutter correction
WO2016061011A3 (en) Camera capture recommendation for applications
WO2015189629A3 (en) Improvements in and relating to the display of images
WO2007092937A3 (en) Adaptive image filter for filtering image information
JP2016516378A5 (en)
WO2012003483A3 (en) A method and apparatus for video processing for improved video compression
WO2007087640A3 (en) Adaptive filtering to enhance video encoder performance
WO2008066699A3 (en) Processing images having color and panchromatic pixels
WO2011060579A8 (en) Method for generating depth maps from monocular images and systems using the same
US9307162B2 (en) Local enhancement apparatus and method to generate high dynamic range images by blending brightness-preserved and brightness-adjusted blocks
WO2019226211A1 (en) Image signal processing for reducing lens flare
EP2151996A3 (en) Image signal processing unit and method of processing image signal
US10757384B2 (en) Desaturation control
WO2017205492A1 (en) Three-dimensional noise reduction
WO2020146911A3 (en) Multi-stage multi-reference bootstrapping for video super-resolution
MX2019010055A (en) Method and device for reconstructing an hdr image.
US10902558B2 (en) Multiscale denoising of raw images with noise estimation
TW200632765A (en) Method and apparatus of image dynamic response re-mapping and digital camera using the same
CN106342194B (en) A kind of Infrared Image Non-uniformity Correction method of ground scene
US20220122228A1 (en) Chrominance Denoising

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18767692

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20197030137

Country of ref document: KR

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 18767692

Country of ref document: EP

Kind code of ref document: A2