JP7395577B2 - 再投影フレームのモーションスムージング - Google Patents

再投影フレームのモーションスムージング Download PDF

Info

Publication number
JP7395577B2
JP7395577B2 JP2021519155A JP2021519155A JP7395577B2 JP 7395577 B2 JP7395577 B2 JP 7395577B2 JP 2021519155 A JP2021519155 A JP 2021519155A JP 2021519155 A JP2021519155 A JP 2021519155A JP 7395577 B2 JP7395577 B2 JP 7395577B2
Authority
JP
Japan
Prior art keywords
pixel data
frame
motion vectors
array
gpu
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2021519155A
Other languages
English (en)
Japanese (ja)
Other versions
JP2022504475A (ja
JP2022504475A5 (cg-RX-API-DMAC7.html
Inventor
ヴラコス、アレックス
ライビー、アーロン
Original Assignee
バルブ コーポレーション
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by バルブ コーポレーション filed Critical バルブ コーポレーション
Publication of JP2022504475A publication Critical patent/JP2022504475A/ja
Publication of JP2022504475A5 publication Critical patent/JP2022504475A5/ja
Application granted granted Critical
Publication of JP7395577B2 publication Critical patent/JP7395577B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/005—General purpose rendering architectures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
    • G09G3/003—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background to produce spatial visual effects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region
    • H04N5/144—Movement detection
    • H04N5/145—Movement estimation
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/02—Improving the quality of display appearance
    • G09G2320/0261—Improving the quality of display appearance in the context of movement of objects on the screen or movement of the observer relative to the screen
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/10—Special adaptations of display systems for operation with variable images
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/10—Special adaptations of display systems for operation with variable images
    • G09G2320/106—Determination of movement vectors or equivalent parameters within the image
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0407—Resolution change, inclusive of the use of different resolutions for different screen areas
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0457—Improvement of perceived resolution by subpixel rendering
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/14—Solving problems related to the presentation of information to be displayed
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/16—Determination of a pixel data signal depending on the signal applied in the previous frame
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00—Aspects of interface with display user

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Transforming Electric Information Into Light Information (AREA)
JP2021519155A 2018-10-09 2019-09-27 再投影フレームのモーションスムージング Active JP7395577B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US16/155,686 US10733783B2 (en) 2018-10-09 2018-10-09 Motion smoothing for re-projected frames
US16/155,686 2018-10-09
PCT/US2019/053528 WO2020076526A1 (en) 2018-10-09 2019-09-27 Motion smoothing for re-projected frames

Publications (3)

Publication Number Publication Date
JP2022504475A JP2022504475A (ja) 2022-01-13
JP2022504475A5 JP2022504475A5 (cg-RX-API-DMAC7.html) 2022-10-03
JP7395577B2 true JP7395577B2 (ja) 2023-12-11

Family

ID=70052297

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2021519155A Active JP7395577B2 (ja) 2018-10-09 2019-09-27 再投影フレームのモーションスムージング

Country Status (6)

Country Link
US (1) US10733783B2 (cg-RX-API-DMAC7.html)
EP (1) EP3857885A4 (cg-RX-API-DMAC7.html)
JP (1) JP7395577B2 (cg-RX-API-DMAC7.html)
KR (1) KR102641272B1 (cg-RX-API-DMAC7.html)
CN (1) CN113170136B (cg-RX-API-DMAC7.html)
WO (1) WO2020076526A1 (cg-RX-API-DMAC7.html)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10962780B2 (en) * 2015-10-26 2021-03-30 Microsoft Technology Licensing, Llc Remote rendering for virtual images
US10152822B2 (en) * 2017-04-01 2018-12-11 Intel Corporation Motion biased foveated renderer
US10319064B2 (en) 2017-04-10 2019-06-11 Intel Corporation Graphics anti-aliasing resolve with stencil mask
CN115842907A (zh) * 2018-03-27 2023-03-24 京东方科技集团股份有限公司 渲染方法、计算机产品及显示装置
US10861215B2 (en) * 2018-04-30 2020-12-08 Qualcomm Incorporated Asynchronous time and space warp with determination of region of interest
US10520739B1 (en) * 2018-07-11 2019-12-31 Valve Corporation Dynamic panel masking
US11381825B2 (en) * 2018-11-27 2022-07-05 Advanced Micro Devices, Inc. Variable rate rendering based on motion estimation
US11030822B2 (en) * 2019-05-15 2021-06-08 Microsoft Technology Licensing, Llc Content indicators in a 3D environment authoring application
US11287947B2 (en) 2019-05-15 2022-03-29 Microsoft Technology Licensing, Llc Contextual input in a three-dimensional environment
US11164395B2 (en) 2019-05-15 2021-11-02 Microsoft Technology Licensing, Llc Structure switching in a three-dimensional environment
US11039061B2 (en) 2019-05-15 2021-06-15 Microsoft Technology Licensing, Llc Content assistance in a three-dimensional environment
US11948242B2 (en) 2019-08-02 2024-04-02 Fmr Llc Intelligent smoothing of 3D alternative reality applications for secondary 2D viewing
US11138804B2 (en) 2019-08-02 2021-10-05 Fmr Llc Intelligent smoothing of 3D alternative reality applications for secondary 2D viewing
US12148120B2 (en) * 2019-12-18 2024-11-19 Ati Technologies Ulc Frame reprojection for virtual reality and augmented reality
US11363247B2 (en) 2020-02-14 2022-06-14 Valve Corporation Motion smoothing in a distributed system
US11651547B2 (en) * 2020-12-10 2023-05-16 Nvidia Corporation Adaptive temporal image filtering for rendering realistic illumination
US12069230B2 (en) * 2020-12-23 2024-08-20 Meta Platforms Technologies, Llc Temporal foveated rendering
CN113066001B (zh) * 2021-02-26 2024-10-22 华为技术有限公司 一种图像处理方法及相关设备
US12200398B2 (en) 2021-03-16 2025-01-14 Samsung Electronics Co., Ltd. Pixel blending for synthesizing video frames with occlusion and watermark handling
US12075018B2 (en) * 2021-04-21 2024-08-27 Microsoft Technology Licensing, Llc Distributed virtual reality
US11587208B2 (en) * 2021-05-26 2023-02-21 Qualcomm Incorporated High quality UI elements with frame extrapolation
US12236608B2 (en) * 2021-09-22 2025-02-25 Samsung Electronics Co., Ltd. Judder detection for dynamic frame rate conversion
EP4224289A1 (en) * 2022-02-08 2023-08-09 Nokia Technologies Oy Mediated reality
CN118647966A (zh) * 2022-02-14 2024-09-13 高通股份有限公司 显示掩模层生成和运行时调整
CN118057460A (zh) * 2022-11-21 2024-05-21 荣耀终端有限公司 图像处理方法及装置
CN118506702A (zh) * 2023-02-15 2024-08-16 台达电子工业股份有限公司 移动物件检测装置及方法
WO2025192345A1 (ja) * 2024-03-12 2025-09-18 株式会社ソニー・インタラクティブエンタテインメント 画像生成装置、画像生成方法および画像生成プログラム
IL311824A (en) * 2024-03-28 2025-10-01 Semiconductor Devices Ltd Correction of non-uniformity for infra-red images
KR102888530B1 (ko) * 2025-03-20 2025-11-21 서울대학교산학협력단 디스플레이 소자 어레이 제조방법 및 디스플레이 패널

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070035706A1 (en) 2005-06-20 2007-02-15 Digital Display Innovations, Llc Image and light source modulation for a digital display system
JP2012073350A (ja) 2010-09-28 2012-04-12 Canon Inc 映像制御装置、及び映像制御方法
US20140307787A1 (en) 2013-04-15 2014-10-16 Santa Clara University Method and apparatus of depth prediction mode selection
US20180041768A1 (en) 2015-02-24 2018-02-08 Lg Electronics Inc. Polygon unit-based image processing method, and device for same
US20190045213A1 (en) 2017-08-03 2019-02-07 Intel Corporation Reference frame reprojection for improved video coding

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07222155A (ja) * 1994-01-31 1995-08-18 Kyocera Corp 画像符号化装置
US6466185B2 (en) 1998-04-20 2002-10-15 Alan Sullivan Multi-planar volumetric display system and method of operation using psychological vision cues
JP4408779B2 (ja) * 2004-09-15 2010-02-03 キヤノン株式会社 画像処理装置
EP2130381A2 (en) 2007-01-23 2009-12-09 Euclid Discoveries, LLC Computer method and apparatus for processing image data
JP2008252591A (ja) * 2007-03-30 2008-10-16 Toshiba Corp 補間フレーム生成装置、補間フレーム生成方法及び放送受信装置
US8208551B2 (en) * 2007-10-31 2012-06-26 Broadcom Corporation Method and system for hierarchically layered adaptive median motion vector smoothing
US20120021828A1 (en) 2010-02-24 2012-01-26 Valve Corporation Graphical user interface for modification of animation data using preset animation samples
US8624891B2 (en) 2011-01-17 2014-01-07 Disney Enterprises, Inc. Iterative reprojection of images
US9277129B2 (en) 2013-06-07 2016-03-01 Apple Inc. Robust image feature based video stabilization and smoothing
US11303900B2 (en) * 2013-12-06 2022-04-12 Mediatek Inc. Method and apparatus for motion boundary processing
US9824498B2 (en) * 2014-12-30 2017-11-21 Sony Interactive Entertainment Inc. Scanning display system in head-mounted display for virtual reality
US10037620B2 (en) * 2015-05-29 2018-07-31 Nvidia Corporation Piecewise linear irregular rasterization
GB2539042B (en) * 2015-06-05 2019-08-21 Imagination Tech Ltd Tessellation method using displacement factors
HK1249270A1 (zh) * 2015-08-19 2018-10-26 威尔乌集团 用於检测和/或校正显示器中的像素亮度和/或色度响应变化的系统和方法
US10026212B2 (en) * 2015-11-20 2018-07-17 Google Llc Electronic display stabilization using pixel velocities
US9978180B2 (en) 2016-01-25 2018-05-22 Microsoft Technology Licensing, Llc Frame projection for augmented reality environments
US10129523B2 (en) * 2016-06-22 2018-11-13 Microsoft Technology Licensing, Llc Depth-aware reprojection
US10297086B2 (en) * 2016-07-04 2019-05-21 Liquid Cinema Inc. System and method for processing digital video
US10861215B2 (en) * 2018-04-30 2020-12-08 Qualcomm Incorporated Asynchronous time and space warp with determination of region of interest

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070035706A1 (en) 2005-06-20 2007-02-15 Digital Display Innovations, Llc Image and light source modulation for a digital display system
JP2012073350A (ja) 2010-09-28 2012-04-12 Canon Inc 映像制御装置、及び映像制御方法
US20140307787A1 (en) 2013-04-15 2014-10-16 Santa Clara University Method and apparatus of depth prediction mode selection
US20180041768A1 (en) 2015-02-24 2018-02-08 Lg Electronics Inc. Polygon unit-based image processing method, and device for same
US20190045213A1 (en) 2017-08-03 2019-02-07 Intel Corporation Reference frame reprojection for improved video coding

Also Published As

Publication number Publication date
JP2022504475A (ja) 2022-01-13
KR20210071015A (ko) 2021-06-15
EP3857885A4 (en) 2022-06-22
US20200111195A1 (en) 2020-04-09
EP3857885A1 (en) 2021-08-04
CN113170136B (zh) 2024-04-23
CN113170136A (zh) 2021-07-23
WO2020076526A1 (en) 2020-04-16
KR102641272B1 (ko) 2024-02-27
US10733783B2 (en) 2020-08-04

Similar Documents

Publication Publication Date Title
JP7395577B2 (ja) 再投影フレームのモーションスムージング
JP7612700B2 (ja) 分散システムにおける動き平滑化
KR102714790B1 (ko) 애플리케이션 렌더링 성능에 근거한 예측 및 스로틀링 조정
CN113272715B (zh) 基于眼睛跟踪的动态渲染时间确定
EP2854099B1 (en) Information processing device and information processing method
JP7822783B2 (ja) ダイナミックパネルマスキング
US10121221B2 (en) Method and apparatus to accelerate rendering of graphics images
US11343486B2 (en) Counterrotation of display panels and/or virtual cameras in a HMD
US12125181B2 (en) Image generation device and image generation method
US20260030764A1 (en) Rendered image data processing and optical flow calculation

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20220922

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20220922

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20231107

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20231129

R150 Certificate of patent or registration of utility model

Ref document number: 7395577

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150