JP7711002B2 - 動的オクルージョンのための高速の手のメッシュ化 - Google Patents

動的オクルージョンのための高速の手のメッシュ化

Info

Publication number
JP7711002B2
JP7711002B2 JP2021576046A JP2021576046A JP7711002B2 JP 7711002 B2 JP7711002 B2 JP 7711002B2 JP 2021576046 A JP2021576046 A JP 2021576046A JP 2021576046 A JP2021576046 A JP 2021576046A JP 7711002 B2 JP7711002 B2 JP 7711002B2
Authority
JP
Japan
Prior art keywords
hand
scene
information
mesh
depth information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2021576046A
Other languages
English (en)
Japanese (ja)
Other versions
JP2022537817A (ja
JPWO2020264149A5 (https=
JP2022537817A5 (https=
Inventor
デイビッド ジェフリー モリノー，
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Magic Leap Inc
Original Assignee
Magic Leap Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Magic Leap Inc filed Critical Magic Leap Inc
Publication of JP2022537817A publication Critical patent/JP2022537817A/ja
Publication of JPWO2020264149A5 publication Critical patent/JPWO2020264149A5/ja
Publication of JP2022537817A5 publication Critical patent/JP2022537817A5/ja
Priority to JP2025075562A priority Critical patent/JP2025107314A/ja
Application granted granted Critical
Publication of JP7711002B2 publication Critical patent/JP7711002B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2215/00—Indexing scheme for image rendering
    • G06T2215/16—Using real world measurements to influence rendering

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Optics & Photonics (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Social Psychology (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Computer Hardware Design (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
  • Image Generation (AREA)
JP2021576046A 2019-06-28 2020-06-25 動的オクルージョンのための高速の手のメッシュ化 Active JP7711002B2 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2025075562A JP2025107314A (ja) 2019-06-28 2025-04-30 動的オクルージョンのための高速の手のメッシュ化

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201962868640P 2019-06-28 2019-06-28
US62/868,640 2019-06-28
PCT/US2020/039601 WO2020264149A1 (en) 2019-06-28 2020-06-25 Fast hand meshing for dynamic occlusion

Related Child Applications (1)

Application Number Title Priority Date Filing Date
JP2025075562A Division JP2025107314A (ja) 2019-06-28 2025-04-30 動的オクルージョンのための高速の手のメッシュ化

Publications (4)

Publication Number Publication Date
JP2022537817A JP2022537817A (ja) 2022-08-30
JPWO2020264149A5 JPWO2020264149A5 (https=) 2023-06-29
JP2022537817A5 JP2022537817A5 (https=) 2023-06-29
JP7711002B2 true JP7711002B2 (ja) 2025-07-22

Family

ID=74042905

Family Applications (2)

Application Number Title Priority Date Filing Date
JP2021576046A Active JP7711002B2 (ja) 2019-06-28 2020-06-25 動的オクルージョンのための高速の手のメッシュ化
JP2025075562A Pending JP2025107314A (ja) 2019-06-28 2025-04-30 動的オクルージョンのための高速の手のメッシュ化

Family Applications After (1)

Application Number Title Priority Date Filing Date
JP2025075562A Pending JP2025107314A (ja) 2019-06-28 2025-04-30 動的オクルージョンのための高速の手のメッシュ化

Country Status (5)

Country Link
US (2) US11238651B2 (https=)
EP (1) EP3991142B1 (https=)
JP (2) JP7711002B2 (https=)
CN (1) CN114026606B (https=)
WO (1) WO2020264149A1 (https=)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11710310B2 (en) * 2019-06-19 2023-07-25 Apple Inc. Virtual content positioned based on detected object
JP7711002B2 (ja) 2019-06-28 2025-07-22 マジック リープ， インコーポレイテッド 動的オクルージョンのための高速の手のメッシュ化
US11373339B2 (en) * 2020-03-18 2022-06-28 Sony Group Corporation Projection-based mesh compression
US11836871B2 (en) * 2021-03-22 2023-12-05 Apple Inc. Indicating a position of an occluded physical object
GB2613840B (en) * 2021-12-16 2025-03-12 Sony Interactive Entertainment Europe Ltd Feature tracking system and method
JP7740048B2 (ja) * 2022-02-14 2025-09-17 トヨタ自動車株式会社 システム、情報処理装置、及び方法
US12154231B2 (en) * 2022-07-20 2024-11-26 Qualcomm Incorporated Late stage occlusion based rendering for extended reality (XR)
CN115439171A (zh) * 2022-08-04 2022-12-06 阿里巴巴（中国）有限公司 商品信息展示方法、装置及电子设备
WO2024191175A1 (en) * 2023-03-15 2024-09-19 Samsung Electronics Co., Ltd. Method and electronic device for estimating a landmark point of body part of subject
US20250148701A1 (en) * 2023-11-06 2025-05-08 Samsung Electronics Co., Ltd. Dynamic overlapping of moving objects with real and virtual scenes for video see-through (vst) extended reality (xr)
US20250173978A1 (en) * 2023-11-27 2025-05-29 Meta Platforms Technologies, Llc Real-time Rendering Of Animated Objects

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006343953A (ja) 2005-06-08 2006-12-21 Canon Inc 画像処理方法、画像処理装置
JP2014514653A (ja) 2011-03-29 2014-06-19 クアルコム，インコーポレイテッド 各ユーザの視点に対する共有デジタルインターフェースのレンダリングのためのシステム
WO2014128788A1 (ja) 2013-02-19 2014-08-28 株式会社ブリリアントサービス 形状認識装置、形状認識プログラム、および形状認識方法
JP2016115148A (ja) 2014-12-15 2016-06-23 キヤノンマーケティングジャパン株式会社 情報処理装置、情報処理システム、情報処理方法、及びプログラム
US20170038850A1 (en) 2012-04-09 2017-02-09 Shahar Fleishman System and method for combining three-dimensional tracking with a three-dimensional display for a user interface
JP2017529635A (ja) 2014-06-14 2017-10-05 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． 仮想および拡張現実を作成する方法およびシステム
JP2018022292A (ja) 2016-08-02 2018-02-08 キヤノン株式会社 情報処理装置、情報処理装置の制御方法およびプログラム
JP2019514101A (ja) 2016-03-24 2019-05-30 クアルコム，インコーポレイテッド 物理環境の視覚画像を仮想現実に統合するための空間関係
WO2019126665A1 (en) 2017-12-22 2019-06-27 Magic Leap, Inc. Viewpoint dependent brick selection for fast volumetric reconstruction

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5886702A (en) * 1996-10-16 1999-03-23 Real-Time Geometry Corporation System and method for computer modeling of 3D objects or surfaces by mesh constructions having optimal quality characteristics and dynamic resolution capabilities
US6879324B1 (en) * 1998-07-14 2005-04-12 Microsoft Corporation Regional progressive meshes
US7961186B2 (en) * 2005-09-26 2011-06-14 Siemens Medical Solutions Usa, Inc. Brick-based fusion renderer
FR2897455A1 (fr) * 2006-02-13 2007-08-17 Univ Hokkaido Nat Univ Corp Dispositif, procede et programme de segmentation de donnees de modele en treillis
US8774512B2 (en) * 2009-02-11 2014-07-08 Thomson Licensing Filling holes in depth maps
US8933925B2 (en) * 2009-06-15 2015-01-13 Microsoft Corporation Piecewise planar reconstruction of three-dimensional scenes
US9110194B2 (en) * 2009-09-14 2015-08-18 Exxonmobil Upstream Research Company System and method for providing a time-based representation of data
KR101697184B1 (ko) 2010-04-20 2017-01-17 삼성전자주식회사 메쉬 생성 장치 및 그 방법, 그리고, 영상 처리 장치 및 그 방법
US20120154400A1 (en) * 2010-12-20 2012-06-21 General Electric Company Method of reducing noise in a volume-rendered image
US8587583B2 (en) * 2011-01-31 2013-11-19 Microsoft Corporation Three-dimensional environment reconstruction
US8840466B2 (en) * 2011-04-25 2014-09-23 Aquifi, Inc. Method and system to create three-dimensional mapping in a two-dimensional game
US8879828B2 (en) * 2011-06-29 2014-11-04 Matterport, Inc. Capturing and aligning multiple 3-dimensional scenes
US9255813B2 (en) * 2011-10-14 2016-02-09 Microsoft Technology Licensing, Llc User controlled real object disappearance in a mixed reality display
US8638989B2 (en) * 2012-01-17 2014-01-28 Leap Motion, Inc. Systems and methods for capturing motion in three-dimensional space
CN103472909B (zh) * 2012-04-10 2017-04-12 微软技术许可有限责任公司 用于头戴式、增强现实显示器的逼真遮挡
US9645394B2 (en) * 2012-06-25 2017-05-09 Microsoft Technology Licensing, Llc Configured virtual environments
US20150024337A1 (en) * 2013-07-18 2015-01-22 A.Tron3D Gmbh Voxel level new information updates using intelligent weighting
US9996974B2 (en) * 2013-08-30 2018-06-12 Qualcomm Incorporated Method and apparatus for representing a physical scene
KR102184766B1 (ko) * 2013-10-17 2020-11-30 삼성전자주식회사 3차원 모델 복원 시스템 및 방법
US10203762B2 (en) * 2014-03-11 2019-02-12 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US10852838B2 (en) * 2014-06-14 2020-12-01 Magic Leap, Inc. Methods and systems for creating virtual and augmented reality
US9865089B2 (en) * 2014-07-25 2018-01-09 Microsoft Technology Licensing, Llc Virtual reality environment with real world objects
WO2016077798A1 (en) * 2014-11-16 2016-05-19 Eonite Perception Inc. Systems and methods for augmented reality preparation, processing, and application
KR102317247B1 (ko) * 2015-06-15 2021-10-26 한국전자통신연구원 영상정보를 이용한 증강현실 기반 손 인터랙션 장치 및 방법
US20170185141A1 (en) * 2015-12-29 2017-06-29 Microsoft Technology Licensing, Llc Hand tracking for interaction feedback
US10565791B2 (en) * 2015-12-29 2020-02-18 Microsoft Technology Licensing, Llc Tracking rigged polygon-mesh models of articulated objects
CN106373085A (zh) * 2016-09-20 2017-02-01 福州大学 一种基于增强现实的智能终端3d手表试戴方法及系统
US10297085B2 (en) * 2016-09-28 2019-05-21 Intel Corporation Augmented reality creations with interactive behavior and modality assignments
US10803616B1 (en) * 2017-04-13 2020-10-13 Facebook Technologies, Llc Hand calibration using single depth camera
US10929654B2 (en) 2018-03-12 2021-02-23 Nvidia Corporation Three-dimensional (3D) pose estimation from a monocular camera
CN109471521A (zh) * 2018-09-05 2019-03-15 华东计算技术研究所（中国电子科技集团公司第三十二研究所） Ar环境下的虚实遮挡交互方法及系统
US11775836B2 (en) 2019-05-21 2023-10-03 Magic Leap, Inc. Hand pose estimation
JP7711002B2 (ja) 2019-06-28 2025-07-22 マジック リープ， インコーポレイテッド 動的オクルージョンのための高速の手のメッシュ化

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006343953A (ja) 2005-06-08 2006-12-21 Canon Inc 画像処理方法、画像処理装置
JP2014514653A (ja) 2011-03-29 2014-06-19 クアルコム，インコーポレイテッド 各ユーザの視点に対する共有デジタルインターフェースのレンダリングのためのシステム
US20170038850A1 (en) 2012-04-09 2017-02-09 Shahar Fleishman System and method for combining three-dimensional tracking with a three-dimensional display for a user interface
WO2014128788A1 (ja) 2013-02-19 2014-08-28 株式会社ブリリアントサービス 形状認識装置、形状認識プログラム、および形状認識方法
JP2017529635A (ja) 2014-06-14 2017-10-05 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． 仮想および拡張現実を作成する方法およびシステム
JP2016115148A (ja) 2014-12-15 2016-06-23 キヤノンマーケティングジャパン株式会社 情報処理装置、情報処理システム、情報処理方法、及びプログラム
JP2019514101A (ja) 2016-03-24 2019-05-30 クアルコム，インコーポレイテッド 物理環境の視覚画像を仮想現実に統合するための空間関係
JP2018022292A (ja) 2016-08-02 2018-02-08 キヤノン株式会社 情報処理装置、情報処理装置の制御方法およびプログラム
WO2019126665A1 (en) 2017-12-22 2019-06-27 Magic Leap, Inc. Viewpoint dependent brick selection for fast volumetric reconstruction

Also Published As

Publication number Publication date
WO2020264149A1 (en) 2020-12-30
EP3991142A4 (en) 2023-06-28
CN114026606B (zh) 2025-06-17
JP2022537817A (ja) 2022-08-30
US20220114788A1 (en) 2022-04-14
EP3991142A1 (en) 2022-05-04
US20200410753A1 (en) 2020-12-31
CN114026606A (zh) 2022-02-08
JP2025107314A (ja) 2025-07-17
US11238651B2 (en) 2022-02-01
EP3991142B1 (en) 2026-04-08
US11620792B2 (en) 2023-04-04

Similar Documents

Publication Publication Date Title
JP7711002B2 (ja) 動的オクルージョンのための高速の手のメッシュ化
JP7573017B2 (ja) 深度情報を用いた高速３ｄ再構築
JP7646788B2 (ja) クロスリアリティシステム
CN114051629B (zh) 密集3d重建数据的缓存和更新
US11587298B2 (en) Caching and updating of dense 3D reconstruction data
EP3729376B1 (en) Method of occlusion rendering using raycast and live depth
US12100098B2 (en) Simple environment solver using planar extraction
JP7667866B2 (ja) 占有率グリッドを使用した場面理解
JP7752134B2 (ja) ３ｄ環境の合成表現を算出するための算出上効率的方法

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20230621

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20230621

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20240729

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20240802

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20241105

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20250129

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20250430

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20250619

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20250709

R150 Certificate of patent or registration of utility model

Ref document number: 7711002

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150