AU2021211677B2 - Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system - Google Patents

Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system Download PDF

Info

Publication number
AU2021211677B2
AU2021211677B2 AU2021211677A AU2021211677A AU2021211677B2 AU 2021211677 B2 AU2021211677 B2 AU 2021211677B2 AU 2021211677 A AU2021211677 A AU 2021211677A AU 2021211677 A AU2021211677 A AU 2021211677A AU 2021211677 B2 AU2021211677 B2 AU 2021211677B2
Authority
AU
Australia
Prior art keywords
data
point cloud
depth
scene
cameras
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
AU2021211677A
Other languages
English (en)
Other versions
AU2021211677A1 (en
Inventor
David Julio COLMENARES
Thomas Ivan Nonn
James Andrew YOUNGQUIST
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Proprio Inc
Original Assignee
Proprio Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Proprio Inc filed Critical Proprio Inc
Publication of AU2021211677A1 publication Critical patent/AU2021211677A1/en
Application granted granted Critical
Publication of AU2021211677B2 publication Critical patent/AU2021211677B2/en
Priority to AU2024202120A priority Critical patent/AU2024202120A1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/77—Retouching; Inpainting; Scratch removal
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/557—Depth or shape recovery from multiple images from light fields, e.g. from plenoptic cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G06V20/647—Three-dimensional [3D] objects by matching two-dimensional images to three-dimensional objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more two-dimensional [2D] image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20092—Interactive image processing based on input by user
    • G06T2207/20104—Interactive definition of region of interest [ROI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30008—Bone
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/41—Medical
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/56—Particle system, point based geometry or rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Signal Processing (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Processing Or Creating Images (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Measurement Of Optical Distance (AREA)
AU2021211677A 2020-01-21 2021-01-21 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system Active AU2021211677B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2024202120A AU2024202120A1 (en) 2020-01-21 2024-04-03 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US202062963717P 2020-01-21 2020-01-21
US62/963,717 2020-01-21
PCT/US2021/014397 WO2021150741A1 (en) 2020-01-21 2021-01-21 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Related Child Applications (1)

Application Number Title Priority Date Filing Date
AU2024202120A Division AU2024202120A1 (en) 2020-01-21 2024-04-03 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Publications (2)

Publication Number Publication Date
AU2021211677A1 AU2021211677A1 (en) 2022-08-25
AU2021211677B2 true AU2021211677B2 (en) 2024-01-04

Family

ID=76857295

Family Applications (2)

Application Number Title Priority Date Filing Date
AU2021211677A Active AU2021211677B2 (en) 2020-01-21 2021-01-21 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system
AU2024202120A Pending AU2024202120A1 (en) 2020-01-21 2024-04-03 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Family Applications After (1)

Application Number Title Priority Date Filing Date
AU2024202120A Pending AU2024202120A1 (en) 2020-01-21 2024-04-03 Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Country Status (6)

Country Link
US (3) US11682165B2 (https=)
EP (1) EP4094186A4 (https=)
JP (2) JP2023511670A (https=)
AU (2) AU2021211677B2 (https=)
CA (1) CA3168799A1 (https=)
WO (1) WO2021150741A1 (https=)

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2021211677B2 (en) 2020-01-21 2024-01-04 Proprio, Inc. Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system
US11836940B2 (en) * 2020-06-15 2023-12-05 Zebra Technologies Corporation Three-dimensional sensor acuity recovery assistance
KR102797392B1 (ko) * 2021-08-03 2025-04-21 레이아 인코포레이티드 깊이 맵을 이용한 뷰 합성 시스템 및 방법
US12016642B2 (en) 2021-09-08 2024-06-25 Proprio, Inc. Constellations for tracking instruments, such as surgical instruments, and associated systems and methods
US12357397B2 (en) 2022-05-09 2025-07-15 Proprio, Inc. Methods and systems for calibrating instruments within an imaging system, such as a surgical imaging system
CN114782438B (zh) * 2022-06-20 2022-09-16 深圳市信润富联数字科技有限公司 物体点云修正方法、装置、电子设备和存储介质
CN115100385A (zh) * 2022-06-22 2022-09-23 众趣（北京）科技有限公司 一种三维场景中家具物品替换方法
GB2624621A (en) * 2022-11-15 2024-05-29 Continental Autonomous Mobility Germany GmbH 3-Dimensional (3D) map generation system and method for creating 3D map of surroundings of a vehicle
EP4734830A1 (en) * 2023-06-28 2026-05-06 Abbott Diabetes Care Inc. Systems, methods, and features for analyte monitoring
US11935209B1 (en) * 2023-07-24 2024-03-19 Illuscio, Inc. Systems and methods for dynamic backfilling of a three-dimensional object
WO2025064955A1 (en) * 2023-09-22 2025-03-27 Proprio, Inc. Methods and systems for generating three-dimensional renderings of a scene using a mobile sensor array, such as neural radiance field (nerf) renderings
WO2025171548A1 (en) * 2024-02-15 2025-08-21 Dental Imaging Technologies Corporation Point cloud registration for intraoral scanning applications
CN120910119B (zh) * 2025-10-09 2025-12-09 兰州交通大学 一种基于cbr的山区铁路隧道绿色施工决策方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170079724A1 (en) * 2012-10-29 2017-03-23 7D Surgical Inc. Integrated illumination and optical surface topology detection system and methods of use thereof
US20180225866A1 (en) * 2015-08-06 2018-08-09 Heptagon Micro Optics Pte. Ltd. Generating a merged, fused three-dimensional point cloud based on captured images of a scene
US10121064B2 (en) * 2015-04-16 2018-11-06 California Institute Of Technology Systems and methods for behavior detection using 3D tracking and machine learning
US20190236796A1 (en) * 2016-10-18 2019-08-01 Photonic Sensors & Algorithms, S.L. Device and method for obtaining distance information from views

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001231019A (ja) * 2000-02-16 2001-08-24 Sony Corp 撮像画像付き表示装置、モニタ装置、テレビ電話装置、プロンプタ付きカメラ装置、テレビ会議装置、並びに、反射エネルギ・モデル化装置
US7389041B2 (en) * 2005-02-01 2008-06-17 Eastman Kodak Company Determining scene distance in digital camera images
JP5057734B2 (ja) * 2006-09-25 2012-10-24 株式会社トプコン 測量方法及び測量システム及び測量データ処理プログラム
DE102006055641B4 (de) * 2006-11-22 2013-01-31 Visumotion Gmbh Anordnung und Verfahren zur Aufnahme und Wiedergabe von Bildern einer Szene und/oder eines Objektes
JP2009300268A (ja) * 2008-06-13 2009-12-24 Nippon Hoso Kyokai <Nhk> ３次元情報検出装置
US8780172B2 (en) * 2009-01-27 2014-07-15 Telefonaktiebolaget L M Ericsson (Publ) Depth and video co-processing
GB2490872B (en) * 2011-05-09 2015-07-29 Toshiba Res Europ Ltd Methods and systems for capturing 3d surface geometry
US9191646B2 (en) * 2011-08-29 2015-11-17 Nokia Technologies Oy Apparatus, a method and a computer program for video coding and decoding
JP6023324B2 (ja) * 2013-06-11 2016-11-09 敦 丹治 外科手術支援システム、外科手術支援装置、外科手術支援方法、外科手術支援プログラムおよび情報処理装置
EP3073894B1 (en) * 2013-11-27 2022-03-30 Children's National Medical Center Corrected 3d imaging
US9737370B2 (en) * 2014-10-14 2017-08-22 Synaptive Medical (Barbados) Inc. Patient reference tool
GB2556727B (en) 2015-06-22 2021-11-03 Synaptive Medical Inc System and method for mapping navigation space to patient space in a medical procedure
WO2017011892A1 (en) * 2015-07-21 2017-01-26 Synaptive Medical (Barbados) Inc. System and method for mapping navigation space to patient space in a medical procedure
WO2017015739A1 (en) * 2015-07-29 2017-02-02 Synaptive Medical (Barbados) Inc. Handheld scanner for rapid registration in a medical navigation system
JP6639276B2 (ja) * 2016-03-10 2020-02-05 キヤノン株式会社 画像処理装置およびその制御方法、撮像装置、プログラム
CN109792537B (zh) * 2016-09-29 2023-06-23 皇家飞利浦有限公司 用于提供图像的装置和方法以及图像处理装置和系统
CN110192390A (zh) * 2016-11-24 2019-08-30 华盛顿大学 头戴式显示器的光场捕获和渲染
US20200057778A1 (en) * 2017-04-11 2020-02-20 Siemens Mobility GmbH Depth image pose search with a bootstrapped-created database
US10395418B2 (en) * 2017-08-18 2019-08-27 Microsoft Technology Licensing, Llc Techniques for predictive prioritization of image portions in processing graphics
JP6623199B2 (ja) * 2017-09-28 2019-12-18 株式会社コロプラ ヘッドマウントデバイスを用いて仮想現実を提供するためにコンピュータで実行されるプログラムおよび情報処理装置
US10460509B2 (en) * 2017-11-07 2019-10-29 Dolby Laboratories Licensing Corporation Parameterizing 3D scenes for volumetric viewing
EP3804328B1 (en) 2018-06-29 2025-02-19 Proprio, Inc. Synthesizing an image from a virtual perspective using pixels from a physical imager array
US10627901B2 (en) * 2018-09-14 2020-04-21 Facebook Technologies, Llc Vergence determination
AU2021211677B2 (en) 2020-01-21 2024-01-04 Proprio, Inc. Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170079724A1 (en) * 2012-10-29 2017-03-23 7D Surgical Inc. Integrated illumination and optical surface topology detection system and methods of use thereof
US10121064B2 (en) * 2015-04-16 2018-11-06 California Institute Of Technology Systems and methods for behavior detection using 3D tracking and machine learning
US20180225866A1 (en) * 2015-08-06 2018-08-09 Heptagon Micro Optics Pte. Ltd. Generating a merged, fused three-dimensional point cloud based on captured images of a scene
US20190236796A1 (en) * 2016-10-18 2019-08-01 Photonic Sensors & Algorithms, S.L. Device and method for obtaining distance information from views

Also Published As

Publication number Publication date
US20210225020A1 (en) 2021-07-22
EP4094186A1 (en) 2022-11-30
US11682165B2 (en) 2023-06-20
AU2021211677A1 (en) 2022-08-25
CA3168799A1 (en) 2021-07-29
US12243162B2 (en) 2025-03-04
US20240005596A1 (en) 2024-01-04
JP2023511670A (ja) 2023-03-22
AU2024202120A1 (en) 2024-05-02
US20250252664A1 (en) 2025-08-07
WO2021150741A1 (en) 2021-07-29
EP4094186A4 (en) 2024-01-24
JP2026047369A (ja) 2026-03-13

Similar Documents

Publication Publication Date Title
AU2021211677B2 (en) Methods and systems for augmenting depth data from a depth sensor, such as with data from a multiview camera system
AU2021414195B2 (en) Methods and systems for registering preoperative image data to intraoperative image data of a scene, such as a surgical scene
US20250104323A1 (en) METHODS AND SYSTEMS FOR GENERATING THREE-DIMENSIONAL RENDERINGS OF A SCENE USING A MOBILE SENSOR ARRAY, SUCH AS NEURAL RADIANCE FIELD (NeRF) RENDERINGS
US7825948B2 (en) 3D video conferencing
US20130335535A1 (en) Digital 3d camera using periodic illumination
US20110187827A1 (en) Method and apparatus for creating a stereoscopic image
US20230196595A1 (en) Methods and systems for registering preoperative image data to intraoperative image data of a scene, such as a surgical scene
US20260114931A1 (en) Methods and systems for calibrating instruments within an imaging system, such as a surgical imaging system
US20250294123A1 (en) Methods for generating stereoscopic views in multicamera systems, and associated devices and systems
Kim et al. Tracking by detection for interactive image augmentation in laparoscopy
Cockshott et al. Experimental 3-D digital TV studio
KR20120056668A (ko) 3차원 정보 복원 장치 및 그 방법
Conen et al. Rectification and robust matching using oriented image triplets for minimally invasive surgery
US12450777B2 (en) Methods and systems for calibrating and/or verifying a calibration of an imaging system such as a surgical imaging system

Legal Events

Date Code Title Description
FGA Letters patent sealed or granted (standard patent)