EP3857451A4 - Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications - Google Patents

Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications Download PDF

Info

Publication number
EP3857451A4
EP3857451A4 EP19864640.8A EP19864640A EP3857451A4 EP 3857451 A4 EP3857451 A4 EP 3857451A4 EP 19864640 A EP19864640 A EP 19864640A EP 3857451 A4 EP3857451 A4 EP 3857451A4
Authority
EP
European Patent Office
Prior art keywords
employing
neural networks
dimensional
data predicted
modeling applications
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP19864640.8A
Other languages
German (de)
French (fr)
Other versions
EP3857451A1 (en
Inventor
David Alan GAUSEBECK
Matthew Tschudy BELL
Waleed K. Abdulla
Peter Kyuhee Hahn
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Matterport Inc
Original Assignee
Matterport Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/141,558 external-priority patent/US11094137B2/en
Application filed by Matterport Inc filed Critical Matterport Inc
Publication of EP3857451A1 publication Critical patent/EP3857451A1/en
Publication of EP3857451A4 publication Critical patent/EP3857451A4/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
EP19864640.8A 2018-09-25 2019-09-25 Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications Pending EP3857451A4 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/141,558 US11094137B2 (en) 2012-02-24 2018-09-25 Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications
PCT/US2019/053040 WO2020069049A1 (en) 2018-09-25 2019-09-25 Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications

Publications (2)

Publication Number Publication Date
EP3857451A1 EP3857451A1 (en) 2021-08-04
EP3857451A4 true EP3857451A4 (en) 2022-06-22

Family

ID=69949995

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19864640.8A Pending EP3857451A4 (en) 2018-09-25 2019-09-25 Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications

Country Status (3)

Country Link
EP (1) EP3857451A4 (en)
CN (1) CN112771539B (en)
WO (1) WO2020069049A1 (en)

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113330490B (en) * 2019-01-31 2024-04-23 华为技术有限公司 Three-dimensional (3D) assisted personalized home object detection
US11157774B2 (en) * 2019-11-14 2021-10-26 Zoox, Inc. Depth data model training with upsampling, losses, and loss balancing
US12052408B2 (en) * 2020-02-26 2024-07-30 Intel Corporation Depth based 3D reconstruction using an a-priori depth scene
WO2021226528A1 (en) * 2020-05-08 2021-11-11 Korman Max Virtual reality film hybridization
US20210397760A1 (en) * 2020-06-19 2021-12-23 Herman Miller, Inc. Generating Space Models And Geometry Models Using A Machine Learning System With Multi-Platform Interfaces
CN113890984B (en) * 2020-07-03 2022-12-27 华为技术有限公司 Photographing method, image processing method and electronic equipment
EP3944183A1 (en) * 2020-07-20 2022-01-26 Hexagon Technology Center GmbH Method and system for enhancing images using machine learning
EP4186033A2 (en) * 2020-07-21 2023-05-31 InterDigital CE Patent Holdings, SAS Map for augmented reality
CN112150608B (en) * 2020-09-07 2024-07-23 鹏城实验室 Three-dimensional face reconstruction method based on graph convolution neural network
US11393179B2 (en) * 2020-10-09 2022-07-19 Open Space Labs, Inc. Rendering depth-based three-dimensional model with integrated image frames
CN112396703B (en) * 2020-11-18 2024-01-12 北京工商大学 Reconstruction method of single-image three-dimensional point cloud model
US11860641B2 (en) * 2021-01-28 2024-01-02 Caterpillar Inc. Visual overlays for providing perception of depth
WO2022176132A1 (en) * 2021-02-18 2022-08-25 株式会社Live2D Inference model construction method, inference model construction device, program, recording medium, configuration device, and configuration method
US11798288B2 (en) 2021-03-16 2023-10-24 Toyota Research Institute, Inc. System and method for generating a training set for improving monocular object detection
US12086997B2 (en) * 2021-04-27 2024-09-10 Faro Technologies, Inc. Hybrid feature matching between intensity image and color image
CN113223173B (en) * 2021-05-11 2022-06-07 华中师范大学 Three-dimensional model reconstruction migration method and system based on graph model
CN113099847B (en) * 2021-05-25 2022-03-08 广东技术师范大学 Fruit picking method based on fruit three-dimensional parameter prediction model
JP7414332B2 (en) * 2021-06-23 2024-01-16 スリーアイ インコーポレーテッド Depth map image generation method and computing device therefor
CN113793255A (en) * 2021-09-09 2021-12-14 百度在线网络技术（北京）有限公司 Method, apparatus, device, storage medium and program product for image processing
CN113962274B (en) * 2021-11-18 2022-03-08 腾讯科技（深圳）有限公司 Abnormity identification method and device, electronic equipment and storage medium
TWI817266B (en) * 2021-11-29 2023-10-01 邦鼎科技有限公司 Display system of sample house
CN114693670B (en) * 2022-04-24 2023-05-23 西京学院 Ultrasonic detection method for weld defects of longitudinal submerged arc welded pipe based on multi-scale U-Net
CN115861572B (en) * 2023-02-24 2023-05-23 腾讯科技（深圳）有限公司 Three-dimensional modeling method, device, equipment and storage medium
US20240303843A1 (en) * 2023-03-07 2024-09-12 Snap Inc. Depth estimation from rgb images
CN117291845B (en) * 2023-11-27 2024-03-19 成都理工大学 Point cloud ground filtering method, system, electronic equipment and storage medium

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2305735C (en) * 1997-12-05 2008-01-08 Dynamic Digital Depth Research Pty. Ltd. Improved image conversion and encoding techniques
US7697749B2 (en) * 2004-08-09 2010-04-13 Fuji Jukogyo Kabushiki Kaisha Stereo image processing device
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
US10706615B2 (en) * 2015-12-08 2020-07-07 Matterport, Inc. Determining and/or generating data for an architectural opening area associated with a captured three-dimensional model
WO2018140656A1 (en) * 2017-01-26 2018-08-02 Matterport, Inc. Capturing and aligning panoramic image and depth data
CN106709481A (en) * 2017-03-03 2017-05-24 深圳市唯特视科技有限公司 Indoor scene understanding method based on 2D-3D semantic data set

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
See also references of WO2020069049A1 *
SU YU-CHUAN ET AL: "Learning Spherical Convolution for Fast Features from 360 deg Imagery", NIPS'17: PROCEEDINGS OF THE 31ST INTERNATIONAL CONFERENCE ON NEURAL INFORMATION PROCESSING SYSTEMS, 2 August 2017 (2017-08-02), XP055919745, Retrieved from the Internet <URL:https://proceedings.neurips.cc/paper/2017/file/0c74b7f78409a4022a2c4c5a5ca3ee19-Paper.pdf> [retrieved on 20220510] *
vol. 11210, 16 September 2018, SPRINGER INTERNATIONAL PUBLISHING, Cham, ISBN: 978-3-030-58594-5, article ZIOULIS NIKOLAOS ET AL: "OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas", pages: 453 - 471, XP055820431, DOI: 10.1007/978-3-030-01231-1_28 *
ZHAO QIANG ET AL: "Distortion-aware CNNs for Spherical Images", PROCEEDINGS OF THE TWENTY-SEVENTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 1 July 2018 (2018-07-01), California, pages 1198 - 1204, XP055879853, ISBN: 978-0-9992411-2-7, Retrieved from the Internet <URL:https://www.ijcai.org/proceedings/2018/0167.pdf> DOI: 10.24963/ijcai.2018/167 *

Also Published As

Publication number Publication date
CN112771539B (en) 2023-08-25
CN112771539A (en) 2021-05-07
WO2020069049A1 (en) 2020-04-02
EP3857451A1 (en) 2021-08-04

Similar Documents

Publication Publication Date Title
EP3857451A4 (en) Employing three-dimensional data predicted from two-dimensional images using neural networks for 3d modeling applications
EP3737997A4 (en) Systems and methods for rendering data from a 3d environment
EP3867883A4 (en) 3d scene synthesis techniques using neural network architectures
EP3850832A4 (en) Motion estimation using 3d auxiliary data
EP3955213A4 (en) Three-dimensional object modeling method, image processing method, and image processing device
EP3449274A4 (en) Three-dimensional imaging systems
EP4066169A4 (en) Systems and methods for constructing a three-dimensional model from two-dimensional images
GB202018574D0 (en) Method for reconstructing three-dimensional space scene on basis of photography
EP3519125A4 (en) Three-dimensional (3d) printing
EP3397455A4 (en) Three-dimensional (3d) printing
EP3436236A4 (en) Forming three-dimensional (3d) electronic parts
EP3408071A4 (en) Three-dimensional (3d) object printing simulator
EP3606084A4 (en) Method for transmitting data about three-dimensional image
EP3455054A4 (en) Three-dimensional (3d) printing
EP3626429A4 (en) Layering method for three-dimensional object data, and 3d printing method and device
EP3408046A4 (en) Three-dimensional (3d) printing
EP3984000A4 (en) Data serialization extrusion for converting two-dimensional images to three-dimensional geometry
EP3421930A4 (en) Three-dimensional shape data and texture information generation system, photographing control program, and three-dimensional shape data and texture information generation method
EP3655233A4 (en) Three-dimensional (3d) printing
EP3445564A4 (en) Three-dimensional (3d) printing
EP3419814A4 (en) Three-dimensional (3d) printing
EP3648912A4 (en) Three-dimensional (3d) printing
EP3750689A4 (en) Three-dimensional forming method
EP3368972A4 (en) Halftoning of object data for a three-dimensional object
EP3445516A4 (en) Three-dimensional (3d) printing

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210423

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20220523

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 3/04 20060101ALI20220517BHEP

Ipc: G06N 7/00 20060101ALI20220517BHEP

Ipc: G06N 3/08 20060101ALI20220517BHEP

Ipc: G06T 7/50 20170101ALI20220517BHEP

Ipc: G06T 7/593 20170101ALI20220517BHEP

Ipc: G06T 17/00 20060101ALI20220517BHEP

Ipc: G06N 3/02 20060101AFI20220517BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20240516