EP4058974A4 - Depth data model training with upsampling, losses, and loss balancing - Google Patents

Depth data model training with upsampling, losses, and loss balancing Download PDF

Info

Publication number
EP4058974A4
EP4058974A4 EP20888621.8A EP20888621A EP4058974A4 EP 4058974 A4 EP4058974 A4 EP 4058974A4 EP 20888621 A EP20888621 A EP 20888621A EP 4058974 A4 EP4058974 A4 EP 4058974A4
Authority
EP
European Patent Office
Prior art keywords
upsampling
losses
data model
model training
depth data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP20888621.8A
Other languages
German (de)
French (fr)
Other versions
EP4058974A1 (en
Inventor
Thomas Oscar DUDZIK
Kratarth Goel
Praveen Srinivasan
Sarah Tariq
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zoox Inc
Original Assignee
Zoox Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/684,568 external-priority patent/US11157774B2/en
Priority claimed from US16/684,554 external-priority patent/US20210150278A1/en
Application filed by Zoox Inc filed Critical Zoox Inc
Publication of EP4058974A1 publication Critical patent/EP4058974A1/en
Publication of EP4058974A4 publication Critical patent/EP4058974A4/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
EP20888621.8A 2019-11-14 2020-11-09 Depth data model training with upsampling, losses, and loss balancing Pending EP4058974A4 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US16/684,568 US11157774B2 (en) 2019-11-14 2019-11-14 Depth data model training with upsampling, losses, and loss balancing
US16/684,554 US20210150278A1 (en) 2019-11-14 2019-11-14 Depth data model training
PCT/US2020/059686 WO2021096806A1 (en) 2019-11-14 2020-11-09 Depth data model training with upsampling, losses, and loss balancing

Publications (2)

Publication Number Publication Date
EP4058974A1 EP4058974A1 (en) 2022-09-21
EP4058974A4 true EP4058974A4 (en) 2023-12-13

Family

ID=75912561

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20888621.8A Pending EP4058974A4 (en) 2019-11-14 2020-11-09 Depth data model training with upsampling, losses, and loss balancing

Country Status (4)

Country Link
EP (1) EP4058974A4 (en)
JP (1) JP2023503827A (en)
CN (1) CN114981834A (en)
WO (1) WO2021096806A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023050381A1 (en) * 2021-09-30 2023-04-06 Guangdong Oppo Mobile Telecommunications Corp., Ltd. Image and video coding using multi-sensor collaboration
CN113591823B (en) * 2021-10-08 2022-03-25 北京的卢深视科技有限公司 Depth prediction model training and face depth image generation method and device
GB2611765B (en) * 2021-10-08 2024-01-31 Samsung Electronics Co Ltd Method, system and apparatus for monocular depth estimation
CN117333626B (en) * 2023-11-28 2024-04-26 深圳魔视智能科技有限公司 Image sampling data acquisition method, device, computer equipment and storage medium

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190295282A1 (en) * 2018-03-21 2019-09-26 Nvidia Corporation Stereo depth estimation using deep neural networks

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9349192B2 (en) * 2012-04-24 2016-05-24 Lg Electronics Inc. Method and apparatus for processing video signal
GB2553782B (en) * 2016-09-12 2021-10-20 Niantic Inc Predicting depth from image data using a statistical model
CN109146944B (en) * 2018-10-30 2020-06-26 浙江科技学院 Visual depth estimation method based on depth separable convolutional neural network
CN110175986B (en) * 2019-04-23 2021-01-08 浙江科技学院 Stereo image visual saliency detection method based on convolutional neural network

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190295282A1 (en) * 2018-03-21 2019-09-26 Nvidia Corporation Stereo depth estimation using deep neural networks

Non-Patent Citations (6)

* Cited by examiner, † Cited by third party
Title
BOVIK A C ET AL: "Image Quality Assessment: From Error Visibility to Structural Similarity", IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE, USA, vol. 13, no. 4, 1 April 2004 (2004-04-01), pages 600 - 612, XP011110418, ISSN: 1057-7149, DOI: 10.1109/TIP.2003.819861 *
CHEN PO-YI ET AL: "Towards Scene Understanding: Unsupervised Monocular Depth Estimation With Semantic-Aware Representation", 2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 15 June 2019 (2019-06-15), pages 2619 - 2627, XP033687468, DOI: 10.1109/CVPR.2019.00273 *
GUAN-HAO CHEN ET AL: "Edge-Based Structural Similarity for Image Quality Assessment", ACOUSTICS, SPEECH AND SIGNAL PROCESSING, 2006. ICASSP 2006 PROCEEDINGS . 2006 IEEE INTERNATIONAL CONFERENCE ON TOULOUSE, FRANCE 14-19 MAY 2006, PISCATAWAY, NJ, USA,IEEE, PISCATAWAY, NJ, USA, 14 May 2006 (2006-05-14), pages II, XP031386534, ISBN: 978-1-4244-0469-8, DOI: 10.1109/ICASSP.2006.1660497 *
JIN HAN LEE ET AL: "From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 24 July 2019 (2019-07-24), XP081615569 *
See also references of WO2021096806A1 *
YEVHEN KUZNIETSOV ET AL: "Semi-Supervised Deep Learning for Monocular Depth Map Prediction", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 9 February 2017 (2017-02-09), XP080747211, DOI: 10.1109/CVPR.2017.238 *

Also Published As

Publication number Publication date
JP2023503827A (en) 2023-02-01
EP4058974A1 (en) 2022-09-21
WO2021096806A1 (en) 2021-05-20
CN114981834A (en) 2022-08-30

Similar Documents

Publication Publication Date Title
EP4058974A4 (en) Depth data model training with upsampling, losses, and loss balancing
EP3709226A4 (en) Model training system and method, and storage medium
EP3876125A4 (en) Model parameter training method based on federated learning, terminal, system and medium
EP3805988A4 (en) Training method for model, storage medium and computer device
EP3703332A4 (en) Graph structure model training and junk account identification
EP3726426A4 (en) Classification training method, server and storage medium
EP3910841A4 (en) Method executed by user equipment, and user equipment
EP3876589A4 (en) User access control method, information transmission method and apparatuses therefor
EP3270239A4 (en) Device characteristic model learning device, device characteristic model learning method, and storage medium
EP3832935A4 (en) Sidelink information transmission method, and terminal
EP4004828A4 (en) Training methods for deep networks
EP3881819A4 (en) Two-dosage-form essence and preparation method therefor
EP3600582A4 (en) Mission-based, game-implemented cyber training system and method
EP3968243A4 (en) Method and apparatus for realizing model training, and computer storage medium
EP3637662A4 (en) Data transmission method, and new as sublayer entity
EP3889846A4 (en) Deep learning model training method and system
EP3965341A4 (en) Rate matching method, device and storage medium
EP3626449A4 (en) Laminate, kit, and method for producing laminate
EP4021928A4 (en) Modified n-810 and methods therefor
EP3949367A4 (en) Systems and methods for providing context data associated with a communications session to the called device
EP3962564A4 (en) Articulated stiffening cannulas, access sets, and methods thereof
EP4027547A4 (en) Communication method, device, and storage medium
EP3992678A4 (en) Display medium, authenticity determination method, and article including display medium
EP3841451A4 (en) Multiview display, system, and method with user tracking
EP4007419A4 (en) Transmission configuration determination method, information configuration method, and device

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220510

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Free format text: PREVIOUS MAIN CLASS: G06T0001400000

Ipc: G06T0007500000

A4 Supplementary search report drawn up and despatched

Effective date: 20231109

RIC1 Information provided on ipc code assigned before grant

Ipc: G06N 20/00 20190101ALI20231103BHEP

Ipc: G06T 7/593 20170101ALI20231103BHEP

Ipc: G06T 7/50 20170101AFI20231103BHEP