CA2916062C - Stereo vision for sensing vehicles operating environment - Google Patents

Stereo vision for sensing vehicles operating environment Download PDF

Info

Publication number
CA2916062C
CA2916062C CA2916062A CA2916062A CA2916062C CA 2916062 C CA2916062 C CA 2916062C CA 2916062 A CA2916062 A CA 2916062A CA 2916062 A CA2916062 A CA 2916062A CA 2916062 C CA2916062 C CA 2916062C
Authority
CA
Canada
Prior art keywords
point cloud
data
vision system
camera
vehicle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CA2916062A
Other languages
English (en)
French (fr)
Other versions
CA2916062A1 (en
Inventor
John H. Posselius
Christopher A. Foster
Bret T. Turpin
Daniel J. Morwood
Thomas M. Petroff
Brad A. Baillio
Chad B. Jeppesen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CNH Industrial America LLC
Autonomous Solutions Inc
Original Assignee
CNH Industrial America LLC
Autonomous Solutions Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CNH Industrial America LLC, Autonomous Solutions Inc filed Critical CNH Industrial America LLC
Publication of CA2916062A1 publication Critical patent/CA2916062A1/en
Application granted granted Critical
Publication of CA2916062C publication Critical patent/CA2916062C/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/156—Mixing image signals
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G05D1/0251—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/15—Processing image signals for colour aspects of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more two-dimensional [2D] image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/25—Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/20—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from infrared radiation only
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0077—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0092—Image segmentation from stereoscopic image signals

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)
CA2916062A 2014-04-18 2015-04-17 Stereo vision for sensing vehicles operating environment Active CA2916062C (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201461981408P 2014-04-18 2014-04-18
US61/981,408 2014-04-18
PCT/US2015/026388 WO2015161208A1 (en) 2014-04-18 2015-04-17 Stereo vision for sensing vehicles operating environment

Publications (2)

Publication Number Publication Date
CA2916062A1 CA2916062A1 (en) 2015-10-22
CA2916062C true CA2916062C (en) 2020-08-11

Family

ID=53190010

Family Applications (1)

Application Number Title Priority Date Filing Date
CA2916062A Active CA2916062C (en) 2014-04-18 2015-04-17 Stereo vision for sensing vehicles operating environment

Country Status (4)

Country Link
US (1) US9819925B2 (https=)
BR (1) BR112015030886B1 (https=)
CA (1) CA2916062C (https=)
WO (1) WO2015161208A1 (https=)

Families Citing this family (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10183667B2 (en) 2015-09-15 2019-01-22 Deere & Company Human presence detection on a mobile machine
US9963246B2 (en) * 2016-03-28 2018-05-08 Amazon Technologies, Inc. Combining depth and thermal information for object detection and avoidance
US10394237B2 (en) 2016-09-08 2019-08-27 Ford Global Technologies, Llc Perceiving roadway conditions from fused sensor data
DE102016119592A1 (de) 2016-10-14 2018-05-03 Connaught Electronics Ltd. Verfahren zum Erkennen von Objekten in einem Umgebungsbereich eines Kraftfahrzeugs unter Berücksichtigung von Sensordaten im infraroten Wellenlängenbereich, Objekterkennungsvorrichtung, Fahrerassistenzsystem sowie Kraftfahrzeug
CN106595631B (zh) * 2016-10-25 2019-08-23 纳恩博（北京）科技有限公司 一种躲避障碍物的方法及电子设备
EP4035943B1 (en) * 2016-11-01 2023-08-30 Panasonic Intellectual Property Corporation of America Display method and display device
CN108733039A (zh) * 2017-04-18 2018-11-02 广东工业大学 一种机器人室内导航定位的方法与装置
US11392133B2 (en) 2017-06-06 2022-07-19 Plusai, Inc. Method and system for object centric stereo in autonomous driving vehicles
US11042155B2 (en) 2017-06-06 2021-06-22 Plusai Limited Method and system for closed loop perception in autonomous driving vehicles
US11573573B2 (en) 2017-06-06 2023-02-07 Plusai, Inc. Method and system for distributed learning and adaptation in autonomous driving vehicles
EP3690398B1 (en) * 2017-09-29 2023-12-27 Panasonic Intellectual Property Corporation of America Three-dimensional data creation method, client device and server
TWI708210B (zh) * 2017-12-25 2020-10-21 宏達國際電子股份有限公司 三維模型重建方法、電子裝置及非暫態電腦可讀取記錄媒體
US11074700B2 (en) 2018-04-23 2021-07-27 Cognex Corporation Systems, methods, and computer-readable storage media for determining saturation data for a temporal pixel
US11385338B2 (en) 2019-03-06 2022-07-12 Cnh Industrial America Llc System and method for disregarding obscured sensor data during the performance of an agricultural operation
WO2020219694A1 (en) * 2019-04-23 2020-10-29 Apple Inc. Systems and methods for resolving hidden features in a field of view
KR102925564B1 (ko) * 2019-05-20 2026-02-11 삼성전자주식회사 첨단 운전자 지원 장치, 이의 캘리브레이션 방법 및 이의 객체를 검출하는 방법
US12155811B2 (en) * 2019-06-11 2024-11-26 Lg Electronics Inc. Dust measurement device
WO2020263972A1 (en) 2019-06-24 2020-12-30 Event Capture Systems, Inc. Methods, systems, and devices for monitoring a web of material translating along a travel path
WO2020263984A1 (en) 2019-06-24 2020-12-30 Event Capture Systems, Inc. Methods, devices, and systems for headlight illumination for semi-autonomous vehicles
WO2021074868A1 (en) * 2019-10-18 2021-04-22 Telefonaktiebolaget Lm Ericsson (Publ) System of morphology recognition for optimizing rf propagation model
US12016257B2 (en) 2020-02-19 2024-06-25 Sabanto, Inc. Methods for detecting and clearing debris from planter gauge wheels, closing wheels and seed tubes
US11567197B2 (en) * 2020-02-20 2023-01-31 SafeAI, Inc. Automated object detection in a dusty environment
US12461083B2 (en) 2020-08-03 2025-11-04 Sabanto, Inc. Methods for improved agricultural procedures
US11917272B2 (en) 2020-10-28 2024-02-27 Semiconductor Components Industries, Llc Imaging systems for multi-spectral imaging
WO2022165705A1 (zh) * 2021-02-04 2022-08-11 深圳市大疆创新科技有限公司 低光环境检测方法及自动驾驶方法
CN113643345A (zh) * 2021-07-27 2021-11-12 数量级(上海)信息技术有限公司 一种基于双光融合的多目道路智能识别方法
US12333751B2 (en) 2022-01-19 2025-06-17 Ford Global Technologies, Llc Object detection
IT202200001616A1 (it) * 2022-01-31 2023-07-31 Cnh Ind Italia Spa Metodo per la gestione di un veicolo agricolo durante il processo di raccolta in una piantagione di alberi da frutto, come un aranceto

Family Cites Families (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003075893A (ja) 2001-09-06 2003-03-12 Murakami Corp 車両用周囲撮像装置
US7995095B2 (en) 2001-10-18 2011-08-09 Autoliv Development Ab Night vision device for a vehicle
DE10301468B4 (de) 2002-01-18 2010-08-05 Honda Giken Kogyo K.K. Vorrichtung zur Beobachtung der Umgebung eines Fahrzeugs
US7834905B2 (en) 2002-06-18 2010-11-16 Bayerische Motoren Werke Aktiengesellschaft Method and system for visualizing the environment of a vehicle with a distance-dependent merging of an infrared and a visual image
ATE386951T1 (de) 2002-08-05 2008-03-15 Elbit Systems Ltd Nachtsichtabbildungssystem und -verfahren zur montage in einem fahrzeug
DE10304703B4 (de) 2003-02-06 2023-03-16 Bayerische Motoren Werke Aktiengesellschaft Verfahren und Vorrichtung zur Sichtbarmachung der Umgebung eines Fahrzeugs mit umgebungsabhängiger Fusion eines Infrarot- und eines Visuell-Abbilds
DE10346482B4 (de) 2003-10-02 2008-08-28 Daimler Ag Vorrichtung zur Verbesserung der Sichtverhältnisse in einem Kraftfahrzeug
CA2505715A1 (en) * 2004-05-03 2005-11-03 Her Majesty In Right Of Canada As Represented By The Minister Of National Defence Volumetric sensor for mobile robotics
JP2005350010A (ja) 2004-06-14 2005-12-22 Fuji Heavy Ind Ltd ステレオ式車外監視装置
DE102004061998A1 (de) 2004-12-23 2006-07-06 Robert Bosch Gmbh Stereokamera für ein Kraftfahrzeug
JP2006203448A (ja) 2005-01-19 2006-08-03 Hitachi Ltd 車載ステレオカメラ装置
WO2006084385A1 (en) * 2005-02-11 2006-08-17 Macdonald Dettwiler & Associates Inc. 3d imaging system
US8519837B2 (en) 2005-09-08 2013-08-27 Johnson Controls Gmbh Driver assistance device for a vehicle and a method for visualizing the surroundings of a vehicle
JP4793638B2 (ja) 2006-03-27 2011-10-12 マツダ株式会社 車両用歩行者検出装置
US8108092B2 (en) 2006-07-14 2012-01-31 Irobot Corporation Autonomous behaviors for a remote vehicle
JP4807152B2 (ja) 2006-06-15 2011-11-02 住友電気工業株式会社 障害物検出システム及び障害物検出方法
JP4816308B2 (ja) 2006-07-31 2011-11-16 住友電気工業株式会社 遠赤外線撮像システム及び遠赤外線撮像方法
JP4267657B2 (ja) 2006-10-31 2009-05-27 本田技研工業株式会社 車両周辺監視装置
DE102006053109B3 (de) 2006-11-10 2008-06-12 Audi Ag Verfahren zur bildhaften Darstellung einer Fahrzeugumgebung und Bilderfassungssystem
KR100846576B1 (ko) 2007-02-28 2008-07-16 (주)한비젼 열적외선 카메라를 장착한 자동차 사이드 미러
US20090180668A1 (en) * 2007-04-11 2009-07-16 Irobot Corporation System and method for cooperative remote vehicle behavior
GB0808340D0 (en) * 2008-05-08 2008-06-18 Univ Edinburgh Remote sensing system
EP2254091B1 (en) 2009-05-19 2020-03-25 Veoneer Sweden AB Vision system and method for a motor vehicle
US8400511B2 (en) 2009-12-04 2013-03-19 Lockheed Martin Corporation Optical detection and ranging sensor system for sense and avoid, and related methods
US20120056982A1 (en) 2010-09-08 2012-03-08 Microsoft Corporation Depth camera based on structured light and stereo vision
US8780179B2 (en) * 2011-05-10 2014-07-15 Southwest Research Institute Robot vision with three dimensional thermal imaging
US9692991B2 (en) * 2011-11-04 2017-06-27 Qualcomm Incorporated Multispectral imaging system
WO2014106755A1 (en) * 2013-01-07 2014-07-10 Bae Systems Plc Image processing
US9449227B2 (en) * 2014-01-08 2016-09-20 Here Global B.V. Systems and methods for creating an aerial image

Also Published As

Publication number Publication date
US20170064278A1 (en) 2017-03-02
WO2015161208A1 (en) 2015-10-22
US9819925B2 (en) 2017-11-14
BR112015030886B1 (pt) 2022-09-27
BR112015030886A2 (https=) 2017-07-25
CA2916062A1 (en) 2015-10-22

Similar Documents

Publication Publication Date Title
CA2916062C (en) Stereo vision for sensing vehicles operating environment
Fritz et al. UAV-based photogrammetric point clouds–tree stem mapping in open stands in comparison to terrestrial laser scanner point clouds
Shalal et al. Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion–Part A: Tree detection
CN118602997A (zh) 一种无人机多维空间面积测量系统
CN110988912A (zh) 自动驾驶车辆的道路目标与距离检测方法、系统、装置
Campos et al. Spatio-temporal analysis for obstacle detection in agricultural videos
Maddern et al. LAPS-II: 6-DoF day and night visual localisation with prior 3D structure for autonomous road vehicles
Nguyen et al. Structure overview of vegetation detection. A novel approach for efficient vegetation detection using an active lighting system
Kuru et al. Vision-based remote sensing imagery datasets from Benkovac landmine test site using an autonomous drone for detecting landmine locations
Auer et al. Simulation-based interpretation and alignment of high-resolution optical and SAR images
CN116957360A (zh) 一种基于无人机的空间观测与重建方法及系统
Lee et al. Design considerations of a perception system in functional safety operated and highly automated mobile machines
Dhrafani et al. Firestereo: Forest infrared stereo dataset for uas depth perception in visually degraded environments
Nowakowski Perception technology for conversion of off-road vehicles for the purposes of unmanned missions
US10907960B1 (en) Calibration system for combined depth and texture sensor
Bradley Vegetation detection for mobile robot navigation
Yilmaz et al. Determination of tree crown diameters with segmentation of a UAS-Based canopy height model
Gong et al. Automated road extraction from LiDAR data based on intensity and aerial photo
Christiansen et al. Advanced sensor platform for human detection and protection in autonomous farming
Sizintsev et al. Multi-sensor fusion for motion estimation in visually-degraded environments
Núñez-Andrés et al. Vegetation filtering using colour for monitoring applications from photogrammetric data
Naimaee et al. Automatic extraction of control points from 3D Lidar mobile mapping and UAV imagery for aerial triangulation
Abdullah S et al. Capabilities of UAV-Based Watershed Segmentation Method for Estimating Tree Crown: A Case Study of Oil Palm Tree
Walter Object-based classification of integrated multispectral and LIDAR data for change detection and quality control in urban areas
Arriola-Valverde et al. Comparison of LiDAR and RGB Sensor Technologies Applied to Close-Range Remote Sensing

Legal Events

Date Code Title Description
EEER Examination request

Effective date: 20171218

MPN Maintenance fee for patent paid

Free format text: FEE DESCRIPTION TEXT: MF (PATENT, 10TH ANNIV.) - STANDARD

Year of fee payment: 10

U00 Fee paid

Free format text: ST27 STATUS EVENT CODE: A-4-4-U10-U00-U101 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE REQUEST RECEIVED

Effective date: 20250325

U11 Full renewal or maintenance fee paid

Free format text: ST27 STATUS EVENT CODE: A-4-4-U10-U11-U102 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE FEE PAYMENT DETERMINED COMPLIANT

Effective date: 20250325

Free format text: ST27 STATUS EVENT CODE: A-4-4-U10-U11-U102 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE FEE PAYMENT PAID IN FULL

Effective date: 20250325

MPN Maintenance fee for patent paid

Free format text: FEE DESCRIPTION TEXT: MF (PATENT, 11TH ANNIV.) - STANDARD

Year of fee payment: 11

U00 Fee paid

Free format text: ST27 STATUS EVENT CODE: A-4-4-U10-U00-U101 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE REQUEST RECEIVED

Effective date: 20260324

U11 Full renewal or maintenance fee paid

Free format text: ST27 STATUS EVENT CODE: A-4-4-U10-U11-U102 (AS PROVIDED BY THE NATIONAL OFFICE); EVENT TEXT: MAINTENANCE FEE PAYMENT PAID IN FULL

Effective date: 20260324