WO2019138677A1 - 情報処理装置及びその制御方法及びコンピュータ可読記憶媒体、並びに、運転制御システム - Google Patents

情報処理装置及びその制御方法及びコンピュータ可読記憶媒体、並びに、運転制御システム Download PDF

Info

Publication number
WO2019138677A1
WO2019138677A1 PCT/JP2018/041667 JP2018041667W WO2019138677A1 WO 2019138677 A1 WO2019138677 A1 WO 2019138677A1 JP 2018041667 W JP2018041667 W JP 2018041667W WO 2019138677 A1 WO2019138677 A1 WO 2019138677A1
Authority
WO
WIPO (PCT)
Prior art keywords
geometric information
image
information
unit
imaging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/041667
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智昭 肥後
小竹　大輔
鈴木　雅博
望 糟谷
誠 冨岡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Publication of WO2019138677A1 publication Critical patent/WO2019138677A1/ja
Priority to US16/921,389 priority Critical patent/US11393123B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the defect indicates that the estimated geometric information and position and orientation are in a state largely different from the ideal geometric information and position and orientation.
  • the cause of the failure is noise due to the image coming from the imaging device or shooting scene environment, estimation due to a scene with insufficient learning in geometric information estimation using a learning model, calibration parameters A variety of factors can be considered, such as something different from the actual one, a failure of the imaging device itself, or the like.
  • FIG. 2 is a block diagram of the information processing device 1 and the operation processing device 12 in the present embodiment.
  • step S 1110 the measurement result processing unit 104 estimates the position and orientation of the current frame by linear interpolation from the position and orientation in the image one frame before and the image two frames before, and uses this as the initial position and orientation.
  • the initial position and orientation is not limited to this, and the initial position and orientation may be determined from an acceleration sensor, a GPS sensor, or the like.
  • the reliability is not limited to the luminance difference.
  • the reliability of the geometric information obtained from the method of Non-Patent Document 1 or the stereo matching degree is used as the reliability. May be
  • FIG. 7 is a flowchart showing the processing procedure of the measurement result processing unit 104 in step S1041 of FIG.
  • the measurement result processing unit 104 calculates the reliability of each of the first, second, and third geometric information.
  • the degree of reliability is obtained by quantifying the degree of reliability of each measurement point or each measurement area with respect to the first, second, and third geometric information. The higher the reliability, the more likely that the measurement point or the three-dimensional position where the measurement area exists is correct.
  • the method of determining the reliability for the first geometric information, when the correspondence between the first image and the second image is determined by stereo, the similarity for each minute region is calculated. The higher the degree of similarity is, the higher the degree of reliability is determined as the region of high reliability.
  • the reliability of the geometric information can be obtained by the method described in Non-Patent Document 1.
  • the usage application of the information processing apparatus 1 is not limited to position and orientation estimation, and may be used for three-dimensional reconstruction.
  • it may be used as a measurement system for generating a CAD model such as an industrial part or a building.
  • the measurement system at this time may further include a three-dimensional model generation unit that generates a three-dimensional model from the geometric information updated by the measurement result processing unit 104.
  • it may be used as an apparatus for acquiring a distance image with high accuracy from a camera which can not acquire a distance image, such as an RGB camera or a camera for acquiring a gray scale image.
  • the geometric information estimation unit 102 estimates a plurality of geometric information.
  • the position and orientation calculation unit 103 calculates a plurality of positions and orientations, and the measurement result processing unit 104 obtains a stable position and orientation, and updates the geometric information. Then, the geometric information and the position and orientation estimated by the cloud server are transferred to the information processing apparatus 1 using the communication unit. With such a configuration, since the information processing apparatus 1 can reduce the calculation load, it is possible to save space with a small-scale computer.
  • the measurement result processing unit 104 determines the presence or absence of the position and orientation defect.
  • step S 2220 the measurement result processing unit 104 performs weighted alignment of the geometric information with reliability obtained in step S 2210 and the geometric information in the surrounding environment map so far using weighted ICP.
  • the measurement result processing unit determines the presence or absence of a defect in the geometric information or the position and orientation, and calculates the geometric information or the position and orientation using the estimation result without the defect.
  • the measurement result processing unit not only makes a determination on the geometric information or the position and orientation, but when there is a defect, the cause of the defect is eliminated or the defect is avoided. To solve the problem. In this manner, even if there is a problem, the automatic driving or driving assistance is performed by robustly estimating the geometric information or the position and orientation.
  • the configuration of the automobile in the third embodiment is the same as that of FIG. 1 shown in the first embodiment, and thus the description thereof is omitted.
  • the calibration calculation unit 309 When the calibration calculation unit 309 receives an instruction to perform calibration from the measurement result processing unit 304, the calibration calculation unit 309 calibrates the first imaging device and the second imaging device to obtain calibration parameters. The calibration parameters of the calibration parameter holding unit (not shown) are updated according to the obtained calibration parameters.
  • Step S3090 is the same processing as step S1000, step S1010, step S1020, step S1030, step S1040, step S1060, step S1070, step S1080, and step S1090, respectively, so the description will be omitted, and a different step S3050, Step S3051 will be described.
  • the learning model holding unit 401 holds a model for estimating geometric information from an image. Further, the learning model holding unit 401 holds the learning model supplied from the learning unit 409 described later. Thereby, the geometric information estimation unit 402 can select an appropriate learning model according to the situation.
  • the measurement result processing unit 404 uses the learning model based on the first image and the second image, and geometric information corresponding to those images. Generate teacher data used when learning. Then, the measurement result processing unit 404 outputs the generated teacher data to the learning unit 409.
  • step S4150 learning unit 409 determines whether to start learning, and when learning is started, the process proceeds to step S4160. If the learning is not started, the process ends. In this case, when the processing for acquiring teacher data is started again, teacher data is additionally held. In addition, it is judged whether the amount of teacher data to be held is sufficient or not.
  • ⁇ Effect> As described above, according to the fourth embodiment, by generating a learning model, geometric information is generated using a generated learning model for a scene for which geometric information can not be estimated by the existing learning model. It is possible to estimate and estimate the stable position and orientation, and to perform automatic driving or driving assistance more safely. In addition, by automatically generating teacher data while the information processing apparatus is processing, it is necessary to separately create or label a large amount of teacher data necessary for learning a learning model. Can be omitted.
  • control unit 55 initializes the system. In addition to the process of step S1000 in the first embodiment, the control unit 55 also reads the activation of the projection unit 54 and the setting of the pattern projected by the projection control unit 509.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Optics & Photonics (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)
PCT/JP2018/041667 2018-01-15 2018-11-09 情報処理装置及びその制御方法及びコンピュータ可読記憶媒体、並びに、運転制御システム Ceased WO2019138677A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/921,389 US11393123B2 (en) 2018-01-15 2020-07-06 Information processing device, control method therefor, non-transitory computer-readable storage medium, and driving control system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018004469A JP7149707B2 (ja) 2018-01-15 2018-01-15 情報処理装置及びその制御方法及びプログラム、並びに、運転制御システム
JP2018-004469 2018-01-15

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/921,389 Continuation US11393123B2 (en) 2018-01-15 2020-07-06 Information processing device, control method therefor, non-transitory computer-readable storage medium, and driving control system

Publications (1)

Publication Number Publication Date
WO2019138677A1 true WO2019138677A1 (ja) 2019-07-18

Family

ID=67218581

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/041667 Ceased WO2019138677A1 (ja) 2018-01-15 2018-11-09 情報処理装置及びその制御方法及びコンピュータ可読記憶媒体、並びに、運転制御システム

Country Status (3)

Country Link
US (1) US11393123B2 (enExample)
JP (1) JP7149707B2 (enExample)
WO (1) WO2019138677A1 (enExample)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2023007559A1 (enExample) * 2021-07-26 2023-02-02
US20230110992A1 (en) * 2021-10-12 2023-04-13 Canon Kabushiki Kaisha Information processing apparatus, control method of information processing apparatus, and storage medium

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10816984B2 (en) 2018-04-13 2020-10-27 Baidu Usa Llc Automatic data labelling for autonomous driving vehicles
WO2019224947A1 (ja) * 2018-05-23 2019-11-28 株式会社ソニー・インタラクティブエンタテインメント 学習装置、画像生成装置、学習方法、画像生成方法及びプログラム
JPWO2020209144A1 (enExample) * 2019-04-09 2020-10-15
JP7249919B2 (ja) * 2019-09-17 2023-03-31 株式会社東芝 推定装置、推定方法及びプログラム
US12196545B2 (en) 2019-10-04 2025-01-14 Nec Corporation Surface property estimation system
US11127164B2 (en) * 2019-10-04 2021-09-21 Mitsubishi Electric Research Laboratories, Inc. Image processing system and method for landmark location estimation with uncertainty
US12181273B2 (en) * 2020-02-21 2024-12-31 Hamamatsu Photonics K.K. Three-dimensional measurement device
JP7498651B2 (ja) * 2020-02-21 2024-06-12 浜松ホトニクス株式会社 三次元計測装置
JP7298562B2 (ja) 2020-07-20 2023-06-27 トヨタ自動車株式会社 車両の周辺検知装置
JP7389729B2 (ja) * 2020-09-10 2023-11-30 株式会社日立製作所 障害物検知装置、障害物検知システム及び障害物検知方法
JP7127901B1 (ja) 2021-05-24 2022-08-30 Ｎｅｃプラットフォームズ株式会社 情報処理端末、制御方法、及びプログラム
CN113487684B (zh) * 2021-07-23 2024-08-06 浙江华睿科技股份有限公司 一种标定参数确定方法、装置、存储介质及电子装置
US20230363610A1 (en) * 2021-12-27 2023-11-16 Trifo, Inc. Occupancy map segmentation for autonomous guided platform with deep learning

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10341458A (ja) * 1997-06-10 1998-12-22 Toyota Motor Corp 車載ステレオカメラの校正方法、および、その方法を適用した車載ステレオカメラ
JP2012058188A (ja) * 2010-09-13 2012-03-22 Ricoh Co Ltd 校正装置、距離計測システム、校正方法および校正プログラム
JP2012134670A (ja) * 2010-12-20 2012-07-12 Nikon Corp 画像演算装置、撮像装置および画像演算プログラム
JP2015084229A (ja) * 2009-10-19 2015-04-30 メタイオ ゲゼルシャフト ミット ベシュレンクテル ハフツングｍｅｔａｉｏ ＧｍｂＨ カメラ姿勢決定方法、及び実環境物体認識方法
JP2015203680A (ja) * 2014-04-16 2015-11-16 キヤノン株式会社 情報処理装置、方法、プログラム
JP2017204145A (ja) * 2016-05-11 2017-11-16 株式会社豊田中央研究所 走行経路生成装置、モデル学習装置、及びプログラム
JP2017202828A (ja) * 2013-05-03 2017-11-16 グーグル インコーポレイテッド 車両の速度を制御するための予測推定

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4985166B2 (ja) 2007-07-12 2012-07-25 トヨタ自動車株式会社 自己位置推定装置
JP5496008B2 (ja) * 2010-08-06 2014-05-21 キヤノン株式会社 位置姿勢計測装置、位置姿勢計測方法、およびプログラム
JP5766620B2 (ja) 2012-01-10 2015-08-19 日本電信電話株式会社 物体領域検出装置、方法、及びプログラム
WO2016084926A1 (ja) * 2014-11-28 2016-06-02 日立マクセル株式会社 撮像システム
JP2017027520A (ja) * 2015-07-27 2017-02-02 キヤノン株式会社 ３ｄデータ生成装置及び方法、プログラム、記憶媒体
EP3470880B1 (en) 2016-06-08 2020-08-19 Panasonic Semiconductor Solutions Co., Ltd. Distance-measuring system and distance-measuring method
US9940729B1 (en) * 2016-11-18 2018-04-10 Here Global B.V. Detection of invariant features for localization
US10957068B2 (en) * 2017-01-06 2021-03-23 Canon Kabushiki Kaisha Information processing apparatus and method of controlling the same

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10341458A (ja) * 1997-06-10 1998-12-22 Toyota Motor Corp 車載ステレオカメラの校正方法、および、その方法を適用した車載ステレオカメラ
JP2015084229A (ja) * 2009-10-19 2015-04-30 メタイオ ゲゼルシャフト ミット ベシュレンクテル ハフツングｍｅｔａｉｏ ＧｍｂＨ カメラ姿勢決定方法、及び実環境物体認識方法
JP2012058188A (ja) * 2010-09-13 2012-03-22 Ricoh Co Ltd 校正装置、距離計測システム、校正方法および校正プログラム
JP2012134670A (ja) * 2010-12-20 2012-07-12 Nikon Corp 画像演算装置、撮像装置および画像演算プログラム
JP2017202828A (ja) * 2013-05-03 2017-11-16 グーグル インコーポレイテッド 車両の速度を制御するための予測推定
JP2015203680A (ja) * 2014-04-16 2015-11-16 キヤノン株式会社 情報処理装置、方法、プログラム
JP2017204145A (ja) * 2016-05-11 2017-11-16 株式会社豊田中央研究所 走行経路生成装置、モデル学習装置、及びプログラム

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2023007559A1 (enExample) * 2021-07-26 2023-02-02
WO2023007559A1 (ja) * 2021-07-26 2023-02-02 日本電信電話株式会社 点群データを生成する装置、方法及びプログラム
JP7670136B2 (ja) 2021-07-26 2025-04-30 日本電信電話株式会社 点群データを生成する装置、方法及びプログラム
US20230110992A1 (en) * 2021-10-12 2023-04-13 Canon Kabushiki Kaisha Information processing apparatus, control method of information processing apparatus, and storage medium
US12347133B2 (en) * 2021-10-12 2025-07-01 Canon Kabushiki Kaisha Information processing apparatus, control method of information processing apparatus, and storage medium

Also Published As

Publication number Publication date
JP2019125112A (ja) 2019-07-25
JP7149707B2 (ja) 2022-10-07
US11393123B2 (en) 2022-07-19
US20200334855A1 (en) 2020-10-22

Similar Documents

Publication Publication Date Title
WO2019138677A1 (ja) 情報処理装置及びその制御方法及びコンピュータ可読記憶媒体、並びに、運転制御システム
WO2019149061A1 (en) Gesture-and gaze-based visual data acquisition system
KR102023588B1 (ko) 로봇 파지용 심층 기계 학습 방법 및 장치
CN111683193B (zh) 图像处理装置
Rambach et al. Learning to fuse: A deep learning approach to visual-inertial camera pose estimation
CN113012219B (zh) 信息处理装置、信息处理方法及计算机可读介质
US9330472B2 (en) System and method for distorted camera image correction
TWI756844B (zh) 自走車導航裝置及其方法
KR20200091298A (ko) 핸드 아이 캘리브레이션 방법 및 시스템
CN113165178B (zh) 使用高级策略模型和经训练的低级策略模型进行机器人导航
JP7536442B2 (ja) 情報処理装置、情報処理方法およびプログラム
US20200159229A1 (en) System and method for creating a single perspective synthesized image
CN113847911A (zh) 信息处理装置、信息处理装置的控制方法和存储介质
JP6770488B2 (ja) 注視対象物推定装置、注視対象物推定方法、およびプログラム
JP7679208B2 (ja) 情報処理装置、情報処理方法およびプログラム
JP2021099384A (ja) 情報処理装置、情報処理方法およびプログラム
JP2025512595A (ja) トレーラベースの車両の運動および経路計画のためのシステムおよび方法
CN111971149A (zh) 记录介质、信息处理设备和信息处理方法
CN118276669B (zh) 确定相对位姿的方法和装置、扩展现实系统、设备和介质
JP7223587B2 (ja) 物体運動推定方法及び物体運動推定装置
CN114858158A (zh) 一种基于深度学习的移动机器人重定位方法
US12282608B2 (en) User interface, control method thereof, and vehicle having user interface
JP2024074625A (ja) 情報処理装置、制御方法、及びプログラム
CN116257134B (zh) 非惯性参考系下手柄头盔追踪方法、装置、设备及介质
JP2021109291A (ja) ロボット制御装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18900545

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18900545

Country of ref document: EP

Kind code of ref document: A1