CN111989537B - 用于在无约束环境中检测人类视线和手势的系统和方法 - Google Patents

用于在无约束环境中检测人类视线和手势的系统和方法 Download PDF

Info

Publication number
CN111989537B
CN111989537B CN201980026218.5A CN201980026218A CN111989537B CN 111989537 B CN111989537 B CN 111989537B CN 201980026218 A CN201980026218 A CN 201980026218A CN 111989537 B CN111989537 B CN 111989537B
Authority
CN
China
Prior art keywords
interest
individual
environment
objects
map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201980026218.5A
Other languages
English (en)
Chinese (zh)
Other versions
CN111989537A (zh
Inventor
S·A·I·斯滕特
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Toyota Motor Corp
Original Assignee
Toyota Research Institute Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toyota Research Institute Inc filed Critical Toyota Research Institute Inc
Publication of CN111989537A publication Critical patent/CN111989537A/zh
Application granted granted Critical
Publication of CN111989537B publication Critical patent/CN111989537B/zh
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
    • G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0268—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means
    • G05D1/0274—Control of position or course in two dimensions specially adapted to land vehicles using internal positioning means using mapping information stored in a memory device
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/19—Sensors therefor
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0381—Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Remote Sensing (AREA)
  • Automation & Control Theory (AREA)
  • Ophthalmology & Optometry (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Electromagnetism (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)
  • Image Processing (AREA)
CN201980026218.5A 2018-04-17 2019-04-11 用于在无约束环境中检测人类视线和手势的系统和方法 Expired - Fee Related CN111989537B (zh)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/955,333 US11126257B2 (en) 2018-04-17 2018-04-17 System and method for detecting human gaze and gesture in unconstrained environments
US15/955,333 2018-04-17
PCT/US2019/026999 WO2019204118A1 (en) 2018-04-17 2019-04-11 System and method for detecting human gaze and gesture in unconstrained environments

Publications (2)

Publication Number Publication Date
CN111989537A CN111989537A (zh) 2020-11-24
CN111989537B true CN111989537B (zh) 2023-01-06

Family

ID=68160332

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201980026218.5A Expired - Fee Related CN111989537B (zh) 2018-04-17 2019-04-11 用于在无约束环境中检测人类视线和手势的系统和方法

Country Status (5)

Country Link
US (1) US11126257B2 (enExample)
EP (1) EP3781896B1 (enExample)
JP (2) JP7675970B2 (enExample)
CN (1) CN111989537B (enExample)
WO (1) WO2019204118A1 (enExample)

Families Citing this family (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3403146A4 (en) * 2016-01-15 2019-08-21 iRobot Corporation AUTONOMOUS MONITORING ROBOT SYSTEMS
DK180639B1 (en) 2018-06-01 2021-11-04 Apple Inc DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT
US10909374B2 (en) * 2018-09-07 2021-02-02 Intel Corporation Technologies for identifying unrecognizable objects in autonomous systems
JP7222216B2 (ja) * 2018-10-29 2023-02-15 株式会社アイシン 運転支援装置
US10833945B2 (en) * 2018-11-13 2020-11-10 International Business Machines Corporation Managing downloading of content
US11221671B2 (en) * 2019-01-31 2022-01-11 Toyota Research Institute, Inc. Opengaze: gaze-tracking in the wild
US10983591B1 (en) * 2019-02-25 2021-04-20 Facebook Technologies, Llc Eye rank
WO2020184733A1 (ko) * 2019-03-08 2020-09-17 엘지전자 주식회사 로봇
US11435820B1 (en) * 2019-05-16 2022-09-06 Facebook Technologies, Llc Gaze detection pipeline in an artificial reality system
US11430414B2 (en) * 2019-10-17 2022-08-30 Microsoft Technology Licensing, Llc Eye gaze control of magnification user interface
CN118885066A (zh) 2019-11-08 2024-11-01 苹果公司 使用多个传感器的基于机器学习的手势识别
US11449802B2 (en) * 2019-11-08 2022-09-20 Apple Inc. Machine-learning based gesture recognition using multiple sensors
CN111105347B (zh) * 2019-11-19 2020-11-13 贝壳找房(北京)科技有限公司 一种生成带深度信息的全景图的方法、装置及存储介质
KR20210067539A (ko) * 2019-11-29 2021-06-08 엘지전자 주식회사 정보 처리 방법 및 정보 처리 장치
KR20220021581A (ko) * 2020-08-14 2022-02-22 삼성전자주식회사 로봇 및 이의 제어 방법
CN112163990B (zh) 2020-09-08 2022-10-25 上海交通大学 360度图像的显著性预测方法及系统
US11999356B2 (en) * 2020-11-13 2024-06-04 Toyota Research Institute, Inc. Cognitive heat map: a model for driver situational awareness
CN112363626B (zh) * 2020-11-25 2021-10-01 广东魅视科技股份有限公司 基于人体姿态和手势姿态视觉识别的大屏幕交互控制方法
US11097414B1 (en) * 2020-12-22 2021-08-24 X Development Llc Monitoring of surface touch points for precision cleaning
KR102857620B1 (ko) * 2021-01-26 2025-09-09 삼성전자주식회사 전자 장치 및 이의 제어 방법
US12093461B2 (en) 2021-02-12 2024-09-17 Apple Inc. Measurement based on point selection
US11869144B1 (en) * 2021-03-03 2024-01-09 Apple Inc. Modeling a physical environment based on saliency
JP7697234B2 (ja) * 2021-03-17 2025-06-24 株式会社リコー 画像処理方法、撮像制御方法、プログラム、画像処理装置および撮像装置
JP7585946B2 (ja) * 2021-04-14 2024-11-19 トヨタ自動車株式会社 遠隔ロボットシステム及び遠隔ロボットシステムの制御方法
US11687155B2 (en) * 2021-05-13 2023-06-27 Toyota Research Institute, Inc. Method for vehicle eye tracking system
CN115695906B (zh) * 2021-07-27 2025-12-09 博泰车联网(南京)有限公司 基于车外景象的视频生成方法、系统、设备及介质
US11887405B2 (en) * 2021-08-10 2024-01-30 Capital One Services, Llc Determining features based on gestures and scale
US12526598B2 (en) * 2021-09-14 2026-01-13 Intel Corporation Methods and apparatus to generate spatial audio based on computer vision
US12457302B2 (en) 2021-09-23 2025-10-28 Intel Corporation Apparatus, systems, and methods for audio and video filtering for electronic user devices
CN114063856A (zh) * 2021-11-17 2022-02-18 塔米智能科技（北京）有限公司 一种身份注册方法、装置、设备和介质
US20230360079A1 (en) 2022-01-18 2023-11-09 e-con Systems India Private Limited Gaze estimation system and method thereof
EP4466594A4 (en) * 2022-02-21 2026-01-21 Iitm Indian Institute Of Tech Madras Iit Madras MULTI-USER EYE METRY DEVICE AND METHOD
EP4276586A1 (en) * 2022-05-12 2023-11-15 Koninklijke Philips N.V. Providing information about an area of interest
US12646510B2 (en) * 2022-05-13 2026-06-02 Apple Inc. Determining whether speech input is intended for a digital assistant
JP7858803B2 (ja) * 2022-09-27 2026-05-14 本田技研工業株式会社 情報処理装置、情報処理方法、およびプログラム
US12243258B1 (en) * 2023-09-11 2025-03-04 Elm Company Gaze target detection method and system
US20250303574A1 (en) * 2024-03-27 2025-10-02 Blue Hill Tech, Inc. Multimodal robot-human interaction via text, voice, and video for robot controls
EP4749401A1 (en) * 2024-11-20 2026-05-27 VoxelSensors SRL System and method for visual awareness
WO2025261994A1 (en) * 2024-06-17 2025-12-26 Voxelsensors Srl System and method for visual awareness
KR102969504B1 (ko) * 2024-10-30 2026-05-27 세종대학교산학협력단 제스쳐를 이용한 모바일 로봇 제어 방법 및 제스쳐를 인식하는 모바일 로봇
CN121424417B (zh) * 2025-12-31 2026-04-07 华南理工大学 基于条件扩散的手势理解与机器人动作生成方法及系统

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4220595B2 (ja) * 1998-08-10 2009-02-04 株式会社日立製作所 欠陥の分類方法並びに教示用データ作成方法
JP3529049B2 (ja) * 2002-03-06 2004-05-24 ソニー株式会社 学習装置及び学習方法並びにロボット装置
CN1867881B (zh) * 2003-09-12 2010-08-18 平蛙实验室股份公司 确定辐射散射/反射件的位置的系统和方法
GB2415562B (en) * 2004-06-23 2007-11-21 Hewlett Packard Development Co Image processing
US8340349B2 (en) * 2006-06-20 2012-12-25 Sri International Moving target detection in the presence of parallax
JP2010268158A (ja) * 2009-05-13 2010-11-25 Fujifilm Corp 画像処理システム、画像処理方法およびプログラム
US8803880B2 (en) * 2009-08-21 2014-08-12 Peking University Image-based lighting simulation for objects
WO2012083989A1 (en) * 2010-12-22 2012-06-28 Sony Ericsson Mobile Communications Ab Method of controlling audio recording and electronic device
WO2012107892A2 (en) 2011-02-09 2012-08-16 Primesense Ltd. Gaze detection in a 3d mapping environment
US9024844B2 (en) * 2012-01-25 2015-05-05 Microsoft Technology Licensing, Llc Recognition of image on external display
JP2013250882A (ja) * 2012-06-01 2013-12-12 Sharp Corp 注目位置検出装置、注目位置検出方法、及び注目位置検出プログラム
US9275460B2 (en) * 2012-10-17 2016-03-01 Google Inc. Reference orientations for viewing panoramic images
JP6171353B2 (ja) * 2013-01-18 2017-08-02 株式会社リコー 情報処理装置、システム、情報処理方法およびプログラム
US9122916B2 (en) * 2013-03-14 2015-09-01 Honda Motor Co., Ltd. Three dimensional fingertip tracking
US9514574B2 (en) * 2013-08-30 2016-12-06 Qualcomm Incorporated System and method for determining the extent of a plane in an augmented reality environment
US10165176B2 (en) * 2013-10-31 2018-12-25 The University Of North Carolina At Chapel Hill Methods, systems, and computer readable media for leveraging user gaze in user monitoring subregion selection systems
WO2015072166A1 (ja) * 2013-11-18 2015-05-21 オリンパスイメージング株式会社 撮像装置、撮像アシスト方法及び撮像アシストプログラムを記録した記録媒体
JP2015116319A (ja) * 2013-12-18 2015-06-25 パナソニックＩｐマネジメント株式会社 診断支援装置、診断支援方法、および診断支援プログラム
JP6448767B2 (ja) * 2014-04-24 2019-01-09 ナント・ホールデイングス・アイ・ピー・エル・エル・シー 画像物体認識におけるロバスト特徴特定
US10416760B2 (en) * 2014-07-25 2019-09-17 Microsoft Technology Licensing, Llc Gaze-based object placement within a virtual reality environment
WO2016095057A1 (en) 2014-12-19 2016-06-23 Sulon Technologies Inc. Peripheral tracking for an augmented reality head mounted device
US10419655B2 (en) 2015-04-27 2019-09-17 Snap-Aid Patents Ltd. Estimating and using relative head pose and camera field-of-view
US10291845B2 (en) * 2015-08-17 2019-05-14 Nokia Technologies Oy Method, apparatus, and computer program product for personalized depth of field omnidirectional video
US10027888B1 (en) * 2015-09-28 2018-07-17 Amazon Technologies, Inc. Determining area of interest in a panoramic video or photo
EP3391339A2 (en) * 2015-12-18 2018-10-24 Iris Automation, Inc. Real-time visual situational awareness system
CA3014670A1 (en) * 2016-03-08 2017-09-14 Nant Holdings Ip, Llc Image feature combination for image-based object recognition
US20170263017A1 (en) 2016-03-11 2017-09-14 Quan Wang System and method for tracking gaze position
JP6629678B2 (ja) * 2016-06-16 2020-01-15 株式会社日立製作所 機械学習装置
US10423914B2 (en) * 2016-07-08 2019-09-24 International Business Machines Corporation Industrial setup composition
US10142686B2 (en) * 2017-03-30 2018-11-27 Rovi Guides, Inc. System and methods for disambiguating an ambiguous entity in a search query based on the gaze of a user

Also Published As

Publication number Publication date
CN111989537A (zh) 2020-11-24
US20190317594A1 (en) 2019-10-17
EP3781896A1 (en) 2021-02-24
WO2019204118A1 (en) 2019-10-24
US11126257B2 (en) 2021-09-21
JP2021522564A (ja) 2021-08-30
EP3781896A4 (en) 2022-01-26
JP7675970B2 (ja) 2025-05-14
JP2024045273A (ja) 2024-04-02
EP3781896B1 (en) 2024-02-14

Similar Documents

Publication Publication Date Title
JP7675970B2 (ja) 非制約環境において人間の視線及びジェスチャを検出するシステムと方法
US12314478B2 (en) Systems and methods of tracking moving hands and recognizing gestural interactions
US20240419254A1 (en) Three Dimensional (3D) Modeling of a Complex Control Object
KR102255273B1 (ko) 청소 공간의 지도 데이터를 생성하는 장치 및 방법
US10762386B2 (en) Method of determining a similarity transformation between first and second coordinates of 3D features
JP6469706B2 (ja) 深度センサを用いた構造のモデル化
WO2019179442A1 (zh) 智能设备的交互目标确定方法和装置
CN104956292A (zh) 多个感知感测输入的交互
CN111163906A (zh) 能够移动的电子设备及其操作方法
WO2019214442A1 (zh) 一种设备控制方法、装置、控制设备及存储介质
WO2021248857A1 (zh) 一种障碍物属性判别方法、系统及智能机器人
JP2016139396A (ja) ユーザーインターフェイス装置、方法およびプログラム
EP3115926A1 (en) Method for control using recognition of two-hand gestures
US9898183B1 (en) Motions for object rendering and selection
CN112711324B (zh) 基于tof相机的手势交互方法及其系统
KR20260052112A (ko) 멀티-카메라 디바이스에서의 객체 깊이 추정을 위한 경계 상자 변환
KR20240057297A (ko) 신경망 모델을 학습시키는 방법 및 전자 장치
송준봉 CEE: Command Everything with Eyes, Multi-modal gaze-based interface for everyday Interaction

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20230504

Address after: Aichi Prefecture, Japan

Patentee after: Toyota Motor Corp.

Address before: California, USA

Patentee before: Toyota Research Institute, Inc.

TR01 Transfer of patent right
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20230106

CF01 Termination of patent right due to non-payment of annual fee