WO2021156914A1 - 注意方向判定装置および注意方向判定方法 - Google Patents

注意方向判定装置および注意方向判定方法 Download PDF

Info

Publication number
WO2021156914A1
WO2021156914A1 PCT/JP2020/003925 JP2020003925W WO2021156914A1 WO 2021156914 A1 WO2021156914 A1 WO 2021156914A1 JP 2020003925 W JP2020003925 W JP 2020003925W WO 2021156914 A1 WO2021156914 A1 WO 2021156914A1
Authority
WO
WIPO (PCT)
Prior art keywords
occupant
unit
face orientation
attention
estimation unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/003925
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
太郎 熊谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to DE112020006670.3T priority Critical patent/DE112020006670B4/de
Priority to PCT/JP2020/003925 priority patent/WO2021156914A1/ja
Priority to JP2021575109A priority patent/JP7267467B2/ja
Publication of WO2021156914A1 publication Critical patent/WO2021156914A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris

Definitions

  • the skeleton estimation unit 5 estimates that the occupant's hand extends toward the adjacent seat. In the example shown in FIG. 8, when the occupant's hand is detected in the skeleton detection area B, the skeleton estimation unit 5 estimates that the occupant's hand extends toward the rear seat. In the example shown in FIG. 8, when the occupant's hand is detected in the skeleton detection area C, the skeleton estimation unit 5 estimates that the occupant's hand extends in the direction of the doorknob.
  • the face orientation detection unit 6 may be configured to detect the visual field or the line of sight of the occupant from the feature points estimated by the face feature point estimation unit 2. That is, the face orientation detection unit 6 calculates the occupant's face orientation angle and the occupant's field of view or line of sight based on the occupant's face feature points estimated by the face feature point estimation unit 2, and calculates the occupant's face orientation.
  • the configuration may be such that the face orientation of the occupant is detected based on the angle and the occupant's field of view or line of sight.
  • the face orientation detection unit 6 determines that the occupant is facing the adjacent seat direction and determines the occupant's line-of-sight direction angle. If the considered face-facing angle exceeds the predetermined angle ⁇ , it is determined that the occupant is facing the rear seat direction. Further, the face orientation detection unit 6 determines that the occupant is facing the door knob direction when the face orientation angle considering the occupant's line-of-sight direction angle is 0 ° to a predetermined angle ⁇ or less, and determines the occupant's line-of-sight direction angle. If the considered face-facing angle exceeds the predetermined angle ⁇ , it is determined that the occupant is facing the rear seat direction.
  • step S105 the skeleton estimation unit 5 estimates the skeleton of the occupant based on the relative position of the occupant's head estimated by the head position estimation unit 3 and the size of each part estimated by the site estimation unit 4. ..
  • step S106 the skeleton estimation unit 5 determines whether the estimated skeleton is detected in the skeleton detection area, and estimates the direction in which the occupant's hand is extended.
  • the direction in which the occupant's hand is extended is the door knob direction estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit.
  • the direction is the door knob direction, it is determined that the direction in which the occupant is paying attention is the door knob direction.
  • Embodiment 2 The configuration of the caution direction determination device 200 according to the second embodiment of the present disclosure will be described. The same or corresponding configurations as those in the first embodiment will be omitted, and only the parts having different configurations will be described.
  • the seating information acquisition unit 21 acquires the seating information of the occupant in the vehicle from the image of the occupant in the vehicle taken by the camera 9.
  • the seating information is information indicating which seat in the vehicle the occupant is seated in.
  • the acquisition of the seating information of the occupant in the vehicle by the seating information acquisition unit 21 is not limited to the configuration acquired from the image of the occupant in the vehicle taken by the camera 9.
  • the seating information acquisition unit 21 may be configured to acquire the seating information of the occupant in the vehicle from the seating sensor.
  • the attention direction determination unit 7 acquires the seating information, the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the seating information.
  • the direction in which the occupant is paying attention is determined based on at least one information of the seating information acquired by the unit 21 or the vehicle information acquired by the vehicle information acquisition unit 22.
  • the attention direction determination unit 7 determines the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the seating acquired by the seating information acquisition unit 21. An example of determination in the case of determining the direction in which the occupant is paying attention based on the information will be described.
  • the attention direction determination unit 7 determines the direction in which the occupant's hand is extended estimated by the skeleton estimation unit 5, the occupant's face orientation detected by the face orientation detection unit 6, and the vehicle acquired by the vehicle information acquisition unit 22. An example of determination in the case of determining the direction in which the occupant is paying attention based on the information will be described.
  • the face orientation detection unit 6 uses a predetermined angle value or a skeleton used when detecting the face orientation of the occupant.
  • the estimation unit 5 may be configured to change the size of each skeleton detection area used when estimating the extending direction of the occupant's hand. For example, when the vehicle information acquired by the vehicle information acquisition unit 22 indicates that the vehicle gear is parking or the ignition is off, the skeleton estimation unit 5 uses each skeleton detection area when the vehicle is running. It may be configured to be larger than.
  • the control unit 10 When the attention direction determination unit 7 does not use the seating information and the vehicle information when determining the direction in which the occupant is paying attention, the control unit 10 outputs the determination result output from the determination result output unit 8.
  • the vehicle-mounted device or the like may be controlled based on at least one piece of seating information acquired by the seating information acquisition unit 21 or vehicle information acquired by the vehicle information acquisition unit 22.
  • control unit 10 controls the in-vehicle device or the like based on the determination result output from the determination result output unit 8 and the seating information acquired by the seating information acquisition unit 21. An embodiment in the case of performing the above will be described.
  • the control unit 10 When the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the direction of the next seat, the control unit 10 acquires the next seat seating information from the seating information acquired by the seating information acquisition unit 21.
  • the control unit 10 When the seating information acquired by the control unit 10 is information indicating that the occupant is not seated in the seat next to the occupant to be detected, the control unit 10 is for turning on the lighting for illuminating the occupant next to the occupant to be detected. Take control.
  • the seating information acquired by the control unit 10 is information indicating that the occupant is seated in the seat next to the occupant to be detected, the control unit 10 turns on the light illuminating the occupant next to the occupant to be detected. No control for
  • the control unit 10 When the control unit 10 receives the determination result that the direction in which the occupant is paying attention is the rear seat direction, the control unit 10 acquires the rear seat seating information from the seating information acquired by the seating information acquisition unit 21. When the seating information acquired by the control unit 10 is information indicating that the occupant is not seated in the rear seat of the occupant to be detected, the control unit 10 turns on the light illuminating the rear seat of the occupant to be detected. Control for. On the other hand, when the seating information acquired by the control unit 10 is information indicating that the occupant is seated in the rear seat of the occupant to be detected, the control unit 10 illuminates the rear seat of the occupant to be detected. No control is performed to turn it on.
  • control unit 10 controls the in-vehicle device or the like based on the determination result output from the determination result output unit 8 and the vehicle information acquired by the vehicle information acquisition unit 22. An embodiment in the case of performing the above will be described.
  • the attention direction determination unit 7 does not use the seating information and the vehicle information when determining the direction in which the occupant is paying attention, and the attention direction determination unit 7 determines the direction in which the attention is directed.
  • the control unit 10 corresponds to the determination result when the vehicle information acquired by the vehicle information acquisition unit 22 indicates that the vehicle is stopped. It may be configured to control the equipment or the like. That is, the control unit 10 is a occupant whose attention direction is determined by the attention direction determination unit 7 in the driver's seat, and the vehicle information acquired by the vehicle information acquisition unit 22 is the vehicle. If it indicates that the vehicle is not stopped, the vehicle-mounted device or the like may not be controlled.
  • FIG. 13 is a flowchart showing an example of the operation of the caution direction determination device 200 shown in FIG. Note that steps S201 to S207 in FIG. 13 correspond to steps S101 to S107 in FIG. 11, and step S211 in FIG. 13 corresponds to step S109 in FIG. 11, so the description thereof is omitted here. Hereinafter, steps S208 to S210 will be described.
  • step S208 the seating information acquisition unit 21 acquires the seating information of the occupant in the vehicle.
  • step S209 the vehicle information acquisition unit 22 acquires vehicle information.
  • the attention direction determination device has a seating information acquisition unit that acquires seating information of the occupant in the vehicle, and the attention direction determination unit extends the occupant's hand estimated by the skeleton estimation unit.
  • the direction in which the occupant is paying attention is determined based on the direction, the occupant's face orientation detected by the face orientation detection unit, and the seating information acquired by the seating information acquisition unit.
  • the attention direction determination device has a vehicle information acquisition unit that acquires vehicle information of the vehicle, and the attention direction determination unit is a direction in which the occupant's hand is extended as estimated by the skeleton estimation unit.
  • the direction in which the occupant is paying attention is determined based on the occupant's face orientation detected by the face orientation detection unit and the vehicle information acquired by the vehicle information acquisition unit.
  • the direction in which the occupant's hand is extended is the direction of the adjacent seat estimated by the skeleton estimation unit, and the occupant's face detected by the face orientation detection unit.
  • the direction is the direction of the next seat and the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the next seat, it is determined that the direction in which the occupant is paying attention is the direction of the next seat.
  • the direction in which the occupant's hand is extended as estimated by the skeleton estimation unit is the rear seat direction
  • the occupant's face orientation detection unit detects it.
  • the face direction is the rear seat direction and the seating information acquired by the seating information acquisition unit indicates that the occupant is not seated in the rear seat
  • the direction in which the occupant is paying attention is the rear seat direction. judge.
  • the attention direction determination device 200 can determine the direction in which the occupant is paying attention. Further, the attention direction determination device 200 according to the second embodiment determines the direction in which the occupants are paying attention in consideration of the information regarding a plurality of occupants by acquiring the seating information, so that the attention direction is erroneously determined. Can be reduced. Further, since the attention direction determination device 200 according to the second embodiment determines the direction in which the occupant is paying attention in consideration of the state of the vehicle by acquiring the vehicle information, the erroneous determination of the attention direction is reduced. can do.
  • Each function of the determination result output unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22 is realized by the processing circuit. That is, the attention direction determination devices 100 and 200 acquire the image of the occupant in the vehicle taken by the camera, estimate the feature points of the occupant's face, estimate the relative position of the occupant's head with respect to the camera, and image.
  • the processing circuit 31 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC (Application Specific Integrated Circuit). , FPGA (Field Processor Gate Array), or a combination thereof.
  • Each function of 21 and the vehicle information acquisition unit 22 may be realized by the processing circuit 31, or each function may be collectively realized by one processing circuit 31.
  • the processing circuit 31 is the processor 32 shown in FIG. 15, the image acquisition unit 1, the face feature point estimation unit 2, the head position estimation unit 3, the site estimation unit 4, the skeleton estimation unit 5, the face orientation detection unit 6, and the attention direction.
  • Each function of the determination unit 7, the determination result output unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22 is realized by software, firmware, or a combination of software and firmware.
  • the software or firmware is written as a program and stored in the memory 33.
  • the processor 32 realizes each function by reading and executing the program recorded in the memory 33. That is, the attention direction determination devices 100 and 200 are programs in which each step from step S101 to step S109 shown in FIG. 11 or each step from step S201 to step S211 shown in FIG. 13 is executed as a result.
  • a memory 33 for storing the above is provided. Further, these programs include an image acquisition unit 1, a face feature point estimation unit 2, a head position estimation unit 3, a site estimation unit 4, a skeleton estimation unit 5, a face orientation detection unit 6, a attention direction determination unit 7, and a determination result output. It can also be said that the computer is made to execute the procedure or method of the unit 8, the seating information acquisition unit 21, and the vehicle information acquisition unit 22.
  • the memory is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EPROM (Electrically Memory) It may be a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a DVD (Digital Versaille Disc), or any other storage medium that will be used in the future.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • flash memory an EPROM (Erasable Programmable Read Only Memory)
  • EPROM Electrically Memory
  • It may be a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a DVD (Digital Versaille Disc), or any other storage medium that will be used in the future.
  • some functions may be realized by dedicated hardware, and other functions may be realized by software or firmware.
  • the processing circuit can realize each of the above-mentioned functions by hardware, software, firmware, or a combination thereof.
  • the caution direction determination device described above is an appropriate combination of not only an in-vehicle navigation device, that is, a car navigation device, but also a PND (Portable Navigation Device) that can be mounted on the vehicle, a server provided outside the vehicle, and the like. It can also be applied to a navigation device constructed as a system or a device other than the navigation device. Even if each function of the attention direction determination device is distributed and arranged in each function for constructing the system, the same effect as that of the above-described embodiment can be obtained.
  • each embodiment can be freely combined, and each embodiment can be appropriately modified or omitted within the scope of the disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
PCT/JP2020/003925 2020-02-03 2020-02-03 注意方向判定装置および注意方向判定方法 Ceased WO2021156914A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
DE112020006670.3T DE112020006670B4 (de) 2020-02-03 2020-02-03 Aufmerksamkeitsrichtung-Beurteilungsvorrichtung und Aufmerksamkeitsrichtung-Beurteilungsverfahren
PCT/JP2020/003925 WO2021156914A1 (ja) 2020-02-03 2020-02-03 注意方向判定装置および注意方向判定方法
JP2021575109A JP7267467B2 (ja) 2020-02-03 2020-02-03 注意方向判定装置および注意方向判定方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/003925 WO2021156914A1 (ja) 2020-02-03 2020-02-03 注意方向判定装置および注意方向判定方法

Publications (1)

Publication Number Publication Date
WO2021156914A1 true WO2021156914A1 (ja) 2021-08-12

Family

ID=77199953

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/003925 Ceased WO2021156914A1 (ja) 2020-02-03 2020-02-03 注意方向判定装置および注意方向判定方法

Country Status (3)

Country Link
JP (1) JP7267467B2 (https=)
DE (1) DE112020006670B4 (https=)
WO (1) WO2021156914A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210300274A1 (en) * 2020-03-30 2021-09-30 Toyota Jidosha Kabushiki Kaisha Occupant position detection system
JP2023127912A (ja) * 2022-03-02 2023-09-14 株式会社デンソーアイティーラボラトリ 降車行動検知装置、降車行動検知方法およびプログラム
JPWO2023242886A1 (https=) * 2022-06-13 2023-12-21

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014522058A (ja) * 2012-06-14 2014-08-28 ソフトキネティック ソフトウェア 三次元オブジェクトのモデリング、フィッティング、およびトラッキング
JP2018092489A (ja) * 2016-12-06 2018-06-14 オムロン株式会社 分類装置、分類方法およびプログラム
JP2018528536A (ja) * 2015-08-31 2018-09-27 エスアールアイ インターナショナルＳＲＩ Ｉｎｔｅｒｎａｔｉｏｎａｌ 運転行動を監視する方法及びシステム
JP2018150694A (ja) * 2017-03-10 2018-09-27 株式会社オートネットワーク技術研究所 駆動制御装置及びコンピュータプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5233977B2 (ja) * 2009-12-11 2013-07-10 株式会社デンソー 乗員姿勢推定装置
DE102012020713A1 (de) * 2012-10-23 2014-04-24 Volkswagen Aktiengesellschaft Verfahren zur Beurteilung der Ablenkung eines Fahrers eines Fahrzeugs

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014522058A (ja) * 2012-06-14 2014-08-28 ソフトキネティック ソフトウェア 三次元オブジェクトのモデリング、フィッティング、およびトラッキング
JP2018528536A (ja) * 2015-08-31 2018-09-27 エスアールアイ インターナショナルＳＲＩ Ｉｎｔｅｒｎａｔｉｏｎａｌ 運転行動を監視する方法及びシステム
JP2018092489A (ja) * 2016-12-06 2018-06-14 オムロン株式会社 分類装置、分類方法およびプログラム
JP2018150694A (ja) * 2017-03-10 2018-09-27 株式会社オートネットワーク技術研究所 駆動制御装置及びコンピュータプログラム

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210300274A1 (en) * 2020-03-30 2021-09-30 Toyota Jidosha Kabushiki Kaisha Occupant position detection system
US11628789B2 (en) * 2020-03-30 2023-04-18 Toyota Jidosha Kabushiki Kaisha Occupant position detection system
JP2023127912A (ja) * 2022-03-02 2023-09-14 株式会社デンソーアイティーラボラトリ 降車行動検知装置、降車行動検知方法およびプログラム
JP7715060B2 (ja) 2022-03-02 2025-07-30 株式会社デンソー 降車行動検知装置、降車行動検知方法およびプログラム
JPWO2023242886A1 (https=) * 2022-06-13 2023-12-21
WO2023242886A1 (ja) * 2022-06-13 2023-12-21 三菱電機株式会社 顔向き推定装置、脇見判断システム、及び顔向き推定方法
JP7523726B2 (ja) 2022-06-13 2024-07-26 三菱電機株式会社 顔向き推定装置、脇見判断システム、及び顔向き推定方法

Also Published As

Publication number Publication date
JPWO2021156914A1 (https=) 2021-08-12
DE112020006670T5 (de) 2022-11-24
JP7267467B2 (ja) 2023-05-01
DE112020006670B4 (de) 2025-12-04

Similar Documents

Publication Publication Date Title
JP7369184B2 (ja) 運転者注意状態推定
US7771049B2 (en) Method and apparatus for detecting sight line vector
JP5406328B2 (ja) 車両用表示装置、その制御方法及びプログラム
US9785235B2 (en) Display control apparatus, display control method of display control apparatus, and eye gaze direction detection system
JP5184596B2 (ja) 脇見判定装置
JP7031072B2 (ja) 認知機能推定装置、学習装置、および、認知機能推定方法
JP2019034574A (ja) 運転者状態把握装置、運転者状態把握システム、及び運転者状態把握方法
JP6479272B1 (ja) 視線方向較正装置、視線方向較正方法および視線方向較正プログラム
WO2021156914A1 (ja) 注意方向判定装置および注意方向判定方法
CN109760689A (zh) 驾驶员监视系统
JP2010033106A (ja) 運転者支援装置、運転者支援方法および運転者支援処理プログラム
CN106406515A (zh) 推测车辆乘客视线的方法和机构、确定对车辆乘客特定的头部运动放大参数的方法和机构及对车辆乘客视线推测的方法和装置
JP4319535B2 (ja) 顔の向き検知装置
CN110310321A (zh) 视线方向估算装置，视线方向估算方法和视线方向估算程序
CN114391129B (zh) 凝视方向的确定
JP6572538B2 (ja) 下方視判定装置および下方視判定方法
JP2016115117A (ja) 判定装置および判定方法
JP7134364B2 (ja) 体格判定装置および体格判定方法
CN113316805A (zh) 使用红外线和可见光监视人的方法和系统
JP7003335B2 (ja) 操作者判定装置および操作者判定方法
KR20150067679A (ko) 차량용 제스처 인식 시스템 및 그 방법
CN120220121A (zh) 车辆儿童模式检测方法、装置、设备、存储介质及计算机程序产品
JP7683546B2 (ja) シートポジション推定装置、シートポジション推定方法及びシートポジション推定用コンピュータプログラム
KR102486161B1 (ko) 차량, 차량의 제어 방법 및 영상 탐색 장치
JP7483160B2 (ja) 乗員監視装置及び乗員監視方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20917350

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021575109

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 112020006670

Country of ref document: DE

WWP Wipo information: published in national office

Ref document number: 112020006670

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20917350

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 112020006670

Country of ref document: DE