WO2024202037A1 - ドライバ監視装置及びプログラムと媒体 - Google Patents

ドライバ監視装置及びプログラムと媒体 Download PDF

Info

Publication number
WO2024202037A1
WO2024202037A1 PCT/JP2023/013629 JP2023013629W WO2024202037A1 WO 2024202037 A1 WO2024202037 A1 WO 2024202037A1 JP 2023013629 W JP2023013629 W JP 2023013629W WO 2024202037 A1 WO2024202037 A1 WO 2024202037A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
monitoring device
vehicle
driver monitoring
line
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/013629
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
スリニヴァサ ディヴァカル バット
プラティック ガルグ
輝 金原
成光 土屋
龍司 岩下
ムディット ソニ
スルビ アガルバル
タルン サティシュ
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Honda Motor Co Ltd
Original Assignee
Honda Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honda Motor Co Ltd filed Critical Honda Motor Co Ltd
Priority to JP2025509619A priority Critical patent/JPWO2024202037A1/ja
Priority to CN202380095710.4A priority patent/CN120917502A/zh
Priority to PCT/JP2023/013629 priority patent/WO2024202037A1/ja
Publication of WO2024202037A1 publication Critical patent/WO2024202037A1/ja
Priority to US19/341,028 priority patent/US20260024358A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60Q—ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
    • B60Q9/00—Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00, e.g. haptic signalling
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02—Alarms for ensuring the safety of persons
    • G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30232—Surveillance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/07—Target detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/08—Detecting or categorising vehicles

Definitions

  • the present invention relates to a driver monitoring device, a program, and a medium for monitoring a driver who drives a vehicle, for example. Regarding the device.
  • Patent Document 1 describes a driving situation monitoring device that receives driving situation data including an image of the driver for authentication taken by the drive recorder, vehicle information, position information, and acceleration information, and links the driver's identification information to the driving situation data and records them.
  • Patent document 1 also describes that the driving status monitoring device generates dangerous driving data in response to the occurrence of an event and transmits warning information to the drive recorder.
  • the driving status monitoring device described in Patent Document 1 monitors the vehicle's status, such as its speed, acceleration, and position. Although it photographs the driver, the image is only used to identify the driver and is not used to monitor the driver's condition.
  • the present invention was made in consideration of the above-mentioned conventional examples, and aims to contribute to safer vehicle operation by monitoring the driver's condition.
  • a driver monitoring device that can be retrofitted to a vehicle, comprising: An image capturing means for capturing an image including a driver of the vehicle; A control means; output means; The control means Identifying the driver's line of sight from the image captured by the imaging means; determining whether the identified line of sight direction is continuously directed below a preset virtual line for more than a predetermined time period; A driver monitoring device is provided, characterized in that when it is determined that the line of sight is continuously directed below the virtual line for more than the specified time, the output means outputs a warning to the driver.
  • the present invention contributes to safer vehicle operation by monitoring the driver's condition.
  • FIG. 1 is a diagram showing a vehicle according to an embodiment
  • 1 is a block diagram of a drive recorder according to an embodiment
  • FIG. 2 is a diagram illustrating a driver's line of sight vector and a virtual line.
  • 11 is a flowchart of a virtual line setting process according to the first embodiment.
  • 4 is a flowchart of a driver monitoring process according to the first embodiment.
  • 13 is a flowchart of a virtual line setting process according to the second embodiment.
  • 13 is a flowchart (part) of a driver monitoring process according to a third embodiment.
  • 13 is a flowchart (part) of a driver monitoring process according to a fourth embodiment. 13 is a flowchart (part) of a driver monitoring process according to the fifth embodiment. 13 is a flowchart (part) of a driver monitoring process according to the fifth embodiment.
  • This embodiment relates to an information processing device for detecting the amount of movement of a moving body to which it is attached.
  • the driver monitoring device will be described as an example in which it is a drive recorder attached to a moving body such as a vehicle.
  • the driver monitoring device may be any other device that is attached to a moving body and can capture images of the inside and outside of the moving body.
  • the driver monitoring device will be described as an example in which the moving body is, for example, a vehicle.
  • the vehicle is typically a four-wheeled vehicle, but the device may be applied to other types of vehicles as long as the vehicle is driven and operated by an occupant.
  • a drive recorder 101 is mounted on a vehicle 100.
  • the drive recorder 101 is mounted near the rearview mirror or on the rearview mirror of the vehicle 100 at a position on the windshield of the vehicle 100 that does not obstruct the driver's field of vision.
  • the mounting position of the drive recorder 101 is not limited to this, and may be any position that can capture an image of the outside world of the vehicle 100.
  • the drive recorder 101 can operate independently, but may also receive power from the vehicle 100.
  • the drive recorder 101 is equipped with an exterior camera 101F that captures the view outside the vehicle, particularly the view in front of the vehicle, and an interior camera 101R that captures the view inside the vehicle, particularly the driver.
  • the interior camera 101R has a viewing angle that is sufficient to capture an image of the face of the driver seated in the driver's seat 102.
  • a virtual line 103 indicating the position of the bottom edge of the windshield is set on the drive recorder 101 (shown by a dotted line in the figure).
  • the line where the dashboard 104 located near the front end of the passenger compartment of the vehicle 100 meets the windshield may be curved, but in this embodiment, the virtual line 103 is a straight line to simplify processing and settings. Of course, it may have a shape that matches the actual bottom edge of the windshield.
  • the virtual line 103 is a line that serves as a reference for determining that the driver is looking outside the vehicle (particularly forward) if he or she is looking above it, if the bottom edge of the windshield is curved as in Figure 1, it may be a straight line extending in the width direction of the vehicle from the point on the curve that is closest to the front of the vehicle.
  • ⁇ Configuration example of the drive radar Figure 2 shows an example of the control configuration of the drive recorder 101 of the embodiment.
  • This drive recorder is retrofitted to the vehicle, and does not require an input signal from the outside.
  • the power source of the drive recorder 101 depends on the vehicle 100, and the user interface may use a device mounted on the vehicle 100 or another device.
  • the drive recorder 101 may have an input/output device for providing a user interface. Since the driver is monitored by the drive recorder in this embodiment, the drive recorder 101 may be called a driver monitoring device, focusing on this point. Alternatively, it may be called a driving support device.
  • the drive recorder 101 includes a control unit 200, an in-vehicle camera 101R, and an outside-vehicle camera 101F. These cameras are sometimes called a shooting unit or an image acquisition unit.
  • Camera interfaces (IF) 201 and 202 are signal interfaces with the in-vehicle camera 101R and the outside-vehicle camera 101F, respectively. Each camera shoots video (video) at a predetermined frame rate, for example, 29 fps (frames per second).
  • the captured images are processed by an image processing unit 203, and further necessary processing is executed by a CPU 204, and the video is saved as a video file in an erasable ROM 209.
  • the video file includes video of a predetermined length of time, for example, about 30 to 60 seconds, and when the predetermined length of recording is completed, a new video file is recorded.
  • the captured image can be referenced frame by frame, and in this example, a frame is sometimes referred to as an image.
  • the processes that the image processing unit 203 can perform may include, for example, a process of converting a captured image of an object into an image from which the object is viewed at a different angle, such as projective transformation.
  • the CPU 204 also executes a program with a procedure that will be described later, and has a built-in timer (not shown).
  • the drive recorder also includes an acceleration sensor, and when acceleration exceeding a predetermined value is detected, control is performed such as stopping the recording of the video file.
  • the explanation focuses on the driving assistance function of this drive recorder, so the explanation of the function and device as a drive recorder is omitted.
  • the communication unit 208 provides wired or wireless communication functions.
  • the communication unit 208 may be connected to a smartphone or the like and the warning may be output thereto.
  • communication for other purposes is also possible.
  • the input unit 206 and the output unit 207 may include an input/output unit as a user interface, and the input/output unit may be realized, for example, by a touch panel constituting an operation unit.
  • an interface may be provided for connecting to a display audio device provided in the vehicle 100.
  • an interface may be provided for connecting to a mobile terminal.
  • the RAM 205 is used as a data memory required for the operation of the CPU 204 and the image processing unit 203, and as a memory for storing programs executed by the CPU 204.
  • the power supply unit 210 is connected to a power source provided by the vehicle 100, and provides power supply suitable for the drive recorder.
  • FIG. 3 is a diagram for explaining the line of sight vector and virtual line of the driver who is the subject of monitoring in this embodiment and other embodiments. Explanation of the symbols explained in Figure 1 will be omitted here.
  • the driver 301 seated in the driver's seat 102 looks at the surroundings, mainly the front, through the windshield while driving forward. However, the driver does not usually look below, that is, at the lower part of the lower end of the windshield, while driving, except for a very brief moment when he or she looks at meters.
  • the drive recorder 101 of this embodiment captures the face of the driver 301 with the in-vehicle camera 101R and identifies a gaze vector 302 that indicates the direction in which the driver is looking.
  • the gaze vector 302 is a vector that starts from either the left or right eye of the driver and indicates the gaze direction. Its magnitude may be an appropriate predetermined value. It may be determined in advance which eye is the starting point, or if only one eye can be recognized, the recognized eye may be the starting point.
  • extension line of the gaze vector 302 (shown by a dotted line in the figure) is above the virtual line 103 that indicates the bottom edge of the front window, the driver is looking outside, but if it is below the virtual line 103, it can be estimated that the driver is not looking outside and may be looking away.
  • a three-dimensional coordinate system is set in the drive recorder 101.
  • This is referred to here as the camera coordinate or camera coordinate system.
  • the origin of the camera coordinate system is the installation position of the drive recorder 101
  • the X axis is set along the optical axis of the in-vehicle camera 101R
  • the Y axis is directed downward toward the vehicle
  • the Z axis is set in the vehicle width direction.
  • This coordinate system makes it possible to identify the position of the driver 301 and the position of the virtual line 103.
  • the drive recorder 101 is installed so that the optical axis of the outside-vehicle camera 101F is parallel to the surface (ground) on which the vehicle 100 runs and is aligned with the straight direction of the vehicle 101. Also, it is assumed that the optical axis of the inside-vehicle camera 101R is coaxial with or parallel to the optical axis of the outside-vehicle camera 101F.
  • the mounting position of the drive recorder 101 relative to the vehicle is set when it is mounted.
  • the mounting position may include, for example, the height of the drive recorder 101 from the ground and an offset relative to the center line of the vehicle in the width direction.
  • the camera is installed as described above because the camera coordinate system and the vehicle coordinate system can be easily converted by simply moving it in parallel. Therefore, the camera may be installed in any way as long as it can capture images of the inside and outside of the vehicle. In that case, parameters necessary to maintain compatibility between the camera coordinate system and the vehicle coordinate system are set. Furthermore, in order to estimate the distance to the driver, the distance to the center of the driver's seat relative to the center line of the vehicle in the width direction of the vehicle (the distance in the depth direction in Figure 3) may be set.
  • the operator first inputs the position of the bottom edge of the front window via the input unit 206 (S401).
  • This position may be, for example, the height and depth from the ground.
  • the depth may be, for example, the distance along the X-axis from the camera mounting position.
  • the front window is a quadratic or cubic surface whose bottom edge is curved, the height and depth may be input based on the front edge.
  • the virtual line 103 that is set will be a straight line that passes through the input position and is parallel to the Z-axis.
  • the input position is converted into the camera coordinate system (S403).
  • the mounting position of the drive recorder 101 is set and stored when it is installed, so the input position is converted so that this position becomes the origin. If the height along the X axis and the depth along the Y axis are input based on the camera position, it is possible to convert into the camera coordinates as shown in FIG. 3 simply by determining the sign of the depth. Note that the units of length are unified to predetermined units such as centimeters or millimeters.
  • the position coordinates of the virtual line obtained in the final step S403 are written and stored in erasable ROM 209 (S405).
  • the coordinates of the virtual line set here are (Xv, Yv).
  • Xv, Yv When divided by virtual line (Xv, Yv), if the Y coordinate value of the intersection between the extension of line of sight vector 302 and the Y-Z plane whose X coordinate value is given by Xv exceeds Yv, then line of sight vector 302 can be presumed to indicate a line of sight looking forward through the front window.
  • the Y coordinate value of the intersection is equal to or less than Yv, then line of sight vector 302 can be presumed to indicate a line of sight looking at the dashboard 104 or below it without looking forward.
  • Fig. 5 An example of the procedure is shown in Fig. 5.
  • the process of Fig. 5 is a loop that is repeated without ending, but it may be interrupted by an interrupt or the like, and may be executed in parallel with other processes.
  • the process of Fig. 5 may also be executed by the CPU 204. This process may be started, for example, when the power of the vehicle 100 is turned on and power is supplied to the drive recorder 101. Alternatively, it may be started when the movement of the vehicle is detected from an image captured by the exterior camera 101F.
  • step S501 it is determined whether the driver monitoring setting is on (S501). This setting is stored in the ROM 209 or the like, and if it is on, driver monitoring is performed, and if it is not on, it is not performed. In step S501, if the monitoring setting is not on, the process loops at S501, but it is also possible to stop the processing in FIG. 5 and start execution after the monitoring setting is turned on.
  • the driver may be able to set the driver monitoring setting to on or off via the input unit 206, for example.
  • step S503 If it is determined that the driver monitoring setting is on, it is determined whether the current speed is equal to or greater than a predetermined speed (S503). If it is below the predetermined speed, monitoring is stopped. If the current speed is equal to or greater than the predetermined speed, the process branches to step S505. If the drive recorder 101 does not have a function for measuring speed (or if it does), step S503 may be skipped. In that case, if the driver monitoring setting is on, the process branches to step S505. For example, the speed can be estimated by the drive recorder 101 as follows.
  • the vehicle speed can be estimated by identifying a stationary target that corresponds between multiple frames, such as a building or a white line on the road, and dividing the difference in distance between the frames of the target by the time difference between the frames.
  • the gaze vector 302 of the driver 301 is identified from the image captured by the in-vehicle camera 101R (S505).
  • the gaze vector 302 may be a vector on the XY plane as shown in FIG. 3, and may not have a Z component.
  • the gaze vector may be identified using an existing method. For example, the driver's eyes may be recognized from the image. This may be achieved by using pattern matching or a trained model that has been machine-learned using features extracted from the image as input.
  • the face direction is identified.
  • the face direction may be identified based on the distortion of an inverted triangle formed by the binoculars and the mouth, using the binocular positions or, in addition, the mouth position as input.
  • the face direction identified in this way may be identified as the gaze vector direction.
  • the magnitude of the gaze vector may be appropriate. This process may be performed by machine learning.
  • the direction in which the eyes are facing may be identified and combined with the face direction to identify the gaze vector.
  • the image is transformed to face the face directly.
  • a part near the eye that is not the eyeball (for example, the inner corner or outer corner of the eye) is identified as the reference position, and the gaze is identified from the positional relationship between the pupil or iris of the eye and the reference position (this will be called a provisional gaze vector).
  • the positional relationship between the pupil or iris of the eye and the reference position and the provisional gaze vector may be associated in advance and saved, and the provisional gaze vector may be identified by referring to this. Since the provisional gaze vector is a vector when facing the driver's face directly, the provisional gaze vector is transformed so that it is tilted to match the direction of the face, and the desired gaze vector 302 is obtained.
  • line of sight vector 302 starting from the eye can be determined, but since it is a vector based on the eye position, it must be converted into the camera coordinate system. To do this, the eye position, which serves as the starting point, is determined in camera coordinates. Since the drive recorder 101 is fixed to the vehicle, if the eyes can be identified from the captured image, the direction of the eyes relative to the in-vehicle camera 101R can be identified. If the in-vehicle camera 101R is a stereo camera, the distance can be determined from the parallax, so the eye position can be identified in polar coordinates with the camera as the origin.
  • the in-vehicle camera 101R is a monocular camera
  • the distance is estimated from the identified eye direction.
  • the Z component value of the driver's position (eye position) in the camera coordinate system is determined by the mounting position of the camera and the seat arrangement of the vehicle. It has been explained that when the drive recorder 101 is mounted, for example, the height of the position of the drive recorder 101 from the ground and the offset with respect to the vehicle center line are set.
  • an offset in the width direction (Z direction) to the center of the driver's seat 102 with respect to the vehicle center line may be set.
  • the Z component value (Zdrv) of the center line in the width direction of the driver's seat 102 in the camera coordinate system It is assumed that the driver's eye position is within a plane (called the driver's seat center plane) parallel to the XY plane identified by this Z component value Zdrv. With this assumption, if a straight line from the camera toward the driver's eye is identified from the image, the distance to the intersection of this straight line and the driver's seat center plane is the distance from the camera to the eye. Once the direction and distance to the eye have been identified, the position in the polar coordinate system can be converted into the camera coordinate system to obtain the starting point of the line of sight vector 302. Of course, this method is just one example and other methods may be used.
  • the gaze vector 302 may be identified based on the face direction, and it may be determined that the gaze vector identification has been successful.
  • line of sight vector 302 is below the virtual line (S509). In other words, it is determined whether the extension of line of sight vector 302 is below virtual line 103. This determination may be made as described above. That is, when a virtual line (Xv, Yv) is set, the intersection of the extension of line of sight vector 302 and the YZ plane whose X coordinate value is given by Xv is found. If the Y coordinate value of this intersection is equal to or less than Yv, it can be estimated that line of sight vector 302 is below virtual line 103, in other words, that driver 301 is not looking ahead but is looking at dashboard 104 or below it.
  • This timer is, for example, built into the CPU 204, and is a timer that measures a predetermined time in order to determine whether the driver has been continuously directing his/her gaze below a preset virtual line for more than a predetermined time.
  • the predetermined time may be set when the timer is started, and an interrupt may be generated when the timer expires, or it may be monitored at regular intervals to see if the predetermined time has been reached. In this embodiment, monitoring is performed within a loop process, so the latter method may be used. The latter method is particularly suitable for the fifth embodiment, which will be described later, since it is possible to change the setting of the predetermined time after the timer has started. This predetermined time may also be called the reference time.
  • the timer is reset and a warning is output from the output unit 207 (S519).
  • This warning may be audible or visual.
  • the warning may be output from the drive recorder 101 itself, or from another device such as a mobile terminal. Note that the timer is stopped by being reset.
  • step S503 If it is determined in step S503 that the speed does not exceed the predetermined speed, or if it is determined in step S507 that the line of sight vector has failed to be identified, or if it is determined in step S509 that the extension of the line of sight vector is on the virtual line, the process branches to step S515.
  • step S515 the timer is reset and the output of the warning is stopped. It may be stopped even if a warning has not been output, or it may be stopped only if a warning has been output. Note that in FIG. 5, if the driver monitoring setting is turned off while a warning has been output in step S519, the warning will remain output, but in preparation for this case, if it is determined that the driver monitoring setting is not on, the process may branch to step S515.
  • the drive recorder 101 can monitor the driver, particularly whether the driver is looking ahead. If the driver is looking below the virtual line for more than a certain period of time, the driver may be gazing at the instrument panel, concentrating on operating it, or operating a mobile terminal. Such situations can be judged by the retrofittable drive recorder, and if there is a possibility, a warning can be issued to the driver, urging him or her to concentrate on driving.
  • the line of sight vector of the driver 301 seated in the driver's seat 102 is identified (S601).
  • the line of sight vector may be a vector on the XY plane, and the value in the Z direction does not need to be identified.
  • the driver is looking at the position where the virtual line 103 of the front and bottom edge of the window is set. It is then determined whether this has been completed (603).
  • “Completed” here means that multiple attempts have been completed. In multiple attempts, it is desirable for the driver to change the height of his or her face for each attempt, for example by changing the depth of his or her seat.
  • Multiple times means at least two times.
  • intersection points of the extension lines of each line of sight vector are identified (S605).
  • the position (X, Y) value of the identified intersection point is saved as the setting value of the virtual line (Xv, Yv) (S607).
  • Fig. 7 shows the driver monitoring procedure in the third embodiment.
  • Fig. 7 shows the procedure to be executed in place of step S519 when it is determined in step S517 in Fig. 5 that the predetermined time has expired. Therefore, the same steps as in Fig. 5 are omitted.
  • the direction of the face is specified, and the level of warning is changed depending on whether only the eyes are facing down the virtual line or the face is also facing down the virtual line.
  • the face direction is identified from the image captured by the in-vehicle camera 101R (S701). This process may be performed as described in step S505 of FIG. 5. However, in step S701, the face direction is identified in three dimensions, including the Z component.
  • a face direction vector is generated in place of the line of sight vector 302, and the intersection point between the extension of this vector and the Y-Z plane whose X coordinate value is given by Xv is found. If the Y coordinate value of this intersection point is equal to or less than Yv, it can be estimated that the face is facing down the virtual line 103.
  • the Z component of the face direction does not need to be used in this determination.
  • the vehicle is turning from the image from the exterior camera 101F (S705).
  • the motion vectors of corresponding targets between multiple frames are detected, and if the average direction is either left or right, it can be determined that the vehicle is turning in that direction.
  • the turning direction identified at this time is stored.
  • the turning direction can be represented, for example, by the sign (positive or negative) of the value obtained by subtracting the Z coordinate value of the end point from the Z coordinate value of the start point of the motion vector. Alternatively, the start point and end point may be reversed.
  • step S707 it is determined whether the stored turning direction and the face direction match. This determination may be made based on whether the sign indicating the turning direction stored in step S705 matches the sign of the value obtained by subtracting the Z component of the end point from the Z component of the start point of the vector indicating the face direction. If the signs match, it can be determined that the turning direction and the face direction match.
  • step S707 If it is determined in step S707 that there is no match, it is determined that the driver is looking below the virtual line based on eye movement alone, and a normal warning is output (S709). If it is determined that there is a match, it can be determined that the face is looking in the direction of the turn, so no warning is issued even if the gaze vector is determined to be pointing below the virtual line. Also, if it is determined in step S703 that the direction of the face is pointing below the virtual line, a stronger warning than normal is output (S711). This is because it is presumed that the driver is concentrating by looking away from the vehicle. The final timer is reset, and the process branches to step S501 (S713). A stronger warning than normal could be, for example, a loud or intermittent sound in the case of audio, or a display in a more noticeable color or flashing in the case of a display.
  • this embodiment can warn the driver more strongly when not only the line of sight but also the face is directed below the virtual line. Also, by not issuing a warning when it is estimated that the driver is looking in the turning direction, it is possible to reduce the annoyance to the driver caused by issuing unnecessary warnings.
  • Fig. 8 shows an example of a procedure executed by the CPU 204 in this case. Fig. 8 starts from the case where it is determined in step S509 of Fig. 5 that the extension line of the line of sight vector 302 is not below the virtual line, and the process other than that is the same as in the first embodiment. The description of the same parts as in the first embodiment will be omitted.
  • step S801 if it is determined in step S509 that the extension of gaze vector 302 is not below the virtual line, the value of the gaze vector is saved (S801).
  • the save location may be RAM 205, with a storage capacity sufficient to save a predetermined amount of gaze vectors reserved for that purpose.
  • Gaze vectors for a predetermined time are gaze vectors identified within a predetermined time, but if the amount of data is too large, it is possible to periodically sample and save not all of them but some of them in step S801. Note that in step S801, if the saved gaze vectors exceed the predetermined time, the oldest one may be deleted and a new gaze vector may be saved.
  • the range of the saved gaze vectors i.e., whether the end points of the gaze vectors are within a predetermined range (S805).
  • the smallest area that includes the end points of all saved gaze vectors is identified, and if its maximum diameter is less than or equal to a threshold, the range of the gaze vectors may be determined to be limited. Or, more simply, if, among all saved gaze vectors, the difference between the maximum Z component and the minimum Z component, and the difference between the maximum Y component and the minimum Y component are both less than or equal to a threshold, the range of the gaze vectors may be determined to be limited. In this case, the threshold for the Z component and the threshold for the Y component may be different.
  • the timer is reset and a warning is output (S807); if not, the timer is reset and the warning is stopped (S809).
  • the predetermined time for collecting the line of sight vector may be several seconds, for example, about 1 to 3 seconds.
  • Figures 9 and 10 show an example of the procedure executed by the CPU 204 in this case.
  • Figure 9 begins with the case where it is determined in step S511 of Figure 5 that the timer has been started, and the process other than that is the same as in the first embodiment. Descriptions of the same parts as in the first embodiment will be omitted.
  • the target object is identified from the image captured by the exterior camera 101F. This identification may be performed by pattern recognition, or may be performed by inputting the image features into a trained model that has previously learned the image features and the target object in the image through machine learning. As a result, it is determined whether the target object has been recognized (S902).
  • the process branches to step S501 in FIG. 5.
  • the process branches to step S501 in FIG. 5.
  • the targets include a preceding vehicle (S903). Whether or not it is a preceding vehicle may be determined, for example, based on its position, size, distance, etc. If it is determined that there is a preceding vehicle, the process branches to preceding vehicle processing in FIG. 10. If it is determined that there is no preceding vehicle, it is determined whether there are more than a predetermined number of targets in the image (S905). If it is determined that there are no targets that exceed the predetermined number, a normal reference time is set as the expiration time of the timer (S907).
  • the normal reference time may be the time determined as the expiration of the timer in step S517 in the first embodiment.
  • a shortened reference time that is shorter than the normal reference time is set as the expiration time of the timer (S907). Thereafter, the process branches to step S517 to determine whether the timer is full, and if it is full, step S519 is executed to output a warning.
  • step S903 If it is determined in step S903 that there is a leading vehicle, the distance to the leading vehicle is identified in step S911 of FIG. 10. If the distance was identified when determining the presence of the leading vehicle, that value may be used.
  • the height of the exterior camera 101F from the ground has already been set, and the direction of its optical axis is assumed to be parallel to the ground and the longitudinal axis of the vehicle. In other words, it is assumed that the vehicle 100 traveling straight travels toward the vanishing point of the image captured by the exterior camera 101F.
  • the distance to a point set on the ground can be estimated based on the image height of the image. This allows the distance to the leading vehicle to be estimated. Note that even if the installation position and direction of the camera are not as assumed above, the distance to the leading vehicle can be estimated by converting the coordinate system as long as the direction and height of the optical axis relative to the vehicle are known.
  • step S9 it is determined whether the identified distance exceeds a predetermined reference distance (S913). If it does, the process branches to step S907 in FIG. 9 to set the normal reference time, and if it is less than the predetermined distance, the process branches to step S909 to set a shortened reference time. In this way, when the distance to the preceding vehicle is less than the predetermined distance, a shortened predetermined time, i.e., a reference time, is set for when the distance exceeds the predetermined distance.
  • the object of monitoring was the driver's behavior of looking down the virtual line.
  • the object of monitoring may also be the driver's distracted driving. That is, for example, if the driver's line of sight vector (or face) is directed outside a specified range in front of the vehicle, a warning may be output.
  • a driver monitoring device that can be retrofitted to a vehicle, comprising: An image capturing means for capturing an image including a driver of the vehicle; A control means; output means; The control means Identifying the driver's line of sight from the image captured by the imaging means; determining whether the identified line of sight direction is continuously directed below a preset virtual line for more than a predetermined time period; A driver monitoring device is provided, characterized in that when it is determined that the line of sight is continuously directed below the virtual line for more than the specified time, the output means outputs a warning to the driver. This makes it possible to detect a situation in which the driver is looking down and to warn the driver.
  • a driver monitoring device comprising: The device further includes an input means, There is provided a driver monitoring device characterized in that the control means sets the virtual line based on the distance and height to the bottom edge of the front windshield of the vehicle relative to a predetermined reference position, which are input by the input means. This makes it possible to determine whether the driver is looking down based on the set virtual line, so that it is possible to easily and reliably grasp whether the driver is looking down.
  • a driver monitoring device comprising: There is provided a driver monitoring device, characterized in that the control means sets the virtual line based on the identified line of sight of the driver. This allows the virtual line to be set in a simple manner.
  • a driver monitoring device According to a fourth aspect of the present invention, there is provided a driver monitoring device according to the third aspect, wherein the control means sets the virtual line based on a plurality of different line-of-sight directions. This allows the virtual line to be set in a simple manner.
  • a driver monitoring device according to any one of the first to fourth aspects, There is provided a driver monitoring device characterized in that the control means specifies, as the line of sight direction, a direction in which the driver's face is facing. This makes it possible to specify the driver's line of sight in a simple manner.
  • a driver monitoring device according to any one of the first to fifth aspects, There is provided a driver monitoring device characterized in that the control means specifies the direction in which the driver's eyes are facing as the line of sight direction. This makes it possible to identify the direction of the driver's line of sight with higher accuracy, and to warn the driver with higher accuracy.
  • a driver monitoring device according to any one of the first to sixth aspects, The control means Further determining the direction of the driver's face;
  • a driver monitoring device is provided, characterized in that when it is determined that the driver's line of sight is continuously directed below the virtual line for more than the specified time period and the identified face direction is directed below the virtual line, the warning is output in a manner different from when the face direction is not directed below the virtual line. This makes it possible to detect not only the driver's line of sight but also the direction of the driver's face, and if it is estimated that the driver is distracted, the manner of the warning can be changed.
  • a driver monitoring device according to any one of the first to seventh aspects,
  • the control means Further determining whether the identified gaze direction is within a certain range for more than a second predetermined time;
  • the driver monitoring device is characterized in that a warning is output by the output means also when it is determined that the line of sight direction is within a certain range for more than the second predetermined time. This makes it possible to issue a warning not only when the driver is looking down, but also when the driver's gaze is fixed.
  • a driver monitoring device according to any one of the first to eighth aspects, Further, a second image capturing means for capturing an image of a front of the vehicle is provided, The control means Detecting a target from an image in front of the vehicle; There is provided a driver monitoring device characterized in that, when more than a predetermined number of targets are detected, the predetermined time is shortened in comparison with a case where no more than the predetermined number of targets are detected. This allows the company to be warned at an earlier stage if there are many obstacles outside the vehicle.
  • a driver monitoring device According to an eleventh aspect of the present invention, there is provided a driver monitoring device according to the ninth aspect, there is provided a driver monitoring device characterized in that the control means does not cause the output means to output the warning when no target object is detected in an image ahead of the vehicle. This allows the warning to be suppressed if there are no obstacles in the vicinity.
  • a driver monitoring device according to any one of the first to eleventh aspects,
  • the control means Further determining the direction of the driver's face;
  • a driver monitoring device characterized in that, when the identified face of the driver is directed outside a predetermined range in front of the vehicle, the warning is output by the output means. This makes it possible to issue a warning even if the driver is distracted.
  • a driver monitoring device comprising: Further, a second image capturing means for capturing an image of a front of the vehicle is provided, The control means determining whether the vehicle is turning based on an image of a front of the vehicle; A driver monitoring device is provided which, when it is determined that the vehicle is turning and the identified face of the driver coincides with the direction in which the vehicle is turning, does not output the warning even if the identified face of the driver is directed outside a specified range in front of the vehicle. As a result, even if the driver is looking away from the vehicle, the warning can be suppressed if the driver is looking in the direction of a turn.
  • a driver monitoring device that functions as a drive recorder that records the latest video captured by a capturing means for a certain period of time. This allows the device to also function as a drive recorder.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Mechanical Engineering (AREA)
  • Business, Economics & Management (AREA)
  • Emergency Management (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2023/013629 2023-03-31 2023-03-31 ドライバ監視装置及びプログラムと媒体 Ceased WO2024202037A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2025509619A JPWO2024202037A1 (https=) 2023-03-31 2023-03-31
CN202380095710.4A CN120917502A (zh) 2023-03-31 2023-03-31 驾驶员监视装置、程序以及介质
PCT/JP2023/013629 WO2024202037A1 (ja) 2023-03-31 2023-03-31 ドライバ監視装置及びプログラムと媒体
US19/341,028 US20260024358A1 (en) 2023-03-31 2025-09-26 Driver monitoring apparatus and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/013629 WO2024202037A1 (ja) 2023-03-31 2023-03-31 ドライバ監視装置及びプログラムと媒体

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/341,028 Continuation US20260024358A1 (en) 2023-03-31 2025-09-26 Driver monitoring apparatus and medium

Publications (1)

Publication Number Publication Date
WO2024202037A1 true WO2024202037A1 (ja) 2024-10-03

Family

ID=92904597

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/013629 Ceased WO2024202037A1 (ja) 2023-03-31 2023-03-31 ドライバ監視装置及びプログラムと媒体

Country Status (4)

Country Link
US (1) US20260024358A1 (https=)
JP (1) JPWO2024202037A1 (https=)
CN (1) CN120917502A (https=)
WO (1) WO2024202037A1 (https=)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009015549A (ja) * 2007-07-04 2009-01-22 Omron Corp 脇見検出装置および方法、並びに、プログラム
JP2009277021A (ja) * 2008-05-15 2009-11-26 Hitachi Ltd 接近物検出装置および接近物検出方法
WO2014054198A1 (ja) * 2012-10-01 2014-04-10 株式会社デンソー 車両の移動方向検出装置
WO2016027411A1 (ja) * 2014-08-22 2016-02-25 株式会社デンソー 車載制御装置
JP2017016568A (ja) * 2015-07-06 2017-01-19 株式会社デンソー 運転者異常検出装置
JP2018022349A (ja) * 2016-08-03 2018-02-08 パナソニックＩｐマネジメント株式会社 情報提示装置
JP2018097515A (ja) * 2016-12-12 2018-06-21 株式会社デンソーアイティーラボラトリ 運転支援装置、運転支援方法、およびプログラム
JP2019091281A (ja) * 2017-11-15 2019-06-13 オムロン株式会社 脇見判定装置、脇見判定方法及び脇見判定のためのプログラム
WO2019188112A1 (ja) * 2018-03-28 2019-10-03 マツダ株式会社 車両用警報装置
WO2020250962A1 (ja) * 2019-06-11 2020-12-17 日本電気株式会社 画像処理装置、画像処理方法、記録媒体
JP2021157831A (ja) * 2018-03-19 2021-10-07 日本電気株式会社 余所見判定装置、余所見判定システム、余所見判定方法、プログラム

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009015549A (ja) * 2007-07-04 2009-01-22 Omron Corp 脇見検出装置および方法、並びに、プログラム
JP2009277021A (ja) * 2008-05-15 2009-11-26 Hitachi Ltd 接近物検出装置および接近物検出方法
WO2014054198A1 (ja) * 2012-10-01 2014-04-10 株式会社デンソー 車両の移動方向検出装置
WO2016027411A1 (ja) * 2014-08-22 2016-02-25 株式会社デンソー 車載制御装置
JP2017016568A (ja) * 2015-07-06 2017-01-19 株式会社デンソー 運転者異常検出装置
JP2018022349A (ja) * 2016-08-03 2018-02-08 パナソニックＩｐマネジメント株式会社 情報提示装置
JP2018097515A (ja) * 2016-12-12 2018-06-21 株式会社デンソーアイティーラボラトリ 運転支援装置、運転支援方法、およびプログラム
JP2019091281A (ja) * 2017-11-15 2019-06-13 オムロン株式会社 脇見判定装置、脇見判定方法及び脇見判定のためのプログラム
JP2021157831A (ja) * 2018-03-19 2021-10-07 日本電気株式会社 余所見判定装置、余所見判定システム、余所見判定方法、プログラム
WO2019188112A1 (ja) * 2018-03-28 2019-10-03 マツダ株式会社 車両用警報装置
WO2020250962A1 (ja) * 2019-06-11 2020-12-17 日本電気株式会社 画像処理装置、画像処理方法、記録媒体

Also Published As

Publication number Publication date
US20260024358A1 (en) 2026-01-22
CN120917502A (zh) 2025-11-07
JPWO2024202037A1 (https=) 2024-10-03

Similar Documents

Publication Publication Date Title
JP7060031B2 (ja) 運転者監視システム
JP4926437B2 (ja) 車両の運転支援装置
WO2019073708A1 (ja) 車両用運転支援装置
US20180229654A1 (en) Sensing application use while driving
JP3228086B2 (ja) 運転操作補助装置
JP4683192B2 (ja) 車両用死角監視装置及び車両用運転支援システム
JP6822325B2 (ja) 操縦支援装置、操縦支援方法、プログラム
JP2014515847A (ja) 運転者覚醒度判定システム及び方法
JP2010143411A (ja) ヘッドアップディスプレイ装置
JP6730096B2 (ja) 乗員状態監視装置
JP2013132970A (ja) ミラー制御装置およびプログラム
JP2015085719A (ja) 注視対象物推定装置
KR20140079947A (ko) 차량용 영상 녹화 장치 및 차량용 영상 녹화 방법
JP6673005B2 (ja) 顔向き推定装置及び顔向き推定方法
JP5223289B2 (ja) 視覚情報呈示装置及び視覚情報呈示方法
JP2009251761A (ja) 運転支援システム
JP5811623B2 (ja) 車両用監視装置
CN116788158A (zh) 乘员状态监视装置
JP2021077137A (ja) 運転者状態推定装置
WO2024202037A1 (ja) ドライバ監視装置及びプログラムと媒体
WO2017150466A1 (ja) 運転者監視システム
JP2019179370A (ja) 車速制御装置
JP2007008382A (ja) 視覚情報呈示装置及び視覚情報呈示方法
JP2024112588A (ja) ドライバモニタ装置、ドライバモニタ方法及びドライバモニタ用コンピュータプログラム
WO2013114871A1 (ja) 運転支援装置及び運転支援方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23930671

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: CN2023800957104

Country of ref document: CN

Ref document number: 202380095710.4

Country of ref document: CN

ENP Entry into the national phase

Ref document number: 2025509619

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025509619

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 202380095710.4

Country of ref document: CN

122 Ep: pct application non-entry in european phase

Ref document number: 23930671

Country of ref document: EP

Kind code of ref document: A1