JP7276082B2 - Driver state estimation device - Google Patents

Description

The technology disclosed herein belongs to the technical field related to a driver's state estimation device.

Recently, the development of automatic driving systems has been promoted nationally. The applicant of the present application believes that, at present, there are roughly two directions for automatic driving systems.

The first direction is a system in which an automobile plays a central role in carrying a passenger to a destination without the need for driver's operation, that is, so-called fully automatic driving of an automobile. On the other hand, the second direction is an automatic driving system based on the premise that a human being will drive the car, for example, to enjoy driving the car.

In the automatic driving system of the second direction, for example, when the driver develops a disease that makes it difficult to drive normally, the car automatically replaces the passenger and performs automatic driving. is assumed. Therefore, the ability to quickly and accurately detect the occurrence of an abnormality in the driver, especially the occurrence of a functional disorder or disease in the driver, will improve the survival rate of the driver and improve the safety of people around them. It is extremely important from the viewpoint of ensuring

As a method of estimating a driver's abnormality, an attempt has been made to estimate the state of the driver based on the line-of-sight movement of the driver of the vehicle, as in Patent Document 1, for example. Specifically, in Patent Literature 1, an image captured by an imaging device that captures the driver's line of sight direction and the surroundings of the vehicle is acquired, and each of a plurality of points on an object existing in the vicinity of the vehicle is captured. Acquisition means for acquiring three-dimensional position information specifying a position relative to the own vehicle; Conspicuity calculation means for calculating conspicuity of the line of sight of the driver; Based on the conspicuity calculated by the conspicuity calculation means and determining means for determining the state of the driver by comparing the extent to which the driver looks at the conspicuous part with a predetermined threshold time.

Further, Patent Document 2 discloses obstacle detection means for detecting a relative positional relationship with respect to the own vehicle of an obstacle present in front, gazing point detection means for detecting the gazing point of the driver of the own vehicle, and obstacle detection means. and a driving skill detection means for detecting the driving skill of the driver on the basis of the driver's viewing behavior with respect to the obstacle detected by the detection means.

Japanese Patent No. 5966640 Japanese Patent No. 4385392

However, when the vehicle is traveling in a place such as an expressway where the external environment does not change much, the driver's line of sight does not change much. Therefore, when estimating the driver's state by comparing the external environment captured by a camera with the line of sight of the driver, as in Patent Document 1, the estimation accuracy decreases in places where the external environment changes little. There is a risk.

On the other hand, it is conceivable to display a head-up display or the like and estimate the driver's condition based on the movement of the driver's line of sight with respect to the display. However, as a result of intensive research by the inventors of the present application, it was found that even when the driver is in a normal state, there is a large difference in the line-of-sight movement time depending on the position of the display.

The technology disclosed herein maximizes the accuracy of estimating the driver's state while the vehicle is running, even in a place where the external environment changes little.

In order to solve the above-mentioned problems, the technology disclosed herein targets a driver state estimation device for estimating the state of a driver based on the line-of-sight movement of the driver of the vehicle. a line-of-sight detection device that detects the line-of-sight direction of the driver, a display device that performs a specific display according to a driving scene of the vehicle in the line-of-sight area of the driver when the vehicle is running, and a display device that detects the line-of-sight direction of the driver. an estimating unit for estimating that the driver has an abnormality when the specific display is not turned to the specific display within a threshold time after displaying the specific display; and a threshold setting unit for setting the threshold time.

In other words, the inventors of the present application conducted extensive research and found that, for example, when a specific display is displayed in the vicinity of the front pillar trim, the line-of-sight movement time becomes relatively long even if the driver is in a normal state. . Therefore, if the threshold time is set according to the position where the specific display is displayed, erroneous determination can be suppressed and the accuracy of estimating the state of the driver can be improved.

Further, according to the above configuration, even when driving in a place such as an expressway where the external environment does not change much, the specific display is displayed by the display device, and the line of sight when the driver sees the specific display. Movement can be used to estimate the state of the driver. Therefore, even in a place where there is little change in the external environment, the accuracy of estimating the state of the driver while the vehicle is running can be made as high as possible.

In one embodiment of the driver state estimating device, the threshold setting unit sets the threshold time to a predetermined angular range in the horizontal direction from a reference position when the line of sight of the driver faces forward. A configuration may be adopted in which the specific display is longer when displayed in a region outside a predetermined angular range in the horizontal direction from the reference position than when the specific display is displayed in the inner region.

That is, inside the predetermined angle range, the specific display is displayed at a position that is easy for the driver to see, so the line-of-sight movement time tends to be short. On the other hand, outside the predetermined angle range, the specific display is displayed at the edge of the driver's field of vision and at a position that is easy for the driver to see, so the line-of-sight movement time tends to be long. Therefore, by setting the threshold time to be longer when the specific display is displayed in an area outside the predetermined angle range than when the specific display is displayed in the area within the predetermined angle range, the driver's condition can be estimated. It is possible to further improve the accuracy of the calculation.

In the one embodiment, the predetermined angle range is 10 degrees, and the threshold setting unit sets the threshold time to when the specific display is displayed in an area with a viewing angle of less than 10 degrees in the horizontal direction from the reference position. Alternatively, the display may be set to be longer when the specific display is displayed in a viewing angle area that is 10 degrees abnormal in the horizontal direction from the reference position.

That is, as a result of intensive research by the inventors of the present application, when a specific display is displayed in an area of 10 degrees or more in the horizontal direction from the reference position when the line of sight of the driver is facing the front, the line of sight It was found that the travel time tends to be longer. Therefore, by setting the predetermined angle range to 10 degrees, it is possible to further improve the accuracy of estimating the state of the driver.

In the driver state estimation device, the display device displays the specific display until the threshold time elapses after displaying the specific display, or until the line-of-sight direction of the driver turns to the specific display. The configuration may be such that the change is made to be noticeable gradually.

According to this configuration, the time required for estimating the driver's condition can be shortened as much as possible. As a result, even when the driver's physical condition is abnormal, it is possible to estimate the physical condition of the driver as soon as possible, and to quickly take measures such as stopping the vehicle on the shoulder of the road. become.

In the driver state estimation device, the line-of-sight area is an area of the front window glass of the vehicle, and the display device is a head-up display that projects a virtual image on the vehicle front side of the front window glass. It may be a configuration.

According to this configuration, devices for estimating the state of the driver can be easily configured. Further, by displaying a specific display within the region of the windshield, the state of the driver can be estimated without interfering with the driver's driving as much as possible.

As described above, according to the technology disclosed herein, the accuracy of estimating the driver's state while the vehicle is traveling can be made as high as possible even in a place where there are few changes in the external environment.

FIG. 2 is a schematic diagram showing a state in which the front side of the vehicle is viewed from the driver's seat side in the interior of the vehicle in which the device for estimating the driver's state according to exemplary Embodiment 1 is mounted. 1 is a front view of a vehicle; FIG. 1 is a block diagram showing the configuration of a driver's state estimation device; FIG. FIG. 4 is a diagram showing a state in which a specific display is displayed in the driver's field of vision; 6 is a diagram showing a state in which a specific display is displayed at a position different from that in FIG. 5 in the driver's field of view. FIG. 5 is a graph showing the line-of-sight movement time when the driver is in a normal state and the line-of-sight movement time when the driver is in a careless state; FIG. 10 is a diagram showing the position of a specific display displayed in the driver's field of view in an experiment using a simulator; 4 is a flowchart showing a processing operation for estimating a driver's state; FIG. 3 is a block diagram showing the configuration of a driver's state estimation device according to Embodiment 2; 9 is a flowchart showing processing operations for estimating a driver's state by a driver's state estimation device according to Embodiment 2;

Exemplary embodiments are described in detail below with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. Also, the left side when the front side is viewed from the rear side is called the left side, and the opposite is called the right side.

FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and a steering wheel 8 is arranged on the right side.

A front window glass 1 is arranged on the front side of the vehicle when viewed from the driver's seat in the vehicle interior. The front window glass 1 is partitioned by a plurality of vehicle components when viewed from the inside of the vehicle compartment. Specifically, the front window glass 1 is partitioned by left and right front pillar trims 2 , a roof trim 3 , and an instrument panel 4 .

The left and right front pillar trims 2 form boundaries on the outer side of the front window glass 1 in the vehicle width direction. Each front pillar trim 2 is arranged along each front pillar. As shown in FIG. 1, the front pillar trims 2 are arranged so as to be inclined upward so as to be spaced apart from each other.

The roof trim 3 forms the upper boundary of the front window glass 1 . The roof trim 3 covers the interior side of the roof panel of the vehicle. A rearview mirror 5 is attached to the center of the front window glass 1 in the vehicle width direction and a slightly lower portion of the roof trim 3 . A vehicle interior camera 101 is provided in the vicinity of the rearview mirror 5 in the roof trim 3 to photograph the interior of the vehicle, particularly the driver's face.

The instrument panel 4 constitutes the lower boundary of the windshield 1 . A meter box and a display 7 are provided on the instrument panel 4 .

The vehicle also has a head-up display 10 (hereinafter referred to as HUD 10) as a display device that displays an image in the driver's field of view. Specifically, the HUD 10 displays a virtual image on the vehicle front side of the front window glass 1 . The HUD 10 displays a virtual image of the image in a predetermined area on the front window glass 1 by projecting the image onto the front window glass 1 from below. In Embodiment 1, the display range of the HUD 10 is set to the entire front window glass 1 .

The vehicle also has side mirrors 6 outside the left and right front pillars in the vehicle width direction. Each side mirror 6 is arranged so that the driver sitting in the driver's seat can see through the side door window.

As shown in FIG. 2, the vehicle is provided with a plurality of cameras 100 for photographing the external environment (only one is shown in FIG. 2). For example, the camera 100 that captures the external environment on the front side of the vehicle is arranged at the front end of the vehicle slightly below the hood 9 of the vehicle. A plurality of cameras 100 are also provided on both sides of the vehicle and also on the rear of the vehicle so that the surroundings of the vehicle can be photographed horizontally at 360 degrees.

(Configuration of driver state estimation device)
The vehicle according to the first embodiment has a function of estimating the driver's state based on the movement of the driver's line of sight. FIG. 3 shows the configuration of the driver's state estimation device 20. As shown in FIG. The driver's state estimation device 20 has a controller 50 having an estimation unit 56 for actually estimating the driver's state. This controller 50 is a well-known microcomputer-based controller, and includes a central processing unit (CPU) that executes programs, and a RAM (Random Access Memory) or ROM (Read Only Memory) for example. It comprises a memory configured to store programs and data, and an input/output bus for inputting/outputting electrical signals.

The controller 50 is connected with various sensors as shown in FIG. The sensors include, for example, a camera 100 that captures the external environment in front of the vehicle, an in-vehicle camera 101 that captures the face of the driver, a vehicle speed sensor 102 that detects the speed of the vehicle, and a global positioning system (Global Positioning System). and a position sensor 103 that detects the position of the vehicle (vehicle position information) using GPS.

The controller 50 has a driving state determination unit 51 that determines the driving scene of the vehicle from the image representing the external environment acquired by the camera 100, the detection result of the vehicle speed sensor 102, and the detection result of the position sensor 103. The driving condition determination unit 51 determines the time zone of driving (daytime, nighttime, etc.) and road conditions (traffic volume, etc.) from the image acquired by the camera 100 . The traveling state determination unit 51 detects whether or not the vehicle is traveling on an expressway based on the detection result of the vehicle speed sensor 102 and the detection result of the position sensor 103 . The traveling state determination unit 51 determines the traveling location (city, suburb, etc.) from the detection result of the position sensor 103 .

Also, the driving state determination unit 51 determines whether or not there is a dangerous object around the vehicle. Dangerous objects include other vehicles braking in front of your vehicle, other vehicles approaching your vehicle from behind in the next lane, and pedestrians in the vicinity of your vehicle in front of your vehicle. be. Further, for example, when the own vehicle is stepping on a white line indicating a roadside strip, the white line is also recognized as a dangerous object.

The controller 50 has a display control section 52 that controls the HUD 10 . The display control unit 52 outputs control signals for controlling information displayed on the HUD 10 . Although details will be described later, the display control unit 52 sends a control signal to the HUD 10 so as to perform a specific display informing of the presence of a dangerous object when a dangerous object exists in a driving scene where the environment outside the vehicle is difficult to change. Output.

The controller 50 has a line-of-sight calculator 53 that calculates the line-of-sight direction of the driver from the driver's face photographed by the vehicle interior camera 101 . For example, the line-of-sight calculation unit 53 calculates the line-of-sight direction of the driver by detecting a change in the driver's pupil from a state in which the driver looks into the lens of the vehicle interior camera 101 as a reference. The line-of-sight direction may be calculated from either the left eye or the right eye of the driver, or may be the average value of the line-of-sight directions (line-of-sight vector) obtained from both eyes of the driver. Further, when it is difficult to calculate the line-of-sight direction of the driver from the change in the driver's pupils, the line-of-sight direction may be calculated from the orientation of the driver's face. The vehicle interior camera 101 and the controller 50 are an example of a line-of-sight detection device.

The controller 50 has an attractiveness calculation unit 55 that calculates an attractiveness indicating how easily the specific display attracts the driver's line of sight when the HUD 10 is caused to display the specific display by the display control unit 52 . The attractiveness calculation unit 55 calculates the attractiveness in consideration of the position where the specific display is displayed and the external environment of the vehicle when the line of sight of the driver turns to the specific display.

In the first embodiment, the attractiveness calculation unit 55 calculates saliency (visual saliency) as the attractiveness. Saliency is a visual feature that consists of saliency that changes moment by moment due to color, brightness, movement, and the like. That is, high saliency means high conspicuity, and a high saliency region is a region that stands out from the surroundings. More specifically, the saliency is high in an area that has a large color difference or luminance difference with respect to the surrounding area, or that has a large movement with respect to the surrounding area. For example, when a red light is projected as a specific display, the specific display tends to be buried in the surrounding scenery at dusk, so the saliency is low, but at night the specific display tends to stand out against the surrounding scenery. , the saliency increases. For calculation of saliency itself, a known computer program published on the Internet or the like can be used.

The controller 50 has an estimating unit 56 that estimates the state of the driver based on the movement of the driver's line of sight. The estimating unit 56 estimates the state of the driver, particularly based on the eye movement time of the driver and the degree of attraction calculated by the degree of attraction calculating unit 55 . Although the details will be described later, the estimation unit 56 estimates that the driver has an abnormality when the eye movement time of the driver is equal to or longer than the threshold time. The abnormal state of the driver includes a careless state of the driver and a state in which the driver is in an abnormal physical condition.

The controller 50 has a threshold value setting unit 57 for setting a threshold time used when estimating the state of the driver by the estimating unit 56 . Calculation of the threshold by the threshold setting unit 57 will be described later.

The controller 50 controls various actuators A, the display 7, the buzzer B, etc. mounted on the vehicle based on the estimation result of the estimation unit 56 . The various actuators A include actuators that constitute an engine system, a brake system, and a steering system.

(Indication of Specific Indication)
The specific display by the display control unit 52 and the HUD 10 is a display for calling the driver's attention. The display control unit 52 performs a specific display when the vehicle is running in a specific scene such as a highway where the external environment does not change much. This is because in a place where there is little change in the external environment, the driver is less stimulated and tends to be in a careless state. In addition, in places where there are few changes in the external environment, the movement of the driver's line of sight is small. Therefore, in order to estimate the state of the driver based on the movement of the driver's line of sight, the specific display actively encourages the movement of the driver's line of sight. Because it is necessary.

The specific indication changes depending on the type of dangerous object. For example, when another vehicle running in front of the host vehicle is determined to be a dangerous object, a ring 60 surrounding the other vehicle is displayed as a specific display, as shown in FIG. This prompts the driver to stop the other vehicle ahead. Further, for example, when another vehicle is approaching from the right rear side of the own vehicle, as shown in FIG. . This urges the driver to check the approaching vehicle from the rear using the mirror. The display range of this specific display is a range that not only alerts the driver but does not interfere with the driver's driving.

The display control unit 52 gradually makes the specific display more conspicuous until the threshold time set by the threshold setting unit 57 elapses after the specific display is displayed, or until the line of sight of the driver turns to the specific display. Change (to increase saliency). For example, the display control unit 52 gradually darkens the color of the specific display, changes the lighting display to blinking display, or shortens the blinking interval in the case of blinking display. Note that the specific display may be changed continuously with the lapse of time, or may be changed at regular time intervals.

(Calculation of threshold time)
In the first embodiment, the threshold setting unit 57 is configured to set the threshold time based on the position where the specific display is displayed by the HUD 10 and the saliency of the specific display when the driver's line of sight is directed to the specific display. It is Hereinafter, description will be made with reference to FIGS. 6 and 7. FIG.

FIG. 6 shows the result of measuring the driver's line-of-sight movement time when the gaze point is displayed in the driver's field of view. The graph shown in FIG. 6 was created by conducting an experiment using a drive simulator.

FIG. 7 schematically shows the experimental conditions that were used to generate the graph shown in FIG. The experiment was conducted by having the subject watch the moving image D by the drive simulator. First, as shown in FIG. 7, the field of view when the driver is seated is displayed. Next, while advancing the moving image D, a white circle is randomly displayed as a gaze point at one of the positions indicated by the numbers in FIG. 7 at regular time intervals. Then, the time required for the subject's line of sight to turn to the fixation point was measured. In addition, the moving image D displayed in the drive simulator displayed a range with a viewing angle of 30 degrees to the left and right from the reference position (0 degrees) when the line of sight of the subject faced the front. In general, the front pillar trim is located at a position 20 degrees to the right side of the vehicle (opposite to the front passenger seat) from the reference position.

In this experiment, when assuming a normal state, the subject was asked to concentrate on the video D without doing another task (such as solving mental arithmetic problems), and when assuming a careless state, the subject was asked to I proceeded with Movie D while requesting another task (this time I was asked to solve a mental arithmetic problem). Experiments were conducted on the same subject under the assumption of a normal state and the case of a careless state.

In the graph of FIG. 6, the horizontal axis is the viewing angle in the horizontal direction. It is on the plus side. This reference position corresponds to the position of the line of sight when the driver sits in the driver's seat and faces the front. The vertical axis on the left indicates the line-of-sight movement time when the subject is assumed to be in a normal state, and the vertical axis on the right side indicates the line-of-sight movement time when the subject is assumed to be in a careless state. Both vertical axes have the same width. Although the line-of-sight movement time differs even at the reference position between the normal state and the careless state, in the graph of FIG. The vertical axis is adjusted so that both curves overlap at the position.

As shown in FIG. 6, it can be seen that the line-of-sight movement time tends to increase as the subject moves away from the reference position regardless of whether the subject is in a normal state or in a careless state. Also, from the graph of FIG. 6, it can be seen that when the viewing angle is ±10 degrees or more, the difference between the line-of-sight movement time in the normal state and the line-of-sight movement time in the careless state increases. This is because a position with a large viewing angle is difficult for the subject to see in the first place, and is difficult for the subject to notice.

In Embodiment 1, the threshold setting unit 57 is configured to set the threshold time according to the position where the specific display is displayed based on this experimental result. Specifically, the threshold setting unit 57 sets the threshold time as a reference position when the line of sight of the driver faces the front, and displays the specific display in an area within a predetermined angular range in the horizontal direction from the reference position. It is set to be longer when the specific display is displayed in a region outside the predetermined angle range in the horizontal direction from the reference position than when the specific display is displayed. In particular, in the first embodiment, the predetermined angle range is set to 10 degrees based on the experimental results described above. That is, in the first embodiment, the threshold setting unit 57 sets the threshold time to 10 degrees horizontally from the reference position longer than when the specific display is displayed in the region with a viewing angle of less than 10 degrees horizontally from the reference position. It is set to be longer when the specific display is displayed in a viewing angle area of 100 degrees or more. More specifically, the threshold setting unit 57 sets the threshold time for estimating that the driver is in the above state to a constant value within a range of less than 10 degrees in the horizontal direction from the reference position. In the range of 10 degrees or more, the threshold time increases as the viewing angle increases.

Various methods can be adopted for setting the threshold time by the threshold setting unit 57 . For example, a map in which the threshold time is set based on the display position of the specific display may be stored in the threshold setting unit 57, and the threshold time may be set by reading the map. Further, for example, the threshold time in the range of viewing angles of less than 10 degrees may be corrected according to the display position to obtain the final threshold time. It should be noted that the threshold time in the viewing angle range of less than 10 degrees is set to 0.44 s to 0.46 s, for example.

(Estimation of Driver's State)
Next, the processing operation for estimating the driver's state will be described based on the flowchart of FIG.

First, in step S101, the controller 50 acquires various data.

In the next step S102, the controller 50 determines whether the driving scene of the vehicle is in a specific state or a specific scene. When the driving scene is not a specific scene, the controller 50 returns. On the other hand, when the driving scene is the specific scene NO, the controller 50 proceeds to step S103.

At step S103, the controller 50 determines whether or not there is a dangerous object around the vehicle. The controller 50 returns when YES that there are no dangerous objects around the vehicle. On the other hand, the controller 50 proceeds to step S104 when the answer is NO that there is a dangerous object around the vehicle.

At step S104, the controller 50 determines whether or not the driver of the vehicle has seen a dangerous object. This determination is made based on whether or not the line-of-sight direction of the driver calculated by the line-of-sight calculation unit 53 has turned toward the dangerous object within a preset set time. If YES that the driver has seen the dangerous object within the set time, the controller 50 proceeds to step S113. Sometimes, the process proceeds to step S105. When the dangerous object is behind the own vehicle, the controller 50 determines that the driver has seen the dangerous object when the line of sight of the driver looks at the rearview mirror 5 or the side mirror 6. It's becoming

At step S105, the controller 50 causes the HUD 10 to display a specific display on the front window glass 1. FIG.

In the next step S106, the controller 50 sets the threshold time according to the display position of the specific display.

In the next step S107, the controller 50 determines whether or not the driver has seen the specific display. This determination is made based on whether or not the line-of-sight direction of the driver calculated by the line-of-sight calculation unit 53 is directed toward the specific display. If YES, the controller 50 proceeds to step S110. On the other hand, if NO, the controller 50 proceeds to step S108.

In step S108, the controller 50 changes the display method of the specific display so as to increase the saliency of the specific display.

In the next step S109, the controller 50 determines whether or not the elapsed time since displaying the specific display is within the threshold time. When the elapsed time is within the threshold time (YES), the controller 50 returns to step S107. On the other hand, when the elapsed time exceeds the threshold time (NO), the controller 50 proceeds to step S112.

In step S110 to which the controller 50 proceeds when the determination in step S107 is YES, the controller 50 calculates the saliency of the specific display when the driver sees the specific display.

In subsequent step S111, the controller 50 determines whether or not the saliency calculated in step S110 is greater than or equal to the set value set according to the display position. When the saliency of the specific display is equal to or greater than the set value and YES, the controller 50 proceeds to step S112. On the other hand, when the saliency of the specific display is less than the set value, NO, the controller 50 proceeds to step S113.

At step S112, the controller 50 estimates that the driver is in an abnormal state. After step S112, the process returns.

At step S113, the controller 50 estimates that the driver is in a normal state. After step S113, the process returns.

As described above, the controller 50 presumes that the driver is in an abnormal state depending on the saliency of the specific display even if the driver views the specific display within the threshold time. For example, when the specific display is red light, the saliency of the specific display is higher than that in the daytime because it is conspicuous when the specific display is performed at night. In other words, if the driver is in a normal state, it is considered that he/she sees the specific display earlier at night than during the day. Even if the driver is in an abnormal state, there is a possibility that the specific display will be seen within the threshold time at night. Therefore, even if the driver looks at the specific display within the threshold time, if the saliency of the specific display is high when the driver looks at it, the driver will judge that it is in an abnormal state. As a result, it is possible to improve the estimation accuracy of the driver's state.

The controller 50 estimates the depth of the abnormal condition when it is estimated that the driver is in an abnormal condition. For example, the controller 50 determines that the driver has failed when the driver does not see the specific display within the threshold time even though the saliency of the specific display is high. Further, when the saliency of the specific display is low and the driver does not see the specific display within the threshold time, for example, the controller 50 displays another display in front of the driver so that the driver can see the display. Depending on the time to time, it can be estimated whether the person is in a state of inattentiveness, a state of reduced alertness, or a failure.

The controller 50, for example, sounds a buzzer B or displays on the display 7 when it is estimated that the driver's arousal level has decreased or the driver is in a careless state. As a result, the controller 50 prompts the driver to wake up and concentrate on driving. Further, when it is estimated that the driver is in trouble, the controller 50, for example, outputs a control signal to various actuators A to drive the vehicle to a safe area such as the shoulder of the road, and to drive the vehicle to a safe area such as a road shoulder. Stop the vehicle in the area.

Therefore, according to the first embodiment, the line-of-sight detection device (the line-of-sight calculation unit 53 and the vehicle interior camera 101) for detecting the line-of-sight direction of the driver and the line-of-sight region of the driver during driving of the vehicle are added to the driving scene of the vehicle. When the display device (HUD 10 and display control unit 52) that performs specific display in response and the line of sight of the driver does not turn to the specific display within the threshold time after the display device displays the specific display , an estimating unit 56 for estimating that the driver has an abnormality, and a threshold setting unit 57 for setting the threshold time according to the position where the specific display is displayed. As a result, the threshold time can be set by reflecting the characteristics of the line-of-sight movement of the driver. As a result, erroneous determination is suppressed, and the accuracy of estimating the state of the driver can be improved.

In addition, even when driving in a place such as a highway where the external environment does not change much, a specific display is displayed on the display device, and the driver uses the movement of the line of sight when looking at the specific display to drive. A person's condition can be estimated. Therefore, even in a place where there is little change in the external environment, the accuracy of estimating the state of the driver while the vehicle is running can be made as high as possible.

Further, in the first embodiment, the threshold setting unit 57 sets the threshold time to a specific display area within a predetermined angular range in the horizontal direction from the reference position when the line of sight of the driver faces the front. is set to be longer when the specific display is displayed in a region out of the predetermined angular range in the horizontal direction from the reference position than when is displayed. In particular, in this embodiment, the predetermined angle range is 10 degrees, and the threshold setting unit 57 sets the threshold time to be longer than when the specific display is displayed in an area with a viewing angle of less than 10 degrees in the horizontal direction from the reference position. , is set to be longer when the specific display is displayed in a viewing angle region that is 10 degrees abnormal in the horizontal direction from the reference position. That is, inside the predetermined angle range, the specific display is displayed at a position that is easy for the driver to see, so the line-of-sight movement time tends to be short. On the other hand, outside the predetermined angle range, the specific display is displayed at the edge of the driver's field of vision and at a position that is easy for the driver to see, so the line-of-sight movement time tends to be long. Therefore, by setting the threshold time to be longer when the specific display is displayed in an area outside the predetermined angle range than when the specific display is displayed in the area within the predetermined angle range, the driver's condition can be estimated. It is possible to further improve the accuracy of the calculation.

Further, in the first embodiment, the display device gradually makes the specific display more conspicuous until the threshold time elapses after the specific display is displayed, or until the line-of-sight direction of the driver turns to the specific display. change to As a result, the time required for estimating the driver's condition can be shortened as much as possible. As a result, even when the driver's physical condition is abnormal, it is possible to estimate the physical condition of the driver as soon as possible, and to quickly take measures such as stopping the vehicle on the shoulder of the road. become.

<Embodiment 2>
Hereinafter, Embodiment 2 will be described in detail with reference to the drawings. In the following description, the same reference numerals are assigned to the same parts as in the first embodiment, and detailed description thereof will be omitted.

FIG. 9 shows the configuration of a driver's state estimation device 20 according to the second embodiment. In Embodiment 2, the controller 50 further has a line-of-sight movement time calculator 254 that calculates the line-of-sight movement time of the driver. When the display control unit 52 causes the HUD 10 to display a specific display, the line-of-sight movement time calculation unit 254 calculates the time from when the specific display is displayed until the driver's line of sight turns to the specific display. The line-of-sight movement time calculator 254 detects the driver's saccade based on the detection result of the line-of-sight calculator 53 and calculates the line-of-sight movement time. A saccade is an eyeball movement (jumping eyeball movement) in which the driver intentionally shifts the line of sight, and is an eyeball movement that moves the line of sight from one gaze point to the next gaze point.

Further, in the second embodiment, the threshold setting unit 57 is configured to set the threshold time in consideration of the saliency of the specific display when the specific display is displayed. As described in the first embodiment, even with the same specific display, the saliency of the specific display varies depending on the external environment. When the specific display is displayed and the saliency of the specific display is low, the specific display is inconspicuous, so even if the driver is in a normal state, it takes time for the line of sight of the driver to turn to the specific display. Therefore, if the threshold time is set based only on the display position of the specific display, there is a risk that the driver may be erroneously assumed to be in an abnormal state even though the driver is in a normal state. In the second embodiment, the threshold setting unit 57 sets the threshold time so that when the saliency of the specific display is low when the specific display is displayed, it is longer than when the saliency of the specific display is high. By doing so, misjudgment is suppressed.

Here, various methods can be adopted for setting the threshold time by the threshold setting unit 57 . For example, the threshold setting unit 57 may store a map in which the threshold time is set based on the display position and saliency of the specific display, and read the map to set the threshold time. Further, for example, by calculating the temporary threshold time according to the display position of the specific display and correcting the temporary threshold time based on the saliency calculated by the attractiveness calculation unit 55, the final threshold time is good too.

Next, the processing operation for estimating the driver's state will be described with reference to the flowchart of FIG.

First, in step S201, the controller 50 acquires various data.

In the next step S202, the controller 50 determines whether the driving scene of the vehicle is in a specific state or a specific scene. When the driving scene is not a specific scene, the controller 50 returns. On the other hand, when the driving scene is the specific scene NO, the controller 50 proceeds to step S203.

In step S203, the controller 50 determines whether or not there is a dangerous object around the vehicle. The controller 50 returns when YES that there are no dangerous objects around the vehicle. On the other hand, the controller 50 proceeds to step S204 when the answer is NO that there is a dangerous object around the vehicle.

At step S204, the controller 50 determines whether or not the driver of the vehicle has seen a dangerous object. This determination is made based on whether or not the line-of-sight direction of the driver calculated by the line-of-sight calculation unit 53 has turned toward the dangerous object within a preset set time. If YES that the driver has seen the dangerous object within the set time, the controller 50 proceeds to step S214. Sometimes, the process proceeds to step S205. When the dangerous object is behind the own vehicle, the controller 50 determines that the driver has seen the dangerous object when the line of sight of the driver looks at the rearview mirror 5 or the side mirror 6. It's becoming

At step S205, the controller 50 causes the HUD 10 to display a specific display on the front window glass 1. FIG.

In the next step S206, the controller 50 determines whether or not the driver has seen the specific display. This determination is made based on whether or not the line-of-sight direction of the driver calculated by the line-of-sight calculation unit 53 faces the specific display. If YES that the driver is looking at the specific display, the controller 50 proceeds to step S208. If NO, the controller 50 proceeds to step S207.

In step S207, the controller 50 changes the display method of the specific display so as to increase the saliency of the specific display.

In the next step S208, the controller 50 determines whether or not the elapsed time since displaying the specific display is earlier than a predetermined time. This predetermined time is a value that is sufficiently larger than a threshold time that will be set later, and is a time during which it can be determined that the driver has lost consciousness. When the elapsed time is before the predetermined time (YES), the controller 50 returns to step S106. On the other hand, when the elapsed time exceeds the predetermined time and NO, the controller 50 proceeds to step S213.

In step S209 to which the controller 50 proceeds when the determination in step S206 is YES, the controller 50 calculates the line-of-sight movement time until the line-of-sight direction of the driver faces the specific display.

In subsequent step S210, the controller 50 calculates the saliency of the specific display when the specific display is displayed.

In the next step S211, the controller 50 calculates a threshold time for estimating the abnormality of the driver based on the position of the specific display displayed in step S205 and the saliency of the specific display calculated in step S210. .

In subsequent step S212, the controller 50 determines whether or not the line-of-sight movement time calculated in step S209 is greater than or equal to the threshold time calculated in step S211. When the line-of-sight movement time is equal to or longer than the threshold time (YES), the controller 50 proceeds to step S213. When the line-of-sight movement time is less than the threshold time, NO, the controller 50 proceeds to step S214.

At step S213, the controller 50 estimates that the driver is in an abnormal state. After step S213, the process returns.

At step S214, the controller 50 estimates that the driver is in a normal state. After step S214, the process returns.

Also in the second embodiment, the controller 50 estimates the depth of the abnormal state when it is estimated that the driver is in an abnormal state. In Embodiment 2, the controller 50 estimates that the driver is in a careless state, for example, when the line-of-sight movement time slightly exceeds the threshold time. Further, the controller 50 estimates that the driver's arousal level is lowered when the line-of-sight movement time relatively greatly exceeds the threshold time (shorter than the predetermined time). This is because if the driver is in a careless state, the concentration of the driver is reduced, and the driver's consciousness is clear, and it does not take much time to notice the specific display. . On the other hand, when the driver's arousal level is low, in addition to the driver's concentration being low, drowsiness tends to close the eyelids, making it difficult to notice the specific display. Further, the controller 50 presumes that the driver is at fault when a predetermined period of time has elapsed without the driver seeing the specific display. This is because it is estimated that the driver cannot notice the specific display.

<Other embodiments>
The technology disclosed herein is not limited to the above-described embodiments, and substitutions are possible without departing from the scope of the claims.

For example, in the above-described embodiment, the threshold setting unit 57 sets the threshold time in consideration of the saliency of the specific display, but the saliency of the specific display does not necessarily have to be considered.

Further, in the above-described embodiment, the specific display is performed when the vehicle is running in a specific scene and there is a dangerous object. Not limited to this, in order to call the driver's attention, a specific display may be performed if there is a dangerous object even if it is not in a specific scene, or if there is no dangerous object in a specific scene, a specific display may be performed. You may make it display. In particular, when the running state of the vehicle is a specific scene, there are few targets for estimating the driver's state, so the specific display may be positively performed to estimate the driver's state.

The above-described embodiments are merely examples, and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the scope of claims, and all variations and modifications within the equivalence range of the claims are within the scope of the present disclosure.

The technology disclosed herein is useful as a driver state estimation device that estimates the state of the driver based on the line-of-sight movement of the driver of the vehicle.

1 front window glass 10 head-up display (display device)
20 Driver state estimation device
50 controller 52 display control unit (display device)
53 line-of-sight calculation unit (line-of-sight detection device)
56 estimator
57 Threshold setting unit 101 Vehicle interior camera (line-of-sight detection device)

Claims (5)

A driver state estimating device for estimating the state of a driver based on the movement of the line of sight of the driver of the vehicle,
a line-of-sight detection device for detecting the line-of-sight direction of the driver;
a display device that performs a specific display according to a driving scene of the vehicle in the line-of-sight area of the driver when the vehicle is driving;
An estimating unit for estimating that an abnormality has occurred in the driver when the line-of-sight direction of the driver does not face the specific display within a threshold time after the display device displays the specific display. and,
A driver's state estimation device, comprising: a threshold setting unit that sets the threshold time according to a position where the specific display is displayed. In the driver state estimation device according to claim 1,
The threshold value setting unit sets the threshold time to the time when the specific display is displayed in a region within a predetermined angle range in the horizontal direction from the reference position when the line of sight of the driver faces the front. Also, the driver's state estimating device is set so that the specific display is longer when the specific display is displayed in a region outside the predetermined angular range in the horizontal direction from the reference position. In the driver state estimation device according to claim 2,
The predetermined angle range is 10 degrees,
The threshold setting unit sets the threshold time to a viewing angle that is 10 degrees abnormal in the horizontal direction from the reference position than when the specific display is displayed in a region with a viewing angle of less than 10 degrees in the horizontal direction from the reference position. A driver's state estimating device characterized in that it is set to be longer when the specific display is displayed in the area of . In the driver state estimation device according to any one of claims 1 to 3,
The display device gradually changes the specific display so that it stands out until the threshold time elapses after displaying the specific display, or until the line of sight of the driver turns to the specific display. A driver state estimation device characterized by: In the driver state estimation device according to any one of claims 1 to 4,
The line-of-sight area is an area of the front window glass of the vehicle,
The driver's state estimation device, wherein the display device is a head-up display that projects a virtual image on the vehicle front side of the front window glass.

Images