JP2021077139A - Driver state estimating device - Google Patents

Abstract

To provide a driver state estimating device which improves an accuracy of estimating a driver's state when a vehicle is running.SOLUTION: A driver state estimating device 20 is comprised of: an image processing unit 52 which synthesizes an image showing a front pillar trim with an image taken by a camera 100 and outputs the synthesized image data; an eye-catching degree calculation unit 53 which calculates an eye-catching degree indicating an ease of attracting a driver's line of sight with respect to a target shown in the synthesized image data, and generates an eye-catching degree map; and an estimating unit 54 which estimates a state of the driver based on the driver's line-of-sight direction and the eye-catching degree map. The eye-catching degree calculation unit calculates an eye-catching degree in a region near the front pillar trim in consideration of an inclination angle of the front pillar trim with respect to a vertical direction when calculating the eye-catching degree.SELECTED DRAWING: Figure 3

Description

The technology disclosed herein belongs to the technical field relating to the driver state estimation device.

Recently, the development of autonomous driving systems is being promoted nationally. At present, the applicant of the present application considers that the autonomous driving system has two main directions.

The first direction is a system in which the automobile takes the lead in transporting the occupants to the destination without requiring the driver's operation, which is the so-called fully automatic driving of the automobile. On the other hand, the second direction is an automatic driving system that is premised on human driving, such as wanting to enjoy driving a car.

In the second directional automatic driving system, for example, when a driver has a disease or the like and a situation occurs in which normal driving is difficult, the automobile automatically changes to a occupant and automatically drives. is assumed. For this reason, how quickly and accurately it is possible to detect the occurrence of an abnormality in the driver, especially the occurrence of a dysfunction or disease in the driver, is the improvement of the lifesaving rate of the driver and the safety including the surroundings. It is extremely important from the viewpoint of ensuring.

As a method of estimating an abnormality of a driver, for example, as in Patent Document 1, an attempt is made to estimate the state of the driver based on the movement of the line of sight of the driver of the vehicle. Specifically, in Patent Document 1, the driver's line-of-sight direction and the image captured by the imaging means for imaging the vicinity of the own vehicle are acquired, and each of the plurality of points on the object existing in the vicinity of the own vehicle is obtained. Based on the acquisition means for acquiring three-dimensional position information that specifies the position with respect to the own vehicle, the conspicuity calculation means for calculating the conspicuity of the driver's line of sight, and the conspicuity calculation means for calculating the conspicuity. A driver state determination device including a determination means for determining a driver's condition by comparing the degree to which the driver directs his / her line of sight to a conspicuous portion and a predetermined threshold value is disclosed.

Japanese Patent No. 5966640

However, there is a difference between the image acquired by the camera and the actual driver's field of view. That is, in general, the driver's field of vision includes vehicle components in addition to the external environment on the front side of the vehicle. As a result of diligent research by the inventors of the present application, it has been found that the place where the driver's line of sight is attracted changes in the driver's field of view depending on the presence or absence of these vehicle components. Therefore, if the degree of conspicuity or the like is simply calculated from the image taken by the camera, the accuracy of estimating the driver's state will be low.

The techniques disclosed herein improve the accuracy of estimating the driver's condition when the vehicle is running.

In order to solve the above-mentioned problems, in the technique disclosed here, the line-of-sight direction of the driver is targeted at a driver state estimation device that estimates the state of the driver based on the movement of the line-of-sight of the driver of the vehicle. A line-of-sight detection device that detects the above, a camera mounted on the vehicle that captures the external environment on the front side of the vehicle, and a front that enters the driver's field of view when the vehicle is traveling with respect to the image captured by the camera. An image processing unit that synthesizes an image showing the pillar trim and outputs the composite image data, and an attraction indicating the ease of attracting the driver's line of sight to the target on the front side of the vehicle based on the composite image data. An attraction degree calculation unit that calculates the degree and generates an attraction degree map, and an estimation unit that estimates the driver's state based on the driver's line-of-sight direction and the attraction degree map detected by the line-of-sight detection device. When calculating the attractiveness of the region near the front pillar trim, the attractiveness calculation unit calculates the attractiveness in consideration of the inclination angle of the front pillar trim with respect to the vertical direction in the composite image data. It was configured.

That is, according to the research by the inventors of the present application, it was found that the angle of the front pillar trim affects the degree of attraction. Specifically, it was found that when the tilt angle of the front pillar trim with respect to the vertical direction (the angle on the acute angle side, hereinafter referred to as the pillar tilt angle) is large, the degree of attraction in the region near the front pillar trim in the driver's field of vision is high. It was. When the pillar tilt angle is small, the target located in the vicinity of the front pillar trim is blocked by the front pillar trim almost at the same time when the vehicle moves forward. On the other hand, when the pillar inclination angle is large, the target visible in the vicinity of the front pillar trim is gradually blocked by the front pillar trim from the lower portion when the vehicle advances. As a result, when the pillar inclination angle is large, it is illusion that the target is moving to the rear side and the upper side. Therefore, when the pillar inclination angle is large, the target visually recognized in the vicinity of the front pillar trim tends to attract the driver's line of sight.

According to the above configuration, since the attractiveness is calculated in consideration of the inclination angle of the front pillar trim with respect to the vertical direction, the accuracy of calculating the attractiveness in the driver's field of view is improved. Therefore, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running.

It should be noted that the degree of attraction referred to here indicates the degree of conspicuity to the surroundings, and is not considered when a skilled driver consciously looks at it.

In the driver state estimation device, the attraction degree calculation unit may be configured such that when the inclination angle of the front pillar trim is large, the attraction degree is increased as compared with the case where the inclination angle is small.

That is, when the pillar inclination angle is large, it tends to attract the driver's line of sight. Therefore, according to the above-described configuration, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

In the driver state estimation device, when the angle formed by the attraction degree calculation unit between the longitudinal direction of the target located in the vicinity region and the longitudinal direction of the front pillar trim in the composite image data is large, the angle is small. It may be configured to increase the degree of attraction compared to the case.

That is, even if the pillar inclination angle is large, if the longitudinal direction of the target is substantially parallel to the longitudinal direction of the front pillar trim, when the vehicle moves forward, the entire target is blocked by the front pillar trim at substantially the same time. That is, if the angle between the longitudinal direction of the target and the longitudinal direction of the front pillar trim is small, it is difficult to give the illusion that the target is moving, and it is difficult to attract the driver's line of sight. Therefore, if the angle between the longitudinal direction of the target and the longitudinal direction of the front pillar trim is small, the degree of attraction is low. Therefore, according to the above-described configuration, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

In one embodiment of the driver state estimation device, the estimation unit is based on the line-of-sight frequency, which is the frequency at which the line-of-sight of the driver detected by the line-of-sight detection device is directed to a region where the degree of attraction is a predetermined value or more. The driver's condition may be estimated.

In particular, in the one embodiment, the estimation unit may be configured to estimate that an abnormality has occurred in the driver when the line-of-sight frequency is equal to or higher than a predetermined frequency.

That is, under normal conditions, the driver drives while looking at a parked vehicle, a pedestrian, a side road, or other visible object to be watched when the vehicle is running. However, when the driver is in a vague state or when the driver becomes ill and cannot actively drive, the driver naturally turns his / her gaze to the area where the degree of attraction is high. That is, when the driver frequently sees the area with a high degree of attraction, there is a high possibility that the driver has an abnormality. Therefore, if the driver's state is estimated based on the driver's line-of-sight frequency, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

As described above, according to the technology disclosed herein, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running. it can.

It is the schematic which shows the front side part in the vehicle interior of the vehicle equipped with the driver state estimation device which concerns on an exemplary Embodiment 1. It is a front view which looked at the vehicle from the front side. It is a block diagram which shows the structure of the driver state estimation device. It is a figure which shows the external environment of the front side of a vehicle photographed by a camera. It is a figure which shows the composite image data, and shows the case where the inclination angle with respect to the vertical direction of a front pillar trim is small. It is a figure which shows the composite image data, and shows the case where the inclination angle with respect to the vertical direction of a front pillar trim is large. It is a figure which shows the distribution of salience calculated based on the composite image data shown in FIG. 5A. It is a figure which shows the distribution of salience calculated based on the composite image data shown in FIG. 5B. It is a graph which shows the salience of the driver's gaze point. It is a graph which shows the salience of the point randomly extracted from the attraction degree map. It is a graph comparing the ratio exceeding the threshold value at a random point and the ratio exceeding the threshold value at the driver's gaze point. It is a flowchart which shows the processing operation of the estimation part. It is a figure which conceptually shows the calculation method of salience in the driver state estimation apparatus which concerns on Embodiment 2. FIG.

Hereinafter, exemplary embodiments will be described in detail with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. The left side when looking at the front side from the rear side is called the left side, and the opposite is called the right side.

<Embodiment 1>
FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and the steering wheel 8 is arranged on the right side.

In the vehicle interior, the front window glass 1 is arranged on the front side of the vehicle when viewed from the driver's seat. The front window glass 1 is partitioned by a plurality of vehicle components when viewed from the vehicle interior side. Specifically, the front window glass 1 is partitioned by the left and right front pillar trims 2, the roof trim 3, and the instrument panel 4.

The left and right front pillar trims 2 form boundaries on the outer side of the front window glass 1 in the vehicle width direction. Each front pillar trim 2 is arranged along each front pillar 12. As shown in FIG. 1, each front pillar trim 2 is arranged so as to be inclined obliquely so as to be separated from each other toward the upper side.

The roof trim 3 constitutes the upper boundary of the front window glass 1. The roof trim 3 covers the vehicle interior side of the roof panel 13 of the vehicle. A rear-view mirror 5 is attached to the center of the front window glass 1 in the vehicle width direction and slightly below the roof trim 3. A vehicle interior camera 101 (see FIG. 3) for photographing the vehicle interior, particularly the driver's face, is provided in the vicinity of the rear-view mirror 5 in the roof trim 3. The vehicle interior camera 101 is composed of a high-performance camera that can detect changes in the driver's pupil.

The instrument panel 4 constitutes the lower boundary of the front window glass 1. The instrument panel 4 is provided with a meter box and a display 7.

Further, the vehicle has side mirrors 6 on the outer side in the vehicle width direction with respect to the left and right front pillars 12, respectively. Each side mirror 6 is arranged so that the driver seated in the driver's seat can see through the window of the side door.

As shown in FIG. 2, the vehicle is provided with a camera 100 for photographing the external environment on the front side of the vehicle. The camera 100 is a front end portion of the vehicle and is arranged slightly below the bonnet 9 of the vehicle.

(Configuration of driver state estimation device)
The vehicle according to the first embodiment has a function of estimating the state of the driver based on the movement of the driver's line of sight. FIG. 3 shows the configuration of the driver state estimation device 20. The driver state estimation device 20 has a controller 50 having an estimation unit 54 for actually estimating the driver state. The controller 50 is a controller based on a well-known microcomputer, and uses a central processing unit (CPU) for executing a program and, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). It includes a memory that is configured to store programs and data, and an input / output bus that inputs and outputs electrical signals.

As shown in FIG. 3, the controller 50 is connected to various sensors. The sensors are, for example, a camera 100 that captures the external environment on the front side of the vehicle, a vehicle interior camera 101 that captures the driver's face, and a sweating sensor 102 that is provided on the steering wheel 8 and detects the sweating state of the driver's hand. And include.

The controller 50 has a line-of-sight calculation unit 51 that calculates the driver's line-of-sight direction from the driver's eyeball photographed by the vehicle interior camera 101. The line-of-sight calculation unit 51 calculates the line-of-sight direction of the driver by detecting a change in the driver's pupil from the state in which the driver looks into the lens of the vehicle interior camera 101, for example. The line-of-sight direction may be calculated from either the driver's left eye or the right eye, or may be the average value of the line-of-sight directions (line-of-sight vectors) obtained from each of the driver's eyes. Further, when it is difficult to calculate the driver's line-of-sight direction from the change in the driver's pupil, the line-of-sight direction may be calculated in consideration of the direction of the driver's face. The vehicle interior camera 101 and the controller 50 are examples of the line-of-sight detection device.

The controller 50 has an image processing unit 52 that performs image processing for synthesizing an image showing the front pillar trim on the image D1 taken by the camera 100 and outputs the composite image data D2. The image processing executed by the image processing unit 52 will be described later.

Based on the composite image data D2, the controller 50 calculates the degree of attraction indicating the ease with which the driver's line of sight is attracted to the target on the front side of the vehicle, and generates the degree of attraction map D3. Has 53. In the first embodiment, the attraction degree calculation unit 53 calculates salience (visual prominence) as the attraction degree. Salience is a visual feature amount consisting of prominence that changes from moment to moment depending on color, brightness, movement, and the like. That is, a high salience means a high degree of prominence, and a high salience area is a area that stands out with respect to the surroundings. More specifically, the salience is high in the region where the color difference or the luminance difference is large with respect to the surrounding region or the movement is occurring with respect to the surrounding region.

The controller 50 operates based on the attraction degree map D3, the driver's line-of-sight direction calculated by the line-of-sight calculation unit 51, the detection result of the sweating sensor 102, and the change in the driver's pupil diameter photographed by the vehicle interior camera 101. It has an estimation unit 54 for estimating the state of a person. As will be described in detail later, the estimation unit 54 estimates whether the driver is in a normal state or an abnormal state from the calculation results of the attraction degree map D3 and the line-of-sight calculation unit 51. When the driver is presumed to be in an abnormal state, the estimation unit 54 determines the driver's state from the detection result of the sweating sensor 102 and the change in the driver's pupil diameter photographed by the vehicle interior camera 101. Estimate in detail. The driver's abnormal state is not limited to the state in which the driver is in an abnormal physical condition, but also includes the driver's absent-minded state and the state in which the driver's arousal level is reduced (drowsiness is increased). ..

The controller 50 controls various actuators A, a display 7, a buzzer B, etc. mounted on the vehicle based on the estimation result of the estimation unit 54. The various actuators A include actuators constituting the engine system, the brake system, and the steering system.

(Image processing by the image processing unit)
Next, the image processing by the image processing unit 52 will be described. The image processing unit 52 synthesizes the image D1 showing the external environment on the front side of the vehicle taken by the camera 100 with the image showing the front pillar trim that enters the driver's field of view when the vehicle is running, and synthesizes the composite image data D2. Performs the process of generating. Further, the image processing unit 52 performs processing on the image D1 captured by the camera 100 to delete pixels unnecessary for processing (calculation of salience) by the attraction degree calculation unit 53 among the elements constituting the image D1. ..

FIG. 4 is an image D1 showing the external environment on the front side of the vehicle taken by the camera 100. The external environment shown in the image D1 includes a roadway 150 and a white line 151 on the roadway 150. Further, in the external environment shown in this image D1, the other vehicle 161 and the wall 162 formed on the right side of the roadway 150, the trees 163 planted in the area on the right side of the wall 162, and the trees 163 It includes hills 164 extending to the area on the right and forests 165 formed on the left side of the roadway 150. Further, the external environment shown in the image D1 includes the building 166 and the sky 167 extending above the building 166 and the forest 165. In the following description, it is assumed that the sky 167 of the image D1 is a cloudy sky and is substantially white.

In the first embodiment, the image processing unit 52 synthesizes an image showing the front pillar trim 2 that enters the driver's field of view when the vehicle is traveling with the image D1. Actually, in addition to the front pillar trim 2, the image processing unit 52 also synthesizes an image showing a vehicle component that enters the driver's field of view when the driver seated in the driver's seat looks at the front side of the vehicle. Specifically, as shown in FIGS. 5A and 5B, the image processing unit 52 includes the front pillar trim 2 on the left side (driver's seat side) and the roof when the driver seated in the driver's seat looks at the front side of the vehicle. An image showing the left part of the trim 3, the left part of the instrument panel 4, the left side mirror 6, the steering wheel 8, and a part of the bonnet 9 is combined with the image D1. The image for compositing can be, for example, photographed in advance from the driver's seat side, and the photographed data can be stored in the memory of the controller 50 as a layer. Then, when synthesizing an image for compositing with the image D1, the image processing unit 52 may read the image from the memory and superimpose the image on the image D1 as a layer.

As shown in FIGS. 5A and 5B, in the composite image, a part of the image D1 is blocked by the vehicle component. The image processing unit 52 outputs the composite image data D2 indicating the composite image to the attraction degree calculation unit 53.

(Generation of attraction map)
The attraction degree calculation unit 53 calculates the salience with respect to the composite image data D2 and generates an attraction degree map (here, a salience map). Specifically, the attractiveness calculation unit 53 calculates salience for a portion of the composite image data D2 that represents the external environment, that is, a portion other than the image synthesized by the image processing unit 52. At this time, the attraction degree calculation unit 53 does not calculate the salience itself for the part of the image (the part of the vehicle component) synthesized by the image processing unit 52, but the salience of the part of the composite image data D2 representing the external environment. It is used to calculate. That is, the attraction degree calculation unit 53 calculates the attraction degree in consideration of the vehicle constituent members that enter the driver's field of view when the vehicle is traveling.

As described above, salience changes depending on the color, brightness, movement, etc. of the target. Therefore, in the first embodiment, the attraction degree calculation unit 53 calculates the salience for each feature such as color-based salience, brightness-based salience, and motion-based salience, and generates an attraction degree map for each feature. , The final attraction map D3 is generated by adding them together.

For example, regarding color-based salience, the attraction degree calculation unit 53 compares when the color difference between the neighborhood region and the vehicle component is large in the region near the vehicle component in the composite image data D2, as compared with when the color difference is small. Then, the salience of the neighboring region is increased. The color difference is calculated by the following formula when the RGB of the color of a certain pixel is (R1, G1, B1) and the RGB of the color of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The attractiveness calculation unit 53 may calculate so that the larger the color difference, the higher the salience continuously, or may set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. Good.

Further, for example, regarding salience based on brightness, the attractiveness calculation unit 53 determines that when the brightness difference between the neighborhood region and the vehicle constituent member is large in the vicinity region of the vehicle constituent member in the composite image data D2, the brightness difference is large. Increase the salience of the neighboring area compared to when it is small. For example, when the combined vehicle component is black, the attractiveness calculation unit 53 increases the salience of the portion of the vicinity region that is close to white because the brightness difference becomes larger as the portion is closer to white. The attractiveness calculation unit 53 may calculate so that the larger the difference in brightness, the higher the salience continuously, or set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. May be good.

In particular, in the first embodiment, when calculating the salience based on the movement, the angle α on the acute angle side of the front pillar trim 2 with respect to the vertical direction in the longitudinal direction (hereinafter referred to as the pillar inclination angle α) is taken into consideration. Specifically, when the pillar inclination angle α is large, the attraction degree calculation unit 53 increases the attraction degree in the vicinity region of the front pillar trim 2 as compared with the case where the pillar inclination angle α is small. In the first embodiment, the pillar inclination angle α is the inclination angle of the boundary portion of the front pillar trim 2 with the front window glass 1, as shown in FIGS. 5A and 5B. Since the extending direction of the boundary portion substantially coincides with the longitudinal direction of the front pillar trim 2, there is no particular problem.

When the pillar inclination angle α is small, the entire target is blocked by the front pillar trim 2 substantially at the same time in the driver's field of view when the vehicle is moving forward. In this case, the target does not move in particular.

On the other hand, when the pillar inclination angle α is large, the target is gradually blocked by the front pillar trim 2 from the lower portion in the driver's field of view when the vehicle moves forward. This gives the driver the illusion that the target is moving backwards and downwards. In other words, the target itself is not moving, but the target appears to be moving in the driver's field of view. Therefore, the area near the front pillar trim 2 becomes more conspicuous than other parts. For this reason, when the pillar inclination angle α is large, the degree of attraction in the vicinity region to the front pillar trim 2 is increased.

The attractiveness calculation unit 53 may calculate so that the larger the pillar inclination angle α is, the higher the salience is continuously. Alternatively, a plurality of threshold values are provided, and the salience is increased each time the pillar inclination angle α exceeds each threshold value. It may be calculated so as to be higher by a certain value. In particular, when the pillar inclination angle α is larger than the first predetermined angle, the salience may be calculated so that the larger the pillar inclination angle α is from the first predetermined angle, the higher the salience.

As described above, the image for compositing can be an image taken from the driver's seat in advance. Therefore, the pillar inclination angle α may be stored in the memory as a value calculated in advance from the image for compositing.

For the calculation of salience itself, a known computer program published on the Internet or the like can be used. In addition, a known computer program can be used for calculating the attraction map for each feature (here, the salency map) and integrating each attraction map.

FIG. 6A is an attraction map D3 showing the distribution of salience when the pillar inclination angle α is small. As shown in FIG. 6A, the portion of the sky 166 separated by the roof trim 3 and the forest 164 is a portion having particularly high salience. This is because when the pillar inclination angle α is small, the color difference and the brightness difference greatly affect the salience. That is, the portion of the sky 166 partitioned by the roof trim 3 and the forest 164 has a large color difference and brightness difference between the roof trim 3 and the forest 164, and thus has high salience.

On the other hand, FIG. 6B is an attraction map D3 showing the distribution of salience when the pillar inclination angle α is large. As shown in FIG. 6B, in addition to the portion partitioned by the roof trim 3 and the forest 164 in the sky 166, the portion near the front pillar trim 2 is a portion having high salience. This is because when the pillar inclination angle α is large, movement is added to the vicinity portion of the front pillar trim 2, particularly the boundary portion, as an element for enhancing salience, in addition to the color difference and the brightness difference.

In this way, by calculating the salience in consideration of the pillar inclination angle α as in the first embodiment, it is possible to accurately calculate the region where the driver is likely to attract the line of sight when the vehicle is running. ..

(Calculation of line-of-sight frequency)
Next, the calculation of the line-of-sight frequency will be described with reference to FIGS. 7 to 9. In this embodiment, the estimation unit 54 calculates the line-of-sight frequency.

FIG. 7 is a plot of the driver's gaze point, that is, the salience of a point beyond the driver's line of sight. In this graph, the line-of-sight calculation unit 51 determines the line-of-sight direction of the driver, applies the gaze point ahead of the line-of-sight direction to the attraction degree map D3, and obtains the salience of the gaze point from the attraction degree map D3. Is generated by. Each of these points is obtained at a predetermined sampling cycle.

On the other hand, FIG. 8 is generated by randomly designating coordinates from the attractiveness map D3 generated by the attractiveness calculation unit 53 and obtaining the salience of the coordinates. Each of these points is obtained at a predetermined sampling cycle.

After creating the graphs of FIGS. 7 and 8, respectively, the ROC (Receiver Operating Characteristic) curve of the ratio exceeding the threshold value at the random point and the ratio exceeding the threshold value at the driver's gaze point is obtained. Specifically, first, as a first step, the threshold value is set to a value larger than the maximum value in the graphs of FIGS. 7 and 8. Next, as a second step, while lowering the threshold value, the ratio of points exceeding the threshold value is obtained for each threshold value. Then, the process of this second step is repeated until the threshold value becomes equal to or less than the minimum value in the graphs of FIGS. 7 and 8.

The graphs shown in FIGS. 7 and 8 are generated by dividing them into predetermined measurement cycles. The measurement cycle can be arbitrarily set, for example, 5 seconds.

After the first and second steps, as the third step, the ratio exceeding the threshold value at the random point (referred to as the first probability) is taken on the horizontal axis and the ratio exceeding the threshold value at the driver's gaze point (referred to as the second probability). For the vertical axis, a graph as shown in FIG. 9 is created. The graph of FIG. 9 shows the gaze probability with respect to the random probability at the same threshold. In the graph of FIG. 9, since both the horizontal axis and the vertical axis are probabilities, the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 9, the broken line indicates that the driver's line of sight is evenly directed to the entire field of view. That is, if the driver's line of sight extends over the entire viewing angle region, the first probability and the second probability do not deviate significantly. Therefore, if the driver's line of sight extends over the entire viewing angle region, the linear shape as shown by the broken line in FIG. 9 is obtained when the first probability and the second probability are compared. On the other hand, when the curve expands convexly upward as shown by the curve C1 in FIG. 9, it means that the driver's gaze point has a high ratio of exceeding the threshold value. That is, when a curve having a shape like the curve C1 in FIG. 9 is calculated, it means that the driver's line of sight tends to see a portion having high salience. On the other hand, when the curve spreads downwardly as shown by the curve C2 in FIG. 9, it means that the driver's gaze point has a low ratio of exceeding the threshold value. That is, when a curve having a shape like the curve C2 in FIG. 9 is calculated, it means that the driver's line of sight tends to see a portion having low salience.

In the present embodiment, the AUC (Area Under Curve) is obtained as the fourth step. AUC is the integrated value of the lower right part of this curve (that is, the area of the part surrounded by the curve and the alternate long and short dash line in FIG. 9). In the present embodiment, this integrated value is used as the line-of-sight frequency. In this way, the accuracy of calculating the line-of-sight frequency can be improved by comparing the salience of the part gazed by the driver with the salience of the coordinates randomly specified from the attraction degree map D3 and calculating the line-of-sight frequency. .. That is, when the region having high salience spreads over the entire map in the attraction degree map D3, the driver sees the portion having high salience no matter which position he / she looks at. In this case, simply calculating the ratio of seeing salience above the threshold value will calculate that the line-of-sight frequency is high even if the driver looks at the entire field of view without bias. On the other hand, if the comparison with the random points is performed as in the present embodiment, even if the area with high salience extends over the entire map, the entire field of view of the driver is randomly viewed. Can be compared if they have the same tendency as. Therefore, it is possible to more accurately determine whether or not there is a bias in the line-of-sight direction of the driver.

(Estimation of driver status)
Next, the processing operation of estimating the driver state by the estimation unit 54 will be described with reference to the flowchart of FIG.

First, in step S101, the estimation unit 54 acquires various data. In particular, the estimation unit 54 includes the calculation result of the line-of-sight calculation unit 51, the attraction degree map D3 generated by the attraction degree calculation unit 53, the image data of the driver's face acquired by the vehicle interior camera 101, and the sweating sensor 102. Acquire the detection result.

In the next step S102, the estimation unit 54 calculates the line-of-sight frequency. The estimation unit 54 calculates the line-of-sight frequency by the calculation method as described above.

In the following step S103, the estimation unit 54 estimates whether or not the driver's attention is reduced. Specifically, the estimation unit 54 estimates that the driver's attention is reduced when the line-of-sight frequency calculated in step S102 is equal to or higher than a predetermined frequency. The estimation unit 54 proceeds to step S104 when the line-of-sight frequency is equal to or higher than a predetermined frequency and YES is estimated to reduce the driver's attention. On the other hand, the estimation unit 54 proceeds to step S106 when the line-of-sight frequency is less than the predetermined frequency and the driver's attention is presumed to be NO.

In step S104, the estimation unit 54 estimates whether or not the driver has a predominant sympathetic nerve. Specifically, the estimation unit 54 estimates that the sympathetic nerve is in a predominant state when the amount of perspiration per unit time detected by the perspiration sensor 102 is equal to or greater than a predetermined amount. On the other hand, the estimation unit 54 estimates that the parasympathetic nerve is in a predominant state when the amount of sweating per unit time detected by the sweating sensor 102 is less than a predetermined amount. The estimation unit 54 proceeds to step S105 when the detection result of the sweating sensor 102 is equal to or greater than a predetermined amount and YES is presumed that the sympathetic nerve is dominant. On the other hand, the estimation unit 54 proceeds to step S107 when the detection result of the sweating sensor 102 is less than a predetermined amount and YES, which is presumed that the parasympathetic nerve is dominant.

In step S105, the estimation unit 54 estimates whether or not the driver is actively driving. Specifically, the estimation unit 54 estimates from the change in the pupil diameter of the driver photographed by the vehicle interior camera 101. More specifically, the estimation unit 54 estimates that the driver is actively driving when there is almost no change in the pupil diameter before and after the movement of the driver's line of sight. On the other hand, the estimation unit 54 estimates that the driver is passively driving when the pupil diameter changes before and after the driver's line-of-sight movement. This is because when the driver is driving passively, the line of sight is moved after being stimulated by the eyes. The estimation unit 54 proceeds to step S109 when YES is presumed that the driver is actively driving. On the other hand, the estimation unit 54 proceeds to step S108 when the driver is presumed to be passively driving NO. Even if the convergence angle, which is the intersection angle between the driver's left eye line of sight and the right eye line of sight, is calculated and it is estimated whether or not the driver is actively driving based on the convergence angle. Good.

In step S106, the estimation unit 54 estimates that the driver is in a normal state. That is, if the driver has some abnormality, the driver's attention is reduced. Therefore, if the driver does not have a decrease in attention, it is presumed that the driver is in a normal state, but if the driver has a decrease in attention, an abnormality occurs in the driver. It can be estimated that it is. After step S106, it returns.

In step S107, the estimation unit 54 estimates that the driver's alertness is reduced (sleepiness is increased). In general, a parasympathetic predominant state predominates over the sympathetic nerve during sleep. Therefore, it can be presumed that the state in which the attention is lowered and the parasympathetic nerve is dominant is the state in which the driver's alertness is lowered. After step S107, it returns.

In step S108, the estimation unit 54 estimates that the driver is in a loose state. That is, since the driver is passively driving while being awake, it can be presumed that the driver is in a distracted state while thinking about something other than driving. After step S108, it returns.

In step S109, the estimation unit 54 estimates that the driver is in an abnormal physical condition. That is, the driver tries to continue normal driving even if the physical condition is abnormal. For this reason, if the driver is awake during normal driving and the driver's attention is reduced despite being actively driving, some abnormality may occur in the driver's physical condition. It can be estimated that there is. Return after step S109.

As described above, the driver's state is estimated by the estimation unit 54. When it is estimated that the driver's alertness is low (step S107) or the driver is in a vague state (step S108), the controller 50 sounds, for example, a buzzer B or displays it on the display 7. Or As a result, the controller 50 encourages the driver to awaken or concentrate on driving. Further, when the estimation unit 54 estimates that the driver is in an abnormal physical condition, the controller 50 outputs control signals to various actuators A, for example, to drive the vehicle to a safe area such as a shoulder. , Stop the vehicle in the safe area.

Therefore, in the first embodiment, the line-of-sight detection device (vehicle interior camera 101 and line-of-sight calculation unit 51) for detecting the driver's line-of-sight direction, the camera 100 mounted on the vehicle and photographing the external environment on the front side of the vehicle, and the camera. The image processing unit 52 that outputs the composite image data D2 by synthesizing the image showing the front pillar trim 2 that enters the driver's field of view when the vehicle is traveling with the image D1 taken by the 100, and the composite image data D2. Based on the above, the attraction degree calculation unit 53 that calculates the attraction degree indicating the ease of attracting the driver's line of sight to the target on the front side of the vehicle and generates the attraction degree map D3, and the line-of-sight detection device It is provided with an estimation unit 54 that estimates the driver's state based on the detected driver's line-of-sight direction and attraction degree map D3, and when the attraction degree calculation unit 53 calculates the attraction degree in the vicinity region of the front pillar trim 2. , The degree of attraction is calculated in consideration of the inclination angle (pillar inclination angle α) of the front pillar trim 2 in the composite image data D2 with respect to the vertical direction.

In particular, in the first embodiment, the attractiveness calculation unit 53 increases the attractiveness when the pillar inclination angle α of the front pillar trim 2 is large as compared with the case where the pillar inclination angle α is small.

That is, when the pillar inclination angle α is small, the target located in the vicinity of the front pillar trim 2 is blocked by the front pillar trim 2 substantially at the same time when the vehicle moves forward. On the other hand, when the pillar inclination angle α is large, the target visually recognized in the vicinity region of the front pillar trim 2 is gradually blocked by the front pillar trim 2 from the lower portion when the vehicle moves forward. As a result, when the pillar inclination angle α is large, it is illusion that the target is moving backward and downward. Therefore, when the pillar inclination angle α is large, the target visually recognized in the vicinity of the front pillar trim 2 tends to attract the driver's line of sight. Therefore, by calculating the degree of attraction in consideration of the pillar inclination angle α, the accuracy of calculating the degree of attraction in the driver's field of view is improved. Therefore, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running.

Further, in the present embodiment, the estimation unit 54 determines when the line-of-sight frequency detected by the line-of-sight detection device is the frequency at which the driver's line-of-sight is directed to a region where the degree of attraction is a predetermined value or more. , It is estimated that the driver has an abnormality. That is, under normal conditions, the driver drives while looking at a parked vehicle, a pedestrian, a side road, or other visible object to be watched when the vehicle is running. However, when the driver is in a vague state or when the driver becomes ill and cannot actively drive, the driver naturally turns his / her gaze to the area where the degree of attraction is high. That is, when the driver frequently sees the area with a high degree of attraction, there is a high possibility that the driver has an abnormality. Therefore, if the driver's state is estimated based on the driver's line-of-sight frequency, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

<Embodiment 2>
Hereinafter, the second embodiment will be described in detail with reference to the drawings. In the following description, the parts common to the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted.

In the second embodiment, the method of calculating the salience by the attractiveness calculation unit 53 is different from that of the first embodiment. Specifically, in the second embodiment, the attractiveness calculation unit 53 determines the angle β (hereinafter, relative angle) formed by the longitudinal direction of the target located in the vicinity region in the composite image data D2 and the longitudinal direction of the front pillar trim 2. When β) is large, the salience is increased as compared with when the relative angle β is small. That is, if the longitudinal direction of the target is substantially parallel to the longitudinal direction of the front pillar trim 2, when the vehicle moves forward, the entire target is blocked by the front pillar trim 2 at substantially the same time. That is, if the relative angle β is small, it is difficult to give the illusion that the target is moving, and it is difficult to attract the driver's line of sight. Therefore, the smaller the relative angle β, the lower the salience. Therefore, by considering the longitudinal direction of the target, the salience in the driver's field of view can be calculated more accurately.

Specifically, as shown in FIG. 11, the attraction degree calculation unit 53 first calculates the longitudinal direction of the target from the composite image data D2. Next, the attractiveness calculation unit 53 calculates the relative angle β formed between the longitudinal direction of the target and the longitudinal direction of the front pillar trim 2. Then, the attractiveness calculation unit 53 calculates salience according to the relative angle β. In the second embodiment as well, as in the first embodiment, the longitudinal direction of the front pillar trim 2 is the direction in which the boundary portion of the front pillar trim 2 with the front window glass 1 extends.

The attractiveness calculation unit 53 may calculate so that the larger the relative angle β is, the higher the salience is continuously. Alternatively, a plurality of threshold values are provided, and the salience is a constant value each time the relative angle β exceeds each threshold value. It may be calculated to be higher. In particular, when the relative angle β is larger than the second predetermined angle, the salience may be calculated so that the larger the relative angle β is from the second predetermined angle, the higher the salience.

In this way, by considering the relative angle β between the target and the front pillar trim 2, the degree of attraction (here, salience) in the driver's visual field can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

The attraction degree calculation unit 53 may calculate the inclination angle (hereinafter, referred to as the target inclination angle) with respect to the vertical direction in the longitudinal direction of the target after calculating the longitudinal direction of the target. In this case, the attraction degree calculation unit 53 calculates the difference between the pillar inclination angle α and the target inclination angle when the target is tilted toward the front pillar trim 2, while the target is on the opposite side of the front pillar trim 2. When tilted to, the sum of the pillar tilt angle α and the target tilt angle is calculated. Then, the attractiveness calculation unit 53 calculates salience by using the calculated value as a relative angle β formed by the longitudinal direction of the target and the longitudinal direction of the front pillar trim 2.

<Other Embodiments>
The technique disclosed herein is not limited to the above-described embodiment, and can be substituted as long as it does not deviate from the gist of the claims.

For example, in the above-described first and second embodiments, the attractiveness calculation unit 53 calculates the attractiveness in consideration of the pillar inclination angle α and the relative angle β between the front pillar trim 2 and the target. In addition to this, the attraction degree calculation unit 53 may further consider the vehicle speed. That is, when the pillar inclination angle α or the relative angle β is large, the moving speed of the target in the driver's field of view changes depending on the vehicle speed, so that the degree of attraction also changes depending on the vehicle speed. Therefore, for example, when the pillar inclination angle α and the relative angle β are large, the attractiveness calculation unit 53 may increase the attractiveness as the vehicle speed increases.

Further, in the above-described first and second embodiments, as the longitudinal direction of the front pillar trim 2, the direction in which the boundary portion of the front pillar trim 2 with the front window glass 2 extends is adopted. Not limited to this, the direction in which the intermediate portion of the front pillar trim 2 in the vehicle width direction extends may be the longitudinal direction of the front pillar trim 2.

Further, in the above-described first and second embodiments, the front pillar trim 2 on the left side (driver's seat side), the left side portion of the roof trim 3, and the instrument panel 4 are used as vehicle components to be considered when calculating the degree of attraction. The left side part, the left side mirror 6, the steering wheel 8, and a part of the bonnet 9 are adopted. Not limited to this, at least the front pillar trim 2 on the left side (driver's seat side) located on the peripheral edge of the front window glass 1 may be considered.

Further, in the above-described first and second embodiments, the line-of-sight frequency is calculated by the estimation unit 54, but the present invention is not limited to this, and a processor for calculating the line-of-sight frequency may be provided separately from the estimation unit 54.

The above embodiments are merely examples, and the scope of the present disclosure should not be construed in a limited manner. The scope of the present disclosure is defined by the scope of claims, and all modifications and changes belonging to the equivalent scope of the scope of claims are within the scope of the present disclosure.

The technique disclosed herein is useful as a driver state estimation device that estimates the driver's state based on the movement of the driver's line of sight of the vehicle.

2 Front pillar trim 20 Driver state estimation device
50 controller
51 Line-of-sight calculation unit (line-of-sight detection device)
52 Image processing unit
53 Invitation degree calculation unit
54 Estimator
100 cameras
101 Interior camera (line-of-sight detection device)
D1 image D2 composite image data D3 attraction map α pillar tilt angle (tilt angle of front pillar trim with respect to vertical direction)
β Relative angle (angle between the longitudinal direction of the target and the longitudinal direction of the front pillar trim)

Claims (5)

It is a driver state estimation device that estimates the state of the driver based on the movement of the line of sight of the driver of the vehicle.
A line-of-sight detection device that detects the driver's line-of-sight direction,
A camera mounted on the vehicle that captures the external environment on the front side of the vehicle,
An image processing unit that combines an image taken by the camera with an image showing a front pillar trim that enters the driver's field of view when the vehicle is running, and outputs composite image data.
Based on the composite image data, an attraction degree calculation unit that calculates an attraction degree indicating the ease of attracting the driver's line of sight to a target on the front side of the vehicle and generates an attraction degree map.
It is provided with an estimation unit that estimates the state of the driver based on the line-of-sight direction of the driver detected by the line-of-sight detection device and the attraction degree map.
When calculating the degree of attraction in the region near the front pillar trim, the degree of attraction calculation unit calculates the degree of attraction in consideration of the inclination angle of the front pillar trim with respect to the vertical direction in the composite image data. Person state estimation device. In the driver state estimation device according to claim 1,
The driver state estimation device is characterized in that when the inclination angle of the front pillar trim is large, the degree of attraction is increased as compared with the case where the inclination angle is small. In the driver state estimation device according to claim 1,
When the angle formed by the longitudinal direction of the target located in the vicinity region and the longitudinal direction of the front pillar trim is large in the composite image data, the attractiveness calculation unit determines the attractiveness as compared with the case where the angle is small. A driver state estimator characterized by increasing the height. In the driver state estimation device according to any one of claims 1 to 3,
The estimation unit estimates the state of the driver based on the line-of-sight frequency at which the line-of-sight of the driver detected by the line-of-sight detection device is directed to a region where the degree of attraction is equal to or higher than a predetermined value. A characteristic driver state estimation device. In the driver state estimation device according to claim 4,
The estimation unit is a driver state estimation device that estimates that an abnormality has occurred in the driver when the line-of-sight frequency is equal to or higher than a predetermined frequency.

Images