JP2021077138A - Driver state estimating apparatus - Google Patents

Abstract

To provide a driver state estimating apparatus that improves accuracy in estimating the state of a driver during vehicle traveling.SOLUTION: A driver state estimating apparatus 20 comprises: a camera 100 as a visual line detection device that detects a visual line direction of a driver; a controller 50; an image processing unit 52 that performs image processing for an image taken by the camera and outputs image data; an eye-catching degree calculation unit 53 that calculates an eye-catching degree indicating easiness in attracting the visual line of the driver, for a target on a front side of a vehicle, on the basis of the image data, and generates an eye-catching degree map; and an estimation unit 54 that estimates the state of the driver on the basis of a visual line direction of the driver detected by the visual line detection device and the eye-catching degree map. The eye-catching degree calculation unit calculates an eye-catching degree, in consideration of vehicle components which come into the visual field of the driver when the vehicle travels.SELECTED DRAWING: Figure 3

Description

The technology disclosed herein belongs to the technical field relating to the driver state estimation device.

Recently, the development of autonomous driving systems is being promoted nationally. At present, the applicant of the present application considers that the autonomous driving system has two main directions.

The first direction is a system in which the automobile takes the lead in transporting the occupants to the destination without requiring the driver's operation, which is the so-called fully automatic driving of the automobile. On the other hand, the second direction is an automatic driving system that is premised on human driving, such as wanting to enjoy driving a car.

In the second directional automatic driving system, for example, when a driver has a disease or the like and a situation occurs in which normal driving is difficult, the automobile automatically changes to a occupant and automatically drives. is assumed. For this reason, how quickly and accurately it is possible to detect the occurrence of an abnormality in the driver, especially the occurrence of a dysfunction or disease in the driver, is the improvement of the lifesaving rate of the driver and the safety including the surroundings. It is extremely important from the viewpoint of ensuring.

As a method of estimating an abnormality of a driver, for example, as in Patent Document 1, an attempt is made to estimate the state of the driver based on the movement of the line of sight of the driver of the vehicle. Specifically, in Patent Document 1, the driver's line-of-sight direction and the image captured by the imaging means for imaging the vicinity of the own vehicle are acquired, and each of the plurality of points on the object existing in the vicinity of the own vehicle is obtained. Based on the acquisition means for acquiring three-dimensional position information that specifies the position with respect to the own vehicle, the conspicuity calculation means for calculating the conspicuity of the driver's line of sight, and the conspicuity calculation means for calculating the conspicuity. A driver state determination device including a determination means for determining a driver's condition by comparing the degree to which the driver directs his / her line of sight to a conspicuous portion and a predetermined threshold value is disclosed.

Japanese Patent No. 5966640

However, there is a difference between the image acquired by the camera and the actual driver's field of view. That is, in general, the driver's field of vision includes vehicle components in addition to the external environment on the front side of the vehicle. As a result of diligent research by the inventors of the present application, it has been found that the place where the driver's line of sight is attracted changes in the driver's field of view depending on the presence or absence of these vehicle components. Therefore, if the degree of conspicuity or the like is simply calculated from the image taken by the camera, the accuracy of estimating the driver's state will be low.

The techniques disclosed herein improve the accuracy of estimating the driver's condition when the vehicle is running.

In order to solve the above-mentioned problems, in the technique disclosed here, the line-of-sight direction of the driver is targeted at a driver state estimation device that estimates the state of the driver based on the movement of the line-of-sight of the driver of the vehicle. A line-of-sight detection device that detects the above, a camera that is mounted on the vehicle and captures the external environment on the front side of the vehicle, and an image processing unit that performs image processing on the image captured by the camera and outputs image data. Based on the image data, the attraction degree calculation unit that generates an attraction degree map by calculating the attraction degree indicating the ease of attracting the driver's line of sight to the target on the front side of the vehicle, and the line of sight. The detection device includes an estimation unit that estimates the driver's state based on the driver's line-of-sight direction and the attraction degree map detected by the detection device, and the attraction degree calculation unit is a unit of the driver when the vehicle is running. The degree of attraction is calculated in consideration of the vehicle components that enter the visibility area.

That is, the field of view of the driver who is actually seated in the driver's seat of the vehicle includes vehicle components such as a roof trim, an instrument panel, a front pillar, and a bonnet. On the other hand, the images mounted on the camera do not include those vehicle components. For this reason, there is a difference between the external environment seen by the driver from the vehicle interior and the external environment photographed by the camera. The degree of attraction of the target changes relatively due to the color difference, the brightness difference, the relative movement, and the like. Therefore, if the degree of attraction is calculated in consideration of the vehicle components that enter the driver's field of view when the vehicle is running, the accuracy of calculating the degree of attraction in the driver's field of view is improved. Therefore, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running.

It should be noted that the degree of attraction referred to here indicates the degree of conspicuity to the surroundings, and is not considered when a skilled driver consciously looks at it.

In the driver state estimation device, the image processing of the image processing unit includes a process of synthesizing an image showing the vehicle constituent member with an image taken by the camera to generate a composite image. The attractiveness calculation unit may be configured to calculate the attractiveness based on the image data based on the composite image.

According to this configuration, the composite image is an image that reproduces the driver's field of view when the vehicle is running. Since the attractiveness calculation unit calculates the attractiveness based on the image data of the composite image, the attractiveness of the target located on the front side of the vehicle can be calculated accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

In the driver state estimation device, the attraction degree calculation unit has at least one of a color difference and a brightness difference between the neighborhood region and the vehicle constituent member in a region near the vehicle constituent member in the driver's view region. When it is large, the degree of attraction of the neighboring region may be increased as compared with the case where at least one of the color difference and the luminance difference is small.

That is, when the color difference or the brightness difference between the vehicle constituent member and the neighborhood region of the vehicle constituent member is large, the neighborhood region is more conspicuous than the other regions, so that the degree of attraction is high. Therefore, according to the above-described configuration, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

In one embodiment of the driver state estimation device, the estimation unit is based on the line-of-sight frequency, which is the frequency at which the line-of-sight of the driver detected by the line-of-sight detection device is directed to a region where the degree of attraction is a predetermined value or more. The driver's condition may be estimated.

In particular, in the one embodiment, the estimation unit may be configured to estimate that an abnormality has occurred in the driver when the line-of-sight frequency is equal to or higher than a predetermined frequency.

That is, under normal conditions, the driver drives while looking at a parked vehicle, a pedestrian, a side road, or other visible object to be watched when the vehicle is running. However, when the driver is in a vague state or when the driver becomes ill and cannot actively drive, the driver naturally turns his / her gaze to the area where the degree of attraction is high. That is, when the driver frequently sees the area with a high degree of attraction, there is a high possibility that the driver has an abnormality. Therefore, if the driver's state is estimated based on the driver's line-of-sight frequency, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

In the driver state estimation device, the vehicle component may be configured to include a roof trim, an instrument panel, and a front pillar trim.

That is, when the vehicle is running, the external environment on the front side of the vehicle that the driver sees occupies most of the area defined by the front window glass. Therefore, the vehicle component that affects the degree of attraction of the external environment on the front side of the vehicle is a member located on the outer edge of the front window glass in the vehicle interior. Therefore, according to the above-described configuration, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

As described above, according to the technology disclosed herein, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running. it can.

It is the schematic which shows the front side part in the vehicle interior of the vehicle equipped with the driver state estimation device which concerns on an exemplary embodiment. It is a front view which looked at the vehicle from the front side. It is a block diagram which shows the structure of the driver state estimation device. It is a figure which shows the external environment of the front side of a vehicle photographed by a camera. It is a figure which shows the composite image data. It is a figure which shows the distribution of salience calculated based on the image taken by a camera. It is a figure which shows the distribution of salience calculated based on the composite image data. It is a graph which shows the salience of the driver's gaze point. It is a graph which shows the salience of the point randomly extracted from the attraction degree map. It is a graph comparing the ratio exceeding the threshold value at a random point and the ratio exceeding the threshold value at the driver's gaze point. It is a flowchart which shows the processing operation of the estimation part.

Hereinafter, exemplary embodiments will be described in detail with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. The left side when looking at the front side from the rear side is called the left side, and the opposite is called the right side.

FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and the steering wheel 8 is arranged on the right side.

In the vehicle interior, the front window glass 1 is arranged on the front side of the vehicle when viewed from the driver's seat. The front window glass 1 is partitioned by a plurality of vehicle components when viewed from the vehicle interior side. Specifically, the front window glass 1 is partitioned by the left and right front pillar trims 2, the roof trim 3, and the instrument panel 4.

The left and right front pillar trims 2 form boundaries on the outer side of the front window glass 1 in the vehicle width direction. Each front pillar trim 2 is arranged along each front pillar 12. As shown in FIG. 1, each front pillar trim 2 is arranged so as to be inclined obliquely so as to be separated from each other toward the upper side.

The roof trim 3 constitutes the upper boundary of the front window glass 1. The roof trim 3 covers the vehicle interior side of the roof panel 13 of the vehicle. A rear-view mirror 5 is attached to the center of the front window glass 1 in the vehicle width direction and slightly below the roof trim 3. A vehicle interior camera 101 (see FIG. 3) for photographing the vehicle interior, particularly the driver's face, is provided in the vicinity of the rear-view mirror 5 in the roof trim 3. The vehicle interior camera 101 is composed of a high-performance camera that can detect changes in the driver's pupil.

The instrument panel 4 constitutes the lower boundary of the front window glass 1. The instrument panel 4 is provided with a meter box and a display 7.

Further, the vehicle has side mirrors 6 on the outer side in the vehicle width direction with respect to the left and right front pillars 12, respectively. Each side mirror 6 is arranged so that the driver seated in the driver's seat can see through the window of the side door.

As shown in FIG. 2, the vehicle is provided with a camera 100 for photographing the external environment on the front side of the vehicle. The camera 100 is a front end portion of the vehicle and is arranged slightly below the bonnet 9 of the vehicle.

(Configuration of driver state estimation device)
The vehicle according to the present embodiment has a function of estimating the state of the driver based on the movement of the driver's line of sight. FIG. 3 shows the configuration of the driver state estimation device 20. The driver state estimation device 20 has a controller 50 having an estimation unit 54 for actually estimating the driver state. The controller 50 is a controller based on a well-known microcomputer, and uses a central processing unit (CPU) for executing a program and, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). It includes a memory that is configured to store programs and data, and an input / output bus that inputs and outputs electrical signals.

As shown in FIG. 3, the controller 50 is connected to various sensors. The sensors are, for example, a camera 100 that captures the external environment on the front side of the vehicle, a vehicle interior camera 101 that captures the driver's face, and a sweating sensor 102 that is provided on the steering wheel 8 and detects the sweating state of the driver's hand. And include.

The controller 50 has a line-of-sight calculation unit 51 that calculates the driver's line-of-sight direction from the driver's eyeball photographed by the vehicle interior camera 101. The line-of-sight calculation unit 51 calculates the line-of-sight direction of the driver by detecting a change in the driver's pupil from the state in which the driver looks into the lens of the vehicle interior camera 101, for example. The line-of-sight direction may be calculated from either the driver's left eye or the right eye, or may be the average value of the line-of-sight directions (line-of-sight vectors) obtained from each of the driver's eyes. Further, when it is difficult to calculate the driver's line-of-sight direction from the change in the driver's pupil, the line-of-sight direction may be calculated in consideration of the direction of the driver's face. The vehicle interior camera 101 and the controller 50 are examples of the line-of-sight detection device.

The controller 50 has an image processing unit 52 that performs image processing on the image D1 captured by the camera 100 and outputs the image data D2. The image processing executed by the image processing unit 52 will be described later.

Based on the image data D2, the controller 50 calculates the degree of attraction indicating the ease with which the driver's line of sight is attracted to the target on the front side of the vehicle, and generates the degree of attraction map D3. Has. In the present embodiment, the attraction degree calculation unit 53 calculates salience (visual prominence) as the attraction degree. Salience is a visual feature amount consisting of prominence that changes from moment to moment depending on color, brightness, movement, and the like. That is, a high salience means a high degree of prominence, and a high salience area is a area that stands out with respect to the surroundings. More specifically, the salience is high in the region where the color difference or the luminance difference is large with respect to the surrounding region or the movement is large with respect to the surrounding region.

The controller 50 operates based on the attraction degree map D3, the driver's line-of-sight direction calculated by the line-of-sight calculation unit 51, the detection result of the sweating sensor 102, and the change in the driver's pupil diameter photographed by the vehicle interior camera 101. It has an estimation unit 54 for estimating the state of a person. As will be described in detail later, the estimation unit 54 estimates whether the driver is in a normal state or an abnormal state from the calculation results of the attraction degree map D3 and the line-of-sight calculation unit 51. When the driver is presumed to be in an abnormal state, the estimation unit 54 determines the driver's state from the detection result of the sweating sensor 102 and the change in the driver's pupil diameter photographed by the vehicle interior camera 101. Estimate in detail. The driver's abnormal state is not limited to the state in which the driver is in an abnormal physical condition, but also includes the driver's absent-minded state and the state in which the driver's arousal level is reduced (drowsiness is increased). ..

The controller 50 controls various actuators A, a display 7, a buzzer B, etc. mounted on the vehicle based on the estimation result of the estimation unit 54. The various actuators A include actuators constituting the engine system, the brake system, and the steering system.

(Image processing by the image processing unit)
Next, the image processing by the image processing unit 52 will be described. For example, the image processing unit 52 performs a process of deleting pixels unnecessary for processing (calculation of salience) by the attraction degree calculation unit 53 among the elements constituting the image D1 with respect to the image D1 captured by the camera 100. .. Further, the image processing unit 52 performs a process of synthesizing another image with the image D1 showing the external environment on the front side of the vehicle taken by the camera 100 to generate a composite image.

FIG. 4 is an image D1 showing the external environment on the front side of the vehicle taken by the camera 100. The external environment shown in the image D1 includes a roadway 150 and a white line 151 on the roadway 150. Further, in the external environment shown in this image D1, the other vehicle 161 and the wall 162 formed on the right side of the roadway 150, the trees 163 planted in the area on the right side of the wall 162, and the wall 162 It includes a hill 164 extending to the area on the right and a forest 165 formed on the left side of the roadway 150. Further, the external environment shown in the image D1 includes the building 166 and the sky 167 extending above the building 166 and the forest 165. In the following description, it is assumed that the sky 167 of the image D1 is a cloudy sky and is substantially white.

In the present embodiment, the image processing unit 52 synthesizes with the image D1 an image showing a vehicle component that enters the driver's field of view when the vehicle is traveling. Specifically, as shown in FIG. 5, the image processing unit 52 has a front pillar trim 2 on the left side (driver's seat side) and a roof trim 3 when the driver seated in the driver's seat looks at the front side of the vehicle. An image showing the left side portion, the left side portion of the instrument panel 4, the left side mirror 6, and the steering wheel 8 is combined with the image D1. The image for compositing can be, for example, photographed in advance from the driver's seat side, and the photographed data can be stored in the memory of the controller 50 as a layer. Then, when synthesizing an image for compositing with the image D1, the image processing unit 52 may read the image from the memory and superimpose the image on the image D1 as a layer. A part of the bonnet 9 may be further considered as a vehicle component.

As shown in FIG. 5, in the composite image, a part of the image D1 is blocked by the vehicle constituent members. The image processing unit 52 outputs the image data D2 showing the composite image to the attraction degree calculation unit 53.

(Generation of attraction map)
The attraction degree calculation unit 53 calculates the salience with respect to the image data D2 of the composite image, and generates an attraction degree map (here, a salience map). Specifically, the attraction degree calculation unit 53 calculates salience for a portion of the image data D2 that represents the external environment, that is, a portion other than the image synthesized by the image processing unit 52. At this time, the attraction degree calculation unit 53 does not calculate the salience itself for the part of the image (the part of the vehicle component) synthesized by the image processing unit 52, but the salience of the part of the image data D2 representing the external environment. It is used for calculation. That is, the attraction degree calculation unit 53 calculates the attraction degree in consideration of the vehicle constituent members that enter the driver's field of view when the vehicle is traveling.

As described above, salience changes depending on the color, brightness, movement, etc. of the target. Therefore, in the present embodiment, the attraction degree calculation unit 53 calculates the salience for each feature such as color-based salience, brightness-based salience, and motion-based salience, and generates an attraction degree map for each feature. By adding them together, the final attraction map D3 is generated.

For example, regarding color-based salience, the attraction degree calculation unit 53 compares when the color difference between the neighborhood region and the vehicle constituent member is large in the vicinity region of the vehicle constituent member in the image data D2, as compared with the case where the color difference is small. To increase the salience of the neighboring area. The color difference is calculated by the following formula when the RGB of the color of a certain pixel is (R1, G1, B1) and the RGB of the color of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The attractiveness calculation unit 53 may calculate so that the larger the color difference, the higher the salience continuously, or may set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. Good.

Further, for example, with respect to salience based on brightness, the attractiveness calculation unit 53 determines that the brightness difference is small when the brightness difference between the neighborhood region and the vehicle constituent member is large in the vicinity region of the vehicle constituent member in the image data D2. Compared to the case, the salience of the neighboring area is increased. For example, when the combined vehicle component is black, the attractiveness calculation unit 53 increases the salience of the portion of the vicinity region that is close to white because the brightness difference becomes larger as the portion is closer to white. The attractiveness calculation unit 53 may calculate so that the larger the difference in brightness, the higher the salience continuously, or set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. May be good.

For the calculation of salience itself, a known computer program published on the Internet or the like can be used. In addition, a known computer program can be used for calculating the attraction map for each feature (here, the salency map) and integrating each attraction map.

FIG. 6 is an attraction map showing the distribution of the salience when the salience is calculated based on the image D1 taken by the camera 100, that is, the image data before synthesizing the vehicle components. On the other hand, FIG. 7 is an attraction map D3 showing the distribution of the salience when the salience is calculated based on the image data D2 of the composite image. Although not clearly shown in FIG. 7, the vehicle body components have a color close to black.

As shown in FIG. 6, in the state where there are no vehicle components, for example, the boundary portion between the tree 163 and the forest 165 with the sky 167 and the white line 151 on the left side of the road 150 are high salience portions. On the other hand, as shown in FIG. 7, in the state where the vehicle components are present, the part partitioned by the roof trim 3 and the forest 165 in the sky 167 and the building 166 located between the front pillar trim 2 and the tree 163 in the sky 167. Is a high part of salience. This is because the presence of the vehicle component takes into consideration the color difference and the brightness difference from the vehicle component. In this way, the portion with high salience changes depending on whether the vehicle component is present or not.

When the vehicle is running, the vehicle components enter the driver's field of view in addition to the external environment on the front side of the vehicle. Therefore, as in the present embodiment, the salience is calculated based on the image data D2 of the composite image obtained by synthesizing the vehicle component with the image D1 taken by the camera 100, so that the driver can see the line of sight when the vehicle is running. It is possible to accurately calculate the area that is easily attracted.

(Calculation of line-of-sight frequency)
Next, the calculation of the line-of-sight frequency will be described with reference to FIGS. 8 to 10. In this embodiment, the estimation unit 54 calculates the line-of-sight frequency.

FIG. 8 is a plot of the driver's gaze point, that is, the salience of a point beyond the driver's line of sight. In this graph, the line-of-sight calculation unit 51 determines the line-of-sight direction of the driver, applies the gaze point ahead of the line-of-sight direction to the attraction degree map D3, and obtains the salience of the gaze point from the attraction degree map D3. Is generated by. Each of these points is obtained at a predetermined sampling cycle.

On the other hand, FIG. 9 is generated by randomly designating coordinates from the attractiveness map D3 generated by the attractiveness calculation unit 53 and obtaining the salience of the coordinates. Each of these points is obtained at a predetermined sampling cycle.

After creating the graphs of FIGS. 8 and 9, respectively, the ROC (Receiver Operating Characteristic) curve of the ratio exceeding the threshold value at the random point and the ratio exceeding the threshold value at the driver's gaze point is obtained. Specifically, first, as a first step, the threshold value is set to a value larger than the maximum value in the graphs of FIGS. 8 and 9. Next, as a second step, while lowering the threshold value, the ratio of points exceeding the threshold value is obtained for each threshold value. Then, the process of this second step is repeated until the threshold value becomes equal to or less than the minimum value in the graphs of FIGS. 8 and 9.

The graphs shown in FIGS. 8 and 9 are generated by dividing them into predetermined measurement cycles. The measurement cycle can be arbitrarily set, for example, 5 seconds.

After the first and second steps, as the third step, the ratio exceeding the threshold value at the random point (referred to as the first probability) is taken on the horizontal axis and the ratio exceeding the threshold value at the driver's gaze point (referred to as the second probability). For the vertical axis, a graph as shown in FIG. 10 is created. The graph of FIG. 10 represents the gaze probability for a random probability at the same threshold. In the graph of FIG. 10, since both the horizontal axis and the vertical axis are probabilities, the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 10, the broken line indicates that the driver's line of sight is evenly directed to the entire field of view. That is, if the driver's line of sight extends over the entire viewing angle region, the first probability and the second probability do not deviate significantly. Therefore, if the driver's line of sight extends over the entire viewing angle region, the linear shape as shown by the broken line in FIG. 10 is obtained when the first probability and the second probability are compared. On the other hand, when the curve expands convexly upward as shown by the curve C1 in FIG. 10, it means that the driver's gaze point has a high ratio of exceeding the threshold value. That is, when a curve having a shape like the curve C1 in FIG. 10 is calculated, it means that the driver's line of sight tends to see a portion having high salience. On the other hand, when the curve spreads downwardly as shown by the curve C2 in FIG. 10, it means that the driver's gaze point has a low ratio of exceeding the threshold value. That is, when a curve having a shape like the curve C2 in FIG. 10 is calculated, it means that the driver's line of sight tends to see a portion having low salience.

In the present embodiment, the AUC (Area Under Curve) is obtained as the fourth step. AUC is the integrated value of the lower right part of this curve (that is, the area of the part surrounded by the curve and the alternate long and short dash line in FIG. 10). In the present embodiment, this integrated value is used as the line-of-sight frequency. In this way, the accuracy of calculating the line-of-sight frequency can be improved by comparing the salience of the part gazed by the driver with the salience of the coordinates randomly specified from the attraction degree map D3 and calculating the line-of-sight frequency. .. That is, when the region having high salience spreads over the entire map in the attraction degree map D3, the driver sees the portion having high salience no matter which position he / she looks at. In this case, simply calculating the ratio of seeing salience above the threshold value will calculate that the line-of-sight frequency is high even if the driver looks at the entire field of view without bias. On the other hand, if the comparison with the random points is performed as in the present embodiment, even if the area with high salience extends over the entire map, the entire field of view of the driver is randomly viewed. Can be compared if they have the same tendency as. Therefore, it is possible to more accurately determine whether or not there is a bias in the line-of-sight direction of the driver.

(Estimation of driver status)
Next, the processing operation of estimating the driver state by the estimation unit 54 will be described with reference to the flowchart of FIG.

First, in step S101, the estimation unit 54 acquires various data. In particular, the estimation unit 54 includes the calculation result of the line-of-sight calculation unit 51, the attraction degree map D3 generated by the attraction degree calculation unit 53, the image data of the driver's face acquired by the vehicle interior camera 101, and the sweating sensor 102. Acquire the detection result.

In the next step S102, the estimation unit 54 calculates the line-of-sight frequency. The estimation unit 54 calculates the line-of-sight frequency by the calculation method as described above.

In the following step S103, the estimation unit 54 estimates whether or not the driver's attention is reduced. Specifically, the estimation unit 54 estimates that the driver's attention is reduced when the line-of-sight frequency calculated in step S102 is equal to or higher than a predetermined frequency. The estimation unit 54 proceeds to step S104 when the line-of-sight frequency is equal to or higher than a predetermined frequency and YES is estimated to reduce the driver's attention. On the other hand, the estimation unit 54 proceeds to step S106 when the line-of-sight frequency is less than the predetermined frequency and the driver's attention is presumed to be NO.

In step S104, the estimation unit 54 estimates whether or not the driver has a predominant sympathetic nerve. Specifically, the estimation unit 54 estimates that the sympathetic nerve is in a predominant state when the amount of perspiration per unit time detected by the perspiration sensor 102 is equal to or greater than a predetermined amount. On the other hand, the estimation unit 54 estimates that the parasympathetic nerve is in a predominant state when the amount of sweating per unit time detected by the sweating sensor 102 is less than a predetermined amount. The estimation unit 54 proceeds to step S105 when the detection result of the sweating sensor 102 is equal to or greater than a predetermined amount and YES is presumed that the sympathetic nerve is dominant. On the other hand, the estimation unit 54 proceeds to step S107 when the detection result of the sweating sensor 102 is less than a predetermined amount and YES, which is presumed that the parasympathetic nerve is dominant.

In step S105, the estimation unit 54 estimates whether or not the driver is actively driving. Specifically, the estimation unit 54 estimates from the change in the pupil diameter of the driver photographed by the vehicle interior camera 101. More specifically, the estimation unit 54 estimates that the driver is actively driving when there is almost no change in the pupil diameter before and after the movement of the driver's line of sight. On the other hand, the estimation unit 54 estimates that the driver is passively driving when the pupil diameter changes before and after the driver's line-of-sight movement. This is because when the driver is driving passively, the line of sight is moved after being stimulated by the eyes. The estimation unit 54 proceeds to step S109 when YES is presumed that the driver is actively driving. On the other hand, the estimation unit 54 proceeds to step S108 when the driver is presumed to be passively driving NO. Even if the convergence angle, which is the intersection angle between the driver's left eye line of sight and the right eye line of sight, is calculated and it is estimated whether or not the driver is actively driving based on the convergence angle. Good.

In step S106, the estimation unit 54 estimates that the driver is in a normal state. That is, if the driver has some abnormality, the driver's attention is reduced. Therefore, if the driver does not have a decrease in attention, it is presumed that the driver is in a normal state, but if the driver has a decrease in attention, an abnormality occurs in the driver. It can be estimated that it is. After step S106, it returns.

In step S107, the estimation unit 54 estimates that the driver's alertness is reduced (sleepiness is increased). In general, a parasympathetic predominant state predominates over the sympathetic nerve during sleep. Therefore, it can be presumed that the state in which the attention is lowered and the parasympathetic nerve is dominant is the state in which the driver's alertness is lowered. After step S107, it returns.

In step S108, the estimation unit 54 estimates that the driver is in a loose state. That is, since the driver is passively driving while being awake, it can be presumed that the driver is in a distracted state while thinking about something other than driving. After step S108, it returns.

In step S109, the estimation unit 54 estimates that the driver is in an abnormal physical condition. That is, the driver tries to continue normal driving even if the physical condition is abnormal. For this reason, if the driver is awake during normal driving and the driver's attention is reduced despite being actively driving, some abnormality may occur in the driver's physical condition. It can be estimated that there is. Return after step S109.

As described above, the driver's state is estimated by the estimation unit 54. When it is estimated that the driver's alertness is low (step S107) or the driver is in a vague state (step S108), the controller 50 sounds, for example, a buzzer B or displays it on the display 7. Or As a result, the controller 50 encourages the driver to awaken or concentrate on driving. Further, when the estimation unit 54 estimates that the driver is in an abnormal physical condition, the controller 50 outputs control signals to various actuators A, for example, to drive the vehicle to a safe area such as a shoulder. , Stop the vehicle in the safe area.

Therefore, in the present embodiment, the line-of-sight detection device (vehicle interior camera 101 and line-of-sight calculation unit 51) that detects the driver's line-of-sight direction, the camera 100 that is mounted on the vehicle and captures the external environment on the front side of the vehicle, and the camera 100. Image processing unit 52 that performs image processing on the image D1 captured by the vehicle and outputs the image data D2, and based on the image data D2, attracts the driver's line of sight to the target on the front side of the vehicle. The state of the driver based on the attraction degree calculation unit 53 that calculates the attraction degree indicating ease and generates the attraction degree map D3, and the driver's line-of-sight direction and the attraction degree map D3 calculated by the line-of-sight detection device. The attraction degree calculation unit 53 calculates the attraction degree in consideration of the vehicle constituent members (front pillar trim 2, etc.) that enter the driver's view area when the vehicle is traveling. As a result, the attractiveness calculation unit 53 calculates the attractiveness in consideration of the driver's actual field of view when the vehicle is running. Therefore, the accuracy of calculating the degree of attraction in the driver's field of view is improved. As a result, when estimating the driver's state based on the movement of the driver's line of sight, it is possible to improve the accuracy of estimating the driver's state when the vehicle is running.

In particular, the vehicle components considered in this embodiment include a roof trim 3, an instrument panel 4, and a front pillar trim 2. That is, when the vehicle is running, most of the external environment on the front side of the vehicle that the driver sees is the area defined by the front window glass 1. Therefore, the vehicle component that affects the degree of attraction of the external environment on the front side of the vehicle is a member located on the outer edge of the front window glass 1 in the vehicle interior. Therefore, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

Further, in the present embodiment, the image processing of the image processing unit 52 includes a process of synthesizing an image showing a vehicle component with an image taken by the camera 100 to generate a composite image, and is an attraction degree calculation unit. 53 calculates the degree of attraction based on the image data based on the composite image. That is, the composite image generated by the image processing unit 52 is an image that reproduces the driver's field of view when the vehicle is running. Since the attraction degree calculation unit 53 calculates the attraction degree based on the image data of the composite image, the attraction degree of the target located on the front side of the vehicle can be calculated with high accuracy. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

Further, in the present embodiment, when at least one of the color difference and the brightness difference between the neighborhood region and the vehicle constituent member is large in the neighborhood region of the vehicle constituent member in the driver's view region, the attraction degree calculation unit 53 causes a color difference and Compared with the case where at least one of the brightness differences is small, the degree of attraction in the neighboring region is increased. That is, when the color difference or the luminance difference between the vehicle component and the vicinity region is large, the vicinity region is more conspicuous than the other regions, so that the degree of attraction is high. Therefore, by considering the color difference and the brightness difference, the degree of attraction in the driver's field of view can be calculated more accurately. As a result, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

Further, in the present embodiment, the estimation unit 54 determines when the line-of-sight frequency detected by the line-of-sight detection device is the frequency at which the driver's line-of-sight is directed to a region where the degree of attraction is a predetermined value or more. , It is estimated that the driver has an abnormality. That is, under normal conditions, the driver drives while looking at a parked vehicle, a pedestrian, a side road, or other visible object to be watched when the vehicle is running. However, when the driver is in a vague state or when the driver becomes ill and cannot actively drive, the driver naturally turns his / her gaze to the area where the degree of attraction is high. That is, when the driver frequently sees the area with a high degree of attraction, there is a high possibility that the driver has an abnormality. Therefore, if the driver's state is estimated based on the driver's line-of-sight frequency, the accuracy of estimating the driver's state when the vehicle is running can be further improved.

The technique disclosed herein is not limited to the above-described embodiment, and can be substituted as long as it does not deviate from the gist of the claims.

For example, in the above-described embodiment, the composite image is generated by the image processing of the image processing unit 52, and the attraction degree calculation unit 53 generates the attraction degree map D3 based on the image data D2 of the composite image. Not limited to this, the attraction degree calculation unit 53 calculates the salience of each pixel from the image data showing the image D1 taken by the camera 100, generates an attraction degree map, and then considers the influence of the vehicle constituent members. The final attraction map D3 may be generated by correcting the salience of the generated attraction map.

Further, in the above-described embodiment, as vehicle components to be considered when calculating the degree of attraction, the front pillar trim 2 on the left side (driver's seat side), the left side portion of the roof trim 3, and the left side portion of the instrument panel 4 are used. The part, the left side mirror 6, and the steering wheel 8 are adopted. Not limited to this, at least the left side (driver's seat side) front pillar trim 2 located on the periphery of the front window glass 1, the left part of the roof trim 3, and the left part of the instrument panel 4 should be considered. Just do it.

Further, in the above-described first and second embodiments, the line-of-sight frequency is calculated by the estimation unit 54, but the present invention is not limited to this, and a processor for calculating the line-of-sight frequency may be provided separately from the estimation unit 54.

The above embodiments are merely examples, and the scope of the present disclosure should not be construed in a limited manner. The scope of the present disclosure is defined by the scope of claims, and all modifications and changes belonging to the equivalent scope of the scope of claims are within the scope of the present disclosure.

The technique disclosed herein is useful as a driver state estimation device that estimates the driver's state based on the movement of the driver's line of sight of the vehicle.

2 Front pillar trim 3 Roof trim
4 Instrument panel 20 Driver state estimation device
50 controller
51 Line-of-sight calculation unit (line-of-sight detection device)
52 Image processing unit
53 Invitation degree calculation unit
54 Estimator
100 cameras
101 Interior camera (line-of-sight detection device)
D1 image D2 image data D3 attraction map

Claims (6)

It is a driver state estimation device that estimates the state of the driver based on the movement of the line of sight of the driver of the vehicle.
A line-of-sight detection device that detects the driver's line-of-sight direction,
A camera mounted on the vehicle that captures the external environment on the front side of the vehicle,
An image processing unit that performs image processing on the image taken by the camera and outputs image data,
Based on the image data, an attraction degree calculation unit that calculates an attraction degree indicating the ease of attracting the driver's line of sight to a target on the front side of the vehicle and generates an attraction degree map.
It is provided with an estimation unit that estimates the state of the driver based on the line-of-sight direction of the driver detected by the line-of-sight detection device and the attraction degree map.
The driver state estimation device is characterized in that the attraction degree calculation unit calculates the attraction degree in consideration of vehicle components that enter the driver's field of view when the vehicle is traveling. In the driver state estimation device according to claim 1,
The image processing of the image processing unit includes a process of synthesizing an image showing the vehicle component with an image taken by the camera to generate a composite image.
The driver state estimation device is characterized in that the attraction degree calculation unit calculates the attraction degree based on image data based on the composite image. In the driver state estimation device according to claim 1 or 2.
When at least one of the color difference and the luminance difference between the neighborhood region and the vehicle constituent member is large in the neighborhood region of the vehicle constituent member in the driver's view region, the attraction degree calculation unit has the color difference and the luminance difference. A driver state estimation device, characterized in that the degree of attraction in the vicinity region is increased as compared with the case where at least one of the two is small. In the driver state estimation device according to any one of claims 1 to 3,
The estimation unit estimates the state of the driver based on the line-of-sight frequency at which the line-of-sight of the driver detected by the line-of-sight detection device is directed to a region where the degree of attraction is equal to or higher than a predetermined value. A characteristic driver state estimation device. In the driver state estimation device according to claim 4,
The estimation unit is a driver state estimation device that estimates that an abnormality has occurred in the driver when the line-of-sight frequency is equal to or higher than a predetermined frequency. In the driver state estimation device according to any one of claims 1 to 5,
The vehicle component is a driver state estimation device including a roof trim, an instrument panel, and a front pillar trim.

Images