WO2020174601A1

WO2020174601A1 - Alertness level estimation device, automatic driving assistance device, and alertness level estimation method

Info

Publication number: WO2020174601A1
Application number: PCT/JP2019/007500
Authority: WO
Inventors: 和樹國廣
Original assignee: 三菱電機株式会社
Priority date: 2019-02-27
Filing date: 2019-02-27
Publication date: 2020-09-03
Also published as: DE112019006953T5; JP7109649B2; JPWO2020174601A1

Abstract

In an alertness level estimation device (10), a face image acquisition unit (11) acquires a captured face image representing the face of an occupant in a vehicle. An upper eyelid opening degree calculation unit (12) calculates the degree of opening of an upper eyelid on the basis of the positions, detected from the face image, of an eye inner corner, an eye outer corner, and the highest point of the upper eyelid. When the degree of opening of the upper eyelid is less than a predetermined threshold value, a downward view determination unit (13) determines whether or not the occupant is in a downward view state, on the basis of the position of the highest point of the upper eyelid detected from the face image and the position of an intersection point of a profile line of a lower eyelid and a vertical line drawn downward from the highest point of the upper eyelid. When it is determined that the degree of opening of the upper eyelid becomes less than the threshold value and the occupant is not in a downward view state, an alertness level estimation unit (14) of the alertness level estimation device (10) estimates that the alertness level of the occupant has decreased.

Description

Arousal level estimation device, automatic driving support device, and arousal level estimation method

The present invention relates to a technique for estimating the wakefulness of a vehicle occupant.

A wakefulness estimation device that estimates the wakefulness of a driver (presence or absence of drowsiness) based on the degree of eye opening obtained from an image of the driver's face taken by an in-vehicle camera is known (for example, Patent Document 1). ..

Drivers sometimes turn their line of sight downward not only to look outside the front of the vehicle, but also to view the screens of meters and navigation devices inside the vehicle (hereinafter, turning the line of sight downward is referred to as "downward viewing"). Since the eye opening when looking down is close to the eye opening when the awakening level decreases, the awakening level estimating device determines that the awakening level decreases when the driver looks down. The challenge is to prevent erroneous decisions.

The arousal level estimation device of Patent Document 1 estimates the arousal level from the degree of eye opening of the driver, determines whether or not the driver is in the downward-looking state, and determines that the driver is in the downward-looking state. In addition, the erroneous determination is prevented by correcting the driver's awakening degree estimated from the eye opening in the direction of increasing the awakening degree.

JP, 2008-171065, A JP, 2011-048531, A JP, 2007-241937, A

In the arousal level estimation device of Patent Document 1, the degree of eye opening of the driver is calculated based on the distance between the highest point of the contour line of the upper eyelid and the lowest point of the contour line of the lower eyelid, Whether or not the driver is in the downward-looking state is determined based on the shapes of the contour line and the contour line of the lower eyelid. In general, the contour of the lower eyelid is less clear than the contour of the upper eyelid, and it is difficult to detect the contour of the lower eyelid with high accuracy. Therefore, in the method of Patent Document 1, there is a concern that the error of the calculated eye opening becomes large and, as a result, the reliability of the estimation result of the awakening degree of the driver becomes low. In Patent Document 1, a light source that projects the shadow of the lower eyelid on the eyeball of the driver is installed in the vehicle, and the shadow is photographed by a camera to improve the detection accuracy of the contour of the lower eyelid. There is.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a wakefulness estimation device capable of highly accurately estimating the wakefulness of a vehicle occupant.

The arousal level estimation device according to the present invention is a face image acquisition unit that acquires a face image that is an image of the face of an occupant of a vehicle, and the positions of the inner and outer corners of the eyes, the position of the outer corners of the eyes, and the highest point of the upper eyelid. When the upper eyelid opening calculation unit that calculates the opening of the upper eyelid based on the position of, the opening of the upper eyelid is smaller than a predetermined opening threshold, the highest point of the upper eyelid detected from the face image. The position and the downward-viewing determination unit that determines whether the occupant is in the downward-looking state based on the position of the intersection point between the vertical line drawn from the highest point of the upper eyelid and the contour line of the lower eyelid, and the opening of the upper eyelid. And a wakefulness estimation unit that estimates that the wakefulness of the occupant has decreased when it is determined that the occupant is smaller than the opening threshold and the occupant is not in the downward viewing state.

In the present invention, the degree of awakening of the occupant of the vehicle is estimated based on the result of determination of the upper eyelid of the occupant and determination of whether the occupant is in the downward-looking state (downward-looking determination). Since the opening degree of the upper eyelid can be calculated without using the position of the lower eyelid, it can be calculated with high accuracy. The position of the contour line of the lower eyelid is used for the downward vision determination, but the downward vision determination is performed with the upper eyelid opening small, that is, with the contour of the lower eyelid bulging and the contour line being clear. Therefore, it can be performed with high accuracy. As a result, the estimation accuracy of the wakefulness of the occupant is also high.

The objects, features, aspects and advantages of the present invention will become more apparent by the following detailed description and the accompanying drawings.

FIG. 3 is a block diagram showing a configuration of an awakening degree estimation device according to the first embodiment. 5 is a flowchart showing an operation of the arousal level estimation device according to the first embodiment. FIG. 6 is a block diagram showing a modified example of the arousal level estimation device according to the first embodiment. It is a figure which shows the hardware structural example of an awakening degree estimation apparatus. It is a figure which shows the hardware structural example of an awakening degree estimation apparatus. FIG. 6 is a block diagram showing a configuration of an automatic driving support device according to a second embodiment.

<Embodiment 1>
FIG. 1 is a block diagram showing the configuration of the arousal level estimation device 10 according to the first embodiment. In the present embodiment, it is assumed that wakefulness estimation device 10 is installed in a vehicle. However, the arousal level estimation device 10 is not limited to one that is permanently installed in the vehicle, and may be built in a portable device that can be brought into the vehicle, such as a mobile phone, a smartphone, or a portable navigation device. Further, part or all of the awakening degree estimation device 10 may be built on a server that can communicate with the vehicle.

As shown in FIG. 1, the alertness estimation device 10 is connected to a camera 21 and a warning device 22 provided in the vehicle.

The camera 21 is for photographing the inside of the vehicle, and is installed at a position where the face of the vehicle occupant can be photographed. In the present embodiment, the camera 21 is assumed to be a wide-angle camera having a shooting range capable of shooting the faces of passengers in all seats of the vehicle. However, for example, when the awakening degree estimation device 10 needs to estimate only the awakening degree of the occupant (driver) in the driver's seat, the image capturing range of the camera 21 may be wide enough to capture only the driver's seat.

The arousal level estimation device 10 estimates the arousal level (presence or absence of drowsiness) of the occupant based on the image of the occupant's face captured by the camera 21 (hereinafter referred to as “face image”), and sends the estimation result to the warning device 22. Output. In the present embodiment, it is assumed that the alertness estimation device 10 estimates the alertness of all occupants captured by the camera 21. In this case, the arousal level estimation device 10 outputs the information on the position of the seat of each occupant together with the estimation result of the arousal level of each occupant. That is, the wakefulness estimation device 10 outputs information such as in which seat the occupant estimated to have high wakefulness is in, and which seat the occupant estimated to have low wakefulness is in.

The warning device 22 issues a warning to the inside of the vehicle according to the wakefulness of the occupant estimated by the wakefulness estimation device 10, and includes a speaker for issuing a warning sound or a warning message, a display for displaying a warning screen, and the like. .. In the present embodiment, it is assumed that the warning device 22 gives a warning when it is estimated that the awakening level of the occupant (driver) in the driver's seat has decreased. However, the operation content of the warning device 22 is not limited, and for example, the warning may be issued even when it is estimated that the wakefulness of an occupant other than the driver has decreased.

As shown in FIG. 1, the arousal level estimation device 10 includes a face image acquisition unit 11, an upper eyelid opening calculation unit 12, a downward vision determination unit 13, and an arousal level estimation unit 14.

The face image acquisition unit 11 acquires the face image of the occupant captured by the camera 21. In the present embodiment, since the camera 21 is a wide-angle camera capable of capturing all seats of the vehicle, the face image acquisition unit 11 uses the face recognition technology to extract the face image for each occupant from the in-vehicle image captured by the camera 21. To extract.

The upper eyelid opening degree calculation unit 12 detects the position of the eyes of the occupant, the position of the outer corners of the eyes, and the position of the highest point of the upper eyelid from the face image of the occupant acquired by the face image acquisition unit 11, and based on their positional relationship. Then, the degree of opening of the upper eyelid (hereinafter referred to as "opening") is calculated. More specifically, the upper eyelid opening degree calculation unit 12 calculates the opening degree of the upper eyelid based on the first distance which is the distance between the straight line connecting the inner corner of the eye and the outer corner of the eye and the highest point of the upper eyelid. To do. As described above, it is difficult to detect the contour of the lower eyelid with higher accuracy than the contour of the upper eyelid. Since the opening degree of the upper eyelid calculated by the upper eyelid opening degree calculation unit 12 can be calculated without using the information of the contour line of the lower eyelid, the accuracy thereof is high. In addition, the highest point of the upper eyelid is the point (vertex) of the upper eyelid that is farthest from the straight line connecting the outer corner of the eye and the inner corner of the eye.

In the present embodiment, the upper eyelid opening degree calculation unit 12 obtains the flatness of the eye by dividing the first distance by the distance between the inner and outer corners of the eye, and the flattening rate is a predetermined reference value. The value obtained by dividing by is calculated as the opening degree of the upper eyelid.

The reference value of the flatness of the eyes may be, for example, a constant value determined by the manufacturer of the alertness estimation device 10 as an average value of the flatness of the eyes. Alternatively, if the wakefulness estimation device 10 can identify an individual occupant by face authentication using a face image or the like, a reference value for each occupant may be registered in the wakefulness estimation device 10. By using different reference values for each occupant, it is possible to suppress the influence of individual differences in eye size on the calculation result of the upper eyelid opening.

The downward viewing determination unit 13 determines that the occupant is in the downward viewing state when the upper eyelid opening of the occupant calculated by the upper eyelid opening calculation unit 12 is smaller than a predetermined threshold (hereinafter referred to as “opening threshold”). A downward view determination is performed to determine whether or not it is. Specifically, when the opening degree of the upper eyelid is smaller than the opening threshold value, the downward viewing determination unit 13 determines the position of the highest point of the upper eyelid and the lower eyelid from the face image acquired by the face image acquisition unit 11. And the position of the intersection of the vertical line drawn from the highest point of the upper eyelid and the contour line of the lower eyelid (hereinafter referred to as the "specific point of the lower eyelid"), the highest point of the upper eyelid and the lower eyelid. It is determined whether or not the occupant is in the downward-looking state based on the positional relationship with the specific point. In addition, the specific point of the lower eyelid may be an intersection of a perpendicular line descending from the highest point of the upper eyelid toward a straight line connecting the inner corner of the eye and the outer corner of the eye and the contour line of the lower eyelid.

Generally, when the eyelid is narrowed or the eyelid is narrowed by a person, the lower eyelid bulges and the contour of the lower eyelid becomes clear, so the facial image shows The contour line can be detected with high accuracy. The position of the contour line of the lower eyelid is necessary for the downward view determination performed by the downward view determination unit 13, but the downward view determination is performed when the opening degree of the upper eyelid is small, and therefore the downward view determination is The position of the contour of the lower eyelid obtained with high accuracy is used. Therefore, the downward-view determination performed by the downward-view determination unit 13 is highly accurate.

In the present embodiment, the downward viewing determination unit 13 determines the second distance, which is the distance between the highest point of the upper eyelid and the specific point of the lower eyelid, when the opening degree of the upper eyelid is smaller than the opening degree threshold value. It is determined whether or not the occupant is in the downward-looking state based on and. That is, when the opening degree of the upper eyelid is smaller than the opening threshold value and the second distance is larger than a predetermined threshold value (hereinafter referred to as “distance threshold value”), the downward viewing determination unit 13 causes the occupant to look downward. It is determined to be in the state.

Hereafter, the condition "the upper eyelid opening is smaller than the opening threshold and the second distance is larger than the distance threshold" may be referred to as "downward viewing condition". The second distance may also be referred to as the “upper and lower eyelid distance”.

The distance threshold may be, for example, a constant value determined by the manufacturer of the arousal level estimation device 10 as the size of the minimum distance between the upper and lower eyelids required for downward viewing. Alternatively, when the wakefulness estimation device 10 can identify an individual occupant by face recognition using a face image or the like, a distance threshold for each occupant may be registered in the wakefulness estimation device 10. By using a different distance threshold for each occupant, it is possible to suppress the result of the downward vision determination from being influenced by the individual difference in the size of the eyes when looking downward.

The awakening degree estimation unit 14 determines the occupant's upper eyelid opening calculated by the upper eyelid opening calculation unit 12 and the result of the downward vision determination of the occupant performed by the downward vision determination unit 13 for the occupant. Estimate if the alertness has decreased. Specifically, the wakefulness estimation unit 14 reduces the wakefulness of the occupant when the occupant's upper eyelid opening is smaller than the opening threshold and it is determined that the occupant is not in the downward-looking state. Presumed to have done. As described above, the upper eyelid opening degree calculation unit 12 can calculate the upper eyelid opening degree of the occupant with high accuracy, and the downward vision determination unit 13 can perform downward vision determination with high accuracy. The estimation of the awakening degree of the occupant by 14 is highly accurate.

Hereafter, the condition that "the occupant's upper eyelid opening is smaller than the opening threshold value and the occupant is not in the downward-looking state" may also be referred to as "awakening degree lowering condition".

As described above, according to the arousal level estimation device 10 according to the first embodiment, the arousal level of the occupant can be estimated with high accuracy in consideration of whether or not the occupant is in the downward vision state. Further, the detection of the contour of the lower eyelid, which is a relatively difficult process, may be performed at least when the downward vision determination is performed (that is, when the opening of the upper eyelid is smaller than the opening threshold). It can also contribute to the reduction of the processing load of the alertness estimation device 10.

FIG. 2 is a flowchart showing the operation of the alertness estimation device 10 according to the first embodiment. The operation of the alertness estimation device 10 will be described below with reference to FIG.

When the ignition of the vehicle is turned on and the awakening degree estimation device 10 is activated, first, the face image acquisition unit 11 acquires the face image of the occupant captured by the camera 21 (step S101). Here, for simplicity of explanation, it is assumed that the image captured by the camera 21 includes only the face image of one passenger. However, if the camera 21 is a wide-angle camera as in the present embodiment, face images of a plurality of passengers may be acquired in step S101. In that case, the following processes of steps S102 to S108 are performed on the face images of the respective occupants.

Next, the upper eyelid opening degree calculation unit 12 detects the position of the eyes of the occupant, the position of the outer corners of the eyes, and the position of the highest point of the upper eyelid from the face image of the occupant acquired in step S101, and establishes their positional relationship. Based on this, the opening degree of the upper eyelid of the occupant is calculated (step S102).

When the opening degree of the occupant's upper eyelid calculated in step S102 is smaller than the predetermined opening degree threshold value (YES in step S103), the downward viewing determination unit 13 determines based on the occupant's face image acquired in step S101. Then, the downward vision determination is performed to determine whether the occupant is in the downward vision state (step S104).

When it is determined that the occupant is not in the downward looking state as a result of the downward looking determination in step S104 (NO in step S105), the awakening degree estimation unit 14 estimates that the awakening degree of the occupant has decreased (step S106).

On the other hand, when the opening of the occupant's upper eyelid calculated in step S102 is larger than the opening threshold value (NO in step S103), or when it is determined in step S104 that the occupant is in the downward-looking state (step S105). YES), the awakening degree estimation unit 14 estimates that the awakening degree of the occupant is high (step S107).

The estimation result of the wakefulness of the occupant in step S106 or S107 is output to the warning device 22 (step S108). As a result, the warning device 22 issues a warning according to the estimation result of the awakening degree of the occupant.

The arousal level estimation device 10 repeatedly executes the above flow. Note that the flow may be constantly executed, for example, after the activation of the awakening degree estimation device 10 (after the ignition of the vehicle is turned on), or may be executed only while the vehicle is running (while the vehicle is stopped). May not be executed). When the flow is constantly executed, the wakefulness of the occupant is estimated before the vehicle starts moving, which can contribute to the improvement of vehicle safety. However, in general, it is not necessary to estimate the awakening degree of the occupant while the vehicle is stopped. Therefore, the flow is executed only while the vehicle is running, and the processing of the awakening degree estimation device 10 is performed. The load and the power consumption may be reduced.

[Modification]
In the wakefulness estimation device 10 of the first embodiment, the wakefulness estimation unit 14 outputs the estimation result of the wakefulness of the occupant to the warning device 22, but the output destination is not limited to the warning device 22, and any output destination. It can be a device. For example, as shown in FIG. 3, the estimation result of the wakefulness of the occupant by the wakefulness estimation unit 14 may be input to the automatic driving control device 23 that is an ECU (Electronic Control Unit) that automatically drives the vehicle.

As the operation of the automatic driving control device 23 in FIG. 3, for example, the alertness estimation unit 14 estimates that the alertness of the occupant (driver) in the driver's seat has decreased, and the warning device 22 gives a warning. If the driver's awakening level remains low, the automatic driving control device 23 automatically evacuates the vehicle to a safe place (for example, a wide shoulder or a parking area) and stops the vehicle. Such an operation is possible.

Further, in the first embodiment, in the downward vision determination unit 13, the occupant has an upper eyelid opening smaller than the opening threshold value, and the occupant's upper and lower eyelid distance (second distance) becomes larger than the distance threshold value. Immediately after the condition (downward viewing condition) is satisfied, it is determined that the occupant is in a downward viewing state. However, the downward-viewing determination unit 13 may determine that the occupant is in the downward-viewing state when the eyes of the occupant have satisfied the downward-viewing condition for a certain length of time. Specifically, it may be determined that the occupant is in the downward-looking state when the ratio of the time when the downward-looking condition is satisfied to a certain time exceeds a predetermined threshold value (time threshold value). As a result, it is possible to suppress erroneous detection of downward vision due to, for example, the influence of noise, which can contribute to further improvement in the estimation accuracy of the arousal level.

There is a correlation between the direction of the occupant's face and the direction of the line of sight. Specifically, when the occupant looks downward, not only the eyes and eyelids move, but also the face tends to move toward the direction of the line of sight. Therefore, the downward-viewing determination unit 13 may determine whether the occupant is in the downward-looking state by taking into consideration the orientation of the occupant's face detected from the face image. For example, the downward vision determination unit 13 causes the eyes of the occupant to satisfy the downward vision condition, and the direction of the occupant's face moves downward (in particular, the direction in which the meter in the vehicle or the screen of the navigation device is located). In this case, it may be determined that the occupant is in the downward-looking state.

Also, there is a correlation between the occupant's face orientation and arousal level. Specifically, when the wakefulness of the occupant decreases, the movement of the face tends to decrease, and the orientation of the face tends to be substantially constant. Therefore, the awakening degree estimation unit 14 may estimate the awakening degree of the occupant by taking into consideration the orientation of the occupant's face detected from the face image. For example, the awakening degree estimation unit 14 satisfies the condition that the opening of the upper eyelid of the occupant is smaller than the opening threshold value and that the occupant is not in the downward-looking state (awakening degree lowering condition), and If the variation in the face orientation is smaller than a predetermined threshold value, it may be estimated that the wakefulness of the occupant has decreased.

∙ Occupants often look down when operating equipment (vehicle equipment) mounted on the vehicle, such as shift levers and navigation devices. Therefore, the downward-viewing determination unit 13 may determine whether or not the occupant is in the downward-looking state by taking into consideration the operation status of the vehicle-mounted device. For example, when the downward vision determination unit 13 determines that the occupant is in the downward vision state when the eyes of the occupant satisfy the downward vision condition and the occupant operates the vehicle-mounted device at a position lower than the face. It may be determined. The information on the operating status of the vehicle-mounted device can be acquired from, for example, the ECU of the vehicle.

Also, it is considered that the occupants operating the in-vehicle device are highly alert. Therefore, the awakening degree estimation unit 14 may estimate the awakening degree of the occupant in consideration of the operation status of the vehicle-mounted device. For example, when the occupant's eyes satisfy the wakefulness reduction condition and the occupant is not operating the in-vehicle device, the wakefulness estimation unit 14 estimates that the wakeup level of the occupant has decreased. May be.

[Example of hardware configuration]
4 and 5 are diagrams each showing an example of the hardware configuration of the arousal level estimation device 10. Each function of the components of the alertness estimation device 10 shown in FIG. 1 is realized by the processing circuit 50 shown in FIG. 4, for example. That is, the arousal level estimation device 10 acquires a face image that is an image of the face of an occupant of the vehicle, and raises it based on the positions of the inner and outer corners of the eyes, the positions of the outer corners of the eyes, and the position of the highest point of the upper eyelid detected from the face image. When the eyelid opening is calculated and the eyelid opening is smaller than a predetermined opening threshold, the position of the highest point of the upper eyelid detected from the face image and the vertical line drawn from the highest point of the upper eyelid It is determined whether the occupant is in the downward-looking state based on the position of the intersection between the contour line of the lower eyelid and the lower eyelid, the opening of the upper eyelid is smaller than the opening threshold, and it is determined that the occupant is not in the downward-looking state. Then, the processing circuit 50 is provided for estimating that the wakefulness of the occupant has decreased. The processing circuit 50 may be dedicated hardware, or a processor that executes a program stored in a memory (Central Processing Unit (CPU), processing device, arithmetic device, microprocessor, microcomputer, It may be configured using a DSP (also referred to as a digital signal processor).

When the processing circuit 50 is dedicated hardware, the processing circuit 50 includes, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable). Gate Array), or a combination of these. Each function of the components of the alertness estimation device 10 may be realized by an individual processing circuit, or those functions may be collectively realized by one processing circuit.

FIG. 5 shows an example of the hardware configuration of the alertness estimation device 10 when the processing circuit 50 is configured using the processor 51 that executes a program. In this case, the functions of the constituent elements of the alertness estimation device 10 are realized by software or the like (software, firmware, or a combination of software and firmware). The software and the like are written as a program and stored in the memory 52. The processor 51 realizes the function of each unit by reading and executing the program stored in the memory 52. That is, the awakening level estimation apparatus 10 acquires, when executed by the processor 51, a process of acquiring a face image that is an image of the face of an occupant of the vehicle, and positions of the inner and outer corners of the eye detected from the face image, and Processing to calculate the opening of the upper eyelid based on the position of the highest point of the upper eyelid, when the opening of the upper eyelid is smaller than a predetermined opening threshold, of the highest point of the upper eyelid detected from the face image A process of determining whether or not the occupant is in the downward-looking state based on the position and the position of the intersection point of the vertical line drawn from the highest point of the upper eyelid and the contour line of the lower eyelid, and the opening degree of the upper eyelid is the opening threshold value. When it is determined that the occupant is smaller and the occupant is not in the downward-looking state, a process of estimating that the wakefulness of the occupant is lowered, and a memory 52 for storing a program to be executed as a result, Prepare In other words, it can be said that this program causes a computer to execute the procedure and method of the operation of the constituent elements of the alertness estimation device 10.

Here, the memory 52 is, for example, RAM (RandomAccessMemory), ROM (ReadOnlyMemory), flash memory, EPROM (ErasableProgrammableReadOnlyMemory), EEPROM (ElectricallyErasableProgrammableReadOnlyMemory), or the like. Volatile semiconductor memory, HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, or any storage medium used in the future. May be.

Above, the configuration of the functions of the components of the awakening degree estimation device 10 being realized by either hardware or software has been described. However, the configuration is not limited to this, and a configuration in which some of the components of the alertness estimation device 10 are implemented by dedicated hardware and another of the components is implemented by software or the like may be used. For example, for some of the constituent elements, the function is realized by the processing circuit 50 as dedicated hardware, and for some of the other constituent elements, the processing circuit 50 as the processor 51 executes the program stored in the memory 52. The function can be realized by reading and executing.

As described above, the arousal level estimation device 10 can realize each function described above by hardware, software, or a combination thereof.

<Second Embodiment>
The second embodiment shows an example in which the awakening degree estimation device 10 described in the first embodiment is applied to an automatic driving support device that supports automatic driving of a vehicle.

Here, the definition of the automation level of automatic driving of a car (automatic driving level) will be explained. According to J3016 (September 2016) of SAE (Society of Automotive Engineers) International and its Japanese translation, JASO TP18004 (February 2018), the automatic driving level of the automatic driving system is defined as follows. Has been done.

Level 0 (No driving automation): Driver performs some or all of the dynamic driving tasks Level 1 (Driving assistance): System performs either vertical or horizontal vehicle motion control subtasks in a limited area Level 2 (Partial driving automation): System executes both vertical and lateral vehicle motion control subtasks in limited area Level 3 (Conditional driving automation): System executes all dynamic driving tasks in limited area However, if it is difficult to continue operation, the driver appropriately responds to the intervention request from the system. Level 4 (advanced operation automation): The system responds to all dynamic driving tasks and when it is difficult to continue operation. Execution in limited area Level 5 (complete operation automation): System executes all dynamic driving tasks and responses to cases where it is difficult to continue operation (that is, not within the limited area). The term means all operational and tactical functions (excluding strategic functions such as itinerary planning and selection of transit points) that must be performed in real time when operating a vehicle in road traffic. In addition, "limited area" means a specific condition (geographical constraint, road constraint, environmental constraint, traffic constraint, speed constraint, time constraint, etc.) that the system or its function is designed to operate. (Including restrictions).

FIG. 6 is a block diagram showing the configuration of the automatic driving support device 30 according to the second embodiment. In FIG. 6, elements having the same functions as those shown in FIG. 1 are designated by the same reference numerals as those in FIG. 1, and description thereof will be omitted here.

As shown in FIG. 6, the automatic driving support device 30 is connected to the camera 21, the automatic driving control device 23, and the notification device 24. The automatic driving control device 23 is an ECU that performs automatic driving of the vehicle, and corresponds to the above-mentioned "system". In the present embodiment, the automatic driving control device 23 performs at least level 3 automatic driving. The notification device 24 notifies the occupant of the vehicle, and includes a speaker that emits a notification sound or a notification message sound, a display that displays an alarm screen, and the like.

Further, the automatic driving support device 30 includes the awakening degree estimation device 10 and an occupant selection unit 31. The arousal level estimation device 10 estimates the arousal level of the occupant of the vehicle as described in the first embodiment, and outputs the estimation result to the occupant selection unit 31.

The occupant selection unit 31 selects the occupant (the occupant to be the driver) to intervene in driving the vehicle in response to the intervention request from the automatic driving support device 30. The intervention request is input from the automatic driving support apparatus 30 to the occupant selection unit 31 when the automatic driving support apparatus 30 is performing level 3 automatic driving and it becomes difficult to continue the automatic driving. When the occupant selection unit 31 receives the intervention request from the automatic driving support device 30, the occupant selection unit 31 confirms the estimation result of the wakefulness of the occupant by the wakefulness estimation device 10, and drives the occupant estimated to have a high wakefulness to drive the vehicle. To be selected as an occupant to intervene in.

The selection result by the passenger selection unit 31 is input to the notification device 24. The notification device 24 notifies the occupant selected by the occupant selection unit 31 to instruct intervention in driving the vehicle.

When the occupant selection unit 31 selects an occupant in the driver's seat, the driving authority of the vehicle can be transferred to the occupant without stopping the vehicle. However, when the occupant selection unit 31 selects a occupant other than the driver's seat, the automatic driving control device 23 stops the vehicle at a safe place, moves the selected occupant to the driver's seat, and then drives the occupant. Transfer the authority. When there are a plurality of occupants estimated to have a high arousal level, the occupant selection unit 31 may select any one of them, but may preferentially select the occupant in the driver's seat.

As described above, the automatic driving support device 30 according to the second embodiment supports the so-called handover process in which the automatic driving support device 30 transfers the driving authority to the occupant, and the driving authority is transferred to the occupant having a low awakening degree. It can be prevented.

Note that, in the present invention, the respective embodiments can be freely combined, or the respective embodiments can be appropriately modified or omitted within the scope of the invention.

Although the present invention has been described in detail, the above description is illustrative in all aspects, and the present invention is not limited thereto. It is understood that innumerable variants not illustrated can be envisaged without departing from the scope of the invention.

10 arousal level estimation device, 11 face image acquisition part, 12 upper eyelid opening calculation part, 13 downward vision determination part, 14 awakening level estimation part, 21 camera, 22 warning device, 23 automatic driving control device, 24 notification device, 30 Automatic driving support device, 31 passenger selection unit, 50 processing circuit, 51 processor, 52 memory.

Claims

A face image acquisition unit that acquires a face image that is an image of the face of a vehicle occupant;
Position of the eyelid detected from the face image, the position of the corner of the eye and the position of the highest point of the upper eyelid, an upper eyelid opening calculation unit that calculates the opening of the upper eyelid,
When the opening of the upper eyelid is smaller than a predetermined opening threshold, the position of the highest point of the upper eyelid detected from the face image and the vertical line and the contour of the lower eyelid lowered from the highest point of the upper eyelid A downward-viewing determination unit that determines whether the occupant is in the downward-viewing state based on the position of the intersection with the line,
An opening degree of the upper eyelid is smaller than the opening threshold value, and, when it is determined that the occupant is not in a downward-looking state, an awakening degree estimation unit that estimates that the awakening degree of the occupant has decreased,
An awakening degree estimation device including.
The upper eyelid opening calculation unit calculates the opening of the upper eyelid based on a first distance which is a distance between a straight line connecting the inner corner of the eye and the outer corner of the eye and the highest point of the upper eyelid,
The arousal level estimation device according to claim 1.
The upper eyelid opening calculation unit divides the first distance by the distance between the inner corner of the eye and the outer corner of the eye to obtain a value obtained by dividing a flatness rate by a predetermined reference value, Calculated as the opening of the upper eyelid,
The arousal level estimation device according to claim 2.
The upper eyelid opening calculation unit uses the reference value of a different value for each occupant,
The arousal level estimation device according to claim 3.
The downward viewing determination unit is based on a second distance, which is a distance between the highest point of the upper eyelid and the intersection of the vertical line drawn from the highest point of the upper eyelid and the contour line of the lower eyelid, The awakening degree estimation device according to claim 1, wherein it is determined whether or not the occupant is in a downward-looking state.
The downward viewing determination unit determines that the occupant is in the downward viewing state when the opening of the upper eyelid is smaller than the opening threshold and the second distance is larger than a predetermined distance threshold. The arousal level estimation device according to claim 5.
The downward-view determination unit uses the distance threshold having a different value for each occupant,
The awakening degree estimation device according to claim 6.
The downward-viewing determination unit predetermines a ratio of a time period in which the opening degree of the upper eyelid is smaller than the opening degree threshold value and the second distance is larger than a predetermined distance threshold value to a certain time. The awakening degree estimation device according to claim 5, wherein it is determined that the occupant is in a downward-looking state when the time threshold exceeds the time threshold.
The downward-view determination unit uses the distance threshold having a different value for each occupant,
The arousal level estimation device according to claim 8.
The downward-viewing determination unit determines whether or not the occupant is in a downward-looking state, taking into consideration the orientation of the face of the occupant detected from the face image.
The arousal level estimation device according to claim 1.
The awakening degree estimation unit estimates the awakening degree of the occupant by adding the orientation of the face of the occupant detected from the face image,
The arousal level estimation device according to claim 1.
The downward-view determination unit determines whether or not the occupant is in the downward-view state by taking into consideration the operation status of the device mounted on the vehicle.
The arousal level estimation device according to claim 1.
The awakening degree estimation unit estimates the awakening degree of the occupant in consideration of an operation state of a device mounted on the vehicle,
The arousal level estimation device according to claim 1.
The awakening degree estimation unit outputs the estimation result of the awakening degree of the occupant to a warning device that warns a occupant of the vehicle,
The arousal level estimation device according to claim 1.
The awakening degree estimation unit outputs an estimation result of the awakening degree of the occupant to an automatic driving control device of the vehicle,
The arousal level estimation device according to claim 1.
An automatic driving support device for supporting automatic driving of a vehicle,
An arousal level estimation device according to claim 1;
When it becomes difficult to continue the automatic driving of the vehicle, based on the estimation result of the awakening degree of the occupant by the awakening degree estimation unit, an occupant selection unit that selects an occupant to intervene in the driving of the vehicle,
An automatic driving support device.
The face image acquisition unit of the arousal level estimation device acquires a face image that is an image of the face of the vehicle occupant,
The upper eyelid opening calculation unit of the awakening degree estimation device, the position of the inner corner of the eye detected from the face image, the position of the outer corner of the eye and the position of the highest point of the upper eyelid, calculates the opening of the upper eyelid,
When the opening degree of the upper eyelid is smaller than a predetermined opening degree threshold value, the downward vision determination unit of the arousal level estimation device, the position of the highest point of the upper eyelid detected from the face image and the upper eyelid Determine whether the occupant is in the downward-looking state based on the position of the intersection of the vertical line drawn from the highest point and the contour of the lower eyelid,
When the opening degree of the upper eyelid becomes smaller than the opening threshold value, and when it is determined that the occupant is not in the downward-looking state, the awakening degree estimation unit of the awakening degree estimation device reduces the awakening degree of the occupant. Presumably,
Arousal level estimation method.