WO2021176633A1 - Driver state estimation device and driver state estimation method - Google Patents

Abstract

A driver state estimation device according to the present invention is provided with: an information collection unit (11) that collects driver-related information related to a driver of a vehicle; a state determination unit (12) that determines, on the basis of a reaction of the driver, whether the driver-related information collected by the information collection unit (11) is driver-related information collected when the driver is in a normal state; and a state estimation unit (13) that estimates whether the driver is in the normal state on the basis of the driver-related information collected by the information collection unit (11), and estimation information that has been reset on the basis of a determination result from the state determination unit (12) indicating that the driver-related information is driver-related information collected when the driver is in the normal state.

Description

Driver state estimation device and driver state estimation method

The present disclosure relates to a driver state estimation device for estimating a driver's state and a driver state estimation method.

Conventionally, there is known a technique for estimating whether or not a driver of a vehicle or the like is currently driving in a normal state based on information acquired while driving in a normal state.
Here, the state in which the driver can be said to be in a normal state differs depending on the driver. Therefore, in order to estimate whether or not the driver is driving in a normal state, it is necessary that the normal state of the driver is determined according to the driver.
For example, Patent Document 1 discloses a technique for determining that a driver is driving in a normal state from the biometric information of the driver.

JP-A-2007-272834

For example, biological information such as heart rate or pupil size varies due to individual differences. Therefore, even when the normal state of the driver is determined from the biological information as in the technique disclosed in Patent Document 1, there is still a problem that the normal state of the driver cannot be accurately determined.

This disclosure is made to solve the above-mentioned problems, and enables the driver to judge that the driver is driving in a normal state with higher accuracy than the case of judging from biometric information. It is an object of the present invention to provide a state estimation device.

The driver state estimation device according to the present disclosure includes an information collecting unit that collects driver-related information related to the driver of the vehicle, and the driver-related information collected by the information collecting unit based on the driver's reaction is driving. The state judgment unit that determines whether the information is driver-related information collected when the person is in a normal state, the driver-related information collected by the information collection unit, and the driver-related information by the state judgment unit are the driver. State estimation that estimates whether or not the driver is in the normal state based on the estimation information that is reset based on the judgment result that is the driver-related information collected when is in the normal state. It is equipped with a part.

According to the present disclosure, it is possible to judge that the driver is driving in a normal state with higher accuracy than the case of judging from biometric information.

It is a figure which shows the configuration example of the driver state estimation apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the image of an example of the estimation information in Embodiment 1. FIG. It is a flowchart for demonstrating operation of the driver state estimation apparatus which concerns on Embodiment 1. FIG. It is a flowchart for demonstrating the specific operation of step ST302 of FIG. It is a flowchart explaining the specific operation of step ST403 of FIG. It is a figure which shows the configuration example of the driver state estimation apparatus which concerns on Embodiment 2. FIG. It is a figure for demonstrating a neural network. It is a flowchart for demonstrating operation of the driver state estimation apparatus which concerns on Embodiment 2. FIG. FIG. 5 is a diagram showing a configuration example of a driver state estimation device and a learning device when the learning unit is provided in a learning device outside the driver state estimation device in the second embodiment. 10A and 10B are diagrams showing an example of the hardware configuration of the driver state estimation device according to the first and second embodiments.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
Embodiment 1.
The driver state estimation device according to the first embodiment is mounted on the vehicle.
The driver state estimation device collects information related to the driver of the vehicle (hereinafter referred to as "driver-related information"), and is based on the collected driver-related information and the stored estimation information. It is estimated whether or not the driver is in a normal state (hereinafter referred to as "normal state"). In the first embodiment, the driver-related information relates to the state of the environment surrounding the driver, such as information about the driver himself, information about the state inside the vehicle driven by the driver, or information about the state outside the vehicle. Contains information.
In the first embodiment, the "normal state" means a state in which the driver can drive the vehicle normally. Specifically, the "normal state" is, for example, a state in which the driver can concentrate on driving, a state in which the driver is awake, a state in which the driver is not exhausted, or a state in which the driver is not frustrated. To say.
In the first embodiment, the estimation information is information in which the driver-related information and the estimation rule for determining whether or not the driver-related information is in a normal state are associated with each other. The estimation information is generated based on, for example, driver-related information that is assumed to be collected during driving by a general driver in a normal state. The estimation information is generated in advance at the time of shipment of the driver state estimation device and is stored in the storage unit of the driver state estimation device.

The driver state estimation device according to the first embodiment collects driver-related information when the driver starts driving the vehicle, and the collected driver-related information is the driver collected by the driver in a normal state. Determine if it is related information (hereinafter referred to as "normal state driver related information"). When the driver state estimation device according to the first embodiment determines that the driver-related information is the normal state driver-related information, the driver state estimation device updates the stored estimation information based on the normal state driver-related information. do. As a result, the driver state estimation device according to the first embodiment uses the stored estimation information as estimation information that matches the driver who is driving. The driver state estimation device estimates whether or not the driver is in a normal state based on the collected driver-related information and the updated estimation information while updating the estimation information.
The details of the determination of whether the information is related to the normal state driver and the update of the determination driver-related information by the driver state estimation device according to the first embodiment will be described later.

FIG. 1 is a diagram showing a configuration example of a driver state estimation device according to the first embodiment.
As shown in FIG. 1, the driver state estimation device 1 includes an information collection unit 11, a state determination unit 12, a state estimation unit 13, a storage unit 14, and an output unit 15. The state determination unit 12 includes an inquiry unit 121, a response acquisition unit 122, and a determination unit 123. The state estimation unit 13 includes an update unit 131 and an estimation unit 132.

The information collecting unit 11 collects driver-related information.
More specifically, the information collecting unit 11 extracts the feature amount from the information collected from the information collecting device (not shown), reflects it in the collected information, and reflects the information after reflecting the feature amount in the driver. Use related information.
The information collecting device is, for example, an image pickup device that images the inside of a vehicle (hereinafter, referred to as an “in-vehicle image pickup device”; not shown). The in-vehicle image pickup device is a camera or the like installed for the purpose of monitoring the inside of the vehicle, and is installed so as to be able to image at least the driver's face. The in-vehicle image pickup device may be shared with, for example, a so-called "driver monitoring system (DMS)".

The information collecting device is, for example, a microphone (not shown) installed for the purpose of collecting sound in the vehicle.

The information collecting device includes, for example, a vehicle speed sensor (not shown), an accelerator opening sensor (not shown), a brake sensor (not shown), a button (not shown), a turn signal (not shown), or a GPS. (Global Positioning System) or the like, which is a device installed in a vehicle to detect an operation performed on the vehicle by a driver or a state of the vehicle (hereinafter referred to as a “vehicle state detection device”; not shown).

Further, the information collecting device is installed in the vehicle, for example, an image pickup device (hereinafter referred to as "outside vehicle image pickup device"; not shown), a sensor (not shown), a LiDAR (not shown), or the like that images the surroundings of the vehicle. It is a device that acquires information around the vehicle (hereinafter referred to as "peripheral information acquisition device"; not shown).

For example, when the information collecting device is an in-vehicle image pickup device, the information collecting unit 11 obtains the driver's body temperature, sweating degree, heartbeat, etc. from the captured image (hereinafter referred to as "in-vehicle image") collected from the in-vehicle image pickup device. The feature amount according to the facial expression, emotion, line of sight, eye opening degree, pupil size, face orientation, or posture is extracted. The information collecting unit 11 uses the in-vehicle image after reflecting the extracted feature amount as driver-related information.

For example, when the information collecting device is a microphone, the information collecting unit 11 uses the voice information collected from the microphone as a feature amount according to the voice uttered by the driver, the voice quality of the driver, or the voice uttered by the passenger. Is extracted. The information collecting unit 11 uses the voice information after reflecting the extracted feature amount as the driver-related information.

For example, when the information collecting device is a vehicle state detecting device, the information collecting unit 11 uses the information collected from the vehicle state detecting device to handle the steering angle, the accelerator opening, the brake opening, the button operation, and the direction instruction. The feature amount according to the operation of the device, the vehicle speed, the acceleration, or the position of the own vehicle is extracted. The information collecting unit 11 uses the information after reflecting the extracted feature amount as the driver-related information.

For example, when the information collecting device is a peripheral information acquisition device, the information collecting unit 11 responds to the distance from another vehicle or whether or not the white line is stepped on from the information collected from the peripheral information acquisition device. Extract the feature amount. The information collecting unit 11 uses the information after reflecting the extracted feature amount as the driver-related information.

The information collecting unit 11 outputs the driver-related information to the state determination unit 12.
The information collecting unit 11 may collect information from a plurality of information collecting devices. In that case, the information collecting unit 11 extracts a feature amount from the collected information for each information collecting device, reflects the extracted feature amount in the collected information, and sets it as driver-related information.
Further, the above-mentioned information collecting device is only an example, and the information collecting device includes various devices and the like capable of collecting driver-related information.

The state determination unit 12 is based on the driver's reaction, and the driver-related information collected by the information collection unit 11 is the driver-related information collected when the driver is in a normal state, that is, the normal state driver-related information. Determine if it is information.
In the first embodiment, the driver's reaction is, for example, the driver's response to an inquiry about the driver's condition made by the driver state estimation device 1 to the driver.

The inquiry unit 121 of the state determination unit 12 makes an inquiry to the driver regarding the driver's condition.
Specifically, the inquiry unit 121 makes the above inquiry by voice, for example. The inquiry unit 121 outputs, for example, a voice message "Are you sleepy" or "Are you tired?" From a speaker (not shown) installed in the vehicle.
When the driver confirms the output voice message, he / she responds, for example, with "yes" or "no".

Further, the inquiry unit 121 may make the above inquiry by displaying, for example. The inquiry unit 121 outputs, for example, a display message "Are you sleepy?" Or "Are you tired?" From a touch panel display (not shown) installed in the vehicle.
When the driver visually recognizes the displayed display message, he / she responds by touching, for example, the "Yes" button or the "No" button. The "Yes" button or the "No" button is displayed together with the display message when, for example, the inquiry unit 121 displays the display message.

The driver may respond to the inquiry by the inquiry unit 121 by a method other than voice or button touch. For example, the driver may use his / her face to respond to the inquiry by the voice message or display of "Are you sleepy?" Output by the inquiry unit 121. Specifically, the response using the face to the inquiry output by the inquiry unit 121 is, for example, a response by nodding, a response by changing the facial expression, a response by changing the direction of the face, and a response by changing the degree of eye opening. It is a response or a response by changing the line of sight.
The driver may respond to the inquiry of the inquiry unit 121 by combining a plurality of methods. For example, the driver may combine a voice response and a face response to the inquiry of the inquiry unit 121.

Further, the inquiry unit 121 may simultaneously make an inquiry by voice and an inquiry by display, for example.
The content of the voice message or the content of the display message described above is only an example. The inquiry unit 121 may make an inquiry to the driver to obtain a response from the driver as to whether or not the condition is normal.

The response acquisition unit 122 of the state determination unit 12 acquires the driver's response to the inquiry made by the inquiry unit 121.
For example, in the above example, when the inquiry unit 121 outputs a voice message "Are you sleepy?", The response acquisition unit 122 acquires the utterance voice of "yes" or "no" by the driver.
The response acquisition unit 122 may acquire the driver's response to the inquiry made by the inquiry unit 121 from the driver-related information as an in-vehicle image collected by the information collection unit 11. For example, when the driver responds by nodding to the voice message "Are you sleepy?" Output by the inquiry unit 121, the response acquisition unit 122 acquires the response from the in-vehicle image. The response acquisition unit 122 may acquire information to the effect that the driver nodded by using a known image recognition technique.
The response acquisition unit 122 outputs the information regarding the acquired response to the determination unit 123. At this time, the response acquisition unit 122 outputs the information regarding the inquiry made by the inquiry unit 121 to the determination unit 123 together with the information regarding the acquired response.

The determination unit 123 of the state determination unit 12 determines whether the driver-related information collected by the information collection unit 11 is normal state driver-related information based on the reaction of the driver (hereinafter referred to as "normal state determination". )I do.
Specifically, the judgment unit 123 is based on the information regarding the response output from the response acquisition unit 122, and the response of the driver acquired by the response acquisition unit 122 is information indicating that the driver is in a normal state. In some cases, the driver determines that the condition is normal.
To give a specific example, for example, when the response acquisition unit 122 outputs information that the driver has responded by voice to the inquiry by voice message "Are you sleepy?", The judgment unit. 123 determines that the driver-related information collected by the information collecting unit 11 is normal state driver-related information.
It should be noted that it is predetermined in advance what kind of inquiry is made to the driver and what kind of response is obtained from the driver to determine that the driver is in a normal state. The determination unit 123 may perform voice recognition or the like by using a known technique such as voice recognition to determine the normal state.

In the above description, the driver's reaction is the driver's response to the inquiry about the driver's state made by the driver state estimation device 1 to the driver, but this is only an example. No. For example, the driver's reaction may be the driver's response to the utterance of the passenger.
Specifically, for example, the determination unit 123 acquires the driver's response to the utterance of the passenger based on the driver-related information as voice information in the vehicle collected by the information collection unit 11. Then, the determination unit 123 determines the normal state based on the acquired driver's response, and when the acquired driver's response is information indicating that the driver is in the normal state, the driver is in the normal state. You may decide that.

For example, the determination unit 123 is in a normal state when the driver can obtain some response to some utterance by the passenger based on the voice information in the vehicle collected by the information collection unit 11. Assuming that the information indicating that the information can be obtained, the driver determines that the vehicle is in a normal state. It is assumed that the information that can identify the utterance voice of the passenger and the information that can identify the utterance voice of the driver are registered in advance, and the determination unit 123 operates based on the information that is registered in advance. The voice of the person or passenger may be specified.

For example, the determination unit 123 analyzes the utterance content using a known voice recognition technique, and when the driver does not utter a content that complains of an abnormal state, the acquired driver's response is The driver may be information indicating that the vehicle is in a normal state. To give a specific example, for example, the judgment unit 123 acquires when the driver responds "OK" to the utterance "Do you want to sleep?" By the passenger based on the voice information collected by the information collection unit 11. The driver's response is determined to be a state indicating that the driver is in a normal state, and the driver is determined to be in a normal state. For example, when the driver responds "I want to sleep" to the utterance "I want to sleep?" By the passenger, the judgment unit 123 determines that the acquired response of the driver is not information indicating the normal state of the driver. , The driver does not judge that it is in a normal state. It should be noted that it is predetermined in advance what kind of response the driver should make to determine that the driver is in a normal state.

Further, for example, the driver's reaction may be the driver's reaction to an external event. For example, the determination unit 123 reacts to some external event that occurs outside the vehicle based on the driver-related information as vehicle peripheral information and the driver-related information as an in-vehicle image collected by the information collection unit 11. Judge whether or not it was possible to obtain. In the first embodiment, the "external event" refers to various events that suddenly occur outside the vehicle, such as jumping out or sudden braking by a vehicle in front. The vehicle peripheral information is an captured image (hereinafter referred to as "vehicle peripheral image") obtained by capturing the surroundings of the vehicle. When the reaction of the driver can be acquired, the determination unit 123 assumes that the information indicating that the driver is in the normal state can be acquired, and determines that the driver is in the normal state.
To give a specific example, for example, when the judgment unit 123 jumps out of the vehicle based on the vehicle peripheral image and the vehicle interior image collected by the information collecting unit 11, the driver moves in the direction in which the pop-out occurs. Determine if you have turned your gaze. When the driver turns his / her line of sight in the direction in which the pop-out occurs, the determination unit 123 determines that the driver has been able to acquire information indicating that the driver is in the normal state, and determines that the driver is in the normal state. For example, if the determination unit 123 does not direct the line of sight in the direction in which the pop-out occurs, the driver cannot acquire the information indicating that the normal state is obtained, and the driver does not determine that the normal state is present.

When the determination unit 123 determines the normal state based on the driver's reaction other than the response to the inquiry made by the inquiry unit 121, in the driver state estimation device 1, the state determination unit 12 may use the inquiry unit 121 and It is not essential to include the response acquisition unit 122.

Further, the determination unit 123 may determine the normal state based on information other than the driver's reaction in addition to the driver's reaction.
For example, the determination unit 123 may determine the normal state based on the driver's biological information in addition to the driver's reaction. The determination unit 123 is information that the information acquired as the reaction of the driver indicates that the driver is in a normal state, and the biometric information of the driver is information that indicates that the driver is in a normal state. In some cases, the driver determines that it is in a normal state. The driver's biological information is included in, for example, the driver-related information collected by the information collecting unit 11.
It is assumed that the conditions for determining what kind of information the biometric information about the driver is in the normal state of the driver are set in advance. For example, as a condition for the driver to determine that the vehicle is in a normal state, "the degree of eye opening is equal to or higher than a preset threshold value" is set.

Further, for example, when the driver's reaction is an utterance by the driver, the determination unit 123 may determine the normal state based on the driver's voice quality in addition to the driver's reaction.
Further, for example, the determination unit 123 may determine the normal state based on the facial expression of the driver in addition to the reaction of the driver.
Further, for example, the determination unit 123 may determine the normal state based on the reaction of the driver and the time until the reaction is acquired. The time until the response is acquired is, for example, the time from when the inquiry unit 121 outputs the query until the response acquisition unit 122 acquires the response, or after the external event occurs, the judgment unit 123. Is the time it takes to get the driver's reaction to the external event. For example, the determination unit 123 acquires information indicating that the driver is in a normal state, and the time until the information is acquired is a preset threshold value (hereinafter referred to as “reaction time determination threshold value”). If it is less than or equal to the following, the driver determines that the vehicle is in a normal state. The determination unit 123 may change the reaction time determination threshold value according to the driver.
By causing the determination unit 123 to determine the normal state based on information other than the driver's reaction in addition to the driver's reaction, the driver state estimation device 1 determines the normal state only from the driver's reaction. It is possible to judge the normal state more accurately than in the case of performing.

The state determination unit 12 associates the driver-related information collected by the information collection unit 11 with the determination result of whether or not the driver-related information can be determined to be normal state driver-related information, and causes the state estimation unit 13 to perform the determination. Output.
Further, when the state determination unit 12 determines that the driver-related information collected by the information collection unit 11 is the normal state driver-related information, the state determination unit 12 stores the normal state driver-related information for each feature amount. Accumulate in 14.

In the first embodiment, the state determination unit 12 performs the normal state determination as described above at an appropriate timing.
For example, in the state determination unit 12, when the determination unit 123 determines the normal state based on the driver's response to the inquiry by the inquiry unit 121, a preset time interval such as a 30-minute interval or an hour interval is used. (Hereinafter referred to as "set time interval"), the normal state is judged. In this case, the inquiry unit 121 may make an inquiry at set time intervals.
The set time interval may be changed according to the elapsed time since the driver starts driving the vehicle.
For example, in the state determination unit 12, the determination unit 123 acquires the response to the driver's external event based on the driver-related information, not the driver's response to the inquiry by the inquiry unit 121, and determines the normal state. In this case, the determination unit 123 constantly determines the normal state.

Further, in the first embodiment, when the determination unit 123 determines that the driver is in the normal state as a result of determining the normal state, the determination unit 123 is preset from the time when it is determined that the driver is in the normal state. It can be determined that the driver-related information collected by the information collecting unit 11 while the conditions are satisfied is the normal state driver-related information. The preset condition may be, for example, "a preset time" or "until the traveling condition changes".

The state estimation unit 13 estimates whether or not the driver is in a normal state based on the driver-related information collected by the information collecting unit 11 and the estimation information stored in the storage unit 14.
More specifically, the state estimation unit 13 is based on the driver-related information collected by the information collecting unit 11 and the determination result by the state determination unit 12 that the driver-related information is the normal state driver-related information. The estimation information is reset, and the estimation information stored in the storage unit 14 is updated. When the estimation information is updated, the state estimation unit 13 estimates whether or not the driver is in a normal state based on the driver-related information collected by the information collection unit 11 and the reset estimation information. do.

When the driver-related information output from the state determination unit 12 is the normal state driver-related information, the update unit 131 of the state estimation unit 13 resets the estimation information based on the normal state driver-related information. do.
When the estimation information is reset, the update unit 131 updates the estimation information stored in the storage unit 14 to the estimation information after the reset. When the update unit 131 resets the estimation information, the estimation unit 132 of the state estimation unit 13 estimates whether or not the driver is in a normal state based on the reset estimation information. The details of the estimation unit 132 will be described later.

A method of resetting the estimation information by the update unit 131 will be described with reference to specific examples.
FIG. 2 is a diagram showing an image of an example of estimation information in the first embodiment.
As shown in FIG. 2, the estimation information is information in which the driver-related information and the estimation rule for estimating that the driver is in a normal state are associated with each other. In FIG. 2, the feature amount information reflected in the driver-related information is also illustrated.
For example, in the estimation information shown in FIG. 2, when the driver-related information is an in-vehicle image reflecting a feature amount showing the driver's facial expression, the driver is not sleeping, in other words, the driver's eyes If it is open, it indicates that the driver can be presumed to be in a normal state.

It is assumed that the storage unit 14 now stores estimation information as shown in FIG. 2, for example. Further, it is assumed that the state determination unit 12 outputs the normal state driver-related information as the vehicle information. The normal state driver-related information reflects a feature amount indicating that the sudden acceleration is 1.5 times per 10 km. That is, the driver is in a normal state even if he / she accelerates suddenly 1.5 times per 10 km.

For example, in the estimation information, the update unit 131 sets the estimation rule that "the sudden acceleration per 10 km is within 1 time", which is associated with the vehicle information, to "the sudden acceleration per 10 km is within 1.5 times". Reset to. As a result, the update unit 131 can set a normal state estimation rule according to the driver.

For example, the update unit 131 may reset the estimation rule based on the normal state driver-related information stored in the storage unit 14. Specifically, when the update unit 131 accumulates the normal state driver-related information reflecting the feature amount indicating that the sudden acceleration is 1.5 times per 10 km, the threshold value or more set in advance is accumulated. In the estimation information, the estimation rule that "the sudden acceleration per 10 km is within 1.5 times", which is associated with the vehicle information, is reset to "the sudden acceleration per 10 km is within 1.5 times". May be good.

For example, the update unit 131 recalculates "n" that "the sudden acceleration per 10 km is within n times" based on the accumulated normal state driver-related information, and resets the estimation rule. May be good. To give a specific example, the update unit 131 may set the number of sudden accelerations per 10 km, which is the largest number, to "n" from the accumulated normal state driver-related information.

For example, in addition to the contents shown in FIG. 2, it is assumed that the usage rule flag is set for each estimation rule in the estimation information. The usage rule flag is a flag for designating an estimation rule used by the estimation unit 132 when estimating whether or not the driver is in a normal state. Here, the estimation unit 132 estimates whether or not the driver is in a normal state according to an estimation rule in which the usage rule flag "1" is set.
In this case, when the normal state driver-related information is output from the state determination unit 12, the update unit 131 may set the usage rule flag based on the normal state operation-related information. In the above example, the update unit 131 outputs the normal state driver-related information reflecting the feature amount indicating the number of sudden accelerations per 10 km from the state determination unit 12, and the number of sudden accelerations per 10 km. The estimation rule for is updated and the usage rule flag "1" is set. As a result, when the estimation unit 132 estimates the normal state of the driver based on the driver-related information reflecting a plurality of different feature quantities, the estimation rule reflecting the characteristics of the driver is preferentially used. , The normal state of the driver can be estimated.

The estimation unit 132 of the state estimation unit 13 determines whether or not the driver is in a normal state based on the driver-related information collected by the information collection unit 11 and the estimation information stored in the storage unit 14. presume. When the update unit 131 updates the estimation information, the estimation unit 132 puts the driver in a normal state based on the driver-related information collected by the information collection unit 11 and the estimation information updated by the update unit 131. Estimate whether or not it is. The estimation unit 132 may acquire the driver-related information collected by the information collection unit 11 from the state determination unit 12.
For example, it is assumed that the estimation information is the content shown in FIG. 2, and the driver-related information collected by the information collecting unit 11 is voice information reflecting a feature amount indicating the voice quality of the driver. In this case, the estimation unit 132 estimates that the driver is in a normal state if there is no change in the voice quality of the driver based on the estimation information. On the other hand, the estimation unit 132 estimates that the driver is not in a normal state when there is a change in the voice quality of the driver. The estimation unit 132 may determine whether or not there is a change in the voice quality of the driver by using a known voice recognition technique based on the accumulated driver-related information.

Here, when the driver-related information acquired from the state determination unit 12 is the normal state driver-related information, the state determination unit 12 has already obtained the reaction of the driver indicating the normal state, and the driver is in the normal state. It is judged to be. Therefore, the estimation unit 132 may estimate that the driver is in a normal state without using the estimation information. In this case, the estimation unit 132 provides estimation information when the driver-related information is not the normal state driver-related information, in other words, when the state determination unit 12 does not determine that the driver is in the normal state. Based on, it is estimated whether or not the driver is in a normal state.
In addition, even if the driver-related information acquired from the state determination unit 12 is the normal state driver-related information, the estimation unit 132 determines whether or not the driver is in the normal state based on the estimation information. May be estimated.

The estimation unit 132 outputs the estimated estimation result of whether or not the driver is in a normal state to the output unit 15.

The storage unit 14 stores estimation information. In addition, the storage unit 14 stores information related to the normal state driver.
In the first embodiment, as shown in FIG. 1, the storage unit 14 is provided in the driver state estimation device 1, but this is only an example. The storage unit 14 may be provided in a place outside the driver state estimation device 1 where the driver state estimation device 1 can be referred to.

The output unit 15 outputs the estimation result of whether or not the driver is in a normal state, which is output from the estimation unit 132, to an external device (not shown). The external device is, for example, an automatic driving control device mounted on a vehicle. Even when the vehicle has an automatic driving function, the driver can drive the vehicle by himself / herself without executing the automatic driving function. The automatic driving control device controls the vehicle based on the above estimation result output from the output unit 15. For example, when the automatic driving control device outputs an estimation result that the driver is in a normal state from the output unit 15 in a state where the vehicle is automatically driving, the automatic driving control device starts from the state where the automatic driving is performed. , The operation control method is shifted to the state where the driver performs manual operation. For example, the automatic driving control device stops the vehicle when the output unit 15 outputs an estimation result indicating that the driver is not in a normal state.

The operation of the driver state estimation device 1 according to the first embodiment will be described.
FIG. 3 is a flowchart for explaining the operation of the driver state estimation device 1 according to the first embodiment.
The information collecting unit 11 collects driver-related information (step ST301).
The information collecting unit 11 outputs the driver-related information to the state determination unit 12.

Based on the driver's reaction, the state determination unit 12 determines whether the driver-related information collected by the information collection unit 11 in step ST301 is the driver-related information collected when the driver is in a normal state. Determine (step ST302).

Here, FIG. 4 is a flowchart for explaining the specific operation of step ST302 of FIG.
The inquiry unit 121 of the state determination unit 12 makes an inquiry to the driver regarding the driver's condition (step ST401).
The response acquisition unit 122 of the state determination unit 12 acquires the driver's response to the inquiry made by the inquiry unit 121 in step ST401 (step ST402).
The response acquisition unit 122 outputs the information regarding the acquired response to the determination unit 123. When the response acquisition unit 122 does not acquire the response, the response acquisition unit 122 outputs information to the effect that the response has not been acquired to the determination unit 123.
Based on the driver's reaction, the determination unit 123 of the state determination unit 12 determines whether the driver-related information collected by the information collection unit 11 in step ST301 of FIG. 3 is normal state driver-related information. (Step ST403).

When the determination unit 123 determines the normal state based on the driver's reaction other than the response to the inquiry made by the inquiry unit 121, the operation of steps ST401 and ST402 described above is performed in the driver state estimation device 1. Not done.

Here, FIG. 5 is a flowchart illustrating a specific operation of step ST403 of FIG.
The determination unit 123 determines whether or not the driver's reaction, which indicates that the driver is in a normal state, has been acquired (step ST501).
If the driver does not obtain a reaction indicating that the driver is in the normal state in step ST501 (when “NO” in step ST501), the determination unit 123 does not determine that the driver is in the normal state (step ST504). .. The determination unit 123 outputs the driver-related information to the state estimation unit 13 in association with the information indicating that the driver did not determine the normal state.

When the driver's reaction indicating that the driver is in the normal state is acquired in step ST501 (when “YES” in step ST501), the determination unit 123 indicates that the driver is in the normal state. , It is determined whether or not information other than the driver's reaction has been acquired (step ST502).

When no information other than the driver's reaction, which indicates that the driver is in the normal state, is acquired in step ST502 (when “NO” in step ST502), the determination unit 123 determines that the driver is in the normal state. Is not determined (step ST504). The determination unit 123 outputs the driver-related information to the state estimation unit 13 in association with the information indicating that the driver did not determine the normal state.

When information other than the driver's reaction, which indicates that the driver is in the normal state, is acquired in step ST502 (when “YES” in step ST502), the determination unit 123 determines that the driver is in the normal state. (Step ST503). The determination unit 123 outputs the driver-related information to the state estimation unit 13 in association with the information indicating that the driver has determined that the state is normal. At this time, the state determination unit 12 stores the driver-related information collected by the information collection unit 11 in step ST301, in other words, the normal state driver-related information, in the storage unit 14 for each feature amount.
In the flowchart of FIG. 5, the operation of step ST502 is not essential.

Returning to the description of the flowchart of FIG.
When the state determination unit 12 determines in step ST302 that the driver-related information collected by the information collection unit 11 in step ST301 is normal state driver-related information (when “YES” in step ST303), the state is determined. The update unit 131 of the estimation unit 13 resets the estimation information based on the normal state driver-related information output from the state determination unit 12 (step ST304). The update unit 131 updates the estimation information stored in the storage unit 14 to the estimation information after resetting. After that, the driver state estimation device 1 ends the operation shown in the flowchart of FIG. This is because the state determination unit 12 obtains the reaction of the driver indicating the normal state, and the driver is determined to be in the normal state.
When the state determination unit 12 cannot determine that the driver-related information collected by the information collection unit 11 in step ST301 is normal state driver-related information (when "NO" in step ST303), the update unit 131 ) Does not reset the estimation information. The operation of the driver state estimation device 1 proceeds to step ST305.

The estimation unit 132 of the state estimation unit 13 is in a normal state of the driver based on the driver-related information collected by the information collection unit 11 in step ST301 and the estimation information stored in the storage unit 14. Estimate whether or not (step ST305).
The estimation unit 132 outputs the estimation result of whether or not the driver is in a normal state to the output unit 15.
The output unit 15 outputs the estimation result of whether or not the driver is in the normal state, which was output from the estimation unit 132 in step ST305, to the external device (step ST306).

In this way, the driver state estimation device 1 determines whether the collected driver-related information is the driver-related information collected when the driver is in the normal state, based on the reaction of the driver. When the driver state estimation device 1 determines that the collected driver-related information is the normal state driver-related information collected when the driver is in the normal state, the driver state estimation device 1 is based on the normal state driver-related information. And reset the estimation information. The driver state estimation device 1 estimates whether or not the driver is in the normal state based on the collected driver-related information and the estimation information reset based on the normal state driver-related information. ..
As a result, the driver state estimation device 1 can determine that the driver is driving in a normal state with higher accuracy than the case of determining from the biological information.

Regarding the operation of the driver state estimation device 1 described in the flowchart of FIG. 3, the operations of step ST301 and steps ST305 to ST306 are basically always performed while the driver is driving the vehicle. On the other hand, the operations of steps ST302 to ST304 are performed at appropriate timings. Therefore, when it is not the timing when the operations of steps ST302 to ST304 are performed, the driver state estimation device 1 skips the operations of steps ST302 to ST304.
Further, the operation of the driver state estimation device 1 described in the flowchart of FIG. 3 is repeated when the driver starts the driver of the vehicle, for example, until the driver finishes driving the vehicle.
The driver state estimation device 1 resets the estimation information stored in advance based on the result of determining the normal state, and makes the estimation information the estimation information suitable for the driver. By repeating the resetting of the estimation information, the estimation information can be made more suitable for the driver.

As described above, according to the first embodiment, the driver state estimation device 1 collects information based on the reaction of the driver and the information collecting unit 11 that collects the driver-related information related to the driver of the vehicle. The driver-related information collected by the unit 11 is a state determination unit 12 that determines whether the driver-related information is collected when the driver is in a normal state, and a driver-related information collected by the information collection unit 11. Based on the estimation information reset based on the determination result that the driver-related information is the driver-related information collected when the driver is in the normal state by the state determination unit 12. It is configured to include a state estimation unit 13 for estimating whether or not the driver is in a normal state. Therefore, the driver state estimation device 1 can determine that the driver is driving in a normal state with higher accuracy than the case of determining from the biological information.

Embodiment 2.
In the first embodiment, the driver state estimation device estimates whether or not the driver is in a normal state based on the estimation information that is reset according to the reaction of the driver.
In the second embodiment, the driver is normal based on a trained model in machine learning (hereinafter referred to as "machine learning model") that the driver state estimator has relearned according to the reaction of the driver. An embodiment of estimating whether or not it is in a state will be described.

The driver state estimation device according to the second embodiment is mounted on the vehicle like the driver state estimation device according to the first embodiment.
FIG. 6 is a diagram showing a configuration example of the driver state estimation device according to the second embodiment.
Regarding the configuration of the driver state estimation device according to the second embodiment, the same reference numerals are given to the same configurations as the driver state estimation device described with reference to FIG. 1 in the first embodiment, and duplicate explanations are omitted. do.
The driver state estimation device 1a according to the second embodiment is provided with a learning unit 16 and a model storage unit 17 instead of the storage unit 14 with the driver state estimation device 1 according to the first embodiment. Is different. Further, the driver state estimation device 1a according to the second embodiment is different from the driver state estimation device 1 according to the first embodiment in that the state estimation unit 13a does not include the update unit 131. Further, the specific operation of the estimation unit 132a in the driver state estimation device 1a according to the second embodiment is different from the specific operation of the estimation unit 132 in the driver state estimation device 1 according to the first embodiment.

The driver state estimation device 1a is a machine that inputs driver-related information in advance at the time of shipment of the driver state estimation device 1a and outputs information for estimating whether or not the driver is in a normal state. It has a learning model. The information for estimating whether or not the driver is in a normal state is, for example, information indicating the degree to which the driver is in a normal state (hereinafter referred to as "normal degree"). In the second embodiment, as an example, the machine learning model is a model that has been trained to output the degree of normality represented by a numerical value from "0" to "1" by inputting driver-related information. It is assumed that the greater the degree of normality, the higher the possibility that the driver is in a normal state. The machine learning model is stored in the model storage unit 17.

The learning unit 16 repeats the machine learning model based on the judgment result by the state judgment unit 12 that the driver-related information is the normal state driver-related information and the driver-related information corresponding to the judgment result. Let them learn. That is, the learning unit 16 relearns the machine learning model based on the determination result that the information is related to the normal state driver and the driver-related information corresponding to the determination result.
The learning unit 16 may train a machine learning model by using a known algorithm for supervised learning as a learning algorithm. Specifically, the learning unit 16 may train a machine learning model composed of a neural network by, for example, so-called supervised learning.

Here, supervised learning is a machine learning model in which a set of data of an input and a teacher label is given to a machine learning model as learning data so that the machine learning model learns the characteristics of the input and estimates the result for the input. Refers to the method of constructing.
A neural network is composed of an input layer composed of a plurality of neurons, an intermediate layer composed of a plurality of neurons, and an output layer composed of a plurality of neurons. The middle layer is also called a hidden layer. The intermediate layer may be one layer or two or more layers.
FIG. 7 is a diagram for explaining a neural network.
For example, in the case of a three-layer neural network as shown in FIG. 7, when a plurality of input values are input to the input layers (X1 to X3), the input values are weighted W1 (w11 to w16) to the input values. ) Is applied to the intermediate layers (Y1 to Y2), and the result output from the intermediate layer is further multiplied by the weights W2 (w21 to w26) to be output from the output layers (Z1 to Z3). The output result depends on the values of the weights W1 and W2.

In the second embodiment, in the machine learning model configured by the neural network, the driver-related information collected by the information collecting unit 11 and the driver-related information by the state determination unit 12 are normal state driver-related information. Learning is performed by so-called supervised learning so that the driver's normality is output using the set of normality based on the judgment result as learning data. In the second embodiment, the degree of normality based on the determination result that the information is related to the driver in the normal state is set to the degree of normality "1" indicating that the driver is in the normal state. That is, in the machine learning model, the driver-related information is input to the input layer, and the weight W1 and the weight W1 and the weight W1 and the normal degree of the driver output from the output layer approach "1" indicating that the normal state is in the normal state. Learn by adjusting W2.

When the learning unit 16 outputs the determination result that the driver-related information is the normal state driver-related information from the state determination unit 12, the driver-related information collected by the information collecting unit 11 and the normal degree ". The set of 1 ”is given to the machine learning model as learning data, and the machine learning model is made to perform the above-mentioned learning. More specifically, when the state determination unit 12 outputs the determination result that the driver-related information is the normal state driver-related information, the learning unit 16 relearns the machine learning model stored in advance. Let me.

In the second embodiment, when the state determination unit 12 determines that the driver-related information collected by the information collection unit 11 is the normal state driver-related information, the state determination unit 12 obtains the normal state driver-related information. Instead of storing it in the storage unit 14, it is output to the learning unit 16 in association with the determination result that the information is related to the normal state driver.

In the second embodiment, the estimation unit 132a of the state estimation unit 13a has a normal driver based on the driver-related information collected by the information collection unit 11 and the machine learning model stored in the model storage unit 17. Estimate whether or not it is in a state. Specifically, the estimation unit 132a inputs the driver-related information collected by the information collection unit 11 into the machine learning model, and estimates whether or not the driver is in a normal state. For example, if the normality output from the machine learning model is equal to or higher than a preset threshold value (hereinafter referred to as “degree determination threshold value”), the estimation unit 132a estimates that the driver is in a normal state.
When the learning unit 16 relearns the machine learning model, the estimation unit 132a operates based on the driver-related information collected by the information collecting unit 11 and the machine learning model relearned by the learning unit 16. Estimate whether a person is in a normal state. The estimation unit 132a may acquire the driver-related information collected by the information collection unit 11 from the state determination unit 12.

The model storage unit 17 stores the machine learning model.
In the second embodiment, as shown in FIG. 6, the model storage unit 17 is provided in the driver state estimation device 1a, but this is only an example. The model storage unit 17 may be provided in a place outside the driver state estimation device 1a where the driver state estimation device 1a can be referred to.

The operation of the driver state estimation device 1a according to the second embodiment will be described.
FIG. 8 is a flowchart for explaining the operation of the driver state estimation device 1a according to the second embodiment.
In the second embodiment, the specific operations of steps ST804 to ST805 of FIG. 8 are different from the specific operations of steps ST304 to ST305 of FIG. 3 described in the first embodiment.
Since the specific operations of steps ST801 to ST803 and step ST806 of FIG. 8 are the same as the specific operations of steps ST301 to ST303 and step ST306 of FIG. 3 described in the first embodiment, respectively. , Omit duplicate description. However, in the second embodiment, as a result of the state determination unit 12 performing the normal state determination in step ST802, the driver-related information collected by the information collection unit 11 in step ST801 is the normal state driver-related information. When it is determined, the normal state driver-related information is associated with the determination result and output to the learning unit 16.

When the state determination unit 12 determines in step ST802 that the driver-related information collected by the information collection unit 11 in step ST801 is normal state driver-related information (when “YES” in step ST803), learning is performed. The unit 16 relearns the machine learning model based on the determination result that the information is related to the normal state driver and the driver-related information corresponding to the determination result (step ST804). The driver-related information corresponding to the determination result is the driver-related information determined by the state determination unit 12 to be the normal state driver-related information. Specifically, the information collection unit in the immediately preceding step ST801. 11 is the driver-related information collected. After that, the driver state estimation device 1a ends the operation shown in the flowchart of FIG. This is because the state determination unit 12 determines that the driver is in a normal state.
When the learning unit 16 cannot determine that the driver-related information collected by the information collecting unit 11 in step ST801 is normal state driver-related information (when “NO” in step ST803). ) Does not retrain the machine learning model. The operation of the driver state estimation device 1a proceeds to step ST805.

In the estimation unit 132a of the state estimation unit 13a, the driver is in a normal state based on the driver-related information collected by the information collection unit 11 in step ST801 and the machine learning model stored in the model storage unit 17. It is estimated whether or not there is (step ST805). Specifically, the estimation unit 132a inputs the driver-related information collected by the information collection unit 11 into the machine learning model, and estimates whether or not the driver is in a normal state. For example, if the degree of normality output from the machine learning model is equal to or greater than the degree determination threshold value, the estimation unit 132a estimates that the driver is in a normal state.
The estimation unit 132a outputs an estimation result of whether or not the driver is in a normal state to the output unit 15.

Regarding the operation of the driver state estimation device 1a described in the flowchart of FIG. 8, the operations of step ST801 and steps ST805 to ST806 are basically always performed while the driver is driving the vehicle. On the other hand, the operations of steps ST802 to ST804 are performed at appropriate timings. Therefore, when it is not the timing when the operations of steps ST802 to ST804 are performed, the driver state estimation device 1a skips the operations of steps ST802 to ST804.
Further, the operation of the driver state estimation device 1a described in the flowchart of FIG. 8 is repeated when the driver starts the driver of the vehicle, for example, until the driver finishes driving the vehicle. The driver state estimation device 1a relearns a machine learning model stored in advance based on the result of determining the normal state, thereby making the machine learning model a machine learning model suitable for the driver. By repeating the re-learning of the machine learning model, the accuracy of the machine learning model can be improved and the model can be made more suitable for the driver.

As described above, the driver state estimation device 1a determines whether the collected driver-related information is the driver-related information collected when the driver is in the normal state, based on the reaction of the driver. .. When the driver state estimation device 1a determines that the collected driver-related information is the normal state driver-related information collected when the driver is in the normal state, the driver state estimation device 1a determines that the collected driver-related information is the normal state driver-related information. The machine learning model is retrained based on the judgment result of the above and the driver-related information corresponding to the judgment result. The driver state estimation device 1a estimates whether or not the driver is in a normal state based on the collected driver-related information and the relearned machine learning model.
As a result, the driver state estimation device 1a can determine that the driver is driving in a normal state with higher accuracy than the case of determining from the biological information.

As described above, the model storage unit 17 may be provided in a place outside the driver state estimation device 1a where the driver state estimation device 1a can be referred to. For example, the model storage unit 17 is provided in a server (not shown) connected to the driver state estimation device 1a via a network, and the driver state estimation device 1a acquires a machine learning model from the server. You may. Here, for example, the server is connected to a plurality of driver state estimation devices 1a, and the model storage unit 17 on the server uses a plurality of machine learning models relearned by the plurality of driver state estimation devices 1a. It may be something that you remember. In this case, the driver state estimation device 1a selects a machine learning model from the model storage unit 17 on the server, and estimates whether or not the driver is in a normal state based on the selected machine learning model. To give a specific example, for example, the driver state estimation device 1a selects a machine learning model relearned by the learning unit 16 when the driver has driven in the past from the model storage unit 17, and the driver is normal. Estimate whether or not it is in a state. In this case, the model storage unit 17 stores the machine learning model in association with information that can identify the driver. The driver state estimation device 1a identifies the machine learning model to be selected based on the information that can identify the driver.
In this way, the driver state estimation device 1a selects the machine learning model that was relearned when the driver drove in the past from the model storage unit 17 on the server, so that the driver drives the vehicle. Even when the above is changed, it is possible to estimate whether or not the driver is in a normal state by using a machine learning model that has been relearned according to the driver.

In the above embodiment 2, the learning unit 16 uses a known algorithm for supervised learning as a learning algorithm to train a machine learning model, but this is only an example. For example, the learning unit 16 may use deep learning as a learning algorithm for learning the extraction of the feature amount itself, or may train the machine learning model according to another known method. .. Other known methods are, for example, genetic programming, functional logic programming, or support vector machines.

Further, in the above second embodiment, the learning unit 16 is provided in the driver state estimation device 1a, but this is only an example. The learning unit 16 may be provided in an external device of the driver state estimation device 1a, which is connected to the driver state estimation device 1a.
FIG. 9 shows a configuration example of the driver state estimation device 1b and the learning device 2 when the learning unit 16 is provided in the learning device 2 outside the driver state estimation device 1b in the second embodiment. It is a figure which shows.
The learning device 2 is mounted on the vehicle like the driver state estimation device 1b.

In FIG. 9, the same reference numerals are given to the configurations similar to those of the driver state estimation device 1a shown in FIG.
As shown in FIG. 9, the learning device 2 may include a learning unit 16, an information collecting unit 11, a state determination unit 12, and a model storage unit 17.
The driver state estimation device 1b and the learning device 2 are connected via a network.
In FIG. 9, the driver state estimation device 1b and the learning device 2 each include an information collecting unit 11, but this is only an example. For example, the learning device 2 does not include the information collecting unit 11, and the learning device 2 includes an information acquisition unit (not shown) that acquires the collected driver-related information from the information collecting unit 11 of the driver state estimation device 1b. You may do so.
Further, in FIG. 9, the state determination unit 12 and the model storage unit 17 are provided in the learning device 2, but this is only an example. The state determination unit 12 and the model storage unit 17 may be provided in, for example, the driver state estimation device 1b. The model storage unit 17 may be provided in, for example, a server connected to the driver state estimation device 1b and the learning device 2 via a network.

As described above, according to the second embodiment, the driver state estimation device 1a collects information based on the reaction of the driver and the information collecting unit 11 that collects the driver-related information related to the driver of the vehicle. The driver-related information collected by the unit 11 is the state determination unit 12 that determines whether the driver-related information is collected when the driver is in a normal state, and the driver-related information by the state determination unit 12. The information collecting unit 11 is added to the machine learning model relearned based on the judgment result that the driver is the driver-related information collected when the driver is in the normal state and the driver-related information corresponding to the judgment result. It is configured to include a state estimation unit 13a for inputting the driver-related information collected by the driver and estimating whether or not the driver is in a normal state. Therefore, the driver state estimation device 1b can determine that the driver is driving in a normal state with higher accuracy than the case of determining from the biological information.

In the above-described first and second embodiments, the driver state estimation devices 1, 1a, 1b or the learning device 2 are in-vehicle devices mounted on the vehicle, and the information collecting unit 11 and the state determination are performed. The unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 are provided in the driver state estimation devices 1, 1a, 1b, or the learning device 2. Not limited to this, it is assumed that a part of the information collecting unit 11, the state determination unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 is mounted on the in-vehicle device of the vehicle. The driver state estimation system may be configured by the in-vehicle device and the server, assuming that the other is provided in the server connected to the in-vehicle device via the network.
Further, the information collecting unit 11, the state determination unit 12, the state estimation units 13 and 13a, the output unit 15, and the learning unit 16 may all be provided in the server. In this case, for example, the information collecting unit 11 collects driver-related information from the in-vehicle device via the network, and the output unit 15 determines whether the driver is in a normal state with respect to the in-vehicle device via the network. Outputs the estimation result of whether or not.

10A and 10B are diagrams showing an example of the hardware configuration of the driver state estimation devices 1 and 1a according to the first and second embodiments.
In the first and second embodiments, the functions of the information collection unit 11, the state determination unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 are realized by the processing circuit 1001. That is, the driver state estimation devices 1 and 1a determine whether the collected driver-related information is normal state driver state information based on the driver's reaction, and reset it based on the judgment result. A processing circuit 1001 for performing control for estimating whether or not the driver is in a normal state is provided based on an inference rule or a machine learning model relearned based on the determination result.
The processing circuit 1001 may be dedicated hardware as shown in FIG. 10A, or may be a CPU (Central Processing Unit) 1005 that executes a program stored in the memory 1006 as shown in FIG. 10B.

When the processing circuit 1001 is dedicated hardware, the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable). Gate Array) or a combination of these is applicable.

When the processing circuit 1001 is the CPU 1005, the functions of the information collection unit 11, the state determination unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 are software, firmware, or a combination of software and firmware. Is realized by. That is, the information collection unit 11, the state determination unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 execute the CPU 1005 that executes the program stored in the HDD (Hard Disk Drive) 1002, the memory 1006, or the like. , System LSI (Lage-Scale Integration) or the like, which is realized by a processing circuit 1001. The program stored in the HDD 1002, the memory 1006, or the like causes the computer to execute the procedures or methods of the information collecting unit 11, the state determining unit 12, the state estimating units 13, 13a, the output unit 15, and the learning unit 16. It can be said that. Here, the memory 1006 is, for example, a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Emergency Memory), an EEPROM (Electrically Emergency Memory), a volatile Memory, etc. A semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versaille Disc), or the like is applicable.

Some of the functions of the information collection unit 11, the state determination unit 12, the state estimation units 13, 13a, the output unit 15, and the learning unit 16 are realized by dedicated hardware, and some by software or firmware. It may be realized. For example, the information collecting unit 11 and the output unit 15 are realized by the processing circuit 1001 as dedicated hardware, and the state determination unit 12, the state estimation units 13, 13a, and the learning unit 16 are processed circuit 1001. Can realize the function by reading and executing the program stored in the memory 1006.
The storage unit 14 and the model storage unit 17 use the memory 1006. Note that this is an example, and the storage unit 14 and the model storage unit 17 may be composed of an HDD 1002, an SSD (Solid State Drive), a DVD, or the like.
Further, the driver state estimation devices 1 and 1a include devices such as a server (not shown), an input interface device 1003 and an output interface device 1004 that perform wired communication or wireless communication.

Further, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component in each embodiment.

Since the driver state estimation device of the present disclosure is configured to be able to judge that the driver is driving in a normal state with higher accuracy than the case of judging from biometric information, the driver's state is estimated. It can be applied to a driver state estimation device.

1,1a, 1b Driver state estimation device, 2 Learning device, 11 Information collection unit, 12 State judgment unit, 121 Inquiry unit, 122 Response acquisition unit, 123 Judgment unit, 13, 13a State estimation unit, 131 Update unit, 132 , 132a estimation unit, 14 storage unit, 15 output unit, 16 learning unit, 17 model storage unit, 1001 processing circuit, 1002 HDD, 1003 input interface device, 1004 output interface device, 1005 CPU, 1006 memory.

Claims (15)

1. An information gathering department that collects driver-related information related to the driver of the vehicle,
   Based on the driver's reaction, the driver-related information collected by the information collecting unit includes a state determination unit that determines whether the driver-related information is collected when the driver is in a normal state.
   Based on the driver-related information collected by the information collecting unit and the determination result by the state determination unit that the driver-related information is the driver-related information collected when the driver is in a normal state. A driver state estimation device including a state estimation unit that estimates whether or not the driver is in a normal state based on the estimation information reset in the above.
2. The driver's reaction is the driver's response to an inquiry about the driver's condition.
   Based on the driver's response, the state determination unit determines whether or not the driver-related information collected by the information collection unit is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 1, further comprising determining.
3. The driver's reaction is the driver's response to the passenger's utterance.
   Based on the driver's response, the state determination unit determines whether or not the driver-related information collected by the information collection unit is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 1, further comprising determining.
4. The driver's reaction is the driver's reaction to an external event.
   The state determination unit
   The driver's reaction to the external event is detected based on the in-vehicle image of the inside of the vehicle, and the driver-related information collected by the information collecting unit is based on the detected reaction of the driver. The driver state estimation device according to claim 1, further comprising determining whether or not the information is driver-related information collected when is in a normal state.
5. The driver-related information collected by the information collecting unit includes biometric information of the driver.
   The state determination unit
   The driver-related information collected by the information collecting unit based on the driver's reaction and the biometric information of the driver is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 1, wherein it determines whether or not the driver state is determined.
6. The state determination unit
   Is the driver-related information collected by the information collecting unit based on the reaction time until the driver's reaction is acquired, the driver-related information collected when the driver is in a normal state? The driver state estimation device according to claim 1, further comprising determining whether or not the driver state is estimated.
7. The driver's reaction is due to the driver's utterance.
   The state determination unit is based on the voice quality of the driver's utterance, and is the driver-related information collected by the information collection unit the driver-related information collected when the driver is in a normal state? The driver state estimation device according to claim 1, further comprising determining whether or not the driver state is estimated.
8. An information gathering department that collects driver-related information related to the driver of the vehicle,
   Based on the driver's reaction, the driver-related information collected by the information collecting unit includes a state determination unit that determines whether the driver-related information is collected when the driver is in a normal state.
   The driver-related information by the state determination unit is based on a determination result that the driver-related information is the driver-related information collected when the driver is in a normal state, and the driver-related information corresponding to the determination result. Driver state estimation provided with a state estimation unit that inputs driver-related information collected by the information collection unit into the machine learning model that has been relearned and estimates whether or not the driver is in a normal state. Device.
9. The driver's reaction is the driver's response to an inquiry about the driver's condition.
   Based on the driver's response, the state determination unit determines whether or not the driver-related information collected by the information collection unit is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 8, further comprising determining.
10. The driver's reaction is the driver's response to the passenger's utterance.
    Based on the driver's response, the state determination unit determines whether or not the driver-related information collected by the information collection unit is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 8, further comprising determining.
11. The driver's reaction is the driver's reaction to an external event.
    The state determination unit
    The driver's reaction to the external event is detected based on the in-vehicle image of the inside of the vehicle, and the driver-related information collected by the information collecting unit is based on the detected reaction of the driver. The driver state estimation device according to claim 8, further comprising determining whether or not the information is driver-related information collected when is in a normal state.
12. The driver-related information collected by the information collecting unit includes biometric information of the driver.
    The state determination unit
    The driver-related information collected by the information collecting unit based on the driver's reaction and the biometric information of the driver is the driver-related information collected when the driver is in a normal state. The driver state estimation device according to claim 8, further comprising determining whether or not the driver state is estimated.
13. The state determination unit
    Is the driver-related information collected by the information collecting unit based on the reaction time until the driver's reaction is acquired, the driver-related information collected when the driver is in a normal state? The driver state estimation device according to claim 8, further comprising determining whether or not the driver state is estimated.
14. The driver's reaction is due to the driver's utterance.
    The state determination unit is based on the voice quality of the driver's utterance, and is the driver-related information collected by the information collection unit the driver-related information collected when the driver is in a normal state? The driver state estimation device according to claim 8, further comprising determining whether or not the driver state is estimated.
15. The steps that the information gathering department collects driver-related information related to the driver of the vehicle,
    A step in which the state determination unit determines, based on the driver's reaction, whether the driver-related information collected by the information collection unit is the driver-related information collected when the driver is in a normal state. When,
    The state estimation unit determines that the driver-related information collected by the information collection unit and the driver-related information by the state determination unit are driver-related information collected when the driver is in a normal state. A driver state estimation method including a step of estimating whether or not the driver is in a normal state based on the estimation information reset based on the determination result of.

Images