JP7483160B2 - Occupant monitoring device and occupant monitoring method - Google Patents

Description

The present disclosure relates to an occupant monitoring device and an occupant monitoring method.

A second task is an action other than driving that a driver takes with at least one hand off the steering wheel. When a driver is performing a second task, there is a risk that the driver will not be able to concentrate on driving and will not be able to respond quickly because both hands are not holding the steering wheel. Therefore, a system is being considered to detect and warn when a second task is occurring.
For example, Patent Document 1 discloses a smoking detection device that detects smoking, which is the second task. The smoking detection device of Patent Document 1 captures the face of the driver with an imaging unit provided inside the vehicle, determines the facial orientation of the driver using the captured image, and sets a presence determination region of a cigarette burning part according to the facial orientation. The smoking detection device of Patent Document 1 detects the driver's smoking by searching for a high-temperature region that is a candidate region of the cigarette burning part in the presence determination region.

JP 2009-23400 A

However, the smoking detection device of Patent Document 1 may mistakenly detect the driver's smoking even when the driver does not have a cigarette in his/her mouth. Specifically, this is the case when the driver holds the steering wheel with both hands while holding a cigarette in his/her hand. In this case, the driver's face and the cigarette are actually far apart, but in the captured image, the driver's face and the cigarette are detected as being close to each other, and the driver's smoking is mistakenly detected. Therefore, the smoking detection device of Patent Document 1 has an issue in that the detection accuracy of the second task is insufficient.

The present disclosure has been made to solve the above-mentioned problems, and aims to provide an occupant monitoring device and an occupant monitoring method that can determine the second task with high accuracy.

The occupant monitoring device of the present disclosure includes an image acquisition unit that acquires an image from an imaging unit that images the driver of the vehicle, a first second task determination unit that uses the image to determine whether a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit that determines a gripping state indicating whether the driver is gripping the steering wheel with both hands, and a second second task determination unit that determines whether a second task is being performed based on the determination result of the first second task determination unit and the determination result of the gripping state determination unit.

The occupant monitoring method according to the present disclosure includes the steps of acquiring an image from an imaging unit that images the driver of the vehicle, using the image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, determining a gripping state indicating whether or not the driver is holding the steering wheel with both hands, and determining whether or not the second task is being performed based on the determination result of whether or not the second task is being performed and the determination result of the gripping state.

According to the present disclosure, whether or not a second task is being performed is determined based on the judgment result of the first second task judgment unit and the judgment result of the gripping state judgment unit, thereby making it possible to judge the second task with high accuracy.

FIG. 1 is a block diagram showing a part of a vehicle equipped with an occupant monitoring device according to a first embodiment of the present invention. FIG. 2 is a schematic diagram showing an example of a hardware configuration of the occupant monitoring device according to the first embodiment. FIG. 3 is a flowchart showing the processing operation of the passenger monitoring device according to the first embodiment. FIG. 4 is a flow chart showing the processing operation of the passenger monitoring device according to the second embodiment. FIG. 5 is a flow chart showing the processing operation of the passenger monitoring device according to the third embodiment. FIG. 6 is a block diagram showing a part of a vehicle equipped with an occupant monitoring device according to a fourth embodiment of the present invention. FIG. 7 is a block diagram showing a part of a vehicle equipped with an occupant monitoring device according to a fifth embodiment of the present invention. FIG. 8 is a flow chart showing the processing operation of the passenger monitoring device according to the fifth embodiment.

Embodiment 1.
The passenger monitoring device 20 in the first embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing a part of a vehicle 30 equipped with the passenger monitoring device 20 in the first embodiment. The passenger monitoring device 20 judges whether or not a driver who drives the vehicle 30 is performing a second task. Here, an example of a car will be described as the vehicle 30, but the vehicle 30 may be any vehicle that is operated by a person, such as a motorcycle, a train, or an airplane. The second task is an action other than driving that the driver performs while driving with at least one hand off the handle. For example, the second task is smoking, operating a mobile terminal, talking using a mobile terminal, eating and drinking, checking documents, and reading. As shown in FIG. 1, the vehicle 30 includes a driving determination unit 1, an imaging unit 2, an image acquisition unit 3, a gripping state determination unit 4, a second task determination unit 5, a memory unit 6, an output unit 7, and a display unit 8. The passenger monitoring device 20 includes a driving determination unit 1, an image acquisition unit 3, a gripping state determination unit 4, a second task determination unit 5, a memory unit 6, and an output unit 7. The second task judgment unit 5 has a first second task judgment unit 5a and a second second task judgment unit 5b.

The driving determination unit 1 acquires driving information from the vehicle 30 indicating whether the vehicle 30 is driving or not. The driving determination unit 1 then uses the driving information to determine whether the vehicle 30 is driving or not, and outputs the determination result to the first second task determination unit 5a. The driving information is, for example, information indicating the vehicle speed. For example, the driving determination unit 1 determines that the vehicle 30 is not driving if the vehicle speed is 0 km/h, and determines that the vehicle 30 is driving if the vehicle speed is not 0 km/h.

The imaging unit 2 is a camera provided inside the vehicle 30, and is installed on the instrument panel, dashboard, center display, A-pillar, or rearview mirror of the vehicle 30. The imaging unit 2 outputs an image of the driver of the vehicle 30 to the image acquisition unit 3.

The image acquisition unit 3 acquires an image from the imaging unit 2, which images the driver of the vehicle 30.

The gripping state determination unit 4 determines the gripping state indicating whether the driver is gripping the steering wheel with both hands. In addition, when the gripping state determination unit 4 determines that the driver is not gripping the steering wheel with both hands, it may further determine which of the following gripping states the driver is in: a state in which the driver is gripping the steering wheel with the right hand, a state in which the driver is gripping the steering wheel with the left hand, or a state in which the driver is not gripping the steering wheel. The gripping state is a state in which the driver's hands are touching the steering wheel. Hereinafter, the state in which the driver is gripping the steering wheel with both hands is referred to as "both hands", the state in which the driver is gripping the steering wheel with the right hand is referred to as "right hand", the state in which the driver is gripping the steering wheel with the left hand is referred to as "left hand", and the state in which the driver is not gripping the steering wheel with both hands is referred to as "no grip". The gripping state determination unit 4 may determine the state in which the driver is gripping the steering wheel with one hand as "one hand" without distinguishing between the "right hand" and the "left hand".

The method by which the gripping state determination unit 4 determines the gripping state will be explained. The gripping state determination unit 4 detects multiple feature points of the driver in the captured image. The feature points are used to generate a so-called skeletal model. The gripping state determination unit 4 then generates a skeletal model of the driver using the detected multiple feature points. Next, the gripping state determination unit 4 uses the generated skeletal model to detect the inclination angle of the driver's shoulders or the inclination angle of the driver's arms. This allows the gripping state determination unit 4 to estimate the direction in which the driver's arms are pointing, and to estimate whether the driver's hands are gripping the steering wheel.

The method by which the gripping state determination unit 4 determines the gripping state is not limited to the above-mentioned method. The gripping state determination unit 4 may determine whether the driver's hands are touching the steering wheel from the captured image, or may determine whether the driver's hands are closer to the steering wheel than a predetermined position.

The second task judgment unit 5 judges whether or not a second task is being performed. The second task judgment unit 5 has a first second task judgment unit 5a and a second second task judgment unit 5b. The first second task judgment unit 5a judges whether or not a second task is being performed when the driving judgment unit 1 judges that the vehicle 30 is driving. The second second task judgment unit 5b judges whether or not a second task is being performed based on the judgment result of the first second task judgment unit 5a and the judgment result of the gripping state judgment unit 4.

The first second task determination unit 5a determines whether or not a second task is being performed using the captured image acquired by the image acquisition unit 3. When the first second task determination unit 5a determines that a second task is being performed, the first second task determination unit 5a may further determine the type of the second task. The type of the second task is at least one of smoking, operating a mobile terminal, conversation using a mobile terminal, eating and drinking, checking documents, and reading.

A method for determining whether or not a second task is being performed by the first second task determination unit 5a and the type of the second task will be described. The determination by the first second task determination unit 5a can be performed, for example, by pattern matching processing. The first second task determination unit 5a stores in advance pattern images used in the pattern matching processing. The pattern images are, for example, an image of a person smoking, an image of a person operating a mobile terminal, an image of a person having a conversation using a mobile terminal, an image of a person eating and drinking, an image of a person checking a document, an image of a person reading, an image of a cigarette, an image of a mobile terminal, an image of food such as rice balls, an image of a container such as a plastic bottle containing a drink, an image of a document, and an image of a book. The first second task determination unit 5a then determines whether or not a second task is being performed and the type of the second task by calculating the similarity between a part of the captured image acquired by the image acquisition unit 3 and the pattern image.

The first second task determination unit 5a may perform the determination using a known algorithm for supervised learning. As an example, a case where a neural network is applied to the learning algorithm will be described. The first second task determination unit 5a determines whether or not a second task is being performed and the type of the second task by so-called supervised learning. Supervised learning refers to a method in which a set of data of input and result (teacher label) is provided to a learning device as learning data, the data is learned from the features of the learning data, and a result is inferred from the input.
The first second task judgment unit 5a has a machine learning model generated in advance according to learning data in which image information and the type of second task are combined, and judges the second task by inputting the captured image acquired by the image acquisition unit 3 into the machine learning model.
As a learning algorithm for generating the machine learning model, deep learning may be used, or the machine learning model may be trained according to other known methods, such as genetic programming, functional logic programming, or support vector machines.

The method of determining whether or not a second task is being performed by the first second task determination unit 5a and the type of the second task is not limited to the above-mentioned method. For example, the imaging unit 2 has an infrared camera, and the image acquisition unit 3 acquires information on the heat distribution of the area imaged by the imaging unit 2. Then, the first second task determination unit 5a may determine whether or not the second task of smoking is being performed by determining the presence or absence of a cigarette based on the heat distribution information.

As described above, the second task is an action other than driving that the driver performs while driving with at least one hand removed from the steering wheel. Therefore, for example, the action of the driver holding a cigarette in one hand and gripping the steering wheel with both hands is not a second task. In order to avoid erroneously determining such an action as a second task, the first second task determination unit 5a may determine that the second task of smoking is being performed when the distance between the driver's mouth and the cigarette estimated from the captured image is equal to or less than a preset threshold, for example, 2 cm or less. For example, a method of estimating the distance between the driver's mouth and the cigarette estimated from the captured image may be to obtain the coordinate distance between the driver's mouth and the cigarette and the coordinate distance between the driver's eyes from the captured image, and estimate the distance between the driver's mouth and the cigarette based on the standard length between the human eyes. Similarly, for other second tasks, the first second task determination unit 5a may determine that the second task of conversation using a mobile device is being performed when the distance between the driver's face estimated from the captured image is equal to or less than a preset threshold, for example, 5 cm or less.

The second second task judgment unit 5b judges, depending on the gripping state judged by the gripping state judgment unit 4, whether or not to cause the first second task judgment unit 5a to carry out a judgment, or whether or not to validate the judgment result by the first second task judgment unit 5a.
In this embodiment, when the first second task determination unit 5a determines that the second task is being performed, and the gripping state determination unit 4 determines that the gripping state is not with "both hands," the second second task determination unit 5b validates the determination result of the first second task determination unit 5a. When the first second task determination unit 5a determines that the second task is being performed, and the gripping state determination unit 4 determines that the gripping state is with "both hands," the second second task determination unit 5b invalidates the determination result of the first second task determination unit 5a.

The gripping state determination unit 4 and the second task determination unit 5 perform the determination at preset time intervals. The preset time intervals are, for example, 0.5 seconds.
The storage unit 6 stores the judgment result of the second task judgment unit 5. The storage unit 6 can store a preset number of past judgment results, and stores at least the judgment results of the second task judgment unit 5 for the previous and latest captured images. The storage unit 6 may further store the judgment result of the gripping state judgment unit 4.

The output unit 7 outputs a signal indicating that the second task is being performed to the display unit 8. The display unit 8 is, for example, a speaker or a display provided in the vehicle 30. When the display unit 8 receives a signal from the output unit 7, it displays at least one of a sound and an image prompting the user to end the performance of the second task.

Next, an example of the hardware configuration of the occupant monitoring device 20 will be described. Figure 2 is a schematic diagram showing an example of the hardware configuration of the occupant monitoring device 20 of embodiment 1. The driving determination unit 1, image acquisition unit 3, gripping state determination unit 4, second task determination unit 5, memory unit 6, and output unit 7 of the occupant monitoring device 20 may be a processing circuit 12 which is dedicated hardware as shown in Figure 2A, or a processor 13 which executes a program stored in memory 14 as shown in Figure 2B.

2A, when each component of the occupant monitoring device 20 is a dedicated hardware, the processing circuit 12 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-programmable Gate Array), or a combination of these. The functions of each component of the occupant monitoring device 20 may be realized by the processing circuit 12, or the functions of each component may be realized together by a single processing circuit 12.

As shown in FIG. 2B, when each component of the occupant monitoring device 20 is a processor 13, the function of each component is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 14. The processor 13 realizes the function of each component of the occupant monitoring device 20 by reading and executing the program stored in the memory 14. That is, each component of the occupant monitoring device 20 has a memory 14 for storing a program that, when executed by the processor 13, results in the execution of each step shown in FIG. 3, FIG. 4, FIG. 5, or FIG. 8 described below. It can also be said that these programs cause a computer to execute the procedure or method of each component of the occupant monitoring device 20.

Here, the processor 13 is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 14 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), or a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

In addition, some of the functions of each component of the occupant monitoring device 20 may be realized by dedicated hardware and some by software or firmware. In this way, the processing circuit 12 in the occupant monitoring device 20 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

Next, we will explain the occupant monitoring method using the occupant monitoring device 20. Figure 3 is a flowchart showing the processing operation of the occupant monitoring device 20 in embodiment 1.

In step S101, the driving determination unit 1 determines whether the vehicle 30 is driving. If the driving determination unit 1 determines that the vehicle 30 is driving (YES in step S101), it outputs the determination result to the first second task determination unit 5a. If the driving determination unit 1 determines that the vehicle 30 is not driving (NO in step S101), the driving determination unit 1 repeats step S101 until it determines that the vehicle 30 is driving. In other words, since the second task is permitted to be performed while the vehicle 30 is stopped, the occupant monitoring device 20 does not perform a determination of the second task.

In step S102, the second task judgment unit 5 causes the image acquisition unit 3 to acquire an image from the imaging unit 2, which images the driver.

In step S103, the gripping state determination unit 4 uses the captured image acquired by the image acquisition unit 3 to determine the gripping state indicating whether the driver is gripping the steering wheel with both hands.

In step S104, the first second task determination unit 5a uses the captured image acquired by the image acquisition unit 3 to determine whether or not the second task is being performed. If the first second task determination unit 5a determines that the second task is being performed, it may further determine the type of the second task. If the first second task determination unit 5a determines that the second task is being performed (YES in step S104), the process proceeds to step S105. If the first second task determination unit 5a determines that the second task is not being performed (NO in step S104), the process proceeds to step S106.

In steps S105 and S106, the memory unit 6 stores the judgment result of the first second task judgment unit 5a. After the processing operation of step S105, the process proceeds to step S107, and after the processing operation of step S106, the process proceeds to step S110.

In steps S107 to S110, the second second task judgment unit 5b judges whether or not the second task is being performed based on the judgment result of the first second task judgment unit 5a and the judgment result of the grasping state judgment unit 4.

In step S107, the second second task determination unit 5b uses the judgment result of the grip state judgment unit 4 to judge whether the grip state satisfies a preset condition. The preset condition of the grip state in this embodiment is a state in which the driver grips the steering wheel with both hands, that is, the grip state is "both hands". If the grip state judgment unit 4 judges that the grip state is "both hands", the driver is gripping the steering wheel with both hands, so both hands are physically occupied. Therefore, the result of the first second task judgment unit 5a judging that the second task is being performed (YES in step S104) is incorrect. Therefore, in step S108, the second second task judgment unit 5b judges that the judgment result of the first second task judgment unit 5a is invalid. Then, in step S109, the second second task judgment unit 5b corrects the judgment result of the first second task judgment unit 5a stored in the memory unit 6 in step S105. More specifically, the second second task judgment unit 5b corrects the latest judgment result of the second task judged by the first second task judgment unit 5a from "the second task is being performed" to "the second task is not being performed."

If the grip state determination unit 4 determines that the grip state is not "both hands" (NO in step S107), the second second task determination unit 5b determines that the judgment result of the first second task determination unit 5a is valid (step S110). Also, in step S104, even if the first second task determination unit 5a determines that the second task is not being performed, the second second task determination unit 5b determines that the judgment result of the first second task determination unit 5a is valid (step S110). If the second second task determination unit 5b determines that the judgment result of the first second task determination unit 5a is valid (step S110), there is no need to perform a processing operation to correct the judgment result of the first second task determination unit 5a stored in steps S105 and S106, so the process proceeds to step S111.

In step S111, the second second task determination unit 5b determines whether the second task has been continuously performed for a preset time in the determination result of the second task stored in the memory unit 6. There are two cases where the second task is determined to be continuously performed for a preset time in the second second task determination unit 5b (YES in step S111). The first case is when the state in which the second task has been performed continues for a preset time when the first second task determination unit 5a determines whether the second task is being performed. The second case is when the same type of second task has been continuously performed for a preset time in the determination result stored in the memory unit 6 when the first second task determination unit 5a determines whether the second task is being performed and the type of the second task. The preset time may be set to a time when it is expected that the driver's concentration on driving will decrease and that he or she will not be able to respond immediately because both hands are not holding the steering wheel if the second task is continuously performed. In addition, the time when it is expected that the driver's concentration on driving will decrease and that he or she will not be able to respond immediately because both hands are not holding the steering wheel differs depending on the type of the second task. Therefore, the preset time may be set differently depending on the type of the second task. For example, the preset time is set to 2 to 3.5 seconds for the second task of smoking, and 3 to 4.5 seconds for the second task of talking using a mobile terminal. In step S109, if the second task is not performed continuously for the preset time, the processing operation of the occupant monitoring device 20 is terminated.

In step S112, the second second task determination unit 5b causes the output unit 7 to output a signal indicating that the second task is being performed to the display unit 8. Then, the display unit 8 displays at least one of a sound and an image that prompts the driver to end the performance of the second task.
This completes the processing operation of the occupant monitoring device 20.

Next, the effects of the occupant monitoring device 20 in this embodiment will be described.
As described above, the occupant monitoring device 20 in this embodiment includes a first second task determination unit 5a that uses a captured image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit 4 that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands, and a second second task determination unit 5b that determines whether or not the second task is being performed based on the determination results of the first second task determination unit 5a and the determination results of the gripping state determination unit 4. This enables the occupant monitoring device 20 to perform highly accurate determination of the second task.

The effects of the passenger monitoring device 20 of the present embodiment described above will now be described in more detail.
For example, suppose that the first second task determination unit 5a determines that the second task of smoking is being performed using a captured image. However, because a cigarette is small compared to a steering wheel, there is a possibility that the second task of smoking is erroneously determined to be performed even when the cigarette is not actually captured in the captured image. In addition, it is difficult to accurately estimate the distance between the cigarette and the driver's mouth from a two-dimensional captured image. Therefore, for example, even when a driver holds a cigarette in his or her hand and grips the steering wheel with both hands, the position of the driver's face and the cigarette are determined to be close on the captured image, so that there is a possibility that the second task of smoking is erroneously determined to be performed. Here, the second task of smoking is described as an example, but the same applies to other second tasks. In other words, the determination result using the captured image in the first second task determination unit 5a may be an erroneous determination.

In this embodiment, an example has been shown in which the grip state determination unit 4 determines the grip state using a captured image. Here, since the handle is larger than, for example, a cigarette, the grip state can be determined with higher accuracy than the second task of a cigarette. In other words, the judgment result of the grip state determination unit 4 is more accurate than the judgment result of the first second task determination unit 5a. Therefore, the second second task determination unit 5b determines whether or not the second task is being performed, taking into account the judgment result of the grip state determination unit 4 in addition to the judgment result of the first second task determination unit 5a. This allows the second task to be determined with higher accuracy.

Furthermore, the second second task determination unit 5b determines whether or not to validate the determination result by the first second task determination unit 5a depending on the gripping state. More specifically, when the first second task determination unit 5a determines that the second task is being performed, if the gripping state determination unit 4 determines that the gripping state is not with "both hands," the second second task determination unit 5b validates the determination result of the first second task determination unit 5a, and invalidates the determination result of the first second task determination unit 5a if the gripping state determination unit 4 determines that the gripping state is with "both hands."
When the driver is holding the steering wheel with both hands, the driver's hands are occupied and the driver is therefore unable to perform the second task. Therefore, when the first second task determination unit 5a determines that the second task is being performed when the driver is holding the steering wheel with both hands, the determination result of the first second task determination unit 5a is erroneous. Therefore, the second second task determination unit 5b invalidates the determination result of the first second task determination unit 5a, thereby enabling the second task to be determined with higher accuracy.

A second task is an action that the driver performs with at least one hand off the steering wheel, and is at least one of the following: smoking, operating a mobile device, talking using a mobile device, eating, drinking, checking documents, and reading. When these second tasks are performed, the driver's concentration on driving decreases and there is a risk that the driver will not be able to respond immediately because both hands are not holding the steering wheel. The occupant monitoring device 20 of this embodiment can accurately determine second tasks that cause the driver's concentration on driving to decrease and there is a risk that the driver will not be able to respond immediately because both hands are not holding the steering wheel.

Furthermore, when the second task has been continuously performed for a preset time, the second second task determination unit 5b causes the output unit 7 to output a signal indicating that the second task is being performed to the display unit 8. In response to the signal from the output unit 7, the display unit 8 provided in the vehicle 30 displays at least one of a sound and an image urging the driver to end the performance of the second task. This urges the driver to end the performance of the second task, allowing the driver to concentrate on driving and strive for safe driving, such as by holding the steering wheel with both hands.

In addition, if the output unit 7 outputs a signal to the display unit 8 each time the first second task determination unit 5a determines that the second task is being performed, the display unit 8 displays the signal too many times, which the driver finds annoying. Therefore, the second second task determination unit 5b causes the output unit 7 to output a signal when the second task is being performed continuously for a preset time. This allows the occupant monitoring device 20 to display the end of the second task when the second task is being performed continuously for a time when it is expected that the driver's concentration on driving will decrease and that he or she will not be able to respond immediately because he or she is not holding the steering wheel with both hands. Therefore, the occupant monitoring device 20 can suppress the annoyance caused by the display unit 8 displaying the signal too many times.

The occupant monitoring device 20 also includes a storage unit 6 that stores the results of the judgments made by the first second task judgment unit 5a and the second second task judgment unit 5b. The first second task judgment unit 5a judges the type of the second task when it is judged that the second task is being performed. The second second task judgment unit 5b outputs a signal to the output unit 7 when the judgment results stored in the storage unit 6 indicate that the same type of second task has been performed continuously for a preset time. The preset time differs depending on the type of second task. Depending on the type of second task, the amount of time that the driver needs to continue performing the second task to lose concentration on driving and to be unable to respond immediately because both hands are not holding the steering wheel differs. Therefore, by setting the preset time to be different depending on the type of second task, the driver can be prompted to end the performance of the second task at the optimal timing for the type of second task being performed.

In addition, in embodiment 1, an example is shown in which the occupant monitoring device 20 is equipped with a memory unit 6, but the judgment result of the second task judgment unit 5 may be transmitted to and stored in a server device provided outside the vehicle 30.

As an example of using the second task determination result by the occupant monitoring device 20, the output unit 7 outputs a signal indicating that the second task is being performed to the display unit 8, and the display unit 8 prompts the driver to end the performance of the second task, but this is not limited to this. For example, the second task determination result by the occupant monitoring device 20 can be used by a company that operates commercial vehicles such as trucks and taxis. The company that operates the commercial vehicles is equipped with a server device for managing the commercial vehicles. The output unit 7 may then output a signal indicating that the second task is being performed to the server device. This allows the management company to monitor whether employees who are drivers of commercial vehicles are performing the second task.

Embodiment 2.
An occupant monitoring device 20 in the embodiment 2 will be described. The configuration of the occupant monitoring device 20 in the embodiment 2 is similar to the configuration of the occupant monitoring device 20 in the embodiment 1. The same components as those in the embodiment 1 are denoted by the same reference numerals.

In this embodiment, the second second task judgment unit 5b validates the judgment result of the first second task judgment unit 5a when the state of the second task changes and the grasping state does not satisfy the preset condition, and invalidates the judgment result of the first second task judgment unit 5a when the state of the second task changes and the grasping state satisfies the preset condition.

A change in the state of the second task refers to a change in whether or not the second task is being performed, or a change in the type of the second task. Specifically, this includes a change from a state in which the second task is not being performed to a state in which the second task is being performed, and a change from the second task of smoking to the second task of talking using a mobile device.

The preset conditions for the gripping state in this embodiment are at least one of the three shown in Table 1.

Table 1 Predefined gripping conditions

The first gripping state condition is when the driver continues to grip the steering wheel with both hands, i.e., when the previous judgment result of the gripping state judgment unit 4 was "both hands" and the latest judgment result of the gripping state judgment unit 4 is "both hands." When the driver continues to grip the steering wheel with both hands, both of the driver's hands are physically occupied, and there is no possibility of the state of the second task changing.

The second gripping state condition is when the driver changes from a state where he/she grips the steering wheel with both hands to a state where he/she does not grip the steering wheel with both hands, that is, when the previous judgment result of the gripping state judgment unit 4 was "both hands" and the latest judgment result of the gripping state judgment unit 4 is either "right hand", "left hand", "one hand" or "no grip". The second pre-set gripping state condition is set when the pre-set time interval judged by the gripping state judgment unit 4 is sufficiently short, for example, shorter than 0.5 seconds. For example, the action of releasing the hands from the steering wheel and the change of the state of the second task cannot be performed simultaneously in a time shorter than 0.5 seconds. If the pre-set time interval judged by the gripping state judgment unit 4 and the first second task judgment unit 5a stored in the memory unit 6 judged by the gripping state judgment unit 4 is long enough to allow the action of releasing the hands from the steering wheel and the change of the state of the second task to be performed simultaneously, for example, longer than 0.5 seconds, the second gripping state condition does not need to be set.

The third gripping state condition is when the driver changes from a state where he/she is not gripping the steering wheel with both hands to a state where he/she is gripping the steering wheel with both hands, that is, when the previous judgment result of the gripping state judgment unit 4 was either "right hand", "left hand", "one hand" or "not gripping" and the latest judgment result of the gripping state judgment unit 4 is "both hands". When the driver changes from a state where he/she is not gripping the steering wheel with both hands to a state where he/she is gripping the steering wheel with both hands, the action of gripping the steering wheel with both hands and the change of the state of the second task cannot be performed simultaneously.

Next, we will explain the occupant monitoring method using the occupant monitoring device 20. Figure 4 is a flowchart showing the processing operation of the occupant monitoring device 20 in embodiment 2.

Steps S201 and S202 are similar to steps S101 and S102, respectively, in the embodiment.

In step S203, the first second task determination unit 5a determines whether or not a second task is being performed, using the captured image acquired by the image acquisition unit 3. If the first second task determination unit 5a determines that a second task is being performed, it may further determine the type of the second task.

In step S204, the grip state determination unit 4 determines the grip state indicating whether the driver is gripping the steering wheel with both hands. Steps S203 and S204 may be performed simultaneously, or step S203 may be performed after step S204.

In step S205, the memory unit 6 stores the judgment result of the first second task judgment unit 5a in step S203 and the judgment result of the grasping state judgment unit 4 in step S204.

In steps S206 to S210, the second second task judgment unit 5b judges whether or not the second task is being performed based on the judgment result of the first second task judgment unit 5a stored in the memory unit 6 and the judgment result of the grasping state judgment unit 4.

First, in step S206, the second second task judgment unit 5b judges whether the state of the second task has changed. That is, the second second task judgment unit 5b judges that the state of the second task has changed if the latest judgment result of the first second task judgment unit 5a is different from the previous judgment result of the first second task judgment unit 5a, and judges that the state of the second task has not changed if the latest judgment result of the first second task judgment unit 5a is the same as the previous judgment result of the first second task judgment unit 5a. A change in the state of the second task means a change in the state of whether the second task is being performed or a change in the type of the second task.

If the state of the second task has changed (YES in step S206), the second second task determination unit 5b determines whether the gripping state satisfies a preset condition (step S207). The preset condition of the gripping state is at least one of the three conditions shown in Table 1 above.

If the grip state satisfies the preset condition (YES in step S207), the second second task judgment unit 5b judges the judgment result of the first second task judgment unit 5a to be invalid (step S208), and proceeds to step S209. In step S209, the second second task judgment unit 5b corrects the judgment result of the first second task judgment unit 5a stored in the memory unit 6 in step S205. More specifically, the second second task judgment unit 5b deletes the judgment result of the latest second task judged by the first second task judgment unit 5a and sets it to no data, or corrects it to the same judgment result as the previous judgment result of the second task.

If the grip state does not satisfy the preset condition (NO in step S207), the second second task judgment unit 5b judges that the judgment result of the first second task judgment unit 5a is valid (step S210). Also, if the state of the second task does not change in step S206, the second second task judgment unit 5b judges that the judgment result of the first second task judgment unit 5a is valid (step S210).

Steps S211 and S212 are similar to steps S111 and S112, respectively, in the embodiment.
This completes the processing operation of the occupant monitoring device 20.

As in the first embodiment, the occupant monitoring device 20 in this embodiment includes a first second task determination unit 5a that uses a captured image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit 4 that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands, and a second second task determination unit 5b that determines whether or not the second task is being performed based on the determination results of the first second task determination unit 5a and the determination results of the gripping state determination unit 4. This allows the occupant monitoring device 20 to perform highly accurate determination of the second task.

Furthermore, the second second task determination unit 5b determines whether or not to validate the determination result by the first second task determination unit 5a depending on the gripping state. More specifically, when the state of the second task changes and the gripping state does not satisfy a preset condition, the second second task determination unit 5b validates the determination result of the first second task determination unit 5a, and invalidates the determination result of the first second task determination unit 5a when the state of the second task changes and the gripping state satisfies the preset condition.
The second second task judgment unit 5b judges that the state of the second task has changed if the latest judgment result of the first second task judgment unit 5a is different from the previous judgment result of the first second task judgment unit 5a, and judges that the state of the second task has not changed if the latest judgment result of the first second task judgment unit 5a is the same as the previous judgment result of the first second task judgment unit 5a.
The preset condition of the gripping state is a condition under which the state of the second task is unlikely to change. Therefore, if the gripping state is in the preset condition when the state of the second task changes, the judgment result of the first second task judgment unit 5a is erroneous. Therefore, the second second task judgment unit 5b invalidates the judgment result of the first second task judgment unit 5a, thereby making it possible to more accurately judge the second task.

Specifically, the preset conditions for the gripping state are at least one of the following: the driver continues to grip the steering wheel with both hands, the state changes from the driver gripping the steering wheel with both hands to the state where the driver is not gripping the steering wheel with both hands, and the state changes from the driver not gripping the steering wheel with both hands to the state where the driver is gripping the steering wheel with both hands. These conditions are conditions under which the state of the second task is not likely to change.

Embodiment 3.
An occupant monitoring device 20 in embodiment 3 will be described. The configuration of the occupant monitoring device 20 in embodiment 3 is similar to the configuration of the occupant monitoring device 20 in embodiment 1 or embodiment 2. The same components as those in embodiment 1 and embodiment 2 are denoted by the same reference numerals.

In the occupant monitoring device 20 of embodiments 1 and 2, after the first second task judgment unit 5a judges the second task and the grasping state judgment unit 4 judges the grasping state, a judgment is made by the second second task judgment unit 5b.
In the occupant monitoring device 20 of the third embodiment, after the grasping state determination unit 4 determines the grasping state, the second second task determination unit 5b performs a determination. Then, the second second task determination unit 5b determines whether or not to cause the first second task determination unit 5a to perform a determination depending on the grasping state. More specifically, if the grasping state does not satisfy a preset condition, the second second task determination unit 5b causes the first second task determination unit 5a to perform a determination, and if the grasping state satisfies the preset condition, the second second task determination unit 5b does not cause the first second task determination unit 5a to perform a determination.

We will now explain the occupant monitoring method using the occupant monitoring device 20. Figure 5 is a flowchart showing the processing operation of the occupant monitoring device 20 in embodiment 3.

Steps S301 to S303 are the same as steps S101 to S103, respectively, in embodiment 1.

In step S304, the memory unit 6 stores the judgment result of the gripping state judgment unit 4.

In step S305, the second second task determination unit 5b determines whether the gripping state satisfies a preset condition. A detailed description of step S305 is omitted, but it is similar to step S207 in embodiment 2. The preset condition of the gripping state is at least one of the following: the driver continues to grip the steering wheel with both hands, the driver changes from a state in which the driver grips the steering wheel with both hands to a state in which the driver is not gripping the steering wheel with both hands, and the driver changes from a state in which the driver is not gripping the steering wheel with both hands to a state in which the driver is gripping the steering wheel with both hands.

If the gripping state satisfies the preset condition (YES in step S305), the second second task judgment unit 5b terminates the processing operation of the occupant monitoring device 20 without causing the first second task judgment unit 5a to perform a judgment.

If the gripping state does not satisfy the preset condition (NO in step S305), the second second task determination unit 5b causes the first second task determination unit 5a to perform a determination (step S306). Next, the first second task determination unit 5a determines whether or not a second task is being performed (step S307). In addition, in step S307, the first second task determination unit 5a may further perform a determination of the type of the second task if the second task is being performed.

Then, in step S308, the memory unit 6 stores the judgment result of the first second task judgment unit 5a.

Steps S309 and S310 are similar to steps S111 and S112 in the first embodiment, respectively.
This completes the processing operation of the occupant monitoring device 20.

As in the first embodiment, the occupant monitoring device 20 in this embodiment includes a first second task determination unit 5a that uses a captured image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit 4 that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands, and a second second task determination unit 5b that determines whether or not the second task is being performed based on the determination results of the first second task determination unit 5a and the determination results of the gripping state determination unit 4. This allows the occupant monitoring device 20 to perform highly accurate determination of the second task.

In addition, the second second task judgment unit 5b judges whether or not to cause the first second task judgment unit 5a to perform judgment depending on the gripping state. More specifically, the second second task judgment unit 5b causes the first second task judgment unit 5a to perform judgment when the gripping state does not satisfy a preset condition, and does not cause the first second task judgment unit 5a to perform judgment when the gripping state satisfies the preset condition. This makes it possible to prevent erroneous judgment by the first second task judgment unit 5a when the gripping state satisfies the preset condition, thereby making it possible to perform judgment of the second task with high accuracy.

In addition, if the grip state satisfies a preset condition, the state of the second task does not change, so the second second task determination unit 5b can omit the determination process in the first second task determination unit 5a. This reduces the processing load on the occupant monitoring device 20.

Embodiment 4.
The occupant monitoring device 20 in the fourth embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing a part of a vehicle 30 equipped with the occupant monitoring device 20 in the fourth embodiment. In the first embodiment, an example was shown in which the gripping state determination unit 4 determines the gripping state using a captured image. In the fourth embodiment, the gripping state determination unit 4 determines the gripping state using a hands-on signal from the hands-on sensor 9. The rest of the configuration of the occupant monitoring device 20 is the same as any one of the first to third embodiments. The same reference numerals are used for the same configurations as those in the first to third embodiments.

As in the first embodiment, the gripping state determination unit 4 determines the gripping state indicating whether or not the driver is gripping the steering wheel with both hands. That is, the gripping state determination unit 4 determines which of the gripping states is "both hands," "right hand," "left hand," "one hand," or "not gripping."

The hands-on sensor 9 is provided on the steering wheel of the vehicle 30, and outputs a hands-on signal to the grip state determination unit 4 indicating one of the following states: when both hands of the driver are touching the steering wheel, when the right hand is touching, when the left hand is touching, when one hand is touching, and when neither hand is touching the steering wheel. Here, an example is shown in which the hands-on sensor 9 is a pressure sensor.

The hands-on sensor 9 is provided on the left and right sides of the steering wheel. The hands-on sensor 9 may be provided on the entire left and right halves of the steering wheel, or may be provided only at a location that is estimated to be gripped. The hands-on sensor 9 detects a current value according to pressure, and when it detects a current value equal to or greater than a preset threshold, it determines that the driver's hands are touching the steering wheel. For example, when both of the hands-on sensors 9 provided on the left and right sides of the steering wheel detect a current value equal to or greater than a preset threshold, the hands-on sensor 9 outputs a hands-on signal indicating that both of the driver's hands are touching the steering wheel. When only the hands-on sensor 9 provided on the right side of the steering wheel detects a current value equal to or greater than a preset threshold, the hands-on sensor 9 outputs a hands-on signal indicating that the driver's right hand or one hand is touching the steering wheel. When only the hands-on sensor 9 provided on the left side of the steering wheel detects a current value equal to or greater than a preset threshold, the hands-on sensor 9 outputs a hands-on signal indicating that the driver's left hand or one hand is touching the steering wheel. If both of the hands-on sensors 9, installed on the left and right sides of the steering wheel, detect a current value above a predetermined threshold value, the hands-on sensor 9 outputs a hands-on signal indicating that neither of the driver's hands are touching the steering wheel.
The gripping state determination unit 4 determines the gripping state using a hands-on signal from the hands-on sensor 9.

The occupant monitoring method by the occupant monitoring device 20 is the same as the occupant monitoring method by the occupant monitoring device 20 in the first to third embodiments. As described above, in steps S107, S207, and S305, the gripping state determination unit 4 determines the gripping state indicating whether or not the driver is gripping the steering wheel with both hands. At that time, the gripping state determination unit 4 determines the gripping state using the hands-on signal of the hands-on sensor 9.

As in the first embodiment, the occupant monitoring device 20 in this embodiment includes a first second task determination unit 5a that uses a captured image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit 4 that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands, and a second second task determination unit 5b that determines whether or not the second task is being performed based on the determination results of the first second task determination unit 5a and the determination results of the gripping state determination unit 4. This allows the occupant monitoring device 20 to perform highly accurate determination of the second task.

The gripping state determination unit 4 also determines the gripping state using a hands-on signal from the hands-on sensors 9, which are pressure sensors provided on the left and right sides of the steering wheel. The hands-on sensors 9 detect a current value equal to or greater than a preset threshold when the driver's hands touch the steering wheel. Therefore, the gripping state can be determined more accurately when the hands-on sensor 9 is used to determine the gripping state, compared to when a captured image is used to determine the gripping state.

Although the hands-on sensor 9 is a pressure sensor, the present invention is not limited to this. For example, the hands-on sensor 9 may be a capacitance sensor or a temperature sensor. When the hands-on sensor 9 is a capacitance sensor, the steering wheel has electrodes, and the gripping state is determined based on the capacitance that changes depending on the distance between the driver's hands and the electrodes. When the hands-on sensor 9 is a temperature sensor, the part of the steering wheel that the driver grips is hotter than the other parts of the steering wheel due to body heat. Therefore, when the temperature measured by the temperature sensor is equal to or higher than a preset threshold value, it is sufficient to determine that the steering wheel is being gripped. In this way, the gripping state determination unit 4 may determine the gripping state using a hands-on signal from the hands-on sensor 9, which is a capacitance sensor or a temperature sensor provided on the left and right sides of the steering wheel. In this case, as in the case where the hands-on sensor 9 is a pressure sensor, the gripping state can be determined more accurately when the gripping state is determined using the hands-on sensor 9 than when the gripping state is determined using a captured image.

Embodiment 5.
The occupant monitoring device 20 in the fifth embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing a part of a vehicle 30 equipped with the occupant monitoring device 20 in the fourth embodiment. The vehicle 30 in this embodiment has an autonomous driving control unit 10 that performs autonomous driving, and an autonomous driving level control unit 11 that controls the autonomous driving level. The same reference numerals are used for the same components as in the first embodiment.

The autonomous driving control unit 10 is composed of a driving assistance ECU and an autonomous driving ECU. The driving assistance ECU is an electronic control device that realizes a driving assistance function that assists the driver in driving operations. The driving assistance ECU enables advanced driving assistance or partial autonomous driving control at level 2 or below in terms of autonomous driving level. The autonomous driving ECU is an electronic control device that realizes an autonomous driving function that can take over driving operations from the driver. The autonomous driving ECU enables autonomous driving of level 3 or above, where the vehicle 30 system is the controlling entity, only within a limited operation design area that has been set in advance.

Here, the autonomous driving levels are explained. When the autonomous driving level is level 0, the driver is the main actor in all driving actions. When the autonomous driving level is level 1, the main actor in either vehicle steering or vehicle driving and vehicle braking among the driving actions is the driving assistance ECU of the vehicle 30, and the main actor in all other driving actions is the driver. When the autonomous driving level is level 2, the main actor in the driving operations of vehicle steering, vehicle driving and vehicle braking is the driving assistance ECU of the vehicle 30. When the autonomous driving level is level 2, the driver is the main actor in the monitoring of the surroundings, and is required to monitor the implementation status of the driving operation by the vehicle 30 and maintain a state in which he/she can perform immediate driving operations such as holding the steering wheel so that he/she can always intervene in the driving operation. When the autonomous driving level is level 3, the main actor in the driving actions of vehicle steering, vehicle driving, vehicle braking and surrounding monitoring is the autonomous driving ECU of the vehicle 30. When the autonomous driving level is level 3, the driver can perform the second task within the range permitted for safety without constantly monitoring the implementation status of the driving action by the autonomous driving ECU. The range permitted from the viewpoint of safety may be determined by law or based on the safety of actual driving. That is, when the autonomous driving level is level 0 to 2, the driver is prohibited from performing the second task, but when the autonomous driving level is level 3, the driver can perform the second task within the range permitted from the viewpoint of safety. Also, when the autonomous driving level is level 4 or higher, the driver can perform the second task.

The autonomous driving level control unit 11 controls switching of the autonomous driving level. The autonomous driving level control unit 11 also transmits a signal indicating the autonomous driving level to the second task determination unit 5b and the autonomous driving control unit 10.

The display unit 8 displays at least one of a sound and an image to prompt the driver to end the second task. At this time, the display unit 8 may also notify the driver that the autonomous driving is ending.

Next, we will explain the occupant monitoring method using the occupant monitoring device 20. Figure 8 is a flowchart showing the processing operation of the occupant monitoring device 20 in embodiment 5.

First, in step S501, the second task determination unit 5b obtains information indicating the current autonomous driving level from the autonomous driving level control unit 11, and determines whether the autonomous driving level is lower than level 3. If the autonomous driving level is lower than level 3, i.e., between level 0 and level 2 (YES in step S501), the processing operation of the occupant monitoring device 20 described in any one of embodiments 1 to 3 is performed, although it is omitted in FIG. 8.

If the autonomous driving level is level 3 or higher (NO in step S501), the second second task determination unit 5b determines whether or not to implement control in the autonomous driving level control unit 11 to change the autonomous driving level from level 3 or higher to level 2 or lower (step S502). For example, control to change the autonomous driving level from level 3 to level 2 is implemented when the autonomous driving level control unit 11 acquires information from the navigation device that the vehicle 30 is scheduled to travel outside the area of a limited operation design area that has been set in advance.

If control to change the autonomous driving level from level 3 or higher to level 2 or lower is not performed (NO in step S502), i.e., if the autonomous driving level is maintained at level 3 or higher, the processing operation of step S502 is repeated. This is because, when the autonomous driving level is level 3 or higher, the execution of the second task is permitted, and therefore, there is no need to determine whether the second task is being executed.

When control is performed to change the autonomous driving level from level 3 or higher to level 2 or lower (YES in step S502), a processing operation of the occupant monitoring device 20 described in any one of embodiments 1 to 3 is performed, although it is omitted in Figure 8.

As in the first embodiment, the occupant monitoring device 20 in this embodiment includes a first second task determination unit 5a that uses a captured image to determine whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed, a gripping state determination unit 4 that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands, and a second second task determination unit 5b that determines whether or not the second task is being performed based on the determination results of the first second task determination unit 5a and the determination results of the gripping state determination unit 4. This allows the occupant monitoring device 20 to perform highly accurate determination of the second task.

In addition, when the level of autonomous driving changes from level 3 or higher to level 2 or lower, the subject of driving changes from the vehicle 30 to the driver. At level 3 or higher, the driver may be performing the second task, and if the level is suddenly lowered to level 2 or lower, there is a risk that the subject of driving will change to the driver while the driver is performing the second task. Therefore, the second second task determination unit 5b determines whether the second task is being performed when the autonomous driving level changes from level 3 or higher, where the subject of driving is the vehicle 30, to level 2 or lower, where the subject of driving is the driver. In this way, it is determined whether the second task is being performed when the subject of driving is changed to the driver. Then, when the subject of driving is changed from the vehicle 30 to the driver, the driver can end the performance of the second task.

In addition, the second second task determination unit 5b does not determine whether or not a second task is being performed when the autonomous driving level is level 3 or higher, where the driving entity is the vehicle 30. As a result, at levels 3 and higher where the performance of the second task is permitted, the occupant monitoring device 20 does not determine whether or not a second task is being performed, thereby reducing the processing load.

Furthermore, the embodiments disclosed in this specification may be freely combined within their scope, and each embodiment may be modified or omitted as appropriate.

1 Driving determination unit, 2 Imaging unit, 3 Image acquisition unit, 4 Gripping state determination unit, 5a First second task determination unit, 5b Second second task determination unit, 6 Memory unit, 7 Output unit, 8 Display unit, 9 Hands-on sensor, 10 Autonomous driving control unit, 11 Autonomous driving level control unit, 12 Processing circuit, 13 Processor, 14 Memory, 20 Occupant monitoring device, 30 Vehicle

Claims (15)

an image acquisition unit that acquires an image from an imaging unit that images a driver of the vehicle;
a first second task determination unit that determines whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed by using the captured image;
a gripping state determination unit that determines a gripping state indicating whether or not the driver is gripping the steering wheel with both hands;
a second second task determination unit that determines whether or not the second task is being performed based on a determination result of the first second task determination unit and a determination result of the gripping state determination unit; and
An occupant monitoring device comprising: The occupant monitoring device of claim 1, characterized in that the second second task judgment unit determines whether or not to cause a judgment to be made by the first second task judgment unit, or whether or not to validate the judgment result by the first second task judgment unit, depending on the gripping state. The second second task judgment unit is characterized in that when the first second task judgment unit judges that the second task is being performed, if the grasping state judgment unit judges that the grasping state is not being performed, the second second task judgment unit validates the judgment result of the first second task judgment unit, and if the grasping state judgment unit judges that the grasping state is being performed, the second second task judgment unit invalidates the judgment result of the first second task judgment unit. The occupant monitoring device according to claim 1 or 2, characterized in that the second second task judgment unit validates the judgment result of the first second task judgment unit when the state of the second task changes and the gripping state does not satisfy a preset condition, and invalidates the judgment result of the first second task judgment unit when the state of the second task changes and the gripping state satisfies the preset condition. The occupant monitoring device according to claim 4, characterized in that the second second task judgment unit judges that the state of the second task has changed when the latest judgment result of the first second task judgment unit differs from the previous judgment result of the first second task judgment unit, and judges that the state of the second task has not changed when the latest judgment result of the first second task judgment unit is the same as the previous judgment result of the first second task judgment unit. The occupant monitoring device according to claim 1 or 2, characterized in that the second second task judgment unit causes a judgment to be made by the first second task judgment unit if the gripping state does not satisfy a preset condition, and does not cause a judgment to be made by the first second task judgment unit if the gripping state satisfies the preset condition. The occupant monitoring device according to any one of claims 4 to 6, characterized in that the preset condition of the gripping state is at least one of the following: the driver continues to grip the steering wheel with both hands; the driver changes from a state in which the driver grips the steering wheel with both hands to a state in which the driver is not gripping the steering wheel with both hands; and the driver changes from a state in which the driver is not gripping the steering wheel with both hands to a state in which the driver is gripping the steering wheel with both hands. The occupant monitoring device according to any one of claims 1 to 7, characterized in that the second task is an action performed by the driver with at least one hand off the steering wheel, and is at least one of the following: smoking, operating a mobile terminal, having a conversation using a mobile terminal, eating and drinking, checking documents, and reading. an output unit that outputs a signal indicating that the second task is being performed to a display unit that displays at least one of a sound and an image prompting the user to finish performing the second task,
9. The occupant monitoring device according to claim 1, wherein the second task determination unit causes the output unit to output the signal when the second task has been continuously performed for a preset time period. a storage unit that stores the determination results of the first second task determination unit and the second second task determination unit;
the first second task determination unit determines a type of the second task when it is determined that the second task is being performed;
the second second task determination unit causes the output unit to output the signal when the determination result stored in the storage unit indicates that the second task of the same type has been continuously executed for the predetermined time period;
10. The passenger monitoring device according to claim 9, wherein the preset time period differs depending on the type of the second task. An occupant monitoring device as described in any one of claims 1 to 10, characterized in that the first second task determination unit uses pattern matching or a neural network to determine whether the second task is being performed. The occupant monitoring device according to any one of claims 1 to 11, characterized in that the gripping state determination unit determines the gripping state using the captured image, or determines the gripping state using a signal from a hands-on sensor, which is either a pressure sensor, a capacitance sensor, or a temperature sensor provided on the left and right sides of the handlebars. The occupant monitoring device according to any one of claims 1 to 12, characterized in that the second second task judgment unit judges whether the second task is being performed when the autonomous driving level changes from level 3 or higher, in which the vehicle is the driving subject, to level 2 or lower, in which the driver is the driving subject. The occupant monitoring device according to any one of claims 1 to 13, characterized in that the second second task judgment unit does not judge whether the second task is being performed when the autonomous driving level is level 3 or higher, in which the vehicle is the driving subject. acquiring an image from an imaging unit that images a driver of the vehicle;
determining whether or not a second task, which is an action other than driving that the driver performs while driving, is being performed by using the captured image;
determining a gripping state indicating whether the driver is gripping a steering wheel with both hands;
determining whether the second task is being performed based on a result of determining whether the second task is being performed and a result of determining the gripping state;
An occupant monitoring method comprising:

Images