JP2022127881A - Operation state detection method - Google Patents

Abstract

To detect a driver's state of being less awakened or inattentive to driving and to enable the driver's state of being idle to be detected.SOLUTION: Driver's current driving ability is evaluated (C) on the basis of a predicted load predicted from a surrounding state of a vehicle (A) and performance of driver's handling a situation (B). By comparing the current driving ability with normal driving ability of the driver, a degree of demonstrating the driving ability is determined (E). Also, whether or not the driver is sleepy or inattentive is determined (X). When the degree of demonstrating the driving ability is equal to or less than a predetermined degree and the driver is not determined to be sleepy or inattentive, the driver is determined to be in a state of being idle (F).SELECTED DRAWING: Figure 1

Description

The technology disclosed herein belongs to the technical field of detecting the driving state of a driver who drives a vehicle.

2. Description of the Related Art In recent years, techniques have been developed for detecting the driving state of a driver who drives a vehicle from, for example, an image of the driver captured by a driver monitor camera, output data from various sensors mounted on the vehicle, and the like. For example, Patent Literature 1 discloses a technique for evaluating the awakening level of a vehicle driver using an error value of vehicle control by the driver.

Japanese Patent No. 6492096

Here, it is possible to detect the driver's reduced arousal and inattentiveness from the image of the driver monitor camera. However, when the driver is absent-minded or thinking and is in a careless state (attentiveness of consciousness), it cannot be distinguished from the normal state from the outside, so it is necessary to detect it from the image of the driver monitor camera. is difficult.

The technology disclosed herein aims to detect a driver's reduced arousal and distraction, and to detect that the driver is in a careless state.

In order to solve the above-mentioned problems, the technology disclosed herein provides a driving state detection method for detecting the state of a driver of a vehicle, wherein information indicating the surrounding situation of the vehicle is used to predict the driver. Using the step (a) of estimating the load, the information indicating the surrounding situation of the vehicle, the information indicating the driving operation of the driver, and the information indicating the behavior of the driver, the corresponding behavior of the driver is determined. and evaluating the current driving ability of the driver based on the predicted load estimated in step (a) and the performance of the corresponding behavior evaluated in step (b). step (c), comparing the driving ability evaluated in step (c) with the normal driving ability of the driver to determine the degree of demonstration of the driving ability, and information indicating the behavior of the driver. Step (e) for determining whether the driver is drowsy driving or distracted driving using and a step (f) of determining that the driver is in a careless state when it is determined in (e) that the driver is not drowsy driving or inattentive driving.

According to this configuration, the current driving ability of the driver is evaluated based on the predicted load estimated from the surrounding conditions of the vehicle and the performance of the response action of the driver. Here, the corresponding action means a driving action that the driver performs in response to the situation outside the vehicle. Then, by comparing the evaluated current driving ability with the normal driving ability of the driver, the degree of demonstration of the driving ability is determined. On the other hand, information indicating the behavior of the driver is used to determine whether the driver is drowsy driving or distracted driving. Then, when it is determined that the degree of manifestation of the driving ability is below a predetermined level and the driver is not drowsy driving or inattentive driving, it is determined that the driver is in a careless state. Therefore, by comprehensively judging the state in which the driver's inherent driving ability is not exhibited and the behavior seen in drowsiness and distracted driving, the driver may be absent-minded or thinking. It can be determined whether the driver is in a careless state, drowsy driving, or inattentive driving.

In the driving state detection method, the step (e) compares the driver's head position with the reference position when the distance between the upper and lower eyelids of the driver becomes smaller than the reference value by a predetermined threshold or more. When the line of sight of the driver deviates by a predetermined threshold or more, or when the deviation between the line of sight of the driver and the traveling direction of the vehicle becomes greater than a predetermined threshold, it is determined that the driver is drowsy driving or distracted driving. It may include steps.

According to this, drowsy driving or inattentive driving is determined when the eyelids are slightly closed, the head is shifted from the normal position, or the line of sight is shifted from the direction of travel. On the other hand, when the driver is in a careless state, the eyelids are open, the head is in a normal position, and the line of sight is looking in the direction of travel. Therefore, careless driving can be distinguished from drowsy driving or inattentive driving.

As described above, according to the technology disclosed herein, it is possible to detect the driver's reduced alertness and distraction, and to detect that the driver is in a careless state.

Functional image of a system that implements the driving state detection method according to the present embodiment Determining method for demonstrating driving ability Flow of processing in the driving state detection method according to the present embodiment Flow of processing in the driving state detection method according to the present embodiment System configuration example for executing the driving state detection method according to the present embodiment

Exemplary embodiments are described in detail below with reference to the drawings.

FIG. 1 is a functional image diagram of a system that implements a driving state detection method according to this embodiment. The system of FIG. 1 receives, as inputs, external sensing information, vehicle driving operation information, and information from a driver monitor camera that captures images of the driver and the like. The external sensing information is, for example, information obtained by external detection means such as a vehicle-mounted camera or an ADAS (Advanced Driver Assistance Systems) vehicle-mounted sensor, and includes information on targets in the external environment. The information on the driving operation is obtained, for example, by an on-vehicle sensor such as an accelerator opening sensor and a brake pressure sensor. From the driver monitor camera, information on the driver's behavior, such as movement of the line of sight, behavior of the head, facial expression, etc., can be obtained.

In the system of FIG. 1, as an output, determination of careless driving and determination of drowsy driving or inattentive driving are performed. Inattentive driving means that the driver is absent-minded or thinking, causing "awareness of distraction" and driving in an inattentive state. "Conscious inattentiveness" refers to a state in which the amount of attention for driving is reduced due to thoughts, etc., and falls below the required amount.

The processing of the system of FIG. 1 will be described in detail.

In block A, the predicted load is calculated using the external sensing information. Here, human cognitive load consists of three components: attentional load, perceived load, and predicted load. The attention allocation load is the load when looking at a place to be confirmed in order to acquire information. The perceptual load is the load on recognizing and interpreting the distance, movement, etc. of an object seen. The prediction load is the load when predicting future situations from perceptual information. In block A, a predictive load model is constructed, and the present predictive load is estimated from external sensing information using the predictive load model.

Now, the predictive load model will be described. Humans are thought to reproduce memories and predict the future from clue information. For example, for the event of deceleration of the preceding vehicle, clue information includes deceleration of the preceding vehicle, uphill, red light, sharp curve, turn signal, surrounding congestion, speed adjustment error of the preceding vehicle, etc. is confirmed from public road driving data. For example, in a driving environment, when the preceding vehicle emits a turn signal, the driver searches the memory that ``the vehicle often decelerates after the blinker'' and predicts that ``the preceding vehicle may decelerate.'' it is conceivable that.

Therefore, the inventors of the present application hypothesized that the predicted load is composed of the spare time and the encounter rate. The margin time is the time from the appearance of a clue (winker in the example above) to the occurrence of an event (deceleration of the preceding vehicle in the example above). 25% deceleration of the preceding vehicle, etc.). That is, the idea is that the longer the time available for recall and the higher the frequency of encounters and the more memorable they are, the lower the prediction load should be.

However, according to experiments conducted by the inventors of the present application, the expected results were not always obtained. Further investigation showed that the predicted load needed to take into account the uncertainty of real-world events. For example, at a red light, the preceding vehicle will almost certainly decelerate, but when the preceding vehicle decelerates, the preceding vehicle will not necessarily decelerate. In other words, the probability that an event will occur with respect to a clue differs depending on the clue.

Therefore, the predictive load model is defined by, for example, the following formula.

In the above equation, uncertainty is reflected in the predictive load model. Through studies by the inventors of the present application, it has been confirmed that this predictive load model reflecting uncertainty can be applied to driving scenes.

Block B evaluates the performance of the driver's risk reduction behavior. A risk reduction action is an action for reducing the risk of an accident occurring, for example, an action of releasing the accelerator in relation to the deceleration of the preceding vehicle. The performance of risk reduction behavior is evaluated, for example, by the time from the appearance of a cue such as a blinker until the accelerator is turned off. In addition to this, the risk reduction behavior includes, for example, the behavior of stepping on the brake, the behavior of looking at the mirror, and the like.

Blocks C, D, and E determine the extent to which the driver's driving ability is demonstrated. In block C, the predicted load calculated in block A and the performance of the risk reduction actions evaluated in block B are combined and output as the current driving ability of the driver. Block D stores the driver's driving performance data output from block C. FIG. Based on the accumulated data, the driver's normal driving ability, that is, the original driving ability is individually learned. Block E compares the driver's current driving ability output from block C with the driver's normal driving ability output from block D.

FIG. 2 is a diagram for explaining a method of determining the degree of driving ability. In FIG. 2, the driving ability is represented on a two-dimensional coordinate system with the predicted load on the horizontal axis and the performance of risk reduction behavior on the vertical axis. FIG. 2(a) is the current driving ability of the driver obtained in block C. FIG. FIG. 2(b) is the normal driving ability of the driver obtained in Block D. FIG. From the accumulated drivability data, a line representing normal drivability has been obtained. Then, as shown in FIG. 2(c), the current driving ability shown in FIG. 2(a) is compared with the normal driving ability shown in FIG. 2(b) to determine the degree of performance of the driving ability. In FIG. 2(c), since the performance of the risk reduction behavior deteriorates under the same predicted load, it is determined that the driving ability is declining, that is, the driving ability is exhibited at a low degree. The degree of exertion may be evaluated, for example, by the distance between the coordinate position of the current driving ability and the line representing the normal driving ability.

Returning to FIG. 1, on the other hand, in block X, it is determined whether or not the driver is drowsy driving or inattentive driving from the image of the driver captured by the driver monitor camera. In block X, for example, the distance between the upper and lower eyelids of the driver, the position of the head, the line-of-sight direction, etc. are used for determination.

In block F, when it is not determined in block X that the driver is not drowsy driving or distracted driving, and the comparison result in block E shows that the degree of driving ability of the driver is below a predetermined level, It is determined that the driver is driving aimlessly.

The data accumulated in block D preferably does not include data when the driver is carelessly driving or drowsy/inattentive driving. In the system of FIG. 1, the noise to the processing performance includes the driver's driving characteristics (safety sensitivity, acceptable risk, etc.), cognitive ability (young people, elderly people, etc.), and driving experience (beginners, veterans, etc.).

3 and 4 are flowcharts showing an example of the flow of processing in the driving state detection method according to this embodiment. FIG. 5 is a block diagram showing an example of system configuration for executing the driving state detection method of FIGS. 3 and 4. In FIG. The controller 1 is composed of, for example, an IC chip with a processor and memory, or a plurality of IC chips with a processor and memory. The controller 1 may also include an AI (Artificial Intelligence) dedicated chip, a GPU (Graphic Processing Unit), or the like.

The system of FIG. 5 includes a vehicle-mounted camera, sensor, and the like as input means for the controller 1 . The front camera 10 is a camera that is installed, for example, in the front part of the vehicle and captures the situation ahead of the vehicle. The in-vehicle camera 11 is a camera installed, for example, in the interior of the vehicle to photograph the situation inside the vehicle, and functions as a driver monitor camera. A vehicle speed sensor 12, an acceleration sensor 13, and a yaw rate sensor 14 are sensors that detect the speed, acceleration, and yaw rate of the vehicle, respectively. The steering angle sensor 15, the steering torque sensor 16, the accelerator opening sensor 17, and the brake pressure sensor 18 are sensors for detecting the steering angle, steering torque, accelerator opening, and brake pressure of the vehicle, respectively. A GPS (Global Positioning Systems) unit 19 is a unit that detects the position of the vehicle. Front camera 10 and GPS unit 19 are examples of means for detecting external conditions. The steering angle sensor 15, the steering torque sensor 16, the accelerator opening sensor 17, and the brake pressure sensor 18 are examples of means for detecting driving operation.

The controller 1 sends image signals to a center display 21 and a HUD (Head-Up Display) 22, sends audio signals to an audio output device 23, and sends control signals to a steering control device 24, a brake control device 25 and an engine control device 26. send.

3 and 4, the controller 1 outputs a careless driving determination flag and a drowsiness/inattentive driving determination flag. When the controller 1 determines that the driver is driving carelessly, the controller 1 sets the careless driving determination flag to "1". When it is determined that the driver is drowsy/inattentive driving, the drowsy/inattentive driving flag is set to "1". When the careless driving determination flag or the drowsiness/inattentive driving flag is set to "1", the controller 1, for example, sends a signal to output a warning image to the center display 21 or the HUD 22, and outputs a warning sound to the audio output device 23. send a signal. Alternatively, it sends a control signal to the brake controller 25 and the engine controller 26 to slow the vehicle.

In the process of FIG. 3, the controller 1 acquires vehicle peripheral information (S11a), driver's driving operation information (S11b), and driver's line of sight/head behavior information (S11c). The surrounding information of the vehicle is acquired based on, for example, images captured by the front camera 10, positional information acquired by the GPS unit 19, and the like. The driving operation information is acquired based on various data detected by the steering angle sensor 15, the steering torque sensor 16, the accelerator opening sensor 17, and the brake pressure sensor 18, for example. The line-of-sight/head behavior information of the driver is acquired, for example, based on the video imaged by the in-vehicle camera 11 .

The controller 1 calculates the predicted load of the driver based on the surrounding information of the vehicle (S12). Here, the predictive load is calculated using the above-described predictive load model with respect to clues such as the blinkers of the preceding vehicle obtained from the surrounding information of the vehicle. The controller 1 then evaluates the performance of the driver's risk reduction behavior (S13). For example, the time until the accelerator is turned off is evaluated with respect to the blinker of the preceding vehicle. Then, the controller 1 evaluates the current driving ability of the driver by combining the predicted load calculated in S12 and the performance of the risk reduction action evaluated in S13 (S14).

The controller 1 determines whether or not the driver's current driving ability obtained in S14 is below the driver's normal driving ability (S15). As described above, the normal driving ability of the driver is individually learned from the cumulative data of the driver's past driving ability in normal times. If not, the process is terminated as no problem. On the other hand, when it is below, it progresses to following step S16.

In S16, the controller 1 determines whether or not the drowsiness/inattentive driving determination flag is "0" (S16). Note that the set flow of the drowsiness/inattentive driving determination flag is described in FIG. 4 and will be described later. When the drowsiness/inattentive driving determination flag is set to "1", the drowsiness/inattentive driving determination flag is output (S19), and the process ends. On the other hand, when the drowsiness/inattentive driving determination flag is "0", the controller 1 sets the inattentive driving determination flag to "1" (S17). Then, a careless driving determination flag is output (S18), and the process is terminated.

FIG. 4 shows a set flow of the drowsiness/inattentive driving determination flag. The controller 1 acquires information about the driver's posture, facial expression, etc. from the video captured by the in-vehicle camera 11 (S21). Based on the information acquired in S21, the difference between the distance between the upper and lower eyelids of the driver and the reference value, for example, the distance in the initial state is calculated (S22). When the calculated difference is equal to or greater than the predetermined threshold (No in S23), the drowsiness/inattentive driving determination flag is set to "1" (S28). When the calculated difference is smaller than the predetermined threshold, the deviation of the driver's head position from the reference position is calculated from the information acquired in S21 (S24). When the calculated deviation is equal to or greater than the predetermined threshold value (No in S25), the drowsiness/inattentive driving determination flag is set to "1" (S28). When the calculated deviation is smaller than the predetermined threshold, the deviation between the line-of-sight direction and the traveling direction of the vehicle is calculated from the information acquired in S21 (S26). When the calculated deviation is greater than or equal to the predetermined threshold value (No in S27), the drowsiness/inattentive driving determination flag is set to "1" (S28). When the calculated deviation is smaller than the predetermined threshold value, the drowsiness/inattentive driving determination flag is not set to "1".

As described above, according to the present embodiment, the current driving ability of the driver is evaluated based on the predicted load estimated from the surrounding conditions of the vehicle and the performance of the driver's risk reduction actions. Then, by comparing the evaluated current driving ability with the normal driving ability of the driver, the degree of demonstration of the driving ability is determined. On the other hand, information indicating the behavior of the driver is used to determine whether the driver is drowsy driving or distracted driving. Then, when it is determined that the degree of manifestation of the driving ability is below a predetermined level and the driver is not drowsy driving or inattentive driving, it is determined that the driver is in a careless state. Therefore, by comprehensively judging the state in which the driver's inherent driving ability is not exhibited and the behavior seen in drowsiness and distracted driving, the driver may be absent-minded or thinking. It can be determined whether the driver is in a careless state, drowsy driving, or inattentive driving.

Also, when the driver's eyelids tend to be closed, the head is displaced from the normal position, or the line of sight deviates from the direction of travel, it is determined that the driver is drowsy driving or inattentive driving. On the other hand, when the driver is in a careless state, the eyelids are open, the head is in a normal position, and the line of sight is looking in the direction of travel. Therefore, careless driving can be distinguished from drowsy driving or inattentive driving.

Note that, in the above-described embodiment, the driver's driving ability is obtained using the performance of the risk reduction behavior. However, the present disclosure is not limited to this, and for example, the driver's driving ability may be evaluated using the behavior of maintaining ride comfort and the performance of compliance behavior. That is, in the present disclosure, the performance of the driver's response behavior is used to evaluate the driving ability. Here, "response behavior" refers to the driving behavior that the driver performs in response to the situation outside the vehicle. is an example of

Regarding ride comfort behavior and legal compliance behavior, predicted load and performance may be evaluated, for example, as follows.

(Action to maintain ride comfort)
Regarding the load (related to pedal operation) that maintains the ride comfort in the longitudinal direction of the vehicle, the load increases as the speed of the vehicle ahead becomes unstable, the load increases as the gradient changes sharply, and the load increases as the driving speed decreases. I can say. Specifically, when the preceding vehicle is running at a constant speed, the own vehicle can also run at a constant speed, so control that causes acceleration and deceleration that affects ride comfort does not occur. That is, the load for maintaining ride comfort is low. On the other hand, when the vehicle ahead is repeatedly accelerating and decelerating, it is necessary to anticipate the situation and operate the pedal early in order to maintain a smooth ride. That is, the load for maintaining ride comfort is high.

In addition, since it is sufficient to keep the speed constant on a flat road, there is no control that causes acceleration or deceleration that affects ride comfort, and the load for maintaining ride comfort is low. On the other hand, in scenes where there are frequent uphills and downhills, it is necessary to operate the accelerator and brakes additionally because the speed will be too high or too low if it is not operated in advance. That is, the load for maintaining ride comfort is high. In addition, when the traveling speed is high, the front and rear gravitational acceleration generated by the access operation and the brake operation becomes small, which facilitates adjustment. Therefore, the load for maintaining ride comfort is low. On the other hand, when the driving speed is low, the longitudinal gravitational acceleration generated by the accelerator operation and the brake operation becomes large, so careful operation is required to maintain the ride comfort. Therefore, the load for maintaining ride comfort is high.

Regarding the load that maintains the ride comfort in the left-right direction (related to steering operation), it can be said that the higher the speed of change in road shape (the tighter the curve, the higher the speed), the higher the load. . Specifically, when driving on a straight road, you don't need to make a big steering operation. That is, the load for maintaining ride comfort is low. On the other hand, the tighter the curve, the greater the need to perform steering operation, and careful operation is required to maintain ride comfort. Therefore, the load for maintaining ride comfort is high. In addition, the faster you drive around a curve, the faster you need to steer, so careful operation is required to maintain a comfortable ride. Therefore, the load for maintaining ride comfort is high.

Performance may be evaluated, for example, by the smoothness of gravitational acceleration applied to the vehicle. For example, when the jerk (differential value of acceleration/deceleration) is small, the performance is evaluated as high. Alternatively, the evaluation may be made based on the smoothness of the connection between the front, rear, left, and right G when traveling on a curve. For example, evaluation is made based on whether the GG diagram is close to a circle. If the connection is smooth, the performance is evaluated as high.

(legal compliance behavior)
・In the case of traffic lights Considering the timing at which the traffic light turns red when passing through an intersection, the load is higher when the traffic light turns red when the vehicle is approaching the intersection. Specifically, for example, when the traffic light turns red 30 seconds before the vehicle approaches an intersection, all that is required is to see the red traffic light and slowly stop. Therefore, the legal compliance load is low. On the other hand, if the traffic light turns red 3 seconds before your vehicle approaches the intersection, you must see the red traffic light and act immediately. Therefore, the legal compliance load is high. In addition, a signal that changes from green to yellow to red has a low load because there is a margin until the signal turns red, and a signal that suddenly changes from flashing yellow to red has a high load.

Performance can be evaluated, for example, by how many seconds passed after turning red (or yellow). For example, the performance is evaluated as high if stopped, and the performance is evaluated as low if the intersection is crossed 10 seconds after turning red or yellow.

・In the case of signs If it is easy to recognize that there is a sign due to its color, visibility, and position, the burden is low. Specifically, for example, when the sign is blurred and the content is not clear even if you look closely, or when a tree branch hangs in front of the sign and makes it difficult to see, the legal compliance load is high. Also, when the sign is upside down and in an unusual place, it can be said that the legal compliance load is high.

Performance can be evaluated, for example, in the case of speed signs, by how far the speed limit is exceeded. For example, if the speed limit is observed, the performance is evaluated as high, and if the speed limit is exceeded, the performance is evaluated as low.

・In the case of the driving lane In the case of the driving lane, if the lane is clearly visible, the load is low, and if it is difficult to see it blurred, the load is high.

Performance may be evaluated, for example, by how far the vehicle is running out of the driving lane. For example, if there is no protrusion, the performance is evaluated to be high, and the greater the amount of protrusion, the lower the performance.

Further, the application of the present invention is not limited to automobiles. This is effective for detecting the driving state of moving bodies other than automobiles, such as trains.

Also, the technology according to the present disclosure may be realized in a form other than a single information processing device. For example, part of the processing executed by the controller 1 may be realized by another information processing device. Further, for example, an information processing device that is not mounted on the moving object, such as a smartphone or tablet held by the driver, may execute part of the processing. Alternatively, the cloud may be configured to execute part or all of the processing.

The above-described embodiments are merely examples, and should not be construed as limiting the scope of the present disclosure. The scope of the present disclosure is defined by the claims, and all modifications and changes within the equivalent range of the claims are within the scope of the present disclosure.

The technology disclosed herein is useful for improving the safety of driving a car, for example, because it can detect a driver's reduced arousal and distraction, and can also detect that the driver is in a careless state.

1 controller 10 front camera 11 in-vehicle camera (driver monitor camera)
12 vehicle speed sensor 13 acceleration sensor 14 yaw rate sensor 15 steering angle sensor 16 steering torque sensor 17 accelerator opening sensor 18 brake pressure sensor 19 GPS unit 21 center display 22 HUD
23 audio output device 24 steering control device 25 brake control device 26 engine control device

Claims (2)

A driving state detection method for detecting the state of a driver of a vehicle,
a step (a) of estimating the predicted load of the driver using information indicating the surrounding situation of the vehicle;
a step (b) of evaluating the performance of the response action of the driver using the information indicating the surrounding situation of the vehicle, the information indicating the driving operation of the driver, and the information indicating the behavior of the driver; ,
step (c) of assessing the driver's current driving ability based on the predicted load estimated in step (a) and the performance of the corresponding action assessed in step (b);
a step (d) of comparing the driving ability evaluated in step (c) with the normal driving ability of the driver to determine the degree of demonstration of the driving ability;
a step (e) of determining whether or not the driver is drowsy driving or distracted driving using the information indicating the behavior of the driver;
When it is determined in step (d) that the degree of manifestation of the driving ability determined in step (d) is below a predetermined level, and in step (e) it is determined that the driver is not drowsy driving or distracted driving, the driver is in a careless state. A driving state detection method comprising a step (f) of determining that the In the operating state detection method according to claim 1,
The step (e) includes
When the distance between the driver's upper and lower eyelids becomes smaller than a reference value by a predetermined threshold or more, when the driver's head position deviates from the reference position by a predetermined threshold or more, or when the driver's line of sight direction determining that the driver is drowsy driving or inattentive driving when a difference between the direction of travel and the traveling direction of the vehicle becomes greater than a predetermined threshold value.

Images