JP7418669B2 - Occupant condition detection device, occupant condition detection system, vehicle control system, and occupant condition detection method - Google Patents

Description

The present disclosure relates to occupant condition detection technology.

Technology that captures images of the interior of a vehicle, uses the occupant's face orientation or head position extracted from the captured image to determine posture collapse, and detects abnormalities such as loss of consciousness in the occupant. is being developed. For example, Patent Document 1 discloses a face orientation detection means for detecting the orientation of the driver's face relative to the front of the vehicle based on an image of the driver's seat captured by an imaging device mounted on the vehicle; , an orientation error state detection means for detecting that the driver is unable to drive when the face orientation detected by the face orientation detection means exceeds an orientation determination time and is larger than a certain threshold; A device for detecting a driver's inability to drive is disclosed.

JP2016-9256A

Some occupants may intentionally or out of habit drive with their posture slumped, such as by leaning on their elbows. According to conventional technology, which determines whether or not the occupant's posture is slumped based on the direction of the occupant's face, it is possible to In this case, it is determined that there is a posture collapse. Therefore, according to the prior art, there have been cases where it has been determined that the vehicle cannot be driven even though it is actually possible for the passenger to drive the vehicle. In other words, there is a problem in that erroneous detection of posture collapse occurs.

The present disclosure has been made to solve such problems, and an object of the present disclosure is to provide an occupant condition detection technique that prevents false detection of posture collapse.

An occupant state detection device according to an embodiment of the present disclosure includes an image acquisition unit that acquires an image of the inside of a vehicle captured by an imaging device; and a posture of an occupant of the vehicle is determined in advance based on the acquired image. a posture type determination unit that determines which posture type the acquired posture type corresponds to among a plurality of posture types including a first posture type obtained by the acquisition of the acquired posture and a second posture type different from the first posture type; A visual information acquisition unit that calculates visual information including visual line information regarding the direction of the passenger's line of sight and blink information regarding the presence or absence of blinking based on the captured image; and an abnormality determining section that uses the information to determine whether or not continued operation of the vehicle by the occupant is possible.

According to the occupant condition detection device according to the embodiment of the present disclosure, it is possible to prevent false detection of posture collapse.

1 is a diagram illustrating a configuration example of an occupant condition detection device and an occupant condition detection system. It is a figure showing the example of composition concerning the modification of the passenger state detection device and the passenger state detection system. 1 is a diagram illustrating a configuration example of a driving control system that is an example of using an occupant condition detection device and an occupant condition detection system. FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant condition detection device, an operation control device, and a learning device. FIG. 2 is a diagram illustrating an example of a hardware configuration of an occupant condition detection device, an operation control device, and a learning device. FIG. 3 is a diagram for explaining the operation of the occupant condition detection device. 3 is a flowchart showing the operation of the occupant condition detection device. It is a flowchart which shows the determination process of the possibility of continuing operation. 3 is a flowchart showing the operation of the operation control system.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that components given the same or similar symbols in the drawings have the same or similar configurations or functions, and overlapping explanations of such components will be omitted.

Embodiment 1.
<Configuration of occupant condition detection system>
An occupant condition detection device and an occupant condition detection system according to the present disclosure will be described with reference to FIGS. 1, 2, and 4 to 7. FIG. 1 shows an occupant condition detection device 1 and an occupant condition detection system SYS. It is a figure showing the example of composition of B. As shown in FIG. 1, the occupant condition detection system SYS. B is the operation control system SYS. C, the steering mechanism 3, and the braking/driving mechanism 4, as well as the vehicle control system SYS. Configure A. Occupant status detection system SYS. B includes an occupant condition detection device 1 and an imaging device 2. The imaging device 2 is arranged at a position where it can image the interior of the vehicle, and captures an image of a vehicle occupant, typically a driver. The imaging device 2 outputs the captured image to the occupant condition detection device 1.

<Configuration of occupant condition detection device>
As shown in FIG. 1, the occupant condition detection device 1 includes an image acquisition section 11, a vehicle information acquisition section 12, a characteristic information detection section 13, and an abnormality determination section 14, and includes an image captured by the imaging device 2. Based on the image, it is determined whether the occupant can continue driving.

(Image acquisition unit)
The image acquisition unit 11 acquires a captured image from the imaging device 2 and outputs the acquired image to the feature information detection unit 13.

(Vehicle information acquisition department)
The vehicle information acquisition unit 12 acquires vehicle information. Examples of vehicle information include vehicle start information indicating the start of the vehicle, vehicle speed information indicating the speed of the vehicle, steering angle information indicating the steering angle, and seat position information indicating the seat position where the occupant is seated. included. The vehicle information acquisition section 12 outputs the acquired vehicle starting information to the image acquisition section 11, and upon receiving the vehicle starting information, the image acquisition section 11 starts image acquisition and image output operations. Further, the vehicle information acquisition unit 12 outputs the acquired vehicle speed information, steering angle information, and seat position information to the abnormality determination unit 14, and the abnormality determination unit 14 determines that the vehicle speed indicated by the vehicle speed information is equal to or higher than a predetermined speed. In addition, an abnormality determination is made when the steering angle indicated by the steering angle information is within a predetermined range. As an example, an abnormality determination is made when the vehicle speed is 25 [km/h] or more and the steering angle is within a range of ±20 degrees. By setting such operating conditions, it is possible to exclude from the determination a situation in which even if the posture of the vehicle occupant collapses, it is not considered abnormal, such as during slow driving or right/left turning driving. Note that the seat position information is used by an abnormality determining section 14A, which will be described later.

(Feature information detection unit)
The feature information detection unit 13 includes a shape information detection unit 131 and a feature amount calculation unit 132, and detects the shape of the body such as the head, face, and eyes, and the amount of change in these shapes from the image output from the image acquisition unit 11. Detect feature information indicating physical characteristics, including.

(Shape information detection unit)
The shape information detection unit 131 acquires the shape related to the physical characteristics of the occupant from the image received from the image acquisition unit 11 . For example, the shape of the occupant's head, face, and eyes is obtained.

(Feature value calculation unit)
The feature amount calculation unit 132 calculates the amount of change in shape acquired by the shape information detection unit 131 as a feature amount.

The feature information detection unit 13 outputs the shape acquired by the shape information detection unit 131 and the feature amount calculated by the feature amount calculation unit 132 to the abnormality determination unit 14 as feature information.

(Abnormality determination section)
The abnormality determination unit 14 includes a posture type determination unit 141, a visual information acquisition unit 142, and a continued operation possibility determination unit 143, and uses the characteristic information output from the characteristic information detection unit 13 to determine whether or not the occupant can continue driving the vehicle. Determine whether it is possible. This determination of the possibility of continued operation is performed based on the vehicle speed information and steering angle information that the abnormality determination unit 14 has acquired from the vehicle information acquisition unit 12. This is performed when the steering angle indicated by is within a predetermined range. For example, an abnormality determination is made when the vehicle speed is 25 [km/h] or more and the steering angle is within a range of ±20 degrees.

(Posture type determination section)
Posture type determination unit 141 determines which posture type the posture of the vehicle occupant corresponds to among a plurality of predetermined posture types based on the captured image. Symptoms such as loss of consciousness or epilepsy may cause the occupant's posture to collapse. Figure 5 (partially revised from March 2018, ``Basic Design Document for Automatic Driver Abnormality Detection System'', Advanced Safety Vehicle Promotion Study Group, Automobile Bureau, Ministry of Land, Infrastructure, Transport and Tourism) is a diagram showing the type of such posture collapse.

As shown in FIG. 5, there are multiple aspects of postural collapse. In Figure 5, the driver falls forward and continues to have his face close to the steering wheel, which is a ``prone'' position, and the driver's face continues to be in a downward facing position. ``Hunting forward,'' a condition in which the driver's upper body is tilted backwards and his or her face is facing upwards; ``Shrimp warp'', which is a condition where the driver's face is tilted to the left or right; ``neck tilted to the side'', which is a condition where the driver's upper body is tilted to the left or right, and the driver's face is also tilted in the same direction. ``Side leaning'' is a condition in which the driver's upper body continues to lean to the left or right, and ``side leaning'' is a condition in which the driver's upper body continues to lean to the left or right. In the present disclosure, "prone", "prone", "backward", and "shrimp" are classified into posture type 1 (first posture type), and "head only falls sideways", "sideways" and " "Side leaning" is classified into posture type 2 (second posture type). Posture type 1 is a posture in which the occupant's head, face, or upper body is tilted along the longitudinal direction of the vehicle, and posture type 2 is a posture in which the occupant's head, face, or upper body is tilted to the right or in the longitudinal direction of the vehicle. The posture is leaning towards the left side.

Posture type determination section 141 determines whether or not the occupant's posture has deteriorated by detecting the position of the head and the direction of the face using the characteristic information. For example, the type of posture collapse is determined by determining whether the difference between the position of the occupant's head and the direction of the face obtained from the current image frame from the reference posture is equal to or greater than a predetermined threshold. The reference posture may be, for example, a predetermined time period of the head position and face direction of the occupant obtained when the vehicle speed is 25 [km/h] or higher and the steering angle is within a range of ±20 degrees. 3 seconds) can be used.

(Visual information acquisition unit)
The visual information acquisition unit 142 includes a line of sight calculation unit 1421 and an eye opening degree calculation unit 1422, and uses the feature information output from the feature information detection unit 13 to acquire visual information that is information regarding the eyes of the occupant. Examples of visual information include information indicating the direction of the line of sight and information indicating the presence or absence of blinking.

(Line of sight calculation unit)
The line of sight calculation unit 1421 uses the feature information output from the feature information detection unit 13 to calculate the passenger's line of sight. The line of sight can be calculated, for example, using the position of the pupil relative to the position of the inner and outer corners of the eyes. The line of sight calculation unit 1421 calculates the calculated direction of the line of sight. Calculation of the direction of the line of sight can be defined by the angle of the line of sight with respect to a virtual straight line connecting the imaging device 2 and the center of either the right or left eye of the occupant. Note that instead of using the feature information output from the feature information detection unit 13, the line of sight may be obtained directly from the face image using a learning model that has learned the relationship between the face image and the direction of the line of sight. good.

Furthermore, the line of sight calculation unit 1421 uses the calculated line of sight to determine whether the occupant is visually checking the front. Determination as to whether or not the occupant is visually checking the front can be made, for example, according to the following procedure.

First, the line of sight calculation unit 1421 estimates the line of sight Pitch when the occupant is visually checking the front. For example, a histogram of line-of-sight Pitch is created, and the line-of-sight Pitch with the highest frequency is set as the line-of-sight during forward visual recognition (hereinafter referred to as "reference line-of-sight") LOSref.

Next, by calculating the difference between the line of sight Pitch of the current image frame and the reference line of sight LOSref, it is determined whether forward vision is instantaneously recognized. If the difference is within a predetermined range, it is determined that the occupant is momentarily visually recognizing the road ahead.

Next, if the condition that the instantaneous forward visibility determination continues to a certain extent is met, it is determined that the occupant is continuously visualizing the front, and it is determined that there is "forward visibility". If the conditions are not met, it is determined that there is no forward visibility. For example, if a 2-sec buffer is prepared to store the instantaneous forward visibility determination results, and 80% or more of the stored determination results indicate forward visibility, it is determined that there is forward visibility, and if less than 80%, then forward visibility is determined. It is determined that there is no forward visibility.

The line-of-sight calculation unit 1421 outputs the result of the determination indicating the presence or absence of forward visibility as line-of-sight information.

(Eye opening degree calculation unit)
The eye opening degree calculation unit 1422 calculates the degree of eye opening of the occupant using the feature information output from the feature information detection unit 13, and determines whether or not the occupant blinks. The degree of eye opening is calculated by, for example, detecting the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the highest point of the upper eyelid, and calculating the distance between the straight line connecting the inner corner and the outer corner of the eye and the highest point of the upper eyelid. The flatness of the eye can be determined by dividing by the distance, and the value obtained by dividing the flatness by a predetermined reference value can be calculated. Note that the highest point of the upper eyelid is the point (apex) of the upper eyelid that is farthest from the straight line connecting the outer corner of the eye and the inner corner of the eye. When the degree of eye opening is less than a predetermined threshold TH1, it is determined that the eyes are closed, and when the degree of eye opening exceeds a predetermined threshold TH2, which is larger than the predetermined threshold TH1, it is determined that the eyes are open, and closed eyes are detected. If eye opening is detected within a predetermined period of time after the blinking is detected, it can be determined that "blinking occurred". If eye opening is not detected within a predetermined time after eye closure is detected, it is determined that there is no blinking.

The eye opening degree calculation unit 1422 outputs information indicating the presence or absence of blinking as blink information.

(Continuous operation possibility determination unit)
The continued driving possibility determining unit 143 allows the occupant to continue driving the vehicle using both or one of the line of sight information and the blink information, depending on the posture type determined by the posture type determining unit 141. Determine whether or not. Specifically, when it is determined that the occupant's posture collapse is posture type 1, the continued operation possibility determination unit 143 determines the possibility of continued operation using line of sight information. More specifically, when the occupant's posture collapse is posture type 1, the continued driving possibility determination unit 143 determines that if the line of sight information indicates that "forward visibility is present", it is possible for the occupant to continue driving. If the line of sight information indicates that there is no forward visibility, it is determined that it is impossible for the occupant to continue driving.

On the other hand, when it is determined that the occupant's posture collapse is posture type 2, the continued operation possibility determination unit 143 determines the possibility of continued operation using the line of sight information and the blink information. More specifically, when the occupant's posture collapse is posture type 2, the continued operation possibility determining unit 143 determines that the line of sight information indicates "forward visibility" and the blink information indicates "blinking". ”, it is determined that it is possible for the occupant to continue driving. Further, when the occupant's posture is posture type 2, the continued operation possibility determining unit 143 determines that the line of sight information is "no forward visibility" or the blink information is "no blinking". When the result is shown, it is determined that it is impossible for the occupant to continue driving.

The result of the determination made by the continued operation possibility determining unit 143 is sent to the driving control system SYS. Output to C.

The occupant condition detection device 1 described above includes a continued operation possibility determination unit 143 that determines whether or not the occupant can continue to drive the vehicle using different visual information depending on the determined posture type. , it is possible to appropriately determine whether or not continued operation is possible depending on the determined posture type. Therefore, according to the occupant condition detection device 1, it is possible to prevent erroneous detection of posture collapse.

<Modified example of occupant status system>
Next, a modification of the occupant status system will be described with reference to FIG. 2. Occupant status system SYS. shown in FIG. The BA includes an occupant condition detection device 1A and an imaging device 2, and further includes a learning device 5. The occupant condition detection device 1A differs from the occupant condition detection device 1 in that it includes a line-of-sight information acquisition section 142A instead of the line-of-sight information acquisition section 142. The line of sight information acquisition section 142A includes a line of sight calculation section 1421A including a learning result utilization section 1421A1 in addition to the eye opening degree calculation section 1422.

(learning device)
The learning device 5 generates and holds a learning model 51 that has learned a reference range that is a normal range of the passenger's line of sight. The learning device 5 generates a learning model 51 by learning the relationship between the position of the seat where the passenger is seated (seat position) and the distribution of the passenger's line of sight during a period when the passenger is considered to be in a normal posture. do. The period during which the occupant is considered to be in a normal posture includes, for example, the period after the occupant gets into the vehicle, the period from the start of driving operation until a predetermined time, and the period when the vehicle restarts after stopping at a traffic light. It is the period from when the It is thought that the range of the passenger's line of sight changes depending on the seat position. Therefore, the relationship between the seat position and the passenger's line of sight is learned. In addition to the seat position, the driving mode of the vehicle, for example, an acceleration mode when the vehicle is accelerating or a constant speed mode where the vehicle travels at a constant speed may be learned as input data. Supervised learning using the passenger's line of sight as training data can be used, but learning may be performed using other methods. The information on the seat position or the driving mode is, for example, acquired by the abnormality determining section 14A from the vehicle information acquisition section and supplied to the learning device 5.

(Learning result utilization department; line of sight calculation department)
In the inference phase, the learning model 51 accepts the current seat position or vehicle driving mode and current line of sight information as input, calculates and outputs a degree of deviation indicating the extent to which the occupant's current line of sight deviates from the reference range. . That is, the abnormality determination unit 14A including the line of sight calculation unit 1421A inputs the line of sight of the occupant into the learning model 51 for inferring the degree of deviation of the line of sight of the occupant from the reference range, and calculates the degree of deviation obtained from the learning model 51. is used to determine whether the occupant is able to continue driving the vehicle. As the deviation degree index, an index that increases as the degree of deviation from the reference range increases is used. The learning model 51 may be provided in the learning device 5 and used, or may be installed and used in the learning result usage section 1421A1. The line-of-sight calculation unit 1421A outputs, as line-of-sight information, a determination result that forward visibility is present when the deviation degree is less than a predetermined threshold, and a determination result that forward visibility is not present when the deviation degree is equal to or greater than the predetermined threshold.

The occupant condition detection device 1A described above also includes the continued operation possibility determining section 143 similarly to the occupant condition detection device 1, so that false detection of posture collapse can be prevented.

<Operation control system configuration>
Hereinafter, with reference to FIG. 3, occupant condition detection system SYS. A driving control system SYS.B provides driving support using the detection results of B. The configuration of C will be explained. Also, for explanation, the occupant state detection system SYS. A vehicle equipped with B is referred to as a first vehicle, and a vehicle other than the first vehicle is referred to as a second vehicle. Note that the first vehicle and the second vehicle may be each expressed as a vehicle without distinguishing between the first vehicle and the second vehicle. In the following, the occupant condition detection system SYS. B is the operation control system SYS. An example in which the occupant condition detection system SYS.C is used as part of the occupant condition detection system SYS. As shown in FIG. 1, operation control system SYS. It may be configured to be able to communicate with C wirelessly or by wire.

FIG. 3 shows the operation control system SYS. It is a block diagram which shows the structure of C. Operation control system SYS. C is the occupant state detection system SYS. A driving control device C3 is provided that controls the driving of the first vehicle by outputting a control signal based on the detection result of B and controlling the steering mechanism 3 and braking/driving mechanism 4 of the first vehicle. In addition, the operation control system SYS. C is a map information storage device C4 that stores map information used for automatic vehicle driving, a surrounding situation monitoring device C5 that monitors the surrounding situation of the vehicle, and a vehicle state acquisition device that obtains information indicating the state of the first vehicle. C6, and an alarm control device C7 that controls warning or notification to the occupants of the vehicle.

In addition, the operation control system SYS. The occupant state detection system SYS. B, driving control device C3, map information storage device C4, surrounding situation monitoring device C5, vehicle status acquisition device C6, and alarm control device C7 are each connected to a communication bus C8, and transmit data via the communication bus C8. Sending and receiving is possible.

The steering mechanism 3 is a mechanism provided in the first vehicle for determining the traveling direction of the first vehicle, and includes, for example, a steering column, a steering shaft, a rack, a pinion, a steering actuator 31, and the like. The braking/driving mechanism 4 is a mechanism for controlling the traveling speed of the first vehicle and switching between forward and backward movement, and includes, for example, an accelerator, a brake, a shift, a braking/driving actuator 41, and the like. The steering actuator 31 that controls the steering mechanism 3 is composed of, for example, an EPS (Electric Power Steering) motor, and the braking/driving actuator 41 that controls the braking/driving mechanism 4 is composed of, for example, an electronically controlled throttle, a brake actuator, etc. configured.

The map information storage device C4 is a storage medium that stores map information including road connection relationships, number of lanes, location information of intersections, and location information of railroad crossings. Here, it is assumed that the map information storage device C4 is installed in the first vehicle, but the map information storage device C4 is configured as a server that transmits map data to the driving control device C3 by communication, for example. Good too. In addition, when configuring the map information storage device C4 as the above-mentioned server, for example, the operation control system SYS. C may be provided with a storage medium, and map information acquired from the map information storage device C4 via an on-vehicle communication device C52, which will be described later, may be stored in the storage medium.

The surrounding situation monitoring device C5 monitors the situation around the first vehicle, and includes a GPS (Global Positioning System) receiver C51, an on-vehicle communication device C52, an external sensor C53, and a navigation system C54.

The GPS receiver C51 receives signals transmitted from GPS positioning satellites and detects the current position of the first vehicle.

The vehicle exterior sensor C53 is composed of, for example, at least one of a camera that images the exterior of the vehicle, a millimeter wave radar, LiDAR, and an ultrasonic sensor, and detects a second vehicle, a pedestrian, and an obstacle that are present around the first vehicle. , or the distance from the first vehicle.

The navigation system C54 calculates a route from the current position of the first vehicle to the destination, and provides guidance on the calculated route. The navigation system C54 has a display section configured with, for example, a liquid crystal display, and is housed within the instrument panel. Furthermore, the navigation system C54 has an operation unit such as a touch panel or physical buttons, and is configured to be able to receive operations from the occupant. Note that the operation unit of the navigation system C54 may be configured with a microphone or the like so as to be able to receive operations using voice emitted by the occupant.

The in-vehicle communication device C52 will be explained. The in-vehicle communication device C52 is, for example, a wireless communication device connected to an antenna for wireless communication. The in-vehicle communication device C52 obtains information on the position of the second vehicle and pedestrians, traffic information, etc. by communicating with the communication device of the second vehicle or the communication device installed on the road. . Here, the traffic information includes, for example, traffic congestion information, traffic regulation information, construction section information, and the like.

Further, the vehicle-mounted communication device C52 can perform vehicle-to-vehicle communication by wireless communication with the vehicle-mounted communication device C52 of a second vehicle located around the first vehicle. In addition, the in-vehicle communication device C52 may perform mobile communication with a base station outside the first vehicle. Further, the on-vehicle communication device C52 is configured to be able to transmit information about the first vehicle output onto the communication bus C8 to the second vehicle, a call center, and the like. Note that the in-vehicle communication device C52 can output information received from the second vehicle and information received from a call center or the like to the communication bus C8. Furthermore, the in-vehicle communication device C52 is configured to be able to acquire information from a server existing outside the vehicle, such as a map information storage device C4 configured as a server that transmits map data to the driving control device C3. Good too.

The vehicle status acquisition device C6 acquires information indicating the status of the first vehicle, and includes a steering angle sensor C61, a vehicle speed sensor C62, a steering torque sensor C63, an accelerator position sensor C64, and a brake position sensor C65.

The steering angle sensor C61 is provided, for example, in the EPS motor or the steering wheel, and detects the steering angle of the first vehicle. The vehicle speed sensor C62 is provided, for example, on a wheel, and detects the traveling speed of the first vehicle.

The steering torque sensor C63 is provided, for example, on the steering wheel, and detects the magnitude of the steering wheel operating force by the driver. The accelerator position sensor C64 detects the amount of depression of the accelerator pedal by the driver. The brake position sensor C65 detects the amount of depression of the brake pedal by the driver.

Next, the alarm control device C7 will be explained. The alarm control device C7 includes an occupant state detection system SYS. Obtain the detection result from B. When the alarm control device C7 obtains a detection result indicating that the occupant is not in a state in which the vehicle can continue to drive, the alarm control device C7 outputs a warning to the occupant via the warning unit C71. The warning unit C71 is, for example, an audio device mounted on the first vehicle, and the warning is a sound or a notification sound outputted from the audio device. Further, the warning section C71 may be configured by the display section of the navigation system C54, and the warning may be displayed as a warning message on the display section of the navigation system C54. As described above, the warning may be anything that can be recognized by the occupant of the first vehicle. Note that the operation control device C3 may be controlled to output a control signal and the alarm control device C7 may issue a warning.

Further, the alarm control device C7 stops the warning if there is a reaction to the warning by the occupant after issuing the warning. Here, the case where there is a reaction to the warning by the occupant means that the occupant condition detection system SYS. If the detection result of B shows that the driver is ready to continue driving, the operation unit installed in the navigation system C54 and the switches installed on the steering wheel accept operations by the occupant. This is a case where it is recognized that there was a conscious action by the occupant in response to the warning. Note that the alarm control device C7 sends a signal indicating that the occupant has responded to the warning to the occupant condition detection system SYS. It may be output to B.

Furthermore, the warning control device C7 detects that the driver is not in a state where the driver can continue to drive, and the occupant state detection system SYS. If the detection result B indicates that the second vehicle To notify a person existing outside the first vehicle, such as a passenger, that the driver of the first vehicle is not in a state where he or she can continue driving the first vehicle. For example, if the occupant does not react to the warning for a predetermined period after the warning is issued, the warning control device C7 activates the notification unit C72 to notify the first person outside the first vehicle. Notify that the driver of the vehicle is not in a condition to continue driving the vehicle.

Note that the alarm control device C7 uses inter-vehicle communication via the above-mentioned in-vehicle communication device C52 to inform the navigation system of the second vehicle that the driver of the first vehicle is not in a state where the driver can continue to drive. The occupants of the second vehicle may be notified by displaying a notification message on the display section of the system C54.

Furthermore, when driving support is executed by the driving control device C3, the notification control section is installed on the vehicle exterior, etc., using the vehicle state acquired by the vehicle state acquisition section or the signal output from the driving control device C3. The notification unit C72, which is composed of the above-mentioned lamps, etc., notifies people outside the first vehicle, such as the occupants of the second vehicle, of the traveling direction of the first vehicle, whether or not there is a lane change, etc. good.

Operation control system SYS. The driving control device C3 of C controls the braking/driving actuator 41 or the braking/driving actuator 41 mounted on the first vehicle to support the driver's driving. The driving control device C3 performs automatic driving control of the first vehicle by outputting a signal to the steering actuator 31, the braking/driving actuator 41, etc., for example. The driving control device C3 controls the first vehicle by, for example, operating a brake to decelerate or stop the first vehicle by controlling a braking/driving actuator 41 composed of an electronically controlled throttle and a brake actuator. Performs drive control. The driving control device C3 performs steering control of the first vehicle, such as by controlling a steering actuator 31 configured with an EPS motor to maintain the lane in which the first vehicle is traveling.

Further, the driving control device C3 has a plurality of driving support functions that support or substitute the driving operation by the driver by controlling the driving force, braking force, steering force, etc. of the first vehicle. For example, the driving support functions include a cruise control function and a lane departure prevention function. In the following description, the cruise control function will be referred to as ACC (Adaptive Cruise Control), and the lane departure prevention function will be referred to as LKA (Lane Keeping Assist).

When the driving control device C3 executes the ACC, the driving control device C3 controls the traveling speed of the first vehicle by adjusting the driving force and the braking force based on the monitoring information of the vehicle in front acquired from the surrounding situation monitoring device C5. If no vehicle in front is detected, the ACC causes the first vehicle to travel at a constant speed at a target speed preset by the driver or the like. On the other hand, if the vehicle in front is detected, the ACC causes the first vehicle to follow the vehicle in front while maintaining the inter-vehicle distance to the vehicle in front.

Further, when the driving control device C3 executes LKA, the driving control device C3 controls the steering force and the steering force based on the shape information of the marking line in the traveling direction acquired from the surrounding situation monitoring device C5. LKA causes the first vehicle to travel along the lane by applying a steering force to the steering wheel in a direction that prevents the first vehicle from approaching the lane marking. Note that the road information output by the surrounding situation monitoring device C5 may be used for vehicle control by ACC and LKA.

Furthermore, when the driving control device C3 executes the emergency evacuation function, it can perform automatic evacuation control (evacuation processing) that automatically stops the first vehicle. When the automatic evacuation control is started, the driving control device C3 causes the surrounding situation monitoring device C5 to search for an evacuation place where the first vehicle is to be stopped. Then, the driving control device C3 moves the first vehicle to the evacuation location set by the search by the surrounding situation monitoring device C5, and stops the first vehicle at this evacuation location. In addition, the above-mentioned evacuation place may be the road shoulder outside the lane where the vehicle is driving on an expressway, and on a general road, it may be a shoulder outside the lane where the vehicle is driving, or a second place such as an intersection, a railroad crossing, a sidewalk, etc. This may be a location that avoids locations where moving objects such as vehicles, trains, and pedestrians are likely to be present.

Here, if the driver is not in a state where he or she can continue driving, there is a high possibility that the operation by the driver is an erroneous operation. For example, if the driver's physical condition suddenly changes and he or she falls down on the steering wheel, there is a high possibility that the steering wheel is operated by mistake, and the driver's physical condition suddenly changes and he or she falls flat on his/her back. If this happens, there is a possibility that the accelerator pedal may be pressed by mistake.

Therefore, if the driving control device C3 executes the driving support function due to the driver not being able to continue driving, the driver's intentional driving operation or the passenger's substitute driving A driving operation (hereinafter referred to as an override operation) for the purpose of preventing an accident, such as an operation, may be enabled, while an erroneous operation by the driver may be disabled. The vehicle state acquisition device C6 may be provided with an override operation detection section C66 that detects an override operation.

The override operation will be explained below. Regarding the acceleration of the vehicle, if the driver is not in a state where he or she can continue driving, the driver's posture may collapse and he or she may accidentally press the accelerator pedal, so the driving support function by the driving control device C3 is executed. In this case, accelerator operation may be disabled. That is, the override operation does not need to include an accelerator operation.

On the other hand, regarding vehicle deceleration, even if the driver is not in a state where he or she can continue driving, a driver who is unconscious may stop the first vehicle in an attempt to avoid collision with an obstacle. There is. Therefore, the override operation detection unit C66 acquires the amount of depression of the brake pedal from the brake position sensor C65, and when the braking force obtained from the depression of the brake pedal is larger than the braking force in the driving support function by the driving control device C3, the override operation detection unit C66 detects the amount of depression of the brake pedal. Pedal operation is detected as an override operation.

Furthermore, regarding the course change of the vehicle, even if the driver is not in a state where he or she can continue driving, a fellow passenger may operate the steering wheel in place of the driver. However, if the driver is not in a condition to continue driving, the driver's posture may collapse and he may fall down on the steering wheel, so the override operation detection unit C66 does not need to detect this erroneous operation as an override operation. good. Therefore, the override operation detection unit C66 detects the condition of the occupant state detection system SYS. From B, the steering wheel operation performed when a detection result indicating that the driver's posture has not deteriorated is detected as an override operation. The override operation detection unit C66 also detects the occupant state detection system SYS. Steering performed when a detection result is obtained from B and surrounding situation monitoring device C5 indicating that the driver or passenger is aware of an obstacle that exists in front or to the side of the first vehicle. Wheel operation is detected as an override operation.

And the operation control system SYS. C replaces the control of the braking/driving actuator 41 or the braking/driving actuator 41 with a driving operation by a fellow passenger or the driver when the override operation detection unit C66 detects an override operation while executing the driving support function. Good too. That is, while the driving support function is being executed, the driving of the vehicle may be controlled by the override operation detected by the override operation detection section C66.

Next, the hardware configurations of the occupant condition detection devices 1 and 1A, the driving control device C3, and the learning device 5 will be described with reference to FIGS. 4A and 4B. As an example, as shown in FIG. 4A, the occupant condition detection device 1 or 1A is realized by a processing circuit 100a. The processing circuit 100a realizes the functions of the functional units included in the occupant condition detection device 1. The processing circuit 100a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of each functional section of the occupant condition detection device 1 may be realized by separate processing circuits, or these functions may be realized by a single processing circuit. The operation control device C3 and the learning device 5 can also be realized with a similar configuration.

As another example, as shown in FIG. 4B, the occupant state detection device 1 or 1A is realized by a processor 100b and a memory 100c. The programs stored in the memory 100c are read out and executed by the processor 100b, thereby realizing the functions of the functional units included in the occupant condition detection device 1. The program is implemented as software, firmware, or a combination of software and firmware. Examples of the memory 100c include RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electric Memory). Non-volatile or volatile semiconductors such as ally-EPROM) Includes memory, magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.

<Operation>
Next, with reference to FIG. 6, the operation of the occupant condition detection device 1 will be described.

In step ST101, the image acquisition unit 11 acquires an image of the occupant captured by the imaging device 2 from the imaging device 2.

In step ST102, the shape information detection unit 131 acquires the shape related to the physical characteristics of the occupant from the image acquired by the image acquisition unit 11.

In step ST103, the feature amount calculation unit 132 calculates the amount of change in the shape of the shape acquired by the shape information detection unit 131 as a feature amount. The shape acquired by the shape information detection section 131 and the feature amount calculated by the feature amount calculation section 132 are outputted to the abnormality determination section 14 as feature information.

In step ST104, the posture type determining section 141 included in the abnormality determining section 14 determines the type of posture collapse of the occupant using the characteristic information. For example, if the type of posture collapse of the occupant falls under Posture Type 1, which includes ``prone,'' ``prone,'' ``back,'' and ``shrimp,'' or ``neck only falls to the side,'' ``sideways,'' and It is determined whether the posture falls under posture type 2, which includes "leaning back".

In step ST105, the visual information acquisition unit 142 included in the abnormality determination unit 14 uses the feature information to calculate visual information including gaze information and blink information.

In step ST106, the continued operation possibility determining unit 143 determines whether or not forward visibility is performed by the line of sight information and/or blink information and the line of sight calculation unit 1421, depending on the posture type determined by the posture type determining unit 141. Using the determination result, it is determined whether the occupant can continue driving the vehicle.

Here, with reference to FIG. 7, details of the continued operation possibility determination process performed by the continued operation possibility determination unit 143 will be described. In step ST1061, the continued operation possibility determination unit 143 determines whether the type of posture collapse of the occupant is posture type 1 or not.

When the type of posture collapse of the occupant is posture type 1, in step ST1062, the continued operation possibility determination unit 143 determines the possibility of continued operation using the line of sight information. More specifically, when the occupant's posture collapse is posture type 1, the continued driving possibility determining unit 143 determines that the occupant is not allowed to continue driving when the line of sight information indicates that "forward visibility is present". If it is determined that there is, and the line-of-sight information indicates that there is no forward visibility, it is determined that it is impossible for the occupant to continue driving.

If the type of posture collapse of the occupant is not posture type 1, that is, if the type of posture collapse is posture type 2, in step ST1063, the continued operation possibility determination unit 143 determines whether continued operation is possible using the line of sight information and blink information. Determine gender. More specifically, when the occupant's posture collapse is posture type 2, the continued operation possibility determining unit 143 determines that the line of sight information indicates "forward visibility" and the blink information indicates "blinking". ”, it is determined that it is possible for the occupant to continue driving, and the line of sight information indicates that there is no forward visibility, or the blink information indicates that there is no blinking. When this occurs, it is determined that it is impossible for the occupant to continue driving.

In addition, although the above process has been explained based on the case where there are two types of posture collapse, Posture Type 1 and Posture Type 2, it is possible to classify into more types and use appropriate indicators to evaluate the possibility of continued operation. You may judge.

Next, the operation control system SYS. The driving support function of C will be explained. FIG. 8 shows the operation control system SYS. 3 is a flowchart showing an example of the operation of a driving support function according to C. In the following explanation, the occupant condition detection system SYS. If it is detected by the driving control system SYS.B that the driver is not in a state where the driver can continue to drive. An example will be described in which C performs an emergency evacuation function of the vehicle as a driving support function. In addition, the operation control system SYS. The operation C is started, for example, after the ignition of the vehicle is turned on.

First, the operation control system SYS. The operation control device C3 of C is an occupant state detection system SYS.C. A detection result regarding the state of the driver of the first vehicle is acquired from B (ST201), and it is determined whether the driver is in a state where the driver can continue driving (ST202). Note that the determination as to whether or not the occupant is able to continue driving is made by the occupant condition detection system SYS. B may go. If the detection result of the occupant's condition indicates that the occupant is in a state where the occupant can continue to drive (ST202: NO), the driving control system SYS. The operation of C proceeds to the process of ST201.

On the other hand, if the detection result regarding the occupant's condition indicates that the occupant is not in a state where the occupant can continue to drive, the alarm control device C7 activates the warning section C71 and starts warning the occupant (ST203). . Here, the warning to the occupant by the warning unit C71 means, for example, the display of a warning message on the display unit of the navigation system C54 installed in the first vehicle, or the display of a warning message from the audio device installed in the first vehicle. This is the output of Note that the warning to the occupant is to notify that the occupant is not in a state where he or she can continue driving, or to notify that a driving support function such as an emergency evacuation function will be performed.

Next, the warning control device C7 determines whether or not a reaction has been obtained from the occupant to the warning (ST204). Here, the determination as to whether or not a reaction has been obtained from the occupant to the warning is made, for example, by checking the occupant condition detection system SYS. This is performed based on whether or not a detection result indicating that the occupant is in a state in which the passenger can continue to drive is obtained from B.

That is, the warning control device C7 activates the occupant state detection system SYS. If a detection result indicating that the occupant is in a state in which the vehicle can continue to drive is obtained from B, it is determined that a reaction has been obtained from the occupant to the warning. On the other hand, after activating the warning unit C71, the warning control device C7 activates the occupant state detection system SYS. If a predetermined period of time has elapsed without a detection result indicating that the occupant is in a continued driving state from B, it is determined that no reaction has been obtained from the occupant to the warning. Note that the determination as to whether or not a reaction has been obtained from the occupant to the warning can be made by, for example, issuing a warning by operating the operating section by the occupant before a predetermined time has elapsed after activating the warning section C71. It may be performed based on whether or not an operation for canceling has been performed. Further, the predetermined time is, for example, 3 seconds.

If a reaction is obtained from the occupant to the warning (ST204: NO), the warning control device C7 stops the warning from the warning unit C71 (ST205), and the driving control system SYS. The operation of C proceeds to the process of ST201. On the other hand, if the occupant does not respond to the warning (ST204: YES), the warning control device C7 activates the notification unit C72 and starts notifying the person present outside the first vehicle ( ST206). Here, the notification to the second vehicle means that the driver of the first vehicle is not in a state where the driver of the first vehicle is not able to continue driving. or to notify that the first vehicle will start a driving support function such as an emergency evacuation function.

The operation control device C3 then starts the emergency evacuation function. First, the driving control device C3 obtains the vehicle state from the vehicle state obtaining device C6 (ST207). Note that the vehicle state is information indicating the state of the first vehicle, and is used for vehicle control in a driving support function such as an emergency evacuation function.

Next, the driving control device C3 acquires the surrounding situation of the first vehicle from the surrounding situation monitoring device C5 (ST208), and sets the position to which the first vehicle is evacuated (ST209). Hereinafter, the position where the first vehicle is evacuated will be referred to as the evacuated position. Note that the evacuation position may be within a range of 150 m from the current position of the first vehicle, or a position where the time required for the first vehicle to move from the current position is within 60 seconds. In this way, it is possible to prevent the number of times the first vehicle enters the intersection from increasing more than necessary as the first vehicle moves to the retreat position.

For example, the driving control device C3 searches for a position where the first vehicle can be safely stopped from the surrounding situation monitoring device C5, and selects the safest position from among the candidate positions where the first vehicle can be stopped safely. Set as position. Here, the evacuation position is defined as a location where there is no possibility of collision with a moving object such as a second vehicle or a pedestrian, or an obstacle, and where the aforementioned moving object is located on the route from the current position of the first vehicle to the evacuation position. and a location where there is no possibility of collision with obstacles. Examples of evacuation positions include road edges such as road shoulders, intersections, and inside lanes other than railroad crossings. Further, when the evacuation position is at the edge of the road such as a road shoulder, a space may be secured for the passenger to escape from the first vehicle.

After setting the retreat position, the driving control device C3 performs vehicle control and stops the first vehicle at the retreat position. Note that in cases such as when it is determined in the process of ST204 that there is no response to the warning, the occupant condition detection system SYS. If the detection result of B is not due to false detection and there is a high possibility that the driver is not in a state where he or she can continue driving, the driving control device C3 will control the ACC or LKA may also be performed. Further, after setting the evacuation position, the driving control device C3 may display the route from the current position of the first vehicle to the set evacuation position on the display unit of the navigation system C54.

First, the driving control device C3 determines whether a course change is required before moving the first vehicle to the retreat position (ST210). If it is not necessary to change the course before moving the first vehicle to the evacuation position (ST210: NO), that is, if the evacuation position is set within the lane in which the first vehicle is currently traveling, the braking/driving The actuator 41 and the brake/drive actuator 41 are controlled to cause the first vehicle to travel within the lane in which it is currently traveling (ST211). Then, the driving control device C3 controls the braking/driving actuator 41 and the braking/driving actuator 41 to decelerate the first vehicle and then stop it at the retreat position (ST212).

Note that in the process of ST211, the driving control device C3 may cause the first vehicle to travel at a speed that allows an emergency stop, such as 10 km/h, for example. Furthermore, if the first vehicle does not need to continue traveling, such as when the retreat position is set near the first vehicle, the process of ST211 can be omitted. Furthermore, in the processing of ST211 and ST212, when the first vehicle is decelerated by the driving control device C3, the deceleration may be, for example, 3 m/s ² or less in order to prevent the passenger from falling.

Then, after the first vehicle is stopped at the retreat position by the driving control device C3, a notification is sent to a call center or the like outside the vehicle via the on-vehicle communication device C52 (ST213). The report via the in-vehicle communication device C52 includes, for example, a message indicating that the driver of the first vehicle is unable to continue driving, the current position of the first vehicle acquired by the surrounding situation monitoring device C5, and The evacuation position set by the operation control device C3 is included. In addition, even when the notification is made using the in-vehicle communication device C52, the driving control device C3 continues the stopped state of the first vehicle, and the alarm control device C7 continues to notify the person who is present outside the first vehicle. You can.

A case will be described in which the driving control device C3 determines in the process of ST210 that a course change is required before moving the first vehicle to the retreat position (ST210: YES). Note that a change in course refers to a case in which the first vehicle is moved to a lane adjacent to the lane in which the first vehicle is traveling, such as a lane change; This includes the case where the first vehicle deviates from the lane in which the first vehicle is traveling.

When determining that a course change is required, the driving control device C3 controls the brake/drive actuator 41 and the brake/drive actuator 41 to change the course of the first vehicle (ST214). Here, in order to enable the second vehicle or pedestrian, etc. present at the destination of the change of course to recognize the change of course of the first vehicle and avoid a collision, the speed of movement of the vehicle in the lateral direction during the change of course is set at 0.3 m. It may be about /s. Further, when changing the course, the alarm control device C7 sends a first notification that the first vehicle will start changing the course by the notification unit C72 at least a predetermined time before the start of the change of course. may be notified to people outside the vehicle.

Next, if no reaction is obtained from the occupant to the alarm from the alarm unit, the notification unit C72 informs the occupant of the first vehicle that the occupant of the first vehicle is in a state in which he or she can continue driving. make a notification indicating that there is no such thing. Note that the notification to the second vehicle by the notification unit C72 is not limited to notification indicating that the occupant of the first vehicle is not in a condition to continue driving; It may be a notification indicating that the emergency evacuation function is started or a notification indicating that the emergency evacuation function of the vehicle is being executed.

Then, similarly to the process in ST212, the driving control device C3 controls the steering actuator 31 and the braking/driving actuator 41 to decelerate the first vehicle and then stop it at the retreat position (ST215). After the first vehicle is stopped at the retreat position by the driving control device C3, the driving control system SYS. The operation of C proceeds to the process of ST213, and a notification is sent to a call center or the like outside the vehicle via the on-vehicle communication device C52.

In addition, the operation control system SYS. C may replace the control of the steering actuator 31 or the braking/driving actuator 41 with a driving operation by a fellow passenger or the driver when an override operation is performed while the driving support function is being executed. That is, for example, when the override operation detection unit C66 detects an override operation during the processing of ST206 to ST215 in FIG. 8, the vehicle may be driven according to the detected override operation.

Note that in the above example, the occupant state detection system SYS. If it is detected by the driving control system SYS.B that the driver is not in a state where the driver can continue to drive. Although the driving control system SYS. C may execute ACC or LKA as a driving support function. In addition, the operation control system SYS. According to C, after executing ACC and LKA, a driving support function may be combined, such as executing an emergency evacuation function when the safety of the surrounding area can be ensured. In this way, when the driver is not in a state where he or she can continue driving, the operation control system SYS. If C makes it possible to prevent accidents, driving support functions can be combined as appropriate.

<Additional notes>
Some aspects of the various embodiments described above are summarized as follows.

(Additional note 1)
The occupant condition detection device (1; 1A) of Supplementary Note 1 includes an image acquisition unit (11) that acquires an image of the interior of a vehicle captured by an imaging device; a posture type determination unit that determines to which posture type the posture corresponds among a plurality of posture types including a predetermined first posture type and a second posture type different from the first posture type; (141; 141A), and a visual information acquisition unit (142; 142A) that calculates visual information including line-of-sight information regarding the direction of the passenger's line of sight and blink information regarding the presence or absence of blinking based on the acquired captured image. , a continued operation possibility determination unit (143) that determines whether or not continued operation of the vehicle by the occupant is possible using different visual information depending on the posture type determined to be applicable. , an abnormality determination section (14; 14A);

(Additional note 2)
The occupant condition detection device (1; 1A) of Supplementary Note 2 is the occupant condition detection device (1; 1A) of Supplementary Note 1, in which the continued operation possibility determination unit determines whether the posture of the occupant is of the first posture type. If it is determined that this applies, the line of sight information is used to determine whether or not the occupant can continue driving the vehicle.

(Additional note 3)
The occupant condition detection device (1; 1A) according to Supplementary Note 3 is the occupant condition detection device (1; 1A) according to Supplementary Note 1 or 2, and the continued operation possibility determination unit determines whether the posture of the occupant is in the second position. When it is determined that the posture type is applicable, it is determined whether or not the occupant can continue driving the vehicle using the line of sight information and the blink information.

(Additional note 4)
The occupant condition detection device (1; 1A) of appendix 4 is the occupant condition detection device (1; 1A) according to any one of appendices 1 to 3, and the second posture type is the head or face of the occupant. , or the upper body is tilted laterally to the right or left with respect to the longitudinal direction of the vehicle.

(Appendix 5)
The occupant condition detection device (1A) according to appendix 5 is the occupant condition detection device (1A) according to any one of appendices 1 to 4, and the abnormality determination unit determines the degree of deviation of the passenger's line of sight from a reference range. The line of sight of the occupant is input into a learning model (51) for inference, and the degree of deviation obtained from the learning model is used to determine whether the occupant is able to continue driving the vehicle. do.

(Appendix 6)
The occupant condition detection device (1; 1A) in Appendix 6 is the occupant condition detection device (1; 1A) according to any one of Appendixes 1 to 5, and indicates the physical characteristics of the occupant from the captured image. The apparatus further includes a feature information detection section (13) that detects feature information, and the visual information acquisition section calculates the visual information from the detected feature information.

(Appendix 7)
The occupant condition detection system (SYS.B) of Appendix 7 includes the occupant condition detection device (1; 1A) of any one of Appendixes 1 to 6 and the imaging device.

(Appendix 8)
The vehicle control system (SYS.A) in Appendix 8 includes the occupant condition detection device (1; 1A) according to any one of Appendixes 1 to 6, and a driving control device (C3) that controls the driving of the vehicle. In the vehicle control system, the occupant condition detection device outputs the result of the determination to the driving control device when it is determined that the vehicle cannot be continuously driven by the occupant, and Based on the determination result, a control signal for outputting a warning to the occupant or evacuation processing of the vehicle is sent to an alarm control device (C7), a steering mechanism (3), or a braking/driving mechanism mounted on the vehicle. (4) Output.

(Appendix 9)
The occupant condition detection method in Appendix 9 is a method for detecting an occupant condition that includes an image acquisition section (11), a posture type determination section (141; 141A), a visual information acquisition section (142; 142A), and a continued operation possibility determination section (143). A passenger state detection method performed by a state detection device, wherein the image acquisition unit acquires a captured image of the interior of the vehicle captured by an imaging device (ST101), and the posture type determination unit includes Based on the captured image, the posture of the occupant of the vehicle is determined to be one of a plurality of posture types including a predetermined first posture type and a second posture type different from the first posture type. a step of determining whether the situation is applicable (ST104), and the visual information acquisition unit acquires visual information including visual line information regarding the direction of the passenger's line of sight and blink information regarding the presence or absence of blinking based on the acquired captured image. In the step of calculating (ST105), the continued operation possibility determination unit determines whether or not it is possible for the occupant to continue driving the vehicle using different visual information depending on the posture type determined to be applicable. A determining step (ST106) is provided.

Note that it is possible to combine the embodiments, or to modify or omit each embodiment as appropriate.

The occupant state detection device of the present disclosure can be mounted on a vehicle and used as a device for detecting whether or not the vehicle occupant can drive the vehicle.

Reference Signs List 1 (1A) Occupant condition detection device, 2 Imaging device, 3 Steering mechanism, 4 Braking/driving mechanism, 5 Learning device, 11 Image acquisition unit, 12 Vehicle information acquisition unit, 13 Characteristic information detection unit, 14 (14A) Abnormality determination unit , 31 steering actuator, 41 braking/driving actuator, 51 learning model, 100a processing circuit, 100b processor, 100c memory, 131 shape information detection unit, 132 feature amount calculation unit, 141 posture type determination unit, 142 (142A) visual information acquisition unit , 143 Continuous driving possibility determination unit, 1421 Line of sight calculation unit, 1421A Line of sight calculation unit, 1421A1 Learning result utilization unit, 1422 Eye opening degree calculation unit, C3 Driving control device, C4 Map information storage device, C5 Surrounding situation monitoring device, C51 GPS Receiver, C52 In-vehicle communication device, C53 External sensor, C54 Navigation system, C6 Vehicle status acquisition device, C61 Rudder angle sensor, C62 Vehicle speed sensor, C63 Steering torque sensor, C64 Accelerator position sensor, C65 Brake position sensor, C66 Override operation detection section, C7 alarm control device, C71 warning section, C72 notification section, C8 communication bus.

Claims (9)

an image acquisition unit that acquires an image of the interior of the vehicle captured by the imaging device;
Based on the acquired captured image, the posture of the occupant of the vehicle is determined to be one of a plurality of posture types including a predetermined first posture type and a second posture type different from the first posture type. a posture type determination unit that determines whether the posture corresponds to the posture type;
a visual information acquisition unit that calculates visual information including line-of-sight information regarding the direction of the passenger's line of sight and blink information regarding the presence or absence of blinking based on the acquired captured image;
a continued operation possibility determination unit that determines whether or not the occupant can continue to drive the vehicle using different visual information depending on the posture type determined to be applicable;
an anomaly determination section comprising;
A passenger condition detection device equipped with When it is determined that the posture of the occupant corresponds to the first posture type, the continued operation possibility determination unit determines whether or not it is possible for the occupant to continue driving the vehicle using the line-of-sight information. determine whether
The occupant condition detection device according to claim 1. When it is determined that the posture of the occupant corresponds to the second posture type, the continued driving possibility determination unit causes the occupant to continue driving the vehicle using the line of sight information and the blink information. determine whether it is possible to
The occupant condition detection device according to claim 1. The second posture type is a posture in which the head, face, or upper body of the occupant is tilted to the right or left side with respect to the longitudinal direction of the vehicle.
The occupant condition detection device according to claim 1. The abnormality determining unit inputs the passenger's line of sight into a learning model for inferring the degree of deviation of the passenger's line of sight from a reference range, and uses the degree of deviation obtained from the learning model to determine whether the passenger's line of sight deviates from a reference range. determining whether it is possible to continue driving the vehicle;
The occupant condition detection device according to claim 1. further comprising a feature information detection unit that detects feature information indicating physical characteristics of the occupant from the acquired captured image,
The visual information acquisition unit calculates the visual information from the detected feature information.
The occupant condition detection device according to claim 1. An occupant condition detection device according to any one of claims 1 to 6;
the imaging device;
Occupant status detection system equipped with An occupant condition detection device according to any one of claims 1 to 6;
a driving control device that controls driving of the vehicle;
A vehicle control system comprising:
The occupant condition detection device outputs a result of the determination to the driving control device when it is determined that the vehicle cannot be continuously driven by the occupant;
Based on the determination result, the driving control device outputs a control signal for outputting a warning to the occupant or performing an evacuation process for the vehicle to an alarm control device, a steering mechanism, or a brake/drive mechanism mounted on the vehicle. do,
Vehicle control system. An occupant condition detection method performed by an occupant condition detection device including an image acquisition section, a posture type determination section, a visual information acquisition section, and a continued operation possibility determination section, the method comprising:
a step in which the image acquisition unit acquires a captured image of the interior of the vehicle captured by an imaging device;
The posture type determining unit determines, based on the acquired captured image, a plurality of postures including a first posture type in which the posture of the occupant of the vehicle is predetermined and a second posture type different from the first posture type. a step of determining which of the posture types the posture corresponds to;
The visual information acquisition unit calculates visual information including visual line information regarding the direction of the passenger's line of sight and blink information regarding the presence or absence of blinking based on the acquired captured image;
a step in which the continued operation possibility determining unit determines whether or not it is possible for the occupant to continue driving the vehicle, using different visual information depending on the posture type determined to be applicable;
A method for detecting occupant status.

Images