WO2024252912A1 - Information processing device, information processing method, and information processing system - Google Patents

Abstract

The present technology relates to an information processing device, an information processing method, and an information processing system that make it possible to more appropriately determine a driver's cognitive function. An information processing device according to the present technology is provided with: a reaction prediction unit that predicts a driver's reaction to an event occurring around the driver; a reaction recognition unit that recognizes the driver's actual reaction to the event; a determination unit that compares a predicted reaction, which is the reaction predicted by the reaction prediction unit, with the actual reaction to determine whether or not the actual reaction is a normal reaction; and an execution unit that executes a prescribed process if it is determined that the actual reaction is not a normal reaction. The present technology can be applied, for example, to a DMS that recognizes driver status.

Description

Information processing device, information processing method, and information processing system

This technology relates to an information processing device, an information processing method, and an information processing system, and in particular to an information processing device, an information processing method, and an information processing system that can more appropriately determine the driver's cognitive function.

In recent years, cases of drivers with impaired cognitive function due to early-stage dementia and other conditions causing traffic accidents have become a social issue, and there is a demand for ways to assess drivers' cognitive function and prevent traffic accidents before they occur.

For example, Patent Document 1 describes a technology that issues a warning to the driver if the driver's gaze is not directed in the correct direction.

JP 2002-83400 A

The technology described in Patent Document 1, for example, cannot accurately assess the driver's cognitive function if the driver looks in the correct direction but does not recognize the danger.

This technology was developed in light of these circumstances, and makes it possible to more appropriately assess a driver's cognitive function.

The information processing device according to the first aspect of the present technology includes a reaction prediction unit that predicts the driver's reaction to an event occurring around the driver, a reaction recognition unit that recognizes the driver's actual reaction to the event, a determination unit that compares the actual reaction with a predicted reaction predicted by the reaction prediction unit to determine whether the actual reaction is normal, and an execution unit that executes a predetermined process when it is determined that the actual reaction is not normal.

In the information processing method according to the first aspect of the present technology, an information processing device predicts the driver's reaction to an event occurring around the driver, recognizes the driver's actual reaction to the event, compares the predicted reaction with the actual reaction to determine whether the actual reaction is normal or not, and executes a predetermined process if it is determined that the actual reaction is not normal.

The information processing system of the second aspect of the present technology has an information processing device including a first sensor used to recognize the internal situation of a vehicle driven by a driver, a reaction prediction unit that predicts the reaction of the driver to an event occurring around the driver, a reaction recognition unit that recognizes the actual reaction of the driver to the event based on sensor data from the first sensor, a determination unit that compares the reaction predicted by the reaction prediction unit with the actual reaction and determines whether the actual reaction is normal or not, and an execution unit that executes a predetermined process if it is determined that the actual reaction is not normal.

In a first aspect of this technology, the driver's reaction to an event occurring around the driver is predicted, the driver's actual reaction to the event is recognized, the predicted reaction is compared with the actual reaction to determine whether the actual reaction is normal, and if it is determined that the actual reaction is not normal, a predetermined process is executed.

In a second aspect of the present technology, the driver's reaction to an event occurring around the driver is predicted, the actual reaction of the driver to the event is recognized based on sensor data from a first sensor used to recognize the internal situation of the vehicle driven by the driver, the predicted reaction is compared with the actual reaction to determine whether the actual reaction is normal, and if it is determined that the actual reaction is not normal, a predetermined process is executed.

FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system. FIG. 2 is a diagram showing an example of a sensing region. FIG. 1 illustrates an example of a vehicle control system for determining a driver's cognitive function. 1 is a block diagram showing a configuration example of a vehicle control system to which the present technology is applied. 4 is a block diagram showing a detailed configuration example of a cognitive function assessment unit. FIG. 4 is a flowchart illustrating a first process performed by the vehicle control system. 10 is a flowchart illustrating a second process performed by the vehicle control system. 10 is a flowchart illustrating a third process performed by the vehicle control system. FIG. 1 illustrates an example of a driver's abnormal reaction to a dangerous event. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer.

Hereinafter, an embodiment of the present technology will be described in the following order.
1. Configuration Example of Vehicle Control System 2. Embodiment

1. Example of vehicle control system configuration
FIG. 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is an example of a mobility device control system to which the present technology is applied.

The vehicle control system 11 is provided in the vehicle 1 and performs processing related to the automated driving of the vehicle 1. This automated driving includes driving automation of levels 1 to 5, as well as remote driving and remote assistance of the vehicle 1 by a remote driver.

The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

The vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 41 is composed of an in-vehicle communication network or bus that complies with digital two-way communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 41 may be used differently depending on the type of data being transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-volume data. In addition, each part of the vehicle control system 11 may be directly connected without going through the communication network 41, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

Note that, hereinafter, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the vehicle control ECU 21 and the communication unit 22 communicating with each other.

The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls all or part of the functions of the vehicle control system 11.

The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 22 can communicate using multiple communication methods.

The following provides an overview of the communications with the outside of the vehicle that can be performed by the communication unit 22. The communication unit 22 communicates with servers (hereinafter referred to as external servers) on an external network via base stations or access points using wireless communication methods such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the Internet, a cloud network, or an operator-specific network. The communication method that the communication unit 22 uses with the external network is not particularly limited as long as it is a wireless communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined distance.

Furthermore, for example, the communication unit 22 can communicate with a terminal present in the vicinity of the vehicle using P2P (Peer To Peer) technology. The terminal present in the vicinity of the vehicle can be, for example, a terminal attached to a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed position in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, vehicle-to-home communication with a home, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian, etc.

The communication unit 22 can, for example, receive from the outside a program for updating software that controls the operation of the vehicle control system 11 (Over the Air). The communication unit 22 can further receive map information, traffic information, information about the surroundings of the vehicle 1, etc. from the outside. For example, the communication unit 22 can also transmit information about the vehicle 1 and information about the surroundings of the vehicle 1 to the outside. Information about the vehicle 1 that the communication unit 22 transmits to the outside includes, for example, data indicating the state of the vehicle 1, the recognition results by the recognition unit 73, etc. Furthermore, for example, the communication unit 22 performs communication corresponding to a vehicle emergency notification system such as e-Call.

For example, the communication unit 22 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (Vehicle Information and Communication System) (registered trademark)) such as a radio beacon, optical beacon, or FM multiplex broadcasting.

The following provides an overview of the communication that the communication unit 22 can perform with the inside of the vehicle. The communication unit 22 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 22 can perform wireless communication with each device in the vehicle using a communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined speed via wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). Not limited to this, the communication unit 22 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 22 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with each device in the vehicle using a communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined speed via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (Mobile High-definition Link).

Here, the in-vehicle device refers to, for example, a device that is not connected to the communication network 41 inside the vehicle. Examples of in-vehicle devices include mobile devices and wearable devices carried by users inside the vehicle, such as the driver, and information devices brought into the vehicle and temporarily installed.

The map information storage unit 23 stores one or both of a map acquired from an external source and a map created by the vehicle 1. For example, the map information storage unit 23 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, etc.

High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 1 from an external server or the like. A point cloud map is a map made up of a point cloud (point cloud data). A vector map is, for example, a map that is adapted for driving automation by associating traffic information such as the positions of lanes and traffic lights with a point cloud map.

The point cloud map and vector map may be provided, for example, from an external server, or may be created by the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described below, and stored in the map information storage unit 23. In addition, when a high-precision map is provided from an external server, etc., map data of, for example, an area of several hundred meters square regarding the planned route along which the vehicle 1 will travel is acquired from the external server, etc., in order to reduce communication capacity.

The location information acquisition unit 24 receives GNSS signals from Global Navigation Satellite System (GNSS) satellites and acquires location information of the vehicle 1. The acquired location information is supplied to the driving automation control unit 29. Note that the location information acquisition unit 24 is not limited to a method using GNSS signals, and may acquire location information using a beacon, for example.

The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors among the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDAR 53, and ultrasonic sensors 54 is not particularly limited as long as it is a number that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

The imaging method of camera 51 is not particularly limited. For example, cameras of various imaging methods, such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera, which are imaging methods capable of distance measurement, can be applied to camera 51 as necessary. However, the present invention is not limited to this, and camera 51 may simply be used for acquiring photographic images, without regard to distance measurement.

Furthermore, for example, the external recognition sensor 25 can be equipped with an environmental sensor for detecting the environment relative to the vehicle 1. The environmental sensor is a sensor for detecting the environment such as the weather, climate, brightness, etc., and can include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

Furthermore, for example, the external recognition sensor 25 includes a microphone that is used to detect sounds around the vehicle 1 and the location of sound sources.

The in-vehicle sensor (in-cabin sensor) 26 includes various sensors used to recognize the situation inside the vehicle, and supplies sensor data from each sensor to each part of the vehicle control system 11. There are no particular limitations on the types and number of the various sensors included in the in-vehicle sensor 26, so long as they are of the types and number that can be realistically installed in the vehicle 1.

For example, the in-vehicle sensor 26 may be equipped with one or more types of sensors including a camera, a depth sensor such as a radar, a seating sensor, a microphone, and a biometric sensor. The camera equipped in the in-vehicle sensor 26 may be a camera using various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. Not limited to this, the camera equipped in the in-vehicle sensor 26 may be a camera simply for acquiring captured images, regardless of distance measurement. The biometric sensor equipped in the in-vehicle sensor 26 is provided, for example, on a seat, steering wheel, etc., and detects various types of biometric information of the user.

The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from each sensor to each part of the vehicle control system 11. There are no particular limitations on the types and number of the various sensors included in the vehicle sensor 27, so long as they are of the types and number that can be realistically installed on the vehicle 1.

For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 27 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor that detects the remaining charge and temperature of the battery, and an impact sensor that detects external impacts.

The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of storage media include magnetic storage devices such as an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access Memory) and/or a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by one or more (or in some cases all) units of the vehicle control system 11. In some embodiments, the storage unit 28 includes an EDR (Event Data Recorder) and/or a DSSAD (Data Storage System for Automated Driving), and may store information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensor 26.

The driving automation control unit 29 controls the driving automation function of the vehicle 1. For example, the driving automation control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

The analysis unit 61 performs analysis processing of the vehicle 1 and the surrounding conditions. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

The self-position estimation unit 71 estimates the self-position of the vehicle 1 based on the sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25, and estimates the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 is based on, for example, the center of the rear wheel pair axle.

The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size, and the occupancy state of objects is shown on a grid-by-grid basis. The occupancy state of objects is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used, for example, in detection processing and recognition processing of the situation outside the vehicle 1 by the recognition unit 73.

The self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and the sensor data from the vehicle sensor 27.

The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (e.g., image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include compounding, integration, fusion, and association.

The recognition unit 73 executes a detection process to detect the situation outside the vehicle 1, and a recognition process to recognize the situation outside the vehicle 1.

For example, the recognition unit 73 performs detection and recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

Specifically, for example, the recognition unit 73 performs detection processing and recognition processing of objects around the vehicle 1. Object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. Object recognition processing is, for example, processing to recognize attributes such as the type of object, and to identify a specific object. However, detection processing and recognition processing are not necessarily clearly separated, and there may be overlap.

For example, the recognition unit 73 detects objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52, the LiDAR 53, or the like into clusters of points. This allows the presence or absence, size, shape, and position of objects around the vehicle 1 to be detected.

For example, the recognition unit 73 detects the movement of objects around the vehicle 1 by performing tracking to follow the movement of clusters of point clouds classified by clustering. This allows the speed and direction of travel (movement vector) of objects around the vehicle 1 to be detected.

For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. The recognition unit 73 may also recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of self-location estimation by the self-location estimation unit 71, and the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the positions and states of traffic lights, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes on which travel is possible, etc.

For example, the recognition unit 73 can perform recognition processing of the environment around the vehicle 1. The surrounding environment that the recognition unit 73 recognizes may include weather, temperature, humidity, brightness, and road surface conditions.

The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 creates the behavior plan by performing route planning and route following processing.

Route planning includes global path planning and local path planning. Global path planning involves planning a rough route from the start to the goal. Local path planning is also called trajectory planning, and involves generating a trajectory that allows safe and smooth progress in the vicinity of vehicle 1 on the planned route, taking into account the motion characteristics of vehicle 1.

Path following is a process of planning operations for safely and accurately traveling along a route planned by a route plan within a planned time. The action planning unit 62 can, for example, calculate the target speed and target angular velocity of the vehicle 1 based on the results of this path following process.

The operation control unit 63 controls the operation of the vehicle 1 to realize the action plan created by the action planning unit 62.

For example, the operation control unit 63 controls the steering control unit 81, the brake control unit 82, and the drive control unit 83 included in the vehicle control unit 32 described below, to perform lateral vehicle motion control and longitudinal vehicle motion control so that the vehicle 1 moves along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 performs control aimed at driving automation, such as driver assistance functions such as collision avoidance or impact mitigation, following driving, maintaining vehicle speed, collision warning for the vehicle itself, and lane departure warning for the vehicle itself, and driving without the operation of the driver or a remote driver.

The DMS 30 performs processes such as authenticating the driver and recognizing the driver's state based on the sensor data from the in-vehicle sensors 26 and the input data input to the HMI 31 (described later). Examples of the driver's state to be recognized include physical condition, alertness, concentration, fatigue, line of sight, level of intoxication, driving operation, posture, cognitive function, etc.

The DMS 30 may also perform authentication processing for users other than the driver and recognition processing for the status of the users. For example, the DMS 30 may also perform recognition processing for the status inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of the status inside the vehicle that may be recognized include temperature, humidity, brightness, odor, etc.

HMI31 inputs various data and instructions, and displays various data to the user.

The following provides an overview of data input using the HMI 31. The HMI 31 is equipped with an input device that allows a person to input data. The HMI 31 generates input signals based on data and instructions input via the input device, and supplies the signals to each part of the vehicle control system 11. The HMI 31 is equipped with input devices such as a touch panel, buttons, switches, and levers. Without being limited to these, the HMI 31 may further be equipped with an input device that allows information to be input by a method other than manual operation, such as voice or gestures. Furthermore, the HMI 31 may use, as an input device, an externally connected device such as a remote control device that uses infrared or radio waves, or a mobile device or wearable device that supports the operation of the vehicle control system 11.

The presentation of data by the HMI 31 will be briefly described below. The HMI 31 generates visual information, auditory information, and tactile information for the user or the outside of the vehicle. The HMI 31 also performs output control to control the output, output content, output timing, output method, etc. of each piece of generated information. The HMI 31 generates and outputs, as visual information, information indicated by images or light, such as an operation screen, vehicle 1 status display, warning display, and monitor image showing the situation around the vehicle 1. The HMI 31 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 31 also generates and outputs, as tactile information, information that is imparted to the user's sense of touch by force, vibration, movement, etc.

The output device from which the HMI 31 outputs visual information may be, for example, a display device that presents visual information by displaying an image itself, or a projector device that presents visual information by projecting an image. Note that the display device may be a device that displays visual information within the user's field of vision, such as a head-up display, a transmissive display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 31 may also use display devices such as a navigation device, instrument panel, CMS (Camera Monitoring System), electronic mirror, lamp, etc., provided in the vehicle 1 as output devices that output visual information.

The output device through which the HMI 31 outputs auditory information can be, for example, an audio speaker, headphones, or earphones.

Haptic elements using haptic technology can be used as an output device for the HMI 31 to output tactile information. Haptic elements are provided on parts that the user touches, such as the steering wheel and the seat.

The vehicle control unit 32 controls each part of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body control unit 84, a light control unit 85, and a horn control unit 86.

The steering control unit 81 detects and controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

The brake control unit 82 detects and controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, and an actuator that drives the brake system.

The drive control unit 83 detects and controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

The body system control unit 84 detects and controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system, an actuator that drives the body system, etc.

The light control unit 85 detects and controls the state of various lights of the vehicle 1. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projection, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

The horn control unit 86 detects and controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, etc.

2 is a diagram showing an example of a sensing area by the camera 51, radar 52, LiDAR 53, ultrasonic sensor 54, etc. of the external recognition sensor 25 in FIG. 1. Note that FIG. 2 shows a schematic view of the vehicle 1 as seen from above, with the left end side being the front end of the vehicle 1 and the right end side being the rear end of the vehicle 1.

Sensing area 101F and sensing area 101B show examples of sensing areas of ultrasonic sensors 54. Sensing area 101F covers the periphery of the front end of vehicle 1 with multiple ultrasonic sensors 54. Sensing area 101B covers the periphery of the rear end of vehicle 1 with multiple ultrasonic sensors 54.

The sensing results in sensing area 101F and sensing area 101B are used, for example, for parking assistance for vehicle 1.

Sensing area 102F to sensing area 102B show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther in the rear of the vehicle 1 than sensing area 101B. Sensing area 102L covers the rear periphery of the left side of the vehicle 1. Sensing area 102R covers the rear periphery of the right side of the vehicle 1.

The sensing results in sensing area 102F are used, for example, to detect vehicles, pedestrians, etc., that are in front of vehicle 1. The sensing results in sensing area 102B are used, for example, for collision prevention functions behind vehicle 1. The sensing results in sensing area 102L and sensing area 102R are used, for example, to detect objects in blind spots to the sides of vehicle 1.

Sensing area 103F to sensing area 103B show examples of sensing areas by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther in the rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 1. Sensing area 103R covers the periphery of the right side of vehicle 1.

The sensing results in sensing area 103F can be used, for example, for recognizing traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing area 103B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing area 103L and sensing area 103R can be used, for example, for surround view systems.

Sensing area 104 shows an example of the sensing area of LiDAR 53. Sensing area 104 covers a position farther in front of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

The sensing results in the sensing area 104 are used, for example, to detect objects such as surrounding vehicles.

Sensing area 105 shows an example of the sensing area of long-range radar 52. Sensing area 105 covers a position farther in front of vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

The sensing results in the sensing area 105 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

The sensing areas of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensors 54 included in the external recognition sensor 25 may have various configurations other than those shown in FIG. 2. Specifically, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. The installation positions of the sensors are not limited to the examples described above. The number of sensors may be one or more.

2. Preferred embodiment
Next, an embodiment of the present technology will be described with reference to FIG. 3 to FIG.

In recent years, cases of drivers with impaired cognitive function due to early-stage dementia and other factors causing traffic accidents have become a social issue, and there is a need to assess the driver's cognitive function to prevent traffic accidents from occurring. This technology makes it possible to assess a driver's cognitive function based on the driver's reaction to events occurring around the driver.

FIG. 3 shows an example of a vehicle control system 11 for determining the driver's cognitive function.

As shown in FIG. 3, the DMS 30 determines the driver's cognitive function by combining sensor data from the camera 51 and LiDAR 53 used to recognize the situation outside the vehicle 1 and sensor data from the in-vehicle sensor 26 used to recognize the situation inside the vehicle 1.

For example, if the camera 51 or LiDAR 53 detects a dangerous event such as another vehicle suddenly emerging in the direction of vehicle 1's travel, DMS 30 uses the in-vehicle sensor 26 to recognize the driver's reaction to the event, and if the reaction is not normal, determines that the driver's cognitive function is abnormal.

If it is determined that the cognitive function is abnormal, the DMS 30 notifies the driver's close relative that the cognitive function of the driver is abnormal, for example, using a device 201 used by the close relative. In the example of Figure 3, the text "There is a problem with cognitive function!" is displayed on the device 201.

FIG. 4 is a block diagram showing an example configuration of a vehicle control system 11 to which this technology is applied.

The vehicle control system 11 in FIG. 4 includes the components described above (communication unit 22, external recognition sensor 25, in-vehicle sensor 26, HMI 31, and recognition unit 73), as well as an information processing unit 301 that determines the driver's cognitive function. Note that FIG. 4 shows the configuration of the portion of the vehicle control system 11 that is related to determining the driver's cognitive function.

The in-vehicle sensor 26 senses the driver's reaction to events occurring around the driver. The in-vehicle sensor 26 is composed of a camera 311, a depth sensor 312, a microphone 313, and a biometric sensor 314.

The camera 311 supplies the captured image, for example of the driver, to the information processing unit 301.

The depth sensor 312 supplies distance measurement data obtained by, for example, measuring the distance to each part of the driver's body to the information processing unit 301.

The microphone 313, for example, collects the driver's voice and supplies the obtained voice data to the information processing unit 301.

The biosensor 314 detects, for example, the driver's biometric information and supplies the obtained biometric data to the information processing unit 301.

The HMI 31 is composed of a guide information generation unit 321, a display control unit 322, and a speaker control unit 323.

The guide information generation unit 321 generates guide information for guiding the driver and supplies the guide information to the information processing unit 301. The guide information indicates, for example, the presence of an object in the traveling direction of the vehicle 1 and the route to the destination. Although not shown in the figure, the guide information generation unit 321 also supplies guide information to the display control unit 322 and the speaker control unit 323.

The display control unit 322 displays the guide information generated by the guide information generation unit 321, for example, on a display provided inside the vehicle.

The speaker control unit 323 outputs a guide voice that expresses the guide information generated by the guide information generation unit 321 in audio form, for example, from a speaker installed inside the vehicle.

As described above, the recognition unit 73 acquires sensor data from the external recognition sensor 25 and performs recognition processing of objects around the vehicle 1 based on the sensor data. The recognition unit 73 also functions as an event detection unit that detects events occurring around the vehicle 1 (outside the vehicle) based on the object recognition results. The recognition unit 73 supplies the detection results of events occurring around the vehicle 1 to the information processing unit 301. The detection results of events occurring around the vehicle 1 include the content of the event, the area in which the event is occurring, the danger level of the event, etc.

The information processing unit 301 is part of the functions of the DMS 30. The information processing unit 301 is composed of a reaction recognition unit 331, a reaction prediction unit 332, and a cognitive function assessment unit 333.

The reaction recognition unit 331 recognizes, for example, the driver's actual reaction to an event that occurs around the driver within a specified period of time based on sensor data from various sensors that make up the in-vehicle sensor 26, and supplies the recognition result of the actual reaction to the cognitive function assessment unit 333. Events that occur around the driver include events that occur around the vehicle 1, events such as various information such as guide information being displayed in the vehicle, events such as various sounds such as guide voices being output in the vehicle, etc.

The reaction recognition unit 331 can also recognize the actual reaction of the driver based on sensor data from the vehicle sensor 27, which is composed of an accelerator sensor, brake sensor, etc.

The response prediction unit 332 predicts the driver's normal response to events occurring around the driver. Specifically, the response prediction unit 332 predicts the driver's normal response to events occurring around the vehicle 1 that are recognized by the recognition unit 73. The response prediction unit 332 also predicts the driver's normal response to the event that guide information is presented to the driver, based on the guide information supplied from the guide information generation unit 321.

The reaction prediction unit 332 can also predict the driver's normal reaction to an event that occurs around the driver based on the driver's attributes. The driver's attributes include, for example, gender, age, and level of driving experience.

The response prediction unit 332 supplies the normal response prediction result to the cognitive function assessment unit 333.

The cognitive function assessment unit 333 compares the actual reaction of the driver recognized by the reaction recognition unit 331 with the predicted reaction, which is the normal reaction of the driver predicted by the reaction prediction unit 332, to assess the cognitive function of the driver. The cognitive function assessment unit 333 controls the communication unit 22, the display control unit 322, and the speaker control unit 323 based on the assessment result of the driver's cognitive function.

FIG. 5 is a block diagram showing a detailed example configuration of the cognitive function assessment unit 333.

As shown in FIG. 5, the cognitive function assessment unit 333 is composed of a comparison unit 351, a reaction rate calculation unit 352, a reaction rate assessment unit 353, a memory unit 354, and an output unit 355.

The comparison unit 351 compares the driver's actual response with the predicted response to determine whether the driver's actual response is normal. If the driver's actual response is normal, the comparison unit 351 determines that the driver's cognitive function is normal, and if the driver's actual response is not normal, the comparison unit 351 determines that the driver's cognitive function is abnormal.

The comparison unit 351 supplies the recognition results of the driver's actual reaction and the prediction results of the driver's normal reaction to the reaction speed calculation unit 352, and supplies the judgment results of the driver's cognitive function to the output unit 355.

The reaction speed calculation unit 352 calculates the actual reaction speed of the driver based on the recognition result of the actual reaction of the driver by the reaction recognition unit 331. The reaction speed indicates, for example, the time from when an event occurs until the driver steps on the brake pedal, or the time from when an event occurs until the driver turns his or her gaze to the area where the event is occurring.

In addition, the reaction speed calculation unit 352 calculates the normal reaction speed of the driver based on the prediction result of the normal reaction of the driver by the reaction prediction unit 332.

The reaction speed calculation unit 352 supplies information indicating the driver's actual reaction speed and information indicating the driver's normal reaction speed to the reaction speed determination unit 353, and supplies information indicating the driver's actual reaction speed to the memory unit 354.

The reaction speed determination unit 353 determines the driver's cognitive function (whether the driver's actual reaction is normal or not) based on the driver's actual reaction speed calculated by the reaction speed calculation unit 352. Specifically, the reaction speed determination unit 353 first compares the driver's actual reaction speed with the normal reaction speed, and determines whether the driver's actual reaction speed is slower than the normal reaction speed.

Next, the reaction speed determination unit 353 obtains information indicating the driver's past reaction speed to the event detected by the recognition unit 73 from the memory unit 354. The reaction speed determination unit 353 compares the driver's actual reaction speed with the past reaction speed, and determines whether the driver's actual reaction speed is slower than the past reaction speed.

If the driver's actual reaction speed is slower than at least one of the normal reaction speed and the past reaction speed, the reaction speed determination unit 353 determines that the driver's cognitive function is abnormal. The reaction speed determination unit 353 supplies the determination result of the driver's cognitive function to the output unit 355.

In the memory unit 354, for example, information indicating the driver's past reaction speed is recorded in association with the event detection results.

The output unit 355 functions as an execution unit that executes a predetermined process when at least one of the comparison unit 351 and the reaction speed determination unit 353 determines that the driver's cognitive function is abnormal. For example, information indicating that the driver's cognitive function is abnormal is transmitted to the device 201 via the communication unit 22, and the information is presented to the driver's close relatives.

The first process performed by the vehicle control system 11 having the above configuration will be described with reference to the flowchart in FIG. 6. In the first process, the driver's cognitive function is determined based on the driver's reaction to the event of the guide voice being presented to the driver.

In step S1, the speaker control unit 323 outputs the guidance voice from the speaker. After the guidance voice is output, the response prediction unit 332 predicts the driver's normal response to the event that the guidance voice has been presented to the driver.

In step S2, the in-vehicle sensor 26 senses the driver's response to the voice guidance. The response recognition unit 331 recognizes the driver's actual response to the voice guidance based on the sensor data of the in-vehicle sensor 26.

In step S3, the cognitive function assessment unit 333 determines whether the driver's actual response to the guidance voice is normal or not.

For example, if the driver steps on the brake pedal within a predetermined period of time after a voice guidance is presented indicating that an object is present in the direction of travel of vehicle 1, the driver's reaction is determined to be normal. Also, if the driver places a hand over their ear or tilts their head within a predetermined period of time after the voice guidance is presented, the driver's reaction is determined to be abnormal.

If it is determined in step S3 that the driver's actual reaction is normal, the process proceeds to step S4, where the cognitive function assessment unit 333 determines that the guidance voice has been properly conveyed to the driver.

On the other hand, if it is determined in step S3 that the driver's actual reaction is not a normal reaction, the process proceeds to step S5, and the cognitive function assessment unit 333 presents the guide information to the driver again. For example, the cognitive function assessment unit 333 may increase the volume of the guide voice and output it again from the speaker, or present the guide information by means other than outputting the guide voice (such as illuminating a lamp or displaying the guide information).

Next, the second process performed by the vehicle control system 11 will be described with reference to the flowchart in FIG. 7. In the second process, the driver's cognitive function is determined based on the driver's reaction to the event that guide information is displayed on the display.

In step S21, the display control unit 322 displays the guide information on the display. After the guide information is displayed, the response prediction unit 332 predicts the driver's normal response to the event that the guide information has been displayed.

In step S22, the in-vehicle sensor 26 senses the driver's reaction to the display of the guide information. The reaction recognition unit 331 recognizes the driver's actual reaction to the display of the guide information based on the sensor data of the in-vehicle sensor 26.

In step S23, the cognitive function assessment unit 333 determines whether the driver's actual response to the display of the guide information is normal or not.

For example, if the driver steps on the brake pedal within a predetermined period of time after guide information indicating that an object is present in the traveling direction of vehicle 1 is displayed, the driver's reaction is determined to be normal. Also, if the driver squints their eyes or moves their face closer to the display within a predetermined period of time after guide information is displayed, the driver's reaction is determined to be abnormal.

If it is determined in step S23 that the driver's actual reaction is normal, the process proceeds to step S24, and the cognitive function assessment unit 333 determines that the guide information has been correctly conveyed to the driver.

On the other hand, if it is determined in step S23 that the driver's actual reaction is not a normal reaction, the process proceeds to step S25, and the cognitive function assessment unit 333 presents the guide information to the driver again. For example, if the guide information is represented by an icon, the cognitive function assessment unit 333 enlarges the icon and displays it again on the display. In addition, the cognitive function assessment unit 333 presents the guide information by means other than displaying the guide information (such as illuminating a lamp or presenting a guide voice).

Next, the third process performed by the vehicle control system 11 will be described with reference to the flowchart in FIG. 8. In the third process, the driver's cognitive function is determined based on the driver's reaction when a dangerous event occurs.

In step S41, the external recognition sensor 25 senses the situation outside the vehicle 1. The recognition unit 73 performs recognition processing of objects around the vehicle 1 based on the sensor data of the external recognition sensor 25, and detects events occurring around the vehicle 1 based on the object recognition results.

In step S42, the recognition unit 73 determines whether or not a dangerous event has been detected, and continues sensing the situation outside the vehicle 1 until a dangerous event is detected.

If it is determined in step S42 that a dangerous event has been detected, the response prediction unit 332 predicts the driver's normal response to the dangerous event. The process then proceeds to step S43, where the in-vehicle sensor 26 senses the driver's response to the dangerous event. The response recognition unit 331 recognizes the driver's actual response to the dangerous event based on the sensor data of the in-vehicle sensor 26.

In step S44, the cognitive function assessment unit 333 determines whether the driver's actual reaction to the dangerous event is normal or not.

Figure 9 shows an example of a driver's abnormal reaction to a dangerous event.

As shown in Figure 9, if, within a specified period of time after the occurrence of a dangerous event, for example, there is no change in facial expression, the driver is slow to turn his/her gaze in the direction of the dangerous event, or the driver turns his/her gaze in that direction but then immediately looks in a different direction, the driver's reaction is determined to be abnormal. Also, if, within a specified period of time after the occurrence of a dangerous event, there is no change in pupil movement or pulse rate, the driver mistakenly believes that others around him/her are at fault and becomes angry, frowns, or does not step on the brake pedal, the driver's reaction is determined to be abnormal.

On the other hand, if, within a specified period of time after the occurrence of a dangerous event, for example, the driver's facial expression changes, the driver quickly turns their gaze in the direction of the dangerous event, or the driver stares in that direction, the driver's reaction is determined to be normal. Also, for example, if, within a specified period of time after the occurrence of a dangerous event, the driver's pupil movement or pulse rate changes, or the driver presses the brake pedal, the driver's reaction is determined to be normal.

In this way, it is determined whether the driver's facial expression, gaze direction, gaze duration, pupil movement, pulse rate, emotional expression, driving operation, etc., as a reaction to an event occurring around the driver, are normal or not. It may be determined whether one type of reaction is normal or not, or it may be determined whether each of multiple types of reactions is normal or not.

Returning to FIG. 8, if it is determined in step S44 that the driver's actual reaction is normal, the process proceeds to step S45, and the cognitive function assessment unit 333 determines that the driver's cognitive function is normal.

On the other hand, if it is determined in step S44 that the driver's actual reaction is not a normal reaction, the process proceeds to step S46, and the cognitive function assessment unit 333 determines that the driver's cognitive function is abnormal.

In step S47, the output unit 355 of the cognitive function assessment unit 333 generates an alert and presents the alert to the driver or a close relative of the driver. For example, an alert indicating that a dangerous event has occurred is presented to the driver, or an alert indicating that the driver's cognitive function is abnormal is presented to a close relative of the driver.

As described above, in this technology, the driver's reaction to an event occurring around the driver is predicted by the reaction prediction unit 332, and the driver's actual reaction to the event is recognized by the reaction recognition unit 331. In addition, the predicted reaction and the actual reaction are compared, and whether or not the actual reaction is normal is determined by the comparison unit 351 and the reaction speed determination unit 353, and if it is determined that the actual reaction is not normal, a predetermined process is executed by the output unit 355.

In the past, for example, if a driver looked in the direction they should have looked but did not recognize any danger, it was difficult to accurately assess the driver's cognitive function. With this technology, the driver's cognitive function is assessed based on the driver's reaction, so even if the driver looked in the direction they should have looked but did not recognize any danger, it is possible to assess the driver's cognitive function based on reactions such as facial expressions.

Note that in Figures 6 to 8, an example has been described in which, when an event occurs around the driver, the driver's cognitive function is immediately determined based on the driver's reaction to the event. When an event occurs around the driver, sensor data from the in-vehicle sensor 26 within a predetermined period of time after the event occurs is recorded, and the driver's cognitive function may be determined at a time other than immediately after the event occurs. For example, when a predetermined time arrives or when the vehicle 1 arrives at the destination, the driver's cognitive function is determined based on the recorded sensor data.

<About computers>
The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program. The information processing unit 301 is, for example, a PC having a configuration similar to that shown in FIG. 12.

CPU 501, ROM 502, and RAM 503 are interconnected via bus 504.

Further connected to the bus 504 is an input/output interface 505. Connected to the input/output interface 505 are an input unit 506 consisting of a keyboard, mouse, etc., and an output unit 507 consisting of a display, speakers, etc. Also connected to the input/output interface 505 are a storage unit 508 consisting of a hard disk or non-volatile memory, a communication unit 509 consisting of a network interface, etc., and a drive 510 that drives removable media 511.

In a computer configured as described above, the CPU 501, for example, loads a program stored in the storage unit 508 into the RAM 503 via the input/output interface 505 and the bus 504 and executes the program, thereby performing the above-mentioned series of processes.

The programs executed by the CPU 501 are provided, for example, by being recorded on removable media 511, or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and are installed in the storage unit 508.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

The effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiment of this technology is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of this technology.

For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

In addition, each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.

Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.

Example of combination of configurations The present technology can also have the following configurations.

(1)
A reaction prediction unit that predicts a reaction of the driver to an event occurring around the driver;
a reaction recognition unit that recognizes an actual reaction of the driver to the event;
A determination unit that compares a predicted reaction, which is a reaction predicted by the reaction prediction unit, with the actual reaction and determines whether the actual reaction is a normal reaction or not;
and an execution unit that executes a predetermined process when the actual reaction is determined to be an abnormal reaction.
(2)
The information processing device according to (1), wherein the reaction recognition unit recognizes the actual reaction within a predetermined period after the occurrence of the event.
(3)
The information processing device described in (1) or (2), wherein the judgment unit compares the predicted reaction with the actual reaction to judge whether the actual reaction is a normal reaction or not, and judges whether the actual reaction is a normal reaction or not based on a speed of the actual reaction.
(4)
The information processing device according to (3), wherein the determination unit compares a speed of the predicted reaction with a speed of the actual reaction to determine whether the actual reaction is a normal reaction.
(5)
The information processing device according to (3) or (4), wherein the determination unit compares a past reaction speed of the driver with a speed of the actual reaction to determine whether the actual reaction is a normal reaction or not.
(6)
The information processing device described in any of (1) to (5), wherein the execution unit, when it is determined that the actual response to the event that specified information is displayed inside the vehicle driven by the driver is not a normal response, presents the information to the driver by means other than displaying the information.
(7)
The information processing device described in any of (1) to (6), wherein the execution unit, when it is determined that the actual response to the event that a voice expressing predetermined information has been output in the vehicle driven by the driver is not a normal response, presents the information to the driver by means other than outputting the voice.
(8)
The information processing device according to any one of (1) to (7), wherein the execution unit transmits, when it is determined that the actual reaction is not a normal reaction, information indicating that the cognitive function of the driver is abnormal to a device external to the vehicle driven by the driver.
(9)
The information processing device according to any one of (1) to (8), wherein the reaction recognition unit recognizes the actual reaction based on sensor data of a first sensor used to recognize an internal situation of a vehicle driven by the driver.
(10)
The information processing device according to any one of (1) to (9), wherein the driver's reaction includes at least one of the driver's facial expression, gaze direction, gaze duration, pupil movement, pulse rate, emotional expression, and driving operation.
(11)
The information processing device described in any of (1) to (10), further comprising an event detection unit that detects the event occurring around the vehicle based on sensor data of a second sensor used to recognize an external situation of the vehicle driven by the driver, wherein the response prediction unit predicts a response of the driver to the event when the event is detected by the event detection unit.
(12)
The information processing device according to any one of (1) to (11), wherein the reaction prediction unit predicts a reaction of the driver to the event based on an attribute of the driver.
(13)
The information processing device according to (12), wherein the driver's attributes include at least one of gender, age, and degree of driving experience.
(14)
An information processing device,
predicting a driver's reaction to events occurring around the driver;
Recognizing an actual response of the driver to the event;
comparing the predicted response to the actual response to determine whether the actual response is a normal response;
and executing a predetermined process when the actual reaction is determined to be not a normal reaction.
(15)
A first sensor used to recognize a situation inside a vehicle driven by a driver;
A reaction prediction unit that predicts a reaction of the driver to an event occurring around the driver;
A reaction recognition unit that recognizes an actual reaction of the driver to the event based on sensor data of the first sensor;
A determination unit that compares the reaction predicted by the reaction prediction unit with the actual reaction and determines whether the actual reaction is a normal reaction;
and an information processing device including: an execution unit that executes a predetermined process when the actual reaction is determined to be an abnormal reaction.
(16)
A second sensor is further provided for recognizing a situation outside the vehicle,
The information processing device further includes an event detection unit that detects the event occurring around the vehicle based on sensor data of the second sensor,
The information processing system according to (15), wherein the reaction prediction unit of the information processing device predicts a reaction of the driver to the event when the event is detected by the event detection unit.

1 Vehicle, 11 Vehicle control system, 22 Communication unit, 25 External recognition sensor, 26 In-vehicle sensor, 30 DMS, 31 HMI, 51 Camera, 53 LiDAR, 73 Recognition unit, 201 Device, 301 Information processing unit, 311 Camera, 312 Depth sensor, 313 Microphone, 314 Biosensor, 321 Guide information generation unit, 322 Display control unit, 323 Speaker control unit, 331 Response recognition unit, 332 Response prediction unit, 333 Cognitive function judgment unit, 351 Comparison unit, 352 Response speed calculation unit, 353 Response speed judgment unit, 354 Memory unit, 355 Output unit

Claims (16)

A reaction prediction unit that predicts a reaction of the driver to an event occurring around the driver;
a reaction recognition unit that recognizes an actual reaction of the driver to the event;
A determination unit that compares a predicted reaction, which is a reaction predicted by the reaction prediction unit, with the actual reaction and determines whether the actual reaction is a normal reaction or not;
and an execution unit that executes a predetermined process when the actual reaction is determined to be an abnormal reaction. The information processing device according to claim 1 , wherein the reaction recognition unit recognizes the actual reaction within a predetermined period after the occurrence of the event. The information processing device according to claim 1 , wherein the determination unit compares the predicted response with the actual response to determine whether the actual response is a normal response, and determines whether the actual response is a normal response based on a speed of the actual response. The information processing device according to claim 3 , wherein the determination unit compares a speed of the predicted reaction with a speed of the actual reaction to determine whether or not the actual reaction is a normal reaction. The information processing device according to claim 3 , wherein the determination unit compares a past reaction speed of the driver with the actual reaction speed to determine whether the actual reaction is a normal reaction or not. 2. The information processing device according to claim 1, wherein when it is determined that the actual response to the event that predetermined information is displayed in the vehicle driven by the driver is not a normal response, the execution unit presents the information to the driver by a means other than displaying the information. 2. The information processing device according to claim 1, wherein when the execution unit determines that the actual response to the event that a voice expressing predetermined information has been output in the vehicle driven by the driver is not a normal response, the execution unit presents the information to the driver by a means other than outputting the voice. The information processing device according to claim 1 , wherein the execution unit transmits, when the actual reaction is determined to be an abnormal reaction, information indicating that the cognitive function of the driver is abnormal to a device external to the vehicle driven by the driver. The information processing device according to claim 1 , wherein the reaction recognition unit recognizes the actual reaction based on sensor data of a first sensor used to recognize an internal situation of a vehicle driven by the driver. The information processing device according to claim 1 , wherein the driver's reaction includes at least one of the driver's facial expression, gaze direction, gaze duration, pupil movement, pulse rate, emotional expression, and driving operation. 2. The information processing device according to claim 1, further comprising an event detection unit that detects the event occurring around the vehicle based on sensor data of a second sensor used to recognize a situation outside the vehicle driven by the driver, and the response prediction unit predicts a response of the driver to the event when the event is detected by the event detection unit. The information processing device according to claim 1 , wherein the reaction prediction unit predicts the reaction of the driver to the event based on an attribute of the driver. The information processing device according to claim 12 , wherein the driver attributes include at least one of gender, age, and degree of driving experience. An information processing device,
predicting a driver's reaction to events occurring around the driver;
Recognizing an actual response of the driver to the event;
comparing the predicted response to the actual response to determine whether the actual response is a normal response;
and executing a predetermined process when the actual reaction is determined to be not a normal reaction. A first sensor used to recognize a situation inside a vehicle driven by a driver;
A reaction prediction unit that predicts a reaction of the driver to an event occurring around the driver;
A reaction recognition unit that recognizes an actual reaction of the driver to the event based on sensor data of the first sensor;
A determination unit that compares the reaction predicted by the reaction prediction unit with the actual reaction and determines whether the actual reaction is a normal reaction;
and an information processing device including: an execution unit that executes a predetermined process when the actual reaction is determined to be an abnormal reaction. A second sensor is further provided for recognizing a situation outside the vehicle,
The information processing device further includes an event detection unit that detects the event occurring around the vehicle based on sensor data of the second sensor,
The information processing system according to claim 15 , wherein the reaction prediction unit of the information processing device predicts a reaction of the driver to the event when the event is detected by the event detection unit.

Images