JP7298510B2 - state estimator - Google Patents

Description

The technology disclosed herein relates to a state estimation device that estimates the state of a subject.

Japanese Unexamined Patent Application Publication No. 2002-200000 discloses a vehicle control device that detects an abnormality of a vehicle driver. This vehicle control device includes a body temperature measurement unit that measures the body temperature of the driver of the vehicle, a running state determination unit that determines whether the running state of the vehicle is abnormal, a measurement result of the body temperature measurement unit and the running state determination. and a vehicle control unit that controls the vehicle based on the determination result of the unit.

JP 2019-73105 A

However, the device of Patent Document 1 cannot estimate an abnormal state that does not involve heat generation. Therefore, it is not possible to estimate a visual field abnormality such as homonymous hemianopia, in which one side of the visual field is abnormal.

The technology disclosed herein has been made in view of this point, and its purpose is to estimate a visual field abnormality state in which one of the left and right sides of the visual field is abnormal.

The technology disclosed herein relates to a state estimation device that estimates the state of a subject. This state estimation device includes a saccade detection unit that detects a saccade of the subject, and a difference between the amplitude of the saccade in the subject's right visual field and the amplitude of the saccade in the left visual field within a predetermined estimation period. A first condition of being equal to or greater than a predetermined amplitude difference threshold, and a frequency difference threshold in which the difference between the frequency of saccades in the right visual field and the frequency of saccades in the left visual field of the subject within the estimation period is predetermined. and an estimating unit for estimating that the state of the subject is a visual field abnormal state in which one of the left and right sides of the visual field is abnormal when both of the above second conditions are satisfied.

As a result of intensive research, the inventor of the present application found that when the subject's condition becomes a visual field abnormality, the difference between the amplitude of the saccades in the right visual field and the amplitude of the saccades in the left visual field of the subject within the estimation period is greater than that in the normal state. Also, the difference between the frequency of saccades in the subject's right visual field and the frequency of saccades in the left visual field during the estimation period is greater than in normal times. Therefore, based on the establishment of both the first condition and the second condition, it is possible to estimate a visual field abnormal state in which one of the left and right sides of the visual field is abnormal.

In the state estimating device, the estimating unit determines that at least one of the first condition and the second condition is not satisfied and the amplitude of the saccade within the entire visual field of the subject within the estimation period is a predetermined amplitude When at least one of a third condition that the frequency of saccades in the entire field of view of the subject within the estimation period is less than or equal to a threshold and a fourth condition that the frequency is less than or equal to a predetermined frequency threshold, the subject It may be configured to estimate that the state of the person is in a state of reduced attention function.

As a result of intensive research, the inventor of the present application found that when the subject is in a state of reduced attention, at least one of the amplitude and frequency of saccades in the entire visual field of the subject within the estimation period becomes lower than in normal times. rice field. Therefore, based on whether the first and second conditions are satisfied and whether the third and fourth conditions are satisfied, it is possible to distinguish between a sudden abnormal state and a state of reduced attention function.

When the state of the subject is estimated to be the abnormal visual field state, the state estimating device performs a first action according to the abnormal visual field state, and the state of the subject is the reduced attention function state. A control unit may be provided that, when it is estimated that there is, performs a second action in accordance with the state of impaired attention function.

With the above configuration, it is possible to appropriately perform an action according to the state of the subject estimated by the estimation unit.

In the state estimation device, the subject may be a driver of a vehicle. The first action may include an action of controlling travel of the vehicle so that the vehicle retreats to a safe area. The second action may include an action of outputting information prompting the driver to take a rest.

With the above configuration, the vehicle can be stopped in the safe area when it is estimated that the driver's condition is an abnormal visual field condition. As a result, the safety of the own vehicle (the vehicle driven by the driver) and other vehicles around the own vehicle can be ensured. Further, when the driver's state is estimated to be in a state of reduced attention function, it is possible to prompt the driver to take a rest. As a result, the condition of the driver can be recovered from the attention function deterioration condition to the normal condition, and the running safety of the vehicle can be ensured.

In the state estimation device, the subject may be a driver of a vehicle. The amplitude difference threshold and the frequency difference threshold may be set based on the amplitude and frequency of the saccade when the driving scene of the vehicle is a predetermined driving scene.

With the above configuration, the amplitude difference threshold and the frequency difference threshold can be appropriately set in consideration of the characteristics of the driver. As a result, it is possible to improve the accuracy of estimating the state of the driver.

According to the technology disclosed herein, it is possible to estimate a visual field abnormality state in which one of the left and right sides of the visual field is abnormal.

1 is a block diagram illustrating the configuration of a vehicle control system according to Embodiment 1; FIG. It is a graph for explaining a saccade. FIG. 4 is a schematic diagram for explaining a field of view; 4 is a block diagram illustrating the configuration of a state estimation unit; FIG. 7 is a graph for explaining saccade detection processing; 9 is a flowchart for explaining saccade detection processing; 7 is a graph for explaining extraction of saccades; 1 is a perspective view illustrating a driving simulator; FIG. Fig. 10 is a graph showing the amplitude of saccades in the right and left visual fields of healthy subjects and homonymous hemianopia patients, respectively. Fig. 3 is a graph showing the difference in amplitude of saccades between the right and left visual fields of healthy subjects and homonymous hemianopia patients, respectively. 1 is a graph showing the frequency of right and left visual field saccades in healthy subjects and homonymous hemianopia patients, respectively. 2 is a graph showing the difference in frequency of saccades between right and left visual fields in healthy subjects and homonymous hemianopia patients, respectively. 5 is a flowchart for explaining state estimation processing; FIG. 11 is a flowchart for explaining a modification of state estimation processing; FIG. 4 is a flowchart for explaining an example of a first operation; 9 is a flowchart for explaining an example of a second operation; 4 is a flowchart for explaining threshold setting processing; FIG. 10 is a front view illustrating the appearance of the display device of Embodiment 2; FIG. 8 is a block diagram illustrating the configuration of a display device according to Embodiment 2;

Hereinafter, embodiments will be described in detail with reference to the drawings. The same reference numerals are given to the same or corresponding parts in the drawings, and the description thereof will not be repeated.

(Embodiment 1)
FIG. 1 illustrates the configuration of a vehicle control system 10 according to the first embodiment. A vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled motor vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels according to the driver's operation (for example, accelerator operation). Assisted driving is driving in which the vehicle travels while assisting the driver's operation. Autonomous driving is driving that runs without driver's operation. The vehicle control system 10 controls the vehicle during assist driving and automatic driving. Specifically, the vehicle control system 10 controls the operation of the vehicle (especially running) by controlling an actuator 11 provided in the vehicle. The vehicle control system 10 is an example of a mobile body control system provided in a mobile body.

In this example, the vehicle control system 10 includes an information acquisition section 20 , a vehicle control device 30 and a notification section 40 . In the following description, the vehicle provided with the vehicle control system 10 is referred to as "own vehicle", and other vehicles existing around the own vehicle are referred to as "other vehicles".

[Actuator]
The actuator 11 includes a drive system actuator, a steering system actuator, a braking system actuator, and the like. Examples of drive system actuators include engines, transmissions, and motors. An example of a braking system actuator is a brake. Steering is an example of a steering system actuator.

[Information Acquisition Unit]
The information acquisition unit 20 acquires various types of information used for vehicle control. In this example, the information acquisition unit 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, a communication unit 24, a vehicle state sensor 25, a driving operation sensor 26, and a driver state sensor 27. including.

<camera>
A plurality of cameras 21 have the same configuration as each other. The plurality of cameras 21 are provided on the vehicle so that the imaging areas of the plurality of cameras 21 surround the vehicle. The plurality of cameras 21 acquires image data representing the external environment by capturing images of the environment (external environment) surrounding the vehicle. Image data obtained by each of the cameras 21 is transmitted to the vehicle control device 30 .

In this example, camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary metal-oxide-semiconductor). Note that the camera 21 may be a monocular camera having a narrow-angle lens, or a stereo camera having a wide-angle lens or a narrow-angle lens.

<Radar>
A plurality of radars 22 have the same configuration as each other. A plurality of radars 22 are provided in the vehicle so that search areas of the plurality of radars 22 surround the vehicle. A plurality of radars 22 detect the external environment. Specifically, the radar 22 detects the external environment by transmitting a search wave toward the external environment of the vehicle and receiving a reflected wave from the external environment. Detection results of the multiple radars 22 are transmitted to the vehicle control device 30 .

For example, the radar 22 may be a millimeter wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared rays. Alternatively, it may be an ultrasonic sensor that transmits ultrasonic waves.

<Position sensor>
The position sensor 23 detects the position (eg latitude and longitude) of the vehicle. For example, the position sensor 23 receives GPS information from the global positioning system and detects the position of the vehicle based on the GPS information. Information (vehicle position) obtained by the position sensor 23 is transmitted to the vehicle control device 30 .

<Communication part>
The communication unit 24 receives information through an external network (such as the Internet) provided outside the vehicle. For example, the communication unit 24 receives communication information from other vehicles (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, and high-precision map information such as a dynamic map. do. Information obtained by the communication unit 24 is transmitted to the vehicle control device 30 .

<Vehicle status sensor>
A vehicle state sensor 25 detects the state of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle state sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. Information (state of the vehicle) obtained by the vehicle state sensor 25 is transmitted to the vehicle control device 30 .

<Driving operation sensor>
The driving operation sensor 26 detects driving operations applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the operation amount of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil pressure sensor detects the operation amount of the brake of the vehicle. Information (driving operation of the vehicle) obtained by the driving operation sensor 26 is transmitted to the vehicle control device 30 .

<Driver status sensor>
The driver state sensor 27 detects the state of the driver riding in the vehicle (for example, the driver's physical behavior, biological information, etc.). Information (state of the driver) obtained by the driver state sensor 27 is transmitted to the vehicle control device 30 . In this example, the driver state sensor 27 includes an in-vehicle camera 28 and a biological information sensor 29 .

《In-vehicle camera》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver by capturing an area including the driver. Image data obtained by the in-vehicle camera 28 is transmitted to the vehicle control device 30 . In this example, the in-vehicle camera 28 is arranged in front of the driver, and the imaging range is set so that the driver's face (especially eyeballs) is positioned within the imaging range. Note that the in-vehicle camera 28 may be provided in goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 is provided inside the vehicle. The biological information sensor 29 detects biological information of the driver. Examples of the driver's biological information include perspiration, heartbeat, blood flow, skin temperature, and the like. Information (driver's biological information) obtained by the biological information sensor 29 is transmitted to the vehicle control device 30 . For example, the biometric information sensor 29 may be arranged at a location that comes into contact with the driver's hand, or may be provided on a member (not shown) attached to the driver's body.

[Vehicle control device]
The vehicle control device 30 is connected to the actuator 11 and each part of the vehicle control system 10 (in this example, the information acquisition part 20, the notification part 40, etc.) so that signal transmission is possible. The vehicle control device 30 controls the actuator 11 and each part of the vehicle control system 10 based on the information obtained by each part of the vehicle control system 10 . Specifically, in assisted driving or automatic driving, the vehicle control device 30 determines a target route on which the vehicle should travel based on various information acquired by the information acquisition unit 20, and travels along the target route. Determine the target motion, which is the motion of the vehicle required to Then, the vehicle control device 30 controls the operation of the actuator 11 so that the motion of the vehicle becomes the target motion. Note that the vehicle control device 30 is an example of a state estimation device.

For example, the vehicle control device 30 is composed of one or more electronic control units (ECUs). The electronic control unit may be composed of a single IC (Integrated Circuit), or may be composed of a plurality of ICs. Also, within an IC, a single core or die may be provided, or multiple cores or dies may be provided in conjunction. The core or die may be composed of, for example, a CPU (processor) and a memory that stores information such as programs for operating the CPU and results of processing by the CPU.

In this example, the vehicle control device 30 has a vehicle behavior recognition section 31 , a driving operation recognition section 32 , an external environment recognition section 33 , a driver state recognition section 34 and a vehicle control section 35 . The vehicle control unit 35 is an example of a control unit that operates according to the state of the driver (subject).

<Vehicle behavior recognition unit>
The vehicle behavior recognition unit 31 estimates vehicle behavior (for example, speed, acceleration, yaw rate, etc.) based on the output of the vehicle state sensor 25 . For example, the vehicle behavior recognition unit 31 uses a learning model generated by deep learning to generate data indicating vehicle behavior from the output of the vehicle state sensor 25 .

<Driving operation recognition part>
The driving operation recognition unit 32 recognizes the driving operation applied to the vehicle based on the output of the driving operation sensor 26 . For example, the driving operation recognition unit 32 uses a learning model generated by deep learning to generate data representing the driving operation applied to the vehicle from the output of the driving operation sensor 26 .

<External environment recognition part>
The external environment recognition unit 33 recognizes the vehicle behavior based on the outputs of the plurality of cameras 21, the outputs of the plurality of radars 22, the output of the position sensor 23, the output of the communication unit 24, and the output of the vehicle behavior recognition unit 31. Recognize the external environment.

For example, the external environment recognition unit 33 uses a learning model generated by deep learning to generate data representing the external environment of the vehicle from the above outputs. In deep learning, a multilayer neural network (Deep Neural Network) is used. Examples of multi-layer neural networks include CNNs (Convolutional Neural Networks).

Specifically, the external environment recognition unit 33 performs image processing on the image data obtained by the plurality of cameras 21 to generate road map data (for example, three-dimensional map data) representing roads on which the vehicle can travel. to generate Also, the external environment recognition unit 33 acquires object information, which is information about objects existing around the vehicle, based on the detection results of the plurality of radars 22 . The object information includes the position coordinates of the object, the speed of the object, and the like. Note that the external environment recognition unit 33 may acquire object information based on image data obtained by a plurality of cameras 21 . Then, the external environment recognition unit 33 integrates the road map data and the object information to generate integrated map data (three-dimensional map data) representing the external environment.

Road map data includes information on road shape, road structure, road gradient, lane markings, road markings, and the like. The object information includes static object information and dynamic object information. Static object information is information about a stationary object that does not change over time. The static object information includes information about the shape of the stationary body, the positional coordinates of the stationary body, and the like. Examples of stationary objects include road signs and structures. Examples of structures include traffic lights, medians, center poles, buildings, signboards, railroad crossings, tunnels, railroad tracks, bus stops, and the like. Dynamic object information is information about a moving object that may change over time. The dynamic object information includes information about the shape of the moving object, the position coordinates of the moving object, the speed of the moving object, and the like. Examples of moving objects include other vehicles and pedestrians.

The high-precision map information received by the communication unit 24 may include road map data and object information. In this case, the external environment recognition unit 33 generates integrated map data based on the road map data and object information included in the high-precision map information, and outputs the information acquisition unit 20 such as the plurality of cameras 21 and the plurality of radars 22. may be configured to appropriately correct the integrated map data based on. For example, when an object recognized based on the outputs of the plurality of cameras 21 and the outputs of the plurality of radars 22 is not included in the integrated map data, the external environment recognition unit 33 stores the object information about the object in the integrated map data. may be added to Further, when an object included in the integrated map data is not recognized based on the outputs of the plurality of cameras 21 and the outputs of the plurality of radars 22, the external environment recognition unit 33 deletes the object information related to that object from the integrated map data. may

<Driver state recognition unit>
The driver state recognition unit 34 recognizes the state of the driver (for example, the driver's state of health, emotions, posture, etc.) based on the output of the driver state sensor 27 . For example, the driver state recognition unit 34 uses a learning model generated by deep learning to generate data indicating the state of the driver from the output of the driver state sensor 27 . In this example, the driver state recognizing section 34 has a state estimating section 300 . State estimator 300 will be described later in detail.

<Vehicle control unit>
The vehicle control unit 35 controls the actuator 11 based on the output of the vehicle behavior recognition unit 31, the output of the driving operation recognition unit 32, the output of the external environment recognition unit 33, and the output of the driver state recognition unit 34. . In this example, the vehicle control unit 35 performs travel control and notification control.

《Driving control》
Travel control is performed in assist driving and automatic driving. In travel control, the vehicle control unit 35 controls travel of the vehicle. In this example, the vehicle control unit 35 performs candidate route generation processing, target route determination processing, and motion control processing in the travel control.

In the candidate route generation process, the vehicle control unit 35 generates one or more candidate routes based on the output of the external environment recognition unit 33 . A candidate route is a route on which the vehicle can travel, and is a target route candidate. In this example, the candidate routes generated by the candidate route generation process include safe routes. A safe route is a driving route that leads to a safe area (eg, a road shoulder).

For example, in the candidate route generation process, the vehicle control unit 35 generates travel map data including roads in front of the vehicle in the traveling direction and objects existing on the roads, based on the output (integrated map data) of the external environment recognition unit 33. (two-dimensional map data). Then, the vehicle control unit 35 generates candidate routes using the state lattice method. Specifically, the vehicle control unit 35 sets a grid area made up of a large number of grid points on the road of the travel map data, and sequentially connects the plurality of grid points in the traveling direction of the vehicle, thereby forming a plurality of travels. Set route. Also, the vehicle control unit 35 gives a route cost to each of the plurality of travel routes. For example, as the safety of a vehicle on a certain travel route increases, the route cost assigned to that travel route decreases. Then, the vehicle control unit 35 selects one or a plurality of travel routes as candidate routes from among the plurality of travel routes based on the route costs assigned to each of the plurality of travel routes.

In the target route determination process, the vehicle control unit 35 selects one or more candidate routes generated in the candidate route generation process based on the output of the driving operation recognition unit 32 and the output of the driver state recognition unit 34. Select a candidate route to be the target route from . For example, the vehicle control unit 35 selects a candidate route that the driver feels most comfortable from among the plurality of candidate routes.

In the motion control process, the vehicle control unit 35 determines the target motion based on the candidate route selected as the target route in the target route determination process, and controls the actuator 11 based on the determined target motion. For example, the vehicle control unit 35 derives a target driving force, a target braking force, and a target steering amount, which are the driving force, the braking force, and the steering amount for achieving the target motion. Then, the vehicle control unit 35 sets the drive command value indicating the target drive force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuator of the drive system, the actuator of the braking system, and the steering system. and to the actuators, respectively.

《Notification control》
In notification control, the vehicle control unit 35 outputs various information for notifying the driver. In this example, the vehicle control unit 35 outputs various information to the notification unit 40 to notify the driver.

[Notification part]
The notification unit 40 is provided inside the vehicle. Then, the notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification section 40 includes a display section 41 and a speaker 42 . The display unit 41 outputs various information as an image. The speaker 42 outputs various information by voice.

[Explanation of terms]
Next, the terms used below will be explained. In the following description, the terms saccade, full field of view, right field of view, left field of view, abnormal visual field condition, impaired attention function condition and normal condition are used.

<Saccade>
A saccade is a jumping eye movement in which a person (eg, a driver) intentionally shifts the line of sight. Specifically, a saccade is an eyeball movement that moves the line of sight from a point of gaze at which the line of sight is stagnant for a predetermined time to the next point of gaze. FIG. 2 shows temporal changes in the position (angle) of a person's line of sight. As shown in FIG. 2, a period sandwiched between two adjacent gaze periods is a saccade period. Note that the gaze period is a period during which the line of sight is considered to be stagnant. The saccade amplitude ds is the moving distance of the line of sight during the saccade period.

For example, the movement speed of the line of sight is calculated based on the change in the movement distance of the line of sight, and the state in which the movement speed of the line of sight is less than a predetermined speed threshold (for example, 40 deg / s) is a predetermined stagnation time (for example, 0 0.1 second) may be extracted as a "gazing period". Then, in the line-of-sight movement in the period sandwiched between two adjacent gaze periods, the movement speed is equal to or greater than a speed threshold (eg, 40 deg/s) and the movement distance is a predetermined distance threshold (eg, 3 deg). The line-of-sight movement described above may be extracted as a “saccade”.

<Full field of view, right field of view, and left field of view>
The full field of view refers to the entire area of the field of view in the horizontal direction. As shown in FIG. 3, the right visual field VR is a visual field area on the right side of the left and right center lines in the entire visual field. The left visual field VL is a visual field area on the left side of the left and right center lines in the entire visual field. The left and right center lines of the visual field are imaginary lines that extend from the center point of the driver's eyes (for example, the position of the nose) toward the front of the face, and can be derived based on the orientation of the driver's face. be.

<Visual field abnormal state>
The visual field abnormal state is an abnormal state in which one of the left and right sides of the visual field becomes abnormal. In the abnormal vision state, it becomes difficult for the driver to continue driving the vehicle. In addition, it is difficult to restore the abnormal visual field state to a normal state by taking a break or the like. Examples of visual field abnormal conditions include homonymous hemianopia.

<Caution function deterioration state>
The reduced attentional function state is a state in which a person's attentional function is reduced due to a factor different from the abnormal visual field state described above. In the attention function degraded state, the driver's operation of the vehicle is hindered. In addition, it is possible to restore the state of impaired attention function to a normal state by taking a break or the like. Examples of states of reduced attention function include decreased alertness, fatigue, and listlessness.

<Normal state>
The normal state is neither the abnormal visual field state nor the reduced attention function state, and is a state in which a person can normally perform an operation that should be performed. In the normal state, the driver operates the vehicle normally.

[Configuration of State Estimation Unit]
FIG. 4 illustrates the configuration of state estimating section 300 . State estimation section 300 has saccade detection section 301 , estimation section 302 , attention level detection section 303 , and setting section 304 .

[Saccade detector]
The saccade detection unit 301 detects saccades of the driver (subject) of the vehicle. In this example, the saccade detection unit 301 detects the line of sight of the driver by performing line-of-sight detection processing on image data obtained by the in-vehicle camera 28 . Note that this line-of-sight detection processing may be processing performed using a learning model (learning model for detecting line-of-sight) generated by deep learning, or processing performed using a known line-of-sight detection technique. may be Further, the line of sight of the driver may be the line of sight of the driver's right eye, the line of sight of the driver's left eye, or the line of sight of the driver's right eye and the line of sight of the left eye. It may be a line of sight derived based on. Then, the saccade detection unit 301 detects saccades of the driver based on the movement of the driver's line of sight.

The saccade detection unit 301 also calculates the amplitude ds and the frequency fs of saccades within a predetermined estimation period P1.

[Saccade detection processing]
Next, the saccade processing performed in the saccade detector 301 will be described with reference to FIGS. 5 and 6. FIG. First, as shown in FIG. 5, the saccade detector 301 sets an estimation period P1. Specifically, the saccade detection unit 301 sets the estimation period P1 such that the estimation period P1 is shifted by a predetermined amount of time. In the example of FIG. 5, the length of the estimation period P1 is "30 seconds" and the predetermined time is "5 seconds".

Also, the saccade detector 301 sets the driver's right visual field and left visual field using a well-known visual field setting technique. For example, the saccade detection unit 301 detects the orientation of the driver's face using a well-known face orientation detection technique, and detects a virtual line extending from the center point of the driver's eyes toward the front of the driver's field of view. set to the left and right center lines of the Then, the saccade detection unit 301 sets the full field of view of the driver based on the orientation of the driver's face, and sets the visual field area on the right side of the left and right center lines of the driver's field of view to the right. The visual field is set to the left visual field, and the visual field area on the left side of the left and right center lines of the driver's visual field in the entire visual field of the driver is set as the left visual field. The setting of the entire field of view, the right field of view, and the left field of view may be performed each time the designated period P1 is set, or may be performed at each predetermined cycle.

Then, saccade detection section 301 performs the processing (steps ST11 to ST14) shown in FIG. 6 each time estimation period P1 is set.

<Step ST11>
First, the saccade detection unit 301 detects the line of sight of the driver (subject). In this example, the saccade detection unit 301 detects the line of sight of the driver by performing line-of-sight detection processing on image data obtained by the in-vehicle camera 28 . For example, the saccade detection unit 301 detects the pupils of the driver in the image (image data) obtained by the in-vehicle camera 28, and detects the line of sight of the driver based on the detected pupils. Next, the saccade detector 301 calculates the moving distance of the line of sight of the driver. Then, the saccade detection unit 301 calculates the speed of the driver's line of sight based on the temporal change in the moving distance of the driver's line of sight. For example, the saccade detection unit 301 calculates the speed of the line of sight of the driver by differentiating the moving distance of the line of sight, which changes over time.

<Step ST12>
Next, the saccade detection unit 301 extracts saccade candidates, which are saccade candidates, based on the movement speed of the line of sight. For example, the saccade detection unit 301 defines a period in which the line-of-sight movement speed is less than a predetermined speed threshold value (for example, 40 deg/s) for a predetermined stagnant time period (for example, 0.1 seconds) as a “gazing period. ” (see FIG. 2). Then, the saccade detection unit 301 detects that the movement speed of the line-of-sight movement in the period sandwiched between the two adjacent gaze periods is equal to or greater than the speed threshold value (for example, 40 deg/s) and that the movement distance is predetermined. A line-of-sight movement that is greater than or equal to a distance threshold (for example, 3 degrees) is extracted as a "saccade candidate."

<Step ST13>
Next, the saccade detector 301 reads a saccade range R10 (see FIG. 7) based on the regression curve L10.

For example, the saccade range R10 is derived as follows. First, the saccade detector 301 derives a regression curve L10 based on a plurality of saccade candidates. Specifically, the saccade detection unit 301 derives a regression curve L10 from a plurality of saccade candidates by the method of least squares. Next, the saccade detection unit 301 derives the first reference curve L11 by shifting the regression curve L10 by a predetermined amount in the direction in which the moving speed increases (increasing direction on the vertical axis in FIG. 7), and shifts the regression curve L10 to A second reference curve L12 is derived by shifting a predetermined amount in the direction in which the moving speed decreases (decreasing direction on the vertical axis in FIG. 7). Then, the saccade detection unit 301 defines a saccade range R10 between the first reference curve L11 and the second reference curve L12. The derivation of the saccade range R10 may be performed periodically.

Then, saccade detection section 301 compares each of the plurality of saccade candidates with saccade range R10 to extract a saccade. Specifically, the saccade detection unit 301 extracts, as a saccade, a saccade candidate included in the saccade range R10 among the plurality of saccade candidates. Note that the saccade detection unit 301 does not extract, as a saccade, a saccade candidate that is not included in the saccade range R10 among the plurality of saccade candidates.

<Step ST14>
Next, the saccade detector 301 calculates the amplitude ds and the frequency fs of saccades within the driver's entire visual field within the estimation period P1. In the following, saccades within the driver's full field of view are referred to as "full saccades". Specifically, the saccade detection unit 301 calculates the average value of the amplitudes ds of all saccades included in the estimation period P1 as "amplitude ds of all saccades in the estimation period P1", A value obtained by dividing the number of all saccades by the time of the estimation period P1 is calculated as the "frequency fs of all saccades within the estimation period P1".

Further, the saccade detection unit 301 detects the saccades in the driver's entire visual field within the estimation period P1, the saccades in the driver's right visual field during the estimation period P1, and the saccades in the driver's left visual field during the estimation period P1. classified as saccades. Hereinafter, saccades in the driver's right visual field are referred to as "right saccades," and saccades in the driver's left visual field are referred to as "left saccades." For example, the saccade detection unit 301 classifies the saccade as a right saccade when the line of sight of the driver at the gazing point of the saccade is directed to the right visual field, and classifies the saccade as a right saccade when the driver's line of sight at the gazing point of the saccade is directed to the left visual field. , classifies that saccade as a left saccade.

Then, the saccade detector 301 calculates the amplitude ds and the frequency fs of the right saccade within the estimation period P1. The calculation of the amplitude ds and frequency fs of the right saccade within the estimation period P1 is the same as the calculation of the amplitude ds and frequency fs of all saccades within the estimation period P1. Further, the saccade detector 301 calculates the amplitude ds and the frequency fs of the left saccade within the estimation period P1. The calculation of the amplitude ds and frequency fs of the left saccade within the estimation period P1 is the same as the calculation of the amplitude ds and frequency fs of all saccades within the estimation period P1.

[Estimation part]
The estimation unit 302 estimates the state of the vehicle driver (subject). In this example, the estimation unit 302 estimates the state of the driver based on the driver's saccades detected by the saccade detection unit 301 . The operation of estimation section 302 will be described later in detail.

[Attention detection unit]
The caution level detection unit 303 detects the level of caution in the external environment of the vehicle. The degree of caution is an index that indicates the number of caution points (attention points that the driver of the vehicle should check while driving) in the external environment of the vehicle. As the number of places to be careful in the vehicle's external environment increases, the degree of caution in the vehicle's external environment increases. The detection result from the attention detection unit 303 is supplied to the setting unit 304 . Examples of caution points include a point where a moving object is expected to jump out and a point where there is an object that can be an obstacle to the vehicle.

For example, the caution level detection unit 303 may detect the level of caution in the external environment of the vehicle as follows. First, the attention detection unit 303 receives information (high-precision map information) obtained by the communication unit 24 and detects a high-precision area from the high-precision map information. A high caution area is an area where there are relatively many caution points that the driver of the vehicle should check while driving (for example, an area where the number of caution points is equal to or greater than a predetermined threshold value). Examples of high caution areas include road environments with poor visibility such as T-junctions, and road environments such as intersections where obstacles (for example, other vehicles) may block the vehicle's travel. area, etc. Also, the alertness detection unit 303 inputs information (vehicle position) obtained by the position sensor 23 . Then, when the position of the vehicle is included in the high caution level region, the caution level detection unit 303 detects that the level of caution in the external environment of the vehicle is a high caution level. When not included in the area, it is detected that the degree of caution in the external environment of the vehicle is low.

Alternatively, the caution level detection unit 303 may detect the level of caution in the external environment of the vehicle as follows. First, the caution level detection unit 303 detects caution points from within the external environment of the vehicle recognized by the external environment recognition unit 33 . Then, when there are relatively many caution points included in the vehicle's external environment (for example, when the number of caution points is equal to or greater than a predetermined threshold value), the caution level detection unit 303 detects the degree of caution in the vehicle's external environment. is a high degree of caution, and when the number of caution points included in the vehicle's external environment is relatively small (for example, when the number of caution points is less than a predetermined threshold value), in the vehicle's external environment Detect that attention level is low attention level.

A driving scene in which the degree of caution in the external environment of the vehicle is high is a driving scene in which there are a relatively large number of caution points that have been revealed, and it is required to direct the line of sight to these caution points in a wide range. It can be said. In addition, in a driving scene in which the degree of caution in the external environment of the vehicle is low, there are relatively few caution points that have become obvious, and it is required to frequently move the line of sight to search for potential caution points. It can be said that it is a driving scene that Below, a driving scene with a high level of caution in the vehicle's external environment is referred to as a "danger confirmation scene", and a driving scene with a low level of caution in the vehicle's external environment is referred to as a "danger search scene". Describe.

[Setting part]
The setting unit 304 sets an abnormal condition used for estimation of the state of the driver (subject) in the estimation unit 302 . In this example, the setting unit 304 sets the threshold included in the abnormal condition based on the driver's saccades detected by the saccade detection unit 301 . Note that the operation of the setting unit 304 will be described later in detail.

[Experiments conducted by the inventor of the present application]
The inventors of the present application conducted the following experiments in order to investigate the relationship between the driver's state and the driver's behavior (particularly, movement of the line of sight).

First, patients with symptoms of homonymous hemianopsia (hereinafter referred to as "homonymous hemianopia patients") were selected as subjects in order to collect data on visual field abnormalities. In addition, patients with attentional dysfunction (hereinafter referred to as "attentionally impaired patients") were selected as subjects in order to collect data on states of attentional impairment. In addition, in order to collect data on normal conditions, healthy subjects without the above symptoms (hereinafter simply referred to as "healthy subjects") were selected as subjects.

Then, as shown in FIG. 8, a driving simulator 60 was used to allow the subjects to simulate driving operations of a vehicle. Specifically, the driving simulator 60 allows the test subject to view a moving image (a moving image showing the external environment of the vehicle that can be seen from inside the vehicle) while the vehicle is running, and observes the subject's behavior during the experiment. Behavior during vehicle driving was simulated. In this experiment, a camera was installed in front of a subject viewing moving images while the vehicle was running, and the camera was set so that the subject's eyeballs were included in the imaging range.

Then, by performing line-of-sight detection processing on the image data obtained by the camera, the subject's line of sight was detected. Further, by performing saccade detection processing on the subject's line of sight obtained by the line-of-sight detection processing, the saccade of the subject was detected, and the amplitude and frequency of the saccade within a predetermined period were calculated. These processes are the same as the processes performed in the saccade detector 301 (see FIG. 6).

The above experiments were performed on multiple subjects.

FIG. 9 shows the saccade amplitudes in the right and left visual fields of a normal subject and a homonymous hemianopia patient, respectively, and FIG. shows the difference between FIG. 11 shows the frequency of saccades in the right and left visual fields in healthy subjects and homonymous hemianopia patients, respectively, and FIG. 12 shows the saccade frequencies in right and left visual fields in healthy subjects and homonymous hemianopia patients, respectively. shows the difference between Hereinafter, the difference in amplitude between right and left saccades of a subject is referred to as "saccade amplitude left-right difference", and the difference in frequency between right and left saccades of a subject is referred to as "saccade frequency difference".

As shown in FIG. 10, the saccade amplitude laterality of the homonymous hemianopia patient is greater than the saccade amplitude laterality of the healthy subject. As shown in FIG. 12, the saccade frequency laterality of homonymous hemianopia patients is greater than that of healthy subjects. This is presumed to be a compensatory action to compensate for the part of the right visual field and left visual field that cannot be seen due to an abnormality.

Attention-impaired patients also tended to have less amplitude and/or frequency of saccades in the entire visual field than normal subjects.

[Knowledge obtained by the inventor of the present application]
The inventors of the present application obtained the following findings from the experiments described above.

(1) The saccade amplitude laterality of homonymous hemianopia patients is greater than that of healthy subjects, and the saccade frequency laterality of homonymous hemianopia patients is greater than that of healthy subjects.

(2) At least one of the amplitude and frequency of saccades in the entire visual field is lower in patients with attention deficits than in healthy subjects.

Based on the above findings, the inventors of the present application have found the following.

(1) When the driver's condition is in the visual field abnormal state, the driver's saccade amplitude left-right difference (difference between the amplitude of the right saccade and the amplitude of the left saccade) becomes larger than that in the normal state, and the frequency of the driver's saccades. The left-right difference (the difference between the frequency of right saccades and the frequency of left saccades) becomes larger than normal.

(2) When the driver's state becomes a state of reduced attention, at least one of the amplitude and frequency of saccades in the driver's entire visual field becomes lower than normal.

[State estimation process]
Next, with reference to FIG. 13, the operation (state estimation processing) of estimation section 302 will be described. The estimating section 302 performs the following steps ST21 to ST23 for each predetermined cycle. For example, the estimation unit 302 performs the following processing every time the saccade detection unit 301 calculates the amplitude ds and the frequency fs of saccades within the estimation period P1.

<Step ST21>
Estimating section 302 determines whether saccade amplitude left-right difference dLR, which is the difference between the amplitude of the right saccade within estimation period P1 and the amplitude of the left saccade within estimation period P1, is greater than or equal to a predetermined amplitude difference threshold value dLRth. judge. Further, the estimation unit 302 determines whether the saccade frequency left-right difference fLR, which is the difference between the frequency of the right saccade within the estimation period P1 and the frequency of the left saccade within the estimation period P1, is equal to or greater than a predetermined frequency difference threshold fLRth. determine whether

For example, the amplitude difference threshold dLRt is set to the saccade amplitude left-right difference dLR when the driver's condition can be considered to be a visual field abnormal condition. Specifically, the amplitude difference threshold dLRt may be set to 0.8 deg. The frequency difference threshold value fLRth is set to the saccade frequency left-right difference fLR when the driver's condition can be considered to be the visual field abnormal condition. Specifically, the frequency difference threshold fLRth may be set to 0.21 times/s.

If both the first condition that "the saccade amplitude left-right difference dLR is equal to or greater than the amplitude difference threshold dLRth" and the second condition that "the saccade frequency left-right difference fLR is equal to or greater than the frequency difference threshold fLRth" are established, step ST22. is performed, and if not, the process of step ST23 is performed.

<Step ST22>
When both the first condition and the second condition are satisfied, the estimating unit 302 estimates that the driver's condition is an abnormal visual field condition. In this example, the estimation unit 302 raises a flag indicating that the visual field is in an abnormal state.

<Step ST23>
If at least one of the first condition and the second condition is not satisfied, the estimating unit 302 does not estimate that the driver's condition is the abnormal visual field condition. In this example, the estimation unit 302 does not raise a flag indicating that the visual field is abnormal.

[Effect of Embodiment 1]
As described above, based on the establishment of both the first condition and the second condition, it is possible to estimate a visual field abnormal state in which one of the left and right sides of the visual field is abnormal.

(Modified example of state estimation processing)
Note that the estimation unit 302 may be configured to perform the state estimation process shown in FIG. 14 . In the state estimation process shown in FIG. 14, steps ST31 to ST33 are performed instead of step ST23 shown in FIG.

<Step ST31>
If at least one of the first condition and the second condition is not satisfied in step ST21, the estimating section 302 determines whether or not the amplitude ds of the saccade within the driver's entire visual field is equal to or less than a predetermined amplitude threshold value dth1. do. The estimating unit 302 also determines whether or not the frequency fs of saccades in the driver's entire field of view is equal to or less than a predetermined frequency threshold fth1.

For example, the amplitude threshold dth1 is set to the amplitude ds of saccades within the driver's entire field of view when the driver's condition can be considered to be in a state of impaired attention function. The frequency threshold fth1 is set to the frequency fs of saccades within the driver's entire field of view when the driver's condition can be considered to be in a state of reduced attention function.

The third condition that ``the amplitude ds of saccades in the driver's entire visual field is equal to or less than the amplitude threshold dth1'' and the fourth condition that ``the frequency fs of saccades in the driver's entire visual field is equal to or less than the frequency threshold fth1'' If at least one of them is satisfied, the process proceeds to step ST32; otherwise, the process proceeds to step ST33.

<Step ST32>
If at least one of the first condition and the second condition is not satisfied in step ST21, and if at least one of the third condition and the fourth condition is satisfied in step ST31, the estimating unit 302 determines that the state of the driver is a reduced attention function state. We assume that In this example, the estimating unit 302 raises a flag indicating a state of impaired attention function.

<Step ST33>
If at least one of the first condition and the second condition is not satisfied in step ST21, and if both the third condition and the fourth condition are not satisfied in step ST31, the estimating section 302 determines that the driver's condition is normal. presume.

〔effect〕
As described above, the sudden abnormal state and the reduced attention function state are distinguished and estimated based on whether the first and second conditions are met and whether the third and fourth conditions are met. can be done.

(Operation of vehicle control unit)
When the estimating unit 302 estimates that the driver is in an abnormal visual field state, the vehicle control unit 35 performs a first operation according to the abnormal visual field state. In this example, the vehicle control unit 35 performs the first operation when the flag indicating the abnormal visual field state is set. Examples of the first operation include an operation of controlling the running of the vehicle so that the vehicle stops in a safe area, an operation of outputting first notification information for notifying that the driver is in a visual field abnormal state, For example, an operation for notifying the surroundings of the vehicle that the driver's condition is an abnormal visual field condition is exemplified. Examples of the operation of outputting the first notification information include an operation of outputting the first notification information to the notification unit 40, and an operation of outputting the first notification information to an information terminal (not shown) outside the vehicle via the communication unit 24. etc. An example of the operation for notifying the surroundings of the vehicle that the driver is in the abnormal vision condition is the operation of blinking the hazard lamps (not shown) of the vehicle.

When the estimation unit 302 estimates that the state of the driver is in the state of reduced attention function, the vehicle control unit 35 performs a second operation according to the state of reduced attention function. In this example, the vehicle control unit 35 performs the second operation when the flag indicating the attention function deterioration state is set. Examples of the second operation include an operation for resolving the driver's reduced attention function state, an operation for outputting second notification information for notifying that the driver is in a reduced attention function state, and the like. be done. Examples of actions for resolving the driver's attention function deterioration state include the action of outputting warning information to encourage the driver to take a break, and the action to encourage the driver to concentrate on driving the vehicle. For example, an operation of outputting alert information may be mentioned. The notification unit 40 receives warning information and/or attention information output from the vehicle control unit 35 . As a result, the notification unit 40 outputs the warning information and/or the alert information in the form of an image and/or sound inside the vehicle. Examples of the operation of outputting the second notification information include an operation of outputting the second notification information to the notification unit 40, and an operation of outputting the second notification information to an information terminal (not shown) outside the vehicle via the communication unit 24. etc.

[First action]
Next, an example of the first operation will be described with reference to FIG. 15 . For example, the vehicle control unit 35 performs the processing of steps ST101 to ST103 below when a flag indicating an abnormal visual field state is set.

<Step ST101>
The vehicle control unit 35 performs evacuation travel control. In evacuation control, the vehicle control unit 35 sets a safe area in the traveling map data including the road in front of the vehicle in the direction of travel and objects existing on the road, and determines a traveling route (evacuation route) whose target position is the safe area. route). Then, the vehicle control unit 35 controls the traveling of the vehicle so that the vehicle travels along the evacuation route and stops in the safe area.

<Step ST102>
The vehicle control unit 35 determines whether or not the vehicle has stopped in the safe area. If the vehicle is stopped in the safe area, the process of step ST103 is performed, and if not, the process of step ST101 is continued.

<Step ST103>
When the vehicle stops in the safe area, the vehicle control unit 35 outputs to the communication unit 24 first notification information indicating that the driver's condition is the visual field abnormal condition. The communication unit 24 transmits the first notification information to an information terminal (not shown) of the medical institution. This first notification information may include position information indicating the position of the vehicle stopped in the safe area.

[Second action]
Next, an example of the second operation will be described with reference to FIG. For example, the vehicle control unit 35 performs the processing of steps ST201 to ST203 below when the flag indicating the attention function deterioration state is set.

<Step ST201>
The vehicle control unit 35 outputs alarm information to the notification unit 40 to prompt the driver to stop the vehicle and take a rest. The notification unit 40 outputs warning information in the form of images and/or sounds. Specifically, the display unit 41 displays warning information. The speaker 42 reproduces the warning information by voice. Thereby, the driver performs a driving operation for stopping the vehicle.

<Step ST202>
The vehicle control unit 35 determines whether or not the vehicle has stopped. If the vehicle is stopped, the process of step ST203 is performed, and if not, the process of step ST201 is continued.

<Step ST203>
When the vehicle stops, the vehicle control unit 35 controls the notification unit 40 so that the output of the warning information by the notification unit 40 is stopped.

〔effect〕
As described above, the operation corresponding to the state of the driver estimated by the estimation unit 302 can be performed appropriately. Specifically, the vehicle can be stopped in a safe area when the driver's condition is estimated to be an abnormal visual field condition. As a result, the safety of the own vehicle (the vehicle driven by the driver) and other vehicles around the own vehicle can be ensured. Further, when the driver's state is estimated to be in a state of reduced attention function, it is possible to prompt the driver to take a rest. As a result, the condition of the driver can be recovered from the attention function deterioration condition to the normal condition, and the running safety of the vehicle can be ensured.

(Operation of setting section)
The setting unit 304 performs threshold setting processing. In the threshold setting process, an abnormal condition used for estimating the state of the driver (subject) in the estimating unit 302 is set. In this example, an amplitude difference threshold dLRth and a frequency difference threshold fLRth are set. Specifically, the amplitude difference threshold dLRth and the frequency difference threshold fLRth are set based on the amplitude ds and the frequency fs of saccades when the driving scene of the vehicle is a predetermined driving scene.

[Threshold setting process]
Next, the threshold setting process will be described with reference to FIG. Setting section 304 performs the processing of steps ST301 to ST303 below at predetermined intervals. Note that the setting unit 304 may be configured to perform the following processing in response to an instruction from the driver.

<Step ST301>
The setting unit 304 determines whether or not the driving scene of the vehicle corresponds to a predetermined driving scene. In this example, the predetermined driving scene is a danger exploration scene. The setting unit 304 determines whether or not the driving scene of the vehicle corresponds to the danger search scene based on the degree of caution in the external environment of the vehicle detected by the caution level detection unit 303 . When the driving scene of the vehicle corresponds to a predetermined driving scene, the process of step ST302 is performed.

<Step ST302>
The setting unit 304 collects the amplitude ds and frequency fs of saccades detected by the saccade detection unit 301 . The amplitude ds and frequency fs of saccades collected in step ST302 are preferably the amplitude ds and frequency fs of saccades when the driver can be considered to be in a normal state.

<Step ST303>
Next, the setting unit 304 sets the amplitude difference threshold dLRth based on the saccade amplitude ds. The setting unit 304 also sets the frequency difference threshold fLRth based on the saccade frequency fs. For example, setting section 304 sets amplitude difference threshold dLRth to “50% of the average value of amplitudes ds of saccades collected in step ST302” and frequency difference threshold fLRth to “frequency fs of saccades collected in step ST302”. 75% of the average value of

〔effect〕
As described above, the amplitude difference threshold dLRth and the frequency difference threshold fLRth can be appropriately set in consideration of the characteristics of the driver. As a result, it is possible to improve the accuracy of estimating the state of the driver.

(Embodiment 2)
18 and 19 illustrate the appearance and configuration of the display device 100 of Embodiment 2. FIG. In this example, the display device 100 constitutes a smart phone. Specifically, the display device 100 includes a housing 100 a , a display section 110 , a speaker 111 , a storage section 112 , an information acquisition section 120 and a control device 130 .

[Chassis]
The housing 100a is formed in the shape of a flat rectangular parallelepiped box. Components of the display device 100 are housed in the housing 100a.

[Display, speaker, and memory]
Display unit 110 displays an image. As shown in FIG. 18, the display unit 110 is formed in a rectangular shape and provided on the front surface of the housing 100a. The speaker 111 reproduces sound. The storage unit 112 stores various information and data.

[Information Acquisition Unit]
The information acquisition unit 120 acquires various information used in the display device 100 . In this example, information acquisition unit 120 includes front camera 121, rear camera 122, operation unit 123, communication unit 124, microphone 125, position sensor 126, state sensor 127, and environment sensor 128. .

<Front camera and rear camera>
The front camera 121 is provided on the front surface of the housing 100a, and the rear camera 122 is provided on the rear surface of the housing 100a. The front camera 121 acquires image data indicating the front area of the display device 100 by capturing an image of the area (front area) extending in front of the display device 100 . The rear camera 122 acquires image data indicating the rear area of the display device 100 by capturing an image of the area (rear area) extending behind the display device 100 . Image data obtained by each of front camera 121 and rear camera 122 is transmitted to control device 130 . For example, the front camera 121 and the rear camera 122 are configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or CMOS (Complementary metal-oxide-semiconductor).

<Operation unit>
The operation unit 123 is operated by the user of the display device. The operation unit 123 outputs a signal according to the operation given by the user. With such a configuration, the user can operate the operation unit 123 to input information. The output of the operation unit 123 is transmitted to the control device 130 . In this example, operation unit 123 includes a touch panel operation unit that forms a touch panel together with display unit 110 . Note that the operation unit 123 may include an operation dial, an operation button, etc. in addition to the touch panel operation unit.

<Communication part>
The communication unit 124 receives information and data through a communication network provided outside the display device 100 (for example, the Internet, a mobile phone line, etc.). For example, the communication unit 124 receives audio data, image data, map information, etc. from another information terminal (not shown). Information and data obtained by communication unit 124 are transmitted to control device 130 .

<microphone>
Microphone 125 converts sound into electrical signals. An electrical signal (audio data) obtained by the microphone 125 is transmitted to the control device 130 .

<Position sensor>
Position sensor 126 detects the position (eg, latitude and longitude) of display device 100 . For example, the position sensor 126 receives GPS information from the global positioning system and detects the position of the display device 100 based on the GPS information. Information (the position of the display device 100 ) obtained by the position sensor 126 is transmitted to the control device 130 .

<Status sensor>
The state sensor 127 detects the state of the display device 100 (eg, acceleration, angular velocity, attitude, etc.). For example, state sensor 127 includes an acceleration sensor, a gyro sensor, and the like. Information (the state of the display device 100 ) obtained by the state sensor 127 is transmitted to the control device 130 .

<Environment sensor>
The environment sensor 128 detects information about the environment surrounding the display device 100 (for example, light, magnetism, etc.). For example, environmental sensors 128 include optical sensors, magnetic sensors, proximity sensors, and the like. Information obtained by the environment sensor 128 (information about the surrounding environment of the display device 100 ) is transmitted to the control device 130 .

〔Control device〕
The control device 130 is connected to each part of the display device 100 (in this example, the display part 110, the speaker 111, the storage part 112, and the information acquisition part 120) so that signal transmission is possible. Then, the control device 130 controls each part of the display device 100 based on the information obtained by each part of the display device 100 . Note that the control device 130 is an example of a state estimation device.

For example, the control device 130 is composed of one or more ICs (Integrated Circuits). A single core or die may be provided within an IC, or multiple cores or dies may be provided in conjunction. The core or die may be composed of, for example, a CPU (processor) and a memory that stores information such as programs for operating the CPU and results of processing by the CPU.

In this example, controller 130 includes controller 131 and state estimator 300 . The control unit 131 is an example of a control unit that operates according to the state of the user (subject).

<Control unit>
The control unit 131 controls each unit of the display device 100 based on information obtained by each unit of the display device 100 . For example, the control unit 131 stores information and data obtained by the information acquisition unit 120 in the storage unit 112 .

In this example, the control unit 131 performs display control, call control, and the like. In display control, the control unit 131 outputs image data to the display unit 110 . Display unit 110 reproduces the image indicated by the image data. In call control, the control unit 131 outputs audio data obtained by the communication unit 124 to the speaker 111 . Speaker 111 reproduces the sound indicated by the sound data. Also, the control unit 131 transmits voice data obtained by the microphone 125 to the communication network via the communication unit 124 .

<State estimation unit>
The configuration of the state estimator 300 in the second embodiment is the same as the configuration of the state estimator 300 in the first embodiment. The state estimation unit 300 estimates the state of the user (subject).

In the second embodiment, the front camera 121 can acquire image data including the face (especially eyeballs) of the user of the display device 100 . The saccade detection unit 301 detects the user's line of sight by performing line-of-sight detection processing on the image data obtained by the front camera 121 . The estimation unit 302 estimates the state of the user based on the user's saccades detected by the saccade detection unit 301 . The operation of the estimating unit 302 of the second embodiment is the same as the operation of the estimating unit 302 of the first embodiment (see FIGS. 13 and 14).

Further, in the second embodiment, when the estimating unit 302 estimates that the driver's condition is an abnormal visual field condition, the control unit 131 performs the first operation according to the abnormal visual field condition. Examples of the first operation include an operation of outputting first notification information for notifying that the user is in the abnormal visual field, For example, an operation for notifying the Examples of the operation of outputting the first notification information include an operation of outputting the first notification information to the display unit 110 and/or the speaker 111, and an operation of outputting the first notification information to another information terminal (not shown) via the communication unit 124. ), and the like. An example of the operation for notifying the surroundings of the display device 100 that the user is in the abnormal visual field state is the operation of blinking a light (not shown) of the display device 100 .

Further, in the second embodiment, when the estimation unit 302 estimates that the state of the driver is in the state of reduced attention function, the control unit 131 performs the second action according to the state of reduced attention function. Examples of the second operation include an operation for resolving the user's attention function deterioration state, an operation for outputting second notification information for notifying that the user is in the attention function deterioration state, and the like. be done. Examples of the operation for resolving the user's attention function deterioration state include an operation of outputting warning information to prompt the user to take a break, and prompting the user to concentrate on operating the display device 100. For example, an operation of outputting alert information for Display unit 110 and/or speaker 111 receives alarm information and alert information output from control unit 131 . As a result, warning information and alert information are output to the user in the form of images and/or sounds. Examples of the operation of outputting the second notification information include an operation of outputting the second notification information to the display unit 110 and/or the speaker 111, and an operation of outputting the second notification information to another information terminal (not shown) via the communication unit 124. ), and the like.

[Effect of Embodiment 2]
The display device 100 of the second embodiment can obtain the same effects as those of the first embodiment. For example, it is possible to estimate a visual field abnormality state in which physical function suddenly deteriorates.

(Other embodiments)
In addition, in the above description, a vehicle is used as an example of a moving object, but the present invention is not limited to this. For example, other examples of moving bodies include ships and airplanes.

Also, in the above description, the case where the display device 100 is a smart phone was taken as an example, but the present invention is not limited to this. For example, the display device 100 may be a television receiver, a game machine, or other device.

Also, the above embodiments may be combined as appropriate. The above embodiments are essentially preferable illustrations, and are not intended to limit the scope of the invention, its applications, or its uses.

As described above, the technology disclosed herein is useful as a state estimation device.

10 vehicle control system 11 actuator 20 information acquisition unit 27 driver state sensor 28 in-vehicle camera (imaging unit)
30 vehicle control device (state estimation device)
35 vehicle control unit (control unit)
300 state estimation unit 301 saccade detection unit 302 estimation unit 303 caution level detection unit 304 setting unit 40 notification unit 41 display unit 42 speaker 100 display device 110 display unit 120 information acquisition unit 121 front camera (imaging unit)
130 control device (state estimation device)
131 control unit

Claims (5)

A state estimation device for estimating the state of a subject who is a driver of a vehicle ,
a saccade detection unit that detects a saccade that is a jumping eye movement that intentionally moves the line of sight of the subject;
During the driving of the vehicle by the subject, the difference between the amplitude of the saccades in the right visual field and the amplitude of the saccades in the left visual field of the subject within a predetermined estimation period is equal to or greater than a predetermined amplitude difference threshold. and the second condition that the difference between the frequency of saccades in the right visual field and the frequency of saccades in the left visual field of the subject within the estimation period is equal to or greater than a predetermined frequency difference threshold. and an estimating unit for estimating that the subject's state is a visual field abnormality state in which one of the left and right sides of the visual field is abnormal when both conditions are satisfied. In claim 1,
The estimator determines that at least one of the first condition and the second condition is not satisfied and the amplitude of the saccade in the entire visual field of the subject within the estimation period is equal to or less than a predetermined amplitude threshold. When at least one of three conditions and a fourth condition that the frequency of saccades in the entire visual field of the subject within the estimation period is equal to or less than a predetermined frequency threshold is established, the subject's state is an attention function. A state estimating device characterized by estimating that it is in a lowered state. In claim 2,
When the state of the subject is estimated to be the abnormal visual field state, performing a first action according to the abnormal visual field state, and when the state of the subject is estimated to be the reduced attention function state (2) a state estimation device comprising: a control unit that performs a second action according to the attention function deterioration state; In claim 3 ,
The first operation includes an operation of controlling travel of the vehicle so that the vehicle evacuates to a safe area;
The state estimation device, wherein the second action includes an action of outputting information for prompting the driver to take a rest. In any one of claims 1 to 4 ,
The state estimation device, wherein the amplitude difference threshold and the frequency difference threshold are set based on the amplitude and frequency of the saccade when the vehicle travels in a predetermined travel scene.

Images