JP2019136165A - Data processing device, monitoring system, awakening system, data processing method, and data processing program - Google Patents

Abstract

To provide a data processing device capable of improving accuracy of acquiring a vestibulo-ocular reflex movement.SOLUTION: A data processing device for performing data processing for monitoring a person includes: a state determination unit for determining whether or not a person or an object that the person operates is in a predetermined state suitable for calculating a vestibulo-ocular reflex movement of the person; an intervention control unit for executing intervention control so that the person or the object is caused to be in the predetermined state when the state determination unit determines that the person or the object is not in the predetermined state; a measurement unit for measuring a pupil motion and a head motion of the person when the intervention control unit executes the intervention control or when the state determination unit determines that the person or the object is in the predetermined state; and a calculation unit for calculating the vestibulo-ocular reflex movement by using data on the pupil motion and the head motion of the person measured by the measurement unit.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a data processing device, a monitoring system, an awakening system, a data processing method, and a data processing program.

  Patent Document 1 discloses a drowsiness sign detection aiming at detecting a sign before a driver of a vehicle, an operator of a facility, etc. is aware of drowsiness using a vestibulo-oculomotor reflex induced by head movement. An apparatus is disclosed.

  The drowsiness sign detection device described in Patent Literature 1 includes a head movement detection unit that detects head movement, an eye movement detection unit that detects eye movement, and head movement data detected by the head movement detection unit. An ideal eye movement angular velocity calculating means for calculating an ideal eye movement angular speed based on the eye movement angular velocity calculating means for calculating an eye rotation angular speed based on the eye movement data detected by the eye movement detecting means, an ideal eye movement angular speed, Vestibulo-ocular reflex (VOR) is detected from the eyeball rotation angular velocity, and drowsiness sign determination means for determining a sign of drowsiness based on the vestibular movement reflex is provided.

[Problems to be solved by the invention]
In Patent Document 1, an experimental system that simulates driving of a vehicle, that is, a driving simulator system is used, and a test task is imposed on the subject such as fixing the upper part of the number plate of the forward vehicle projected on the screen as a point of gaze. The test results are disclosed.

  However, the pseudo experimental environment using the driving simulator system is greatly different from the actual driving environment of the vehicle. As a result of verification in the actual vehicle driving environment (actual vehicle environment), the present inventor has found that it is extremely difficult to accurately monitor the vestibulo-ocular reflex movement in the actual vehicle environment. In addition to the actual vehicle environment, there is a problem that it is difficult to accurately monitor the vestibulo-oculomotor reflex movement in various actual environments such as an operation environment such as equipment and a work environment.

Japanese Patent No. 5255063

Means for solving the problems and their effects

  The present invention has been made in view of the above problems, and provides a data processing device, a monitoring system, a wakefulness system, a data processing method, and a data processing program that can improve the monitoring accuracy of the vestibulo-oculomotor reflex movement in a real environment. The purpose is to do.

In order to achieve the above object, a data processing device (1) according to the present disclosure is a data processing device that performs data processing for monitoring a person,
A state determination unit for determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculation of the person's vestibulo-ocular reflex movement;
An intervention control unit that performs intervention control so that the person or the object is in the predetermined state when it is determined by the state determination unit that the predetermined state is not present;
When the intervention control is performed by the intervention control unit, or when the state determination unit determines that the intervention control unit is in the predetermined state, the measurement unit that measures the pupil movement and head movement of the person,
And a calculation unit that calculates the vestibulo-ocular reflex movement using data of the pupil movement and head movement of the person measured by the measurement unit.

  According to the data processing device (1), if it is determined that the person is not in the predetermined state, the intervention control makes the person or the object a state suitable for calculating the person's vestibulo-ocular reflex movement, in other words, Then, it becomes possible to promote a state where the signal noise (SN) ratio of the data used for calculating the vestibulo-oculomotor reflex movement is increased. When the intervention control prompts the SN ratio to be increased or when it is determined that the SN ratio is in the predetermined state, the human pupil movement and head movement are measured. Since the vestibular movement reflex movement is calculated, the monitoring accuracy of the vestibular movement reflex movement in a real environment can be improved. In addition, when it is determined that the signal is in the predetermined state, in other words, in the state where the signal-to-noise ratio is assumed to be high, the vestibulo-oculomotor reflex movement is calculated without performing the intervention control. It is not necessary to perform the intervention control, and power saving can be realized. Note that the signal noise signal indicates eye movement due to the vestibulo-ocular reflex movement, and noise indicates eye movement other than the vestibulo-ocular reflex movement (for example, saccade movement, vergence movement, etc.).

  Further, the data processing device (2) according to the present disclosure includes a drowsiness determination unit that determines the sleepiness level of the person using the vestibulo-oculomotor reflex movement calculated by the calculation unit in the data processing device (1). It is characterized by having.

  According to the data processing device (2), the sleepiness level in the real environment can be accurately determined by the sleepiness determination unit.

  In addition, the data processing device (3) according to the present disclosure includes, in the data processing device (2), a wake-up control unit that performs control for waking up the person based on the sleepiness level determined by the sleepiness determination unit. It is characterized by having.

  According to the data processing device (3), the awakening control unit can perform control to appropriately awaken the person according to the sleepiness level.

Moreover, the data processing device (4) according to the present disclosure is any one of the data processing devices (1) to (3).
After the intervention control is performed by the intervention control unit, the data measured by the measurement unit includes a data determination unit that determines whether or not the data is suitable for calculation of the vestibulo-oculomotor reflex movement,
The calculation unit calculates the vestibulo-ocular reflex movement using the data determined by the data determination unit as the suitable data among the data measured by the measurement unit. It is characterized by.

  According to the data processing device (4), the vestibular ocular reflex movement is calculated using the data determined to be the suitable data among the data measured after the intervention control is performed. Therefore, the monitoring accuracy of the vestibulo-ocular reflex movement after the intervention control is performed can be further improved.

  In the data processing device (5) according to the present disclosure, in the data processing device (4), the data determination unit weights and determines whether the data is the appropriate data, and the calculation unit Is characterized in that the vestibulo-ocular reflex movement is calculated in consideration of the weighting.

  According to the data processing device (5), it is possible to finely determine whether the data is the suitable data by the weighting. In addition, since the vestibular ocular reflex movement is calculated in consideration of the weighting, the vestibular ocular reflex movement corresponding to the real environment can be monitored using not only a part of the data but also all of the data.

  In the data processing device (6) according to the present disclosure, in any of the data processing devices (1) to (5), the object is a vehicle, and the person is a driver who drives the vehicle. It is a feature.

  According to the data processing device (6), since the object is a vehicle and the person is a driver who drives the vehicle, the driver's vestibulo-ocular reflex movement can be accurately monitored in an actual vehicle environment. Can do.

  Further, the data processing device (7) according to the present disclosure is characterized in that, in the data processing device (6), the predetermined state includes a state in which the vehicle goes straight.

  According to the data processing device (7), the predetermined state includes a state in which the vehicle goes straight. Therefore, intervention control can be performed by the intervention control unit so that the vehicle moves straight. Then, the eyeball movement other than the vestibulo-ocular reflex movement, for example, a state in which a saccade movement or a vergative movement is less likely to occur, in other words, the data measured in a state where the noise component of the vestibulo-ocular reflex movement is reduced. It is possible to accurately monitor the vestibulo-oculomotor reflex movement.

  Further, in the data processing device (8) according to the present disclosure, in the data processing device (6), the predetermined state causes the driver's head to be displaced or vibrated up, down, left, right, front, back, yaw, or pitch. It is characterized by including a state.

  According to the data processing device (8), the predetermined state includes a state in which the head of the driver is displaced or vibrated in the vertical, horizontal, front / rear, yaw, or pitch directions. Therefore, intervention control can be performed by the intervention control unit so that the driver's head is displaced or vibrated in the vertical, horizontal, longitudinal, yaw, or pitch directions. Then, using the data measured in a state where the driver's head is likely to vibrate, in other words, the signal component of the vestibulo-oculomotor reflex movement, particularly in a state where displacement is increased, the vestibulo-oculomotor reflex movement is accurately performed. Can be monitored.

  Further, the data processing device (9) according to the present disclosure is characterized in that, in the data processing device (6), the predetermined state includes a state in which the driver gazes at a specific portion.

  According to the data processing device (9), the predetermined state includes a state in which the driver gazes at a specific location. Therefore, intervention control can be performed by the intervention control unit so as to cause the driver to gaze at a specific location. And, using the data measured in a state in which eye movements other than the vestibular movement reflex movement hardly occur, in other words, in a state where the noise component of the vestibular movement reflex movement is reduced, the vestibular movement reflex movement is accurately monitored. can do.

A monitoring system (1) according to the present disclosure includes any one of the data processing devices (1) to (9) and a camera that captures an image including the person,
The measurement unit of the data processing device measures the pupil movement and head movement of the person using the image acquired from the camera.

  According to the monitoring system (1), since any one of the data processing devices (1) to (9) and the camera are configured, any of the data processing devices (1) to (9) is configured. Thus, it is possible to provide a system that can be easily introduced in various real environments.

In the monitoring system (2) according to the present disclosure, in the monitoring system (1), the measurement unit extracts the human face region from the image,
Extracting the positions of the corners of the eyes and the eyes from the extracted face area;
The head movement is measured from the movement state of the extracted positions of the corners of the eyes and the eyes.

  According to the monitoring system (2), since the measurement unit measures the head movement from the movement state of the positions of the corners of the eyes and the eyes extracted from the face region, the measurement of the head movement is performed. Such a load can be reduced, and the speed of the measurement process can be increased.

Moreover, the wakefulness system which concerns on this indication is comprised including the said data processing apparatus (3) and the wakefulness apparatus controlled by the said wakefulness control part of this data processing apparatus (3).
According to the awakening system, since the awakening device is controlled by the awakening control unit, the person can be awakened by the awakening device.

A data processing method according to the present disclosure is a data processing method for monitoring a person,
A state determination step of determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculating the vestibulooculomotor reflex movement of the person;
An intervention control step for performing an intervention control so that the person or the object is in the predetermined state when it is determined by the state determination step that the state is not in the predetermined state;
A measurement step of measuring the human pupil movement and head movement when the intervention control is performed or when the state determination step determines that the human body is in the predetermined state;
A step of calculating the vestibular ocular reflex movement using the data of the human pupil movement and head movement measured in the measurement step.

  According to the data processing method described above, when it is determined that the person or the object is not in the predetermined state, the person or the object is in a state suitable for calculation of the person's vestibulo-oculomotor reflex movement, in other words, It is possible to induce a state in which the signal-noise (SN) ratio of the data used for calculating the vestibulo-oculomotor reflex movement is high. Then, when it is determined that the state is induced so as to increase the SN ratio or the predetermined state where the SN ratio is high, the human pupil movement and head movement are measured, and the vestibule Since the oculomotor reflex movement is calculated, the monitoring accuracy of the vestibular oculomotor reflex movement in a real environment can be improved.

A data processing program according to the present disclosure is a data processing program for causing at least one computer to execute data processing for monitoring a person,
Said at least one computer,
A state determination step of determining whether or not a person or an object operated by the person is in a predetermined state suitable for calculation of the person's vestibulo-ocular reflex movement;
An intervention control step for performing an intervention control so that the person or the object is in the predetermined state when it is determined by the state determination step that the state is not in the predetermined state;
A measurement step of measuring the human pupil movement and head movement when the intervention control is performed or when the state determination step determines that the human body is in the predetermined state;
The calculation step of calculating the vestibular ocular reflex motion is executed using data of the human pupil movement and head movement measured in the measurement step.

  According to the data processing program, if it is determined that the person is not in the predetermined state, the intervention control step causes the person or the object to be in a state suitable for calculation of the person's vestibulo-oculomotor reflex movement, in other words, It is possible to induce a state in which the signal-noise (SN) ratio of the data used for calculating the vestibulo-oculomotor reflex movement is high. Then, when it is determined that the state is induced so as to increase the SN ratio or the predetermined state where the SN ratio is high, the human pupil movement and head movement are measured, and the vestibule Since the oculomotor reflex movement is calculated, it is possible to realize data processing that can improve the monitoring accuracy of the vestibular oculomotor reflex movement in a real environment.

It is the schematic which shows the example which applied the data processor which concerns on embodiment to the monitoring system. It is a block diagram which shows an example of the hardware constitutions of the monitoring system which concerns on embodiment. It is a flowchart which shows the processing operation which the data processor which concerns on embodiment performs.

  Embodiments of a data processing device, a monitoring system, an awakening system, a data processing method, and a data processing program according to the present invention will be described below with reference to the drawings.

[Application example]
FIG. 1 is a schematic diagram illustrating an example in which a data processing apparatus according to an embodiment is applied to a monitoring system.
The monitoring system 1 includes a data processing device 10 that is mounted on a vehicle 2 and performs data processing for monitoring the driver 3 of the vehicle 2 and a camera 20 that captures an image including the face of the driver 3. Has been.

  The vehicle 2 is an automobile, but may be a vehicle such as a motorcycle, and the type of the vehicle 2 is not particularly limited. Further, the vehicle 2 may be an autonomous driving vehicle. Self-driving vehicles are level 1 (driver assistance), level 2 (partial automatic driving), level 3 (conditional automatic driving), level 4 (altitude) in the automatic driving levels presented by the American Automobile Engineering Association (SAE). The vehicle may be any level of (automatic driving) and level 5 (fully automatic driving).

  The data processing apparatus 10 includes at least one control unit and a storage unit. The control unit includes one or more hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU). The storage unit may be a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a semiconductor device such as a nonvolatile memory or a volatile memory. It is composed of one or more storage devices capable of storing data.

Further, the data processing device 10 is connected to various devices mounted on the vehicle 2, such as an in-vehicle sensor 30, a start switch 40, a navigation device 50, an electronic control unit (ECU) 60, or a vibration device 70. It is configured to be possible. The data processing device 10 can acquire various detection data detected by the in-vehicle sensor 30, an ON / OFF signal of the start switch 40, and the navigation device 50, the electronic control unit 60, the vibration device 70, etc. A predetermined control signal or the like can be output.
In the navigation device 50, the electronic control unit 60, or the vibration device 70, a predetermined control operation is executed based on a predetermined control signal acquired from the data processing device 10.

  The electronic control unit 60 includes one or more electronic control units that control each part of the vehicle 2 such as a driving unit, a braking unit, a steering unit, or a suspension unit. The data processing device 10 may be configured to be able to communicate with the navigation device 50, the electronic control unit 60, or the vibration device 70 via an in-vehicle network such as a CAN (Controller Area Network).

  One or more vibration devices 70 are disposed, for example, on the lower part, the back, or the headrest of the seat of the driver 3, and the upper body or the head of the driver 3 is vertically, front-back, left-right, or It is a device that can vibrate continuously or intermittently in a predetermined cycle in the yaw or pitch direction.

  As described in the Background Art section, in an actual vehicle environment, the road surface condition, the behavior of the vehicle, and the movement of the driver's head and eyes are not constant. Therefore, the data indicating the driver's pupil movement and head movement are different from the vestibulo-ocular reflex movement (signal component), for example, components such as saccade movement (also called impulsive eye movement) and convergence movement. Many (noise components) are included. As a result, the signal noise (SN) ratio of the data used for the calculation of the vestibulo-oculomotor reflex movement tends to deteriorate, and it is difficult to accurately detect the vestibulo-oculomotor reflex movement of the driver. The vestibular eye movement is an involuntary eye movement that suppresses blurring of the retinal image by moving the eyeball in the opposite direction of the head movement. is there.

  In order to solve the problem, the data processing apparatus 10 according to the present embodiment is configured so that the state of the driver 3 or the vehicle 2 is in a predetermined state suitable for calculating the vestibulooculomotor reflex movement in order to improve the SN ratio. If not, predetermined intervention control is performed on the driver 3 or the vehicle 2 so as to be in the predetermined state.

For example, in order to suppress eye movements other than the vestibular eye movement, the driver 3 or the vehicle 2 so that the vehicle 2 moves straight or the driver 3 gazes at a specific location. Intervention control is performed on
Alternatively, in order for the driver 3 to easily cause the vestibulo-oculomotor reflex movement, the driver 3 or the driver 3 is moved so that the head of the driver 3 is displaced or vibrated in the vertical, left and right, front and rear, yaw, or pitch directions. Intervention control may be performed on the vehicle 2.

  Then, the data processing device 10 analyzes the image captured by the camera 20 after the intervention control, measures the pupil movement and head movement data of the driver 3, and uses these measured data to determine the vestibular movement. Calculate reflex motion. By these processes, the detection accuracy of the vestibulo-ocular reflex movement in an actual vehicle environment is improved.

[Configuration example]
FIG. 2 is a block diagram illustrating an example of a hardware configuration of the monitoring system 1 according to the embodiment.
The data processing apparatus 10 includes a data acquisition unit 11, a control unit 12, and a storage unit 13.

  The data acquisition unit 11 is connected to the in-vehicle sensor 30, the start switch 40, the navigation device 50, the electronic control unit 60, the vibration device 70, and the like in addition to the camera 20, and various signals and data are exchanged with these devices. It is configured to include an interface circuit, a connection connector, etc. for performing exchanges.

  The control unit 12 includes a state determination unit 12a, an intervention control unit 12b, a measurement unit 12c, a data determination unit 12d, a calculation unit 12e, a drowsiness determination unit 12f, and a wakefulness control unit 12g. The storage unit 13 includes an image storage unit 13a, an acquisition data storage unit 13b, a measurement data storage unit 13c, and a program storage unit 13d.

  The image storage unit 13 a stores driver image data acquired from the camera 20. Data acquired from the in-vehicle sensor 30, the navigation device 50, the electronic control unit 60, or the like is stored in the acquired data storage unit 13b. In the measurement data storage unit 13c, data such as the driver's pupil movement and head movement measured by the measurement unit 12c is stored in association with each image stored in the image storage unit 13a. The program storage unit 13d stores a data processing program executed by each unit of the control unit 12, data necessary for execution and determination processing of the program, and the like.

  The control unit 12 performs processing for storing image data, measurement data, and the like in the storage unit 13, reads programs and data stored in the storage unit 13, and executes these programs. When the control unit 12 executes these programs in cooperation with the storage unit 13, the state determination unit 12a, the intervention control unit 12b, the measurement unit 12c, the data determination unit 12d, the calculation unit 12e, the sleepiness determination unit 12f, And the operation | movement of each part of the arousal control part 12g is implement | achieved.

  The camera 20 is a device that captures an image including the face of the driver 3 and includes, for example, a lens unit, an image sensor unit, a light irradiation unit, and a control unit that controls these units (not shown). The image sensor section includes, for example, an image sensor such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS), a filter, a microlens, and the like.

  The image pickup element section includes an infrared sensor such as a CCD, a CMOS, or a photodiode that can form a picked-up image by receiving ultraviolet or infrared light, in addition to those that can form a picked-up image by receiving light in the visible region. May be.

  The light irradiation unit includes a light emitting element such as an LED (Light Emitting Diode), and an infrared LED or the like may be used so that the state of the driver can be imaged regardless of day or night. The control unit includes, for example, a CPU, a memory, an image processing circuit, and the like.

  The control unit controls the imaging element unit and the light irradiation unit, emits light (for example, near infrared rays) from the light irradiation unit, and controls the imaging element unit to capture the reflected light. Do. The camera 20 captures an image at a predetermined frame rate (for example, 30 to 60 frames per second), and the image data captured by the camera 20 is output to the data processing device 10.

  Although the camera 20 is composed of one unit, it may be composed of two or more units. The camera 20 may be configured separately from the data processing device 10 (separate housing), or may be configured integrally with the data processing device 10 (same housing). The camera 20 may be a monocular camera or a stereo camera.

  The installation position of the camera 20 in the vehicle interior is not particularly limited as long as it can capture at least the field of view including the face of the driver 3. For example, in addition to the vicinity of the center of the dashboard of the vehicle 2, it may be installed in a steering part, a steering column part, a meter panel part, a position in the vicinity of a rearview mirror, an A pillar part, or a navigation device 50. Information including the specifications of the camera 20 (view angle, number of pixels (vertical x horizontal), etc.) and position and orientation (mounting angle, distance from a predetermined origin (center position of handle, etc.), etc.) 10 may be stored.

  The in-vehicle sensor 30 includes an out-of-vehicle sensor 31, an acceleration sensor 32, a gyro sensor 33, a steering sensor 34, and the like, but may include other sensors.

The vehicle exterior sensor 31 is a sensor that detects an object existing around the vehicle 2. In addition to moving objects such as other vehicles, bicycles, and people, the object may include road markings such as white lines, guardrails, median strips, and other structures that affect the traveling of the vehicle 2. The vehicle exterior sensor 31 includes at least one of a front monitoring camera, a rear monitoring camera, a radar (Radar), a rider, that is, Light Detection and Ranging, Laser Imaging Detection and Ranging (LIDAR), and an ultrasonic sensor. Has been. The detection data of the object detected by the outside sensor 31 may be output to the electronic control unit 60 in addition to being output to the data processing device 10.
As the front monitoring camera and the rear monitoring camera, a stereo camera, a monocular camera, or the like can be adopted. The radar detects a position, a direction, a distance, and the like of an object by transmitting radio waves such as millimeter waves around the vehicle and receiving radio waves reflected by the object existing around the vehicle. The rider detects the position, direction, distance, and the like of the object by transmitting laser light around the vehicle and receiving light reflected by the object existing around the vehicle.

  The acceleration sensor 32 is a sensor that detects the acceleration of the vehicle 2, and a triaxial acceleration sensor that detects acceleration in three directions of the XYZ axes may be used, or a biaxial, uniaxial acceleration sensor may be used. . As the triaxial acceleration sensor, a semiconductor type acceleration sensor such as a piezoresistive type in addition to the capacitance type may be used. The acceleration data detected by the acceleration sensor 32 may be output to the navigation device 50 or the electronic control unit 60 in addition to being output to the data processing device 10.

  The gyro sensor 33 is an angular velocity sensor that detects a rotational angular velocity (for example, a yaw rate) of the vehicle 2. The rotational angular velocity signal detected by the gyro sensor 33 may be output to the data processing device 10 or may be output to the navigation device 50 or the electronic control unit 60.

  The steering sensor 34 is a sensor that detects a steering amount with respect to the steering of the vehicle 2. For example, the steering sensor 34 is provided on a steering shaft of the vehicle 2 and detects a steering torque or steering angle given to the steering by the driver 3. A signal detected by the steering sensor 43 according to the steering operation of the driver 3 may be output to the electronic control unit 60 in addition to being output to the data processing device 10.

  The navigation device 50 includes a control unit, a display unit, a voice output unit, an operation unit, a map data storage unit, a GPS reception unit, and the like (not shown). The navigation device 50 determines the road or lane on which the vehicle 2 travels based on the position information of the vehicle 2 measured by the GPS receiver and the map information in the map data storage unit, and determines the destination from the current position of the vehicle 2. The route is calculated on the display unit (not shown), and voice output such as route guidance is output from the voice output unit (not shown). The configuration may be such that the position information of the vehicle 2, the information on the traveling road, the information on the planned traveling route, and the like obtained by the navigation device 50 are output to the data processing device 10.

  The awakening device 80 is a device controlled by the awakening control unit 12g of the data processing device 10, and executes an operation for waking up the driver 3 based on a control signal from the awakening control unit 12g. The awakening device 80 is, for example, an alarm device that issues a warning to the driver 3 by sound or light, an air conditioner that blows cold air, hot air, or gas containing an odor component or odor component, etc. to the driver 3, a steering, or a seat belt. It can be configured with a vibration device or the like that vibrates.

Next, the configuration of each unit of the data processing apparatus 10 will be described.
The image data of the camera 20 acquired by the data acquisition unit 11 is stored in the image storage unit 13a. Moreover, the data of the vehicle-mounted sensor 30, the navigation device 50, or the electronic control unit 60 acquired by the data acquisition unit 11 is stored in the acquisition data storage unit 13b.

The state determination unit 12a reads data used for state determination from the image storage unit 13a or the acquired data storage unit 13b, and using the read data, the state of the driver 3 or the vehicle 2 is suitable for calculating the vestibulo-oculomotor reflex movement. A process for determining whether or not the predetermined state is present is performed.
In the predetermined state, the head of the driver 3 is likely to vibrate (in other words, the signal component of the vestibulo-oculomotor reflex movement, particularly the displacement increases), for example, the head of the driver 3 moves up and down. , A left-right or front-rear direction, a yaw or pitch direction, and the like are easily displaced or vibrated.

  Further, the predetermined state is a state in which eye movements other than the vestibulo-ocular reflex movement (for example, saccade movement, vergence movement, etc.) are unlikely to occur (in other words, the noise component of the vestibulo-ocular reflex movement is reduced). For example, a state where the vehicle 2 is traveling on a straight road or a state where the driver 3 is gazing at a specific location is included.

  When the state determination unit 12a determines that the predetermined state is present, a first determination signal indicating that the predetermined state is present is output to the measurement unit 12c. On the other hand, when the state determination unit 12a determines that the predetermined state is not reached, a second determination signal indicating that the state is not in the predetermined state is output to the intervention control unit 12b.

  When the intervention control unit 12b acquires the second determination signal from the state determination unit 12a, the state of the driver 3 or the vehicle 2 is the predetermined state suitable for the calculation of the vestibular eye movement of the driver 3 Intervention control is performed so that an intervention execution signal indicating that the intervention control has been executed is output to the measurement unit 12c.

An example of intervention control performed by the intervention control unit 12b will be described.
(1) The intervention control unit 12b performs intervention control to output a route change signal that instructs the navigation device 50 to change the route so as to travel on a straight road. When traveling on a straight road, the driver 3 is usually facing forward, so that the line of sight is easily determined in one direction. Therefore, in the data processing apparatus 10, it is possible to measure head movement and pupil movement in a state in which eye movement other than the vestibulo-oculomotor reflex movement hardly occurs by the intervention control.

  The route change signal may be, for example, a control signal for changing the current location of the vehicle 2 and a planned travel location ahead of a predetermined distance (for example, 500 m) from the current location to a route on a straight road. Further, the route change signal is, for example, a straight extension line from the current location of the vehicle 2 to a first scheduled travel location ahead of a predetermined distance (for example, 500 m) and a predetermined distance from the first planned travel location ( For example, it may be a control signal for changing the route to a road where the angle formed with the straight line to the second planned traveling point ahead of 500 m) is smaller than a predetermined angle.

  Further, when the navigation device 50 acquires the route change signal from the data processing device 10, the navigation device 50 determines whether or not the traveling road of the vehicle 2 is a straight road that satisfies the condition specified by the route change signal.

  When the set route of the navigation device 50 does not satisfy the condition specified by the route change signal, the navigation device 50 performs a route search for each intersection, for example, and matches the condition specified by the route change signal. Repeat the process of changing to root. By the intervention control, the state of the vehicle 2 can be brought into a state suitable for the calculation of the vestibular eye movement.

(2) When the surrounding safety is confirmed by the vehicle outside sensor 31, the intervention control unit 12b performs a control to output a notification signal that prompts the navigation device 50 to gaze the driver 3 at a predetermined location. Also good. The confirmation of the surrounding safety indicates that, for example, a preceding vehicle or a succeeding vehicle is not detected 100 m forward and rearward of the vehicle 2 and no other obstacle is detected within 5 m around the vehicle. However, the method of confirming the surrounding safety is not limited to this.
When the navigation device 50 receives the notification signal from the data processing device 10, for example, the navigation device 50 outputs a sound such as “please look closely at the front” or displays the information on the screen. By the intervention control, it is possible to guide the eye state of the driver 3 to a state suitable for calculating the vestibulo-oculomotor reflex movement, that is, a state in which the front is watched.

(3) The intervention control unit 12b may control the navigation device 50 to output a notification signal that prompts the driver 3 to move the head.
When the navigation device 50 receives the notification signal from the data processing device 10, for example, it outputs a sound such as “Please move your head up and down or left and right while looking forward” or display it on the screen. Execute control to By the intervention control, the state of the head of the driver 3 can be guided to a state suitable for calculating the vestibulo-oculomotor reflex movement, that is, a state of vibrating the head.

(4) The intervention control unit 12b may perform control for the vibration device 70 to output a control signal for vibrating the driver's 3 seat in the vertical direction, the horizontal direction, the front-rear direction, or the pitch direction at a predetermined cycle. .
When the control device receives the control signal from the data processing device 10, the vibration device 70 executes, for example, control for vibrating the driver's 3 seat up and down, left and right, front and rear, or in the pitch direction at a predetermined vibration cycle. By the intervention control, the head of the driver 3 can be brought into a state suitable for the calculation of the vestibulo-oculomotor reflex movement, that is, the head is vibrated.

(5) The intervention control unit 12b may perform control to output a control signal for controlling the suspension control unit 61 to easily vibrate the vehicle 2.
When the suspension control unit 61 receives the control signal from the data processing device 10, for example, the suspension control unit 61 adjusts the damping force of the active suspension mechanism for a certain period to make the vehicle body easy to swing up and down or to the left and right. Execute. By the intervention control, the head of the driver 3 can be brought into a state suitable for the calculation of the vestibulo-oculomotor reflex movement, that is, the head is vibrated.

  In addition, in the intervention control part 12b, you may comprise so that intervention control in any one of said (1)-(5) may be performed, and said (1)-said according to the state of the vehicle 2 or the driver | operator 3. You may comprise so that two or more intervention control of (5) may be performed combining suitably.

  When the measurement unit 12c acquires the first determination signal from the state determination unit 12a, the measurement unit 12c performs a process of measuring the pupil movement and head movement of the driver 3 from the image captured by the camera 20. For example, the measurement process is performed for each frame of the image, but may be performed at predetermined frame intervals. Thereafter, the measurement unit 12c performs processing for storing the measured pupil movement and head movement data in the measurement data storage unit 13c, and also instructs the calculation unit 12e to calculate the vestibulo-oculomotor reflex movement (calculation instruction signal). ) Is output.

  Moreover, the measurement part 12c performs the process which measures the driver | operator's 3 pupil movement and head movement, even when the said intervention execution signal is acquired from the intervention control part 12b. In this case, the measurement unit 12c performs a process of storing data relating to the measured pupil movement and head movement in the measurement data storage unit 13c, and the measurement data is calculated to the vestibulo-ocular reflex movement to the data determination unit 12d. A signal (determination instruction signal) for instructing whether or not the data is suitable for the data is output.

An example of pupil movement measurement processing performed by the measurement unit 12c will be described.
First, the measurement unit 12c detects the face (face area) of the driver 3 from an image captured by the camera 20 by template matching. The face area may be detected using a face template image prepared in advance. Next, the measurement unit 12c detects the position of the pupil from the face area by template matching for the face area of the driver 3 detected from the image. The pupil position may be detected using a pupil template image prepared in advance. Then, the measuring unit 12c detects the position of the pupil for each frame of the image, and measures the pupil movement from the position change (movement amount) of the pupil for each frame.

Next, an example of the head movement measurement process performed by the measurement unit 12c will be described.
First, the measurement unit 12c extracts the face (face area) of the driver 3 from an image captured by the camera 20 by template matching. The face area may be detected using a face template image prepared in advance. Moreover, you may use the data of the driver | operator's 3 face area detected by the process which measures the said pupil movement.

Next, the measurement unit 12c detects the position of the eye from the face region by template matching for the face region of the driver 3 detected from the image. The eye position may be detected by using an eye template image prepared in advance. Coordinates indicating the positions of the corners of the eyes and the eyes are linked in advance to the eye template image. From the coordinates of the corners of the eyes and the eyes, the positions of the eyes and the eyes of the driver 3 in the image can be detected. Since the positions of the corners of the eyes and the eyes do not move depending on the opening / closing operation of the eyes such as blinking, it can be estimated that the position changes of the corners of the eyes and the eyes are moved by the head movement.
Then, the measuring unit 12c detects the positions of the corners of the eyes and the eyes of the driver 3 for each frame of the image, and measures the head movement from the positional changes (movement amounts) of the corners of the eyes and the eyes for each frame.

In addition, when detecting the position of the driver | operator's 3 eye | corner and eye corner from an image, you may detect in combination with the distance image data containing three-dimensional position information other than using two-dimensional image data.
In order to acquire the distance image data, for example, the monitoring system 1 may be equipped with a three-dimensional image measurement unit. The three-dimensional image measurement unit acquires a three-dimensional image (distance image) in which each pixel of a captured image has a value of distance to the target (information on depth). The three-dimensional image measurement unit may be a passive measurement unit such as a stereo method, or may be an active measurement unit that projects light such as an optical radar or pattern light.

  By combining the two-dimensional image and the distance image, the change in the position of the driver's eye corner and the eye is due to the parallel movement of the head (movement in the vertical or horizontal direction), or rotational movement (yaw) Or movement in the pitch direction) can be accurately detected. With this configuration, it becomes possible to measure the pupil movement and the head movement more accurately, and the monitoring accuracy of the vestibulo-ocular reflex movement can be further increased.

  Moreover, the measurement process of the pupil movement and the head movement of the driver 3 is not limited to the above-described method, and various known methods can be employed. For example, as disclosed in International Publication No. 2006/051607 pamphlet and Japanese Patent Application Laid-Open No. 2007-249280, feature points of facial organs are detected for each frame of an image, and feature points of facial organs are detected. The face orientation may be obtained from the position, and the head movement may be measured from the change (movement amount) of the face orientation for each frame.

When the data determination unit 12d acquires the determination instruction signal from the measurement unit 12c, the data measured by the measurement unit 12c after the intervention control is performed by the intervention control unit 12b is suitable for calculating the vestibular eye movement. To determine whether the data is correct.
When the data determination unit 12d determines that the data is suitable for the calculation of the vestibular eye movement, the data determination unit 12d outputs a signal (calculation instruction signal) instructing the calculation unit 12e to calculate the vestibular movement movement. .
On the other hand, when the data determination unit 12d determines that the data is not suitable for calculating the vestibular eye movement, the data determination unit 12d outputs a signal (intervention instruction signal) instructing the intervention control unit 12b. .

An example of the determination process performed by the data determination unit 12d will be described.
(1) The data determination unit 12d determines whether or not pupil movement and head movement data can be acquired by the measurement unit 12c. In the first place, if the pupil movement and head movement data cannot be acquired, the vestibulo-oculomotor reflex movement cannot be calculated.

  Therefore, when the pupil movement and the head movement are measured by template matching in the measurement unit 12c, the similarity between the face area extracted from the image and the face template image, or the eye area and the eye template extracted from the image The similarity with the image is determined. If each similarity is lower than a predetermined threshold, the position of the head (eye, ie, the corner of the eye and the eye) or the position of the pupil has not been properly acquired from the image, that is, suitable for calculating the vestibulo-oculomotor reflex movement. It may be determined that the acquired data has not been acquired.

(2) The data determination unit 12d may determine whether the pupil movement data is data including a lot of noise components such as eye movements other than the vestibulo-oculomotor reflex movement, that is, saccade movement.
For example, when the pupil's momentum is greater than the head's momentum, or when the pupil moves or rotates following the movement or rotation direction of the head, eye movement such as the rotation speed or rotation angle of the eyeball When it is larger than the predetermined threshold value, it may be determined that the pupil movement data includes a lot of noise components such as saccade movement and is not suitable for the calculation of the vestibular eye movement.

  In addition, when the face moves greatly, the driver 3 is not in a state of focusing on a certain direction, so the head movement such as the rotation speed and the rotation angle of the face is larger than a predetermined threshold. In this case, it may be determined that the head movement data includes a lot of noise components and is not head movement data suitable for the calculation of the vestibulo-oculomotor reflex movement.

  (3) The data determination unit 12d acquires vehicle speed data when the measurement unit 12c measures pupil movement and head movement, and whether or not the vehicle speed data is smaller than a predetermined speed, or the vehicle speed data is It may be determined whether or not the data measured by the measurement unit 12c is data suitable for calculation of the vestibulo-ocular reflex movement by determining whether or not the speed is higher than a predetermined speed.

In general, when the vehicle speed is high, the driver 3 tends to concentrate on a narrow range in front. On the other hand, when the vehicle speed is low, the driver 3 tends to voluntarily overlook a wide range in order to ensure the surrounding safety. When calculating the vestibular eye movement, it is preferable to use data on eye movements and head movements when the driver 3 is concentrating on a narrow range.
Therefore, when the vehicle speed data is smaller than a predetermined speed (for example, slow speed), the data determination unit 12d determines that the data is not suitable for calculating the vestibular eye movement. Alternatively, when the vehicle speed data is larger than a predetermined speed, it may be determined that the data is suitable for calculating the vestibulo-ocular reflex movement.

  Furthermore, the data determination unit 12d may perform weighting according to the vehicle speed on the data of the eye movement and the head movement, instead of performing the binary determination using the predetermined speed as a threshold value. For example, if the vehicle speed is 0 to 20 km / h, the weighting factor is 0.2; if the speed is 20 to 40 km / h, the weighting factor is 0.5; if the speed is 40 to 60 km / h, the weighting factor is 0.8; The coefficient 1.0 may be stored in association with the eye movement and head movement data, and the vestibular eye movement may be calculated in consideration of the weight coefficient.

  (4) The data determination unit 12d acquires steering data when the measurement unit 12c measures pupil movement and head movement, and determines whether the steering data is greater than a predetermined steering angle. Alternatively, it may be determined whether or not the data measured by the measurement unit 12c is data suitable for calculating the vestibulo-oculomotor reflex movement.

  When calculating the vestibular eye movement, it is preferable to use data on eye movements and head movements in a state where the driver 3 is concentrating on a narrow area ahead. The driver 3 is more likely to concentrate on a narrow area ahead when traveling on a straight road than when traveling on a curved road. Therefore, the data determination unit 12d may determine that the data is not suitable for calculating the vestibular eye movement when the steering data is larger than a predetermined steering angle.

  (5) The data determination unit 12d acquires the position data or traveling road data of the vehicle 2 when the measurement unit 12c measures the pupil movement and the head movement, and the vehicle 2 is traveling on the straight road. It may be determined whether or not the data measured by the measurement unit 12c is data suitable for calculating the vestibulo-oculomotor reflex movement.

  The driver 3 is more likely to concentrate on a narrow area ahead when traveling on a straight road than when traveling on a curved road. Therefore, the data determination unit 12d may determine that the data is not suitable for calculating the vestibular eye movement when the vehicle 2 is not traveling on a straight road.

  (6) The data determination unit 12d acquires the periphery monitoring data acquired by the vehicle outside sensor 31 when the measurement unit 12c measures the pupil movement and the head movement, and the obstacle around the vehicle 2 By determining whether or not a vehicle or the like exists, it may be determined whether or not the data measured by the measurement unit 12c is data suitable for calculating the vestibulo-oculomotor reflex movement.

  When there is a preceding vehicle or obstacle that moves relative to the vehicle 2, for example, when approaching, the driver 3 tends to track the preceding vehicle or obstacle that is moving relatively. The eyes of the driver 3 are actively moving. The state in which the eye is actively moved is not in a state suitable for calculating the vestibulo-ocular reflex movement. Therefore, the data determination unit 12d may determine that the data is not suitable for the calculation of the vestibular eye movement when the preceding vehicle or an obstacle moving relative to the vehicle 2 is detected.

  (7) The data determination unit 12d acquires the direction of the line of sight of the driver 3 when measuring the pupil movement and the head movement by the measurement unit 12c, and based on the direction of the line of sight of the driver 3, the measurement unit You may determine whether the data measured by 12c are data suitable for calculation of a vestibular movement reflex movement. As a method for detecting the direction of the line of sight of the driver 3 from the image obtained by capturing the face of the driver 3, a known line of sight detection method is employed.

  For example, when the driver 3 is looking at a distant place such as the direction of the horizon, it is highly likely that the driver 3 is looking at the front. Therefore, for example, the direction of the line of sight is the front of the vehicle (reference direction). If the angle is within a predetermined angle (for example, ± 5 degrees in the vertical direction or ± 5 degrees in the horizontal direction), it may be determined that the front is concentrated.

  In addition, when the driver 3 is looking at an operation unit or a display unit in the vehicle such as the navigation device 50, there is a high possibility that the driver 3 is gazing at a narrow range. Therefore, for example, when the direction of the line of sight of the driver 3 is the installation direction of the navigation device 50 or the like, it may be determined that the data is suitable for calculating the vestibular eye movement. However, when calculating the vestibulo-oculomotor reflex movement from the measurement data while gazing at the equipment in the vehicle, it should be applied to an autonomous driving vehicle having an autonomous driving level of SAE level 3 or higher from the viewpoint of safety. Is preferred.

(8) The data determination unit 12d determines whether the data is suitable for the calculation of the vestibulo-ocular reflex movement based on the amount of movement (parallel movement or rotation movement) of the head movement measured by the measurement unit 12c. Also good.
The vestibulo-ocular reflex movement is an eye movement that does not occur unless the head of the driver 3 moves. Therefore, the data determination unit 12d determines that the data is not suitable for calculating the vestibulo-oculomotor reflex movement when the momentum of the head movement (parallel movement or rotation movement) measured by the measurement unit 12c is smaller than the predetermined momentum. May be.

(9) The data determination unit 12d acquires acceleration data of the vehicle when the pupil movement and head movement are measured by the measurement unit 12c, and is measured by the measurement unit 12c based on the acceleration data of the vehicle 2 It may be determined whether or not the data is data suitable for calculating the vestibulo-ocular reflex movement.
When a predetermined acceleration is generated in the up / down, left / right or front / rear direction of the vehicle 2, the head of the driver 3 can easily move in the up / down, left / right or pitch direction.

Therefore, the data determination unit 12d may determine that the acceleration data of the vehicle 2 is data suitable for the calculation of the vestibulo-oculomotor reflex movement when the acceleration data of the vehicle 2 is larger than the threshold value at which the driver 3 is likely to generate head movement.
Further, the data determination unit 12d has a fixed relationship such that the vibration of the vehicle 2 obtained from the acceleration data of the vehicle 2 and the head movement of the driver 3 are vibrating in the same direction and at the same frequency. If it is, it is determined that the data is suitable for the calculation of the vestibulo-oculomotor reflex movement.
As the acceleration data of the vehicle 2, data from the acceleration sensor 32 equipped on the vehicle 2 can be used, and the speed of the vehicle 2 is obtained from the time series change of the distance to the object recognized by the outside sensor 31, and the speed Acceleration data obtained from the above may be used.

  In addition, in the data determination part 12d, you may comprise so that determination in any one of said (1)-(9) may be performed, and according to the state of the vehicle 2 or the driver | operator 3, said (1)-( You may comprise so that 2 or more determinations of 9) may be performed combining suitably.

When the calculation unit 12e acquires the calculation instruction signal from the measurement unit 12c, the calculation unit 12e reads out the pupil movement data and the head movement data of the driver 3 from the measurement data storage unit 13c, and uses these data to perform the vestibulo-ocular reflex movement. Processing to calculate is performed.
Further, when the calculation unit 12e acquires the calculation instruction signal from the data determination unit 12d, the calculation unit 12e reads out the data of the pupil movement and the head movement of the driver 3 from the measurement data storage unit 13c, and uses these data to read the vestibular eye movement Processing for calculating the reflex motion is performed. After performing the calculation process, the calculation unit 12e outputs calculation data (parameters) related to the vestibulo-oculomotor reflex movement to the drowsiness determination unit 12f.

The parameter relating to the vestibulo-ocular reflex movement calculated by the calculation unit 12e includes, for example, at least one of VOR gain, residual standard deviation, and phase difference data.
VOR gain, in principle, means the degree of response of pupil movement (rotational angular velocity) to head movement (rotational angular velocity), expressed as pupil movement (rotational angular velocity) / head movement (rotational angular velocity). Can do.

For example, the VOR gain is expressed by the equation [Equation 2] as the coefficient G of the regression model of the equation [Equation 1] where the objective variable is the eyeball rotation angular velocity e (t) and the explanatory variable is the ideal eyeball angular velocity h (t) and the constant term dc. Can be obtained by least square estimation. Here, ε (t) is the residual of the regression model. Further, τ is a delay time of the eye movement with respect to the ideal eye movement.
The eyeball rotation angular velocity e (t) can be obtained by obtaining an eyeball movement angle based on the pupil movement data measured by the measurement unit 12c and differentiating the eyeball movement angle. The ideal eyeball angular velocity h (t) can be obtained by obtaining a head movement angle based on the head movement data measured by the measurement unit 12c and differentiating the head movement angle. The VOR gain may be calculated in at least one direction of the driver 3 in the front-rear direction, the upper-lower direction, the left-right direction, the yaw direction, and the pitch direction.

Further, the residual standard deviation (SDres) can be calculated by the equation [Equation 3].

  The VOR gain and the residual standard deviation are over the second time shorter than the first time, with the data of the first time (for example, several tens of seconds) as one segment so that sufficient estimation accuracy can be obtained. You may calculate the value in each segment for every 3rd time shorter than 2nd time, giving a lap. In general, when the driver 3 is drowsy, the VOR gain decreases and the residual standard deviation tends to increase. Therefore, in order to accurately determine a sign of sleepiness, a change rate such as a decrease rate of the VOR gain or a change rate such as an increase rate of the residual standard deviation may be obtained.

  When the drowsiness determination unit 12f acquires the calculation data related to the vestibular movement reflex movement from the calculation unit 12e, the drowsiness determination unit 12f determines the drowsiness of the driver 3 using the acquired calculation data related to the vestibular movement reflex movement. For example, the drowsiness level of the driver 3 may be determined by comparing with a predetermined threshold value using at least one of the parameters of VOR gain, residual standard deviation, and phase difference. The determination may be made in consideration of other parameters. After determining the sleepiness of the driver 3, the sleepiness determination unit 12f outputs the determination result (for example, sleepiness level) of the driver 3 to the awakening control unit 12g.

The awakening control unit 12g performs a process of outputting a control signal for awakening the driver 3 to the awakening device 80 based on the sleepiness level acquired from the sleepiness determination unit 12f.
When the wake-up device 80 is configured by an alarm device that issues a warning to the driver 3 with sound or light, for example, the wake-up control unit 12g outputs a control signal for operating the alarm device for a predetermined period to the alarm device. To do.
In addition, when the awakening device 80 is configured by an air conditioner that blows cold air, warm air, or gas containing a fragrance component or odor component to the driver 3, the awakening control unit 12g operates the air conditioner for a predetermined period. A control signal to be output to the alarm device.
When the awakening device 80 is configured by a vibration device that vibrates a steering wheel, a seat belt, or the like, the awakening control unit 12g outputs a control signal for operating the vibration device for a predetermined period to the vibration device.

  Moreover, you may perform the process which outputs the control signal for awakening the driver | operator 3 with respect to the navigation apparatus 50 or the vibration apparatus 70. FIG. In this case, the control signal includes a control signal for outputting a warning sound or a warning display for awakening the driver 3 to the navigation device 50, a control signal for vibrating the seat for a predetermined period by the vibration device 70, and the like. It is.

[Operation example]
FIG. 3 is a flowchart showing processing operations performed by the control unit 12 in the data processing apparatus 10 according to the embodiment. This processing operation is executed after the start switch 40 of the vehicle 2 is turned on, for example.

  In step S1, the control unit 12 performs a process of starting the camera 20. The camera 20 captures a predetermined number of frames per second. The control unit 12 captures these captured images in time series, and executes this process every frame or every frame at a predetermined interval.

  In step S <b> 2, the data acquisition unit 11 performs a process of acquiring data from the camera 20, the in-vehicle sensor 30, or the navigation device 50. For example, in addition to acquiring image data of the driver 3 from the camera 20, peripheral monitoring data or the like may be acquired from the in-vehicle sensor 30, or road data including the shape (straight line, curve, etc.) of the traveling road may be acquired. You may acquire from the navigation apparatus 50.

  In step S3, the state determination unit 12a uses the data acquired in step S2 to determine whether the state of the driver 3 or the vehicle 2 is in a predetermined state suitable for calculating the vestibular eye movement (VOR). Determine. The predetermined state is as described above.

If the state determination part 12a determines in step S3 that the state of the driver 3 or the vehicle 2 is not in the predetermined state, the control unit 12 executes step S4.
In step S4, the intervention control unit 12b performs intervention control so that the driver 3 or the vehicle 2 is in the predetermined state, and then the control unit 12 executes step S5. In step S4, the intervention control described in any one of (1) to (5) described above may be performed, or the above (1) to (1) may be performed depending on the state of the vehicle 2 or the driver 3. You may comprise so that two or more intervention control of (5) may be performed combining suitably.

  In step S5, the measurement unit 12c performs a process of measuring the pupil movement of the driver 3, and in the next step S6, the measurement unit 12c performs a process of measuring the head movement of the driver 3 and then performs control. Unit 12 executes step S7. Note that the order of steps S5 and S6 may be changed.

  In step S7, the data determination unit 12d determines whether or not the data of the driver's 3 pupil movement and head movement measured in steps S5 and S6 is applicable to the calculation of the vestibulo-oculomotor reflex movement. In step S7, the above-described determinations (1) to (9) may be performed. Depending on the state of the vehicle 2 or the driver 3, the above (1) to (9) may be used. You may comprise so that 2 or more determinations of 9) may be performed combining suitably.

  In step S7, if the data determination unit 12d determines that the data is not applicable to the calculation of the vestibular eye movement, the control unit 12 returns to the process of step S4 and repeats the process. On the other hand, if the data determination unit 12d determines in step S7 that the data can be applied to the calculation of the vestibular eye movement, the control unit 12 executes step S10.

On the other hand, if the state determination unit 12a determines in step S3 that the state of the driver 3 or the vehicle 2 is in the predetermined state, the control unit 12 executes step S8.
In step S8, the measurement unit 12c performs a process of measuring the pupil movement of the driver 3, and then the control unit 12 executes step S9. The pupil movement measurement process is the same as that in step S5.

  In Step S9, measurement part 12c performs processing which measures driver's 3 head movement, and control unit 12 performs Step S10 next. The head movement measurement process is the same as that in step S6. Note that the order of steps S8 and S9 may be changed.

  In step S10, the calculation unit 12e calculates the vestibular eye movement, and then the control unit 12 executes step S11. The calculated data of the vestibulo-ocular reflex movement includes at least one of VOR gain, residual standard deviation, and phase difference.

In step S11, the sleepiness determination unit 12f calculates the sleepiness level of the driver 3, and in the next step S12, the sleepiness determination unit 12f determines whether the sleepiness level is smaller than a predetermined threshold. If it is determined in step S12 that the sleepiness level is equal to or greater than a predetermined threshold, the control unit 12 executes step S13.
In step S13, the awakening control unit 12g performs a process of outputting a predetermined control signal for waking up the driver 3 to the navigation device 50 or the vibration device 70, and then the control unit 12 executes step S14.

  In step S14, it is determined whether or not the start switch 40 is turned off. If it is determined that the start switch 40 is not turned off, the control unit 12 returns to the process of step S2. On the other hand, if it is determined in step S14 that the start switch 40 has been turned off, the control unit 12 executes step S15. In step S15, the control unit 12 stops driving the camera 20 and ends the process.

[Action / Effect]
According to the data processing device 10 according to the above-described embodiment, when the predetermined state is not suitable for the calculation of the vestibular eye movement, it is suitable for the calculation of the vestibular eye movement of the driver 3 by the intervention control by the intervention control unit 12b. In other words, in other words, the driver 3 or the vehicle 2 can be urged to a state in which the signal noise (SN) ratio for calculating the vestibular eye movement can be increased.
Since the driver's 3 pupil movement and head movement are measured and the vestibulooculomotor reflex movement is calculated in a state in which the signal-to-noise ratio is urged to be high, the vestibular ocular reflex movement is calculated even in an actual vehicle environment. Monitoring accuracy can be improved.

  Further, according to the data processing device 10, the vestibulo-oculomotor reflex movement is performed using the suitable data determined by the data determination unit 12d among the data measured after the intervention control by the intervention control unit 12b. Since it is calculated, the monitoring accuracy of the vestibulo-ocular reflex movement after the intervention control is performed can be further increased.

  Further, according to the data processing device 10, when it is determined that the predetermined state is present, in other words, in a state where the SN ratio is assumed to be high, without performing intervention control by the intervention control unit 12b, The vestibulooculomotor reflex movement can be calculated using the data of the pupil movement and head movement of the driver 3 measured by the measurement unit 12c, and the burden on the driver 3 due to the intervention control can be reduced. .

  Further, according to the data processing device 10, since the drowsiness determination unit 12f is provided, the drowsiness level of the driver 3 in the actual vehicle environment can be accurately determined, and the awakening control unit 12g is provided. Thus, it is possible to perform control to appropriately wake the driver 3 in accordance with the sleepiness level.

Moreover, according to the monitoring system 1 provided with the data processor 10 and the camera 20, it is possible to provide a monitoring system that can be easily introduced in an actual vehicle environment.
Moreover, the system which can awaken the driver | operator 3 appropriately in a real vehicle environment by the awakening system provided with the data processing apparatus 10 and the awakening apparatus 80 can be provided.

[Modification]
As mentioned above, although embodiment of this invention was described in detail, the above description is only the illustration of this invention in all the points. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention.

(Modification 1)
The control unit 12 of the data processing apparatus 10 does not have to include all the units illustrated in FIG. 2. In another embodiment, the control unit 12 includes at least the intervention control unit 12b, the measurement unit 12c, and the calculation unit 12e. A second configuration further comprising a state determination unit 12a in the first configuration, a third configuration further comprising a data determination unit 12d in the first configuration or the second configuration A fourth configuration further including a drowsiness determination unit 12f in any one of the first to third configurations, or a fifth configuration further including an arousal control unit 12g in the fourth configuration. Also good.

(Modification 2)
In the processing operation performed by the control unit 12 described with reference to FIG. 3, the pupil movement measurement process in step S8 and the head movement measurement process in step S9 are executed between step S2 and step S3. Then, after the head movement measurement process, the determination process of step S3 is performed, and if it is determined in step S3 that the predetermined state is present, the process proceeds to step S10 to calculate the vestibular ocular reflex movement. May be. In this case, the data determination process of step S7 may be executed instead of the state determination process of step S3.

(Modification 3)
In the data determination process in step S7, binary determination is performed as to whether or not the pupil movement and head movement data of the driver 3 can be applied to the calculation of the vestibulo-oculomotor reflex movement. In this embodiment, instead of binary determination, weighting determination according to an applicable state may be performed. Then, the process proceeds to step S10 after the weighting determination, and the vestibular ocular reflex movement may be calculated using the pupil movement and head movement data weighted in step S7.

  When such a configuration is employed, it is possible to finely determine whether the data measured by the measurement unit 12c is data suitable for calculating the vestibulo-oculomotor reflex movement by the weighting. Further, by calculating the vestibular ocular reflex motion in consideration of the weighting, the vestibulo ocular reflex motion corresponding to the actual vehicle environment can be monitored using not only a part of the data but also all.

(Modification 4)
Moreover, although the said embodiment demonstrated the case where the monitoring system 1 and the data processing apparatus 10 were mounted in the vehicle 2, the monitoring system 1 and the data processing apparatus 10 are not limited to vehicle-mounted use.
In another embodiment, for example, the monitoring system 1 and the data processing device 10 are installed in a factory or an office, and a person who operates equipment installed in the factory, a person who performs a predetermined work on a desk, etc. It can be widely applied to a system for monitoring sleepiness. In this case, a product operated by a person in a factory includes, for example, a production apparatus. In addition, for example, office equipment such as a personal computer can be cited as a thing that a person operates in an office.

[Appendix]
Embodiments of the present invention can also be described as in the following supplementary notes, but are not limited thereto.
(Appendix 1)
A data processing device (10) for performing data processing for monitoring a person,
A state determination unit (12a) for determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculating the vestibulooculomotor reflex movement of the person;
An intervention control unit (12b) that performs intervention control so that the person or the object is in the predetermined state when the state determination unit (12a) determines that the predetermined state is not present;
When the intervention control is performed by the intervention control unit (12b) or when the state determination unit (12a) determines that the state is in the predetermined state, the human pupil movement and head movement are measured. A measuring unit (12c) to perform,
A calculation unit (12e) for calculating the vestibulo-oculomotor reflex movement using data of the human pupil movement and head movement measured by the measurement unit (12c). Processing equipment.

(Appendix 2)
A data processing device (10);
A camera (20) for capturing an image including a person,
The measurement unit (12c) of the data processing device (10)
A monitoring system (1), which measures the pupil movement and head movement of the person using an image acquired from a camera (20).

(Appendix 3)
A data processing method for monitoring a person,
A state determination step (S3) for determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculation of the person's vestibulo-ocular reflex movement;
An intervention control step (S4) for performing an intervention control so that the person or the object is in the predetermined state when it is determined in the state determination step (S3) that the predetermined state is not present;
A measurement step (S5, S6 or S8, S9) for measuring the human pupil movement and head movement when the intervention control is performed or when it is determined in the predetermined state in the state determination step. )When,
A data processing method comprising: a step including a calculation step (S10) for calculating the vestibulo-oculomotor reflex movement using data of the human pupil movement and head movement measured in the measurement step .

(Appendix 4)
A data processing program for causing at least one computer to execute data processing for monitoring a person,
Said at least one computer,
A state determination step (S3) for determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculation of the person's vestibulo-ocular reflex movement;
An intervention control step (S4) for performing an intervention control so that the person or the object is in the predetermined state when it is determined in the state determination step (S3) that the predetermined state is not present;
A measurement step (S5, S6 or S8, S9) for measuring the human pupil movement and head movement when the intervention control is performed or when it is determined in the predetermined state in the state determination step. )When,
A data processing program for executing the calculation step (S10) for calculating the vestibular eye movement using the data of the pupil movement and head movement of the person measured in the measurement step.

1 monitoring system 2 vehicle 3 driver 10 data processing device 11 data acquisition unit 12 control unit 12a state determination unit 12b intervention control unit 12c measurement unit 12d data determination unit 12e calculation unit 12f sleepiness determination unit 12g awakening control unit 13 storage unit 13a image Storage unit 13b Acquired data storage unit 13c Measurement data storage unit 13d Program storage unit 20 Camera 30 In-vehicle sensor 31 Outside sensor 32 Acceleration sensor 33 Gyro sensor 34 Steering sensor 40 Start switch 50 Navigation device 60 Electronic control unit (ECU)
61 Suspension control unit 70 Vibrating device 80 Awakening device

Claims (14)

A data processing device that performs data processing for monitoring a person,
A state determination unit for determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculation of the person's vestibulo-ocular reflex movement;
An intervention control unit that performs intervention control so that the person or the object is in the predetermined state when it is determined by the state determination unit that the predetermined state is not present;
When the intervention control is performed by the intervention control unit, or when the state determination unit determines that the intervention control unit is in the predetermined state, the measurement unit that measures the pupil movement and head movement of the person,
A data processing apparatus comprising: a calculation unit configured to calculate the vestibulo-ocular reflex movement using data of the human pupil movement and head movement measured by the measurement unit.   The data processing apparatus according to claim 1, further comprising a drowsiness determination unit that determines the sleepiness level of the person using the vestibulo-ocular reflex movement calculated by the calculation unit.   The data processing apparatus according to claim 2, further comprising an awakening control unit that performs control for waking up the person based on the sleepiness degree determined by the sleepiness determination unit. After the intervention control is performed by the intervention control unit, the data measured by the measurement unit includes a data determination unit that determines whether or not the data is suitable for calculation of the vestibulo-oculomotor reflex movement,
The calculation unit calculates the vestibulo-ocular reflex movement using the data determined by the data determination unit as the suitable data among the data measured by the measurement unit. The data processing device according to any one of claims 1 to 3. The data determination unit weights and determines whether the data is suitable;
The data processing apparatus according to claim 4, wherein the calculation unit calculates the vestibular eye movement by taking the weighting into account. The object is a vehicle;
The data processing apparatus according to claim 1, wherein the person is a driver who drives the vehicle.   The data processing apparatus according to claim 6, wherein the predetermined state includes a state in which the vehicle travels straight.   The data processing apparatus according to claim 6, wherein the predetermined state includes a state in which the driver's head is displaced or vibrated in the vertical, horizontal, longitudinal, yaw, or pitch directions.   The data processing apparatus according to claim 6, wherein the predetermined state includes a state in which the driver gazes at a specific location. A data processing device according to any one of claims 1 to 9,
A camera for capturing an image including the person,
The measurement unit of the data processing device is
A monitoring system for measuring pupil movement and head movement of the person using the image acquired from the camera. The measurement unit is
Extracting the human face region from the image;
Extracting the positions of the corners of the eyes and the eyes from the extracted face area;
The monitoring system according to claim 10, wherein the head movement is measured from the extracted movement state of the positions of the corner of the eye and the eye. A data processing device according to claim 3;
A wake-up system comprising: a wake-up device controlled by the wake-up control unit of the data processing device. A data processing method for monitoring a person,
A state determination step of determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculating the vestibulooculomotor reflex movement of the person;
An intervention control step for performing an intervention control so that the person or the object is in the predetermined state when it is determined by the state determination step that the state is not in the predetermined state;
A measurement step of measuring the human pupil movement and head movement when the intervention control is performed or when the state determination step determines that the human body is in the predetermined state;
A data processing method, comprising: a step of calculating the vestibulo-ocular reflex movement using data of the human pupil movement and head movement measured in the measurement step. A data processing program for causing at least one computer to execute data processing for monitoring a person,
Said at least one computer,
A state determination step of determining whether or not the person or an object operated by the person is in a predetermined state suitable for calculating the vestibulooculomotor reflex movement of the person;
An intervention control step for performing an intervention control so that the person or the object is in the predetermined state when it is determined by the state determination step that the state is not in the predetermined state;
A measurement step of measuring the human pupil movement and head movement when the intervention control is performed or when the state determination step determines that the human body is in the predetermined state;
A data processing program that executes a calculation step of calculating the vestibulo-oculomotor reflex movement using the data of the human pupil movement and head movement measured in the measurement step.

Images