JP6432702B1 - Calculation system, index calculation method, and computer program - Google Patents

Abstract

An object of the present invention is to provide a calculation system capable of obtaining information about a person's arousal level with higher accuracy in a moving body.
A camera D that captures a face of a driver D in a moving body M, a gyro sensor 2 that measures motion data of the moving body M, and a face image captured by the camera 11 are used to drive the driver D. A first calculation unit 31 that calculates the eye movement of the person, and a second calculation that calculates the head movement of the person using the movement data of the moving body measured by the gyro sensor 2 and the face image taken by the camera 11. And a calculation system including the calculation unit 32.
[Selection] Figure 5

Description

The present invention relates to a calculation system , an index calculation method, and a computer program.

  In recent years, a driver's head movement or eye movement is detected from a driver's face image captured by a camera, and the driver's head movement or eye movement is detected based on the detected driver's head movement or eye movement. Techniques relating to various driving assistances or interventions for detecting a decrease in arousal level and issuing a warning or arousing driving attention are disclosed.

  For example, in Patent Document 1 below, as a wakefulness estimation device, a driver's blinking time is detected, and a blinking reference time unique to the driver at the time of awakening is increased by a predetermined ratio to evaluate a long blink. In addition to setting a blink evaluation time to detect, a long blink is detected according to the blink evaluation time, and the occurrence ratio is determined from the total number of blinks within a predetermined time and the number of long blinks, and regression using a sleepiness prediction model is performed. A technique for evaluating a driver's arousal level according to a prediction formula obtained by analysis is disclosed.

  Further, in Patent Document 2 below, as a drowsiness sign detection device, head movement is detected by head movement detection means, eye movement is detected by eye movement detection means, and an ideal eye movement angular velocity calculation unit is used to detect the head movement. The ideal eye movement angular velocity is calculated based on the movement data, the eye rotation angular velocity calculation unit calculates the eye rotation angular speed based on the eye movement data, and the drowsiness sign determination means calculates the vestibular moving eye from the ideal eye movement angular velocity and the eye rotation angular velocity. A technique is disclosed in which reflection is detected and a sign of sleepiness before a driver of a vehicle, an operator of a machine, etc. is aware of sleepiness based on this vestibulo-oculomotor reflection is disclosed.

[Problems to be solved by the invention]
In the technique disclosed in Patent Document 1 below, the driver's blink time is detected, and the driver's arousal level is evaluated from the total number of blinks within a predetermined time and the number of long blinks. For this reason, although it is possible to detect the lowered degree of arousal, it is impossible to detect a sign of a decrease in arousal, that is, a sign of sleepiness before realizing sleepiness.

  In the technique disclosed in Patent Document 2 below, a sign of sleepiness before a driver of a vehicle, an operator of a machine, etc. is aware of sleepiness is determined based on vestibulo-oculomotor reflection. Vestibulo-Ocular Reflex (VOR) is a reflective eye movement that rotates the eyeball in the opposite direction at approximately the same speed as the head movement to obtain a blur-free field of view. The decrease in the gain of the vestibulo-oculomotor reflex and the increase in the residual standard deviation are effective as an index indicating a sign of drowsiness since they occur before awareness of sleepiness. Patent Document 2 describes an experimental example using a driving simulator system that simulates driving of a car, and allows a subject to report introspection about sleepiness, gain of vestibular oculomotor reflex (VOR gain), residual standard It is described that a sign of sleepiness was determined from a change in deviation (SDres).

  However, in the device described in Patent Document 2, the subject sits on the driver seat equipped with the vibration device after the eyeball rotation imaging device, the acceleration sensor, and the gyro sensor are mounted on the head, and is displayed on the front screen. Experiments have been conducted only in a state in which the state of gazing at the tail is maintained. With an actual passenger moving body, it is not possible to attach a gyro sensor or a photographing device to the driver's head every time he gets on. Therefore, in a system that evaluates the driver's arousal level while driving an actual moving body, it is normal to attach the gyro sensor or the photographing device inside the moving body rather than attaching it to the driver's head. Become.

However, when the imaging device is attached to the inside of the moving body, only driver information in the coordinate system (moving body coordinate system) within the moving body can be obtained. The eye movement of the driver who is driving the moving body is a movement belonging to the absolute coordinate system when the driver is gazing at the front outside the vehicle. For this reason, it is difficult to accurately calculate the vestibulo-oculomotor reflex if head movement cannot be detected in the absolute coordinate system.
However, as described above, when the photographing apparatus is attached to the inside of the moving body, only driver information in the moving body coordinate system can be obtained.
For this reason, there is a problem that it is extremely difficult to detect information about awakening of a driver who is getting on and actually traveling with high accuracy.

Japanese Patent Laid-Open No. 10-272960 WO2010 / 032424

Means for solving the problems and their effects

The present invention has been made in view of the above problems, and provides a calculation system , an index calculation method, and a computer program capable of obtaining information on a person's arousal level with higher accuracy in a moving moving object. For illustrative purposes.

In order to achieve the above object, the calculation system (1) according to the present disclosure includes:
An imaging unit that is installed on a moving body and photographs a face of a person in the moving body;
A sensor installed on the moving body and measuring motion data of the moving body;
A first calculation unit that calculates an eye movement of the person using an image of a face imaged by the imaging unit;
A second calculation unit that calculates head movement of the person in the moving body using movement data of the moving body measured by the sensor and a face image captured by the imaging unit;
A third calculation unit that calculates an index related to awakening of a person in the moving body using the eye movement calculated by the first calculation unit and the head movement calculated by the second calculation unit. When,
It is characterized by having.

According to the calculation system (1), motion data of the moving object by the image of the face of the person in the mobile by the imaging unit installed in the moving body is captured, the sensor installed in the moving body Is measured, and the first calculation unit calculates the eye movement of the person based on the face image of the person. The second calculation unit calculates the head movement of the person based on the movement data of the moving body and the face image.
Therefore, the head movement considering the movement of the moving body is calculated. For this reason, even when at least the imaging unit and the sensor are arranged on a moving body and the moving body is moving , accurate information on head movement can be obtained.
With the configuration in which the imaging unit and the sensor are installed on a moving body, and further including the third calculation unit, the movement of the moving body without mounting the imaging unit or sensor on the driver's head It is possible to calculate an index relating to the awakening of the person in the moving body with high accuracy based on the information on the head movement in consideration of the above.

In addition, the calculation system ( 2 ) according to the present disclosure includes the calculation system ( 1 )
A first determination unit that determines whether or not the person is in an external gaze state in which the person is gazing at an external gaze point of the moving body, using a face image captured by the imaging unit;
The third calculation unit calculates the index using an eye movement and a head movement when it is determined that the first determination unit is in an external gaze state in which an external gaze is being observed. It is said.

The calculation system ( 2 ) performs the calculation of the awakening-related index when the person is gazing at the gaze point outside the moving body. According to the calculation system ( 2 ), a high-accuracy index relating to arousal is obtained by calculating an index relating to driver's arousal based on information on head movements and eye movements when gazing outside the vehicle. Can do. As an index related to awakening in a state of gazing outside the vehicle, for example, each value of VOR (vestibulo-oculomotor reflex) can be mentioned.

In addition, the calculation system ( 3 ) according to the present disclosure is any of the calculation system ( 1 ) and ( 2 ),
A second discriminating unit that discriminates whether or not the eye movement and the head movement are in the second state moving in the same direction using the face image taken by the imaging unit;
The third calculation unit includes the eye movement and the head in the second state among the eye movement calculated by the first calculation unit and the head movement calculated by the second determination unit. The index is calculated by excluding exercise.

  In general, when a person consciously changes the gaze direction, the movement direction of the eyeball and the movement direction of the face are the same direction. By excluding such information during exercise in the same direction, it is possible to obtain a highly accurate index relating to arousal.

Moreover, the calculation system ( 4 ) according to the present disclosure is any one of the calculation systems ( 1 ) to ( 3 ),
The wakefulness index is at least one of the gain of vestibular ocular reflex, the phase difference between the angular velocity of the head and the angular velocity of the eyeball, and the residual standard deviation of the vestibular ocular reflex, or two or more of these values It is a calculation value based on the value of.

According to the calculation system ( 4 ), by using each value related to VOR (vestibulo-oculomotor reflex) known as an index of wakefulness reduction, or an operation value based on these values, an index excellent in correlation with wakefulness is obtained. Can be calculated.

In addition, the index calculation method (1) relating to awakening according to the present disclosure includes:
A method for calculating an index related to awakening of a person in a moving body,
An imaging step of capturing an image of the face of the person in the moving body by an imaging unit installed in the moving body;
A measurement step of measuring motion data of the moving body being moved by a sensor installed on the moving body;
A first calculation step of calculating an eye movement of the person in the moving body using an image of the face imaged in the imaging step;
A second calculation step of calculating a head movement of the person in the moving body using the face image taken in the imaging step and the movement data of the moving body measured in the measurement step; ,
Third calculation step for calculating an index relating to the awakening of the person in the moving body using the eye movement calculated in the first calculation step and the head movement calculated in the second calculation step. When,
It is characterized by having.

According to the index calculation method (1) relating to awakening, head movement can be obtained regardless of the movement state of the moving body. Thereby, in the third calculation step, it is possible to more accurately calculate an index relating to the driver's arousal in the moving body.

In addition, the computer program (1) for calculating an index relating to awakening according to the present disclosure is:
A computer program for causing at least one computer to execute an index calculation process related to awakening of a person in a moving body,
Said at least one computer,
An image acquisition step of acquiring an image of the face of the person in the moving body imaged by an imaging unit installed in the moving body ;
An information acquisition step of acquiring movement data of the moving body being measured , measured by a sensor installed on the moving body ;
A first calculation step of calculating an eye movement of the person in the moving body using the face image acquired in the image acquisition step ;
A second calculation step of calculating a head movement of the person in the moving body using the face image acquired in the image acquisition step and the movement data of the moving body acquired in the information acquisition step ; ,
Third calculation step for calculating an index relating to the awakening of the person in the moving body using the eye movement calculated in the first calculation step and the head movement calculated in the second calculation step. When,
It is characterized by executing.

According to the computer program (1) for calculating the index related to awakening, the head movement can be acquired regardless of the movement state of the moving body. As a result, in the third calculation step, it is possible to cause the computer to calculate the index relating to the driver's arousal in the moving body more accurately.

It is a schematic plan view which shows the example which applied the calculation system containing the information processing apparatus which concerns on embodiment to a mobile body. It is a schematic perspective view which shows the calculation system which concerns on embodiment. It is a schematic side view which shows the structural example regarding the processing operation of an imaging part in the information processing apparatus which concerns on embodiment. It is a block diagram which shows an example of the hardware constitutions of the information processing apparatus which concerns on embodiment. It is a block diagram which shows the flow of the information processing of the calculation system which concerns on embodiment. It is a block diagram which shows the example of a data structure in the flow of information processing. It is a flowchart which shows an example of the whole calculation processing operation | movement of the VOR information of the information processing apparatus which concerns on embodiment. It is a flowchart which shows an example of the head movement calculation process in information processing. It is a flowchart which shows an example of the eye movement calculation process in the information processing apparatus which concerns on embodiment. It is a flowchart which shows an example of the calculation process of VOR information in the information processing apparatus which concerns on embodiment. It is a schematic plan view which shows the example which applied another calculation system containing the information processing apparatus which concerns on embodiment to a mobile body.

  Hereinafter, embodiments of a calculation system, an information processing apparatus, a driving support system, an index calculation method, and a computer program according to the present invention will be described with reference to the drawings.

[Application example]
FIG. 1 is a schematic plan view illustrating an example in which a calculation system including an information processing apparatus according to an embodiment is applied to a moving body M, and FIG. 2 is a schematic perspective view illustrating a calculation system mounted on the moving body M. It is.

In this example, the moving body M is a vehicle that carries a person and has a visual recognition part (for example, a windshield) that allows the outside to be seen through. The moving body M is provided with a seating portion (for example, a driver's seat) for the driver D to sit, and the driver D is seated on the seating portion of the moving moving body M via the visual recognition unit. An arbitrary gazing point P outside the moving body M can be visually recognized. The gazing point P may actually be farther from the moving body M than the illustrated position.
In the example shown in FIG. 2, the driver D sits on the seat with the upper body including the head DH facing the viewing portion, and the pupil 101 (FIG. 3) of the eyeball is the point of sight P at the tip of the viewing portion. Facing the direction.

As the gazing point P, for example, the tail portion of the vehicle ahead, or a distant road sign arranged around the road shown in FIG. The road sign shown in FIG. 1 is stationary in an absolute coordinate system outside the moving body M.
2 indicates one facial organ point of the face for calculating the orientation of the person's face. In order to calculate the face orientation, the information processing apparatus 10 (FIG. 4) prepares a face model composed of a large number of facial organ point information, and estimates the facial orientation from the positional information of a plurality of organ points. The amount of movement of the face can be calculated from the amount of change in the direction of the face.

For example, as shown in FIG. 3, the line-of-sight vector of the driver D includes the position of the pupil center 11 </ b> A in the image portion of the eyeball, and the reflection position of the irradiation light (corneal reflection image 121 in the eyeball image portion). ) And can be specified by. It is also possible to calculate the movement (eye movement) by differentiating the line-of-sight vector.
The eye movement can also be calculated from the movement of the pupil 101 in the image portion of the eyeball.
In many cases, the irradiation light is invisible light, and in the case of invisible light, the image captured by the camera 11 is an image of a person based on reflected light of the invisible light.

[Configuration example]
[Configuration example of calculation system]
As illustrated in FIG. 4, the calculation system 1 includes a camera 11 as an imaging unit that captures the face of a driver D who is a person in the moving body M, an information processing apparatus 10 that performs various types of information processing, and a moving body. A gyro sensor 2 that measures M motion data is included.
The calculation system 1 includes a camera 11, an information processing apparatus 10, and a gyro sensor 2, and can be regarded as constituting a part of a so-called drive monitoring system.
The calculation system 1 can also be expressed by a functional configuration block as shown in FIG.
A first calculation unit 31 that calculates an eye movement of the person using an image of a face photographed by the camera 11;
A second calculation unit 32 that calculates the head movement of the person relative to the absolute coordinate system using the movement data of the moving body M measured by the gyro sensor 2 and the face image captured by the camera 11;
A third calculation unit that calculates an index related to the awakening of the person in the moving body M using the eye movement calculated by the first calculation unit 31 and the head movement calculated by the second calculation unit 32. 33.

  The calculation system 1 is equipped with a camera 11 as an imaging unit that captures an image including a person's head at predetermined time intervals, and an irradiation unit 3 that irradiates the eyeball of the driver D with irradiation light. The device 10 is built in. The calculation system 1 is, for example, fixed at a position in the moving body M and facing the driver D (FIG. 1).

  The camera 11 as an imaging unit includes, for example, a lens unit (not shown), an imaging element unit, an interface unit, a control unit that controls these units, and the like. The image sensor section includes an image sensor such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS), a filter, a microlens, and the like. The image sensor section includes a sensor that can form a captured image by receiving light in the visible region, and can also include a sensor such as a CCD, CMOS, or photodiode that can form a captured image by receiving ultraviolet light, infrared light, or near infrared light. Good.

  The camera 11 photographs the driver D. The head movement of the driver D in the moving body M can be measured by the image taken by the camera 11. Further, the image taken by the camera 11 has a resolution that can specify the direction of the eyeball and line of sight of the person in the image, and the eyeball movement of the driver D can be calculated.

The number of cameras 11 may be one as shown in FIG. 2, and in a modified configuration example described later, the camera 11 is constituted by two or more cameras including the auxiliary camera 4 as shown in FIG. 11. The camera 11 may be a monocular camera or a stereo camera.
The camera 11 captures an image at a predetermined frame rate (for example, 30 to 60 frames per second), and data of the image captured by the camera 11 is output to the information processing apparatus 10.

The gyro sensor 2 acquires motion information associated with the movement of the moving body M. For example, the gyro sensor 2 shown in FIG. 2 is a three-dimensional gyro sensor, and is fixed to the calculation system 1 fixed to the front part of the moving body M (FIG. 1). In this example, the measured motion information is the rotational momentum of the moving body M with respect to the external absolute coordinate system. As the contents of the motion information, the rotation amount around each axis of the XYZ axes that are three-dimensionally orthogonal to each other, and the rotation angular velocities of the roll, pitch, and yaw per unit time are acquired.
The gyro sensor 2 may be installed on the moving body M without unnecessarily swinging, and the installation position on the moving body M is not particularly limited.

The irradiation unit 3 includes a light emitting element such as an LED (Light Emitting Diode), and an infrared LED, a near infrared LED, or the like is employed so that the state of the driver can be photographed regardless of day or night.
For example, as illustrated in FIG. 2, the irradiating unit 3 irradiates irradiation light from a predetermined position in the moving body M toward a driver seat where a person is present. By irradiating the irradiation light in a certain direction, a point on the eyeball in a certain direction is reflected as a cornea reflection image 121. The cornea reflection image 121 is formed at a point where the distance from the fixed position of the irradiation unit 3 to the human eyeball is the shortest distance. Moreover, when irradiation light is a non-visible light ray, it does not disturb the movement of a person's eyes | visual_axis. An image photographed by the camera 11 is an image including a corneal reflection image 121.

[Configuration example of information processing device]
FIG. 4 is a block diagram illustrating a functional configuration example of the calculation system 1 including the information processing apparatus 10 according to the embodiment.
The information processing apparatus 10 includes an input / output interface (I / F) 110, a control unit 12, and a storage unit 13.

  For example, the camera 11, the gyro sensor 2, the irradiation unit 3, the operation control device 30, the notification device 37, and the like are connected to the input / output I / F 110, and the input / output I / F 110 receives signals from these external devices. It is configured to include an interface circuit, a connection connector, etc. for performing exchanges. The operation control device 30 can control the moving body M based on various information supplied from the information processing device 10. For example, the operation control device 30 may control the operation of the moving body M, or may control the output from a speaker or a display provided on the moving body M.

  The control unit 12 includes an image acquisition unit 12a, a detection unit 12b, an information acquisition unit 12c, a first determination unit 12d, a second determination unit 12e, and a reference determination unit 12f, and further includes a calculation unit 12g and a processing unit. 12h, the reference changing unit 12i, and the like may be included. The control unit 12 includes, for example, a CPU (Central Processing Unit), a memory, an image processing circuit, and the like. By reading a predetermined control program stored in the storage unit 13 and the like, the CPU interprets and executes it. That work is realized. The control unit 12 controls the imaging device unit and the irradiation unit 3 to irradiate light (for example, near infrared rays) from the irradiation unit 3 and controls the imaging device unit to photograph the reflected light. The control unit 12 includes one or more hardware processors such as a CPU and a graphics processing unit (GPU).

  The storage unit 13 includes an image storage unit 13a, a face orientation storage unit 13b, a parameter storage unit 13c, a reference storage unit 13d, and a program storage unit 13e. The storage unit 13 stores data by a semiconductor element such as a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), a flash memory, and other nonvolatile memory or volatile memory. It is comprised with one or more memory | storage devices which can memorize | store. Note that the control unit 12 may include a RAM and a ROM.

The image of the driver D acquired from the camera 11 by the image acquisition unit 12a is stored in the image storage unit 13a.
In the face direction storage unit 13b, information on the face direction and movement amount of the driver D of each image detected by the detection unit 12b is stored in association with each image stored in the image storage unit 13a. .
The parameter storage unit 13c stores various parameters used for determination by the second determination unit 12e, such as a face direction threshold value and a movement amount threshold value.
The reference storage unit 13d stores information related to the reference of the face direction of the driver D determined by the reference determination unit 12f, for example, a value indicating the reference (reference value).
The program storage unit 13e stores an information processing program executed by each unit of the control unit 12, data necessary for executing the program, and the like.

[Operation example]
The information processing apparatus 10 is built in the calculation system 1 mounted on the moving body M, and can acquire various types of information related to the traveling state of the moving body M from an in-vehicle device (not shown). Is configured to output.

  As one of information processing for grasping the state of the driver D in the moving body M, the information processing apparatus 10 calculates a face model composed of a large number of facial organ point information in order to calculate the face orientation. Prepare the face direction from the position information of multiple organ points. The amount of movement of the face can be calculated from the amount of change in the direction of the face. Further, the information processing apparatus 10 as one of information processing for grasping the state of the driver D in the moving body M is an eyeball on at least one side of the driver D that serves as a reference for the direction of the eyeball of the driver D. In addition, a process of grasping the pupil included in the eyeball and grasping the amount of movement of the pupil in the eyeball is also executed, and the eye movement can be detected.

  For example, as shown in FIG. 2, the head movement of the driver D includes a pitch angle that is an angle (vertical direction) around the X axis (left and right axis) of the driver D, a Y of the face, At least one of the yaw (Yaw) angle that is an angle around the axis (vertical axis) (left-right orientation) and the roll angle that is an angle (left-right tilt) around the Z axis (front-rear axis) of the face is include.

  The motion information of the moving body M may include, for example, vehicle speed information or steering information acquired from in-vehicle devices connected by a CAN (Controller Area Network) in addition to the speed information of each three-dimensional axis of the moving body M. Good. When the exercise information includes information acquired from the in-vehicle device, it is possible to determine whether the vehicle is in a specific running state based on the acquired information. Examples of the specific traveling state include a traveling state in which the vehicle speed of the moving body M is in a predetermined vehicle speed range and is in a non-steering state, in other words, a traveling state in which the moving body M is traveling straight. Such a running state can be said to be a state suitable for calculating an index related to the awakening of the driver D.

  As the “head movement calculation process”, the information processing apparatus 10 detects the head DH of the driver D based on the image taken by the camera 11 and further uses the head movement reference of the head DH. A process for detecting a certain facial organ point DO is executed.

As the facial organ point DO information, for example, a value obtained by combining the positional information of the facial organ point DO and the orientation information of the facial organ point DO is used.
Among these, as the position information of the facial organ point DO, for example, a combination of values of the width position Xd, the height position Yd, and the front and rear position Zd indicating the three-dimensional axial position of the face of the driver D is used. Further, the orientation information of the facial organ point DO includes, for example, a pitch Pd that is a rotation angle around the X axis among three-dimensional axes, a yaw Yd that is a rotation angle around the Y axis, and a rotation angle around the Z axis. A combination of values consisting of the roll Rd is used. These are information indicating the position of the head of the driver D and information indicating the head movement of the driver D.

  A combination of at least one of the width position Xd, height position Yd, front and rear position Zd, pitch Pd, yaw Yd, and roll Rd from one or more images taken by the camera 11, and an average value thereof Alternatively, a combination of values such as the mode value can be used as movement information of the facial organ point DO.

  In the present embodiment, the information processing apparatus 10 acquires an image captured by the camera 11, calculates the head movement of the driver D from the acquired image, and acquires movement information of the moving body M. And the 1st determination part 12d of the information processing apparatus 10 determines whether the driver | operator D exists in a specific state (henceforth a "1st state") based on the acquired said information (FIG. 7). Specifically, it can be said that the first state is a state in which the driver D is gazing outside the moving body M (here, outside the vehicle).

  The first determination unit 12d of the information processing device 10 corrects the amount of change in the head movement of the driver D in the moving body M according to the movement information of the moving body M itself, and then sets a preset threshold value. It is good also as what discriminate | determines whether it exceeds.

  The control unit 12 performs processing for storing various data in the storage unit 13. The control unit 12 reads various data and various programs stored in the storage unit 13 and executes these programs. The control unit 12 cooperates with the storage unit 13 to operate the image acquisition unit 12a, the detection unit 12b, the information acquisition unit 12c, the first determination unit 12d, the second determination unit 12e, and the reference determination unit 12f, The operations of the calculation unit 12g, the processing unit 12h, and the reference changing unit 12i are realized.

  The image acquisition unit 12a performs a process of acquiring an image of the driver D photographed at a predetermined frame rate from the camera 11, and performs a process of storing the image acquired from the camera 11 in the image storage unit 13a.

  The detection unit 12b reads the image stored in the image storage unit 13a every frame or every frame at a predetermined interval, performs a process of detecting the head DH of the driver D from these images, and detects the detected driver D The movement information of the head DH is stored in the face direction storage unit 13b in association with the image.

  The movement information of the head DH of the driver D includes an angle indicating the direction of the face of the driver D detected by the image processing, for example, at least one of the above yaw angle, pitch angle, and roll angle. Contains information. Further, the movement information of the head D of the driver D includes position information related to the facial organ points DO such as the eyes, the base of the nose, the tip of the nose, the mouth, the eyebrows, and the top of the head. For example, the facial organ point DO is identified by image processing as a feature point indicating a representative or characteristic position of each organ of the face, and is detected as a reference point for head movement.

The information acquisition unit 12c performs a process of acquiring exercise information of the moving body M via the operation control device 30, and outputs the acquired information to the first determination unit 12d. The process of the information acquisition unit 12c and the image acquisition unit 12a is not necessarily limited. For example, each of these treatments may be performed in parallel. In addition, you may make it the information acquisition part 12c acquire the said information from each part of a vehicle-mounted system (not shown) not via the operation control apparatus 30. FIG.
The motion information of the moving body M includes at least the rotational angular velocity detected by the gyro sensor 2.
The information acquisition unit 12c may acquire acceleration / deceleration information of the moving body M, inclination information of the moving body M, and the like.

  Further, the information acquisition unit 12c may acquire the position information of the moving body M and the map information around the moving body M. These pieces of information include, for example, the vehicle position determined by the navigation device and map information around the vehicle position.

  Furthermore, the information acquisition unit 12c may acquire information related to a monitoring object such as another moving body or a person existing around the moving body M, particularly in the traveling direction. Such information includes, for example, information on the type of object and the distance to the object detected by the periphery monitoring sensor.

  The first determination unit 12d determines whether or not the condition of the first state related to the driver D is satisfied. The first determination unit 12d uses the face image captured by the camera 11 to determine whether or not the first state is based on the line-of-sight vector. The determination result is output as a binary signal from the first determination unit 12d to the second determination unit 12e, for example. The first determination unit 12d is an example of a first determination unit according to the present disclosure.

  The second determination unit 12e determines whether or not the current state is not suitable for calculating VOR information described later. In the present embodiment, as an example, a state where the movement direction of the eyeball and the movement direction of the face are the same direction is referred to as a state unsuitable for calculation of VOR information, and this is also referred to as a “second state”. For example, the second determination unit 12e uses the face image captured by the camera 11 to determine whether or not the eye movement direction and the face movement direction are the same direction, thereby determining whether or not the second state is established. Can be determined. The determination result is output as a binary signal from the second determination unit 12e to the reference determination unit 12f, for example. The second determination unit 12e is an example of a second determination unit according to the present disclosure.

  The reference determination unit 12f analyzes the image element ID 20 (FIG. 6) including the head DH of the driver D when the driver D is determined to be in the first state and the second state. Further, the image element ID 10 (FIG. 6) of the eyeball included in the head DH of the driver D is extracted by image analysis. Based on the image element ID 10 of the eyeball, processing for specifying the position and orientation of the pupil 101 is performed. Then, based on the change in the positional information and orientation information of the facial organ point DO of each image taken continuously, the process of determining the “eye movement” of the driver D is performed, and the determined head movement is used as the reference storage unit. Store in 13d.

  Specifically, first, the change in the direction of the face of the driver D and the direction of the line of sight determined by the second determination unit 12e is not the same direction, “a non-codirectional movement state (that is, the second state)” It is determined whether or not a state that can be estimated to be present continues for a predetermined period or longer. That is, the second determination unit 12e gives a determination result indicating that the vehicle is in the “non-coaxial motion state” for each frame of the captured image, and whether or not the determination result has continued for a predetermined number of frames or not. Is determined.

  If it is determined that the image has been continuously acquired for a predetermined period or longer, the reference determination unit 12f then displays the eyeball of the driver D corresponding to the image acquired in the predetermined period from the face orientation storage unit 13b (the embodiment of FIG. 3). Then, the image element ID 10 of the left eye that becomes the facial organ point DO) is read out, and the eye movement of the driver D is determined from the image element ID 10 of the eyeball of each frame read out continuously or intermittently.

  Since the eye movement is information related to the position of the pupil 101 in the eyeball, for example, it can be obtained by specifying the movement of the pupil 101 in the image element of the eyeball based on the image element of the eyeball photographed by the camera 11. it can. By specifying the change in the position information of the pupil 101, the eye movement of the driver D is determined.

The above-described detection unit 12b, information acquisition unit 12c, first determination unit 12d, second determination unit 12e, reference determination unit 12f, and face orientation storage unit 13b cooperate to provide information on VOR (hereinafter "VOR information"). The calculation process is executed.
On the other hand, the calculation unit 12g, the processing unit 12h, and the reference storage unit 13d cooperate to execute processing for grasping (monitoring) the state of the driver D.

  The calculation unit 12g reads an image including the head from the face orientation storage unit 13b determined by the reference determination unit 12f, reads out the eye movement from the reference storage unit 13d, and calculates VOR information from the head movement and the eye movement. And the calculation result is output to the processing unit 12h.

The processing unit 12h performs predetermined processing based on the face direction (for example, deviation from the reference value) of the driver D with respect to the reference of the face direction calculated by the calculation unit 12g.
For example, the predetermined process is performed by determining whether or not the user is in the side-by-side state. When the side-by-side state is in the side-by-side state (for example, when the face direction in the left-right direction is shifted from the reference value by a predetermined value or more) 37 may be a process of instructing the notification, or may be a process of storing information on the orientation of the face in the looking-aside state in the storage unit 13 or a process of outputting to the driving control device 30 without performing the notification process.

  Further, the processing unit 12h performs a determination process as to whether or not a decrease in the arousal level of the driver D is predicted based on the calculation result of the calculation process of the VOR information, and when it is determined that a decrease in the arousal level is predicted Alternatively, a process of outputting a detection signal to the notification device 37 and instructing notification regarding the awakening level reduction into the moving body M may be performed.

Further, the processing unit 12h may perform a process of storing the VOR information in the storage unit 13 when the driver D's arousal level is lowered, instead of the notification process or together with the notification process.
In addition, the processing unit 12h performs a determination process of the moving state of the moving body M, and detects when the moving body M is clearly stopped or when the moving body M is clearly moving right or left. The notification process or the storage process to the storage unit 13 may be performed only when the above is not the case.

  In addition, the predetermined process is a state in which it is possible to take over by determining whether or not the driver D is ready to take over to manual operation when switching from the automatic operation mode to the manual operation mode. (For example, when the deviation of the face direction from the reference value is within a predetermined range suitable for manual operation), the operation control device 30 is provided with a signal permitting switching to manual operation. The process to output may be sufficient.

The reference changing unit 12i performs a process of changing the reference of the face orientation determined by the reference determining unit 12f, for example, a reference value, and stores the changed reference value in the reference storage unit 13d.
For example, the difference between the face direction of the driver D calculated by the calculation unit 12g and the reference of the face direction when a specific driving state (for example, a straight driving state) is within a predetermined range. The difference is reduced when the face is in a certain state (for example, the face is deviated in a certain direction from the reference value, but is not deviated as much as looking aside) for a certain period or intermittently. The reference of the face orientation may be changed, for example, corrected. In addition, the process performed by the reference changing unit 12i includes a process of changing the reference of the face orientation when the driver D changes.

  In the calculation system according to the embodiment, when the driver D is seated in the driver's seat and is in a state of gazing at the external gaze point P, the driver D is based on the driver's D eye movement and head movement. D's VOR information is calculated.

  FIG. 7 is a flowchart illustrating an example of the entire VOR information calculation processing operation performed by the control unit 12 of the information processing apparatus 10 according to the embodiment.

First, in step S1, the control unit 12 performs a process of acquiring an image captured by the camera 11, and then in step S2, the image element ID 20 including the head of the driver D is extracted from the acquired image. After detecting the facial organ points included in the image element, a process of calculating the head movement of the driver D in the moving body M coordinate system is performed. The camera 11 captures a predetermined number of frames per second.
The control unit 12 captures these captured images in time series, and executes this process every frame or every frame at a predetermined interval.
Next, in step S <b> 3, the control unit 12 extracts the image element ID 10 including the eyeball of the driver D from the acquired image, and then performs a process of calculating eye movement.

Next, in step S4, the vehicle motion is calculated based on the information obtained by the gyro sensor 2, and then the process proceeds to step S5.
In step S5, in order to obtain the head movement of the driver D with respect to the absolute coordinate system, the moving body M calculated in step S4 is set to the value of the head movement of the driver D in the vehicle coordinate system calculated in step S2. A process of adding vehicle motion values in the absolute coordinate system is performed. For example, the addition process is a process of adding a value indicating the head movement at a certain time to a value indicating the vehicle movement at the time.
Next, in step S6, the first determination unit 12d determines whether or not the driver D is in the first state. If it is determined that the driver D is in the first state, the process proceeds to step S7. If it is determined that the current state is not in the first state, the present process is terminated.
Next, in step S7, processing for calculating VOR information of the driver D based on the head movement and eye movement converted to the absolute coordinate system when the driver D is determined to be in the first state. Then, the process proceeds to step S8.
In step S8, processing for storing and storing the VOR information in the storage unit 13 is performed. At the same time, data older than 5 seconds is erased, and rewriting to VOR information within the latest 5 seconds is performed.

Thereafter, in step S9, it is determined whether or not data for 5 seconds has been stored. If it is determined that data has been stored, the process proceeds to step S10. If it is not determined that data has been stored, this process is terminated.
In step S10, a process of excluding data for which the correlation between the head movement and the eye movement is determined to be “in the same direction movement state” in the second determination unit 12e is performed, and then the process proceeds to step S11. In S11, an output process of only the VOR information determined as “non-co-directional movement state” is performed, and then this process ends.

FIG. 8 is a flowchart showing details of the head movement calculation process (the process of step S2 in the flowchart shown in FIG. 7) performed by the control unit 12 of the information processing apparatus 10 according to the embodiment.
First, in step S21, the control unit 12 performs a process of acquiring an image photographed by the camera 11. Next, in step S22, whether or not the head DH of the driver D can be detected based on the acquired image. If it is determined that the head DH of the driver D can be detected, the process proceeds to step S23. On the other hand, if it is determined that the head DH of the driver D cannot be detected, the process ends.
In step S23, it is determined whether or not the facial organ point DO can be detected from the head DH of the driver D based on the acquired image. If it is determined that the facial organ point DO can be detected, the process proceeds to step S24. If it is determined that the facial organ point DO cannot be detected, the process is terminated.

In step S24, the “face momentum”, which is the amount of movement of the position of the detected facial organ point, is calculated as the momentum in the vehicle coordinate system of the moving body M (that is, the relative facial momentum with respect to the moving body M).
Next, the process proceeds to step S25. In step S25, the head movement in the vehicle coordinate system is calculated based on the movement amount of the facial organ point position, and then the process proceeds to step S26.
In step S26, the process which memorize | stores the information which shows the calculated head movement in the memory | storage unit 13 is performed, and this process is complete | finished after that.

FIG. 9 is a flowchart showing details of the eye movement calculation process (the process of step S3 in the flowchart shown in FIG. 7) performed by the control unit 12 of the information processing apparatus 10 according to the embodiment.
First, in step S31, the control unit 12 performs a process of acquiring an image captured by the camera 11, and then proceeds to step S32. In step S32, the head D of the driver D is based on the acquired image. If it is determined whether or not the head DH of the driver D can be detected, the process proceeds to step S33. If it is determined that the head DH of the driver D cannot be detected, the process ends. .

In step S33, it is determined whether or not the eyeball DO can be detected from the head DH of the driver D based on the acquired image. If it is determined that the eyeball DO can be detected, the process proceeds to step S34. If it is determined that it cannot be detected, this process is terminated.
In step S34, it is determined whether or not the pupil 101 can be detected from the eyeball DO of the driver D. If it is determined that the pupil 101 can be detected, the process proceeds to step S35, and if it is determined that the pupil 101 cannot be detected, The process ends.
In step S35, the amount of movement of the pupil is calculated, and then the process proceeds to step S36. In step S36, the amount of movement of the eyeball is calculated based on the amount of movement of the pupil, and then the process proceeds to step S37. Based on the movement amount of the eyeball, eye movement is calculated.
Next, the process proceeds to step S38, and in step S38, a process of storing the calculated eye movement in the storage unit 13 is performed, and then this process ends.

FIG. 10 is a flowchart showing details of the VOR information calculation process (step S7 in the flowchart shown in FIG. 7) performed by the control unit 12 of the information processing apparatus 10 according to the embodiment.
First, in step S71, a gain (VOR gain) of vestibular eye movement reflection is calculated.
Next, in step S72, the phase difference between the head angular velocity and the eyeball angular velocity is calculated.
Next, in step S73, a residual standard deviation is calculated based on the VOR gain and the phase difference.
Next, in step S74, a process for storing these VOR information in the storage unit 13 is performed, and then this process is terminated.

[Other application examples]
FIG. 11 shows another application example of the calculation system including the information processing apparatus according to the embodiment. This application example is provided with two cameras as an imaging unit, and the irradiation unit 3 applies an invisible light image to the driver D's eyeball as well as to the driver D's back, and the application example shown in FIG. Is different.

  In the application example illustrated in FIG. 11, the imaging unit includes two cameras including the camera 11 and the auxiliary camera 4 at different positions. By acquiring images including the head of the driver D from two different positions, it is possible to reliably obtain a change in the value in the depth direction of the captured image.

  Further, in the application example shown in FIG. 11, a time display image of invisible light whose display changes in synchronization with the irradiation time is irradiated on the back of the driver D. The time display image is composed of, for example, an image of a clock whose time changes. By including the time display image in the image captured by the camera, it is possible to reliably perform synchronization between the captured image and the capture time of each image. In addition, the time display image should just be an image which can identify time. For example, the time display image is not limited to an image such as a clock that allows a person to recognize the time, and may be an image encoded such as a barcode or a two-dimensional code.

  As mentioned above, although embodiment of this invention was described in detail, the said description is only the illustration of this invention in all the points. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention.

  Although the case where the calculation system is applied to a vehicle has been described in the above-described embodiment, the calculation system can be applied to other mobile bodies other than the vehicle, for example, a ship, a diving machine, and an aircraft.

[Appendix]
Embodiments of the present invention may be described as in the following supplementary notes, but are not limited thereto.
[Appendix 1]
A first calculation unit (31) that calculates an eye movement of a person using an image of a face photographed by a camera (11) installed on a moving body;
Motion data of the mobile body measured by a sensor (2) installed on the mobile body, and
A second calculation unit (32) for calculating a head movement of the person using an image of a face photographed by the camera (11);
An information processing apparatus comprising:

[Appendix 2]
A method for calculating an index relating to awakening of a person in a moving body M,
An imaging step (S1) for capturing an image of a person's face in the moving body M with a camera (11) fixed to the moving body M;
A measurement step (S4) of measuring motion data of the moving moving body M by the gyro sensor (2) installed on the moving body M;
A first calculation step (S3) for calculating eye movements of a person in the moving body M using the face image captured in the imaging step (S1);
Using the face image photographed in the imaging step (S1) and the motion data of the moving body M measured in the measuring step (S4), the head motion of the person in the moving body M is a value in the absolute coordinate system. A second calculation step (S5) to calculate as
Third calculation for calculating an index relating to the awakening of the person in the moving body M using the eye movement calculated in the first calculation step (S3) and the head movement calculated in the second calculation step. Step (S7);
An index calculation method characterized by comprising:

[Appendix 3]
An information processing apparatus (10) and a control means for controlling the moving body using the eye movement calculated by the first calculation unit and the head movement calculated by the second calculation unit. A driving support system characterized by comprising.

[Appendix 4]
A computer program for causing at least one computer to execute an index calculation process related to awakening of a person in a moving body,
Said at least one computer,
An imaging step (S1) for capturing an image of a person's face in the moving body M with a camera (11) fixed to the moving body M;
A measurement step (S4) of measuring motion data of the moving moving body M by the gyro sensor (2) installed on the moving body M;
A first calculation step (S3) for calculating eye movements of a person in the moving body M using the face image captured in the imaging step (S1);
Using the face image photographed in the imaging step (S1) and the motion data of the moving body M measured in the measuring step (S4), the head motion of the person in the moving body M is a value in the absolute coordinate system. A second calculation step (S5) to calculate as
Third calculation for calculating an index relating to the awakening of the person in the moving body M using the eye movement calculated in the first calculation step (S3) and the head movement calculated in the second calculation step. Step (S7);
A computer program for calculating an index relating to awakening, characterized in that

DESCRIPTION OF SYMBOLS 1 Calculation system 2 Gyro sensor 3 Irradiation part 4 Auxiliary camera 10 Information processing apparatus 11 Camera 11A Pupil center 12 Control unit 13 Storage unit 30 Operation control apparatus 31 1st calculation part 32 2nd calculation part 33 3rd calculation part 37 Notification device 101 Pupil 110 Input / output interface 121 Cornea reflection image M Moving object D Driver DO Facial organ point XYZ Three-dimensional axis in absolute coordinate system

Claims (6)

An imaging unit that is installed on a moving body and photographs a face of a person in the moving body;
A sensor installed on the moving body and measuring motion data of the moving body;
A first calculation unit that calculates an eye movement of the person using an image of a face imaged by the imaging unit;
A second calculation unit that calculates head movement of the person in the moving body using movement data of the moving body measured by the sensor and a face image captured by the imaging unit;
A third calculation unit that calculates an index related to awakening of a person in the moving body using the eye movement calculated by the first calculation unit and the head movement calculated by the second calculation unit. When,
A calculation system comprising: A first determination unit that determines whether or not the person is in a first state of gazing at an external gazing point of the moving body, using a face image captured by the imaging unit;
Wherein the third calculating unit, using the eye movement and head movement when the it is determined that the first state by the first determination unit, according to claim 1, wherein the calculating the index The calculation system described. A second discriminating unit that discriminates whether or not the eyeball and the head are moving in the same direction using the face image taken by the imaging unit;
The third calculation unit includes the eye movement and the head in the second state among the eye movement calculated by the first calculation unit and the head movement calculated by the second calculation unit. movement to the exclusion of, claim 1 or claim 2, wherein calculating system and calculates the index. The wakefulness index is at least one of the gain of vestibular ocular reflex, the phase difference between the angular velocity of the head and the angular velocity of the eyeball, and the residual standard deviation of the vestibular ocular reflex, or two or more of these values The calculation system according to any one of claims 1 to 3 , wherein the calculation system is an operation value based on the value of. A method for calculating an index related to awakening of a person in a moving body,
An imaging step of capturing an image of the face of the person in the moving body by an imaging unit installed in the moving body;
A measurement step of measuring motion data of the moving body being moved by a sensor installed on the moving body;
A first calculation step of calculating an eye movement of the person in the moving body using an image of the face imaged in the imaging step;
A second calculation step of calculating a head movement of the person in the moving body using the face image taken in the imaging step and the movement data of the moving body measured in the measurement step; ,
Third calculation step for calculating an index relating to the awakening of the person in the moving body using the eye movement calculated in the first calculation step and the head movement calculated in the second calculation step. When,
An index calculation method relating to awakening, characterized by comprising: A computer program for causing at least one computer to execute an index calculation process related to awakening of a person in a moving body,
Said at least one computer,
An image acquisition step of acquiring an image of the face of the person in the moving body imaged by an imaging unit installed in the moving body;
An information acquisition step of acquiring movement data of the moving body being measured , measured by a sensor installed on the moving body;
A first calculation step of calculating an eye movement of the person in the moving body using the face image acquired in the image acquisition step ;
A second calculation step of calculating a head movement of the person in the moving body using the face image acquired in the image acquisition step and the movement data of the moving body acquired in the information acquisition step ; ,
Third calculation step for calculating an index relating to the awakening of the person in the moving body using the eye movement calculated in the first calculation step and the head movement calculated in the second calculation step. When,
A computer program for calculating an index relating to awakening, characterized in that

Images