JP2012085746A - Attentional state determination system, method, computer program, and attentional state determination device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determinate the attentional state of a user with enough accuracy even when there are a lot of situation changes by shortening an analysis time interval of eye fixation-related potential (EFRP).SOLUTION: An attentional state determination system 1 includes: a brain wave measurement section 20 for measuring a brain wave signal; an eye movement measuring section 30 for measuring the movement of an eye ball; a saccade detector 40 for detecting a plurality of eye ball stopping start times that are when the saccade of the eye ball ends and a plurality of movement amounts of the saccade by using the movement of the eye ball; a classification section 60 for specifying a direction of each saccade based on the movement amount of each saccade, and classifying that the direction of the each specified saccade, that adds and averages the eye ball stopping potential obtained from the brain wave signal for each classified direction, assuming the eye ball stopping start time corresponding to each saccade as the origin; an attention amount determining section 70 for determining an attentional state of each classified direction based on the eye fixation-related potential which has been added and averaged; and an integrally determination section 80 for determining the attentional state by using the determination results in the attentional state of each direction.

Description

  The present invention relates to an apparatus for determining an attention state of an operator who performs an operation such as driving a car based on an electroencephalogram. Specifically, the present invention relates to an apparatus and a method for determining a state of caution for each situation in a driving operation of an automobile whose environment changes from moment to moment, and a computer program executed in such an apparatus.

  In recent years, there is an increasing need for a technology that supports safe driving that can provide assistance in accordance with a driver's state after grasping the physical state and psychological state of the driver in real time. As a method for objectively and quantitatively evaluating the state of the driver, attempts have been made to quantify the degree of arousal using physiological indices such as brain waves and blinks.

  For example, Patent Document 1 discloses a technique for estimating the arousal level from a sleep waveform pattern of an electroencephalogram or an α wave component. Patent Document 2 discloses a technique for capturing a driver's face, detecting blinking from the obtained image, and estimating a decrease in arousal level from the eye-closing time of blinking.

  In contrast to these, the inventors of the present application do not simply capture the state of the driver while driving, but a so-called distracted state in which attention is not directed to driving despite being awake. It is necessary to treat it as a driving caution state that includes For this purpose, a technique for measuring / evaluating the attention state for driving itself is required instead of the conventional detection of drowsiness based on the arousal level.

  In addition, there is a method of determining the driver's attention state by detecting the driver's line of sight and the movement of the face with a camera directed to the driver. For example, in Patent Document 3, the optimal gaze position that the driver should be aware of is determined from the surrounding situation of the driver's own vehicle, and is determined as the gaze point detected from the driver's line of sight and facial movement. A technique for determining a state of attention distribution with respect to driving by comparing with an optimal gaze position to be noted is disclosed.

  If such an index focusing on the driver's gaze position or the like is used, it is possible to evaluate distraction due to not looking at the position where the driver should be careful. However, it is not possible to capture a so-called state of consciousness aside that attention is not directed to driving even though the driver is looking forward. Here, the state of consciousness aside indicates, for example, a state in which other things are being considered, or attention is focused on music or conversation and consciousness is not concentrated on driving.

  On the other hand, as a state of caution for driving including a state of consciousness aside, research is being conducted to examine how much attention is being paid to a subject being watched. Specifically, research using an electroencephalogram related potential (EFRP) of an electroencephalogram has been performed. “Eye-holding related potential” means the end time of rapid eye movement (saccade) when a person is working or is looking at things freely, that is, the brain's occurrence related to the start time of eye-holding. A transient potential change. The positive potential component that appears predominantly in the back of the head about 100 milliseconds from the eye stop time is called the “lambda reaction”, and changes depending on the degree of attention concentration on the subject that the person is watching. It has been known.

  For example, in Patent Document 4, the end time of a saccade is detected based on an eye movement signal, and each time the saccade ends, an EFRP is calculated by averaging the brain waves within a certain period from that time, and the amount of attention is calculated. A technique for discriminating is disclosed.

JP-A-7-108848 Japanese Patent No. 3127760 JP 2004-178367 A JP 2002-272893 A

  However, in the conventional technique described in Patent Document 4, it is necessary to add a sufficient number of eyeball retention-related potential (EFRP) waveforms in order to remove noise. Many EFRP waveforms need to be acquired from long-term electroencephalogram data. In this case, in an urban area where the environment changes from moment to moment, attention amount matching according to the situation cannot be performed. Hereinafter, the above-described problem will be described in detail.

  In order to evaluate the length of the analysis section required in the conventional method using the lambda reaction of EFRP, the present inventors conducted an attention amount discrimination experiment for 17 student experiment participants (experimental experiment). Details and a method for calculating the discrimination rate will be described later). The attention state is determined by comparing the amplitude of the lambda response (maximum value included in the range from 50 milliseconds to 150 milliseconds) with a threshold value, which is a feature quantity of EFRP, either “concentration state” or “diffuse state”. This was done by determining whether it was in a state.

  The EFRP for discrimination was calculated by the method used in Patent Document 4. Specifically, EFRP included in a certain period (analysis section) was extracted, and those waveforms were averaged to calculate one EFRP waveform.

  As a result of calculating the discrimination rate using the discrimination method, the discrimination rate is 72.3% when the analysis interval is 60 seconds, and the discrimination rate is 79.6% when the analysis interval is 120 seconds. Met.

  If the analysis interval is lengthened in this way, the discrimination accuracy can be improved. This is because if the analysis interval is lengthened, the number of EFRPs included in the analysis interval increases and the number of additions increases. As the number of additions increases, the background electroencephalogram and random noise components are canceled out, and only the lambda response of EFRP is conspicuously extracted, thereby improving the discrimination accuracy.

  Therefore, in the conventional method, in order to increase the accuracy of attention state determination to 80% or more, it is necessary to set the analysis interval to 120 seconds or more in order to further increase the number of EFRPs to be averaged.

  If you drive in the city for 120 seconds by car, it is likely to go through multiple intersections. At intersections, actions such as turning left and right, stopping, etc. are performed, and the driver's forward scenery changes greatly at each intersection. It is considered that the driver's attention state changes from moment to moment according to the situation. In the analysis interval every 120 seconds, the attention state between a plurality of intersections is collectively determined, so the attention state according to the situation cannot be determined.

  The present invention has been made in view of the above problems, and its purpose is to shorten the analysis period of the eyeball-related potential (EFRP), which requires minutes, and is sufficient even when there are many changes in conditions such as urban areas. It is to discriminate the driving attention state with accuracy.

  An attention state determination system according to the present invention includes an electroencephalogram measurement unit that measures a user's electroencephalogram signal, an eye movement measurement unit that measures the movement of the user's eyeball, and the eyeball saccade using the movement of the user's eyeball. A saccade detection unit that detects a plurality of movements of the saccade and a saccade detection unit that detects a plurality of movements of the saccade, and identifies the direction of each saccade based on the movement of each saccade, and classifies the direction of each identified saccade A classification unit that adds and averages, for each of the classified directions, an eyeball stationary potential extracted from the user's brain wave signal starting from the eyeball retention start time corresponding to each saccade. An attention amount determination unit that determines an attention state for each classified direction based on the eyeball stop-related potential, and a determination result of the attention state for each direction Used, and a integration identifying unit for discriminating the attentional state of the user.

  The attention amount determination unit determines, for each of the directions, whether it is a concentrated state or a distraction state, and the integrated determination unit determines whether the attention state and the distraction state are determined by the attention amount determination unit. Any of the determined states may be determined as the user's attention state.

  The integrated determination unit may determine the user's attention state by determining a large number of determination results for each direction by the attention amount determination unit.

  The saccade detection unit may detect a time point when the amount of movement of the user's eyeball becomes smaller than a predetermined threshold as the eyeball retention start time.

  The attention amount discriminating unit compares the amplitude value of the lambda response of the eyeball retention-related potential averaged for each direction with a predetermined threshold value to determine the attention state for each classified direction. It may be determined.

  The attention amount determination unit may use, as an amplitude value of a lambda reaction, a local maximum value included in 100 ± 100 milliseconds of the eyeball retention-related potential that has been averaged starting from the eyeball retention start time.

  The integrated determination unit may determine the user's attention state at regular intervals.

  The attention state determination system further includes a situation detection unit that detects a timing at which the external environment has changed, and the integrated determination unit has received the user's attention at a timing at which the external environment detected by the situation detection unit has changed. The state may be determined.

  The situation detection unit may detect the timing of approaching an intersection by acquiring current location information and map information, and comparing the current location information with location information of an intersection on the map.

  The integrated determination unit may determine the user's attention state by determining a majority of determination results for each direction.

  The attention state determination apparatus according to the present invention uses an eye movement measurement of a user measured by an eye movement measurement unit that measures the movement of an eye, and uses an eye movement start time that is a time when the eye saccade is ended and the saccade. A saccade detection unit that detects a plurality of movement amounts of the saccade, a classification unit that identifies the direction of each saccade based on the movement amount of each saccade, and classifies the direction of each identified saccade, for each classified direction A classification unit that adds and averages eye-holding potentials cut out from the user's electroencephalogram signal measured by using an electroencephalogram measurement unit that measures an electroencephalogram signal starting from the eye-ball retention start time corresponding to each saccade; A attention amount determination unit that determines a caution state for each classified direction based on the eye-holding-related potential, and determination of the caution state for each direction With fruit, and a integration identifying unit for discriminating the attentional state of the user.

  The attention state determination method according to the present invention includes a step of measuring a user's brain wave signal, a step of measuring a movement of the user's eyeball, and a time at which the saccade of the eyeball is completed using the movement of the eyeball of the user. A step of detecting a plurality of eyeball stop start times and a movement amount of the saccade, a direction of each saccade based on the movement amount of each saccade, and a step of classifying the direction of each identified saccade, For each direction, the step of adding and averaging the eyeball retention potentials cut out from the user's brain wave signal starting from the eyeball retention start time corresponding to each saccade, and the addition averaged based on the eyeball retention related potential, Using the step of determining the attention state for each classified direction and the determination result of the attention state for each direction, the user Comprising the steps of determining the attention state.

  A computer program according to the present invention is a computer program executed by a computer installed in an attention state determination device, wherein the computer program receives a user's brain wave signal from the computer; Based on the step of receiving the movement, the step of detecting a plurality of eyeball stop start times, which are times when the saccade of the eyeball has ended, and the amount of movement of the saccade, using the movement of the eyeball of the user, and the amount of movement of each saccade Identifying the direction of each saccade, and classifying the direction of each identified saccade, wherein for each classified direction, the eyeball stop start time corresponding to each saccade is used as a starting point from the user's brain wave signal Adding and averaging the cut-out eyeball stationary potentials; Determining the attention state for each classified direction on the basis of the addition-averaged eye-holding related potential, and determining the user's attention state using the determination result of the attention state for each direction; Is executed.

  According to the present invention, the eye-holding potential (EFRP) obtained by averaging for each direction of the saccade is discriminated from the attention state, and the final attention state is discriminated using the attention amount discrimination result for each direction. Thereby, the attention state can be determined in a short analysis interval. By shortening the analysis section, it becomes possible to determine the state of caution in more detail, even in urban areas where the environment changes from moment to moment, or in caution states between intersections.

It is a figure which shows the experimental condition and the content of the subject. It is a figure which shows the electrode position of the international 10-20 method. (A)-(c) is a flowchart of the discrimination | determination procedure of the conventional attention state, (d) is the discrimination | determination rate which shows a discrimination | determination result. It is a figure which shows the electrode position for electrooculogram measurement. It is a figure which shows the example of a saccade direction determination. (A) And (b) is a figure which shows distribution of the discrimination rate according to a saccade direction. It is a figure which shows the structure of the caution state discrimination | determination system 1 by embodiment of this invention. The figure which shows an example of the specific structure of an attention state discrimination device It is a figure which shows a head mounted display (HMD). 3 is a flowchart showing a processing procedure of the attention state determination system 1; It is a figure which shows an example of the relationship between eye movement data, brain wave data, and discrimination | determination timing. It is a figure which shows the example of the correspondence of the electric potential of EOG and the movement angle of an eyeball. (A) is a figure which shows the example of the measured eye movement data, electroencephalogram data, and eyeball stop start time, (b) is a figure which shows the example of cut out EFRP, (c) is added for every direction It is a figure which shows the example of EFRP performed, (d) is a figure which shows the waveform by which the EFRP which started from the end time of the saccade of the right direction was added and averaged. 5 is a flowchart showing a detailed procedure of processing of an EFRP classification unit 60. It is a figure which shows the example of EFRP in process and its saccade information. It is a figure which shows an example of the direction classification of a non-uniform angle. It is a figure which shows an example of the attention state discrimination | determination result output from the attention amount discrimination | determination part. 5 is a flowchart illustrating a detailed procedure of processing of an integrated determination unit 80. (A) is a figure which shows the example of weighting when the car navigation system is installed in the car, (b) shows an example of the attention state discrimination | determination result for every direction discriminate | determined by the attention amount discrimination | determination part 70. FIG. (A) is a figure which shows the comparison result at the time of aligning an analysis area to 120 second like the past, (b) is using the data of the same experiment, maintaining a fixed discrimination | determination precision, It is a figure which shows the result of having evaluated how much an analysis area can be shortened. It is a figure which shows the block configuration of the caution state discrimination | determination system 2 concerning the modification of this embodiment. It is a flowchart of the whole process of the caution state discrimination | determination system 2 which concerns on the modification of embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  Compared with the conventional attention state determination method, the inventors of the present application analyze the tendency of the attention state determination accuracy in each direction when EFRP is classified and averaged for each saccade direction. did. As a result, the inventors of the present application have a bias in the accuracy of attention state discrimination for each saccade direction, the direction with low discrimination accuracy differs for each person and situation, and it is not possible to predict the direction in advance. We found the characteristic that it is possible.

  First, the contents obtained from the EFRP measurement experiment contents and the experiment results conducted by the present inventors for this analysis will be described.

  The experiment participants were 17 males with an average age of 22.1 ± 1.8 years. The inventors of the present application conducted an experiment by a double task method in which the experiment participants had two tasks performed in parallel.

  The first problem is a driving problem. A driving simulator (Mitsubishi Precision Co., Ltd., hereinafter abbreviated as “DS”) was used to drive an urban course of about 4 minutes. The road conditions of the city course were set to be congested so that they could run freely within the speed limit. On the course, vehicles were placed in front of and behind the vehicle on the same lane, vehicles were also placed in the opposite lane, and pedestrians were placed. Participants in the experiment ran along a predetermined route according to the instructions displayed on the car navigation system screen. However, the directions were confirmed only by visual observation of the screen by the participants themselves, and no voice guidance was provided.

  The second problem is a cognitive load problem. In order to experimentally divert the attention resources of the participants from driving, we conducted a task of answering spoken questions with oral answers. Under mild cognitive load conditions (simple question load), questions were asked about the profiles of individual participants in the experiment that could be answered without thinking (for example, “What is your name?”, “What is your age?”, Etc.). In severe cognitive load conditions (difficult question load), questions that require visual thinking, such as hindering attention to driving and thinking of a map, were asked (for example, “A country facing the Mediterranean Please "). This cognitive load experimentally simulates conversations related to thoughts and memories while driving.

  Next, experimental conditions will be described. In this experiment, the electroencephalogram during driving was measured under the two conditions shown in FIG. The first condition is a driving concentration condition. Under the driving concentration condition, the experiment participant performs the DS operation (driving load) and the simple question load in parallel. Since the simple question load is not so much cognitive load, it is considered that the experiment participants can concentrate on driving. The second condition is a distraction condition. Under the distraction condition, the experiment participant performs the DS operation (driving load) and the difficult question load in parallel. Since the difficulty question load has a large cognitive load, the experiment participants have to devote a lot of attention resources to accomplishing this task, and as a result, it is thought that they are distracted from driving.

  Participants in the experiment were equipped with electroencephalographs (Teac, polymate AP-1124), and the electrodes were placed according to the international 10-20 electrode method. FIG. 2 shows the electrode positions for the international 10-20 method. This will be described using the symbols shown in FIG. The lead-out electrode is arranged at the back of the head Oz (between O1 and O2), the reference electrode is arranged at each of the mastoids A1 and A2 near the left and right ears, and the ground electrode is arranged at the forehead. The electroencephalogram was measured by measuring the potential of the occipital region Oz based on the average of the potentials of the left and right mastoids A1 and A2. The inventors of the present application applied a bandpass filter processing of 1 Hz to 15 Hz on the electroencephalogram data measured at a sampling frequency of 500 Hz and a time constant of 3 seconds. Then, electroencephalogram data from -300 milliseconds to 600 milliseconds was extracted from the saccade end time, that is, the eyeball retention start time, and baseline correction was performed with a potential value of 0 milliseconds.

  The discrimination rate was estimated by analyzing EFRP in the above experiment. Here, the discrimination rate is an index indicating how much discrimination between the two states of the driving concentration state and the distraction state can be performed. When the time width TW of the analysis section is TW = 120 seconds and the time shift amount TS of the next analysis section is 10 seconds, it can be correctly determined as the driving concentration from the lambda reaction amplitude value of the driving concentration condition (when driving + simple question loading). And the average value of the ratio that can be correctly determined as distraction from the lambda response amplitude value under the distraction condition (during driving + difficult question loading).

  Based on the above discrimination rate calculation method, the discrimination rate when using the conventional method and the discrimination rate when EFRP is classified for each saccade direction were calculated.

  Hereinafter, discrimination using the conventional method will be described with reference to FIG.

  FIGS. 3A to 3C are flowcharts of a conventional attention state determination procedure. First, all eyeball stop start times included in the analysis section 120 seconds are detected, and an electroencephalogram (EFRP: eyeball stop related potential) waveform starting from each eyeball stop start time is extracted (FIG. 3A). All the EFRP waveforms included in 120 seconds are averaged to calculate an EFRP averaged waveform (FIG. 3B). Then, by comparing the lambda response (maximum value included in greater than 0 and less than 200 milliseconds) of the EFRP average waveform with the threshold, it is determined that the feature amount is greater than or equal to the threshold value, and the case where the feature value is less than or equal to the threshold value is diffused. Is determined. Generally, the vertical axis of the graph of the EFRP waveform is described with the downward direction being positive and the upward direction being negative. In the waveform example of FIG. 3B, since the maximum value of the EFRP average waveform is larger than the threshold value, the determination result is determined as “concentration” (FIG. 3C). When the above determination method was repeatedly determined while shifting by the time shift amount (10 seconds), the determination rate was 79.6%. Further, as shown in FIG. 3D, when the analysis interval was shortened to 60 seconds, it was 72.3%.

  As can be seen from FIG. 3D, in order to obtain an accuracy of about 80% by the conventional method, the analysis interval needs to be 120 seconds in order to ensure a sufficient number of additions.

  Next, EFRP classification for each saccade direction will be described. In the discrimination in which the EFRP is classified for each saccade direction, the eyeball stop start time included in the analysis interval 120 seconds is detected, and the EFRP is extracted for each saccade direction.

  Here, the saccade detection and the saccade direction classification method will be described. The saccade was detected by measuring the horizontal eyeball movement amount and the vertical eyeball movement amount using an EOG (Electro-oculogram) method. The “EOG method” is a method of measuring eye movement from changes in potential of electrodes arranged on the left and right and top and bottom of the eyeball, utilizing the property that the cornea of the eyeball is positively charged with respect to the retina. FIG. 4 shows an example of electrode positions for measuring eye movement by the EOG method. As shown in FIG. 4, electrodes were placed at the top and bottom (V1, V2) and left and right (H1, H2) positions of the experiment participant's eyes. The potential difference between H1 and H2 and V1 and V2 was measured, and the amount of movement in the horizontal direction and the vertical direction was measured based on the magnitude of the potential.

  Next, a saccade detection method will be described. According to the conventional literature (Hiroshi Miyata et al., New Physiological Psychology 1, 1998, p256, Kitaoji Shobo), the time required for saccade is usually 20 milliseconds or more and 70 milliseconds or less, and the speed of saccade is 300 degrees in terms of viewing angle. Per second or more and 500 degrees per second or less. Therefore, an eye movement in which the eye movement direction is the same continuously for a predetermined time (for example, 20 milliseconds or more and 70 milliseconds or less) and the average angular velocity of the predetermined time is 300 degrees / second or more is defined as a saccade. By detecting the end time, the eyeball stop start time can be detected.

  Next, the saccade direction classification method will be described. The saccade direction is classified into 8 directions using a saccade detection threshold (300 degrees / second), a horizontal movement amount, and a vertical movement amount. FIG. 5 shows a direction classification pattern. “◯” in FIG. 5 indicates that the movement amount exceeds the threshold value, and “X” indicates the movement amount that the movement amount is equal to or less than the threshold value. If only the horizontal direction (eg right direction) exceeds the threshold, the direction is classified as “right”, and if only the vertical direction (eg upward direction) exceeds the threshold, the direction is classified as “up”. Is done. When both the horizontal direction (for example, right direction) and the vertical direction (for example, upward direction) exceed the threshold, the direction is classified as “upper right”. This determination is performed in four combinations of top, bottom, left, and right, and the directions are classified.

  For example, consider a saccade with a horizontal eyeball movement of 400 degrees / second in the right direction and a vertical movement of 50 degrees / second in the vertical direction. In this saccade, only the eyeball movement amount in the right direction exceeds the threshold value. Therefore, the direction is determined to be “right”.

  The EFRP addition average waveform is calculated for each saccade direction, and the discrimination rate for each direction is calculated by comparing the lambda response with a threshold value. For example, when determining the discrimination rate in the right direction, the eye stop time at the time of saccade in the right direction included in the analysis interval 120 seconds is detected, and the EFRP waveform is extracted. The extracted EFRP waveforms are averaged, a rightward EFRP average waveform is calculated, and the lambda reaction is extracted. Then, the discrimination rate in the right direction is calculated by comparing the lambda response of the right direction EFRP addition average waveform with the threshold value.

  FIG. 6 shows the result of the discrimination rate calculated for each direction by the method performed by the inventors. FIG. 6A shows the discrimination rate by direction of two experiment participants (experiment participant A and experiment participant B) out of the 17 persons. The radar chart in the figure shows the discrimination rate in 8 directions. The radius of the chart indicates the discrimination rate, the center is 0%, the outside is 100%, and dotted lines are written in 20% increments.

  The discrimination rate of the experiment participant A in the conventional method that does not distinguish the direction was 57.4%. Looking at the discrimination rate for each direction of the experiment participant A, the discrimination rates in the upper right, upper, left, and lower right directions exceed 80%. On the other hand, it can be seen that the discrimination rate in the right, upper left, and lower left directions is as low as about 50%. Similarly, experiment participant B finds that the discrimination rate of the conventional method is 50.0%, while the discrimination rate in the upper and lower left directions is high and the discrimination rate in the lower and lower right directions is low.

  Therefore, although the discrimination accuracy is low in the conventional method, it can be seen that when the accuracy is viewed for each saccade direction, the discrimination accuracy distribution varies, and the direction in which the discrimination accuracy is low varies from person to person.

  For the same experiment participant A, the discrimination rate was calculated under the condition that the road condition of the DS driving task as the first task was changed. In each road situation, for example, the position and number of vehicles in the vicinity of the traveling course and the own vehicle are different. FIG. 6B shows the discrimination result of the experiment participant A obtained under three different road conditions. The radar chart in the figure is the same as the format in FIG. According to FIG. 6B, it is understood that the discrimination rate is different even in the same experiment participant. That is, in road condition 1, the right, upper left, and lower left directions are low, in road condition 2, the upper and lower left directions are low, and in road condition 3, the right, lower left, and lower right directions are low.

  As shown in FIG. 6B, it can be seen that the direction in which the discrimination rate is low differs depending on the situation even for the same person. It can also be seen that it is difficult to predict a direction with a low discrimination rate in advance.

  From the above results, it was found that when the accuracy for each saccade direction is viewed, there is a direction with low discrimination accuracy, the direction varies depending on the person and the situation, and cannot be predicted in advance.

  Considering the reasons for the above knowledge, when driving, depending on the individual and circumstances, it can be biased to the part that attracts the driver's attention such as other cars and pedestrians, when attention is biased in a specific direction or when a specific direction is seen It can be said that the noise contamination rate increases. Thereby, it is considered that the accuracy varies for each direction of the saccade.

  In view of the experimental results described so far and the knowledge obtained from the experimental results, EFRPs are classified by saccade direction, and the final determination result is derived by integrating the determination results for each direction. The idea is that the discrimination accuracy is improved if the analysis interval is the same, and that the analysis interval can be shortened if the same discrimination accuracy is maintained.

  Hereinafter, the details of the embodiment of the present invention configured based on this idea and the effect of analyzing the above data based on the embodiment of the present invention will be described with reference to the drawings, taking as an example the attention state determination during driving of the automobile. While explaining. In addition, along with an example of driving, the user of the attention state determination device and the system will be described below as a driver.

(Embodiment 1)
FIG. 7 shows a configuration of the attention state determination system 1 according to the present embodiment.

  The attention state determination system 1 determines an attention amount for each direction in which a saccade has occurred, using a brain wave signal of the driver 10, more specifically, an eyeball retention related potential which is one component of the brain wave signal. Then, by combining these results, the driver's 10 attention state for driving is finally determined. The “attention state for driving” in the present embodiment means a state where attention is focused on driving or attention is distracted. The attention state determination system 1 displays a warning or the like on a display screen (not shown) so as to call attention according to the attention state of the driver 10 for driving, or accumulates the determination result of the attention state in the storage device. .

  The attention state determination system 1 includes an electroencephalogram measurement unit 20 that measures the brain wave of the driver 10, an eye movement measurement unit 30 that measures the movement of the eyeball of the driver 10, and an attention state determination device 100. Note that the block of the driver 10 is shown for convenience of explanation.

  The attention state determination apparatus 100 includes a saccade detection unit 40, a classification unit 45, an attention amount determination unit 70, and an integrated determination unit 80.

  Hereinafter, each component will be schematically described, and then each will be described in detail.

  The saccade detection unit 40 detects saccade information such as the saccade end time (eyeball stop start time) and the saccade movement amount from the eyeball movement (eyeball movement) by the eyeball movement measurement unit 30. As described later, the saccade direction can be specified based on the saccade movement amount. Therefore, the saccade detection unit 40 may obtain the saccade direction and output it by including it in the saccade information.

  The classification unit 45 acquires the brain wave signal of the user 10 measured by the brain wave measurement unit 20 and acquires saccade information from the saccade detection unit 40. The classification unit 45 cuts out an eyeball retention-related potential starting from the eyeball retention start time from the brain wave signal and saccade information of the user 10. Subsequently, the classification unit 45 obtains and classifies the saccade direction corresponding to the eyeball retention-related potential EFRP cut out using the saccade information, and adds and averages the eyeball retention-related potential EFRP for each classified direction.

  The attention amount determination unit 70 determines a caution state for each direction, and determines a caution state for each direction.

  The integrated determination unit 80 determines the attention state using the determination processing result for each direction, and accumulates or outputs the result.

  Hereinafter, each component will be described in detail.

  The electroencephalogram measurement unit 20 is an electroencephalograph that measures an electroencephalogram signal that is a potential change at an electrode attached to the driver's 10 head. The inventors of the present application envision a wearable electroencephalograph in the future. Therefore, the electroencephalograph may be a head-mounted electroencephalograph. When using the driving state determination system 1, it is assumed that the driver 10 is wearing an electroencephalograph in advance.

  Electrodes are arranged in the electroencephalogram measurement unit 20 so as to come into contact with a predetermined position of the head when worn on the head of the driver 10. For example, at the electrode position of the international 10-20 method shown in FIG. 2, electrodes are arranged on the back of the head (O1 or O2 or Oz in the middle), the left earlobe (A1), the right earlobe (A2), and the forehead.

  According to conventional literature (Hiroshi Miyata et al., New Physiological Psychology 1, 1998, p262, Kitaoji Shobo), the lambda component that appears around 100 milliseconds starting from the eyeball stop start time reflects the cognition and attention. It is said that it will appear dominant. However, measurement is also possible at Pz (the center of the top of the head) around the back of the head, and electrodes may be arranged at this position. This electrode position is determined from the reliability of signal measurement, the ease of mounting, and the like. It is also possible to reduce the number of electrodes. For example, it is possible to measure an electroencephalogram even when the number of electrodes is three, that is, the electrodes Oz, A1, and the ground electrode.

  As a result, the electroencephalogram measurement unit 20 can measure the electroencephalogram of the driver 10. The measured electroencephalogram signal is sampled so that it can be processed by a computer, and data for a predetermined period of time is temporarily stored in the storage unit of the electroencephalogram measurement unit 20 and can be updated as needed. In addition, the influence of the noise mixed in an electroencephalogram signal can be reduced by carrying out the low-pass filter process of the electroencephalogram signal measured in the electroencephalogram measurement part 20 previously, for example by 30 Hz.

  The eye movement measuring unit 30 measures the movement of the eye movement. The eye movement is measured based on, for example, an EOG (electro-oculogram) method.

  As described above, FIG. 4 shows an example of the electrode position of the eye movement measuring unit 30 that measures the eye movement by the EOG method. The electrodes H1 and H2 are mounted at the left and right temple positions, and the horizontal potential is measured from the potential difference between the two electrodes. The electrodes V1 and V2 are attached to the upper and lower positions of the left or right eyeball, and the vertical potential is measured from the potential difference between the two electrodes.

  In the case of measuring by the EOG method, the eye movement measuring unit 30 determines the horizontal and vertical directions of the eyeball from the potential change amount in the horizontal direction and the vertical direction from the potential change amount and eyeball movement angle correspondence table shown in FIG. Guess the amount of movement in the direction. The amount of movement of the line of sight can be estimated from the amount of potential change in the horizontal direction and the amount of potential change in the vertical direction. The saccade detection unit 40, which will be described later, compares the amount of movement of the line of sight with a threshold value, and when the movement of the line of sight exceeding the threshold is detected, detects the movement of the line of sight as a saccade and starts the end of eye movement Judge as time.

  In the above example, the movement of the eyeball is measured using the EOG method, but the method for measuring the eyeball movement is not limited to this. For example, the eyeball may be irradiated with near infrared rays, and the movement of the eyeball may be measured using the reflected image (corneal reflection image) (corneal reflection method). In this case, the line-of-sight movement amount is calculated based on the horizontal and vertical eyeball movement amounts measured from the movement of the reflected image, and is compared with a threshold value. Alternatively, a camera for photographing the eyeball may be installed in the vehicle, the position of the cornea may be detected by image processing, and the amount of movement of the eyeball in the horizontal direction and the vertical direction may be measured. In addition, when using image processing, by detecting the position of the cornea (for example, pixel coordinates in the image) in the captured image, and using a correspondence table between the movement amount of the cornea coordinates and the movement angle of the eyeball, Measure the amount of eyeball movement in the horizontal and vertical directions.

  Refer to FIG. 7 again.

  The classification unit 45 includes an EFRP cutout unit 50 and an EFRP classification unit 60.

  The EFRP cutout unit 50 receives the saccade information from the saccade detection unit 40, and specifies the eyeball stop start time based on the saccade information. Then, the EFRP cutout unit 50 cuts out an electroencephalogram signal starting from the saccade end time, and extracts a waveform (EFRP waveform) of the eyeball stationary potential.

  The EFRP classification unit 60 receives the saccade information from the saccade detection unit 40, and classifies each saccade direction (saccade direction) from the saccade information. Then, the EFRP classification unit 60 adds and averages the cut out EFRP waveforms for each classified saccade direction.

  The attention amount determination unit 70 determines the attention state for each direction based on the EFRP for each saccade direction that is averaged by the EFRP classification unit 60. The attention amount determination unit 70 according to the present embodiment determines which of the two attention states, concentrated or diffused, by comparing the amplitude value of the lambda response of the EFRP with a threshold value that is held in advance.

  The integrated determination unit 80 determines the attention state of the driver 10 based on the determination result for each saccade direction in the attention amount determination unit 70. The integrated determination unit 80 according to the present embodiment also finally determines which of the two attention states of concentration or distraction is applicable. The attention state determination result is fed back to the driver 10 using an output device (not shown) held by the integrated determination unit 80. An example of the output device is a speaker for calling the driver by voice, operation sound, warning sound, or the like. Other examples of output devices are car navigation system monitors, head-mounted displays, or head-up displays (HUD) that can present text and images. In addition, the integrated determination unit 80 may hold a recording device (HDD or the like), and may record and store the determination results in the recording device.

  Next, a specific configuration of the attention state determination system 1 will be described.

  Hereinafter, an example in which the attention state determination system 1 is incorporated in a head mounted display (hereinafter also referred to as “HMD”) will be described.

  FIG. 8 shows an example in which the attention state determination system 1 is configured on a head mounted display. In FIG. 8, the same reference numerals are assigned to components having the same functions as the reference numerals assigned to the respective components shown in FIG.

  The HMD is an eyeglass type and has a band on the back of the head in order to fix the HMD to the head of the driver 10. A part of the electrodes of the electroencephalogram measurement unit 20 is provided on the band on the back of the head.

FIG. 9 shows names of each part of the HMD. In the example of FIG. 9, the name of each part is the same as the general name of each part in the glasses. Hereinafter, names of main parts of the glasses will be described.
A portion that is caught in the ear of the user 10 and fixes the HMD main body is referred to as a “first cell portion 101”. A portion in contact with the nose root of the user 10 is referred to as a “nose pad portion 104”. The frame portion of the spectacle lens is referred to as a “rim portion 102”. Further, a portion connecting and supporting the front cell portion 101 and the rim portion 102 is referred to as a “temple portion 103”. The tip cell portion 101, the rim portion 102, the temple portion 103, and the nose pad portion 104 are provided symmetrically on the left and right sides of the HMD.

  As can be understood from FIGS. 8 and 9, an electrode group used by the electroencephalogram measurement unit 20 to measure an electroencephalogram is arranged in the center of the two front cell units 101 and the occipital band. Yes. In addition, an electrode group used by the eye movement measurement unit 30 to measure the eye movement is arranged above and below the temple part 103 and the rim part 102.

  When the driver 10 wears the HMD, the electrodes of the electroencephalogram measurement unit 20 are arranged so as to naturally touch the position of the binaural earlobe (mastoid) and the back of the head. Further, the electrodes of the eye movement measuring unit 30 are arranged so that the electrodes naturally come into contact with the driver's 10 mechanism and the upper and lower positions of the left eye. The main bodies of the electroencephalogram measurement unit 20 and the eye movement measurement unit 30 may be integrally formed with one electrode, or may be provided as an independent circuit at the position of the rim unit 102 or the like. .

  In addition, the contact resistance of the electrodes of the electroencephalogram measurement unit 20 and the eye movement measurement unit 30 is suppressed by a silver-silver chloride electrode. For the electrode in contact with the back of the head, a mechanism may be provided in which protrusions are provided on the electrode surface to prevent hair and the electrode directly contacts the scalp, and it is not always necessary to apply a paste to the electrode.

  In the example of FIG. 4, the eye movement measurement unit 30 is attached to the upper, lower, left, and right eyes of the left eye, but the electrode position for EOG measurement is not limited to this. For example, the electrode of either the left or right temple part 103 may be arranged on the nose pad part 104 to measure the EOG in the horizontal direction. Alternatively, the electrode of the rim portion 102 may be arranged in the vertical direction on the temple portion 103 to measure the EOG in the vertical direction.

  The saccade detection unit 40, the EFRP cutout unit 50, the EFRP classification unit 60, the attention amount determination unit 70, and the integration determination unit 80 are arranged in the left and right temple units 103 so that each has the connection relationship shown in FIG. Connected by internal wiring (cable).

  In the above configuration, an example in which all configurations of the attention state determination system 1 are arranged as independent parts on the HMD has been described, but the configuration is not limited to this. For example, only the electroencephalogram measurement unit 20 and the eye movement measurement unit 30 are arranged on the HMD, and the saccade detection unit 40, the EFRP cutout unit 50, the EFRP classification unit 60, the attention amount determination unit 70, and the integrated determination unit 80 are separated from each other. The attention state determination system 1 may be configured by communicating between the HMD and another terminal via a cable or wirelessly. Further, the saccade detection unit 40, the EFRP extraction unit 50, the EFRP classification unit 60, the attention amount determination unit 70, and the integration determination unit 80 may be processed in the same CPU or DSP.

  In addition, when the eye movement measurement unit 30 measures the eye movement using a camera or the like instead of the EOG method, a camera for taking a cornea reflection image by the eyeball or near infrared irradiation is used as the HMD rim unit 102 or the temple. What is necessary is just to arrange | position in the part 103 or a vehicle interior.

  Next, the operation of the attention state determination system 1 will be described.

  FIG. 10 is a flowchart showing a processing procedure of the attention state determination system 1. Hereinafter, the operation of the attention state determination system 1 will be described with reference to the flowchart of FIG.

  In step S <b> 10, driving is started by the driver 10. When driving is started, attention state determination processing by the driving attention state determination system 1 is also started. The start of driving is specified, for example, by starting the engine, releasing the side brake, or stepping on the accelerator.

  In step S <b> 20, the electroencephalogram measurement unit 20 measures the electroencephalogram of the driver 10. The measured electroencephalogram is sampled so as to be processed by a computer, and sent to the EFRP cutout unit 50. In order to reduce the influence of noise mixed in the electroencephalogram, in the present embodiment, the electroencephalogram measurement unit 20 performs a bandpass filter process of, for example, 1 Hz to 15 Hz on the measured electroencephalogram.

  In step S <b> 30, the eye movement measurement unit 20 measures the eye movement of the driver 10. As shown in FIG. 4, the eye movement measuring unit 30 attaches four electrodes to the face of the driver 10 and measures the electrooculogram in the horizontal and vertical directions by the EOG method. The measurement method is the same as the above-described experimental method. The measured electrooculogram is sampled so as to be processed by a computer and sent to the saccade detection unit 40. Steps S20 and S30 can be performed in parallel.

  In step S81, the integrated determination unit 80 determines whether it is time to determine the attention state. In the present embodiment, the attention state determination timing is set at an interval of 10 seconds. The integrated determination unit 80 controls the determination timing so that the attention state determination is performed every 10 seconds. If it is not the discrimination timing, the electroencephalogram measurement in step S20 and the eye movement measurement in step S30 are continued. When it is determined by the integrated determination unit 80 that the determination timing is reached, processing for determining the attention state after step S40 is started.

  FIG. 11 shows an example of the relationship between eye movement data, brain wave data, and discrimination timing. The horizontal axis in the figure is time, and the discrimination timing is indicated by white arrows. In this embodiment, the discrimination timing is executed with a time shift amount of every 10 seconds irrespective of the electroencephalogram data and the eye movement data.

  Refer to FIG. 10 again. In step S40, the saccade detection unit 40 analyzes the eye movement data of the time interval to be analyzed (hereinafter referred to as “analysis interval”), and the saccade end timing (that is, the eyeball stop start time) included in the analysis interval. Is detected. In the present embodiment, the analysis interval is 40 seconds. FIG. 11 shows an example of an analysis interval. The analysis interval at the discrimination timing (a) is 40 seconds immediately before the discrimination timing (a), and is a range indicated by the arrow in the analysis interval (a). The reason for setting the length of the analysis section to 40 seconds is that the present inventors considered the average intersection interval to be 40 seconds. When an automobile is moving in a city area at a speed of 40 km / h, the distance traveled in 40 seconds is about 440 m, which is considered to be shorter than the distance between intersections or traffic lights.

  The saccade detection unit 40 detects a saccade based on, for example, the saccade detection condition of the above experiment (eye movement with a movement time of 20 milliseconds to 70 milliseconds and an average angular velocity of 300 degrees / second or more). The saccade end time is detected as the eyeball stop start time, and the horizontal movement amount and the vertical movement amount are accumulated as saccade information for each eyeball stop start time. When the eye movement measuring unit 30 is measured by the EOG method, the velocity of the eyeball can be measured by the amount of change in potential. FIG. 12 shows an example of the correspondence between the EOG potential and the eyeball movement angle. From the correspondence relationship of FIG. 12, it is possible to associate the amplitude value of the eyeball potential with the amount of movement of the eyeball. From these relationships, it is determined whether or not the saccade generation condition is satisfied from the potential amount of eye movement, and the eyeball stop start time is detected.

It is also conceivable that saccades are generated in an oblique direction. In the detection of the saccade in the oblique direction, the movement amount in the horizontal direction and the movement amount in the vertical direction are used. The moving amount of the eye movement in the oblique direction can be calculated by (((moving amount in the horizontal direction) ² + (moving amount in the vertical direction) ² ) ^1/2 . The saccade detection is performed by comparing the movement amount of the eye movement in the oblique direction and the saccade detection condition (average angular velocity is 300 degrees / second or more). The eyeball stop start time is detected based on the detected saccade end time.

  FIG. 13A shows an example of measured eye movement data, electroencephalogram data, and eyeball stop start time. As shown in FIG. 13A, a plurality of eyeball retention start times may be included in one analysis section. The saccade detection unit 40 detects all these eyeball stop start times t1, t2,..., T8.

  As a saccade detection method, first, each horizontal and vertical saccade is detected independently, and then a set of horizontal and vertical saccades with overlapping time intervals is integrated into one. Good. Alternatively, a method may be used in which horizontal eye movement data and vertical eye movement data are first combined to obtain vector data, and then saccade detection is performed based on the direction and size data of the vector data.

  Simultaneously with the detection of the eyeball stop start time, the saccade movement direction and, if necessary, the saccade movement amount are also extracted. The saccade movement direction can be obtained as the direction (angle) of the vector by using the vector data described above. On the other hand, the amount of movement of the saccade corresponds to all detected eyeball retention start times based on information on the amount of movement from the beginning of the saccade to the end of the saccade (potential (μV) for the EOG method and angle (degree) for the corneal reflection method). And extracted.

  The “saccade movement amount” is a concept including the amount of movement in the horizontal direction and the amount of movement in the vertical direction for determining the direction. The position at the start of the saccade and the position at the end of the saccade. Note that it is not just a distance between.

  The eyeball stop start time, the saccade movement direction, and the saccade movement amount are transmitted to the EFRP classification unit 60 as saccade information.

  In step S50 of FIG. 10, the EFRP cutout unit 50 cuts out the eyeball retention related potential (EFRP) included in the analysis section. Specifically, the electroencephalogram data from −300 milliseconds to 600 milliseconds is cut out as EFRP, starting from each eyeball stop start time extracted in step S40.

  FIG. 13B shows an example of the cut out EFRP. Baseline correction is performed on the cut out EFRP so that the potential at the eyeball retention start time (0 milliseconds) is 0 μV.

  The above-mentioned “cutout” has a meaning of associating the electroencephalogram data with the information on the eyeball stop start time. Therefore, the present invention is not limited to the extraction of the electroencephalogram data in the analysis section for processing and copying it to another area of the memory. For example, it also includes using the electroencephalogram data of a predetermined section of the recorded electroencephalogram data as it is.

  In step S60 of FIG. 10, the EFRP classification unit 60 classifies the cut out EFRP for each saccade direction, and adds and averages the EFRPs in the respective directions. FIG. 13C shows an example of EFRP added for each direction. Details of processing of the EFRP classification unit 60 such as saccade direction classification and addition averaging will be described later. In this embodiment, as an example, the saccade direction is classified into eight directions.

  In step S70 of FIG. 10, the attention amount determination unit 70 determines the attention state for each of the EFRP waveforms averaged for each direction in step S60.

  First, the attention amount determination unit 70 measures the amplitude value of a lambda reaction that is a positive component in the vicinity of about 100 milliseconds in the EFRP addition average waveform in each direction. This will be specifically described with reference to FIG. FIG. 13D shows a waveform obtained by averaging the EFRPs starting from the end time of the saccade in the right direction. The horizontal axis of the graph is time, the unit is milliseconds, the vertical axis is potential, the unit is μV, and the downward direction is a positive value. In addition, FIG. 13D also shows an example of the lambda response amplitude. In the present embodiment, the amplitude of the lambda reaction is calculated based on the amplitude of the maximum value in, for example, 50 milliseconds to 150 milliseconds. Note that the waveform may reflect the effects of waveform drift and noise mixing. Therefore, the amplitude of the lambda response may be extracted by calculating the section average value instead of the local maximum value of 50 milliseconds to 150 milliseconds.

  The attention amount determination unit 70 determines the attention state by comparing the measured lambda response amplitude value with a preset threshold value. In the present embodiment, two attention states of concentration and distraction are discriminated. As a tendency of the lambda response amplitude, it is known that the amplitude increases when concentrating (Yagi “Application of Eyeball-Related Potential to Industrial Scenes”, Psychological Review, vol. 45 (1), 103-117.2. -2). Using this feature, the attention state is determined based on the threshold set in the attention amount determination unit 70. In the determination, when the lambda response amplitude exceeds the threshold, it is determined as “concentration”, and when it is below the threshold, it is determined as “diffuse”.

  Now, taking the case where the threshold is set to 2.5 μV as an example, attention state determination of the EFRP added average waveform in FIG. 13D will be described. As the lambda response amplitude of the EFRP added average waveform, a local maximum value in the vicinity of 60 milliseconds is extracted as the local maximum value included in the range from 50 milliseconds to 150 milliseconds. The maximum amplitude value 3.3 μV is the lambda response amplitude. Since the lambda response amplitude value 3.3 μV exceeds the threshold value of 2.5 μV, the attention amount determination unit 70 determines that the EFRP addition average waveform in FIG. 13D means “concentration”. .

  The attention amount determination unit 70 determines the attention amount for every classified saccade direction, and outputs the determination result to the integrated determination unit 80. In this embodiment, since eight directions are classified, eight attention state determination results are output from the attention amount determination unit 70.

  In step S80 in FIG. 10, the integration determination unit 80 integrates the determination results for each saccade direction determined in step S70, and determines the driver's 10 attention state. Detailed processing of the integrated determination unit 80 will be described later.

  Next, details of the processing of the EFRP classification unit 60 performed in step S60 of FIG. 10 will be described with reference to FIG.

  FIG. 14 is a flowchart showing a detailed procedure of processing of the EFRP classification unit 60.

  In step S601, the EFRP classification unit 60 receives the saccade information measured in step S40 and the EFRP waveform data extracted in step S50.

  In step S602, the EFRP classification unit 60 selects one unprocessed EFRP waveform from the received EFRP waveforms. Here, “processing” means processing for classifying EFRPs according to the saccade direction.

  In step S603, the EFRP classification unit 60 identifies the saccade direction from the saccade information of the eyeball stop start time that is the starting point of the selected EFRP. The EFRP and the saccade information are associated with the time information of the eyeball stop start time, and the saccade information holds the movement amount in the horizontal direction and the movement amount in the vertical direction. The EFRP classification unit 60 specifies the saccade direction by the method in the above-described experiment using the horizontal movement amount and the vertical movement amount. That is, as shown in the example of FIG. 5, the saccade movement amount is compared with the threshold value (300 degrees / second) in the horizontal direction and the vertical direction, and the direction is determined by combination.

  FIG. 15 shows an example of EFRP being processed and its saccade information. FIG. 15 also shows a state in which the EFRP processed so far is classified on the storage device according to the saccade direction. The EFRP being processed at the top is determined to be in the right direction from the amount of movement of the saccade.

  The saccade direction classification method described above is an example. The classification method is not limited to the above. For example, the EFRP classification unit 60 may obtain a saccade direction vector from the amount of movement in the horizontal direction and the amount of movement in the vertical direction, and may perform classification based on the angle of the vector. The method of calculating the angle of the saccade direction vector can be calculated by, for example, vector angle (radian) = arctan (vertical direction movement amount / horizontal direction movement amount). When classifying into 8 directions using the angle of the vector, for example, when the right direction is 0 degrees and the angle around the counterclockwise direction is positive, −22.5 to 22.5 degrees is “right”, 22 A region divided into 45 degrees may be associated with a direction such that 5 to 67.5 degrees is “upper right” and 67.5 to 112.5 degrees is “up”. The same applies to “upper left”, “left”, and “lower left”. Which region is divided is determined by the value of the vector angle.

  In step S604, the EFRP classification unit 60 accumulates the EFRP waveform for each corresponding classification according to the direction classified in step S603. When classified into 8 directions, EFRP is accumulated for every 8 directions. In the example of FIG. 15, since the EFRP being processed is classified as “right”, the EFRP waveform is accumulated in the “right” group among the eight categories.

  In step S605, the EFRP classification unit 60 checks whether all the EFRPs received in step S601 have been classified. If all the EFRPs are classified, the process proceeds to step S606. If unprocessed EFRP remains, the process returns to step S602 and is repeated until all EFRPs are classified.

  In the above description, the number of direction classifications is eight directions, but this is an example. It is not limited to this. For example, EFRP may be classified into 16 directions more finely. In the case of 16 directions, 360 degrees are divided into 16 (each direction is 22.5 degrees), and the area is determined such that “−11.25 to 11.25 degrees” is “right”. The direction of the saccade direction can be determined depending on which of the 16 regions the direction (angle) of the saccade direction vector determined from the vertical direction movement amount and the horizontal direction movement amount is included.

  In addition, when the amount of movement in the vertical direction cannot be measured due to restrictions on the number of electrodes, the saccade direction may be classified into two directions only in the horizontal direction. In this case, the “right” and “left” saccade directions are classified according to the amount of horizontal movement.

  Further, the classification of the saccade direction may not be divided equally as described above. FIG. 16 shows an example of direction classification of non-uniform angles. When driving, speedometers and car navigation devices are present at the bottom of the field of view, so it is considered that downward saccades occur when looking at the devices. Therefore, it may be divided so as to distinguish the movement of the line of sight with respect to the front of the vehicle and the movement of the line of sight with respect to the driving equipment. In the example of FIG. 16, the movement of the line of sight toward the front of the vehicle is classified into four areas (direction classifications 1 to 4) from 0 ° to 180 °, and the movement of the line of sight with respect to the driving equipment is classified from 180 ° to 360 °. ing. The classification method of the saccade direction is the same as described above, and is determined according to which direction classification the direction (angle) of the saccade direction vector determined from the vertical direction movement amount and the horizontal direction movement amount belongs.

  In step S606, the EFRP classification unit 60 calculates the average of the EFRP waveforms classified and accumulated for each direction. In the present embodiment, since eight directions are classified, one EFRP addition average waveform in one direction and a total of eight EFRP addition average waveforms are calculated.

  In step S <b> 607, the EFRP classification unit 60 transmits the calculated EFRP addition average waveform to the attention amount determination unit 70.

  The attention amount determination unit 70 compares the lambda response amplitude in the EFRP average waveform for each of the classified saccade directions with the threshold value. When the lambda response amplitude exceeds the threshold value, it is “concentration”. Judged as “diffuse”. FIG. 17 shows an example of the attention state determination result output from the attention amount determination unit 70. As shown in the drawing, discrimination results of “concentration” or “diffuse” are held for each of the eight directions.

  Next, details of the processing of the integrated determination unit 80 performed in step S80 will be described. FIG. 18 is a flowchart showing a detailed procedure of the processing of the integrated determination unit 80.

  In step S801, the integrated determination unit 80 acquires the attention state determination result for each direction determined by the attention amount determination unit 70.

  In step S802, the integrated determination unit 80 calculates a ratio of determination results. In the example of FIG. 17, the determination result of “concentration” is three, and the determination result of “diffuse” is five. Therefore, concentration: diffuse = 3: 5.

  In step S <b> 803, the integration determination unit 80 determines the magnitude of the “concentration” ratio. When the ratio of “concentration” is higher than “diffuse”, the process proceeds to step S805. In the data of FIG. 17, since the ratio of “concentration” is lower than “diffuse”, the process proceeds to step S804.

  In step S804, the integrated determination unit 80 determines the magnitude of the diffuse ratio. If the “concentration” ratio is lower than the “diffuse” ratio, the process proceeds to step S806. If the ratio of “concentration” is not lower than the ratio of diffuse, it is determined that “concentration” = “diffuse”, and the process proceeds to step S807.

  In step S805, the integrated determination unit 80 determines that the driver's 10 attention state is “concentration”.

  In step S806, the integrated determination unit 80 determines the attention state of the driver 10 as “diffuse”.

  In step S807, the integrated determination unit 80 determines the attention state of the driver 10 using the determination result in the specific direction. According to the above experiments conducted by the inventors of the present application, as a result of investigating the discrimination rate by direction of 17 test participants, the accuracy in the lower right direction tended to increase. In the present embodiment, the result of the left is reflected, and the weight of the determination result in the lower right direction is increased. We decided to adopt the discrimination result.

  Note that the processing when the ratio of “concentration” and “diffuse” in the attention state determination results by direction is the same is not limited to the above, and depends on how the attention state determination device of the present embodiment is used, the characteristics of attention, etc. It is determined. For example, when the attention state determination device wants to detect without being distracted, if the ratio of “concentration” and “distraction” is the same, distraction may be output. Further, if it is assumed that the attention state does not change as the characteristic of the state transition of the attention state, the predetermined attention state can be made the same as the previous determination result. As a result, a stable result can be output even in an intermediate state between “diffuse” and “concentration”. In the case of the same number, it may be output as a caution state of “neither” as a state where it is neither “concentrated” nor “diffuse”.

  In the above-described processing of the integrated determination unit 80, it has been described that the attention state of the driver 10 is determined based on the ratio of “concentration” and “diffuse”, but is not necessarily limited to this determination method. For example, instead of the ratio, the final attention state may be determined by directly comparing the number determined as “concentrated” with the number determined as “diffuse”. In that case, the final attention state may be determined by determining a large number of determination results for each classified direction.

  Alternatively, a weight may be set in advance for each direction according to the situation, and the determination may be made based on the weight. FIG. 19A shows an example of weighting when a car navigation system is installed in a car. The value in each direction indicates the magnitude of the weight in that direction. The weight in the lower left is 0.5, and the weight in the other directions is 1. In a car equipped with a car navigation system, it is considered that when the line of sight is moved to the lower left, many line-of-sight movements for looking at the car navigation system are included. When discriminating the amount of attention to driving, the weight on the lower left is set small so that the amount of attention when looking at the car navigation system is not reflected so much.

  Hereinafter, a determination method using weighting will be described. An example of the attention state determination result for each direction determined by the attention amount determination unit 70 is shown in FIG. In the attention state determination result, the vertical and horizontal directions are determined to be diffuse, and the other four directions are determined to be concentrated. For each direction, the weight is divided into concentrated and diffused to determine whether the sum of the concentrated or diffused weights is greater. In the example of FIG. 19, concentration (1 + 1 + 0.5 + 1) = 3.5 and diffuse (1 + 1 + 1 + 1) = 4 are determined to be diffuse.

  Further, as described above, the weighting may be steadily set according to the in-vehicle environment or the individual, or may be dynamically changed according to the road environment or the driving operation. For example, an expressway can be considered as an example of weighting according to the road environment. On expressways, there are few cases of jumping out from the left and right, so the amount of attention in the left and right direction is often small. Therefore, by setting a small weight in the left-right direction, it is possible to make it difficult to reflect the influence of a decrease in the attention amount in the left-right direction in driving attention amount determination. Further, as an example of weighting according to the driving operation, it is possible to operate the steering wheel. For example, when turning the steering wheel to the right, the car travels in the right direction, so much attention should be paid to the right side. On the contrary, on the left side, the amount of attention decreases as much attention resources are allocated to the right side. Therefore, when turning right, it is considered that the influence of attentional bias can be alleviated by increasing the saccade right direction weight or decreasing the saccade left direction weight.

  Note that the above-described method using weighting can also be applied to the case where the final attention state is determined by directly comparing the number determined to be “concentrated” with the number determined to be “diffuse”. . That is, depending on the direction determined as “concentration” or “diffuse”, a number greater than 1 may be assigned, or a number less than 1 may be assigned. The number itself corresponds to the weight.

  In step S808, the integrated determination unit 80 performs feedback to the driver 10 based on the attention state determination result. When the integrated determination unit 80 holds a video output device such as a display, the determination result of “concentration” or “diffuse” may be displayed on the display. In addition, when a voice output device such as a speaker connected to the integrated determination unit 80 is used and the attention amount is reduced and it is determined that the state is distracted, a warning sound is output and attention is given to the driver and passengers. You may be notified of a distracted state. Further, when the integrated determination unit 80 holds the communication means, the result is transmitted to the outside of the attention state determination device. For example, when it is determined that the attention is distracted, the result is output to the control unit of the automobile. The device may be controlled such as automatically applying a brake. In addition, in a distracted state, the information presented to the driver 10 (engine speed, car navigation system information, etc.) is filtered so as not to be shown to the narrowed driver so as to create an environment that can concentrate more on driving. May act.

 Next, the effect when the attention state is determined by the method of the present embodiment will be described using the analysis result. This analysis result is obtained by using the same data as the experimental data (for 17 experiment participants) described at the beginning of the present embodiment, when analyzed by the conventional method, and when analyzed by the method of the present embodiment. It is the result of comparing.

 First, FIG. 20A shows a comparison result when the analysis interval is aligned to 120 seconds as in the conventional case. While the discrimination rate by the conventional method is 79.6%, the discrimination rate by the method of this embodiment is 90.2%, and a very good result was obtained. According to this result, it is understood that the analysis section can be shortened in this embodiment while maintaining the discrimination rate of 80% or more.

  Next, FIG. 20B shows the result of evaluating how much the analysis interval can be shortened while maintaining a certain discrimination accuracy using data of a similar experiment. In the experiments of the present inventors, when the method of this embodiment for classifying and discriminating in eight directions of the saccade direction was used, an evaluation was made as to how far the entire analysis section can be shortened.

  From the table of FIG. 20B, when the analysis interval is 120 seconds, a high discrimination rate of 90.2% is obtained. Hereinafter, 83.1% at 60 seconds, 81.5% at 40 seconds, and 30 seconds. It can be seen that the identification accuracy gradually decreases as the analysis interval is shortened in order of 76.8%. Here, it exceeds 80%, which is the standard of identification accuracy in the conventional method, in the case where the analysis interval is 40 seconds or more.

  From the above results, it was found that according to the method of the present embodiment, a discrimination rate equivalent to the conventional method (about 80%) can be maintained even in the analysis interval of 40 seconds.

  When this 40-second analysis section is converted into the time when the car is driven, if the speed is 40 km / h, the distance traveled in 40 seconds is about 440 m, and the caution state while driving in the section of about 440 m is determined. It will be possible. In the case of the conventional 120 seconds, it can be said that the level of detail in determining the attention state can be improved when compared with running at a distance of 1.3 km or more. For example, it is considered that the attention state can be determined for each intersection if measurement is performed every 40 seconds even in an urban area.

  As described above, according to the configuration and the processing procedure according to the present embodiment, the EFRP is classified and added based on the direction of the saccade in the apparatus that determines the driver's state and performs the safe driving support, and is determined by each classified EFRP. By determining the final attention state based on the result, the time required for the attention state determination can be shortened, and more detailed attention state determination can be performed even in an environment where the situation frequently changes, such as an urban area.

  In the present embodiment, the example in which the attention state determination result by the integrated determination unit 80 is fed back to the driver 10 has been described. However, when the integrated determination unit 80 includes a storage device such as an HDD. May store the determination result in the storage device. When the discrimination results are accumulated, it is conceivable that analysis and use are performed as basic data for safe driving guidance for the driver 10 after the driving is completed. For example, by taking the ratio of the time determined to be diffuse during driving and the driving time, and indicating the frequency of the distraction state during driving to the driver 10 after the driving is completed, it is possible to alert the next driving.

  Note that the attention state determination device according to the present embodiment is not limited to the determination of the attention state during driving of the vehicle, but can also be applied to the determination of the attention state in factory work or monitoring work that requires concentration in the line of sight. For example, in factory work, how to move the line of sight and how to pay attention to an object in the field of view varies depending on the individual and the work process. It is conceivable that the noise contamination rate when looking at is increased. Therefore, also in the above case, it is considered that the analysis section can be shortened by classifying the EFRP for each saccade direction and determining the attention state.

  In the present embodiment, as an example of the determination of the driver's 10 attention state, an example in which two states of a concentrated state and a diffuse state are determined has been described. There may be an application example in which it is necessary to determine the state between concentration and distraction, such as when a warning is given before the driver 10 becomes distracted. The case where the state is determined in multiple stages between concentration and distraction as shown on the left is also within the scope of the present invention. When determining states in multiple stages, the attention amount determination unit 70 determines the attention state based on the amplitude of the lambda reaction.

  In the present embodiment, an example has been described in which the analysis section has a fixed length and the integrated determination unit 80 determines the attention state at a fixed determination timing. Considering that the external environment does not change so much from the intersection to the intersection, it can be considered that the same attention state exists between the intersection and the intersection. In addition, the incidence of accidents at intersections is very high, and safe driving support according to the state of caution at intersections is considered to be highly effective in preventing accidents.

  Therefore, an example of a caution state determination system for effectively supporting safe driving at an intersection will be described. Hereinafter, a modification of the embodiment in the case where the above-described analysis section is between intersections will be described.

  FIG. 21 shows a configuration of the attention state determination system 2 according to a modification of the present embodiment. In FIG. 21, the same components as those in FIG. 7 are denoted by the same reference numerals, and the description thereof is omitted. In FIG. 21, the attention state determination system 2 includes a situation detection unit 90 that detects a timing when the external environment is switched, in other words, a timing when the external environment is changed.

  The situation detection unit 90 is connected to a vehicle-mounted GPS, a car navigation system, or the like, and acquires current location information and map information. The situation detection unit 90 detects the timing at which the vehicle approaches the intersection by comparing the current location with the map information. The detection result is transmitted to the integrated determination unit 81 of the attention state determination device 110. The integrated determination unit 81 receives the detection result from the situation detection unit 90, and performs processing according to the detection result, as will be described later. Except for performing processing according to the detection result, the operation of the integrated determination unit 81 is the same as the operation of the integrated determination unit 80 of the previous embodiment. Note that the attention state determination device 110 and the attention state determination device 100 (FIG. 7) are different in that they have the integrated determination unit 81 or the integrated determination unit 80.

  FIG. 22 is a flowchart of the overall processing of the attention state determination system 2 according to a modification of the above embodiment. The same steps as those in FIG. 10 are denoted by the same step numbers and description thereof is omitted.

  In step S90, the situation detection unit 90 determines whether or not the own vehicle is near the intersection. For example, the situation detection unit 90 acquires information on the current location and map information from the car navigation system, and compares the distance between the coordinates of the current location and the coordinates of the point where the road intersects in the traveling direction. It is determined whether the distance is within a certain value (for example, 50 m before). When it is determined that the host vehicle is about to approach the intersection, the situation detection unit 90 transmits an instruction for determination timing to the integrated determination unit 80.

  In step S <b> 82, the integrated determination unit 80 confirms whether an instruction for determination timing has been transmitted from the situation detection unit 90. If it has not been transmitted, the process returns to step S20 again, and the electroencephalogram measurement is continued. When the determination timing is instructed, the processing from step S40 onward is executed with the analysis period from the time when the previous determination timing is instructed to the time when the current determination timing is instructed.

  By determining the caution state before the intersection by the configuration and processing procedure of the above-described modification, it is possible to support safe driving according to the caution state at the intersection. For example, if it is determined that attention is distracted immediately before entering the intersection, the integrated determination unit 80 may send a control signal to the vehicle at that time or when entering the intersection, and support such as greatly decelerating the vehicle can be considered. It is done. In addition, the car navigation system may be controlled, and voice guidance for instructing the timing of turning right and left may be frequently performed to call attention. Alternatively, safe driving support such as reducing the information that is not related to the right or left turn at the intersection displayed on the screen of the car navigation system so that the user can concentrate on the displayed information can be considered.

  As described above, the configuration and processing procedure of the above-described caution state determination device modification enables safe driving support according to the caution state in an accident-prone area such as an intersection. According to the above-described configuration, safety in intersection traveling is improved.

  In the above example, the situation detection unit 90 is linked to the car navigation system and detects the position of the intersection from the map information. However, the present invention is not limited to this. For example, a right / left turn operation may be detected based on an angle at which the handle is turned and the handle is turned, and a point where the right / left turn operation is started may be detected as a timing approaching an intersection.

  In addition, when applied to factory work instead of during operation, information such as work processes may be acquired from equipment in the factory to detect process switching timing.

  The present invention can also be implemented as a computer program. Such a computer program includes instructions for performing the procedures illustrated by the flowcharts of FIGS. 10, 14, 18 and 22, for example. The computer program is recorded on a recording medium such as a CD-ROM and distributed as a product to the market, or transmitted through an electric communication line such as the Internet. Note that the attention state determination apparatus according to the above-described embodiments and modifications can also be realized as hardware such as a DSP in which a computer program is incorporated in a semiconductor circuit.

  The attention state determination system according to the present invention is useful for attention state determination in a driving operation in which the situation frequently changes. It is also effective for determining the state of caution when working in a factory where concentration on work is required. For example, when the attention state determination system is configured as a head-mounted display type device, it is effective as a safety support device during bicycle driving or walking. Furthermore, by associating a TV viewing attention state with an interest in a TV program, it can be applied to marketing applications such as program interest measurement.

DESCRIPTION OF SYMBOLS 10 User 20 Electroencephalogram measurement part 30 Eye movement measurement part 40 Saccade detection part 50 EFRP extraction part 60 EFRP classification | category part 70 Attention amount discrimination | determination part 80 Integrated discrimination | determination part 90 Situation detection part

Claims (13)

An electroencephalogram measurement unit that measures a user's electroencephalogram signal;
An eye movement measurement unit that measures the movement of the user's eyeball;
A saccade detection unit that detects a plurality of eyeball stop start times and movement amounts of the saccades using the movement of the user's eyeballs;
A classification unit that identifies the direction of each saccade based on the amount of movement of each saccade, and classifies the direction of each identified saccade, and for each classified direction, determines the eyeball stop start time corresponding to each saccade. A classifying unit that adds and averages eye-holding potentials cut out from the user's brain wave signal as a starting point;
An attention amount determination unit that determines an attention state for each classified direction based on the eye-holding-related potential that has been averaged;
An attention state determination system comprising: an integrated determination unit configured to determine the user's attention state using a determination result of the attention state for each direction. The attention amount determination unit determines, for each direction, whether it is in a concentrated state or a distracted state,
The attention state determination system according to claim 1, wherein the integrated determination unit determines a state determined by either the concentration state or the distraction state determined by the attention amount determination unit as the attention state of the user. .   The attention state determination apparatus according to claim 2, wherein the integrated determination unit determines the attention state of the user by deciding a majority of determination results for each direction by the attention amount determination unit.   The attention state determination system according to claim 1, wherein the saccade detection unit detects, as the eyeball stop start time, a time point when the amount of movement of the user's eyeball becomes smaller than a predetermined threshold.   The attention amount discriminating unit compares the amplitude value of the lambda response of the eyeball retention-related potential averaged for each direction with a predetermined threshold value to determine the attention state for each classified direction. The attention state determination system according to claim 2, wherein the determination is performed.   6. The attention amount determination unit uses a local maximum value included in 100 ± 100 milliseconds of the eye-holding-related potential obtained by averaging from the eye-holding start time as a starting point, as an amplitude value of a lambda reaction. Caution state determination system described in 1.   The attention state determination system according to claim 1, wherein the integration determination unit determines the attention state of the user at regular time intervals. It further includes a situation detection unit that detects when the external environment changes,
The attention state determination system according to claim 1, wherein the integration determination unit determines the user's attention state at a timing when the external environment detected by the situation detection unit changes.   The said situation detection part detects the timing which approaches an intersection by acquiring the information of present location and map information, and comparing the information of the said present location with the positional information on the intersection on a map, Attention state determination system.   The attention state determination system according to claim 1, wherein the integrated determination unit determines the user's attention state by deciding a majority of determination results for each direction. Saccade detection that detects a plurality of eyeball stop start times and movement amounts of the saccades, which are times when the saccades of the eyeballs have ended, using the eyeball movements of the user measured by an eyeball movement measurement unit that measures eyeball movements And
A classification unit that identifies the direction of each saccade based on the amount of movement of each saccade, and classifies the direction of each identified saccade, and for each classified direction, determines the eyeball stop start time corresponding to each saccade. As a starting point, a classification unit that adds and averages eye-holding potentials cut out from the user's electroencephalogram signal measured using an electroencephalogram measurement unit that measures an electroencephalogram signal;
An attention amount determination unit that determines an attention state for each classified direction based on the eye-holding-related potential that has been averaged;
An attention state determination apparatus comprising: an integrated determination unit configured to determine the user's attention state using a determination result of the attention state for each direction. Measuring a user's brain wave signal;
Measuring the movement of the user's eyeball;
Using the movement of the user's eyeball to detect a plurality of eyeball stop start times that are times when the saccade of the eyeball has ended and a movement amount of the saccade;
Identifying the direction of each saccade based on the amount of movement of each saccade, and classifying the direction of each identified saccade, starting from the eyeball stop start time corresponding to each saccade for each classified direction Adding and averaging eyeball stationary potentials cut out from the user's brain wave signal,
Determining a state of caution for each classified direction based on the addition-averaged eye-holding-related potential; and
And a step of determining the attention state of the user using a determination result of the attention state for each direction. A computer program executed by a computer mounted on the attention state determination device,
The computer program is for the computer.
Receiving a user's electroencephalogram signal;
Receiving eye movements of the user;
Using the movement of the user's eyeball to detect a plurality of eyeball stop start times that are times when the saccade of the eyeball has ended and a movement amount of the saccade;
Identifying the direction of each saccade based on the amount of movement of each saccade, and classifying the direction of each identified saccade, starting from the eyeball stop start time corresponding to each saccade for each classified direction Adding and averaging eyeball stationary potentials cut out from the user's brain wave signal,
Determining a state of caution for each classified direction based on the addition-averaged eye-holding-related potential; and
And a step of determining the user's attention state using a determination result of the attention state for each direction.

Images