JP7131491B2 - Saccade detection device, saccade detection method, saccade detection program - Google Patents

Description

The present disclosure relates to technology for detecting a saccade of a monitored person.

Conventionally, in the technique disclosed in Patent Document 1 for detecting a saccade, the output when the saccade is detected is filtered based on the probability of non-occurrence of the saccade. Here, the saccade non-occurrence probability is calculated according to the amplitude appearing in the saccade signal indicating the saccade motion.

Japanese Patent No. 5613533

In the technique disclosed in Patent Document 1, when the amplitude of the saccade signal at a single time is large, the probability of non-occurrence of saccades is calculated to be low. That is, when the motion amplitude of the saccade at a single time is large, it is determined that the probability of occurrence of the saccade is high. However, even if the motion amplitude at a single time is large, it may be determined that the saccade occurrence probability is low, depending on the motion amplitudes at previous and subsequent times, for example.

An object of the present disclosure is to provide a saccade detection device that improves saccade detection accuracy.

Another object of the present disclosure is to provide a saccade detection method that improves saccade detection accuracy.

Still another object of the present disclosure is to provide a saccade detection program that improves saccade detection accuracy.

Technical means of the present disclosure for solving the problems will be described below. It should be noted that the symbols in parentheses described in the claims and this column indicate the correspondence with specific means described in the embodiments described in detail later, and limit the technical scope of the present disclosure. not something to do.

A first aspect of the present disclosure is
A saccade detection device (1) for detecting a saccade of a person to be monitored (2),
a first extraction block (12) for extracting a candidate motion that is a saccade candidate as the eyeball motion of the monitoring subject;
a second extraction block (13) for extracting an interlocking movement that is assumed to be interlocked with the candidate movement extracted by the first extraction block as facial movements other than the eye movement of the monitoring subject;
a prediction block (20) for predicting a saccade occurrence probability (P) according to the correlation pattern between the candidate motion extracted by the first extraction block and the interlocking motion extracted by the second extraction block;
an adjustment block (30) for adjusting a judgment threshold (J) for judging whether or not the candidate motion extracted by the first extraction block is a saccade according to the probability of occurrence predicted by the prediction block;
a judgment block (40) for judging truth based on the judgment threshold adjusted by the adjustment block.

A second aspect of the present disclosure is
A saccade detection method executed by a processor (6) to detect a saccade of a monitored person (2), comprising:
extracting a candidate motion that is a saccade candidate as the eyeball motion of the person to be monitored (S102);
As facial movements excluding eye movements of the monitoring subject, extracting interlocking movements that are assumed to be interlocked with the extracted candidate movements (S103);
estimating the probability of saccade occurrence (P) according to the correlation pattern between the extracted candidate motion and the extracted interlocking motion (S104);
Adjusting a judgment threshold (J) for judging whether or not the extracted candidate motion is a saccade according to the predicted occurrence probability (S105);
It includes judging truth or falsehood based on the adjusted judgment threshold (S106, S107, S108).

A third aspect of the present disclosure is
A saccade detection program stored in a storage medium (5) and comprising instructions to be executed by a processor (6) to detect a saccade of a person to be monitored (2),
the instruction is
As the eye movements of the monitoring subject, candidate movements that are saccade candidates are extracted (S102),
As facial movements excluding the eyeball movements of the monitoring subject, extracting interlocking movements that are assumed to be interlocked with the extracted candidate movements (S103);
estimating the probability of saccade occurrence (P) according to the correlation pattern between the extracted candidate motion and the extracted interlocking motion (S104);
adjusting a judgment threshold (J) for judging whether or not the extracted candidate motion is a saccade according to the predicted probability of occurrence (S105);
It includes determining the truth based on the adjusted determination threshold (S106, S107, S108).

Whether or not the candidate motion is a saccade in the above first to third modes is determined based on a determination threshold adjusted according to the occurrence probability of the saccade. Here, the probability of saccade occurrence is the correlation between the candidate saccade movements among the eye movements of the monitored subject and the linked movements assumed to be linked to the candidate facial movements other than the eye movements of the monitored subject. Predicted, depending on the pattern. This involves not only candidate motions at a single time, but also considering correlation patterns with facial motions linked thereto, and predicting the saccade occurrence probability and adjusting the judgment threshold. can be accurately determined. Therefore, it is possible to improve the saccade detection accuracy.

1 is a block diagram showing the overall configuration of a saccade detection device according to a first embodiment; FIG. 2 is a block diagram showing the detailed configuration of the saccade detection device according to the first embodiment; FIG. It is a table for explaining probability prediction data according to the first embodiment. 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a graph for explaining prediction conditions and prediction processing according to the first embodiment; 4 is a flow chart showing a saccade detection method according to the first embodiment; It is a table for explaining probability prediction data according to the second embodiment. It is a graph for explaining prediction conditions and prediction processing according to the second embodiment.

A plurality of embodiments will be described below with reference to the drawings. Note that redundant description may be omitted by assigning the same reference numerals to corresponding components in each embodiment. When only a part of the configuration is described in each embodiment, the configurations of other embodiments previously described can be applied to other portions of the configuration. Moreover, not only the combinations of the configurations explicitly specified in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined even if they are not specified unless there is a particular problem with the combination.

(First embodiment)
As shown in FIG. 1, the saccade detection device 1 according to the first embodiment detects, in particular, saccades among the eyeball movements of the person to be monitored 2 .

Specifically, the saccade detection device 1 may be mounted on a vehicle to detect the saccade of the passenger who is the person to be monitored 2 . In this case, the saccade detection result is used, for example, for driving control of the vehicle. The saccade detection device 1 may be connected to a computer to detect the saccade of the user who is the monitored person 2 . In this case, the saccade detection result is used, for example, for computer operation control. Of course, the application form of the saccade detection device 1 may be a form other than these.

The saccade detection device 1 includes an information acquisition unit 3 and an electronic control unit 4 . The information acquisition unit 3 acquires target information about the monitored person 2 . Acquisition of the target information is executed according to instructions from the person to be monitored 2 or others, or according to control from the electronic control unit 4 or other units. The target information includes eyeball information related to the eyeball movements of the monitoring target person 2 and face information related to the facial movements of the monitoring target person 2 excluding the eyeball movements.

The information acquisition unit 3 may be, for example, a camera or the like that captures an image of at least the eyeballs of the face of the person to be monitored 2 . In this case, the image captured by the camera and generated is the target information. The information acquisition unit 3 may be, for example, an electrooculogram measurement device that outputs a waveform of at least eyeball movement among the movements of the monitored person 2 . In this case, the electrooculogram measured by the electrooculogram measuring device is the target information. The information acquisition unit 3 may be a combination of these cameras and an electrooculogram measuring device. When the saccade detection device 1 is mounted on a vehicle, a device mounted on the vehicle, such as an in-vehicle camera, may be used as the information acquisition unit 3 . In this case, the saccade detection device 1 consists only of the electronic control unit 4 .

The electronic control unit 4 is connected to the information acquisition unit 3 via at least one of, for example, a LAN (Local Area Network), a wire harness, an internal bus, and the like. The electronic control unit 4 is a dedicated computer including at least one memory 5 and at least one processor 6 .

The memory 5 stores computer-readable programs and data non-temporarily, and includes at least one type of non-transitory tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium. medium). The processor 6 includes, as a core, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU.

Processor 6 executes a plurality of instructions contained in a saccade detection program stored in memory 5 . Thereby, the electronic control unit 4 constructs a plurality of functional blocks for detecting the saccade of the person to be monitored 2 . In this way, in the electronic control unit 4, the saccade detection program stored in the memory 5 for detecting the saccade of the person to be monitored 2 causes the processor 6 to execute a plurality of instructions, thereby constructing a plurality of functional blocks. be done.

The functional blocks built by the electronic control unit 4 include an extraction block 10, a prediction block 20, an adjustment block 30 and a decision block 40, as shown in FIG. The extraction block 10 has a feature detection section 11, a candidate extraction section 12, and a linked extraction section 13 as sub-functional blocks.

Target information acquired by the information acquisition unit 3 is input to the feature detection unit 11 . The feature detection unit 11 detects feature points of the eyeballs and face of the monitored person 2 based on the input target information. The feature detection unit 11 generates feature data representing the detected eyeball and face feature points. The eyeball feature data includes, for example, the line-of-sight direction (that is, the line-of-sight angle) of the eyeball as an index representing the eyeball movement. The facial feature data includes, for example, the direction of the face and the movement of the eyelids as indices representing facial movements.

The eyeball feature data generated by the feature detection section 11 is input to the candidate extraction section 12 . The candidate extracting unit 12 extracts, from the eyeball movements of at least one eye of the monitoring subject 2, candidate movements that are predicted to be saccade candidates based on the inputted eyeball feature data. The candidate movements may include short and high-speed eyeball movements that occur when the monitored person 2 intentionally changes the line-of-sight direction. Candidate movements may include slight, rapid eye movements that occur involuntarily during gaze (ie, fixation) of the monitored subject 2 .

The candidate extraction unit 12 compares the specific physical quantity calculated based on the eyeball feature data with a predetermined standard threshold value. The standard threshold is set in advance to a critical value of a specific physical quantity assumed for the candidate motion. If the specific physical quantity is within the allowable range from the standard threshold value, the candidate extraction unit 12 extracts the candidate motion corresponding to the same physical quantity.

The candidate extraction unit 12 may compare the motion amplitude calculated from the change distance of the line-of-sight direction (that is, the movement distance of the line-of-sight) with the standard threshold. In this case, a standard threshold value is set for at least one of, for example, a lower limit value and an upper limit value in the allowable range of motion amplitude. The candidate extraction unit 12 may compare the line-of-sight velocity (that is, the line-of-sight angular velocity in the case of angle conversion) calculated from the amount of change in the line-of-sight direction over time with the standard threshold value. In this case, a standard threshold value is set for at least one of the lower limit value and the upper limit value of the allowable range of the radial velocity, for example. The candidate extracting unit 12 may compare the line-of-sight acceleration calculated by the time differential value with respect to the amount of change in the line-of-sight direction over time with a standard threshold value. In this case, a standard threshold value is set for at least one of the lower limit value and the upper limit value of the allowable range of the line-of-sight acceleration.

The candidate extracting unit 12 stores the extracted candidate data representing the candidate motion in the memory 5 in association with time data (that is, time stamp) representing the occurrence time of the motion. At this time, the candidate extraction unit 12 generates candidate data based on the inputted eyeball feature data. The candidate data may include motion amplitude, which is the amplitude of the extracted candidate motion. The candidate data may include a direction of motion, which is the direction of the extracted candidate motion. The candidate data may include a time interval (that is, a time difference) between the current motion, which is the extracted candidate motion, and the previous motion, which is the previous candidate motion extracted earlier. As described above, among the sub-functional blocks of the extraction block 10, the candidate extraction unit 12 corresponds to the "first extraction block".

The feature data of the face generated by the feature detection section 11 is input to the interlocked extraction section 13 . The interlocking extracting unit 13 extracts interlocking movements interlocking with the candidate movements extracted by the candidate extracting unit 12 from the facial movements of the person to be monitored 2 based on the inputted facial feature data. At this time, the interlocking extracting unit 13 extracts the facial movement occurring in the monitoring target person 2 as an interlocking movement in a range including substantially the same time as the candidate movement and from the candidate movement to the set time before and after the candidate movement. The interlocking movement may include face orientation variation in which the monitoring subject 2 changes the face orientation. The interlocking movement may include blinking, which is caused by movement of the eyelids on the face of the monitoring subject 2 .

The interlocking extracting unit 13 converts the extracted interlocking data representing the type of interlocking movement into time data (i.e., time stamp) representing the occurrence time of the movement and candidate data of candidate movements interlocking with the same movement. They are associated and stored in the memory 5 . The type of interlocking motion represented by the interlocking data means the classification of the facial motion of the monitoring subject 2, such as the above-described change in face orientation and eye blinking. As described above, among the sub-functional blocks of the extraction block 10, the interlocked extraction unit 13 corresponds to the "second extraction block".

The prediction block 20 reads from the memory 5 candidate data and associated data associated with each other and time data associated with each of these data. The prediction block 20 predicts a saccade probability P, which is the occurrence probability of saccades, based on the read data. The saccade probability P means the probability that the candidate exercise extracted by the candidate extraction unit 12 will occur in the monitored subject 2 as a saccade. For the saccade probability P, a standard probability as an initial value is set in advance. Therefore, in the following description, reducing the saccade probability P means a process of predicting the saccade probability P to a smaller value by multiplying the standard probability by a prediction coefficient of less than 1. On the other hand, increasing the saccade probability P in the following description means a process of predicting the saccade probability P to a higher value by multiplying the standard probability by a prediction coefficient exceeding 1.

The prediction block 20 reads from the memory 5 probability prediction data 21 for predicting the saccade probability P as shown in FIG. The probability prediction data 21 is generated in at least one format of, for example, a table, a map, an arithmetic expression, etc., and stored in the memory 5 in advance. The probability prediction data 21 includes at least one of sets 22 to 27 shown in FIG. 3 as data sets of prediction conditions and prediction processes.

The data sets 22 to 24 are defined together with the prediction process using correlation patterns between candidate motions extracted at different times by the candidate extraction unit 12 as prediction conditions. On the other hand, the data sets 25 to 27 are defined together with the prediction process using the correlation pattern between the candidate motion extracted by the candidate extractor 12 and the interlocked motion extracted by the interlocked extractor 13 as a prediction condition.

Specifically, the prediction conditions and prediction process of the data set 22 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 22 include the time interval Ti between the current exercise and the previous exercise among the extracted candidate exercises. Therefore, the prediction processing of the data set 22 reduces the saccade probability P of the current exercise under the prediction condition that the time interval Ti is equal to or less than the first set interval T1. The prediction process of the data set 22 also reduces the saccade probability P of the current exercise even under prediction conditions where the time interval Ti is equal to or greater than the second set interval T2.

In this way, the data set 22 defines a prediction condition including a correlation pattern regarding the time interval Ti between candidate motions and a prediction process according to the same condition. Note that the first set interval T1 is set to, for example, 62 ms. The second set interval T2 is set to 67.5 seconds, for example.

The prediction conditions and prediction process of the data set 23 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 23 include correlations between the motion directions Dp and Dl of the current motion and the previous motion among the extracted candidate motions. Therefore, the prediction processing of the data set 23 reduces the saccade probability P of the current motion under the same-direction motion prediction condition in which the motion direction Dp of the current motion matches the motion direction Dl of the previous motion. Furthermore, the prediction conditions of data set 23 include the number of times the directions of motion Dp and Dl match consecutively. Therefore, in the prediction processing of the data set 23, the degree of decrease in the saccade probability P is increased as the number of consecutive motions in the same direction with the motion directions Dp and Dl increases as a prediction condition.

In this way, the data set 23 defines a prediction condition including a correlation pattern regarding motion directions Dp and Dl between candidate motions, and a prediction process according to the same condition. It is determined whether or not the movement directions Dp and Dl are coincident, for example, only in the lateral direction of the person to be monitored 2, and the saccade probability P needs to be lowered.

The prediction conditions and prediction process of the data set 24 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 24 include the correlations of the motion amplitudes Ap and Al of the current motion and the previous motion among the extracted candidate motions, together with the correlations with the time intervals Ti. Therefore, in the prediction process of the data set 24, in the case of FIG. is equal to or less than the third set interval T3, the saccade probability P of the current exercise is increased.

On the other hand, in the prediction process of the data set 24, in the case of FIG. Under the prediction condition that Ti is equal to or less than the fourth set interval T4, the saccade probability P of the current exercise is reduced. Similarly, the prediction process of the data set 24 is also performed in the case of FIG. Under the prediction condition that the interval Ti is equal to or smaller than the fourth set interval T4, the saccade probability P of the current exercise is reduced.

Furthermore, the prediction processing of the data set 24 is performed in the case of FIG. is greater than the fourth set interval T4 and equal to or less than the fifth set interval T5, the saccade probability P of the current exercise is increased. Similarly, the prediction process of the data set 24 is also performed in the case of FIG. Under the prediction condition that the interval Ti exceeds the fourth set interval T4 and is equal to or smaller than the fifth set interval T5, the saccade probability P of the current exercise is increased.

In this way, the data set 24 defines prediction conditions including correlation patterns regarding motion amplitudes Ap and Al and time intervals Ti between candidates, and prediction processing according to the same conditions. Note that the first set amplitude A1 is set to, for example, 16 degrees in terms of angle. The second set amplitude A2 is set to, for example, 8 degrees in terms of angle. The third set interval T3 is set to, for example, 102 ms. The fourth set interval T4 is set to 167 ms, for example. The fifth set interval T5 is set to 276 ms, for example.

The prediction conditions and prediction process of the data set 25 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 25 include the correlation between the motion amplitude Ap of the current motion among the extracted candidate motions and the face orientation variation among the extracted interlocking motions. Therefore, in the prediction process of the data set 25, when the motion amplitude Ap of the current motion is equal to or greater than the third set amplitude A3, the saccade probability P of the current motion is increased under the prediction condition in which the current motion and the face orientation change are linked. Let

The data set 25 defines a prediction condition including such a correlation pattern regarding the motion amplitude Ap and face direction variation, and a prediction process according to the same condition. It should be noted that, for example, when the time interval from the current motion is within a range of 1 second or the like, the face orientation change is determined to be an interlocking motion that requires an increase in the saccade probability P. Also, the third set amplitude A3 is set to, for example, 20 degrees in angle conversion.

The prediction conditions and prediction process for the data set 26 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 26 include the correlation between the motion amplitude Ap of the current motion among the extracted candidate motions and the blink among the extracted interlocking motions. Therefore, in the prediction process of the data set 26, when the motion amplitude Ap of the current motion is equal to or greater than the fourth set amplitude A4, the saccade probability P of the current motion is increased under the prediction condition that the current motion and blinking are linked. .

The data set 26 defines prediction conditions including such correlation patterns regarding motion amplitude Ap and eye blinks, and prediction processing corresponding to the same conditions. Note that blinking is determined as an interlocking movement that requires an increase in the saccade probability P when the time interval from the current movement falls within a range of 500 ms, for example. The fourth set amplitude A4 is set to, for example, 20 degrees in terms of angle, and is set to the same value as the third set amplitude A3 in the first embodiment.

The prediction conditions and prediction processing of the data set 27 are defined based on the experimental results shown in FIG. The prediction conditions of the data set 27 include the correlation between the motion direction Dp of the current motion among the extracted candidate motions and the variation direction Df of the face orientation variation among the extracted interlocking motions. Therefore, the prediction processing of the data set 27 increases the saccade probability P of the current motion under the prediction condition that the motion direction Dp of the current motion and the change direction Df of the face orientation change match. On the other hand, the prediction processing of the data set 27 reduces the saccade probability P of the current motion under the prediction condition that the motion direction Dp of the current motion and the change direction Df of the face orientation change are different.

The data set 27 defines prediction conditions including such correlation patterns regarding movement direction Dp and changes in face orientation, and prediction processing according to the same conditions. It should be noted that, for example, when the time interval from the current movement is within a range of 1 second or the like, the change in face direction is determined as an interlocking movement that requires an increase or decrease in the saccade probability P. Further, the directions Dp and Df are judged to be a coincidence that requires an increase in the saccade probability P or a difference that requires a decrease in the same probability P only in the lateral direction of the person to be monitored 2, for example.

The prediction block 20 shown in FIG. 2 determines the saccade probability P of the current motion among the candidate motions by executing the prediction process for all the stored sets in the memory 5 among the data sets 22-27. At this time, the prediction coefficient to be multiplied by the standard probability is set to the same value for all the prediction conditions of the memory set in the memory 5, or to a different weighted value for each prediction condition of the memory set. By multiplying the standard probability by the prediction coefficient set in each of the memory sets, the prediction of the saccade probability P for the current exercise is determined.

The adjustment block 30 receives the confirmed saccade probability P of the prediction. The adjustment block 30 adjusts the determination threshold J according to the input saccade probability P. FIG. The determination threshold value J is a reference value when determining whether or not the current motion among the candidate motions is a saccade by the determination block. A standard threshold used also in the candidate extraction unit 12 is set as the saccade authenticity determination threshold J in correspondence with the standard probability of the saccade probability P. FIG.

Therefore, as the saccade probability P predicted by the prediction block 20 decreases below the standard probability, the adjustment block 30 adjusts the determination threshold value J to the side where false determinations are more likely to be made than in the case of the standard threshold value. As a result, as the degree of decrease in the saccade probability P from the standard probability increases, the determination threshold value J is adjusted to a value that widens the range of the specific physical quantity that results in a false determination. On the other hand, as the saccade probability P predicted by the prediction block 20 rises above the standard probability, the adjustment block 30 adjusts the determination threshold J to the side where a true determination is more likely to be made than in the case of the standard threshold. As a result, the determination threshold value J is adjusted to a value that widens the allowable range of the specific physical quantity used for true determination as the degree of increase in the saccade probability P from the standard probability increases.

A decision block 40 receives the decision threshold J adjusted for the current exercise. A determination block 40 determines whether the saccade of the current motion among the candidate motions is true or false based on the input determination threshold value J. FIG. Specifically, the determination block 40 compares the specific physical quantity of the current exercise, which has been compared with the standard threshold value in the candidate extraction unit 12, with the determination threshold value J of the same exercise. Therefore, the decision block 40 decides whether or not the specific physical quantity is within the allowable range from the decision threshold value J. If the specified physical quantity is within the allowable range, a true determination is made that the current motion is a saccade. On the other hand, if the specific physical quantity is not within the allowable range, a false determination that the current exercise is not a saccade is made.

The flow of the saccade detection method in which the electronic control unit 4 detects the saccade of the person to be monitored 2 will be described below with reference to FIG. This flow starts when the information acquisition unit 3 acquires the target information. In the flow described later, "S" means multiple steps of the flow executed by multiple instructions included in the saccade detection program.

In S<b>101 , based on the target information acquired by the information acquisition unit 3 , the feature detection unit 11 detects the feature points of the eyeballs and face of the monitored person 2 . Further, in S101, the feature detection unit 11 generates feature data so as to represent the detected feature points of the eyeball and face.

In S<b>102 , the candidate extraction unit 12 extracts saccade candidate movements among the eyeball movements of the monitored subject 2 based on the eyeball feature data generated in S<b>101 . At this time, specifically, when the specific physical quantity calculated based on the eyeball feature data is within the allowable range from the standard threshold value, the eyeball movement corresponding to the physical quantity is extracted as the candidate movement. Further, in S102, candidate data of the current exercise, which is the extracted current candidate exercise, and time data of the same exercise are associated with each other by the candidate extraction unit 12 and stored in the memory 5. FIG. It is assumed that candidate data and time data of the previous exercise are stored in the memory 5 at the time of extracting the current exercise.

In S103, based on the facial feature data generated in S101, the interlocking extraction unit 13 extracts an interlocking movement interlocked with the current exercise extracted in S102. In S103, the interlocking data of the extracted interlocking exercise, the candidate data of the current exercise interlocking with the same exercise, and the time data of the same exercise are stored in the memory 5 in association with each other by the interlocking extraction unit 13 . It is assumed that the interlocking data and the time data of the interlocking motion with respect to the previous exercise are stored in the memory 5 at the time of extracting the interlocking motion with respect to the current motion.

In S104, the data stored in S102 and S103 as well as the prestored probability prediction data 21 (see FIG. 3) are read from the memory 5 by the prediction block 20. FIG. Further, in S104, the prediction block 20 predicts the saccade probability P of the current motion among the candidate motions based on the read data.

In S105, the adjustment block 30 adjusts the determination threshold J for the current exercise among the candidate exercises so as to conform to the saccade probability P predicted in S104. In S106, the determination block 40 determines whether the saccade of the current motion among the candidate motions is true or false based on the determination threshold value J adjusted in S105. Specifically, at this time, it is determined whether or not the specific physical quantity compared with the standard threshold value in S102 is within the allowable range from the determination threshold value J.

If the specific physical quantity is within the allowable range in S106, the process proceeds to S107. In S107, the decision block 40 makes a true decision that the current motion is a saccade among the candidate motions. On the other hand, when the specific physical quantity is not within the allowable range in S106, the process proceeds to S108. In S108, the decision block 40 makes a false decision that the current motion is not a saccade among the candidate motions.

If a determination is made in one of S107 and S108, this flow ends. The result of the authenticity determination as described above may be utilized for, for example, state estimation in vehicle operation control or computer operation control by executing a program different from the saccade detection program.

(Effect)
The effects of the first embodiment described above will be described below.

In the first embodiment, whether or not a candidate motion is a saccade is determined based on a determination threshold value J adjusted according to a saccade probability P, which is the occurrence probability of a saccade. Here, the saccade probability P is the candidate motion of the saccade among the eye movements of the monitoring subject 2 and the linked motion assumed to be linked to the candidate motion among the facial movements excluding the eye movements of the subject 2. Predicted depending on the correlation pattern. This means that the prediction of the saccade probability P and the adjustment of the judgment threshold J are performed in consideration of not only the candidate motion at a single time but also the correlation pattern with the interlocking facial motion. can be accurately determined. Therefore, it is possible to improve the saccade detection accuracy.

The correlation patterns used for predicting the saccade probability P in the first embodiment include patterns related to the motion amplitude Ap of the candidate motion and the variation of the face orientation of the linked motion. According to this, not only the motion amplitude Ap of the candidate motion but also the correlation pattern between the motion amplitude Ap of the face orientation variation linked to the candidate motion and the factors affecting the saccade probability P can be considered. Therefore, it is possible to achieve accurate authenticity determination based on the determination threshold value J according to the saccade probability P, thereby contributing to improvement in detection accuracy.

The correlation patterns used for predicting the saccade probability P in the first embodiment include patterns related to the motion amplitude Ap of the candidate motion and blinking among the linked motions. According to this, not only the motion amplitude Ap of the candidate motion but also the correlation pattern of the motion amplitude Ap of the blink associated with the candidate motion can be considered among the factors that affect the saccade probability P. Therefore, it is possible to achieve accurate authenticity determination based on the determination threshold value J according to the saccade probability P, thereby contributing to improvement in detection accuracy.

The correlation patterns used for predicting the saccade probability P in the first embodiment include patterns related to the motion direction Dp of the candidate motion and the face orientation variation of the linked motion. According to this, not only the motion direction Dp of the candidate motion, but also among the factors that influence the saccade probability P, in particular, the correlation with the face orientation change (in the first embodiment, its direction of change Df) linked to the same motion. Patterns can be considered. Therefore, it is possible to achieve accurate authenticity determination based on the determination threshold value J according to the saccade probability P, thereby contributing to improvement in detection accuracy.

(Second embodiment)
As shown in FIG. 11, the second embodiment is a modification of the first embodiment.

The probability prediction data 2021 of the second embodiment utilized by the prediction block 20 includes a data set 2028 that replaces the data sets 25, 26 of the first embodiment. In the data set 2028, prediction conditions and prediction processing different from those of the data sets 25 and 26 of the first embodiment are defined based on the experimental results shown in FIG.

The prediction conditions of the data set 2028 include the correlation between the motion amplitude Ap of the current motion among the candidate motions extracted by the candidate extraction unit 12 and the combination of face orientation fluctuation and blinking among the extracted interlocking motions. I'm in. In the prediction process of the data set 2028, when the motion amplitude Ap of the current motion is equal to or greater than the fifth set amplitude A5, under the prediction condition that the motion of the current time, the face direction change, and the eye blink are linked, the saccade probability P of the current motion to raise On the other hand, the prediction process of the data set 2028 is based on the prediction condition that the current motion, the change in face direction, and the blink are linked when the motion amplitude Ap of the current motion is equal to or greater than the sixth set amplitude A6 and less than the seventh set amplitude A7. Then, the saccade probability P of the current exercise is lowered.

Furthermore, the prediction processing of the data set 2028 predicts that when the motion amplitude Ap of the current motion is equal to or greater than the fifth set amplitude A5, there will be no interlocking between the current motion and the change in face orientation and there will be interlocking between the current motion and blinking. In the condition, the saccade probability P of the current exercise is lowered. On the other hand, in the prediction processing of the data set 2028, when the motion amplitude Ap of the current motion is equal to or greater than the sixth set amplitude A6 and less than the seventh set amplitude A7, the current motion and the face orientation change are not interlocked and the current motion and the instantaneous motion are detected. Under the prediction condition that there is interlocking with the eyes, the saccade probability P of the current exercise is increased.

In this way, the data set 2028 defines prediction conditions including correlation patterns for specific combinations of facial motions and motion amplitudes Ap, and prediction processing according to the same conditions. The fifth set amplitude A5 is set to, for example, 20 degrees in terms of angle. The sixth set amplitude A6 is set to, for example, 5 degrees in terms of angle. The seventh set amplitude A7 is set to, for example, 10 degrees in angle conversion.

The correlation patterns used for predicting the saccade probability P in the above second embodiment include patterns related to the motion amplitude Ap of the candidate motion and the combination of face orientation variation and blinking among the linked motions. As a result, the correlation pattern between the candidate motion and the motion amplitude that greatly affects the saccade probability P of the candidate motion can be considered for the combination of the face orientation variation and the eye blink linked to the candidate motion. Therefore, it is possible to improve the accuracy of the authenticity determination based on the determination threshold value J according to the saccade probability P, thereby contributing to the improvement of the detection accuracy.

(Other embodiments)
Although a plurality of embodiments have been described above, the present disclosure is not to be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope of the present disclosure. can be done.

Specifically, the electronic control unit 4 of Modification 1 of the first and second embodiments may be a dedicated computer that includes at least one of a digital circuit and an analog circuit as a processor. Here, especially digital circuits include, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). at least one of Such digital circuits may also include memory storing programs.

In Modification 2 of the first and second embodiments, all of data sets 22-24 may be omitted. In modification 3 of the first and second embodiments, the target information acquired by the information acquisition unit 3 may not include face information. In this case, all of the data sets 25 to 27 may be omitted together with the interlocked extraction section 13 and S103 because the feature detection section 11 does not detect facial feature points in S101. In Modified Example 4 of the first and second embodiments, the prediction conditions of the data set 24 in the probability prediction data 21, 2021 may include only the correlation between the motion amplitudes Ap and Al.

In modification 5 of the first and second embodiments, the saccade true/false may be determined together with the correlation with the motion amplitude instead of the binary determination. In this specific example, when the saccade probability P that the motion amplitude Ap of the current motion is greater than or equal to the first set amplitude A1 increases under the prediction condition that the time interval Ti in the data set 24 is less than or equal to the third set interval T3, , the amplitude Ap is less than the second set amplitude A2. Under these circumstances, the saccade probabilities P and Pp are lowered in the first and second embodiments, making it easier for false determinations to be made. On the other hand, in modification 5, it is assumed that there is an error in the motion amplitude Ap based on the feature data, so even if the current motion is a saccade with the first set amplitude A1 or more, the true determination is made. It is good.

In Modified Example 6 of the first and second embodiments, even though the person to be monitored 2 is awake, even if the determination threshold value J is adjusted, if the false determination of the saccade continues for a predetermined period of time or longer, the determination It may be assumed by block 40 that there is a high probability of false positives. In this case, by executing a program separate from the saccade detection program, use of the authenticity determination result may be restricted in, for example, state estimation in vehicle operation control or computer operation control.

In Modified Example 7 of the first and second embodiments, each process in subsequent blocks 20, 30, and 40 is executed after waiting for extraction block 10 to extract a plurality of candidate motions within a predetermined period of time. good too. In this case, the current motion may be selected in descending order of the candidate motion with the highest saccade probability, and the processes in subsequent blocks 20, 30, and 40 may be executed. Alternatively, the current motion may be selected in descending order of the likelihood of being a candidate, such as a candidate motion having less variation in pupil feature points.

1 saccade detection device 2 monitored person 5 memory 6 processor 10 extraction block 10 block 12 candidate extraction unit 13 interlocking extraction unit 20 prediction block 30 adjustment block 40 determination block Ap motion amplitude Dp motion direction, J decision threshold, P probability

Claims (7)

A saccade detection device (1) for detecting a saccade of a person to be monitored (2),
a first extraction block (12) for extracting a candidate motion that is a candidate for the saccade as the eyeball motion of the monitoring subject;
a second extraction block (13) for extracting an interlocking movement assumed to be interlocked with the candidate movement extracted by the first extraction block as the facial movement of the monitoring subject excluding the eyeball movement;
A prediction block (20) for predicting the occurrence probability (P) of the saccade according to the correlation pattern between the candidate motion extracted by the first extraction block and the interlocking motion extracted by the second extraction block. )When,
An adjustment block (J) that adjusts a judgment threshold (J) for judging whether or not the candidate motion extracted by the first extraction block is the saccade according to the occurrence probability predicted by the prediction block ( 30) and
a judgment block (40) for judging the truth based on the judgment threshold adjusted by the adjustment block (40). 2. The saccade detection apparatus according to claim 1, wherein said correlation pattern includes a pattern relating to motion amplitude (Ap) of said candidate motion and face direction variation among said linked motions. 3. The saccade detection device according to claim 1, wherein the correlation pattern includes a pattern relating to motion amplitude (Ap) of the candidate motion and blinking of the interlocking motion. 4. The saccade detection device according to claim 2, wherein the correlation pattern includes a pattern related to the motion amplitude (Ap) of the candidate motion and a combination of face orientation variation and blinking in the interlocking motion. 5. The saccade detection device according to claim 1, wherein the correlation pattern includes a pattern relating to the motion direction (Dp) of the candidate motion and a face direction change in the interlocking motion. A saccade detection method executed by a processor (6) to detect a saccade of a monitored person (2), comprising:
extracting a candidate motion that is a candidate for the saccade as the eyeball motion of the monitoring subject (S102);
extracting an interlocking motion that is assumed to be interlocked with the extracted candidate motion as the face motion of the monitoring subject excluding the eyeball motion (S103);
predicting the occurrence probability (P) of the saccade according to the correlation pattern between the extracted candidate motion and the extracted interlocking motion (S104);
adjusting a determination threshold (J) for determining whether the extracted candidate motion is the saccade according to the predicted probability of occurrence (S105);
A saccade detection method comprising judging the authenticity based on the adjusted judgment threshold (S106, S107, S108). A saccade detection program stored in a storage medium (5) and comprising instructions to be executed by a processor (6) to detect a saccade of a person to be monitored (2),
Said instruction
extracting a candidate motion that is a candidate for the saccade as the eyeball motion of the monitoring subject (S102);
extracting an interlocking motion that is assumed to be interlocked with the extracted candidate motion as the face motion of the monitoring subject excluding the eyeball motion (S103);
predicting the occurrence probability (P) of the saccade according to the correlation pattern between the extracted candidate motion and the extracted interlocking motion (S104);
adjusting a determination threshold (J) for determining whether the extracted candidate motion is the saccade according to the predicted occurrence probability (S105);
determining the authenticity based on the adjusted determination threshold (S106, S107, S108).

Images