JP2021019943A - Saccade detection device, saccade detection method, and saccade detection program - Google Patents

Abstract

To provide a saccade detection device that can improve saccade detection accuracy.SOLUTION: A saccade detection device 1 for detecting saccade of a person 2 to be monitored includes: an extraction block 10 for extracting an upwardly convex inflection point 2f, in particular, of sites interlocking with an eye movement of the person 2 to be monitored as an upper eyelid interlocking part 2e in an upper eyelid 2b reflected in a face image 7 taken of a face 2a of the person 2 to be monitored; and a detection block 20 for tracing the movement of the upper eyelid interlocking part 2e extracted by the extraction block 10 and detecting saccade on the basis of the movement tracing result.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to a technique for detecting a saccade of a monitored person.

As a conventional technique, Patent Document 1 discloses a technique for detecting a saccade for a driver who is a vehicle monitoring target. In the disclosed technique of Patent Document 1, first, the pupil region of the driver's eye is extracted from the face image showing the driver's face. Therefore, in the technique disclosed in Patent Document 1, the eyeball rotation speed is calculated by time-differentiating the eyeball rotation angle corresponding to the position of the extracted pupil region, and then the saccade is detected based on the calculation result. ..

Japanese Unexamined Patent Publication No. 2011-115450

However, in the technique disclosed in Patent Document 1, when a facial image is taken for a driver with fine eyes, it is difficult to accurately extract the pupil region where the upper eyelid and the lower eyelid overlap from the facial image. Further, in the technique disclosed in Patent Document 1, when the reflected light on the surface of the cornea is reflected in the facial image and photographed, it is difficult to accurately extract the pupil region where the Purkinier image overlaps from the facial image. In these cases, the position of the pupil region where the extraction is inaccurate is used for the saccade detection, so that the detection accuracy is lowered.

An object of the present disclosure is to provide a saccade detection device that improves the detection accuracy of saccades. Another object of the present disclosure is to provide a saccade detection method that improves the accuracy of saccade detection. Yet another object of the present disclosure is to provide a saccade detection program that improves the accuracy of saccade detection.

Hereinafter, the technical means of the present disclosure for solving the problems will be described. The scope of claims and the reference numerals in parentheses described in this column indicate the correspondence with the specific means described in the embodiment described in detail later, and limit the technical scope of the present disclosure. It is not something to do.

The first aspect of the present disclosure is
A saccade detection device (1) that detects the saccade of the monitored person (2).
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), an extraction block that extracts a part linked to the eye movement of the monitored person as an upper eyelid interlocking part (2e). (10) and
A detection block (20, 2020) that tracks the movement of the upper eyelid interlocking portion extracted by the extraction block and detects the saccade based on the movement tracking result is provided.

The second aspect of the present disclosure is
It is a saccade detection method that is executed by the processor (6) to detect the saccade of the monitored person (2).
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), the portion linked to the eye movement of the monitored person is extracted as the upper eyelid interlocking portion (2e) (S101). ~ S105),
This includes tracking the movement of the extracted upper eyelid interlocking portion and detecting the saccade based on the movement tracking result (S106 to S110, S211 and S2112).

The third aspect of the present disclosure is
A saccade detection program that is stored in a storage medium (5) to detect a saccade of a monitored person (2) and includes an instruction to be executed by a processor (6).
The instruction is
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), the portion linked to the eye movement of the monitored person is extracted as the upper eyelid interlocking portion (2e) (S101). ~ S105),
This includes tracking the movement of the extracted upper eyelid interlocking portion and detecting the saccade based on the movement tracking result (S106 to S110, S211 and S2112).

According to these first to third aspects, the upper eyelid interlocking portion interlocking with the eye movement is extracted from the upper eyelid in which the face of the monitored person is reflected in the photographed face image. At this time, the fineness of the eyes of the person to be monitored and the reflected light on the surface of the cornea are unlikely to affect the imaging of the upper eyelid, so that the upper eyelid interlocking part, which is a specific part of the upper eyelid, is accurate from the facial image. Easy to extract. Therefore, it is possible to improve the detection accuracy of the saccade based on the motion tracking result of the upper eyelid interlocking portion extracted accurately.

It is a block diagram which shows the whole structure of the saccade detection apparatus by 1st Embodiment. It is a block diagram which shows the detailed structure of the saccade detection apparatus by 1st Embodiment. It is a schematic diagram for demonstrating the information acquisition unit of 1st Embodiment. It is a schematic diagram for demonstrating the feature extraction part of 1st Embodiment. It is a schematic diagram for demonstrating the feature correction part of 1st Embodiment. It is a schematic diagram for demonstrating the feature correction part of 1st Embodiment. It is a schematic diagram for demonstrating the opening / closing determination part of 1st Embodiment. It is a schematic diagram for demonstrating the interlocking extraction part of 1st Embodiment. It is a schematic diagram for demonstrating the interlocking extraction part of 1st Embodiment. It is a schematic diagram for demonstrating the interlocking extraction part of 1st Embodiment. It is a schematic diagram for demonstrating the motion tracking part and the timing detection part of 1st Embodiment. It is a schematic diagram for demonstrating the amplitude detection part of 1st Embodiment. It is a flowchart which shows the saccade detection method by 1st Embodiment. It is a block diagram which shows the detailed structure of the saccade detection apparatus by 2nd Embodiment. It is a flowchart which shows the saccade detection method by 2nd Embodiment.

Hereinafter, a plurality of embodiments will be described with reference to the drawings. By assigning the same reference numerals to the corresponding components in each embodiment, duplicate description may be omitted. When only a part of the configuration is described in each embodiment, the configurations of the other embodiments described above can be applied to the other parts of the configuration. Further, not only the combination of the configurations specified in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined even if the combination is not specified.

(First Embodiment)
As shown in FIG. 1, the saccade detection device 1 according to the first embodiment detects saccades for at least one of the left and right eyeballs of the monitored person 2. The saccade may include a short-time, high-speed eye movement that occurs when the monitored person 2 intentionally changes the line-of-sight direction. The saccade may include tremor and rapid eye movements that occur involuntarily during gaze (ie, fixation) by the monitored subject 2.

The saccade detection device 1 may be mounted on the vehicle to detect the saccade of the occupant as the monitored person 2. In this case, the detection result of the saccade is used, for example, for driving control of the vehicle. The saccade detection device 1 may detect the saccade of the user as the monitored person 2 by being connected to the computer. In this case, the saccade detection result is used, for example, for computer operation control. Of course, the application form of the saccade detection device 1 may be a form other than these.

The saccade detection device 1 includes an information acquisition unit 3 and an electronic control unit 4. The information acquisition unit 3 acquires the face information I of the monitored person 2. The acquisition of face information I by the information acquisition unit 3 is executed according to an instruction from the monitored person 2 or another person, or according to a control from the electronic control unit 4 or another unit. The face information I is information relating to the state of a plurality of parts of the face 2a of the monitored person 2.

As shown in FIG. 2, the information acquisition unit 3 is mainly composed of an image pickup device 3a such as a camera that captures the face 2a of the monitored person 2 from a predetermined fixed point. As shown in FIG. 3, in the information acquisition unit 3, the face image 7 of the monitored person 2 generated as a result of shooting by the imaging device 3a is acquired as the face information I. In the face image 7 generated by the image pickup apparatus 3a, at least the upper eyelid 2b, the inner corner 2c, and the outer corner 2d of the eyes are projected in the face 2a of the monitored person 2.

When the saccade detection device 1 is mounted on a vehicle, an image pickup device such as an in-vehicle camera mounted on the vehicle may be used as the image pickup device 3a of the information acquisition unit 3. When the information acquisition unit 3 uses the image pickup device 3a mounted on the vehicle, the saccade detection device 1 may be composed of only the electronic control unit 4.

The electronic control unit 4 is connected to the information acquisition unit 3 via, for example, at least one of a LAN (Local Area Network), a wire harness, an internal bus, and the like. As shown in FIG. 1, the electronic control unit 4 is a dedicated computer including at least one memory 5 and one processor 6.

The memory 5 non-transitory tangible storage of at least one kind of non-transitory tangible storage, for example, a semiconductor memory, a magnetic medium, an optical medium, or the like, which stores a computer-readable program and data non-temporarily. medium). The processor 6 includes, for example, at least one of a CPU (Central processing unit), a GPU (Graphics Processing Unit), a RISC (Reduced Instruction Set Computer) -CPU, and the like as a core.

The processor 6 executes a plurality of instructions included in the soccerd detection program stored in the memory 5. As a result, the electronic control unit 4 constructs a plurality of functional blocks for detecting the soccer of the monitored person 2. In this way, in the electronic control unit 4, a plurality of functional blocks are constructed by causing the processor 6 to execute a plurality of instructions by the saccade detection program stored in the memory 5 in order to detect the saccade of the monitored person 2. Will be done.

As shown in FIG. 2, the functional block constructed by the electronic control unit 4 includes an extraction block 10 and a detection block 20. The extraction block 10 has a feature extraction unit 11, a feature correction unit 12, an open / close determination unit 13, and an interlocking extraction unit 14 as sub-function blocks.

The face information I acquired by the information acquisition unit 3 is input to the feature extraction unit 11. The feature extraction unit 11 detects the feature points of the face 2a in the monitored person 2 based on the face information I. The feature extraction unit 11 extracts a plurality of feature points by image processing such as deep learning or pattern matching based on the face image 7 of the face information I. As shown by the white circles in FIG. 4, the plurality of feature points extracted based on the face image 7 include the coordinate points that specify the upper eyelid 2b, the inner corner 2c, and the outer corner 2d of the face 2a of the monitored person 2. , At least included.

The feature extraction unit 11 shown in FIG. 2 generates feature data S representing a plurality of extracted feature points. The extraction of the feature points and the generation of the feature data S are executed in chronological order for each acquisition frame of the face information I by the information acquisition unit 3. Therefore, the feature extraction unit 11 stores the generated feature data S in the memory 5 in association with the corresponding face information I and the time information (that is, the time stamp).

The face information I and the feature data S stored by the feature extraction unit 11 are input to the feature correction unit 12. The feature correction unit 12 corrects the horizontality and frontality of the upper eyelid 2b reflected in the face image 7 of the face information I with respect to the feature data S. The horizontality correction and the frontal degree correction of the upper eyelid 2b are executed in chronological order for each acquisition frame of the face information I. Therefore, the feature correction unit 12 updates the face information I and the feature data S stored in the memory 5 based on the correction results of the horizontal degree and the front degree.

In the levelness correction of the upper eyelid 2b, the feature correction unit 12 sets the coordinates of each feature point reflected in the face image 7 in the feature data S as shown by a white circle in FIG. 5, at a position where the inner corner 2c and the outer corner 2d are horizontal. Rotate to normal coordinates when it becomes a relationship. At this time, the normal coordinates are estimated based on, for example, face information I representing the tilt angle of the face 2a. Such horizontality correction to the normal coordinates makes it possible to reduce the deviation of the feature points due to the inclination of the face 2a.

In the frontal degree correction of the upper eyelid 2b, the feature correction unit 12 sets the coordinates of each feature point reflected in the face image 7 in the feature data S as shown by the white circle in FIG. 6 with the face 2a of the monitored person 2 in front. Convert to normal coordinates when facing. At this time, the normal coordinates are estimated based on, for example, face information I representing the horizontal rotation angle of the face 2a, the degree of eye opening, the eyeball diameter, and the like. Such frontal correction to normal coordinates is performed on the upper eyelid 2b that can be approximated to the deviation of the feature points due to the orientation of the face 2a and the degree of eye opening, particularly the three-dimensional line segment L on the spherical eyeball (see FIG. 9 described later). The deviation caused by distortion can be reduced.

As shown in FIG. 2, the face information I and the feature data S stored and updated by the feature correction unit 12 are input to the open / close determination unit 13. The open / close determination unit 13 determines the open / close of the upper eyelid 2b reflected in the face image 7 of the face information I. The opening / closing determination of the upper eyelid 2b is executed in chronological order for each acquisition frame of the face information I. Therefore, the open / close determination unit 13 stores the feature data S and the face information I associated with the time information in the memory 5 in association with a flag indicating the open / close determination result.

The opening / closing determination of the upper eyelid 2b by the opening / closing determination unit 13 is performed by each coordinate of a plurality of feature points (that is, each coordinate point) in which the upper eyelid 2b reflected in the face image 7 is specified in the feature data S as shown by a white circle in FIG. )based on. From each coordinate of these plurality of feature points, it is possible to determine whether or not the shape of the upper eyelid 2b on the face 2a of the monitored person 2 is an upward convex shape. Therefore, when the shape of the upper eyelid 2b is an upward convex shape, the open / close determination unit 13 determines that the upper eyelid 2b is in an open eye state (hereinafter, referred to as an open determination). On the other hand, when the shape of the upper eyelid 2b is a downward convex shape (see FIG. 7) or a substantially linear shape, the determination that the upper eyelid 2b is in the closed eye state (hereinafter referred to as the closed determination) is determined to be open / closed. Part 13 is down.

As shown in FIG. 2, the face information I and the feature data S updated by the feature correction unit 12 are input to the interlocking extraction unit 14 in accordance with the opening determination by the open / close determination unit 13. The interlocking extraction unit 14 extracts, as shown in FIGS. 8 and 9, a portion of the face information I that is interlocked with the eye movement of the monitored person 2 in the upper eyelid 2b reflected in the face image 7 as the upper eyelid interlocking unit 2e. .. The extraction of the upper eyelid interlocking unit 2e is executed in chronological order for each frame in which the opening determination of the upper eyelid 2b is made among the acquisition frames of the face information I. Therefore, the interlocking extraction unit 14 stores the extraction result of the upper eyelid interlocking unit 2e in association with the feature data S and the face information I associated with the time information together with the open determination flag in the memory 5.

The upper eyelid interlocking unit 2e extracted by the interlocking extraction unit 14 is approximated from the plurality of feature points (see the black circles in the same figure) of the upper eyelid 2b reflected in the face image 7 in the feature data S as shown in FIG. On the three-dimensional line segment L, it is set to an inflection point (that is, an upper vertex) 2f that is convex upward. This depends on the fact that the feature point in which the amount of interlocking with the eye movement is substantially maximum in the upper eyelid 2b is the inflection point 2f. Therefore, in the extraction of the upper eyelid interlocking unit 2e, the interlocking extraction unit 14 determines the coordinates of the inflection point 2f reflected in the face image 7 based on the three-dimensional line segment L as shown by the white circles in FIGS. 9 and 10. At this time, the three-dimensional line segment L is converted into a three-dimensional or higher two-dimensional regression curve that passes through the coordinates of the plurality of feature points of the upper eyelid 2b, the inner corner of the eye 2c, and the outer corner of the eye 2d, thereby forming the three-dimensional line segment L on the two-dimensional regression curve. The coordinates of the curve point 2f are fixed.

In the extraction of the upper eyelid interlocking unit 2e, the interlocking extraction unit 14 has an inflection point 2f from one of the inner corner 2c and the outer corner 2d (FIG. 10 is an example of the inner corner 2c) shown in the face image 7 as shown in FIG. The separation distance in the horizontal direction up to is calculated as the position specific distance Dp. It can be said that the position specifying distance Dp specifies the position of the upper eyelid interlocking portion 2e, which is the inflection point 2f, starting from one of the inner corner 2c and the outer corner 2d of the eye. In the extraction of the upper eyelid interlocking unit 2e, the interlocking extraction unit 14 calculates the horizontal separation distance between the inner corner 2c and the outer corner 2d of the eye shown in the face image 7 as the reference distance Db as shown in FIG. These distances Dp and Db are calculated based on the coordinates of the upper eyelid interlocking portion 2e, the inner corner 2c, and the outer corner 2d of the feature data S.

In the extraction of the upper eyelid interlocking unit 2e, the interlocking extraction unit 14 makes the same distance Dp non-dimensional by dividing the position specific distance Dp by the reference distance Db as shown in FIG. The interlocking extraction unit 14 stores the non-dimensionalized position specific distance (hereinafter referred to as the non-dimensionalization distance) Dp / Db in the memory 5 as the extraction result of the upper eyelid interlocking unit 2e.

The face information I and the feature data S stored and updated by the feature correction unit 12 are not input to the interlocking extraction unit 14 shown in FIG. 2 in accordance with the closing determination by the opening / closing determination unit 13. Therefore, among the frames for acquiring the face information I, the interlocking extraction unit 14 does not extract the upper eyelid interlocking unit 2e in the frame for which the closing determination is made. That is, when the monitored person 2 closes his / her eyes, the extraction of the upper eyelid interlocking unit 2e by the interlocking extraction unit 14 is skipped. As a result, when the eyes of the monitored person 2 are closed, the detection of the saccade based on the motion tracking result of the upper eyelid interlocking unit 2e, which is premised on the extraction of the upper eyelid interlocking unit 2e, is also skipped.

The detection block 20 has a motion tracking unit 21, a timing detection unit 22, and an amplitude detection unit 23 as sub-functional blocks. The detection block 20 jointly of these units 21, 22, and 23 tracks the movement of the upper eyelid interlocking unit 2e extracted by the interlocking extraction unit 14, and detects the soccer of the monitored person 2 based on the movement tracking result. .. At this time, the eyeball in which the saccade is detected may be at least one of the left and right eyes. Therefore, for example, the saccade may be detected by limiting the upper eyelid interlocking portion 2e to the one-sided eyeball that can be easily extracted based on the installation position of the imaging device 3a or the like. On the other hand, assuming that it is difficult to extract the upper eyelid interlocking portion 2e with one eyeball based on the orientation of the face 2a, for example, both eyeballs are selected as saccade detection targets. The detection results for each eyeball may be synthesized.

The extraction result of the upper eyelid interlocking unit 2e stored by the interlocking extraction unit 14 in accordance with the opening determination by the opening / closing determination unit 13 is input to the motion tracking unit 21. The motion tracking unit 21 tracks the motion speed based on the non-dimensionalization distance Dp / Db of the upper eyelid interlocking unit 2e extracted by the interlocking extraction unit 14 in chronological order, and then saccades based on the speed tracking result. Executes the judgment of. These movement speed tracking and soccerd determination are executed in chronological order for each extracted frame of the upper eyelid interlocking portion 2e according to the opening determination of the upper eyelid 2b among the acquisition frames of the face information I. .. Therefore, the motion tracking unit 21 stores the speed tracking result according to the extraction result of the upper eyelid interlocking unit 2e in the memory 5 in association with the flag representing the determination result of the saccade.

In the speed tracking of the upper eyelid interlocking unit 2e, the motion tracking unit 21 sets the tracking period from each tracking execution frame to the past tracking execution frame retroactive in the set time. At this time, the tracking period is set to the time required to track the movement speed of the upper eyelid interlocking portion 2e and detect the saccade in the horizontal direction. Therefore, the motion tracking unit 21 calculates the motion speed of the upper eyelid interlocking unit 2e in each of the multiple frames within the tracking period based on the dimensionless distance Dp / Db stored in the memory 5. At this time, the motion tracking unit 21 may suppress the fluttering of the calculation result by calculating the motion speed of the upper eyelid interlocking unit 2e through a filter that averages a predetermined number of frames. The motion tracking unit 21 executes the soccerd determination process by tracking the motion speed of the upper eyelid interlocking unit 2e calculated for each frame within the tracking period as shown in FIG.

In the determination of saccade, the movement tracking unit 21 determines the presence or absence of saccade detection based on the time transition indicated by the movement speed of the upper eyelid interlocking unit 2e as the speed tracking result within the tracking period. Specifically, the motion tracking unit 21 sets the first to third velocity thresholds V1 to V3 shown in FIG. At this time, the second speed threshold value V2 is set to be larger than the first speed threshold value V1 and smaller than the third speed threshold value V3. Therefore, after the movement speed of the upper eyelid interlocking portion 2e exceeds the first speed threshold value V1, further exceeds the second speed threshold value V2, and then does not exceed the third speed threshold value V3, the second speed threshold value V2 and When a specific transition below the first velocity threshold value V1 is shown, the motion tracking unit 21 determines whether or not the soccerd is detected. On the other hand, when the motion speed of the upper eyelid interlocking unit 2e indicates a transition other than this specific transition, the motion tracking unit 21 determines that the saccade has not been detected.

In the timing detection unit 22 shown in FIG. 2, the speed tracking result of the upper eyelid interlocking unit 2e stored by the movement tracking unit 21 is input in accordance with the detection presence determination by the unit. The timing detection unit 22 detects the start timing Ts and the end timing Te of the saccade based on the time transition shown in FIG. These timing detections are performed in chronological order for each frame in which the face information I acquisition frame is determined to be detected through speed tracking from the extraction of the upper eyelid interlocking unit 2e according to the opening determination of the upper eyelid 2b. Is executed. Therefore, the timing detection unit 22 stores the timing detection result in the memory 5 in association with the speed tracking result and the detection presence determination flag associated with each other.

In the timing detection, the timing detection unit 22 estimates that the timing at which the movement speed of the upper eyelid interlocking unit 2e exceeds the first speed threshold value V1 is the start timing Ts of the saccade. On the other hand, in the timing detection, the timing detection unit 22 estimates that the timing when the movement speed of the upper eyelid interlocking unit 2e falls below the first speed threshold value V1 is the end timing Te of the saccade.

In the amplitude detection unit 23 shown in FIG. 2, the speed tracking result and the timing detection result stored in association with each other by the timing detection unit 22 in accordance with the detection presence determination are input. The amplitude detection unit 23 detects the amplitude of the saccade by based on the dimensionless distance Dp / Db stored in the memory 5 at the frames corresponding to the start timing Ts and the end timing Te, respectively. The detection of the saccade amplitude is performed for each frame in which the timing detection is executed from the extraction of the upper eyelid interlocking portion 2e according to the opening determination of the upper eyelid 2b, the speed tracking and the detection presence determination among the acquisition frames of the face information I. , Executed in chronological order. Therefore, the amplitude detection unit 23 stores the amplitude detection result in the memory 5 in association with the speed tracking result and the detection presence determination flag associated with the timing detection result.

In detecting the saccade amplitude, the amplitude detection unit 23 calculates the motion amplitude of the upper eyelid interlocking unit 2e between the start timing Ts and the end timing Te. As shown in FIG. 12, the motion amplitude of the upper eyelid interlocking portion 2e is the difference between the dimensionless distance Dp / Db in the corresponding frame of the start timing Ts and the dimensionless distance Dp / Db in the corresponding frame of the end timing Te. Calculated by ΔD.

In detecting the saccade amplitude, the amplitude detection unit 23 calculates the saccade amplitude based on the calculated motion amplitude of the upper eyelid interlocking unit 2e. Specifically, the amplitude detection unit 23 converts the motion amplitude of the upper eyelid interlocking unit 2e into the rotation angle of the eyeball of the monitored person 2. At this time, the conversion coefficient that determines the conversion function from the motion amplitude to the rotation angle depends on the unique dimensions of each of the monitored subjects 2, such as the size of the upper eyelid 2b and the eyeball diameter. Therefore, the conversion coefficient may be set by performing calibration for each individual of the monitored person 2, or may be set based on the assumed average size of the monitored person 2. In the former calibration, the monitored person 2 is made to visually recognize two horizontally separated points without moving the face 2a, and the distance between the inflection points 2f at the time of visually recognizing each point and the eyeball are the centers. The inflection factor may be set based on the horizontal angle between each point.

The flow of the soccerd detection method in which the electronic control unit 4 detects the soccerd of the monitored person 2 by the joint use of the blocks 10 and 20 described so far will be described below with reference to FIG. This flow is executed for each acquisition frame of face information I by the information acquisition unit 3. In the flow described later, "S" means a plurality of steps of the flow executed by a plurality of instructions included in the saccade detection program.

In S101, the feature extraction unit 11 extracts a plurality of feature points of the face 2a of the monitored person 2 based on the face information I acquired by the information acquisition unit 3. In S101, the feature extraction unit 11 further generates feature data S representing the extracted plurality of feature points.

Next, in S102, the horizontality of the upper eyelid 2b reflected in the face image 7 of the face information I is corrected with respect to the feature data S by the feature correction unit 12. In the following S103, the front degree of the upper eyelid 2b reflected in the face image 7 of the face information I is corrected with respect to the feature data S by the feature correction unit 12.

Further, in S104, the opening / closing of the upper eyelid 2b reflected in the face image 7 of the face information I is determined by the opening / closing determination unit 13. As a result, when the closing determination of the upper eyelid 2b is made, the extraction of the upper eyelid interlocking portion 2e and the detection of the saccade are skipped by ending the flow this time. On the other hand, when the upper eyelid 2b is determined to be open, the process proceeds to S105.

In S105, in the upper eyelid 2b reflected in the face image 7 of the face information I, the inflection point 2f linked with the eye movement of the monitored subject 2 is extracted by the interlocking extraction unit 14 as the upper eyelid interlocking unit 2e. At this time, the position-specific distance Dp that specifies the position of the extracted upper eyelid interlocking portion 2e is made dimensionless toward the motion tracking of the upper eyelid interlocking portion 2e.

In the following S106, the motion speed of the upper eyelid interlocking unit 2e extracted in the current flow and the motion velocity of the upper eyelid interlocking unit 2e extracted from the current flow in the past flow within the tracking period are combined with the motion tracking unit 21. Calculated by At this time, in the calculation of the motion velocity, the non-dimensionalization distance Dp / Db is used as the position-specific distance Dp for each frame made non-dimensional by the reference distance Db. As a result, the motion speed in the current frame and the motion speed in the past plurality of frames within the tracking period retroactively from the current frame are calculated as the motion tracking target of the upper eyelid interlocking portion 2e.

Further, in S107, the motion speed of the upper eyelid interlocking unit 2e calculated for each frame within the tracking period is tracked in chronological order by the motion tracking unit 21. Further in S108, the motion tracking unit 21 executes the detection determination of the saccade based on the speed tracking result of the upper eyelid interlocking unit 2e. As a result, if it is determined that the saccade has not been detected, the flow ends this time. On the other hand, if it is determined that the saccade is detected, the process proceeds to S109.

In S109, the timing detection unit 22 detects the start timing Ts and the end timing Te of the saccade based on the speed tracking result of the upper eyelid interlocking unit 2e. In the following S110, the saccade amplitude is detected by the amplitude detection unit 23 based on the motion amplitude of the upper eyelid interlocking unit 2e between the frames corresponding to the start timing Ts and the end timing Te, respectively. At this time, the dimensionless distance Dp / Db is used for the calculation of the motion amplitude that is executed first to detect the soccerd amplitude. As described above, when the timing detection and the amplitude detection of the saccade are completed in sequence, the flow ends this time.

(Action effect)
The effects of the first embodiment described above will be described below.

According to the first embodiment, in the upper eyelid 2b in which the face 2a of the monitored person 2 is reflected in the photographed face image 7, the upper eyelid interlocking portion 2e linked with the eye movement is extracted. At this time, since the fineness of the eyes of the monitored person 2 and the reflected light on the surface of the cornea are unlikely to affect the imaging of the upper eyelid 2b, the upper eyelid interlocking portion 2e, which is a specific part of the upper eyelid 2b, is used. It is easy to accurately extract from the face image 7. Further, the upper eyelid interlocking portion 2e can be easily extracted from the face image 7 with higher contrast than the pupil region of the eyeball. From these facts, it is possible to improve the detection accuracy of the saccade based on the motion tracking result of the upper eyelid interlocking portion 2e extracted accurately.

According to the first embodiment, among the upper eyelids 2b reflected in the face image 7, the inflection point 2f having a large interlocking amount with the eye movement is extracted as the upper eyelid interlocking portion 2e. According to this, it is possible to improve the extraction accuracy of the upper eyelid interlocking portion 2e and the detection accuracy of the saccade based on the motion tracking result of the upper eyelid interlocking portion 2e.

According to the first embodiment, among the physical quantities representing the movement of the upper eyelid interlocking portion 2e, paying attention to the movement speed at which the difference is likely to appear depending on the presence or absence of detection of the saccade, it is based on the tracking result for the movement speed. Then, the detection of the saccade will be executed. According to this, since the presence or absence of detection of saccade can be accurately determined, it is possible to improve the detection accuracy of saccade.

According to the first embodiment, the position specifying distance Dp that specifies the position of the upper eyelid interlocking portion 2e starting from one of the inner corner 2c and the outer corner 2d reflected in the face image 7 is the reference distance Db between the inner corner 2c and the outer corner 2d. Is made dimensionless by. Here, assuming that the face 2a of the monitored person 2 moves back and forth without eye movement, the fluctuation rates of the position-specific distance Dp and the reference distance Db in the face image 7 substantially match. The resulting error in the position-specific distance Dp can be canceled by the non-dimensionalization. Therefore, according to the fact that the movement speed of the upper eyelid interlocking portion 2e is tracked based on the non-dimensionalized position-specific distance Dp (that is, the non-dimensionalization distance Dp / Db), soccer based on the tracking result of the movement speed It is possible to improve the detection accuracy of the data.

According to the first embodiment, in the monitored person 2, the horizontality of the upper eyelid 2b is easily affected by the inclination of the face 2a. Therefore, if the horizontality of the upper eyelid 2b reflected in the face image 7 is corrected and then the upper eyelid interlocking portion 2e is extracted, the extraction error due to the inclination of the face 2a can be reduced. Therefore, it is possible to improve the detection accuracy of the saccade based on the motion tracking result of the extracted upper eyelid interlocking portion 2e.

According to the first embodiment, in the monitored person 2, the frontal degree of the upper eyelid 2b is easily affected by the orientation of the face 2a and the degree of eye opening. Therefore, the frontness of the upper eyelid 2b reflected in the face image 7 is corrected, and then the upper eyelid interlocking portion 2e is extracted. Therefore, the upper eyelid interlocking portion 2e is extracted due to the orientation of the face 2a and the degree of eye opening. The error can be reduced. Therefore, it is possible to improve the detection accuracy of the saccade based on the motion tracking result of the extracted upper eyelid interlocking portion 2e.

(Second Embodiment)
As shown in FIG. 14, the second embodiment is a modification of the first embodiment.

The motion tracking unit 2021 of the detection block 2020 according to the second embodiment executes a blink (that is, blink) determination between the tracking of the exercise speed and the determination of the saccade. In the motion tracking unit 2021 due to the determination of the blink of an eye, the face information I and the feature data S stored and updated by the feature correction unit 12 and the closing determination result of the upper eyelid 2b stored as a flag by the opening / closing determination unit 13 in the past. Is entered. In the determination of the blink of an eye, the motion tracking unit 2021 pays attention to the closing determination frame that has been determined to be closed in the past by the opening / closing determination unit 13 among the plurality of frames within the tracking period.

The eye closure determined by the open / close determination unit 13 includes eye closure associated with a blink within the permissible time in which the saccade can be detected, and eye closure in which the saccade is difficult to detect and exceeds the permissible time. Therefore, in the determination of the blink of an eye, in addition to being based on the speed tracking result, the motion tracking unit 2021 determines the shape of the upper eyelid 2b reflected in the face image 7 in the closing determination frame and the number of closing determination frames (that is, closed). Based on the determination period), the presence or absence of blink detection within the tracking period is determined. When the blink is not detected, the motion tracking unit 2021 executes the saccade determination process as in the first embodiment. At this time, if the blinking is not detected due to the occurrence of eye closure exceeding the allowable time within the tracking period, the movement speed of the upper eyelid interlocking portion 2e indicates a transition other than the specific transition, so that the soccer is not performed. The determination of detection will be made.

When the blink detection is present, the motion tracking unit 2021 executes a saccade determination process different from that of the first embodiment. Specifically, the motion tracking unit 2021 extracts one frame before and after the frame in which the blink is generated from the plurality of frames within the tracking period as the front and rear frames. The motion tracking unit 2021 compares the dimensionless distances Dp / Db stored in the memory 5 in each of the extracted front and rear frames as the motion tracking result of the upper eyelid interlocking unit 2e. As a result, when the difference ΔD of the non-dimensionalization distance Dp / Db between the front and rear frames is out of the permissible range, the motion tracking unit 2021 changes the position of the upper eyelid interlocking unit 2e due to the saccade during the blink of an eye. By presuming that it has not occurred, it is determined that the saccade has not been detected. On the other hand, when the difference ΔD of the non-dimensionalization distance Dp / Db between the front and rear frames is within the permissible range, the motion tracking unit 2021 causes the position change of the upper eyelid interlocking unit 2e due to the saccade during the blink of an eye. By presuming that it was, it is judged that there is saccade detection. At this time, the amplitude detection unit 23 of the detection block 2020 according to the second embodiment mimics the difference ΔD of the non-dimensionalization distance Dp / Db between the front and rear frames with the motion amplitude of the upper eyelid interlocking unit 2e. The saccade amplitude is calculated in the same manner as in the embodiment.

The flow of the saccade detection method according to the second embodiment will be described below with reference to FIG. In the flow of the second embodiment, S101 to S107 are executed in the same manner as in the first embodiment.

In 2111 following S107 in the flow of the second embodiment, the motion tracking unit 2021 executes the detection determination based on the speed tracking result of the blink of an eye within the tracking period. As a result, when it is determined that the blink is not detected, S108 to S110 are executed as in the first embodiment. On the other hand, if it is determined that the blink detection is present, the process proceeds to S2112.

In S2112, by comparing the dimensionless distances Dp / Db between the front and rear frames sandwiching the blink generation frame within the tracking period, the detection determination of the saccade based on the difference ΔD of the distances Dp / Db is performed. It is executed by the exercise tracking unit 2021. At this time, whether or not the saccade is detected is based on the non-dimensionalization distance Dp / Db between the front and rear frames, which is the motion tracking result of the upper eyelid interlocking portion 2e, and the difference ΔD between them is the position of the upper eyelid interlocking portion 2e by the saccade. It is determined by estimating whether or not it is fluctuating. As a result, if it is determined that the saccade has not been detected, the flow ends this time. On the other hand, when it is determined that the saccade is detected, the difference ΔD of the non-dimensional distance Dp / Db between the front and rear frames is imitated as the motion amplitude of the upper eyelid interlocking portion 2e, so that the first implementation is performed. S110 is executed in the same manner as in the form.

(Action effect)
The effects of the second embodiment described above will be described below.

According to the second embodiment, the saccade detection associated with the blink of the monitored person 2 is based on the motion tracking result of the upper eyelid interlocking portion 2e extracted before and after the blink, and the position change of the upper eyelid interlocking portion 2e by the saccade. Is realized by estimating. According to this, even if the extraction of the upper eyelid interlocking portion 2e is skipped due to the closing of the eyes accompanying the blinking of the monitored person, the motion tracking result of the upper eyelid interlocking portion 2e before and after the blinking is effectively utilized. , Saccade can be detected presumably. Therefore, it is possible to suppress poor detection of saccades caused by disturbances such as blinks and enhance robustness.

(Other embodiments)
Although the plurality of embodiments have been described above, the present disclosure is not construed as being limited to those embodiments, and is applied to various embodiments and combinations within the scope of the gist of the present disclosure. Can be done.

The electronic control unit 4 of the modification according to the first and second embodiments may be a dedicated computer configured to include at least one of a digital circuit and an analog circuit as a processor. Here, particularly digital circuits include, for example, ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), CPLD (Complex Programmable Logic Device) and the like. Of these, at least one. Further, such a digital circuit may include a memory for storing a program.

In the modified examples of the first and second embodiments, at least one of the horizontality correction of S102 and the frontalness correction of S103 may be skipped. In the modification according to the first embodiment, the opening / closing determination of S104 may be skipped.

In the modified examples relating to the first and second embodiments, feature points other than the inflection point 2f in the upper eyelid 2b may be extracted as the upper eyelid interlocking portion 2e. In the modified examples relating to the first and second embodiments, the position specifying distance Dp is not made dimensionless in S105, and the value as it is may be used in S106, S110, and S2112.

In the modified examples relating to the first and second embodiments, the result of tracking the physical quantity other than the motion velocity representing the motion of the upper eyelid interlocking portion 2e may be used for the soccerd detection in S106 to S110, S2111, S2112. .. At this time, for physical quantities other than the motion speed, for example, the coordinates of the upper eyelid interlocking portion 2e, the position specific distance Dp, the non-dimensionalization distance Dp / Db, the motion acceleration, and the like are adopted.

In the modified examples relating to the first and second embodiments, the amplitude detection of S110 is skipped, and when the saccade detection determination and the timing detection are sequentially executed in S108 and S109, the flow ends this time. May be good. In the modified examples relating to the first and second embodiments, the timing detection of S109 and the amplitude detection of S110 are both skipped, so that if the saccade detection determination is executed in S108, even if the flow ends this time. Good.

1 Saccade detector, 2 Monitored person, 2a face, 2b upper eyelid, 2c inner corner, 2d outer corner of eye, 2e upper eyelid interlocking part, 2f inflection point, 5 memory, 6 processor, 7 face image, 10 extraction block, 20 , 2020 detection block, Db reference distance, Dp position specific distance

Claims (10)

A saccade detection device (1) that detects the saccade of the monitored person (2).
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), a portion linked to the eye movement of the monitored person is extracted as the upper eyelid interlocking portion (2e). Extraction block (10) and
A soccerd detection device including a detection block (20, 2020) that tracks the movement of the upper eyelid interlocking portion extracted by the extraction block and detects the soccerd based on the movement tracking result. The soccerd detection device according to claim 1, wherein the upper eyelid interlocking portion is an inflection point (2f) of the upper eyelid reflected in the face image. The saccade detection device according to claim 1 or 2, wherein the detection block detects the saccade based on a tracking result with respect to a movement speed of the upper eyelid interlocking portion. The position specifying distance (Dp) for specifying the position of the upper eyelid interlocking portion starting from one of the inner and outer corners of the eye reflected in the face image is dimensionlessed by the reference distance (Db) between the inner and outer corners of the eye.
The saccade detection device according to claim 3, wherein the detection block tracks the motion speed based on the non-dimensionalized position-specific distance (Dp / Db). The saccade detection device according to any one of claims 1 to 4, wherein the extraction block corrects the horizontality of the upper eyelid reflected in the face image, and then extracts the upper eyelid interlocking portion. The saccade detection device according to any one of claims 1 to 5, wherein the extraction block corrects the frontality of the upper eyelid reflected in the face image, and then extracts the upper eyelid interlocking portion. The saccade detection device according to any one of claims 1 to 6, wherein the extraction block skips extraction of the upper eyelid interlocking portion when the eyelid of the monitored person reflected in the face image is closed. With the blink of the monitored person, the detection block (2020) is the upper eyelid by the saccade based on the motion tracking result of the upper eyelid interlocking portion extracted before and after the blink by the extraction block. The saccade detection device according to claim 7, wherein the saccade is detected by estimating the position variation of the interlocking unit. It is a saccade detection method that is executed by the processor (6) to detect the saccade of the monitored person (2).
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), a portion linked to the eye movement of the monitored person is extracted as an upper eyelid interlocking portion (2e). (S101 to S105),
A saccade detection method comprising tracking the extracted movement of the upper eyelid interlocking portion and detecting the saccade based on the movement tracking result (S106 to S110, S2111, S2112). A saccade detection program that is stored in a storage medium (5) to detect a saccade of a monitored person (2) and includes an instruction to be executed by a processor (6).
The command is
In the upper eyelid (2b) in which the face (2a) of the monitored person is reflected in the captured face image (7), a portion linked to the eye movement of the monitored person is extracted as the upper eyelid interlocking portion (2e). (S101 to S105),
A soccerd detection program including tracking the movement of the extracted upper eyelid interlocking portion and detecting the saccade based on the movement tracking result (S106 to S110, S2111, S2112).

Images