JP4452833B2 - Gaze movement detection method and gaze movement detection apparatus - Google Patents

Description

  The present invention relates to a line-of-sight movement detection method and a line-of-sight movement detection device for detecting line-of-sight movement of a subject.

  2. Description of the Related Art In recent years, a technology for detecting a human gaze direction has been developed for the purpose of monitoring a driver's driving situation in a car or the like and performing a gaze operation of a personal computer. For example, a technique for calculating the gaze direction of a subject using a first camera that measures the position of the pupil of the subject and a second camera that acquires the distance and angle between the center of the pupil and the corneal reflection point Is disclosed (see Patent Document 1 below). This technique determines the line-of-sight direction from the relative positional relationship between the pupil center and the corneal reflection center in order to distinguish whether the eyeball has rotated or the head itself has moved when the pupil movement of the subject is detected. Is. This is because when the subject is looking at the optical axis direction of the camera, the positions of the pupil center and the corneal reflection center are substantially the same, and when the subject rotates the eye, the pupil center and the corneal reflection center are changed according to the rotation angle. Based on the finding that the distance between

On the other hand, a technique related to a pointing method for moving a cursor on the screen of a personal computer by a line of sight is also known (see Non-Patent Document 1 and Patent Documents 2 to 4 below). These techniques detect a pupil from a face image captured by a camera. In addition, by detecting the pupil and nostril using a camera (see Patent Document 5 below) that obtains the position of the subject's pupil center and corneal reflection center by using one camera and the two cameras. A method for detecting the direction of the head (see Patent Document 6 and Non-Patent Document 2 below) is also disclosed.
JP 2005-185431 A JP 2005-182247 A JP 2005-352580 A JP 2005-348832 A JP 2005-230049 A JP 2005-266868 A Y. Ebisawa, D. Ishima, S. Inoue, Y. Murayama, “PupilMouse: Cursor Control by Head Rotation Using Pupil Detection Technique” (USA), Proceedings of International Conference on Computing, Communications and Control Technologies: CCCT'04, August 14 -17, 2004-Austin, Texas, USA, pp. 209-214 Y. Ebisawa, Y. Nurikabe, “Face Pose Estimation Based on 3D Detection of Pupils and Nostrils”, 2005 IEEE INTERNATIONAL CONFERENCE ONVIRTUAL ENVIRONMENTS, HUMAN-COMPUTER INTERFACES, AND MEASUREMENT SYSTEMS, 2005, pp.92-97

  However, as described above, when detecting the line-of-sight direction by detecting the position of the center of corneal reflection and the center of the pupil, it is necessary to relatively increase the resolution of the camera used to increase the detection accuracy. The imaging range of the camera tends to be small. Therefore, when monitoring a driver whose head tends to move, even if the entire face needs to be photographed in a wide range, the detection range is set to a relatively narrow range in order to ensure detection accuracy. It will be limited. Further, in this case, the allowable angle between the line connecting the camera and the pupil and the line of sight is relatively narrow, about ± 30 to ± 40 degrees in the horizontal direction. This is because, as the imaging angle increases, the corneal reflection protrudes from the corneal region and the white eye shines, making it impossible to detect the corneal reflection in effect. Therefore, when monitoring the movement of the line of sight of the subject and there is no need to detect the line of sight with high accuracy, the significance of detecting corneal reflection is small. Similarly, when detecting the position of the feature point of the subject's face using two cameras, the detection range tends to be narrow.

  On the other hand, it is also possible to determine the eye movement of the subject by monitoring the position of the eye such as the pupil of the subject, but just by monitoring the pupil position when the movement of the head itself is added, It is difficult to stably determine whether the movement is caused by the movement of the line of sight or the movement of the head.

  Therefore, the present invention has been made in view of such problems, and provides a gaze movement detection method and a gaze movement detection apparatus that can stably detect a gaze movement of a subject in a wide range. Objective.

  In order to solve the above-described problem, a gaze movement detection method of the present invention is a gaze movement detection method that captures a face image of a target person and detects the presence or absence of the gaze movement of the target person based on the face image. Detecting a face image of a person for a plurality of frames, detecting a pupil position and a nostril position of the subject for each of the plurality of frames, and a pupil position and a nostril detected for the plurality of frames Based on the position, the calculation step of calculating the amount of change in the relative position of the pupil with respect to the nostril of the subject between the previous and next frames in the plurality of frames, and the line of sight using the calculated amount of change and a predetermined threshold And a determination step for determining presence or absence of movement.

  Alternatively, the line-of-sight movement detection device of the present invention is a line-of-sight movement detection device that captures a face image of a subject and detects the presence or absence of the subject's eye movement based on the face image. Based on the means for capturing a plurality of frames, detecting the pupil position and nostril position of the subject for each of the plurality of frames, and the pupil position and nostril position detected for the plurality of frames, Means for calculating the amount of change in the relative position of the pupil with respect to the nostril of the subject between frames before and after the plurality of frames, and means for determining the presence or absence of eye movement using the calculated amount of change and a predetermined threshold value With.

  According to such a line-of-sight movement detection method and line-of-sight movement detection apparatus, a plurality of frames are acquired as moving images of the subject's face image, the pupil position and nostril position in the plurality of frames are detected, and The change in the relative position of the pupil with respect to the nostril of the subject is calculated, and the presence / absence of the eye movement is determined based on the magnitude of the change, so the presence / absence of the eye movement in a wide range can be determined even for a face image using one imaging means. In addition to being able to determine, by monitoring the pupil position based on the nostril position, it is possible to realize stable detection of gaze movement even for a subject whose head itself such as a driver tends to move.

  In the calculation step, the change amount of the pupil position and the change amount of the nostril position between the previous and next frames are obtained, and the change amount of the relative position is calculated by taking the difference between the change amount of the pupil position and the change amount of the nostril position. It is preferable. In this case, it is possible to efficiently calculate the change in the relative position of the pupil and nostril while referring to the moving image of the subject, and the real-time property of the eye movement detection is improved.

  In the calculation step, the change amount of the relative position is continuously calculated over a plurality of frames, and the moving average value of the change amount is derived. In the determination step, the change amount of the relative position is calculated as the moving average value. It is also preferable to determine the presence or absence of eye movement by comparing with a threshold value set as a reference. In this way, it is possible to detect the movement of the line of sight with higher accuracy by setting an appropriate threshold value according to the magnitude of the displacement of the rotation angle of the eyeball.

  According to the line-of-sight movement detection method and line-of-sight movement detection apparatus according to the present invention, it is possible to stably detect the line-of-sight movement of a subject in a wide range.

  Hereinafter, preferred embodiments of a line-of-sight movement detection method and a line-of-sight movement detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

  The line-of-sight movement detection method and line-of-sight movement detection apparatus of the present invention are for detecting whether the direction of the eye (line-of-sight direction) of the observation subject has changed. The line-of-sight direction here refers to eye movement based on the face direction. The present invention is used, for example, to determine whether a driver or a driver is in an awake state during driving in a vehicle typified by an automobile such as a large truck. Specifically, when the driver is driving normally, the gaze changes frequently to pay attention to the surroundings, but when the driver is drowsy, the driver's gaze tends to stop, so the gaze Based on the frequency of movement, the driver's arousal state can be determined.

  First, the configuration of the eye movement detection device 1 according to the present invention will be described with reference to FIG. FIG. 1 is a plan view showing the positional relationship between the line-of-sight movement detection device 1 and the subject A. As shown in the figure, the line-of-sight movement detection device 1 includes a single camera 2 that captures the face image of the subject A and a light source 3a provided in the vicinity of the optical axis L1 of the imaging lens on the front surface 2a of the camera 2. And a light source 3b provided at a position away from the optical axis L1 of the front surface 2a of the camera 2, and an image processing terminal 4 connected to the camera 2 and processing image data generated by the camera 2. This line-of-sight movement detection apparatus 1 implements the line-of-sight movement detection method according to the present invention.

  The camera 2 is not limited to a specific type as long as it is an imaging means capable of generating the face image of the subject A, but an image sensor such as a CCD or CMOS is used in that the image data can be processed with high real-time properties. Use the built-in digital camera. The camera 2 can continuously capture the face image of the subject A as a moving image, and outputs a plurality of frame images constituting the moving image to the image processing terminal 4 in real time.

  The light source 3a is configured to be able to irradiate illumination light having a near-infrared light component toward a range covering the subject A located on the optical axis L1 along the optical axis L1 of the camera 2. The light source 3b is fixed at a position where the distance from the optical axis L1 is further away from the light source 3a, and irradiates illumination light having a near-infrared light component toward a range covering the subject A along the optical axis L1. It is configured to be possible. Here, the illumination light emitted from the light sources 3a and 3b is light having different wavelength components (for example, center wavelengths of 850 nm and 950 nm) that cause a luminance difference in the pupil portion, and the light source 3b The distance from the optical axis L1 may be fixed at a position equal to the light source 3a. In this case, the configuration of the light source can be simplified and reduced in size while causing a luminance difference in the pupil portion.

  The camera 2 and the light sources 3a and 3b are used for the purpose of preventing reflection of reflected light in the face image when the subject A is wearing glasses and easily detecting the pupil and nostril of the subject A. It is preferable to be provided at a position lower than the height of the face of the person A (for example, an inclination angle of 20 degrees to 30 degrees with respect to the horizontal plane of the optical axis L1). As the installation position, when installed in a vehicle, it may be under the dashboard on the driver's seat side, or above the dashboard when the seat position of a truck or the like is high.

  The image processing terminal 4 is an information processing terminal that performs image processing on a face image composed of a plurality of frame images output from the camera 2, and is a computer terminal that includes an arithmetic processing device such as a CPU and a storage device such as a ROM and a RAM. It is. The image processing terminal 4 executes a line-of-sight movement detection process for detecting the presence or absence of the line-of-sight movement of the subject A and controls the imaging of the subject A by the camera 2.

  Next, before describing each processing described above, the type of eye movement (rotation movement of the eyeball) of the subject will be described. In general, eye movements that can be detected when a face is photographed with a wide-field video camera include (1) saccade, (2) smooth pursuit movement, and follow-up movement. ), (3) vestibular reflex, and (4) vergence eye movement.

  Saccade is a high-speed, step-like motion that occurs when you want to change the object you want to see for a stationary object. If you are driving, this happens when you see the sign in the vicinity of the field of view while looking at the car ahead and you want to see the sign to see its details. Also, sliding eye movement occurs when trying to keep looking at a moving object. In this case, the eyeball rotation speed is smaller than that of the saccade. In vestibular eye movements, basically, when the head rotates while looking at one point, the eye rotates at the same rotational speed in the opposite direction to the rotation of the head and is projected onto the retina of the eye. This is an exercise to prevent the image from blurring. This movement also occurs in a direction in which the retinal image is not blurred even when the head is translated left and right up and down. This eye movement also has a lower eyeball rotation speed than saccade. In addition, the vergence eye movement moves both eyes when looking at nearby objects (both eyes rotate inward) and away when looking at distant objects (both eyes outward). To rotate). Therefore, this movement is a movement in which both eyes rotate in opposite directions. This eye movement is also a slow movement and is slower than saccade.

  The above four types of movements are generated while driving the vehicle, and saccades are often mixed in eye movements other than saccades. At this time, since a saccade, which is a high-speed motion, is interrupted by a relatively smooth motion other than the saccade, the saccade is easier to detect than other motions. Here, it should be noted that exercises other than saccades are usually slow motions while driving, but saccades are relatively fast, and the momentary movement of the pupil when the line of sight is moved. It is a point that appears prominently. This saccade is completed in about 40 msec even with a rotation of about 10 degrees, so even a normal 30 frame / second camera can fit within 2 frames. Therefore, even if other eye movements are mixed, if the rotation angle is more than a certain degree (for example, 10 degrees or more), the saccade can be sufficiently distinguished from other movements. Easy to judge. There are cases where the head is rotated when moving the line of sight from one direction to another, but even in such a case, the rotational speed of the eyeball is faster than the rotational speed of the head. The saccade appears clearly.

  Since the saccade described above occurs when the driver shifts his / her attention, it is apparent that the saccade is generated when the driver is awake. Eye movements other than saccades do not occur unless they are awake, but in this case as well, there is a high probability that saccades coexist. For example, a saccade always occurs when a driver looks at the front, rearview mirror, room mirror, meter, etc. while driving. Conversely, if saccade does not occur during driving, it may be determined that at least the arousal state has decreased.

  Therefore, the image processing terminal 4 calculates the amount of change in the relative position of the pupil with respect to the nostril of the subject A (line-of-sight movement amount) as described below, and then determines whether or not the line-of-sight movement has occurred, and wakes up the driver. Determine the state.

  First, the image processing terminal 4 controls the camera 2 to continuously capture a face image of the subject A, receives a moving image composed of a plurality of frames generated as a result, and targets each of the plurality of frames. In addition, the left and right pupil center positions and the left and right nostril center positions of the subject A are detected. FIG. 2 is a conceptual diagram illustrating a detection state of the subject A by the image processing device 4.

(Detection of pupil center)
The image processing terminal 4 obtains a bright pupil image and a dark pupil image by alternately lighting the light sources 3a and 3b and generating a frame synchronized with each lighting when a face image composed of a plurality of frames is captured. The bright pupil image is an image obtained with the irradiation of the light source 3a, and the luminance of the pupil portion is relatively bright. On the other hand, the dark pupil image is an image obtained with the irradiation of the light source 3b, and the luminance of the pupil portion is relatively dark. These two types of images are obtained due to differences in the intensity of reflected light from the pupil accompanying irradiation of illumination light from the two light sources 3a and 3b. Here, in the case of a camera employing field scanning, the light sources 3a and 3b are turned on in synchronization with the field signal of the camera 2 to separate the bright pupil image and the dark pupil image between the odd field and the even field. May be. Then, after taking the difference between the bright pupil image and the dark pupil image, the range of the pupil portion is determined. By performing such difference processing, it is possible to detect a highly robust pupil.

  After that, the contour of the detected pupil is specified, an ellipse that can be approximated to the contour is calculated, and the center of the ellipse is obtained as the center position of the pupil. Alternatively, the position of the pupil center may be calculated using the center of gravity method after binarizing the frame image that has been subjected to the difference processing. At this time, if there is a moving target such as an eyelid in the image, the area other than the pupil may appear brightly, so the selection of the size of the image area when obtaining the center of gravity becomes a problem. Therefore, as described in JP-A-2005-348832, the position of the pupil center may be calculated using a separability filter. That is, the center coordinate that maximizes the degree of separation is obtained using a pattern close to a circle.

(Detection of nostril center)
The image processing terminal 4 detects the two-dimensional coordinates of the left and right nostril centers with reference to the bright pupil image or the dark pupil image. That is, the center point of the left and right pupil centers is obtained, and a large window whose center substantially coincides with the nostril position when the subject A is assumed to face the front is set at a position below that. Detect nostrils with. Then, 0.8% of pixels with lower luminance are detected by the P-tile method in the large window of the frame image, and converted into a binary image composed of HIGH pixels and LOW pixels. After that, the detected binarized image is repeatedly expanded and contracted (morphological processing) to clarify the area in the image, and then the labeling process is performed to select the two areas from the larger one. The center, aspect ratio, and area of the rectangle formed from the top, bottom, left, and right end points are calculated. Here, the expansion process is a process of converting a target pixel into a HIGH pixel when at least one of the eight pixels in the vicinity of the target pixel is present in the binary image, and the contraction process is a binary process. This is processing for converting a target pixel into a LOW pixel when at least one of the eight pixels near the target pixel is present in the image. If the aspect ratio is smaller than 0.5 or larger than 0.7, and the entire image size is 640 × 240 pixels, the area is smaller than 100 pixels or larger than 300 pixels, the region indicating the nostril image Judge that is not. Otherwise, a small window of 30 × 30 pixels is set around the center of the rectangle, and 5% pixels from the lower brightness by the P-tile method are targeted in the small window of the original frame image. To extract. Thereafter, the above morphological process and labeling process are repeated to obtain a region having the maximum area. If the area of the region is 130 pixels or more and 70 pixels or less, it is determined that the image is not a nostril image. Otherwise, it is determined that the image is a nostril image, and the center of the rectangle formed from the upper, lower, left and right end points of the region is determined. Seek as the center. As a result, when two nostril centers are detected, the correspondence between the left and right nostrils is determined from the size of each coordinate value.

  As described above, when the nostril detection is performed using the large window and the small window, an optimum threshold value can be given for detecting each of the two nostrils having different imaging conditions, and the nostril can be detected reliably. Each nostril position is predicted using a predictive model such as a Kalman filter for the purpose of preventing false recognition of the nostrils when shadows that make it difficult to detect the nostrils appear in the image and responding to quick movement of the nostrils. And the position of the small window may be set.

If only one of the right and left nostrils can be detected during such nostril detection, the distance measured in the distance deriving step and the detected positions of the left and right pupil centers and the center of one nostril are used. Then, the position of the other nostril center is estimated. Now, the coordinates of the right pupil center are (x _RP , y _RP ), the coordinates of the left pupil center are (x _LP , y _LP ), the coordinates of the right nostril center are (x _RN , y _RN ), and the coordinates of the left nostril center are _Assume that (x _LN , y _LN ), the distance between the left and right pupil centers measured in advance is D _P0 , the distance between the left and right nostril centers is D _N0 , and the left nostril cannot be detected. Ramp rate I _P, and the pupil center distance D _P between left and right pupils in the face image at this time, the following equation (1) and (2);
I _P = (y _RP −y _LP ) / (x _RP −x _LP ) (1)
^{_{D P = {(x RP -x}} LP) 2 + (y RP -y LP) 2} 1/2 ... (2)
Can be considered.

Here, since the line connecting the left and right pupil centers and the line connecting the left and right nostril centers are considered to be always parallel, the slope ratio I _N = I _P between the left and right nostrils. Further, the nostrils center distance D _N on the face image, the distance D _P0 between the left and right pupil _center, the distance D _N0, and pupil center distance D _P between the left and right nostrils center, formula (3);
D _N = (D _N0 / D _P0 ) × D _P (3)
It is obtained by. The nostril center distance _DN is expressed by the following formula (4);
^{_{D N = {(x RN -x}} LN) 2 + (y RN -y LN) 2} 1/2 ... (4)
Therefore, the coordinates of the center of the left nostril (x _LN , y _LN ) are the following formulas (5) and (6);
^{_{x LN = x RN - {D}} N 2 / (1 + I N 2)} 1/2 ... (5)
^{_{y LN = y RN - {(}} D N 2 .I N 2) / (1 + I N 2)} 1/2 ... (6)
Can be obtained. Conversely, when the right nostril center is not detected, the following formulas (7) and (8);
^{_{x RN = x LN + {D}} N 2 / (1 + I N 2)} 1/2 ... (7)
^{_{y RN = y LN + {(}} D N 2 .I N 2) / (1 + I N 2)} 1/2 ... (8)
Can be obtained.

(Calculation of eye movement)
The image processing terminal 4 calculates the line-of-sight movement amount of the subject A based on the positions of the left and right pupil centers and the positions of the left and right nostrils in the previous and next frame images in each frame image. In this case, it is impossible to realistically think that the other nostril moves greatly with respect to one nostril. Therefore, it is sufficient that one nostril can be detected. Even if both nostrils can be detected, the position of either the left or right nostril center may be selected as the position that becomes the reference for eye movement. Alternatively, the midpoint of the position of the center of the left and right nostrils may be used as a reference. However, if the reference position is switched from one nostril position to the other nostril position or midpoint position, a change in the relative position of the pupil cannot be detected accurately. Therefore, in the following, a variation Dif _P of variation Dif _N and pupil center position of the nostrils center position between the front and rear frame images of the plurality of frame images obtained successively in time.

Specifically, when the left and right nostrils are detected, the amount of change for each center of the left and right nostrils is calculated. If (i) both nostrils continue to be detected, the amount of change in the center of each nostril between the previous and next frames is calculated, and the average of these is defined as the amount of change Dif _{N in the} nostril center. When (ii) only one nostril is continuously detected, the inter-frame variation of only nostrils center of the side that has been detected and the variation Dif _N nostril center. (Iii) When both nostrils have been detected up to the previous frame and only one of the nostrils is detected in the current frame, the coordinates of the nostril center on the side detected in the current frame and 1 One prior to the change amount of the coordinates of the nostril center on the same side as the variation Dif _N nostrils center of the frame. (iv) When only one nostril is detected up to the previous frame, and both nostrils are detected in the current frame, the coordinates of the nostril center detected in the previous frame and the current nostril are detected. the variation of the coordinates of the nostril center on the same side of the frame and the variation Dif _N nostril center.

On the other hand, in the detection of the center of the pupil, one pupil may not be detected or both pupils may not be detected. Therefore, the pupil center change amount Dif _P is calculated in the same manner as the nostril. That is, when the left and right pupils are detected, the amount of change for each of the left and right pupil centers is calculated. When (i) both the pupil is continuously detected, the average of each of the inter-frame variation of both the pupil center and the movement amount Dif _P of the pupil center. When (ii) only one of the pupils is continuously detected, the inter-frame variation of only the pupil center is detected and the variation Dif _P of the pupil center. (Iii) When both pupils are detected up to the previous frame and only one pupil is detected in the current frame, the coordinates of the center of the pupil detected in the current frame and the previous frame are detected. the amount of change to the same side of the pupil center coordinates in the frame and the variation Dif _P of the pupil center. (iv) When only one pupil is detected up to the previous frame and both pupils are detected in the current frame, the coordinates of the pupil center detected in the previous frame and the current pupil center are detected. the amount of change in the pupil center coordinates of the same side of the frame and the variation Dif _P of the pupil center. (v) When the state where both pupils are not detected continues, the pupil center change amount Dif _{P is set} to zero, and at the same time, “flag = 0” is set. (Vi) When both pupils have not been detected in the previous frame and both or one of the pupils has been detected in the current frame, the change amount Dif _P of the pupil center is set to zero and “flag = 0” ". Here, “flag = 0” is a flag that is set in the image processing terminal 4 in order to distinguish between the case where both pupils or one pupil are detected and no pupil movement and the case where both pupils are not detected. In the above cases (i) to (iv), “flag = 1” is set.

  Note that in the pupil change amount calculation processing, the difference from the case of the nostrils is (v) and (vi) because both pupils may disappear due to blinking. Therefore, the process is more complicated than in the case of a nostril. However, if the image processing does not work correctly for some reason, both nostrils may not be detected as with the pupil, so the same processing as in (v) (vi) can be performed when calculating the amount of change in the nostrils. It is preferable to carry out in combination.

Then, the image processing terminal 4 calculates the following expression (9) from the calculated change amount Dif _N of the nostril center position and the change amount Dif _P of the pupil center position in the calculated plurality of frame images;
Dif = Dif _P −Dif _N (9)
Using a determined continuously vision shift amount Dif by taking the variation Dif _N and Dif _P Kano difference. At this time, if “flag = 0” in the corresponding frame, it is determined that the line-of-sight movement amount Dif could not be detected.

(Calculation of eye movement)
The image processing terminal 4 determines the presence or absence of eye movement by comparing the obtained eye movement amount Dif with a preset threshold value, and outputs an alarm to a buzzer sounding means or the like when it is determined that there is no eye movement. To notify outside.

3 (a) shows a state in which the subject A is viewed front the camera 2, the horizontal positions of the left and right pupil center detected when you move alternately head to the left and right X _PL, X It is a graph showing temporal changes in _PR , left and right nostril center positions X _NL and X _NR , left and right pupil position average value X _PA , and left and right nostril position average value X _NA . FIG. 3B shows temporal changes in the relative position DX between the pupil and the nostril at this time and the line-of-sight movement amount Dif calculated by the image processing terminal 4 (= relative speed of the pupil with respect to the nostril). As shown in the figure, the left and right nostrils move almost the same and the left and right pupils move almost the same, so the absolute value of the line-of-sight movement amount Dif is not so large. Here, for simplicity, the left and right nostrils and the left and right pupils are shown as being always detected.

On the other hand, FIG. 4A shows the positions X _PL and X _{PR of} the left and right pupil centers detected when high-speed saccade eye movements are mixed with the movement of the head, and the center of the right and left nostrils. position _{X NL, X NR,} is a graph showing mean values _{X PA} of the left and right pupil position, and the time variation of the average value _{X NA} of the left and right nostrils position. FIG. 4B shows the relative position DX between the pupil position and the nostril position at this time, and the line-of-sight movement amount Dif calculated by the image processing terminal 4. Thus, it can be seen that the absolute value of the line-of-sight movement amount Dif increases momentarily when the saccade is generated. In this case, a portion where saccade is generated can be detected relatively easily by providing a certain positive and negative threshold. However, when the size of the saccade (the eyeball rotation angle displacement from the start to the end of the saccade) is small, the line-of-sight movement amount Dif at the time of occurrence of the saccade is larger than the movement amount due to the movement of the head. Since it is not detected, it is difficult to detect with a certain threshold. Also, noise tends to be difficult to detect with a certain threshold due to the presence of noise in the position data to be detected.

  Therefore, the image processing terminal 4 sets an appropriate variable threshold as follows, and detects the presence or absence of eye movement based on it.

That is, the image processing terminal 4 calculates the moving average value Dif _A by performing a moving average process on the temporal change of the line-of-sight movement amount Dif. Thereafter, a predetermined constant threshold width Th is set based on the signal waveform S (t) of the moving average value Dif _A after the moving average, and two threshold waveforms Th ₊ (t) and Th _- find a (t).
_{Th + (t) = S (} t) + Th, Th - (t) = S (t) -Th ... (10)

Finally, when the signal waveform of the line-of-sight movement amount Dif before the moving average is V (t), the image processing terminal 4 has the following formula (11);
_{V (t)> Th + (} t) or V (t) <Th - (t) ... (11)
The presence / absence of eye movement is determined based on whether or not the above condition is satisfied. In addition, the moving average value Dif _A may be output as a signal in which a delay due to the moving average process occurs after a signal indicating the line-of-sight movement amount Dif is generated. In such a case, the image processing terminal 4 applies Equation (11) even if the signal waveform is generated as V (t) by delaying the signal of the line-of-sight movement amount Dif by half the moving average width. May be.

In addition, when noise is mixed in the signal waveform of the line-of-sight movement amount Dif, the image processing terminal 4 performs the first moving average process on the line-of-sight movement amount Dif to obtain the signal waveform V (t of the movement average value Dif _A. ), And the moving average value Dif _A is subjected to a second moving average process to calculate a signal waveform S (t) of the moving average value Dif _AA , and the signal waveforms V (t) and S (t) are used. The presence or absence of eye movement can also be determined by applying equations (10) and (11). In this case, even if the noise level mixed in the process of image processing or signal processing is large in the line-of-sight movement amount Dif, it is possible to accurately detect the presence or absence of the line-of-sight movement.

FIG. 5A is a graph showing the time change of the line-of-sight movement amount Dif before and after the moving average, and FIG. 5B is a graph showing the time change of the line-of-sight movement amount Dif and the threshold values Th ₊ and Th ₋ . In this example, the occurrence of saccade can be normally detected from the level of the line-of-sight movement amount Dif.

  According to the line-of-sight movement detection device and the line-of-sight movement detection method described above, a plurality of frames are acquired as a moving image of the face image of the subject A, and the pupil center position and nostril center position in the plurality of frames are detected, Since the change Dif of the relative position of the pupil center with respect to the nostril center between the previous and next frames is calculated and the presence or absence of gaze movement is determined based on the magnitude of the change, a wide range can be obtained even for a face image using one camera. Thus, it is possible to determine the presence or absence of line-of-sight movement, and by monitoring the pupil position based on the nostril position, it is possible to realize stable detection of line-of-sight movement even for a subject whose head itself is likely to move.

  In addition, by detecting the nostril position as a reference for detecting eye movement, it is easy to create a dark condition with respect to the surroundings at the time of detection, and it is not necessary to use the template matching method with some contrivance in image processing Therefore, it is possible to detect the movement of the line of sight more easily.

  Compared to the long-known EOG (electro-oculogram) method for attaching eye electrodes to the face and measuring the movement of the line of sight relative to the face direction, attach accessories to the face. Since it is not necessary, there is little resistance for the user, and it is particularly suitable for monitoring a driver of a car or the like.

In addition, the change amount Dif _P of the pupil position between the previous and next frames and the change amount Dif _N of the nostril position are obtained, and the difference between the change amount of the pupil position and the change amount of the nostril position is obtained, thereby changing the relative position change amount Dif. Therefore, it is possible to efficiently calculate the change in the relative position of the pupil and nostril while referring to the moving image of the subject, and the real-time property of eye movement detection is improved.

In addition, the relative position change amount Dif is continuously calculated over a plurality of frames, the moving average value Dif _A of the change amount is derived, and the relative position change amount Dif is determined based on the moving average value Dif _A. Since the presence or absence of eye movement is determined by comparing with the set threshold values Th ₊ and Th ₋ , the eye movement with higher accuracy can be achieved by setting an appropriate threshold according to the magnitude of the displacement of the rotation angle of the eyeball. Can be detected.

It is a top view which shows the positional relationship of the gaze movement detection apparatus concerning a suitable one Embodiment of this invention, and a subject. It is a conceptual diagram which shows the detection state of the subject by the image processing apparatus of FIG. (A) is a graph showing the temporal changes in the left and right pupil center positions, the left and right nostril center positions, the average value of the pupil positions, and the average value of the nostril positions, and (b) is the graph of the pupil position and the nostril position. It is a graph which shows the time change of a relative position and a gaze movement amount. (A) is a graph showing the position of the left and right pupil centers, the position of the center of the right and left nostrils, the average value of the pupil positions, and the time variation of the average value of the nostril positions, and (b) is the relative relationship between the pupil and the nostrils. It is a graph which shows a time change of a position and a gaze movement amount. (A) is a graph which shows the time change of the gaze movement amount before and behind the moving average, and (b) is a graph which shows the time change of the gaze movement amount and the threshold value.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Eye movement detection apparatus, 2 ... Camera, 3a, 3b ... Light source, 4 ... Image processing terminal.

Claims (4)

A gaze movement detection method for capturing a face image of a subject and detecting the presence or absence of the gaze movement of the subject based on the face image,
Detecting a face image of the subject for a plurality of frames, and detecting a pupil position and a nostril position of the subject for each of the plurality of frames;
Based on the pupil position and the nostril position detected for the plurality of frames, the amount of change in the relative position of the pupil with respect to the subject's nostril between the previous and next frames in the plurality of frames is calculated. A calculating step to
A determination step of determining the presence or absence of the eye movement using the calculated change amount and a predetermined threshold;
A line-of-sight movement detection method comprising: In the calculating step, a change amount of the pupil position and a change amount of the nostril position between the preceding and following frames are obtained, and the relative difference is obtained by taking a difference between the change amount of the pupil position and the change amount of the nostril position. Calculate the amount of change in position,
The line-of-sight movement detection method according to claim 1. In the calculating step, the change amount of the relative position is continuously calculated over the plurality of frames, and a moving average value of the change amount is derived,
In the determination step, the presence or absence of the eye movement is determined by comparing the amount of change in the relative position with a threshold value set based on the moving average value.
The method of detecting gaze movement according to claim 1 or 2. A gaze movement detection device that captures a face image of a subject and detects presence or absence of gaze movement of the subject based on the face image,
Means for capturing the face image of the subject for a plurality of frames and detecting the pupil position and nostril position of the subject for each of the plurality of frames;
Based on the pupil position and the nostril position detected for the plurality of frames, the amount of change in the relative position of the pupil with respect to the subject's nostril between the previous and next frames in the plurality of frames is calculated. Means to
Means for determining the presence or absence of the eye movement using the calculated change amount and a predetermined threshold;
A line-of-sight movement detection device comprising:

Images