JP4491604B2 - Pupil detection device - Google Patents

Description

  The present invention relates to a pupil detection device that can replace a mouse as an input device, and a pointing device using the device. In particular, the present invention relates to an apparatus for detecting a pupil for a subject wearing spectacles without being disturbed by reflection of spectacles of a light source.

Various pointing methods based on the viewpoint (the direction of the user's line of sight) have been proposed so far.
However, when pointing depending on the viewpoint, the viewpoint must be moved to the location on the display device where you want to point, but the cursor (pointer) always appears at the position of the viewpoint while searching for the position to be viewed. There is a problem that it is difficult to move the cursor to the position where the cursor is to be moved. In addition, when a cursor is always presented at a viewpoint position, it becomes a visual stimulus, and there is a problem that it is difficult to see an object to be seen.
According to the movement of the head, the position where the cursor appears is different from the one you are looking at, and the cursor is at the target position while looking at the cursor in peripheral vision while looking at the necessary place on the front display device Can be moved.
An example of a conventional pointing system based on head movement is shown below.
The technology described in Non-Patent Document 1 (named “Eagle eye”) installs a video camera next to a personal computer and captures a part of the face specified in advance by image processing while capturing a face image. This is a method of tracking by template matching and moving the cursor on the personal computer screen according to the movement. As the tracking target, the tip of the nose, the tip of the chin, the black eye, etc. are tried. However, even if the template is updated, there is a possibility that it may be difficult to cope with a sudden decrease in ambient brightness, and problems such as a gradual shift in the target to be tracked are likely to occur.

The present inventor has proposed a pointing device based on pupil position detection in order to eliminate these drawbacks (Japanese Patent Application No. 2003-419110: filed on Dec. 17, 2003). This technique illuminates the face with an invisible infrared light source, shoots with a video camera at a magnification that makes the entire width of the face easier, and directly detects two or one pupil in the image taken by the video camera It is a method. The cursor on the monitor such as a personal computer is moved according to the movement of the pupil in the image caused by the movement of the head, the menu displayed on the monitor is selected, and characters are input using the keyboard software. Or switch between multiple windows. Further, the device can generate a signal corresponding to pressing of the left and right buttons of the mouse, and there is no need for another input operation device for click operation.
However, when the subject (user, operator) is using glasses, pupil detection may be hindered by the reflection of the glasses.
"The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access for People With Severe Disabilities", (Margrit Betke, James Gips, and Peter Fleming), IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 10, No. 1, March 2002

  The inventor detects the position of each of the two pupils in the frame with a single camera having a wide field of view, and responds to the movement of the pupil that moves in the image when the subject performs head movement. In the previous application, a method of moving the cursor (pointer) was proposed. In this method, the left and right pupils are distinguished, the mouse is moved when one eye is momentarily closed, or the head is moved with one eye closed, and the cursor is moved by moving the other pupil. This made it possible to drag. In order to realize this method, it was necessary to make it easy to detect the pupil. For this purpose, the light source (3) close to the aperture (2) and the light source (4) near the aperture (2) of the camera (1) are alternately turned on in synchronization with the video field (FIGS. 2 and 3). The difference between (FIG. 1-a) and the dark pupil image (FIG. 1-b) was performed in real time, the background portion was approximately canceled out, and the pupil portion was lifted to detect the pupil (FIG. 1-c). FIG. 3 shows an example of the arrangement of light sources using LEDs (light emitting diodes; 5). In this pupil detection method, only a light source close to the camera aperture is provided to obtain a bright pupil image and the pupil is detected by image processing, or only a light source far from the camera aperture is provided to obtain a dark pupil image. Compared to the method of detecting the pupil by processing, it is less affected by the surrounding brightness and the like, and the pupil is much easier to detect. Furthermore, since the image is simple, it is advantageous in that the amount of image processing can be greatly reduced.

  Furthermore, in this pupil detection method, even when the subject wears glasses or sunglasses for correcting vision, the spectacle reflection brightness is so bright that it is saturated in the image and easily distinguishable from other parts. By performing the image difference after lowering the brightness of the eyeglass, the reflection of the glasses can be removed at the image processing stage (FIG. 4). 4 (a) is a bright pupil image in which spectacle reflection is reflected, FIG. 4 (b) is a dark pupil image in which spectacle reflection is reflected, and FIG. 4 (c) is a difference after reducing the luminance of the high luminance part. It is an image with However, when the spectacle reflection image completely covers the pupil, the pupil is wiped out by spectacle reflection, and it is impossible to detect the pupil at the stage of image processing. Even in such a case, specular reflection can be removed to some extent by installing a polarizing filter in front of the light source and installing a polarizing filter in which the polarization direction is perpendicular to the polarizing filter described above in front of the opening of the camera. However, it is difficult to completely remove the spectacle reflection at this time, and if a part of the spectacle reflection remains, the spectacle reflection image becomes an image similar to the pupil, which is difficult to distinguish.

In addition, when gaze detection is performed without attaching any markers or the like to the face, the gaze direction may be detected based on the position of the pupil using the corneal reflection image of the light source as a reference point. That is, the line-of-sight direction is estimated from the relative coordinates of the pupil and corneal reflection in the camera image. Since the corneal reflection is different in size from the spectacle reflection but has similar properties, the use of a polarizing filter cannot be used because the corneal reflection is removed when the spectacle reflection is removed.
If the light source is not arranged near the camera opening, for example, as shown in FIG. 5, one light source (6) is installed at left and right positions away from the camera (1) opening, and they are alternately lit. There is a way to make it. In FIG. 5, 7 is a spectacle lens, and 8 is a cornea. In this case, even if one eyeglass reflection (10) of either light source hides the pupil (9), the eyeglass reflection of the other light source often does not hide the pupil, so the eyeglass reflection hides the pupil. The pupil can be detected using the image that is not present. However, as mentioned above, this method cannot be used if one of the light sources must be installed near the camera aperture.

Now, as shown in FIG. 6, two cameras (1) are used, and a light source (6) is prepared near the opening of each camera. These light sources are alternately turned on in synchronization with the video frame or field. When the right light source is turned on, the right camera obtains a bright pupil image (FIG. 6D), and the left camera obtains a dark pupil image (FIG. 6B). Conversely, when the left light source is lit, a dark pupil image (FIG. 6E) is obtained with the right camera, and a bright pupil image (FIG. 6C) is obtained with the left camera. In general, the pupil can be easily detected in the difference image as described above by subtracting the bright pupil image and the dark pupil image obtained by each camera. However, when wearing spectacles, when there is a spectacle lens (7) as shown in FIG. 6, spectacle reflection (10) may cover the pupil (9). In that case, the pupil cannot be detected. In this example, when the right light source is turned on, the eyeglass reflection overlaps the pupil in both the left and right camera images. In such an example, the eyeglass reflection overlaps the pupil in both the difference image obtained from the left camera and the difference image obtained from the right camera, and therefore the pupil cannot be detected by both cameras.
Next, as shown in FIG. 7, two cameras are used, and a light source (3) close to the opening and a light source (4) near the opening are attached to each camera (1), and these light sources are turned on alternately. Think. Then, four eyeglass reflections (10) appear for one camera, and the pupil (9) becomes difficult to detect.

  Therefore, as shown in FIGS. 8 and 9, as in the method shown in FIG. 7, the near-infrared light source is located at a position near and far from the respective openings of the two cameras (1) driven in synchronization. (3, 4) are installed, their light sources are alternately turned on in synchronization with the video field (frame), and images based on image differences are obtained from the bright and dark pupil images obtained from each camera. The pupil is detected by processing. Up to this point, the method is the same as that shown in FIG. The first solution proposed in the present invention is based on the use of light sources having different wavelengths for a pair of light sources attached in the vicinity of the left camera and a pair of light sources attached in the vicinity of the right camera. The second solution is to take a picture with the left camera (open the electronic shutter of the left camera) while one of the two light sources on the left is lit. Shooting with the right camera while is lit (opening the electronic shutter of the right camera).

8 and 9 show a method using light sources having different wavelengths. Unlike the method of FIG. 7, light sources having different wavelengths are used for a pair of light sources attached near the left camera and a pair of light sources attached near the right camera. For example, one uses a light source with a center wavelength of 810 nm and the other uses a light source with an 850 nm. Then, a sufficiently narrow band-pass filter (11) having the same center wavelength as each light source is attached to each camera opening so as not to interfere with each other's images. When the wavelength ranges of the respective light sources overlap, a band pass filter is attached in front of the light sources to narrow the wavelength range of the light sources, and the near infrared rays are irradiated to the head. Thereby, the left and right optical systems are independent and do not interfere with each other. In FIGS. 8 and 9, in the camera image, a spectacle reflection image by a light source close to the opening and a spectacle reflection image by light far from the opening are simultaneously expressed as one image. Light source control means for controlling blinking of the light source is not shown.
In addition, light source control by time-division driving can be considered instead of light sources having different wavelengths.
While one of the two light sources on the left is lit, take a picture with the left camera (open the electronic shutter of the left camera), and to the right while one of the two light sources on the right is lit By adopting a time-division light source control method such as shooting with a camera (opening the electronic shutter of the right camera), the same effect as wavelength limitation can be obtained.

  In the method described so far, basically, when the spectacle reflection image overlaps the pupil image, the fact that the pupil cannot be detected is not different from the conventional method (image of the right camera in FIG. 9). However, probabilistically, it is rare that the pupils are hidden simultaneously in the images of both cameras. That is, the pupil can always be detected from at least one of the images of the two cameras. This is used to increase the robustness of the pupil mouse. Here, FIG. 8 shows a case in which the pupil can be detected simultaneously by both cameras, and FIG. 9 shows an example in which the pupil can be detected by the left camera from the subject and cannot be detected by the right camera. .

Up to this point, two cameras have been described. However, it is self-evident that the use of three or more cameras can further prevent the eyeglass reflection and the pupil from overlapping.
In addition, the three-dimensional positions of the two pupils can be detected by triangulation from the face images captured by the two cameras. In addition, it is possible to analyze the surroundings of each of the two pupils detected in each of the two facial images in detail, detect two accurate pupil centers and two corneal reflection centers, and obtain two eye gazes. . The intersection of two lines of sight is the viewpoint, and a three-dimensional viewpoint can be measured from the three-dimensional position of the pupil and the line of sight. In the application to control the position of the cursor from the viewpoint position thus obtained, it is necessary to capture the object with at least two cameras when capturing the pupil and corneal reflection and measuring the three-dimensional position. In this case, three or more cameras are arranged.
In addition, it is effective to arrange a large number of cameras even when the subject rotates his head at a wide angle or moves his head over a wide range.

  In detecting the movement of the pupil and creating the movement amount information, (1) the position of the pupil is set as the cursor position (named control by absolute amount), and (2) the movement amount of the pupil per unit time is determined by the cursor position. The amount of movement per unit time from the current position (named as control by relative amount). (3) The amount of movement of the cursor per unit time is generated in proportion to the deviation from the origin of the pupil (depending on the integral amount). 3) control). The above (1) corresponds to the conventional joystick, and the cursor position is determined by an amount corresponding to the displacement amount of the lever (control of the absolute position). (2) corresponds to the current mouse or trackball, and the cursor moves by an amount commensurate with the amount of rotation (control of the relative position). The above (3) corresponds to a stick-like controller found in current notebook personal computers, and the cursor continues to move while the lever is tilted (position determination by integration). It can be said that the cursor control by the cursor key corresponds to the above (3).

Further, depending on the application, (4) it is conceivable to move the cursor by differentiation of position information. In actual use, a dead zone is provided, or when the movement is large, a large amount of movement may be created as an acceleration function.
When detecting pupil movement and creating movement amount information, it is necessary to consider the output signal depending on whether the mouse interface is used to connect to other systems, the joystick interface, or another stick interface. It will be understood that there are a wide variety of conversion modes.

The connection interface is preferably a PCI bus interface when the control means is inserted into the PCI slot, and is preferably a USB interface when an external adapter is used.
Furthermore, it is possible to use the personal computer itself as the control means, and in that case, it is conceivable to provide a light source irradiation (flashing) control part separately.
It is possible to divide the control function of the control means as desired, and to distribute functions as appropriate, such as a part executed by software, a part executed by hardware, a part executed by the personal computer itself, and a part executed by the PCI board. Will be understood by those skilled in the art.

Correspondence with the left and right push buttons is also important for substituting the mouse with pupil detection.
The detection state of the pupil detected from the face image is as follows: (a) Both eyes are open and both pupils are detected (right pupil detection, left pupil detection), (b) Only the right eye is open, When only the pupil is detected (right pupil detection, left pupil non-detection), (c) When only the left eye is open and only the left pupil is detected (right pupil non-detection, left pupil detection), (d) There are four ways when both eyes are closed and both pupils are not detected (right pupil not detected, left pupil not detected).
Information output from a general-purpose two-button mouse signal includes mouse button information (down (pressed) for the right button), UP (released) and DOWN / UP for the left button) and cursor movement information (horizontal and vertical). Direction).
For example, if one eye is closed and then opened, clicking is performed, and if the head is moved while closing one eye, dragging is started, and if the eyes that are closed thereafter are opened, the dragging is terminated.
For this purpose, it is assumed that processing for opening and closing the right eye and opening and closing the left eye is made to correspond to UP and DOWN of the right button and UP and DOWN of the left button of the general-purpose two-button mouse. This correspondence can be interchanged between left and right.
However, if both eyes are closed by blinking, it is not the subject's intention and is treated as an exception. Normally, the state where both buttons of the general-purpose mouse are pressed is not used. Therefore, when both eyes are closed, it is considered that both buttons are UP. However, the pupil movement amount is assumed to be zero. Thus, when both eyes are closed, this corresponds to a state in which no mouse is operated (a state in which the hand is released). Such a situation is treated not only when both eyes are closed, but also when the face is not reflected in the camera.

(1) When blinking, both pupils are not detected, but when the right and left eyes are closed, there may be a shift of one to two frames depending on the individual. This is the same when opening the eyes after blinking. In such a case, simply matching the changes in the detection state of both pupils to the left and right buttons of a general-purpose mouse results in the same situation as if the left and right buttons were pressed slightly at different times. When this happens, many application software will behave differently from the user's intention.
(2) Further, when the head moves quickly, the pupil also moves fast, and the pupil image may be blurred at that time, resulting in a decrease in peak luminance value. As a result, the pupil may not be detected. This can be for both pupils when eyes are open (when the cursor is simply moved) or for one pupil. In that case, if it is determined that the detection state of the pupil has changed, button information different from the intention of the subject is output.
In order to prevent the problems (1) and (2) above, the following measures are taken.
Even if the detection state (detection, non-detection) of the pupil obtained from the current image frame changes from the previous frame to another state, the same state continues for several frames (eg, 3 frames, 100 ms) after the change. It is assumed that the button information corresponding to the pupil state after the change is output for the first time.

In pupil detection from a face image, when both pupils are detected, the left and right pupils are determined from the X coordinate of each pupil. Therefore, when either eye is closed from the state where both pupils are open, it is determined from the comparison of coordinates which of the left and right pupils has changed from the detection state to the non-detection state. Also, when opening from one eye to the other from a state where both eyes are closed, both eyes are detected in such a change of state because it is not known which eye is opened first until both eyes are opened. Until it is determined whether the pupil is left or right, the left and right of both pupils are distinguished only after both pupils are detected. Therefore, if both pupils have not been detected before, the button operation will remain UP on both the left and right until it is determined that both pupils have been detected (even after both pupils have been detected. It is assumed that the movement of the cursor follows the amount of movement of the pupil of one eye.
On the other hand, there may be a transition from a state where only one pupil is detected to a state where only the other pupil is detected. This occurs when one eye is closed during dragging, and when it is determined that the closed eye has just opened at the end of dragging, the pupil that is open for some reason is no longer detected. However, assuming that it is impossible for the user to intentionally change from the state where only one eye is closed to the state where only the other eye is closed, the other pupil that should be detected is detected for some reason. Accordingly, it is determined that the state has changed from the state in which one eye is open to the state in which both pupils are detected. Thereafter, the cursor is moved in correspondence with the movement of only the detected pupil.

  When both pupils continue to be detected (corresponding to simple cursor movement), it is desirable to use the movement amount of the center of gravity of both pupils between successive frames as the pupil movement amount, and to correspond to the cursor movement amount. When only one pupil is continuously detected, the detected pupil movement amount between successive frames is made to correspond to the cursor movement amount as the pupil movement amount. Note that while the user (subject) does not have to move during the head movement, he / she intends to open both eyes regardless of his / her intention. When only the pupil can be detected (also occurs when the eyeglass reflection overlaps the pupil image, etc.), the movement amount of one pupil corresponds to the cursor movement amount during that time. In addition, it is possible to detect both pupils in a shorter time than the specified number of frames, although the user intends to open one eye regardless of the intention during the head movement (although it does not necessarily move). In the absence, the pupil does not move during that time, and the cursor does not move.

Although the user is trying to move the cursor, the pupil does not move during several frames in which the pupil cannot be detected. In this case, the head may turn around in a state where the cursor does not move, which makes it difficult for the user to deal with it later. Therefore, the difference between the pupil position immediately before (one frame before) and the pupil position immediately after the several frames where the pupil could not be detected is made to correspond to the cursor movement amount as the pupil movement amount.
When both pupils are detected continuously, if both pupils cannot be detected in a shorter period of time than the specified number of frames, regardless of the user's intention ( In the case of the occurrence of the However, this is not applied when one pupil is continuously detected. In this case, the movement amount of one pupil is made to correspond to the cursor movement amount as the pupil movement amount.

When the subject is nervous, the head vibrates, the pupil vibrates accordingly, and the cursor vibrates. (This is not a problem with pupil detection, but the problem of trying to move the cursor with the movement of the head itself). This is solved by the method described later. First, a method that can be generally considered will be described.
Usually, the correspondence between the obtained pupil movement amount and cursor movement amount
Cx = kx · Px,
Cy = ky ・ Py
Can be associated with each other. Here, Px and Py are the horizontal and vertical pupil movement components (with positive and negative) in the image, respectively, and Cx and Cy are the horizontal and vertical cursor movement amounts in the personal computer screen. Kx and ky are coefficients for selecting a favorite value depending on the subject.

In moving the pupil seen from the camera by rotating or moving the head, if kx and ky are set large, the cursor moves greatly with the movement of the small head, but instead, the subject stops the head. The cursor vibrates due to the vibration of the head even though he intends to be. Conversely, if kx and ky are set to a small value, the cursor cannot be moved greatly unless a large head movement is performed. Even if an appropriate value is selected as much as possible, when the cursor is moved to a desired position, the cursor vibrates and alignment is difficult.
As a solution,
Cx = kx · Px · | Px | b−1
Cy = ky · Py · | Py | b-1
Is to correspond. When b> 1, the orders of Px and Py are higher than in the case of the simple proportionality, and when b = 2, the square of the pupil movement amount is made to correspond to the cursor movement amount. By such association, pupil vibration is suppressed to almost zero, and pupil movement due to head movement is emphasized. Here again, kx and ky are set according to the subject's preference. In this way, when the order is increased, the cursor moves greatly even with a small head movement when the head is moved relatively quickly, and conversely, when the head is moved slowly, the cursor can move only slightly. become. Therefore, it is suitable for both when it is desired to move the cursor greatly and when the cursor is slightly displaced from the desired position and when it is desired to align it with the desired position.

However, even if b = 2, if the vibration of the head adversely affects the movement of the cursor, a so-called dead zone is provided. That is, when Px and Py are equal to or less than the specified threshold values, Px = 0 and Py = 0 (or Cx = 0 and Cy = 0) are set. In this way, the cursor does not move at all with a small head vibration in the dead zone range.
Or, after providing a dead zone, record the integral value of all pupil movements during small pupil movements within the dead zone range, and if there is a large pupil movement that exceeds the dead zone, set the integral value Use to move the cursor. When the cursor is controlled according to the direction in which the face is directed, the cursor can be instantaneously directed substantially in the direction in which the face is directed in this way.
Furthermore, if the dead zone and an exponential function with the dead zone boundary as the origin are used, the cursor does not move at all with a small vibration of the head, and can gradually move as the head moves larger than the dead zone.

  When you click or start dragging, close your eyes from the open state. When the drag is finished, the eye is opened from the closed state. In this way, in a short time for opening and closing the eyes (for example, between two frames), generally, a human moves his eyes unless care is taken. If no special processing is performed, the cursor moves according to the movement of the eyes, and the cursor moves to a position different from the user's intention. Will end up. In order to prevent this problem, it is desirable that the coordinates of the cursor position at the time when the user moves the cursor to the target position and closes or opens the eyes become the input information. Therefore, for example, the amount of cursor movement over the recent several frames is always recorded, and the time is traced back by several frames from the frame where it is determined that the start or end of the click or drag has occurred (this is referred to as frame B). A process for returning the cursor to the cursor position in the frame (referred to as frame E) is performed. That is, the value obtained by subtracting the integral value of the pupil movement amount from frame B to frame E from the pupil movement amount in frame B is replaced with a new pupil movement amount and used for cursor movement.

Some users are not good at opening and closing one eye. For such a user, the part corresponding to the button operation of the pupil mouse can be substituted by another button pressing device (including a general-purpose mouse, a keyboard, etc.) or can be used in combination. For example, it is useful when controlling a cursor on a large display.
It is also useful to use a pupil mouse and a general-purpose manual mouse in combination.
A general-purpose manual mouse is good at fine cursor movement, but in order to move a large cursor, it is difficult to use it because it is necessary to move the hand greatly or to lift it back several times. On the other hand, in the pupil mouse, the cursor moves quickly for a large head movement (rotation). For fine work, the pupil mouse is kept from acting and the cursor is moved by the manual mouse.

  In addition, the cursor is often lost on a large display, making it difficult to use. By adjusting the pupil mouse and moving the cursor to a large extent in accordance with the rotation of the head, the cursor can be moved in a direction almost facing the head, which makes it easier to find the cursor. In order to do the above, the dead zone is set to be large, and the cursor is set not to move at all with small pupil movement. Then, the integrated value of all pupil movements is recorded, and when there is a large pupil movement exceeding the dead zone again, the pupil value is moved using the integrated value. By doing so, it is possible to point the cursor instantaneously in the direction of the face.

So far, the description has been given assuming that the closed state of the eye is directly associated with the mouse button press. In this case, the OS (or application software) determines that it is double-clicked by the second blink of one eye, and executes a corresponding operation. On the other hand, there is a user who is difficult to perform an operation corresponding to dragging by moving the head with one eye open. In order to cope with this, when the second blink of one eye is detected, if it is determined that the mouse button is locked in a pressed state, a drag operation can be performed with both eyes open.
Usefulness is enhanced by configuring the correspondence in the means for associating the opening / closing information of the left and right eyes with the pressing state information of the left and right push button switches of the pointing device in accordance with the use situation.
As another form, it is also possible to discriminate the repetition cycle of one-eye blinking, for example, to make two blinks in the early cycle double-click, and to make the second blink in the late cycle push-down lock. . Furthermore, a modification that generates various button press signals including the locked state is conceivable depending on conditions that are easy for the user to operate, such as a press lock with three blinks.
The detection control means for performing the functions of image processing and conversion to a mouse substitute signal such as pupil position detection, blink detection based on the presence or absence of the pupil, and conversion from the pupil movement amount to the cursor movement amount are not shown.

So far, it has been described that the pupil is detected by a narrow-angle camera, but a wide-angle camera can be used instead of the narrow-angle camera.
When a wide-angle camera is used, two pupils are detected from one camera image by photographing both eyes. From each camera image, the pupil can be easily detected, the closed eye can be clearly seen, and which eye is closed is also known. However, when wearing spectacles, the pupil may not be detected even though one of the two camera images has eyes open due to spectacle reflection.
The determination method in this case will be described below.
When the two pupils are detected in both of the 1.2 cameras, the average amount of movement between the four pupil frames is reflected in the movement amount of the cursor.
In both 2.2 cameras, when only the left pupil or only the right pupil is detected, it is determined that the left eye or the right eye is open and the right eye or the left eye is closed.

3. If both cameras cannot detect both pupils, it is understood that both eyes are closed, but this is a blink or if the subject is not in front of the camera, it is determined that nothing has happened. To do.
4. If one camera detects both pupils and the other camera detects only one pupil, it is determined that both eyes are open and a total of three pupils or two The average inter-frame movement amount of the pupils (two in total) detected by both cameras is reflected in the cursor movement amount.
5. If one camera detects only the right pupil and the other camera detects only the left pupil, it interprets that both eyes are open and moves these two pupils between frames. The average amount is reflected in the amount of cursor movement.
6. With one camera, the right or left pupil is detected, and with the other camera, when neither pupil is detected, only the right or left eye is open and the left or right eye is closed The amount of movement of one pupil in total between frames is reflected in the amount of movement of the cursor.
Increasing the number of cameras can also increase the amount of pupil movement. At this time, instead of reflecting the simple average of the pupil movement amount to the cursor movement, the average is obtained by removing the maximum and minimum of the obtained pupil movement amounts, or in the case of an odd number of data, the center value In the case of an even number, it may be possible to introduce a statistical approach such as using the average of the two at the center as the pupil movement amount.

[Removal of cursor movement caused by eye rotation]
Next, removal of the effect of eyeball rotation will be described.
The pupil mouse originally associates the temporal change in the coordinates of the pupil moving in the video camera image with the movement of the head and the cursor movement. By changing the coefficient for associating pupil movement with cursor movement (the expression is described separately), the degree of cursor movement for the same pupil movement changes, but the face is shown on the video camera. Similarly to the above coefficient, the enlargement ratio at the time also greatly affects the cursor movement.
If you want to be able to use the head even if it moves in a fairly wide range (for example, 30cmx30cmx30cm), you can reduce the magnification of the video camera and always detect both pupils even if the head moves greatly. There must be. On the other hand, if you want to move a large cursor with a small neck movement, for example, for a patient who does not have the neck movement as desired, the cursor must move smoothly with a small head movement. Rather than increasing the coefficient, it is ideal to use a video camera with a larger enlargement factor. For example, the image may be captured so that the width of the face matches the width of the camera image. In that case, even if a pupil movement of the same size as in the case where the magnification rate is small (even if the same head movement) occurs in the space, the pupil movement in the camera image is performed when the magnification rate is small. As a result, a sufficiently large cursor movement can be obtained even with a small head movement.

However, when the enlargement ratio is increased, a phenomenon occurs in which the cursor moves due to eye movement even if there is no movement of the head. The reason is that even if the head is stationary, the pupil moves with the eyeball rotation, and it moves even in the video camera image. When using with a small magnification, it is common to move the head greatly, so the pupil movement due to head movement is sufficiently small compared to the pupil movement due to eye rotation and the latter is ignored. Even if it can be done, it cannot be ignored if the enlargement ratio is increased. As a result, even if the subject does not move at all to observe various places on the personal computer screen, the cursor moves in small increments each time the eyes are moved. In some cases, this may make the subject feel uncomfortable or cause an erroneous operation.
A method for solving this problem is described below.
As shown in FIG. 10, the eyeball (20) is composed of a double sphere whose center is shifted, and an eye model in which one of them is a corneal sphere (23) can be considered. Now, since the size of the eyeball is usually sufficiently smaller than the distance from the camera to the eyeball, it is assumed that the eyeball is at infinity from the camera for convenience. Further, it is assumed that the light source (6) is at the center of the opening of the camera (1). When the line of sight (25) faces the camera as shown in FIG. 10 (b), the corneal reflection (24) of the light source appears near the center of the pupil (9) (right in FIG. 10 (b)). Further, an eyeball rotation center (21) exists on the extension line of the corneal reflection (24) and the pupil center (22). Here, as shown in FIG. 10A, when the line of sight is directed to a position deviated from the camera, the corneal reflection (24) and the pupil center (22) are off. Furthermore, the center of eyeball rotation (21) is also shifted.

Here, as long as the head does not move, the center of eyeball rotation hardly moves even when the eyeball rotates. Therefore, if the movement of the center of eyeball rotation corresponds to the cursor movement instead of the pupil, the cursor will not move unless the head moves even if the eyes move.
However, since the center of the eyeball does not appear in the image, it is necessary to estimate the position. This is described below. First, in FIG. 10, only one pupil is represented, but generally two pupils are simultaneously captured. In a video camera having a resolution of about 640 × 480 pixels or 320 × 240 pixels, the light source is placed around the opening of the camera when the magnification of the face is equal to or smaller than the width of the image. Even when arranged, the light source appears as one light spot as shown in FIG.

Now, as shown in FIG. 11, assuming that the distance from the center of eyeball rotation to the center of the cornea is r and the distance from the center of eyeball rotation to the center of the pupil is R, the line-of-sight direction is deviated by θ from the direction in which the camera exists. . Here, the line of sight is a straight line passing through the center of eyeball rotation, the center of the corneal sphere, and the center of the pupil, and the point of the line of sight is the viewpoint viewed by the subject.
At this time, when the distance from the eyeball rotation center to the pupil center when viewed from the camera direction is p, and the distance from the eyeball rotation center to the pupil center is g,
It is. Therefore,
In other words, 0 <k = r / R <1. Where human standard dimensions are
Since it is known that r = 5.9 mm and R = 9.7 mm, k≈0.6 is obtained therefrom.

In the image, if the estimated center of rotation of the eyeball is E (Xe, Ye) and the pupil center and corneal reflection obtained by image analysis are P (Xp, Yp) and G (Xg, Yg), respectively,
In the same way as equation (3) between vector EG and vector EP
The relationship holds.

Therefore, it is transformed as follows,

Get. By using the equation (7), the eyeball rotation center (Xe, Ye) can be estimated from the corneal reflection coordinates (Xg, Yg) and the pupil center coordinates (Xp, Yp) in the image. Here, the value of k is different for each individual, but since there is no significant difference, k = 0.6 is given. Compared to the movement of the pupil center (Xp, Yp) between frames corresponding to the cursor movement, the movement of the eye movement (eye rotation) clearly moves to the cursor movement when the eye rotation center coordinates (Xe, Ye) correspond. The effect of. Of course, it is possible to estimate k by the following method by fixing the head and intentionally causing eye movement.

From equation (3)
Therefore, fix the head or make it still as much as possible, move the eyes to make the subject look at multiple positions, plot the change in the corneal reflection position Δg and the change in the pupil center Δp, Find k from the slope.
As described above, k may be obtained experimentally for each subject, or an appropriate value (for example, 0.6) may be given from the beginning. The method of obtaining the eyeball rotation center by the equation (7) and moving the cursor in accordance with the movement of the eyeball rotation center instead of the pupil center is not dependent on the eye rotation, but the cursor control which depends only on the rotation of the head. it can.

However, when the head is rotated greatly, the corneal reflection may be hidden by the eyelids or eyelashes, or the angle θ with respect to the camera is too large, and the corneal reflection may not appear, and the white eye may shine instead. In such a case, corneal reflection cannot be observed from the camera, and the center of eyeball rotation cannot be estimated.
The processing in such a case will be described below.
(i) In two consecutive video frame images, if corneal reflection appears in both images,
This (ΔXe, ΔYe) is used for cursor movement instead of pupil movement. Here, (ΔXe, ΔYe), (ΔXg, ΔYg), and (ΔXp, ΔYp) are eyeball rotation center movement, pupil center movement, and corneal reflection movement, respectively, between successive frames.
(ii) When corneal reflection is not detected in one or both of two consecutive video frame images, pupil center movement (ΔXp, ΔYp) between two frames is used for cursor movement.

[Regarding image processing to detect the pupil center with high accuracy]
Next, improvement in detection accuracy of the pupil center will be described.
When controlling the cursor movement with respect to the movement of the pupil, it is desirable to detect the center of the pupil with high resolution. For example, in a generally used template matting method, since coordinates can be obtained only for each pixel, smooth cursor movement is difficult. Thus, the center of the pupil is detected with high accuracy by the method described later, but the premise of the previous image processing will be described.
Basically, it is difficult to detect the center of the pupil with high accuracy by performing the same image processing on all images. In addition, since images having characteristics similar to those of the pupil may appear in all the images, it is necessary to prevent erroneous recognition.
Therefore, once the pupil is detected as a brighter part than the difference image (the high accuracy of the center coordinates at that time is not required), a window is provided there, and the pupil is searched only within the window from the next frame. . This eliminates the need for image processing other than that for the window, so that a large amount of time can be spent on image processing calculations within the window. As for the window size, if the entire length of the face is just enough to fit in the image, a rectangular window of about 21 × 21 pixels can be given with an image with a resolution of 320 × 240 pixels, and the entire width of the face is As long as it is just within the image, it may be about 25 × 25 pixels. In any case, the size of this window must take into account the speed at which the head and pupil move. In a normal video camera with 30 frames per second (NTSC system), the pupil moves greatly during the time of one frame, so the pupil position in the next frame is predicted using a prediction model such as a Kalman filter. It is necessary to set the above window centered on the.

Now, let us consider a little about how to calculate the pupil center with high accuracy. When determining the pupil center, a method of determining the pupil center by the center of gravity method considering the luminance of the difference image is conceivable. The center of gravity method can be expected to have high resolution. However, even in the difference image, as in the example shown in FIG. 12A, there is a case where the pupil is at the center and the eyelid is observed around the pupil (because the difference image is used, the eyelid moves. If there is, it appears). At this time, even if an attempt is made to obtain a mere luminance centroid as the center of the pupil, there is a problem as to which range of image centroid calculation should be performed. Therefore, the centroid method is difficult to use. Furthermore, a method of obtaining the center of gravity after binarizing the image as shown in FIG. 12A is conceivable, but further accuracy cannot be expected.
Recently, in order to detect black eyes close to a circle, a separation degree filter (K.Fukui: "Edge extraction method based on separability of image features", IEICE Trans. Inf. & Syst., Vol. E78-D, No. 12 , pp.1533-1538 (1995)) (M. Yuasa, O. Yamaguchi, K. Fukui: "Precise pupil contour detection based on minimizing the energy of pattern and edge", IEICE Trans. Inf. & Syst., Vol.E87-D, No.1, pp.105-112 (2004), Tsuyoshi Kawaguchi, Mohammed Lizon, Daisuke Hidaka: "Detection of both eyes from human face by Hough transform and separability filter", Science theory D-II, Vol.J84-DII, No.10, pp.2190-2200 (2001)).
In this case, as shown in FIG. 13, a double circle mask is used, and the inner region is defined as region 1 and the outer region is defined as region 2, and the coordinates that maximize the degree of separation in the image are obtained. However, the method of obtaining the coordinate with the maximum degree of separation can detect the coordinate only with the resolution of the pixel unit, and hinders smooth cursor movement. Therefore, a method of obtaining the center of gravity of a narrow range centering on the pixel showing the maximum value can be considered, but since the target is several pixels, the resolution and the like cannot be expected so much. When the range is widened, there is a possibility that a pixel showing a separation value that greatly affects the center of gravity exists in the range, and thus accuracy cannot be expected.

Further, the separability filter generally has a problem that it requires a lot of calculation time. Therefore, the following method is taken.
Basically, the fact that the pupil is circular and the radius is variable in the difference image and the brightness is higher than the surroundings is used.
The pupil feature P (x, y) at the coordinates (x, y) of the difference image is defined as the following equation, and is determined for the window.
Here, I (x, y) is the intensity (Intensity) of the difference image at the coordinates (x, y). S (x, y) will be described next. In order to shorten the calculation time, only the calculation on the radial mask having the shape shown in FIG. 14 extended by five pixels in eight directions is performed (if the calculation time is long, it may be more than five pixels). Also, it is not necessarily 8 directions). That is, the pixel is overlapped so that the center pixel of the mask pattern in FIG. 14 is positioned at the target pixel (x, y) of the difference image to be analyzed, and the calculation is performed only on the pixel on the difference image under the mask. The pixel shown in FIG. 14 is divided into a region corresponding to L pixels (a region in a square indicated by a broken line; region 1) and an outer region (region 2) from the center pixel. Each time L is changed (1 to 5), the degree of separation s is calculated (described later), and L (Lm) when the s value is maximized is determined. Further, the s value at that time was set as the degree of separation indicating the maximum at each coordinate (x, y), and was replaced with S (x, y) and used in the equation (1). Here, it should be noted that even when the difference image in the window includes the eyelid (eyelid) as shown in FIG. 12A, the image representing P (x, y) is shown in FIG. Thus, the eyelid disappears and only the central part of the pupil has a high value. (The circle drawn in FIG. 14 schematically shows the outline of the pupil. The pupil is usually a circle, whereas the boundary between the regions 1 and 2 is set to be a square. (In practice, it goes without saying that it is better to change the length of each radial mask so that it is as circular as possible, but in this example it is shown in a simplified manner.)

Further, the maximum value of P (x, y) is obtained within the window, and when it does not exceed the threshold value (experience value), it is determined that the pupil does not exist, and when it exceeds, it is determined that the pupil exists. By this method, when the pupil existing in the previous frame disappears in the current frame, it is determined. Furthermore, it is used for the trigger (trigger) to cancel the window. If it is determined that a pupil is present, the center of gravity of P (x, y) in a square region having a side of about 2 Lm to 4 Lm around the coordinates indicating the maximum value is calculated and set as the final pupil center.
However, as described in the section [Removal of cursor movement accompanying eye rotation], when the enlargement ratio of the video camera is large, the length of each line of the radial mask is 5 pixels including the center. . In order to accurately determine the center of the pupil that appears large, the problem can be solved by increasing the length of this line, but this will increase the calculation time. In order to solve this problem, as shown in FIG. 15, the pixels closer to the center of the mask are increased in place of the pixels closer to the center range, and as a result, the number of pixels in the mask is not changed. By such a device, it is possible to cope with the enlargement ratio of the video camera without increasing the calculation time.

In addition, when there are two regions 1 and 2 in the image, the degree of separation η can be calculated by a linear discriminant method using information from regions 1 and 2 in equations (11) to (13).
Here, Pi is a feature (for example, luminance) of an image at one pixel i, and upper line P1 and upper line P2 are average values of regions 1 and 2, respectively. The upper line P is an average over both regions, and n1 and n2 are the number of pixels in regions 1 and 2, respectively. (See K.Fukui: "Edge extraction method based on separability of image features", IEICE Trans. Inf. & Syst., Vol. E78-D, No. 12, pp.1533-1538 (1995))

(1) This method uses two cameras, but it is not a stereo camera, so it can be used to point the two cameras appropriately within the range of moving the head and does not require camera calibration. It's easy to do.
(2) Although the maximum movement of up to two pupils is associated with the cursor in one camera, it can be associated with up to four in the two cameras, so it is highly stable by moving the cursor. Sex is obtained. In particular, when dragging, the effect is great because only one pupil movement is originally detected.
(3) Even if there is no spectacle reflection and the eyes are open, even when the pupil cannot be detected for some reason, it can be compensated.
(4) In order to improve the pupil detection rate, once the pupil is detected, a small window around it is given, and in the next frame, only the inside of the window is searched to detect the pupil. In order to respond to rapid pupil movement, the pupil detection position in the current frame is predicted from the previous frame and the pupil position in the previous frame, and a window is provided to greatly improve the pupil detection rate. To do. In the case of only one camera, when the pupil cannot be detected even though the eyes are open, the window can only be canceled or the pupil can be detected by enlarging the window. However, if the window is enlarged, the possibility that a part other than the pupil will be erroneously detected as the pupil increases, and the correct pupil detection rate decreases. If two cameras are used and the distance between them is not too large, the positional relationship between the left and right pupils is the same for both cameras, so the amount of movement of the pupil between frames in one camera image is the other. It is not much different from that of other cameras. Therefore, assuming that the pupil movement equivalent to the pupil movement in the camera image where the pupil is detected also occurs in the camera image where the pupil is not detected, the dynamic window is set according to the movement amount. Keep giving for a while (several frames). In the meantime, if the spectacle reflection image passes through the pupil image and can be detected again within the window, the cursor moves normally as if nothing had happened. As described above, since the left and right pupils obtained from the two cameras can be easily matched (matching), the pupil cannot be detected even though the eyes are open due to spectacle reflection or the like. ,It is valid.

The figure which shows the example of the pupil detection by the difference of a bright pupil image and a dark pupil image. (A) is a bright pupil image, (b) is a dark pupil image, and (c) is a difference image. The figure which shows the example of arrangement | positioning of the light source in a camera The figure which shows another example of arrangement | positioning of the light source in a camera The figure which shows the difference by the image with spectacles reflection. (A) is a bright pupil image in which a spectacle reflection image is shown, (b) is a dark pupil image in which a spectacle reflection image is shown, and (c) is a diagram showing an image obtained by reducing the luminance of a high-luminance portion. . The figure which shows the example which provided the light source in the right and left of a camera. (A) is the positional relationship between the camera, the light source and the pupil, (b) is a part of the camera image when the left light source is lit, and (c) is a part of the camera image when the right light source is lit. Show. The figure which shows the example which provided two cameras. (A) is a positional relationship between the camera, the light source, and the pupil, (b) is a part of the left camera image (dark pupil) when the right light source is turned on, and (c) is a left camera image when the left light source is turned on. (D) is a part of the right camera image (bright pupil) when the right light source is lit, and (e) is a part of the right camera image (dark pupil) when the left light source is lit. ). The figure which shows the example which provided two cameras and provided two light sources in each. The figure which shows the example which provided the band pass filter The figure which shows a case where spectacles reflection corresponds with a pupil The figure which shows the concept for calculating | requiring the eyeball center. (A) shows a state where the line of sight is directed to a position deviated from the camera, and (b) shows a state where the line of sight faces the camera. Diagram showing details for finding eyeball center The figure which shows the example from which the eyelid and the pupil were extracted. (A) shows a state where the eyelid is reflected, and (b) shows a state where the eyelid image is removed by the arithmetic processing. Diagram showing the concept of separability filter The figure which shows the luminance extraction point in a separability filter The figure which shows the modification of the brightness | luminance extraction point in a separability filter

Explanation of symbols

1 Camera 2 Camera opening 3 Light source close to camera opening 4 Light source far from camera opening 5 Light emitting diode (LED)
6 Light source 7 Eyeglass lens 8 Cornea 9 Pupil 10 Eyeglass reflection 11 Band pass filter

Claims (6)

  One video camera (1), a first light source (3) near the opening of the video camera, a second light source (4) far from the opening, and the first and second light sources An imaging system consisting of light source control means for alternately lighting in synchronization with the field or frame of the video camera and detection means for detecting the pupil from the difference between images obtained at each lighting time is N (where N is N> 1). An integer) set, a pupil detection device in which the wavelength of the light source differs between these sets, and each video camera is provided with a band-pass filter corresponding to the wavelength of each light source.   One video camera (1), a first light source (3) near the opening of the video camera, a second light source (4) far from the opening, and the first and second light sources An imaging system consisting of light source control means for alternately lighting in synchronization with the field or frame of the video camera and detection means for detecting the pupil from the difference between images obtained at each lighting time is N (where N is N> 1). An integer) set, a pupil detection device in which the light emission timings of the light sources differ between these sets, and each video camera is provided with a shutter function corresponding to the light emission timing of each light source.   In a photographed image of a subject wearing glasses, sunglasses, etc., when the pupil cannot be detected in the image of one of the cameras due to the reflection of the glasses of the light source, and the pupil can be detected in the image of the other camera The pupil detection device according to claim 1, further comprising detection control means for performing pupil detection using the image.   The video camera has an angle of view at which both pupils of the subject can be photographed. When the information on the presence or absence of the left and right pupils from the two camera images does not match, the detection is performed according to the movement of the pupil of the camera image that can be detected Until the pupil is detected, the pupil extraction window continues to be given while predicting the pupil position. When the same pupil in both the left and right cannot be detected in both images, The pupil detection device according to claim 3, further comprising determination means for determining that the eyes are closed.   The pupil detection device according to claim 3, wherein the detection control means has a function of making the opening / closing of the left and right eyes correspond to pressing state information of the left and right push button switches of the pointing device.   The pupil detection device according to claim 3, wherein the detection control means has a function of creating a signal for controlling a cursor position on the display from the obtained pupil position.