JP2005352580A - Device and method for pointer control signal generation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for pointer control signal generation that outputs a signal corresponding to a left/right button depression signal of a mouse by movement of the head and moves a cursor. <P>SOLUTION: There is provided the device for pointer control signal generation that detects whether the eye is open or closed and outputs a signal corresponding to a left/right button depression signal of a mouse by movement of the head. To detect whether the eye is open or closed, the face is irradiated by an invisible infrared light source and photographed by a video camera with such a magnification that the face is photographed to the overall width with a margin, and the pupil in a picture photographed by the video camera is detected. In combination of movement of the cursor on a monitor of a personal computer etc., in timing to movement of the pupil in the picture caused by movement of the head, a choice in a menu displayed on the monitor can be made, characters can be inputted by using a software keyboard, and a plurality of windows can be switched. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a signal generation method and apparatus corresponding to pressing of left and right buttons in a pointing device that can be substituted for a mouse or joystick as an input device in a system such as a computer, a game machine, or an industrial device.

Conventional method and problems:
Various pointing methods based on the viewpoint (user's line of sight) have been proposed so far (for example, http://www.ericainc.com/ (Human-computer interaction using eye-gaze input.pdf), http: //www.eyegaze.com/ etc.)
However, when pointing depending on the viewpoint, the viewpoint must be moved to the location on the display device where the user wants to point, but the pointer (hereinafter referred to as the `` cursor '') remains at the position of the viewpoint while searching for the position to be viewed. Since it always appears, there is a problem that it is difficult to move the cursor to the position where the cursor is to be moved. In addition, when a cursor is always presented at a viewpoint position, it becomes a visual stimulus, and there is a problem that it is difficult to see an object to be seen.

According to the movement of the head, the position where the cursor appears is different from the one you are looking at, and the cursor is at the target position while looking at the cursor in peripheral vision while looking at the necessary place on the front display device Can be moved.
Example of conventional pointing system with head movement:
・ Eagle eye (refer to Non-Patent Document 1 and Web address http://www.cs.bc.edu/~gips/CM/)
Eagle eye installs a video camera on the side of the computer, captures a face image, tracks part of the face specified in advance by template processing by image processing, and on the computer screen according to the movement This is a method of moving the cursor. As the tracking target, the tip of the nose, the tip of the chin, the black eye, etc. are tried. However, even if the template is updated, there is a possibility that it may be difficult to cope with a sudden decrease in ambient brightness, and problems such as a gradual shift in the target to be tracked are likely to occur.
・ Origin Instruments Corporation (see http://www.orin.com/access/)
The Origin Instruments method detects the movement of the marker attached to the head and moves the cursor accordingly. The problem with this is that the markers must be attached one by one. For this reason, it is generally difficult to use.

-Attach a marker made of a sticker to the face of the face, etc., illuminate the face with an infrared light source installed next to the video camera, detect the marker, detect the two-dimensional position of the marker that moves with the movement of the head, There is a method to move the cursor on the PC screen. The problem with this is that the marker must be attached to the face.
-Method using a magnetic sensor: This is inconvenient because it is necessary to attach a magnetic sensor to the head.
・ Method using ultrasonic waves: It is necessary to attach an ultrasonic sensor to the head.
Further, these prior arts cannot perform an operation equivalent to a click with a mouse, and it is necessary to use another auxiliary input device for a speedy click operation.
"The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access for People With Severe Disabilities", (Margrit Betke, James Gips, and Peter Fleming), IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 10, No. 1, March 2002

  Conventional devices that move the cursor on a monitor such as a personal computer with the movement of the head require separate auxiliary input means for selecting a menu, dragging, and clicking.

According to the present invention, the open / closed state of the eye is related to the operation of the left and right buttons of the mouse, and the same operation as that of the mouse is performed using only the head.
Linking the movement of the head in the horizontal and vertical directions with the movement of the cursor in the vertical and horizontal directions can use a technique such as tracking a marker attached to the forehead as in the prior art. More preferably, by detecting the eyes, particularly the pupil, as described later, there is no sense of incongruity of wearing the marker, and convenience is improved. It is more useful to provide signal synthesizing means so that a normal pointing device can be used together in case the cursor is difficult to move only by moving the head.
The generation of signals corresponding to the left and right buttons of the mouse is linked to the opening / closing operation of the eye (eye). In case it is difficult to open and close one eye, other auxiliary input devices such as a lever switch can be used in combination.
The following description focuses on detection of opening / closing of eyes by pupil detection, but it goes without saying that various modifications such as black eye detection and white eye detection are possible.
In addition, the movement of the head will be described mainly by the detection of pupils. Obviously, other methods such as extraction by pattern matching or luminance separation may be adopted.

The present invention detects an open / closed state of an eye and outputs the open / closed state as a signal corresponding to a left / right button press signal of a mouse.
The eye open / closed state is detected by illuminating the face with an invisible infrared light source, shooting with a video camera at a magnification that allows the entire width of the face to be easily entered, and two or one of the images taken by the video camera. A method of directly detecting the pupil can be considered. In addition, it is possible to use a technique based on black eye detection or white eye detection, or a technique for detecting the opening / closing of the eye by detecting an image change of the entire eye.
The cursor on the monitor such as a personal computer is moved according to the movement of the pupil in the image caused by the movement of the head, the menu displayed on the monitor is selected, and characters are entered using the soft keyboard. Or switch between multiple windows. Further, the device can generate a signal corresponding to pressing of the left and right buttons of the mouse, and there is no need for another input operation device for click operation.

In pupil detection, a light source installed near the optical axis of the camera and a light source installed far away are turned on alternately for each field, and a bright image and a dark image of the pupil are obtained, and image difference calculation is performed. This facilitates pupil detection.
The principle of obtaining bright and dark images of the pupil is based on the phenomenon that light reflected through the pupil of the eyeball is reflected by the retina and returned to the original light source. This is based on the knowledge that reflected light cannot be recognized even when observed at a position off the axis.

FIG. 1 shows the configuration of the entire system.
The camera 1 is a general-purpose CCD video camera sensitive to near-infrared rays, one light source 2 near the opening of the camera 1 (portion through which light passes), and another one away from the opening of the camera. Two light sources 3 are provided. The camera 1, the light source 2 and the light source 3 are connected to the control device 4. The control device 4 is connected to a personal computer (hereinafter referred to as “personal computer”) 5 having a monitor 6. The connection interface is preferably a PCI bus interface when the control device is inserted into the PCI slot, and is preferably a USB interface when an external adapter is used.
Furthermore, it is also possible to use the personal computer itself as a control device, and in that case, it is conceivable to provide a light source irradiation (flashing) control part separately.
It is possible to divide the control functions of the control device as desired, and to distribute functions as appropriate, such as a part executed by software, a part executed by hardware, a part executed by the personal computer itself, and a part executed by the PCI board. Will be understood by those skilled in the art.
Further, the present invention is not limited to a personal computer, and is a system having a pointer position control interface displayed on a display by a signal from a pointing device and an interface for transmitting an operation instruction by a user push button associated with the pointer position to software. It is clear that it is applicable. For example, when applied to a television, an optical remote control signal is used as an interface to enable operations corresponding to some push buttons on the remote control.

The target person (user) 7 is caused to alternately emit the light source 2 in the vicinity of the camera and the light source 3 away from the camera, and a reflected image from the pupil is captured by the camera 1. The controller 4 calculates the difference between the image from the light source 2 and the image from the light source 3, and transfers the obtained change in the pupil position to the personal computer 5 through the USB interface. In the personal computer 5, the signal from the control device 4 is regarded as equivalent to a mouse which is a kind of pointing device, and the position of the cursor 8 displayed on the monitor 6 is controlled.
Further, the control device 4 detects an operation of closing only one eye and generates a signal corresponding to pressing of the left and right buttons.
Here, the control device controls the light emission of the light source, but there is no problem in operation as long as it can capture an image signal in synchronization with light emission / extinction of the light source, for example, in synchronization with the commercial frequency. A flashing light source and a control device that captures an image in synchronization with the commercial frequency can be used.

The system including the camera 1, the light source 2 in the vicinity, the light source 3 away from the camera, and the control device 4 has the same functions as those of a normal personal computer mouse. Therefore, it is preferable to output signals corresponding to the direction and speed of movement, two buttons, and the like in exactly the same format as in an ordinary mouse. Note that this pupil mouse and normal mouse can be used in combination. Connect this system to another personal computer via a USB cable. Therefore, it looks like a normal mouse for a personal computer. If you connect a normal mouse to the same computer at the same time, you can use this system and the mouse together.
Here, the connection interface is exemplified as USB, but the present invention is not limited to this, and PS / 2, RS232C, a parallel port, and the like can also be used.
In addition, although it has been described that the amount of movement of the head is detected by pupil detection and the cursor is moved, detection of markers and magnetic bodies provided on the head (including the face), as well as nostrils, nasal head, chin Obviously, other methods such as extracting the lips, mouth, corners of the mouth, between the eyebrows, the eyes, the corners of the eyes, the black eyes, and the white eyes by pattern matching or luminance separation may be adopted.

In order to input characters with this system, it is preferable to use a software keyboard as described below.
A keyboard input tool as shown in FIG. 2 is displayed on the display screen of the personal computer by software. Move the cursor to the desired character and click to enter the character into general application software such as Word (trademark) (Web address http://orin.com/access/softype/index. see htm). If you move the cursor to a place other than this keyboard input tool, you can do the same as a normal mouse operation.
As a modification of this, the cell of the character is arranged in a matrix, the selected cell is in a high luminance (or low luminance, luminance inversion, blinking, bordering) state, and the cell selected by the cursor movement signal moves up, down, left and right The form to be included is included. In this case, a display form in which the cursor icon is hidden is allowed.

First, the features and limitations of pupil detection will be described.
In order to prevent the pupil from misrecognizing other parts, the change in the image of the eye part of the raw image immediately after the blink (when the pupil reappears) is separated by several frames (for example, 4 frames apart from each other). Detected by subtracting raw images from each other). At this time, a change in the image of the eye portion is detected. Once detected, a window centered on each pupil is provided, and a pupil search is performed only within the window. In subsequent frames, if it is determined that the pupil is present again in the given window, it is determined that the pupil is present, and either eye is closed and the pupil is no longer detected in the corresponding window. , It is determined that the corresponding pupil has disappeared. If either the left or right pupil is detected in the previous frame and only one pupil is detected from the current frame, it is determined that the same pupil as the previous frame has been detected.

On the way, if the pupil is not detected even if the eye is open for some reason, the window is canceled and pupil search is performed on the entire image. However, when one of the pupils is detected, no search is performed in the window given to that pupil. Once the eyes are closed and then reopened, the pupil is again detected and a window is provided to detect changes in the image of the eye. Therefore, generally, if the act of closing and reopening the eyes is not performed, the non-detected pupil remains undetected even if the eyes are opened. Both pupils come to be detected again once, for example, by blinking normally.
In addition to this, if the pupil is not detected even though the eyes are open, if the face is shaken in small increments, the image changes in the eyes, resulting in the same effect as blinking, The pupil is detected again without blinking. Such an effect also occurs while moving the head quickly. Due to the rapid movement, the pupil image may not be clearly visible due to blurring, but the pupil may not be detected temporarily. The pupil may be detected again when the speed decreases.

  In detecting the movement of the head and creating movement amount information, (1) the position of the head is set as the cursor position (named control by absolute amount), (2) the movement amount of the head per unit time The amount of movement per unit time from the current position of the cursor (referred to as control by relative amount). (3) The amount of movement of the cursor per unit time is generated in proportion to the deviation from the origin of the head. There are three possible cases (named control by integration amount). The above (1) corresponds to the conventional joystick, and the cursor position is determined by an amount corresponding to the displacement amount of the lever (control of the absolute position). (2) corresponds to the current mouse or trackball, and the cursor moves by an amount commensurate with the amount of rotation (control of the relative position). The above (3) corresponds to a stick-like controller found in current notebook personal computers, and the cursor continues to move while the lever is tilted (position determination by integration). It can be said that the cursor control by the cursor key corresponds to the above (3).

Further, depending on the application, (4) it is conceivable to move the cursor by differentiation of position information. In actual use, a dead zone is provided, or when the movement is large, a large amount of movement may be created as an acceleration function.
When detecting movement of the head and creating movement amount information, the output signal is considered depending on whether it is connected to another system using a mouse interface, or a joystick interface or another stick interface. It will be appreciated that there are a wide variety of conversion forms that are necessary.
Further, the movement of the head includes a parallel movement and a rotational movement, and the parallel movement is easier to detect, but the head movement of the present invention is not limited to the parallel movement. For example, various detection methods are conceivable as detection of rotational movement, such as a method of inferring based on the distance between pupils and a position of the corner of the eye as a point that can be tracked even when the eyes are closed.

The detection state of the pupil detected from the face image is as follows: (a) Both eyes are open and both pupils are detected (right pupil detection, left pupil detection), (b) Only the right eye is open, When only the pupil is detected (right pupil detection, left pupil non-detection), (c) When only the left eye is open and only the left pupil is detected (right pupil non-detection, left pupil detection), (d) There are four ways when both eyes are closed and both pupils are not detected (right pupil not detected, left pupil not detected).
Information output from a general-purpose two-button mouse signal includes mouse button information (down (pressed) for the right button), UP (released) and DOWN / UP for the left button) and cursor movement information (horizontal and vertical). Direction).
FIG. 3 shows a method for making the opening / closing operation of both eyes of the subject correspond to the operation of the general-purpose mouse. For example, if one eye is closed and then opened, clicking is performed, and if the head is moved while closing one eye, dragging is started, and if the eyes that are closed thereafter are opened, the dragging is terminated.
However, since clicks and drags are only judged by an operating system such as Windows (registered trademark) on a personal computer, in this method, the right button of a general-purpose two-button mouse is used to open and close the right eye and open the left eye. It is assumed that processing corresponding to UP and DOWN of the left button and UP and DOWN of the left button is performed.
However, if both eyes are closed by blinking, it is not the subject's intention and is treated as an exception. Normally, the state where both buttons of the general-purpose mouse are pressed is not used. Therefore, when both eyes are closed, it is considered that both buttons are UP. However, the pupil movement amount is assumed to be zero. Thus, when both eyes are closed, this corresponds to a state in which no mouse is operated (a state in which the hand is released). Such a situation is treated not only when both eyes are closed, but also when the face is not reflected in the camera.

(1) When blinking, both pupils are not detected, but when the right and left eyes are closed, there may be a shift of one to two frames depending on the individual. This is the same when opening the eyes after blinking. In such a case, simply matching the changes in the detection state of both pupils to the left and right buttons of a general-purpose mouse results in the same situation as if the left and right buttons were pressed slightly at different times. When this happens, many application software will behave differently from the user's intention.
(2) Further, when the head moves quickly, the pupil also moves fast, and the pupil image may be blurred at that time, resulting in a decrease in peak luminance value. As a result, the pupil may not be detected. This can be for both pupils when eyes are open (when the cursor is simply moved) or for one pupil. In that case, if it is determined that the detection state of the pupil has changed, button information different from the intention of the subject is output.
In order to prevent the problems (1) and (2) above, the following measures are taken.
Even if the detection state (detection, non-detection) of the pupil obtained from the current image frame changes from the previous frame to another state, the same state continues for several frames (eg, 3 frames, 100 ms) after the change. It is assumed that the button information corresponding to the pupil state after the change is output for the first time.

In pupil detection from a face image, when both pupils are detected, the left and right pupils are determined from the X coordinate of each pupil. Therefore, when either eye is closed from the state where both pupils are open, it is determined from the coordinate comparison whether the left or right pupil has changed from the detection state to the non-detection state. In addition, when opening from one eye to the other after closing both eyes, it is not possible to know which eye is open first until both eyes are opened. Until it is determined whether the pupil is left or right, the left and right of both pupils are distinguished only after both pupils are detected. Therefore, if both pupils have not been detected before, the button operation remains in the UP state on both sides until it is determined that both pupils have been detected (even after both pupils have been detected. It is assumed that the movement of the cursor follows the amount of movement of the pupil of one eye.
On the other hand, there may be a transition from a state where only one pupil is detected to a state where only the other pupil is detected. This occurs when one eye is closed during dragging, and when it is determined that the closed eye has just opened at the end of dragging, the pupil that has been opened for some reason is no longer detected. However, assuming that it is impossible for the user to intentionally change from the state where only one eye is closed to the state where only the other eye is closed, the other pupil that should be detected is detected for some reason. Accordingly, it is determined that the state has changed from the state in which one eye is open to the state in which both pupils are detected. Thereafter, the cursor is moved in correspondence with the movement of only the detected pupil.

  When both pupils continue to be detected (corresponding to mere cursor movement), it is desirable to use the movement amount of the center of gravity of both pupils between successive frames as the pupil movement amount, and to correspond to the cursor movement amount (Example 6). reference). When only one pupil is continuously detected, the detected pupil movement amount between successive frames is made to correspond to the cursor movement amount as the pupil movement amount (see Example 6). In addition, as described in the first embodiment, during the head movement (although it is not always necessary), the user intends to open both eyes regardless of the intention. When only one pupil can be detected in a shorter time (also occurs when spectacle reflection overlaps the pupil image, etc.), the movement amount of one pupil corresponds to the cursor movement amount during that time. In addition, as described in the first embodiment, the user intends to open one eye during the head movement (although not necessarily moving), regardless of the intention, but for a specified number of frames. When both pupils cannot be detected in a shorter time, it is assumed that the pupil does not move during that time, and the cursor does not move.

Although the user is trying to move the cursor, the pupil does not move during several frames in which the pupil could not be detected. In this case, the head may turn around in a state where the cursor does not move, which makes it difficult for the user to deal with it later. Therefore, the difference between the pupil position immediately before (one frame before) and the pupil position immediately after several frames in which the pupil could not be detected is made to correspond to the cursor movement amount as the pupil movement amount.
When both pupils are detected continuously, if both pupils cannot be detected in a shorter period of time than the specified number of frames, regardless of the user's intention ( In the case of the occurrence of the However, this is not applied when one of the pupils continues to be detected. In this case, the movement amount of one pupil is made to correspond to the cursor movement amount as the pupil movement amount (however, as in the second embodiment) It is preferable that the left and right buttons are output as UP.

When the subject is nervous, the head vibrates, the pupil vibrates accordingly, and the cursor vibrates. (This is not a problem with pupil detection, but the problem of trying to move the cursor with the movement of the head itself). This is solved by the method described later. First, a method that can be generally considered will be described.
Usually, the correspondence between the obtained pupil movement amount and cursor movement amount
Cx = kx · Px,
Cy = ky · Py (1)
Can be associated with each other. Here, Px and Py are pupil movement components in the horizontal and vertical directions (with positive and negative), respectively, and Cx and Cy are cursor movement amounts in the horizontal and vertical directions on the personal computer screen, respectively. Kx and ky are coefficients for selecting a favorite value depending on the subject.

In moving the pupil seen from the camera by rotating or moving the head, if kx and ky are set large, the cursor moves greatly with the movement of the small head, but instead, the subject stops the head. The cursor vibrates due to the vibration of the head even though he intends to be. Conversely, if kx and ky are set to a small value, the cursor cannot be moved greatly unless a large head movement is performed. Even if an appropriate value is selected as much as possible, when the cursor is moved to a desired position, the cursor vibrates and alignment is difficult.
As a solution,
Cx = kx · Px · | Px | ^b−1 ,
Cy = ky · Py · | Py | ^b-1 (2)
Is to correspond. When b> 1, the orders of Px and Py are higher than in the case of equation (1), and when b = 2, the square of the pupil movement amount is made to correspond to the cursor movement amount (a in FIG. 4). By such association, pupil vibration is suppressed to almost zero, and pupil movement due to head movement is emphasized. Here again, kx and ky are set according to the subject's preference. In this way, when the order is increased, the cursor moves greatly even with a small head movement if the head is moved relatively quickly, and conversely, if the head is moved slowly, the cursor can move only slightly. become. Therefore, it is suitable for both when it is desired to move the cursor greatly and when the cursor is slightly displaced from the desired position and when it is desired to align it with the desired position.

However, even if b = 2, if the vibration of the head adversely affects the movement of the cursor, a so-called dead zone is provided. That is, when Px and Py are equal to or less than the specified threshold values, Px = 0 and Py = 0 (or Cx = 0 and Cy = 0) (c in FIG. 4 (when b = 1)) or FIG. D of 4 (when b = 2)). In this way, the cursor does not move at all with a small head vibration in the dead zone range.
Or, after providing a dead zone, record the integral value of all pupil movements during small pupil movements within the dead zone range, and if there is a large pupil movement that exceeds the dead zone, set the integral value Used to move the pupil (c, d in FIG. 4). When the cursor is controlled according to the direction in which the face is directed, the cursor can be instantaneously directed substantially in the direction in which the face is directed in this way.
Furthermore, if the relationship as shown in FIGS. 4B and 4E is used, the cursor does not move at all with a small vibration of the head, and can gradually move as the head moves larger than the dead zone. In particular, in e of FIG. 4, when the dead zone is exceeded, the cursor moves a little, and when the head moves more than that, the cursor movement can be rapidly increased.

  When you click or start dragging, close your eyes with your eyes open. When the drag is finished, the eye is opened from the closed state. In this way, in a short time when the eyes are opened and closed (for example, between two frames), generally, a human moves his eyes unless he / she is careful. If no special processing is performed, the cursor moves according to the movement of the eyes, and the cursor moves to a position different from the user's intention, so click or drag a place different from where you want to click or drag. Will end up. In order to prevent this problem, it is desirable that the coordinates of the cursor position at the time when the user moves the cursor to the target position and closes or opens the eyes become the input information. Therefore, for example, the amount of cursor movement over the recent several frames is always recorded, and the time is traced back by several frames from the frame where it is determined that the start or end of the click or drag has occurred (this is referred to as frame B). A process for returning the cursor to the cursor position in the frame (referred to as frame E) is performed. That is, the value obtained by subtracting the integral value of the pupil movement amount from frame B to frame E from the pupil movement amount in frame B is replaced with a new pupil movement amount and used for cursor movement.

Some users are not good at opening and closing one eye. For such use, the part corresponding to the button operation of the pupil mouse can be substituted by another button pressing device (including a general-purpose mouse, a keyboard, etc.) or used together. For example, it is useful when controlling a cursor on a large display.
It is also useful to use a pupil mouse and a general-purpose manual mouse in combination.
A general-purpose manual mouse is good at fine cursor movement, but in order to move a large cursor, it is difficult to use it because it is necessary to move the hand greatly or to lift it back several times. On the other hand, in the pupil mouse, the cursor moves quickly for a large head movement (rotation). For fine work, the pupil mouse is kept from acting and the cursor is moved by the manual mouse.

In addition, the cursor is often lost on a large display, making it difficult to use. By adjusting the pupil mouse and moving the cursor to a large extent in accordance with the rotation of the head, the cursor can be moved in a direction almost facing the head, which makes it easier to find the cursor.
In order to do the above, the dead zone described at the end of the sixth embodiment is set to be large, and the cursor is set not to move at all by small pupil movement. Then, the integrated value of all pupil movements is recorded, and when there is a large pupil movement exceeding the dead zone again, the pupil value is moved using the integrated value. By doing so, it is possible to point the cursor instantaneously in the direction of the face.

So far, the description has been given assuming that the closed state of the eye is directly associated with the mouse button press. In this case, the OS (or application software) determines that it is double-clicked by the second blink of one eye, and executes a corresponding operation. On the other hand, there is a user who is difficult to perform an operation corresponding to dragging by moving the head with one eye open. In order to cope with this, when the second blink of one eye is detected, if it is determined that the mouse button is locked in a pressed state, a drag operation can be performed with both eyes open.
Usefulness is enhanced by configuring the correspondence in the means for associating the opening / closing information of the left and right eyes with the pressing state information of the left and right push button switches of the pointing device in accordance with the use situation.
As another form, it is also possible to discriminate the repetition cycle of one-eye blinking, for example, to make two blinks in the early cycle double-click, and to make the second blink in the late cycle push-down lock. . Furthermore, a modification that generates various button press signals including the locked state is conceivable depending on conditions that are easy for the user to operate, such as a press lock with three blinks.

  By connecting to the TV system or attaching an eye open / close detection function to the TV itself, the channel is changed to increase each time only the right eye (left eye) is closed and opened, or only the left eye (right eye) is closed and opened. It is possible to change the channel in a decreasing direction each time. Further, for example, the volume may be increased until it is opened if only the right eye (left eye) is closed for 1 second or longer, and the volume is decreased until it is opened if only the left eye (right eye) is closed for 1 second or longer. As a modification of this, the right eye (left eye) may be assigned to channel control and the left eye (right eye) assigned to volume control. When connecting to a television system, an interface using a light remote control signal may be useful.

The present invention provides a system having a pointer position control interface displayed on a display by a signal from a pointing device such as a personal computer or a game machine, and an interface for transmitting an operation instruction by a user push button associated with the pointer position to software. Control signals are generated, and it is possible to perform operations equivalent to a mouse without using a hand on a personal computer, and various actions can be performed on a game machine by using it together with a controller. I don't know.
Depending on the application, cursor control may not be necessary. In this case, various modifications such as connection to the system using only the interface relating to the push button are easy.

System configuration diagram Display example of keyboard on monitor by software The figure which shows correspondence with opening and closing movement of both eyes of the test subject and movement of the general purpose mouse Figures showing some examples of the proportional relationship between the amount of movement of the head and the amount of movement of the cursor

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2, 3 Light source 4 Control apparatus 5 Personal computer 6 Monitor 7 Target person (user)
8 Cursor

Claims (13)

  This is a method for generating a control signal for a system having a pointer position control interface displayed on a display by a signal from a pointing device and an interface for transmitting an operation instruction by a user push button associated with the pointer position to software. Detecting the movement of the head and creating movement amount information; detecting the opening and closing of the left and right eyes; creating opening and closing information; and right and left push buttons of the pointing device A pointer control signal generation method comprising the steps of associating with switch pressing state information and outputting the movement amount information and pressing state information as control signals.   2. The pointer according to claim 1, wherein the step of associating associates the closed state of the left eye with a pressed state of the left (or right) button and associates the closed state of the right eye with a pressed state of the right (or left) button. Control signal generation method.   The step of creating the opening / closing information does not immediately change the opening / closing information even if a change in the opening / closing state of the eye is detected, and confirms that the opening / closing state continues for a while and changes the opening / closing information to a new state. The pointer control signal generating method according to claim 1.   The step of creating the opening / closing information does not change the opening / closing information until an open state of both eyes is detected when it is detected that a transition from a non-detection state of both eyes to a one-eye detection state is detected. The pointer control signal generation method described.   The step of creating the movement amount information is to detect the movement of the head by detecting the position of the eyes to create a movement amount, and when both eyes cannot be detected, the pointer is not moved. 2. The pointer control signal generation method according to claim 1, wherein the movement amount information is set.   The step of creating the movement amount information is to detect the movement of the head by detecting the position of the eye and create a movement amount, and both eyes or one eye are detected until immediately before, and temporarily When both eyes can no longer be detected, the pointer is not moved until detection is possible, and when the detection is possible again, movement amount information is created to move the pointer to a position corresponding to the new eye position. 2. The pointer control signal generating method according to claim 1, wherein the pointer control signal is generated.   2. The pointer control according to claim 1, wherein the step of creating the movement amount information is to create movement amount information so that a nonlinear proportional relationship is provided between the movement amount of the head and the movement amount of the pointer. Signal generation method.   Further, in order to correct the movement of the head accompanying the opening and closing of the eye, the step of correcting the movement amount information from the state at the time of opening and closing of the eye to the past to be the amount of movement in a stable state immediately before the opening and closing. The pointer control signal generation method according to claim 1.   2. The pointer control signal generating method according to claim 1, further comprising a step of synthesizing a signal from an input means such as a push button switch or a lever switch with opening / closing information of both eyes.   2. The pointer control signal generating method according to claim 1, further comprising a step of combining a signal from a pointing device such as a mouse or a joystick with head movement amount information.   The pointer control signal generation method according to claim 1, wherein the movement of the head is detected by detecting a position of a marker fixed to the head.   The movement of the head is selected by selecting one or more facial features such as pupils, nostrils, nasal head, chin, lips, mouth, mouth corners, eyebrows, eyes, corners of eyes, black eyes and white eyes, and their positions. 2. The pointer control signal generation method according to claim 1, wherein said pointer control signal generation method is performed by detecting the above. A device that generates a control signal for a system having a pointer position control interface displayed on a display by a signal from a pointing device and an interface for transmitting an operation instruction by a user push button associated with the pointer position to software. Means for detecting movement of the head and creating movement amount information, means for detecting opening and closing of the left and right eyes and creating opening and closing information, and right and left push buttons of the pointing device A pointer control signal generator comprising means for associating with switch pressing state information and means for outputting the movement amount information and the pressing state information as control signals.

Images