JP2007310914A - Mouse alternating method, mouse alternating program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mouse alternating method operating a program with a GUI as a medium without requiring a handicapped person lame in the both hands to wear something. <P>SOLUTION: Image information of facial organs, such as eyes and a mouth is cut out while an operator faces a camera and registered as an image A in a template To, and the position of the image A in an input image is simultaneously registered as an operation reference point. The coordinates of an area where the template To coincides the best in the input image are detected as template position coordinates in real time. The position coordinates of a mouse pointer in the next frame is calculated from the template position coordinates and the coordinates of the operation reference point to move the mouse pointer. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a mouse replacement method, a mouse replacement program, and a recording medium on which the program is recorded.

  A program that requires cooperation with a user in various types of information processing using a personal computer (PC) often processes user input through a GUI (graphical user interface). For example, a feature is that many functions such as starting and ending a program are realized by GUI operations that are easy to understand intuitively, such as instruction, selection, and movement of objects such as icons and menus.

  Many of these operations require input devices using hand movement functions such as a mouse for selecting objects and a keyboard for inputting characters. In order for a user with a disability in the functions of both hands to use it as it is, a dedicated means is required. In particular, various devices have been developed as input means for personal computers for people with physical disabilities, such as methods that substitute mouse operations using the remaining functions of the body, such as obstacles that can move the head at will A method for instructing a button or icon on the screen using the movement has been developed.

  In the alternative method of the mouse using the movement of the head, products such as a head space pointer and head master plus equipped with various sensors on the head and a mechanism for detecting the vertical and horizontal movements, There are products such as a method of attaching a reflective marker to the head like a head mouse and extracting it from a moving image photographed by a CCD camera or the like to a marker region by image processing.

  In addition, there is a method of pointing by attaching a sensor or a marker to a movable part of the body as disclosed in JP-A-10-320108 and JP-A-7-1441098.

  It should be noted that the mounting position of the sensor or marker in these products or inventions may be a movable position of the body in principle, and can be operated even if it is mounted on, for example, a hand or a foot.

In either method, the extracted sensor position or the coordinate position of the marker is displayed as a pointer on the monitor of the personal computer after coordinate conversion as appropriate. Thereby, objects such as icons and buttons on the screen can be instructed by the vertical and horizontal movements of the markers and sensors.
Japanese Patent Laid-Open No. 10-320108 Japanese Unexamined Patent Publication No. 7-141098

  However, in the operation of the PC by the conventional method described above, it is necessary to attach various sensors, a headset, a marker, and the like. Although these have been improved and lightweight and compact ones have been developed, they are necessary only during the operation of the PC, so they must be worn and fixed each time they are used, especially when used by people with physical disabilities. And the burden on the caregiver.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a mouse replacement method that allows a user with a disability in both hands to operate a program mediated by a GUI without the need for a wearing accessory. .

  In order to achieve the above object, the present invention uses coordinates obtained by tracking a specific facial organ from an image obtained by photographing a face with a video camera or the like using image features of the facial organ such as eyes and mouth. The mouse is moved by moving the mouse pointer based on the above, and by selecting the object such as an icon by detecting a change in facial expression such as intentionally closing the eyes and opening the mouth, and issuing a command instruction to the object, etc. The alternative function is realized.

  The mouse replacement method of the present invention takes four approaches.

  Specifically:

The mouse alternative method of the present invention comprises:
An image obtained by cutting out a facial organ from one frame of an input image obtained by photographing a face part of a computer operator is registered as a first template,
At the same time, the position of the image in the input image is registered as an operation reference point, which is a reference point for calculating the amount and direction of movement of the pointer.
Detecting a template position coordinate which is a coordinate of an area where the first template is best matched in an input image input in frame units via an imaging device;
Calculate the position coordinate of the mouse pointer in the next frame from the template position coordinates and the coordinates of the operation reference point,
Move the mouse pointer to the coordinates,
Image information on a different facial expression of the same facial organ as the first template is cut out from the input image obtained by photographing the operator and registered as a second template,
Detect the second template in the input image,
When the second template is continuously detected over a predetermined time or frame number, the value of the expression counter is counted with this as one unit,
A pre-registered instruction command is issued to the OS according to the counted expression counter value.

According to this configuration, image information of the operator's facial organ is used as the first template image, and by tracking the information, the mouse pointer of the PC is moved in conjunction with the operator moving the face up, down, left, and right It becomes possible to make it.
Further, according to this configuration, the operator performs a predetermined operation such as intentionally closing his eyes or opening his mouth for a certain period of time once or more, so that a single click or double click according to the number of times. The command command related to the mouse button can be executed.

  In another embodiment of the present invention, pre-registered instruction commands are sequentially presented to the operator according to a certain rule, and when the desired instruction command is presented, the operator changes the facial expression over a certain time. The instruction command presented when the two templates are continuously observed over a certain time or a certain number of frames is issued to the OS.

  According to this configuration, a plurality of instruction commands relating to mouse buttons such as single click and double click are periodically presented to the operator, while the operator is in accordance with the desired instruction command being presented. By performing a predetermined operation once, a desired one can be selected from a plurality of instruction commands, and an instruction command can be issued to an object such as an icon or button indicated by the mouse pointer at that time. .

According to another embodiment of the present invention, a template position coordinate relating to the first template and an operation reference point are compared with an input image from an imaging device, and the position of the template position coordinate with the operation reference point as the origin is determined. The coordinates are converted into coordinates with the center of the PC screen as the origin, and a mouse pointer is displayed at the coordinates.

  According to this configuration, since the mouse pointer is moved by an amount proportional to the amount of movement of the face in the direction in which the operator moves the face with the center of the PC screen as a reference, the operator simply moves the face. The mouse pointer can be moved to any coordinates on the screen.

  According to another embodiment of the present invention, a template position coordinate related to the first template and an operation reference point are compared with an input image from an imaging device, and the template position coordinate is determined from the direction and distance from the operation reference point. Determines the direction of movement of the mouse pointer and the amount of movement per frame.

  According to this configuration, the moving direction of the mouse pointer is set according to the direction in which the operator moves the face, and the moving speed of the mouse pointer is set according to the amount of movement of the face. The mouse pointer can be moved to any coordinates on the screen.

  According to another embodiment of the present invention, the mouse replacement method can be realized by a computer program.

  Further, according to another embodiment of the present invention, it can be realized as a recording medium on which a mouse substitute program is recorded.

  Next, embodiments of the present invention will be described with reference to the drawings.

<< First Embodiment >>
The first embodiment tracks a specific facial organ of the operator, moves the mouse pointer from the center of the screen by an amount proportional to the amount of facial movement, and simultaneously changes facial expressions in the same facial organ of the operator An instruction command is issued to the OS when an error is detected.

  FIG. 1 is a block diagram of an apparatus for carrying out the mouse replacement method of the first embodiment of the present invention.

  The information processing apparatus 1 includes a main control device 11, a storage device 12, a display device 13, a moving image acquisition device 14, and a bus 15, and a camera 2 is connected to the moving image acquisition device 14.

  The main control device 1 is composed of an MPU, a RAM, and the like, and manages the information processing device 1 in an integrated manner. For this reason, in the main control apparatus 1, a template image registration unit 21, a template matching unit 22, a pointer coordinate calculation unit 23, a command transmission determination unit 24, and an OS 26 are activated as a mouse substitute program 20. In addition, an application program 27 such as word processing software is also running on the main control device 11. The OS 26 has a function of executing various commands relating to the mouse such as movement of the mouse pointer and single click and double click of the mouse button.

  The storage device 12 is composed of a hard disk device or the like. The storage device 12 stores a mouse substitute program 20, an OS 26, an application program 27, and the like.

  The display device 13 is composed of a liquid crystal monitor or the like, and displays the processing result in the main control device 11 including the movement of the pointer.

  The moving image acquisition device 14 has a function of sequentially acquiring the moving image obtained from the camera 2 in the RAM included in the main control device 11 in units of frames. A commercially available video capture board can be used as the moving image acquisition device 14, and a commercially available CCD camera or the like can be used as the camera 2.

  Next, the facial organ template, the operation reference point, and the template position coordinates will be described with reference to FIG.

  The operator's head is photographed by the camera 2, and an image including each organ of the face is captured as an input image by the moving image acquisition device 14 into the main control device 11 as shown in FIG. 2. It is assumed that the camera 2 is photographing the operator's head in a fixed state.

  FIG. 2A shows an input image in a state where the operator is facing the front of the camera 2. In this state, image information of facial organ parts such as eyes and mouth is cut out and registered as an image A in the template To. . The following figures use the eye region as a template. At the same time, the position of the image A in the input image at the time of registration is registered as an operation reference point. The operation reference point is a point used as a reference for calculating the movement amount and the moving direction of the pointer, and its coordinates are equal to the initial values of template position coordinates described later.

  In FIG. 2A, the lower right coordinate of the image A is used as a specific operation reference point, but this is not limited as long as the coordinate is based on a specific position in the cut image. For example, coordinates where the center of the image is located may be used.

  Here, as shown in FIG. 2B, when the operator moves his / her face up / down / left / right, the facial organs such as the eyes, mouth, and nose of the operator also move. Therefore, when template matching is performed by superimposing the template To on the input image, the region where the template To matches best in the input image changes as the facial organ moves. In the following description, the coordinates of the area where the template image A is the best match on the input image are set as the template position coordinates. The specific position of the template position coordinate is a coordinate based on the same position as the reference coordinate used in the above-described definition of the operation reference point in the template image. The template position coordinates are converted into a specific amount of movement of the mouse pointer by a pointer coordinate calculation unit 23 described later.

  Also, as shown in FIG. 2 (c), the image information of the same facial organ part in a different expression (closed eyes, open mouth, etc.) from that at the time of shooting of the image A is cut out with the same size as the image A, and the image B Register to template Tc. FIG. 2 (c) shows a state where the eyes are closed.

  Next, functions of the mouse substitute program 20 will be described.

  As described above, the mouse substitute program 20 includes the template image registration unit 21, the template matching unit 22, the pointer coordinate calculation unit 23, and the command transmission determination unit 24.

  First, the template image registration unit 21 registers, as a template To, an image A obtained by cutting out facial organs such as eyes and mouth from one frame of a moving image obtained by photographing the face part of the operator. At the same time, the position of the image A in the input image is registered as an operation reference point. Furthermore, an image B obtained by cutting out image information of different facial expressions in the same facial organ as when the image A was captured is registered as a template Tc.

The template matching unit 22 performs template matching in the input image using the given template image. As a result of template matching, the template position coordinates in the input image are calculated. Specifically, in one frame of a moving image that is sequentially updated, the degree of coincidence between the template image and a region where the template overlaps in the input image is sequentially calculated while moving the template image by a minute distance. Output the coordinates of the matching area.

  As a measure of the degree of coincidence, there are a method for obtaining a residual of density between corresponding pixels of a template image and an input image, a method for obtaining a cross-correlation coefficient, and the like. When the residual is used, it means that the value is best matched at the position where the value is minimum, and when the cross-correlation coefficient is used, it means that the value is best matched at the position where the value is maximum.

  If the template size is M × N, and the regions on the input image where the template To and the template To overlap are T (m, n) and I (m, n), respectively, the residual R and the cross-correlation coefficient C Is given by:

The meaning of the value differs depending on the scale of matching used, but in the following explanation, it is assumed that the larger the matching value, the better the matching. Further, when calculating the degree of coincidence, the image obtained from the camera 2 can be used as it is, but the degree of coincidence is calculated after pre-processing such as grayscale image extraction and extraction is performed on the image once obtained. May be.

  After the scanning of one frame is completed by the template matching unit 22, the coordinates in the input image having the highest degree of matching with the template are recorded as new template position coordinates.

  The pointer coordinate calculation unit 23 compares the template position coordinates obtained by the template matching unit 22 with the coordinates of the operation reference point, and calculates the position coordinates of the mouse pointer in the next frame based on the movement amount and movement direction of the template. Then, the OS 26 is notified.

  If the distance between the operation reference point and the template position coordinate exceeds a certain value, the area that best matches the template To in the current input image is cut out, and in the next frame, instead of the pre-recorded image information A Register as a template To. In subsequent frames, the search is performed while updating the template To as needed. On the other hand, when the distance from the operation reference point returns within a predetermined value, the image information A recorded in advance is registered again as the template To, and the search process in the subsequent frames is performed.

The command transmission determination unit 24 uses a template Tc registered in advance near the template position coordinates.
Search for. The template matching described above is used as the search method. When the operator changes the facial expression over a predetermined time and the template Tc is continuously detected over a predetermined time or the number of frames, the command transmission determining unit 24 counts this as one unit, and within a predetermined time. A pre-registered instruction command is issued to the OS 26 according to the counted value. Here, the instruction command is a command corresponding to depression, release, etc. of a mouse button, and by notification to the OS 26, selection of an object such as an icon currently pointed by the mouse pointer, and a command for that object It is an instruction.

  When the distance between the template position coordinate corresponding to the template To and the operation reference point exceeds a certain value, an area that best matches the template Tc in the vicinity of the template position coordinate is cut out and recorded in advance. Instead of the information B, it is registered as a template Tc in the next frame, and a search is performed while updating the template Tc as needed in subsequent frames. On the other hand, when the distance between the template position coordinates and the operation reference point returns within a certain value, the image information B recorded in advance is registered again as the template Tc, and the search processing in the subsequent frames is performed.

  The above operation is repeated for each frame.

  Therefore, if the operator moves the face, the mouse pointer on the display device 3 moves according to the direction, and if the facial expression is changed to a pre-registered facial expression for a certain period of time, an object such as an icon pointed to by the mouse pointer is displayed. A predetermined command can be issued. This makes it possible to substitute a mouse for PC operation based on facial orientation and facial expression.

  Next, processing in the mouse replacement method of the first embodiment will be described with reference to the flowchart of FIG.

  In the following processing, the camera 2 is fixed, and it is assumed that the face portion of the operator is photographed.

  First, the image information obtained by photographing the facial organs such as the eyes, mouth, and nose while the PC operator is stationary with respect to the camera 2 is recorded as the image A, and the image A is registered in the template To. The face organs of the same operator are photographed, the obtained image information is recorded as an image B, and the image B is registered in the template Tc (step 101).

  The position coordinates in the input image of the template To are recorded as reference points (XO, YO), and the upper limit Xmax, Ymax of the movement of the template To in the input image when the operator moves the face up / down / left / right The lower limits Xmin and Ymin are recorded (step 102).

  The initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves the face is set to “OFF”, and at the same time, a facial expression flag indicating whether or not there is a region that matches the template Tc Is set to “OFF” (step 103).

  The subsequent steps 104 to 134 are continuously executed at a constant period while the frame is updated.

  It is determined whether or not the frame has been updated (step 104). If the frame is not updated, the process is terminated (step 105).

On the other hand, when the frame is updated, the distance flag is determined (step 106). If the distance flag is “OFF”, it is considered that the facial organ is within a certain distance from the reference point, and image A is registered in template To and image B is registered in template Tc (step 107). If the distance flag is “ON”, it is considered that the facial organ is more than a certain distance from the reference point, the image An is registered in the template To, and the image Bn is registered in the template Tc (step 108). Here, the image An and the image Bn are image information of regions that best match the template To and the template Tc, respectively, in the past frames, and in the t-th frame processing, the images in the t−1-th frame processing Information is used as a template.

  When a template is created by photographing the front face of the operator, even if the face is moved only by a minute range, the distortion of the face area in the input image is small, so that a region with a high degree of coincidence is easily extracted. On the other hand, when the face is moved greatly, the distortion of the image in each organ part increases, so that it is difficult to detect by template matching using a single template. For this reason, when the image moves more than a certain range, the image information of the area that best matches the template in the past frame is used as a new template.

  The degree of coincidence P is calculated (step 109). The specific calculation uses the above-described residual and cross-correlation coefficient.

  The degree of coincidence P is compared, and the coordinates (x, y) of the region having the highest degree of coincidence with the template To in the input image are recorded as template position coordinates (step 110).

  A distance D between the template position coordinates (x, y) and the reference point (XO, YO) is calculated (step 111).

If the template movement amounts in the x-axis and y-axis directions are dx and dy, respectively,
dx = x−XO
dy = y−YO
And the distance D is obtained by the following equation.

The distance D is compared with the distance threshold Th (step 112). If D ≧ Th, the distance flag is set to “ON” (step 113). Further, the region having the highest matching degree P with the template To in the input image is registered as a new template image An (step 114). On the other hand, if D <Th, the distance flag is set to “OFF” (step 115).

  In the subsequent flowcharts, it is determined whether to send an instruction to the OS 26.

  The degree of coincidence Q with the template Tc is calculated in the vicinity of the template position coordinate (x, y) (step 116). The specific calculation of the coincidence Q uses the above-described residual and cross-correlation coefficient.

  The degree of coincidence Q is compared with a threshold value T (step 117). Note that the degree of coincidence Q may be compared with the degree of coincidence P with the template To.

  Here, if Q ≧ T, the facial expression flag is determined (step 118).

  If the facial expression flag is “ON”, the process proceeds to step 121. On the other hand, if the facial expression flag is “OFF”, the facial expression flag is set to “ON” (step 119), and at the same time, the time t1 at this time is recorded (step 120).

  Whether or not to update the template Tc is determined by the distance flag (step 121). If the distance flag is “ON”, the region that best matches the template Tc is cut out and registered as a new template image Bn (step 122).

  On the other hand, if Q <T, the facial expression flag is determined as in step 118 (step 123).

  If the facial expression flag is “OFF”, the process proceeds to step 129.

  If the facial expression flag is already “ON”, the facial expression flag is set to “OFF” (step 124). At the same time, the time t2 at this time is recorded (step 125).

  It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 126). If the difference between the time t2 and the time t1 is not within a certain value, the process proceeds to step 129. If the difference between the time t2 and the time t1 is within a certain value, the timer is activated and turned “ON” (step 127). The facial expression counter is incremented (step 128).

  It is determined whether or not the timer is activated (step 129). If the timer has not started, the routine proceeds to step 133. If the timer is running, the timer value is determined (step 130).

  If the timer value exceeds a certain value K, a predetermined command is issued to the OS 26 (step 131), and the facial expression counter is reset (step 132).

  If the timer value does not exceed the constant value K, the process proceeds to step 133. Detecting when the change in facial expression is detected, that is, when the facial expression flag is “ON” continuously detected for a certain period of time, the operator intentionally changes the facial expression, Increment facial expression counter. The expression counter value is increased by repeatedly executing the expression change (for example, closing the eyes a plurality of times). Thereafter, a command corresponding to the value of the facial expression counter is sent when the facial expression change is not detected, that is, when the facial expression flag is continuously detected to be OFF.

  The above is a flow for determining whether to send an instruction to the OS 26.

  Next, pointer movement amounts Px and Py are calculated (step 133). Let the vertical and horizontal screen sizes of the display device 13 be H and W, respectively. Also, the upper and lower limits of template movement in the input image when the operator moves his / her face up / down / left / right are (Xmin, Ymin) and (Xmax, Ymax), respectively. The template movement amounts in the y-axis direction are dx and dy, respectively.

  The relationship between the template movement range and the vertical and horizontal screen sizes of the display device 13 is expressed as follows using constants Cx and Cy.

W = Cx (Xmax – Xmin)
H = Cy (Ymax-Ymin)
The constants Cx and Cy represent the ratio of the moving range of the template and the horizontal and vertical screen sizes of the display device 13, and the coordinates of the pointer position on the display device 13 are represented by Cx and Cy and the template moving amount (dx and dy). And defined as follows.

Px = Cx × dx
Py ＝ Cy × dy
The template position coordinates in each frame are obtained by the above formula. This means that the amount of movement of the template is enlarged or reduced and projected onto the display device 13, so that when the operator faces the front and stops the face, the pointer stops at the center of the screen, while the face When the upper limit Xmax, Ymax and the lower limits Xmin, Ymin of the movement of the template To in step 102 are reached, the pointer moves to the edge of the screen in the direction in which the face is moved.

  The pointer is moved to the coordinates Px, Py on the display device 13 (step 134).

  In addition to measuring the passage of a certain time with a timer, the number of frames of the moving image obtained from the camera 2 is counted, and when the count exceeds a certain value, it can be considered that there has been an intentional facial expression change.

<< Second Embodiment >>
In the second embodiment, a specific facial organ of the operator is tracked, and the mouse pointer is moved in the direction in which the operator faces the face. At this time, the moving speed of the mouse pointer is controlled in accordance with the amount of movement of the face. In addition, an instruction command is issued to the OS 26 in response to detecting a change in the facial expression of the operator when facing the front direction.

  FIG. 4 is a hardware configuration diagram of the information processing apparatus in which the mouse replacement method of the second embodiment is implemented. The first embodiment shown in FIG. 1 is different from the pointer coordinate calculation unit 23A and the command transmission determination unit 24A. Is different.

  Processing in the alternative method of the mouse according to the second embodiment will be described with reference to the flowchart of FIG.

  In the following processing, the camera 2 is fixed, and it is assumed that the face portion of the operator is photographed.

  First, image information of the facial organs such as the eyes, mouth, and nose is recorded as image A while the PC operator is stationary with respect to camera 2, and then the facial organs of the same operator are photographed with different facial expressions. The obtained image information is recorded as an image B, and the image B is registered in the template Tc (step 201).

  The position coordinates in the input image of the template To are recorded as reference points (XO, YO) (step 202).

  Set the initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves the face to “OFF”, and at the same time, the facial expression flag indicating whether or not there is a region that matches the template Tc Is set to “OFF” (step 203).

  The subsequent steps 204 to 236 are continuously executed at a constant period while the frame is updated.

  It is determined whether or not the frame has been updated (step 204).

  If the frame is not updated, the process ends (step 205).

  On the other hand, if the frame is updated, the distance flag is determined (step 206). If the distance flag is “OFF”, the facial organ is considered to be within a certain distance from the reference point, and the image A is registered in the template To (step 207). If the distance flag is “ON”, it is considered that the facial organ is more than a certain distance from the reference point, and the image An is registered in the template To (step 208). Here, the image An is image information of a region that best matches the template in the past frame, and the image information in the t-1st frame processing is used as a template in the t-th frame processing.

  When a template is created by photographing the front side, even if only a small range of the face is moved, the distortion of the face area in the captured image is small, so that a region with a high degree of coincidence is easily extracted. On the other hand, when the face is moved greatly, the distortion of the image in the organ part increases, so that it is difficult to detect with a single template. For this reason, when moving more than a certain range, image information of an area that best matches the template To in the past frame is used as a new template To.

  The degree of coincidence P is calculated (step 209). The specific calculation uses the above-described residual and cross-correlation coefficient. The degree of coincidence P is compared and the coordinates (x, y) in the input image of the area that best matches the template To are recorded as the template position coordinates (step 210).

  The distance D between the template position coordinates (x, y) and the operation reference point (X0, Y0) and the direction θ from the operation reference point of the template position coordinates are calculated (step 211).

If the amount of template movement in the x-axis and y-axis directions is dx and dy, respectively
dx = x-X0
dy = y-Y0
And the distance D is obtained by the following equation.

The direction θ can be obtained by the following equation: θ = tan-1 (dy / dx)
The distance D is compared with a predefined threshold value T1 (step 212). If 0 ≦ D <T1, the moving amount V of the mouse pointer is set to V1 (step 213). If 0 ≦ D <T1, the process proceeds to step 214, and the thresholds T1 and T2 (T1 <T2) defined in advance are compared with the distance D (step 214). If T1 ≦ D <T2, the moving amount V of the mouse pointer is set to V2 (step 215). If T1 ≦ D <T2 is not satisfied, the moving amount V of the mouse pointer is set to V3 (step 216).

  The distance D is compared with the distance threshold Th (step 217). If D ≦ Th, the distance flag is set to “ON” (step 218). Further, the region having the highest matching degree P with the template To in the input image is registered as a new template image An (step 219). On the other hand, if D <Th, the distance flag is set to “OFF” (step 220).

  In the subsequent flowcharts, it is determined whether to send an instruction to the OS 26.

  The degree of coincidence Q with the template Tc is calculated in the vicinity of the template position coordinate (x, y) (step 221). The specific calculation of the coincidence Q uses the above-described residual and cross-correlation coefficient.

  The degree of coincidence Q is compared with a threshold value T (step 222). Note that the degree of coincidence Q may be compared with the degree of coincidence P with the template To.

  Here, if Q ≧ T, the facial expression flag is determined (step 223). If the facial expression flag is “ON”, the process proceeds to step 236. On the other hand, when the facial expression flag is “OFF”, the facial expression flag is set to “ON” (step 224), and at the same time, the time t1 at this time is recorded (step 225).

  On the other hand, if Q <T, the facial expression flag is determined in the same manner as in step 223 (step 226). If the facial expression flag is “OFF”, the process proceeds to step 232. If the facial expression flag is already “ON”, the facial expression flag is set to “OFF” (step 227). At the same time, the time t2 at this time is recorded (step 228). It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 229). If the difference between the time t2 and the time t1 is within a certain value, the timer is activated and turned “ON” (step 230). Further, the facial expression counter is incremented (step 231). If the difference between the time t2 and the time t1 is not within a certain value, the process proceeds to step 232, and it is determined whether or not the timer is activated (step 232). If the timer has not been started, the process proceeds to step 236. If the timer is activated, the timer value is determined (step 233). If the timer value exceeds a certain value K, a predetermined command is issued to the OS 26 (step 234), and the facial expression counter is reset (step 235). If the value of the timer does not exceed the constant value K, the process proceeds to step 236.

  Detecting when the change in facial expression is detected, that is, when the facial expression flag is “ON” continuously detected for a certain period of time, the operator intentionally changes the facial expression, Increment facial expression counter.

  When various commands are sent to the OS 26 using the number of times the facial expression has changed, it is necessary to distinguish between continuous input and individual input. Therefore, a facial expression counter is used.

  The expression counter value is increased by repeatedly executing the expression change (for example, closing the eyes a plurality of times). Thereafter, a command corresponding to the value of the facial expression counter is sent when the facial expression change is not detected, that is, when the facial expression flag is continuously detected as “OFF”.

  The above is a flow for determining whether to send an instruction to the OS 26.

  The mouse pointer is moved in the direction θ obtained in step 211 by the movement amount V obtained in steps 212 to 216 (step 236).

  If V1 = stop, V2 = small movement amount, and V3 = large movement amount in steps 212 to 216, when the threshold value is larger than a certain value, it is considered that the face has been moved intentionally and the mouse pointer is moved to the direction of the face. The mouse pointer can be prevented from moving when the threshold value is smaller than a certain value, that is, when the movement amount of the face is very small. As a result, it is possible to prevent the mouse pointer from moving unintentionally due to unintended face movements.

  In this embodiment, the amount of movement of the mouse pointer is determined according to the magnitude relationship between the distance D and the two threshold values T1 and T2 (T1 <T2). Needless to say, by finely setting the amount of movement of the mouse pointer according to the size relationship, the speed control of the mouse pointer can be performed more finely.

  Furthermore, in the present embodiment, the mouse pointer is moved in an arbitrary direction depending on the face direction, but fine adjustment of the face direction may be difficult depending on the movement ability of the operator. In such a case, 8 directions or 16 directions including up / down / left / right and diagonal directions are determined in advance, and the operator moves only the rough movement by moving the mouse pointer in the direction closest to the value of θ. The mouse pointer can be moved to a desired position.

<< Third Embodiment >>
In the third embodiment, a specific facial organ of the operator is tracked, and the mouse pointer is moved in the direction in which the operator faces the face. At this time, the moving speed of the mouse pointer is controlled in accordance with the amount of movement of the face. In addition, an instruction command is issued to the OS in response to detecting a change in the expression of the operator when facing the front direction.

  FIG. 6 is a hardware configuration diagram of the information processing apparatus in which the mouse replacement method of the third embodiment is implemented. In addition to the first embodiment of FIG. 25, and the command transmission determination unit 24B is different.

  The input auxiliary area display unit 25 will be described with reference to FIG.

  FIG. 7A is a list of instruction commands presented by the input auxiliary area display unit 25, and periodically presents a plurality of instruction commands related to mouse buttons such as single click and double click to the operator. FIG. 7B shows an example of auxiliary display, in which operation names such as single click and double click are sequentially highlighted at regular time intervals. In FIG. 7B, a hatched rectangle represents an instruction command that is currently valid. The individual instruction commands are repeatedly presented at a predetermined cycle, and the operator performs a predetermined operation once in accordance with the presentation of the desired instruction command, so that the instruction command presented at that time is changed to the OS 26. Notify

  The operation in the alternative method of the mouse according to the third embodiment will be described with reference to the flowchart of FIG.

  In the following processing, the camera 2 is fixed, and it is assumed that the face portion of the operator is photographed.

  First, record the image information of the facial organs such as eyes, mouth, nose, etc. while the PC operator is stationary with respect to the camera as image A, and then photograph the facial organs of the same operator with different facial expressions, The obtained image information is recorded as an image B, and the image B is registered in the template Tc (step 301).

  The position coordinates in the input image of the template To are recorded as reference points (X0, Y0) (step 302).

  Set the initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves the face to “OFF”, and at the same time, the facial expression flag indicating whether or not there is a region that matches the template Tc Is set to “OFF” (step 303).

The subsequent steps 304 to 332 are continuously executed at a constant period while the frame is updated.

  It is determined whether or not the frame has been updated (step 304).

  If the frame is not updated, the process ends (step 305).

  On the other hand, when the frame is updated, the distance flag is determined (step 306). If the distance flag is “OFF”, the facial organ is considered to be within a certain distance from the reference point, and the image A is registered in the template To (step 307). If the distance flag is “ON”, it is considered that the facial organ is more than a certain distance from the reference point, and the image An is registered in the template To (step 308). Here, the image An is image information of an area that best matches the template To in the past frame, and the image information in the t−1th frame processing is used as the template To in the t-th frame processing.

  The degree of coincidence P is calculated (step 309). The specific calculation uses the above-described residual and cross-correlation coefficient.

  The degree of coincidence P is compared and the coordinates (x, y) in the input image of the area that best matches the template To are recorded as the template position coordinates (step 310).

  A distance D between the template position coordinates (x, y) and the operation reference point (X0, Y0) and a direction θ from the operation reference point of the template position coordinates are calculated (step 311).

If the amount of template movement in the x-axis and y-axis directions is dx and dy, respectively
dx = x-X0
dy = y-Y0
And the distance D is obtained by the following equation.

  The direction θ is obtained by the following formula

θ = tan-1 (dy / dx)
The distance D is compared with a predefined threshold value T1 (step 312). If 0 ≦ D <T1, the moving amount V of the mouse pointer is set to V1 (step 313). If 0 ≦ D <T1, the process proceeds to step 314, where the defined threshold value T1 and distance threshold value T2 (T1 <T2) are compared with the distance D (step 314). If T1 ≦ D <T2, the movement amount V of the mouse pointer is set to V2 (step 315). If T1 ≦ D <T2 is not satisfied, the moving amount V of the mouse pointer is set to V3 (step 316).

  The distance D is compared with the distance threshold Th (step 317). If D ≦ Th, the distance flag is set to “ON” (step 318). Further, the region having the highest matching degree P with the template To in the input image is registered as a new template image An (step 319). On the other hand, if D <Th, the distance flag is set to “OFF” (step 320).

  In the subsequent flowcharts, it is determined whether to send an instruction to the OS 26.

  The degree of coincidence Q with the template Tc is calculated in the vicinity of the template position coordinate (x, y) (step 321). The specific calculation of the coincidence Q uses the above-described residual and cross-correlation coefficient.

  The degree of coincidence Q is compared with a threshold value T (step 322). Note that the degree of coincidence Q may be compared with the degree of coincidence P with the template To.

  If Q ≧ T, the facial expression flag is determined (step 323). When the facial expression flag is “ON”, the process proceeds to step 332. On the other hand, if the facial expression flag is “OFF”, the facial expression flag is set to “ON” (step 324), and at the same time the time t1 is recorded (step 325). Further, the instruction command presented at this time is held (step 326).

On the other hand, if Q <T, the facial expression flag is determined in the same manner as in step 323 (step 327). If the facial expression flag is “OFF”, the process proceeds to step 332. If the facial expression flag is already “ON”, the facial expression flag is set to “OFF” (step 328). At the same time, the time t2 at this time is recorded (step 329). It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 330). When the difference between the time t2 and the time t1 is not within a certain value, the process proceeds to step 332. If the difference between the time t2 and the time t1 is equal to or greater than the coincidence value, the instruction command held in step 326 is notified to the OS 26 (step 331).
Detecting the time when the facial expression change is detected, that is, when the facial expression flag is “ON” being detected continuously for a certain period of time, the operator intentionally changes the facial expression, The instruction command presented at that time is notified to the OS.

  The above is the flow for determining whether to send an instruction to the OS 26.

  The mouse pointer is moved in the direction θ obtained in step 311 by the movement amount V obtained in steps 312 to 316 (step 332).

  In steps 312 to 316, if V1 = stop, V2 = small movement amount, and V3 = large movement amount, when the threshold value is larger than a certain value, it is considered that the face has been moved intentionally, and the mouse pointer is changed to the face orientation. It can be moved in the direction by a distance corresponding to the amount of face orientation, and the mouse pointer can be prevented from moving when the threshold value is smaller than a certain value, that is, when the movement amount of the facial organ is very small. . As a result, it is possible to prevent the mouse pointer from moving unintentionally due to unintended face movements.

  In this embodiment, the amount of movement of the mouse pointer is determined according to the magnitude relationship between the distance D and the two threshold values T1 and T2 (T1 <T2). Needless to say, by finely setting the amount of movement of the mouse pointer according to the size relationship, the speed control of the mouse pointer can be performed more finely.

  Furthermore, in the present embodiment, the mouse pointer is moved in an arbitrary direction depending on the face direction, but fine adjustment of the face direction may be difficult depending on the movement ability of the operator. In such a case, 8 directions or 16 directions including up / down / left / right and diagonal directions are determined in advance, and the operator moves only the rough movement by moving the mouse pointer in the direction closest to the value of θ. The mouse pointer can be moved to a desired position.

The mouse substitute program 20 may be recorded on a portable recording medium such as a floppy (registered trademark) disk, a CD-ROM, a magneto-optical disk, or a DVD.
[The invention's effect]

  As described above, the present invention has the following effects.

  Using the image information of the facial organs such as the eyes, mouth, and nose of the PC operator as the first template image, and tracking this, the operator moves the face up, down, left, and right to move the PC mouse pointer. It can be moved up, down, left and right, and by moving the head up, down, left, and right, the pointer can be moved / rested on the display and can function as a pointing device that replaces the conventional mouse pointer.

  By registering image information of facial organs such as eyes, mouth, nose, etc. when the expression is intentionally changed, such as closing eyes or opening the mouth, as a second template image, the operator can keep constant When an intentional operation such as intentionally closing eyes or opening a mouth is performed over time, this can be detected and an instruction command corresponding to a mouse click can be issued to the OS.

  By performing a pre-registered operation such as closing eyes or opening a mouth for a predetermined time, one of mouse commands presented periodically can be selected.

  When the operator moves his / her face up / down / left / right, the mouse pointer can be moved up / down / left / right with the center of the PC screen as the origin.

  The moving direction of the mouse pointer can be controlled by the direction in which the operator moves the face, and the moving speed of the mouse pointer can be controlled by the amount of movement of the face.

  When the operator moves the face and moves the face by a minute distance from the reference point, the same image information is used as a template to speed up the process. The region of the current frame that most closely matches the template is sequentially registered as a time template, and the image information used as the template is changed, thereby improving the matching accuracy.

It is a block diagram of the information processing apparatus with which the mouse substitution method of the first embodiment of the present invention is implemented. It is a schematic diagram of template registration. It is a flowchart of the process in 1st Embodiment. It is a block diagram of the information processing apparatus with which the mouse | mouth substitute method of the 2nd Embodiment of this invention is implemented. It is a flowchart of the process in 2nd Embodiment. It is a block diagram of the information processing apparatus with which the mouse | mouth substitution method of the 3rd Embodiment of this invention is implemented. It is a schematic diagram of the instruction selection by auxiliary display. It is a flowchart of the process in 3rd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Camera 11 Main controller 12 Storage apparatus 13 Display apparatus 14 Moving image acquisition apparatus 15 Bus 20 Mouse substitute program 21 Template image registration part 22 Template matching part 23, 23A, 23B Pointer coordinate calculation part 24, 24A, 24B Command sending judgment part 25 Input auxiliary area display part 26 OS
27 Application programs 101-134, 201-236, 301-332 steps

Claims (6)

An image obtained by cutting out a facial organ from one frame of an input image obtained by photographing a face part of a computer operator is registered as a first template,
At the same time, the position of the image in the input image is registered as an operation reference point, which is a reference point for calculating the amount and direction of movement of the pointer.
Detecting a template position coordinate which is a coordinate of an area where the first template is best matched in an input image input in frame units via an imaging device;
Calculate the position coordinate of the mouse pointer in the next frame from the template position coordinates and the coordinates of the operation reference point,
Move the mouse pointer to the coordinates,
Image information on a different facial expression of the same facial organ as the first template is cut out from the input image obtained by photographing the operator and registered as a second template,
Detect the second template in the input image,
When the second template is continuously detected over a predetermined time or frame number, the value of the expression counter is counted with this as one unit,
A mouse replacement method of issuing an instruction command registered in advance according to a value of the counted facial expression counter to the OS. While the pre-registered instruction command is sequentially presented to the operator according to a certain rule, when the desired instruction command is presented, the operator changes the facial expression over a certain time, and the second template is displayed for a certain time or a certain number. The mouse replacement method according to claim 1 , wherein the presented instruction command is issued to the OS in response to continuous observation over a plurality of frames. A template position coordinate relating to the first template and an operation reference point are compared with an input image from the imaging device, and the position of the template position coordinate with the operation reference point as the origin is the coordinate with the center of the computer screen as the origin The mouse substitute method according to claim 1 , wherein the mouse pointer is displayed at the coordinates. A template position coordinate related to the first template and an operation reference point are compared with an input image from the imaging device, and the direction and distance from the operation reference point of the template position coordinate are used to move the mouse pointer and per frame. The mouse replacement method according to claim 1 , wherein the movement amount is determined. A mouse replacement program for executing the mouse replacement method according to any one of claims 1 to 4 on a computer. The recording medium which recorded the mouse | mouth substitute program which performs the mouse | mouth substitute method of any one of Claim 1 to 4 in the computer.