JP2002323956A - Mouse alternating method, mouse alternating program and recording medium recording the same program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mouse alternating method, with which a program can be operated with a GUI as a medium without requiring an object to be worn by a person having difficulties in both hands. SOLUTION: In the state of directing an operator to the front to a camera, the image information of face organ parts such as eyes and a mouth is segmented and registered in a template To as an image A and at the same time, the position of the image A in an input image is registered as an operation reference point. The coordinates of an area, where the template To is best matched, in the input image are detected in real time as template position coordinates. The position coordinates of a mouse pointer in the next frame are calculated from the template position coordinates and the coordinates of the operation reference point and the mouse pointer is moved.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a mouse replacement method,
The present invention relates to a mouse replacement program and a recording medium on which the program is recorded.

[0002]

2. Description of the Related Art Programs that require cooperation with a user in various types of information processing using a personal computer (PC) often process user input via a GUI (graphical user interface). For example, many features, such as starting and ending a program, are realized by GUI operations that are intuitive and easy to understand, such as pointing, selecting, and moving objects such as icons and menus.

Many of these operations require an input device using a hand movement function, such as a mouse for selecting an object and a keyboard for inputting characters, and thus include commercially available application programs. In order to use these programs as they are for users with impaired functions of both hands, dedicated means is required. In particular, various devices have been developed as input means of personal computers for the physically handicapped, and methods of using the remaining functions of the body to replace the mouse operation, for example, obstacles that can voluntarily move the head A method of instructing a button or an icon on a screen using the movement has been developed for a person.

In an alternative method of the mouse using the movement of the head, various sensors are mounted on the head and a head space pointer or a head master plus equipped with a mechanism for detecting the up, down, left and right movements thereof is used. There are products and methods such as a method in which a reflective marker is attached to the head like a head mouse and a moving image captured by a CCD camera or the like is extracted into a marker area by image processing.

[0005] In addition, there is a method of pointing by attaching a sensor or a marker to a movable part of the body as disclosed in JP-A-10-320108 and JP-A-7-141098.

[0006] In principle, the mounting position of the sensor or marker in these products or the invention may be any movable position of the body, and the operation is possible even when the sensor or marker is mounted on a hand or foot, for example.

[0007] In either method, the position of the extracted sensor,
Alternatively, the coordinate position of the marker is displayed as a pointer on the monitor of the personal computer after the coordinate conversion is appropriately performed. Thereby, an object such as an icon or a button on the screen can be indicated by the vertical or horizontal movement of the marker or the sensor.

[0008]

However, the operation of the PC according to the above-described conventional method requires mounting various sensors, headsets, markers, and the like. Although these are improved and lightweight and compact ones are being developed,
Since it is necessary only when the PC is operated, it is necessary to perform mounting and fixing each time the user uses the personal computer.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a GUI for a handicapped person who does not have a handicap.
It is an object of the present invention to provide a mouse alternative method that enables the operation of a program through the mouse.

[0010]

In order to achieve the above-mentioned object, the present invention utilizes a feature of a face organ such as an eye or a mouth to specify a specific face from an image obtained by photographing the face with a video camera or the like. Move the mouse pointer based on the coordinates obtained by tracking the organ, further select objects such as icons by detecting changes in facial expressions such as intentionally closing eyes, opening mouth, etc. By issuing a command instruction or the like, a mouse alternative function is realized.

That is, according to the mouse replacement method of the present invention, an image obtained by cutting out a face organ from one frame of an input image obtained by capturing a face portion of a computer operator is registered as a first template, and at the same time, the image in the input image is registered. Is registered as an operation reference point, which is a reference point for calculating the moving amount and moving direction of the pointer, and the first template best matches in the input image input in frame units via the imaging device. The template position coordinates, which are the coordinates of the area, are detected in real time, the position coordinates of the mouse pointer in the next frame are calculated from the template position coordinates and the coordinates of the operation reference point, and the mouse pointer is moved to the coordinates.

According to this configuration, the image information of the operator's face organ is used as the first template image, and by tracking this, the mouse of the PC is linked with the operator moving the face up, down, left and right. The pointer can be moved.

In an embodiment of the present invention, image information of the same facial organ as the first template in different facial expressions is cut out from an input image of the operator of the PC and registered as a second template. The second template is searched in the input image at the same time as the search for the first template, and the operator changes the expression over a certain period of time, so that the second template is continuously detected for a predetermined time or number of frames. In this case, this is counted as one unit, and a pre-registered instruction command is issued to the OS according to the value counted within a predetermined time.

According to this configuration, the operator intentionally performs one or more predetermined operations such as closing his eyes or opening his / her mouth for a certain period of time. It is possible to execute a command command related to a mouse button such as a click.

In another embodiment of the present invention, instruction instructions registered in advance are sequentially presented to an operator according to a predetermined rule, and when a desired instruction instruction is presented, the operator changes his / her expression over a certain period of time. Accordingly, the instruction command presented when the second template is continuously observed over a certain period of time or over a certain number of frames is transmitted to the OS.
Issue for

According to this configuration, a plurality of instruction commands related to mouse buttons such as a single click and a double click are presented to the operator periodically, while the desired instruction command is presented. The operator performs a predetermined action
By performing the operation twice, a desired one is selected from a plurality of instruction commands, and an instruction command can be issued to an object such as an icon or a button indicated by the mouse pointer at that time.

According to another embodiment of the present invention, the template position coordinates of the input image from the imaging device are compared with the operation reference point for the first template, and the template position coordinates having the operation reference point as the origin are compared. Is converted to coordinates having the origin at the center of the PC screen, and a mouse pointer is displayed at the coordinates.

According to this configuration, the operator moves the mouse pointer in the direction in which the operator moves the face by an amount proportional to the amount of movement of the face with reference to the center of the PC screen. You can move the mouse pointer to any coordinates on the screen simply by moving it.

According to another embodiment of the present invention, the template position coordinates of the first template and the operation reference point are compared with respect to the input image from the imaging device, and the direction of the template position coordinates from the operation reference point is determined. The moving direction of the mouse pointer and the moving amount per frame are determined from the distance and the distance.

According to this configuration, the moving direction of the mouse pointer is set according to the direction in which the operator moves the face, and the moving speed of the mouse pointer is set according to the magnitude of the moving amount of the face. You can move the mouse pointer to any coordinates on the screen simply by moving it.

According to another embodiment of the present invention, a template position coordinate for the first template and an operation reference point are compared with an input image from an imaging device, and a distance between the template position coordinate and the operation reference point is determined. Exceeds a certain value, an area that best matches the first template in the current frame is cut out instead of the image information recorded in advance, and registered as a template in the next frame.
In the subsequent frames, the search is performed while updating the template as needed, and when the distance from the operation reference point returns within a certain value, the pre-recorded image information is registered again as the first template, and Perform search processing.

According to this configuration, when the movement range of the operator's face is within a predetermined area, the processing speed is increased by using the same image information as a template, while the movement of the operator's face is When the amount exceeds the predetermined range, the region that best matches the template in the current frame is registered as the next template, and the image information used as the template is sequentially changed to increase the matching accuracy. be able to.

[0023]

Next, embodiments of the present invention will be described with reference to the drawings.

First Embodiment In a first embodiment, a specific facial organ of an operator is tracked, and a mouse pointer is moved from the center of the screen by an amount proportional to the amount of movement of the face. An instruction command is issued to the OS upon detection of a facial expression change in the same facial organ of the user.

FIG. 1 is a block diagram of an apparatus for implementing the mouse replacement method according to the first embodiment of the present invention.

The information processing device 1 includes a main control device 11, a storage device 12, a display device 13, a moving image acquiring device 14, and a bus 15, and the camera 2 is connected to the moving image acquiring device 14.

Main controller 1 is composed of an MPU, a RAM, etc.
The information processing apparatus 1 is generally managed. For this reason, the main control device 1 includes a template image registration unit 21 and a template matching unit
2, a pointer coordinate calculation unit 23, an instruction transmission determination unit 24,
OS 26 is running. The main controller 11 also operates an application program 27 such as word processing software. The OS 26 has a function of executing various commands related to the mouse, such as moving a mouse pointer and single-clicking and double-clicking a mouse button.

The storage device 12 comprises a hard disk device or the like. The storage device 12 stores a mouse replacement program 20, an OS 26, an application program 27, and the like.

The display device 13 is composed of a liquid crystal monitor or the like, and displays the processing result in the main control device 11 including the movement of the pointer.

The moving image acquisition device 14 has a function of sequentially taking a moving image obtained from the camera 2 into a RAM provided in the main control device 11 in frame units. Note that a commercially available video capture board can be used as the moving image acquisition device 14, and a commercially available CCD camera or the like can be used as the camera 2.

Next, the template of the facial organ, the operation reference point, and the template position coordinates will be described with reference to FIG.

The head of the operator is photographed by the camera 2, and an image including the respective organs of the face is taken into the main controller 11 by the moving image acquisition device 14 as an input image as shown in FIG. It is assumed that the camera 2 captures the operator's head in a fixed state.

FIG. 2A shows an input image in a state where the operator faces the camera 2 in front. In this state, image information of a face organ portion such as an eye and a mouth is cut out, and a template To Register with. The following figures use the eye part as a template. At the same time, the position of the image A in the input image at the time of registration is registered as an operation reference point. The operation reference point is a reference point for calculating the moving amount and moving direction of the pointer, and its coordinates are equal to initial values of template position coordinates described later.

In FIG. 2A, the lower right coordinate of the image A is used as a specific operation reference point. However, if the coordinates are based on a specific position in the cut-out image, this limit applies. Instead, for example, the coordinates where the center of the image is located may be used.

Here, as shown in FIG. 2B, when the operator moves his / her face up / down / left / right, the facial organs such as the eyes, mouth and nose of the operator also move. Therefore, the template To
Are superimposed on each other and the template matching is performed, the region where the template To best matches in the input image changes with the movement of the face organ. In the following description, the coordinates of the region where the template image A best matches on the input image will be referred to as template position coordinates. Note that the specific position of the template position coordinates uses coordinates based on the same position as the reference coordinates used in the definition of the operation reference point in the template image. The template position coordinates are
This is converted into a specific mouse pointer movement amount by a pointer coordinate calculation unit 23 described later.

Further, as shown in FIG. 2C, image information of the same facial organ portion in a different expression (close eyes, open mouth, etc.) from that at the time of photographing of image A is cut out in the same size as image A, Register as image B in template Tc. FIG.
(C) is a state in which the eyes are closed.

Next, the function of the mouse alternative program 20 will be described.

As described above, the mouse replacement program 20 is composed of the template image registration unit 21, the template matching unit 22, the pointer coordinate calculation unit 23, and the command transmission determination unit 24.

First, the template image registration unit 21 registers, as a template To, an image A obtained by cutting out a face organ such as an eye and a mouth from one frame of a moving image obtained by photographing the face of the operator. At the same time, the position of the image A in the input image is registered as an operation reference point. Furthermore, an image obtained by cutting out image information of different facial expressions in the same facial organ as when the image A was captured
Register B as template Tc.

The template matching unit 22 performs template matching in the input image using the given template image. As a result of template matching,
Calculate the template position coordinates in the input image.
More specifically, in one frame of a moving image that is sequentially updated, the degree of coincidence between the template image and an area where the template overlaps in the input image is sequentially calculated while moving the template image by a small distance, and the template is best. Output the coordinates of the matching area.

As a measure of the degree of coincidence, there are a method of obtaining a residual of density between corresponding pixels of the template image and the input image, a method of obtaining a cross-correlation coefficient, and the like. When the residual is used, it means the best match at the position where the value is minimum, and when the cross-correlation coefficient is used, it means that the best match is at the position where the value is maximum.

Note that the size of the template is M × N,
Assuming that the template To and the regions on the input image where the template To overlap are T (m, n) and I (m, n), respectively, the residual R and the cross-correlation coefficient C are given by the following equations.

[0043]

(Equation 1) The significance of the value differs depending on the scale of the matching degree used, but in the following description, it is assumed that the larger the value of the matching degree, the better the matching. When calculating the coincidence, the image obtained from the camera 2 can be used as it is. However, the image obtained once is subjected to pre-processing such as shading and extraction, and then the coincidence is calculated. You may.

The template matching unit 22
After the scanning of the frame is completed, the coordinates in the input image at which the degree of coincidence with the template is maximum are recorded as new template position coordinates.

The pointer coordinate calculator 23 compares the template position coordinates obtained by the template matching unit 22 with the coordinates of the operation reference point, and determines the position of the mouse pointer in the next frame based on the amount and direction of movement of the template. The coordinates are calculated and notified to the OS 26.

If the distance between the operation reference point and the template position coordinates exceeds a certain value, an area which best matches the template To in the current input image is cut out.
Instead of the image information A recorded in advance, it is registered as a template To in the next frame. In the subsequent frames, the search is performed while updating the template To as needed. On the other hand, when the distance from the operation reference point returns within a certain value, the pre-recorded image information A is registered again as the template To, and the search processing in the subsequent frames is performed.

The command sending determination unit 24 searches for a template Tc registered in advance near the template position coordinates. The search method uses the above-described template matching. When the template Tc is continuously detected for a predetermined time or the number of frames by the operator changing the expression over a certain period of time, the command transmission determination unit 24 counts this as one unit, and within a certain period of time. An instruction command registered in advance is issued to the OS 26 according to the counted value. Here, the instruction command is a command corresponding to depression and release of a mouse button, etc., and is notified to the OS 26 to select an object such as an icon currently pointed to by the mouse pointer and to issue a command for the object. It is for giving instructions.

When the distance between the template position coordinates corresponding to the template To and the operation reference point exceeds a certain value, an area that best matches the template Tc near the template position coordinates is cut out and recorded in advance. Is registered as a template Tc in the next frame in place of the image information B
Search while updating Tc as needed. On the other hand, when the distance between the template position coordinates and the operation reference point returns to within a certain value, the pre-recorded image information B is copied again to the template Tc.
And performs a search process in the subsequent frames.

The above operation is repeatedly executed for each frame.

Therefore, when the operator moves the face, the mouse pointer on the display device 3 moves according to the direction,
By changing the expression to a previously registered expression for a certain period of time, a predetermined command can be issued to an object such as an icon pointed by the mouse pointer. As a result, it becomes possible to substitute the mouse in the operation of the PC according to the direction and expression of the face.

Next, processing in the mouse replacement method of the first embodiment will be described with reference to the flowchart of FIG.

In the following processing, it is assumed that the camera 2 is fixed and an operator's face is photographed.

First, while the PC operator is stationary with respect to the camera 2, image information obtained by photographing a face organ such as an eye, a mouth, and a nose is recorded as an image A, and the image A is registered in a template To. The facial organs of the same operator are photographed with different expressions, the obtained image information is recorded as an image B, and the image B is registered in the template Tc (step 10).
1).

The position coordinates of the template To in the input image are recorded as reference points (XO, YO), and the template To in the input image when the operator moves the face up, down, left, and right.
The upper limit Xmax, Ymax and the lower limit Xmin, Ymin of the movement are recorded (step 102).

The initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves his / her face is set to “OFF”, and at the same time, the presence or absence of an area matching the template Tc is determined. The initial value of the indicated expression flag is set to "OFF" (step 103).

The subsequent steps 104 to 134 are continuously executed at a constant period while the frame is updated.

It is determined whether or not the frame has been updated (step 104). If the frame has not been updated, the process ends (step 105).

On the other hand, if the frame has been updated, the distance flag is determined (step 106). When the distance flag is “OFF”, it is considered that the face organ is within a certain distance from the reference point, and the image A is registered in the template To and the image B is registered in the template Tc (step 107). When the distance flag is “ON”, it is considered that the facial organ is apart from the reference point by a certain distance or more, and the image An is registered in the template To, and the image Bn is registered in the template Tc (step 1).
08). Here, the image An and the image Bn are past frames, which are image information of the regions that best match the template To and the template Tc, respectively, and in the case of the t-th frame processing, the image in the t−1-th frame processing The information is used as a template.

When a template is created by photographing the front of the operator, even if the face is moved by a very small range, the distortion of the face region in the input image is small, so that a region having a high degree of coincidence is easily extracted. On the other hand, when the face is greatly moved, the distortion of the image in each organ part becomes large, and thus it becomes difficult to perform the detection by template matching using a single template. Therefore, when the movement is larger than a certain range, the image information of the area that best matches the template in the past frame is used as a new template.

The degree of coincidence P is calculated (step 109).
Specific calculation uses the above-described residual and cross-correlation coefficients.

The degree of coincidence P is compared, and the coordinates (x, y) of the area having the highest degree of coincidence with the template To in the input image are recorded as template position coordinates (step 110).

The distance D between the template position coordinates (x, y) and the reference point (XO, YO) is calculated (step 11).
1).

Assuming that the template movement amounts in the x-axis direction and the y-axis direction are dx and dy, respectively, dx = x−XO dy = y−YO, and the distance D can be obtained by the following equation.

[0064]

(Equation 2) The distance D is compared with the distance threshold Th (step 11).
2). If D ≧ Th, the distance flag is set to “ON” (step 113). Further, an area having the highest matching degree P with the template To in the input image is registered as a new template image An (step 11).
4). On the other hand, if D <Th, the distance flag is set to "OFF" (step 115).

In the subsequent flowcharts, it is determined whether or not an instruction is to be sent to the OS 26.

The degree of coincidence Q with the template Tc is calculated near the template position coordinates (x, y) (step 11).
6). The specific calculation of the degree of coincidence Q uses the above-described residual and cross-correlation coefficients.

The degree of coincidence Q is compared with a threshold value T (step 117). Note that the matching degree Q is set to the matching degree P with the template To.
It may be compared with.

If Q ≧ T, the expression flag is determined (step 118).

If the expression flag is "ON", step 121
Move to On the other hand, when the expression flag is "OFF", the expression flag is set to "ON" (step 119), and at the same time, the time t1 is recorded (step 120).

Whether or not to update the template Tc is determined by the distance flag (step 121). If the distance flag is "ON", an area that best matches the template Tc is cut out and registered as a new template image Bn (step 122).

On the other hand, if Q <T, the expression flag is determined as in step 118 (step 123).

If the expression flag is "OFF", step 12 is executed.
Move to 9.

If the expression flag is already "ON", the expression flag is set to "OFF" (step 124). At the same time, the time t2 at this time is recorded (step 125).

It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 126). If the difference between time t2 and time t1 is not within a certain value, step 129
Move to If the difference between time t2 and time t1 is within a certain value, the timer is started and turned on (step 12).
7). Increment the expression counter (Step 1)
28).

It is determined whether or not the timer is activated (step 1).
29). If the timer is not running, step 133
Move to If the timer is running, the value of the timer is determined (step 130).

If the timer value exceeds the fixed value K, OS2
6 is issued (step 131), and the expression counter is reset (step 132).

If the value of the timer does not exceed the fixed value K, the flow shifts to step 133. Detecting that the operator intentionally changed the facial expression in response to the time when the facial expression change is detected, that is, the state in which the facial expression flag is “ON” is continuously detected for a certain period of time, Increment the expression counter. By repeatedly performing the change of the facial expression (for example, closing the eyes a plurality of times), the value of the facial expression counter increases. After that, when no change in facial expression is detected,
In other words, an instruction corresponding to the value of the expression counter is sent when the OFF state of the expression flag is continuously detected.

The above is the flow of determining whether to send an instruction to the OS 26.

Next, the movement amounts Px and Py of the pointer are calculated (step 133). The horizontal and vertical screen sizes of the display device 13 are H and W, respectively. The upper and lower limits of template movement in the input image when the operator moves the face up, down, left, and right are (Xmin, Ymin) and (Xmax, Ymax), respectively. The template movement amounts in the y-axis direction are dx and dy, respectively.

The relationship between the moving range of the template and the horizontal and vertical screen sizes of the display device 13 is expressed as follows using constants Cx and Cy.

W = Cx (Xmax−Xmin) H = Cy (Ymax−Ymin) The constants Cx and Cy are the moving range of the template and the display device 13.
Of the vertical and horizontal screen size of the display device 13
The coordinates of the upper pointer position are defined as follows using Cx, Cy and the template movement amount (dx, dy).

Px = Cx × dx Py = Cy × dy The template position coordinates in each frame are obtained by the above equation. This means that the amount of movement of the template is enlarged or reduced and projected on the display device 13. With this, when the operator faces the front and stops the face, the pointer stops at the center of the screen while the face stops. When the upper limit Xmax and Ymax and the lower limit Xmin and Ymin of the movement of the template To in step 102 are reached, the pointer moves to the end of the screen in the direction in which the face is moved.

The pointer is moved to the coordinates Px and Py on the display device 13 (step 134).

In addition to counting the elapse of a certain time by a timer, the number of frames of a moving image obtained from the camera 2 is counted, and when the count exceeds a certain value, it is considered that an intentional change in facial expression has occurred. Can be.

Second Embodiment A second embodiment tracks a specific facial organ of an operator,
The operator moves the mouse pointer in the direction in which his / her face is turned. At this time, the moving speed of the mouse pointer is controlled according to the magnitude of the moving amount of the face. Further, an instruction command is issued to the OS 26 upon detection of a change in the facial expression of the operator when the operator is facing the front.

FIG. 4 is a hardware configuration diagram of an information processing apparatus in which the method for substituting a mouse according to the second embodiment is carried out. The pointer coordinate calculation unit 23A differs from the first embodiment shown in FIG.
And only the command transmission determination unit 24A is different.

The processing in the mouse replacement method according to the second embodiment will be described with reference to the flowchart in FIG.

In the following processing, it is assumed that the camera 2 is fixed and the face of the operator is photographed.

First, image information obtained by photographing a facial organ such as an eye, a mouth and a nose while the PC operator is stationary with respect to the camera 2 is recorded as an image A, and subsequently, the facial organ of the same operator is displayed with different expressions. Is recorded, and the obtained image information is recorded as an image B, and the image B is registered in the template Tc (step 201).

The position coordinates of the template To in the input image are recorded as reference points (XO, YO) (step 20).
2).

The initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves his / her face is set to “OFF”, and at the same time, the presence or absence of an area matching the template Tc is determined. The initial value of the indicated expression flag is set to "OFF" (step 203).

The subsequent steps 204 to 236 are continuously executed at a constant period while the frame is updated.

It is determined whether or not the frame has been updated (step 204).

If the frame has not been updated, the process ends (step 205).

On the other hand, if the frame has been updated, the distance flag is determined (step 206). If the distance flag is “OFF”, it is considered that the face organ is within a certain distance from the reference point, and the image A is registered in the template To (step 207). If the distance flag is "ON", it is considered that the facial organ is apart from the reference point by a certain distance or more, and the image An is registered in the template To (step 208). Here, the image An is image information of a region in the past frame that best matches the template, and in the case of the t-th frame processing, the image information in the (t−1) -th frame processing is used as a template.

When a template is created by photographing the front, even if the face is moved only in a small range, the distortion of the face area in the captured image is small, so that an area having a high degree of coincidence is easily extracted. On the other hand, when the face is largely moved, the distortion of the image in the organ part becomes large, so that it is difficult to detect the single template. Therefore, when the movement is larger than a certain range, the image information of the region that best matches the template To in the past frame is used as a new template To.

The degree of coincidence P is calculated (step 20).
9). Specific calculation uses the above-described residual and cross-correlation coefficients. By comparing the degree of coincidence P, the coordinates (x, y) in the input image of the area that best matches the template To are recorded as template position coordinates (step 210).

The distance D between the template position coordinates (x, y) and the operation reference point (X0, Y0) and the direction θ of the template position coordinates from the operation reference point are calculated (step 21).
1).

Assuming that the template movement amounts in the x-axis direction and the y-axis direction are dx and dy, respectively, dx = x−X0 dy = y−Y0, and the distance D is obtained by the following equation.

[0100]

(Equation 3) The direction θ is obtained by the following equation: θ = tan ⁻¹ (dy / dx) The distance D is compared with a predetermined threshold value T1 (step 212). If 0 ≦ D <T1, the moving amount V of the mouse pointer is set to V1 (step 213). 0 ≦ D
If not, the process proceeds to step 214, where the distance D is compared with threshold values T1 and T2 (T1 <T2) defined in advance (step 214). If T1 ≦ D <T2, the moving amount V of the mouse pointer is set to V2 (step 21).
5). If T1 ≦ D <T2, the moving amount V of the mouse pointer is set to V3 (step 216).

The distance D is compared with the distance threshold Th (step 217). If D ≦ Th, the distance flag is set to “O
N ”is set (step 218). Further, an area having the highest matching degree P with the template To in the input image is registered as a new template image An (step 219). Set the distance flag to “OF
F "(step 220).

In the subsequent flowcharts, it is determined whether or not an instruction to the OS 26 has been sent.

The degree of coincidence Q with the template Tc is calculated near the template position coordinates (x, y) (step 22).
1). The specific calculation of the degree of coincidence Q uses the above-described residual and cross-correlation coefficients.

The degree of coincidence Q is compared with a threshold value T (step 222). Note that the matching degree Q is set to the matching degree P with the template To.
It may be compared with.

Here, if Q ≧ T, the expression flag is determined (step 223). When the expression flag is “ON”,
Move to step 236. On the other hand, if the expression flag is “OF
F ”, the expression flag is set to“ ON ”(step 2).
24) At the same time, the time t1 at this time is recorded (step 225).

On the other hand, if Q <T, the expression flag is determined in the same manner as in step 223 (step 226). If the expression flag is “OFF”, the process proceeds to step 232. If the expression flag is already "ON", the expression flag is set to "OFF" (step 227). At the same time t
2 is recorded (step 228). It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 22).
9). If the difference between time t2 and time t1 is within a certain value,
The timer is started and turned on (step 230). Further, the expression counter is incremented (step 23).
1). If the difference between time t2 and time t1 is not within a certain value,
The process proceeds to step 232 to determine whether or not the timer is activated (step 232). If the timer has not started, the process proceeds to step 236. If the timer is running,
The value of the timer is determined (step 233). If the value of the timer exceeds the fixed value K, a predetermined command is issued to the OS 26 (step 234), and the expression counter is reset (step 235). If the timer value does not exceed the fixed value K, the process proceeds to step 236.

The time when the change of the facial expression is detected, that is, the fact that the state where the facial expression flag is “ON” is continuously detected for a certain period of time triggers the fact that the operator intentionally changes the facial expression. Detect and increment the expression counter.

Various commands are executed using the number of times the expression has changed.
When transmitting to the OS 26, it is necessary to distinguish between continuous input and individual input. Therefore, an expression counter is used.

By repeatedly executing the change of the expression (for example, closing the eyes a plurality of times), the value of the expression counter increases. Thereafter, an instruction corresponding to the value of the expression counter is transmitted when the expression change is not detected, that is, when the state in which the expression flag is “OFF” is continuously detected.

The above is the flow of determining whether to send an instruction to the OS 26.

The mouse pointer is moved in the direction θ obtained in step 211 by the movement amount V obtained in steps 212 to 216 (step 236).

In steps 212 to 216, if V1 = stop, V2 = movement amount small, V3 = movement amount, when the threshold is larger than a certain value, it is regarded that the face has been intentionally moved and the mouse pointer is moved. Move the mouse pointer in the direction of the direction by the distance corresponding to the amount of the face direction, and do not move the mouse pointer when the threshold value is smaller than a certain value, that is, when the movement amount of the face is very small. Can be. This can prevent the mouse pointer from moving without permission due to unintended face movement.

In this embodiment, the amount of movement of the mouse pointer is determined according to the magnitude relationship between the distance D and the two thresholds T1 and T2 (T1 <T2). Needless to say, by finely setting the movement amount of the mouse pointer according to the magnitude relationship with the distance D, the speed control of the mouse pointer can be performed more finely.

Further, in this embodiment, the mouse pointer is moved in an arbitrary direction depending on the direction of the face. However, fine adjustment of the direction of the face may be difficult depending on the movement ability of the operator. In such a case, eight directions or 16 directions including up, down, left, right, and diagonal directions are determined in advance, and by moving the mouse pointer in the direction closest to the value of θ, the operator can combine only rough movements. To move the mouse pointer to a desired position.

Third Embodiment In a third embodiment, a specific facial organ of an operator is tracked, and a mouse pointer is moved in a direction in which the operator faces his / her face.
At this time, the moving speed of the mouse pointer is controlled according to the magnitude of the moving amount of the face. Further, an instruction command is issued to the OS upon detection of a change in the facial expression of the operator when facing the front direction.

FIG. 6 is a hardware configuration diagram of an information processing apparatus in which the mouse replacement method according to the third embodiment is implemented. The mouse replacement program 20 according to the first embodiment shown in FIG.
In addition, an input auxiliary area display section 25 is provided, and a command transmission determination section 24B is different.

The input auxiliary area display section 25 will be described with reference to FIG.

FIG. 7A is a list of instruction commands presented by the input auxiliary area display section 25. A plurality of instruction commands relating to mouse buttons such as a single click and a double click are periodically presented to the operator. I do. FIG. 7B shows an example of an auxiliary display in which operation names such as a single click and a double click are sequentially highlighted at predetermined time intervals. In FIG. 7B, a hatched rectangle indicates a currently valid instruction command. Each instruction command is repeatedly presented at a predetermined cycle, and the operator performs a predetermined operation once in accordance with the presentation of the desired instruction command, so that the instruction command presented at that time is transmitted to the OS 26. Notify.

The operation of the mouse alternative method according to the third embodiment will be described with reference to the flowchart of FIG.

In the following processing, it is assumed that the camera 2 is fixed and the face of the operator is photographed.

First, image information obtained by photographing a facial organ such as an eye, a mouth, a nose, etc. while the operator of the PC is stationary with respect to the camera is displayed as an image.
Recorded as A, then photographed the same operator's face organ with different facial expressions, recorded the obtained image information as image B,
Register image B in template Tc (step 30)
1).

The position coordinates of the template To in the input image are recorded as reference points (X0, Y0) (step 30).
2).

The initial value of the distance flag indicating whether or not the distance from the reference point exceeds a certain value when the operator moves his / her face is set to “OFF”, and at the same time, the presence or absence of an area matching the template Tc is determined. The initial value of the indicated expression flag is set to "OFF" (step 303).

The following steps 304 to 332 are continuously executed at a constant period while the frame is updated.

It is determined whether or not the frame has been updated (step 304).

If the frame is not updated, the process ends (step 305).

On the other hand, if the frame has been updated, the distance flag is determined (step 306). If the distance flag is "OFF", it is considered that the face organ is within a certain distance from the reference point, and the image A is registered in the template To (step 307). If the distance flag is "ON", it is considered that the facial organ is apart from the reference point by a certain distance or more, and the image An is registered in the template To (step 308). Here, the image An is the image information of the region that best matches the template To in the past frame, and at the time of the t-th frame processing, the image information in the (t−1) -th frame processing is used as the template To.

The degree of coincidence P is calculated (step 30).
9). Specific calculation uses the above-described residual and cross-correlation coefficients.

The magnitude of the matching degree P is compared and the template To
(X, y) in the input image of the area that best matches
Are recorded as template position coordinates (step 31).
0).

The distance D between the template position coordinates (x, y) and the operation reference point (X0, Y0) and the direction θ of the template position coordinates from the operation reference point are calculated (step 31).
1).

Assuming that the amounts of template movement in the x-axis direction and the y-axis direction are dx and dy, respectively, dx = x−X0 dy = y−Y0, and the distance D is obtained by the following equation.

The direction θ is obtained by the following equation.

[0133]

(Equation 4) ^{θ = tan -1 (dy / dx} ) distance is compared with a threshold T1 that previously defined as D (step 312). If 0 ≦ D <T1, the moving amount V of the mouse pointer is set to V1 (step 313). 0 ≦ D
If not, the process proceeds to step 314, where the defined threshold value T1 and distance threshold value T2 (T1 <T2) are compared with the distance D (step 314). If T1 ≦ D <T2, the moving amount V of the mouse pointer is set to V2 (step 315). If T1 ≦ D <T2, the moving amount V of the mouse pointer is set to V3 (step 316).

The distance D is compared with the distance threshold Th (step 317). If D ≦ Th, the distance flag is set to “O
N ”is set (step 318). Further, an area having the highest matching degree P with the template To in the input image is registered as a new template image An (step 319). Set the distance flag to “OF
F "(step 320).

In the subsequent flow charts, it is determined whether or not an instruction is sent to the OS 26.

The degree of coincidence Q with the template Tc is calculated near the template position coordinates (x, y) (step 32).
1). The specific calculation of the degree of coincidence Q uses the above-described residual and cross-correlation coefficients.

The coincidence Q is compared with a threshold T (step 322). Note that the matching degree Q is set to the matching degree P with the template To.
It may be compared with.

If Q ≧ T, the expression flag is determined (step 323). When the expression flag is “ON”,
Move to step 332. On the other hand, if the expression flag is “OF
In the case of "F", the expression flag is set to "ON" (step 3
24) At the same time, the time t1 at this time is recorded (step 325). Also, the instruction command presented at this time is held (step 326).

On the other hand, if Q <T, the expression flag is determined in the same manner as in step 323 (step 327). If the expression flag is “OFF”, the flow shifts to step 332. If the expression flag is already "ON", the expression flag is set to "OFF" (step 328). At the same time t
2 is recorded (step 329). It is determined whether or not the difference between time t2 and time t1 is within a certain value (step 33).
0). If the difference between time t2 and time t1 is not within a certain value,
Move to step 332. If the difference between the time t2 and the time t1 is equal to or greater than the coincidence value, the instruction command held in step 326 is notified to the OS 26 (step 331).
When the state of "N" is detected continuously for a certain period of time, it is detected that the operator has intentionally changed his / her expression, and an instruction command presented at that time is notified to the OS. .

The above is the flow of determining whether to send an instruction to the OS 26.

The mouse pointer is moved in the direction θ obtained in step 311 by the movement amount V obtained in steps 312 to 316 (step 332).

If V1 = stop, V2 = movement small, V3 = movement large in steps 312 to 316, when the threshold value is larger than a certain value, it is regarded that the face has been intentionally moved and the mouse pointer is moved. Move the mouse pointer in the direction of the direction by a distance corresponding to the amount of the face direction, and do not move the mouse pointer when the threshold value is smaller than a certain value, that is, when the movement amount of the face organ is very small. be able to. This can prevent the mouse pointer from moving without permission due to unintended face movement.

In this embodiment, the amount of movement of the mouse pointer is determined by the distance D and the two threshold values T1 and T2 (T1 <T
It is determined according to the magnitude relationship with 2), but by preparing a large number of thresholds and finely setting the amount of movement of the mouse pointer according to the magnitude relationship with the distance D, the speed control of the mouse pointer can be more finely controlled. It goes without saying that you can do it.

Further, in this embodiment, the mouse pointer is moved in an arbitrary direction depending on the direction of the face. However, fine adjustment of the direction of the face may be difficult depending on the movement ability of the operator. In such a case, eight directions or 16 directions including up, down, left, right, and diagonal directions are determined in advance, and by moving the mouse pointer in the direction closest to the value of θ, the operator can combine only rough movements. To move the mouse pointer to a desired position.

The mouse replacement program 20 may be recorded on a portable recording medium such as a floppy (registered trademark) disk, CD-ROM, magneto-optical disk, or DVD.

[0146]

As described above, the present invention has the following effects.

According to the first aspect of the present invention, image information of a facial organ such as an eye, a mouth, and a nose of a PC operator is used as a first template image, and by tracking the image information, the operator can move the face up and down. It is possible to move the mouse pointer of the PC up, down, left and right by moving it left and right, and by moving the head up, down, left and right, the pointer is moved / stopped on the display. Can perform functions.

According to a second aspect of the present invention, image information of facial organs such as eyes, mouth, and nose when an expression is intentionally changed, such as closing an eye or opening a mouth, is registered in advance as a second template image. By doing so, if the operator intentionally performs an intentional operation such as closing his eyes or opening his mouth for a certain period of time, this is detected and an instruction command corresponding to a mouse click is issued to the OS. can do.

According to the third aspect of the present invention, by performing a predetermined operation such as closing the eyes or opening the mouth for a predetermined time, one of the mouse commands presented periodically can be selected. .

According to the invention of claim 4, the operator can move the mouse pointer up, down, left, and right with the center of the PC screen as the origin by moving the face up, down, left, and right.

According to the fifth aspect of the present invention, the moving direction of the mouse pointer can be controlled by the direction in which the operator moves the face, and the moving speed of the mouse pointer can be controlled by the magnitude of the moving amount of the face.

According to a sixth aspect of the present invention, the operator moves his face,
When the face is moved by a very small distance from the reference point, the processing is accelerated by using the same image information as a template.On the other hand, when the face is moved by a certain value or more, the face is moved to the template within the current frame. The accuracy of matching can be improved by sequentially registering the most matching areas as time templates and changing the image information used as templates.

[Brief description of the drawings]

FIG. 1 is a block diagram of an information processing apparatus in which a mouse replacement method according to a first embodiment of the present invention is performed.

FIG. 2 is a schematic diagram of template registration.

FIG. 3 is a flowchart of a process according to the first embodiment.

FIG. 4 is a block diagram of an information processing apparatus in which a mouse replacement method according to a second embodiment of the present invention is implemented.

FIG. 5 is a flowchart of a process according to the second embodiment.

FIG. 6 is a block diagram of an information processing apparatus in which a mouse replacement method according to a third embodiment of the present invention is performed.

FIG. 7 is a schematic diagram of instruction selection by auxiliary display.

FIG. 8 is a flowchart of a process according to the third embodiment.

[Explanation of symbols]

Reference Signs List 1 information processing device 2 camera 11 main control device 12 storage device 13 display device 14 moving image acquisition device 15 bus 20 mouse alternative program 21 template image registration unit 22 template matching unit 23, 23A, 23B pointer coordinate calculation unit 24, 24A, 24B Command sending determination unit 25 Input auxiliary area display unit 26 OS 27 Application program 101-134, 201-236, 301-332 Step

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (Reference) G06T 7/20 300 G06T 7/20 300B 7/60 150 7/60 150B F-term (Reference) 5B057 CA08 CA12 CA16 CE09 DA07 DB02 DB09 DC30 DC33 DC36 DC39 5B068 AA01 BD09 BE08 CC17 CD06 5B087 AA00 BC12 BC13 BC26 BC32 DD03 DE07 5L096 BA18 EA35 FA66 FA69 HA02 JA09 KA03 KA15

Claims (8)

[Claims] An image obtained by cutting out a face organ from one frame of an input image obtained by photographing a face part of an operator of a computer is registered as a first template, and at the same time, the position of the image in the input image is determined by the amount of movement of a pointer. And a template position which is registered as an operation reference point which is a reference point for calculating a moving direction, and which is a coordinate of an area where the first template best matches in an input image input in frame units via an imaging device. A mouse substitution method comprising: detecting coordinates in real time; calculating coordinates of a mouse pointer in a next frame from the coordinates of the template position and the coordinates of the operation reference point; and moving the mouse pointer to the coordinates. 2. The method according to claim 1, further comprising the step of:
The image information of the same facial organ as that of the template in different facial expressions is cut out and registered as a second template, and the second template is detected in the input image simultaneously with the search for the first template, and the operator If the second template is detected continuously over a predetermined time or number of frames by changing the expression over time, this is counted as one unit, and the second template is counted according to the value counted within the predetermined time. The mouse replacement method according to claim 1, wherein a pre-registered instruction command is issued to the OS. 3. While a pre-registered instruction command is sequentially presented to an operator according to a certain rule, when a desired instruction command is presented, the operator changes his / her expression for a certain period of time, and the second template is displayed. 2. The system according to claim 1, wherein the instruction instruction is issued to the OS when the observation is continuously performed for a predetermined time or a predetermined number of frames.
Or the mouse replacement method according to 2. 4. An input image from the imaging device is compared with a template position coordinate of a first template and an operation reference point, and a position of the template position coordinate having the operation reference point as an origin is set at the center of the computer screen. 2. The method according to claim 1, wherein the coordinates are converted to coordinates having the origin as an origin, and a mouse pointer is displayed at the coordinates. 5. An image input from the imaging device, the template position coordinates of the first template are compared with an operation reference point, and a mouse pointer is moved based on the direction and distance of the template position coordinates from the operation reference point. The mouse replacement method according to claim 1, wherein the direction and the amount of movement per frame are determined. 6. A method according to claim 1, wherein a template position coordinate of the first template is compared with an operation reference point with respect to an input image from the imaging device, and a distance between the template position coordinate and the operation reference point exceeds a predetermined value. Instead of the image information recorded in advance, an area that best matches the first template is cut out in the current frame, registered as the first template in the next frame, and searched in the subsequent frames while updating the template as needed. 2. The mouse according to claim 1, wherein when the distance from the operation reference point returns within a certain value, the pre-recorded image information is registered again as the first template, and a search process is performed in subsequent frames. Alternative method. 7. A mouse replacement program for executing the mouse replacement method according to claim 1 on a computer. 8. A recording medium in which a mouse replacement program for executing the mouse replacement method according to claim 1 is recorded on a computer.