JP2005301583A - Typing input device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a typing input device which is available in an arbitrary place for inputting characters with a sense of feeling similar to the input of characters by typing being the characteristics of an existing keyboard. <P>SOLUTION: This typing input device is provided with: a camera mounted on the wrist of a user to photograph the finger of the user; a sensor mounted on the arm of the user to recognize the inclination of the backward and forward and right and left directions of the arm of the user; and an information processor for detecting the position of the hand of the user by the inclination sensor, and for deciding typing keys, detecting the position of the fingertip of the user from an image photographed by the camera and detecting the typing operation based on the detected position of the hand and the detected position of the fingertip. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a typing input device.

  Characteristic features of the keyboard, which is an existing character input device for personal computers (PCs), include the ability to input characters by hitting keys with fingers, so-called typing, and one-to-one correspondence between keys and characters (one degree of freedom). And the like. In addition to using all fingers for input, the number of times the key is pressed only once until the character to be input appears, so if you master it to some extent, you can input characters quickly and reliably There are advantages.

  However, a disadvantage is that the keyboard as a whole needs to have a certain size, and is unsuitable for carrying around or using in any place. Simply downsizing makes it difficult to use the keys, making it a little difficult to use and cannot maintain ease of use.

  On the other hand, with the recent development of hardware technology, the miniaturization and weight reduction of PC bodies and peripheral devices are progressing dramatically. For this reason, wearable computing that uses a PC regardless of time and place is gradually being realized by wearing reduced-sized computing devices on the body.

  However, a character input device represented by a keyboard that has been conventionally used as a character input means to a computer becomes difficult to use for a user as the size is reduced as described above, and is portable and functional. However, there was a problem that it was difficult to achieve both.

  Under such circumstances, instead of an actual keyboard, several proposals have been made for a virtual keyboard (virtual keyboard) that has a function equivalent to or similar to a keyboard and mobility that is not found in an existing keyboard.

  A first type of conventional virtual keyboard is a keyboard-mounted type. This is literally to attach a miniaturized keyboard or button to a part of the body and clothes etc., but the number of keys that can be arranged is limited due to the area problem of the wearing part, and when using it There is a problem that troublesomeness arises because the hand must be moved to the mounting site.

  A second type of conventional virtual keyboard is a desktop projection type. This is because the keyboard image is projected onto a flat surface and characters can be input by touching the projected surface with a finger. However, it cannot be used without a stable and sufficiently large flat surface such as a desk. There was a problem that the place of use would be limited.

  A third type of conventional virtual keyboard is a glove-mounted type. In this method, characters are input from the movement of the finger and hand by attaching the device to the hand or finger. However, since the entire hand is restrained, the transition to other work using the finger is not smooth. There was a problem.

  In view of the above, the present invention overcomes the drawbacks of the prior art, enables character input with a feeling similar to typing by characterizing the existing keyboard, and can be used in any place An object of the present invention is to provide a typing input device.

  According to the first aspect of the present invention, there is provided a camera attached to a user's wrist so as to photograph a user's finger, a sensor for detecting a position and / or inclination of the user's hand, and information detected by the sensor. An information processing apparatus that determines a key that can be pressed, detects a position of a user's fingertip for each frame from an image captured by the camera, and detects a keystroke action based on the detected position of the hand and the fingertip; A typing input device comprising:

  The invention described in claim 2 is the typing input device according to claim 1, wherein the sensor is a tilt sensor.

  According to a third aspect of the present invention, in the typing input device according to the first aspect, the sensor is a triaxial sensor.

  A fourth aspect of the present invention is the typing input device according to any one of the first to third aspects, wherein the camera includes an infrared transmission filter.

  The invention according to claim 5 is the typing input apparatus according to claim 4, further comprising an infrared irradiation light source for irradiating the user's hand with infrared rays.

  According to a sixth aspect of the present invention, in the typing input device according to any one of the first to fifth aspects, the information processing apparatus binarizes an image photographed by the camera and binarizes the image. This is a typing input device characterized in that outline extraction and thinning are performed on an image, and the position of a user's fingertip is detected based on these.

  The invention according to claim 7 is the typing input apparatus according to claim 6, wherein the information processing apparatus determines a downward end point as a fingertip in the thinned image. .

  According to an eighth aspect of the present invention, in the typing input device according to the sixth or seventh aspect, the information processing apparatus searches for a downward arched portion in the contour extracted image, and within a certain section. The typing input device is characterized in that the lowest point is determined as a fingertip.

  According to a ninth aspect of the present invention, in the typing input device according to any one of the first to eighth aspects, the information processing device causes the fingertip to abruptly be displayed on the screen based on the fingertip position detected for each frame. Moved downward, the distance that the finger moved is longer than the distance that all other fingers moved, the distance moved is more than a certain distance, and the time from when it starts to move until the movement direction changes is constant The typing input device determines that the finger has performed a typing operation when the time is within the time.

  According to a tenth aspect of the present invention, in the typing input device according to any one of the first to ninth aspects, the information processing device detects each finger according to the inclination of the user's arm detected by the sensor. The typing input device is characterized in that a key candidate assigned to the key is changed.

  The invention according to claim 11 is the typing input device according to any one of claims 1 to 10, wherein the information processing device displays a virtual keyboard on a display device and displays typed keys. It is a typing input device characterized by changing.

  The invention according to claim 12 is the typing input device according to claim 11, wherein the information processing device distinguishes and displays key candidates assigned to each finger on the virtual keyboard. This is a typing input device.

  According to the present invention, even if there is no physical keyboard, the same typing input as using a full keyboard can be performed without choosing a use situation, such as standing in the air, on a knee, or on a train. Moreover, since the virtual keyboard is software, the keyboard size and key layout can be easily customized and switched, and can be applied to various operation panels. In addition, since typing can be done without touching anything, it can be expected to be applied to user interfaces in medical settings and extreme environments.

  FIG. 1 is a diagram showing an example of the configuration of a typing input device according to the present invention. The typing input device 1 includes a tilt sensor 2, a small camera 3, an infrared light irradiation light source 4, and an arm band 5. The tilt sensor 2, the small camera 3, and the infrared irradiation light source 5 are attached to the armband 5 and attached to the user's arm. The tilt sensor 2 and the small camera 3 are connected to an information processing device (not shown). Power is supplied from the outside to the tilt sensor 2, the small camera 3, and the infrared irradiation light source 5, or an internal power supply is built in.

  The tilt sensor 2 is, for example, a three-axis sensor or a tilt sensor, recognizes the tilt of the user's arm in the front-rear and left-right directions, and transmits the information to the information processing apparatus. The information processing apparatus regards the degree of tilt as the amount of hand movement from the home position, and selects a key candidate that can be typed. FIG. 2 is a diagram for explaining the movement of key candidate selection. The gray part indicates the range of keys that can be typed. As shown in this figure, with the home position at the center of the keyboard, tilting the wrist back and forth will divide the keys above or below the home position, and tilting the wrist to the left or right will place the left or left key from the home position. Make a distinction.

  The small camera 3 is arranged so as to photograph a fingertip under the wrist, and transmits the obtained image to the information processing apparatus. In order to capture the entire finger in the field of view from the position under the wrist, it is desirable to attach a fisheye lens to the small camera 3. Further, in order to observe the movement of the fingertip from the video imaged by the small camera 3, it is necessary to stably acquire the hand image. Therefore, the infrared light irradiation light source 4 is disposed so as to irradiate infrared light under the small camera 3, and an infrared transmission filter is attached to the small camera 3. In this way, since the infrared transmission filter blocks visible light, the hand close to the small camera 3 is projected very brightly, and the image captured by the small camera 3 can be obtained only by threshold processing. It is possible to binarize in other areas.

  The information processing apparatus performs processing such as hand region extraction, fingertip extraction, detection of the fingertip position in the video sent from the small camera 3, and determination of a typing operation when the fingertip movement satisfies certain conditions. To recognize typing action. FIG. 3 is a flowchart illustrating a process of extracting a hand region and detecting a fingertip, and FIG. 4 is a flowchart illustrating a process of recognizing a typing operation from the movement of the fingertip. Hereinafter, each process will be described with reference to these drawings.

  First, referring to FIG. 3, a process of extracting a hand region and detecting a fingertip will be described. FIG. 5 is a diagram showing an image processed in the operation shown in FIG. In step s101, the information processing apparatus binarizes the video obtained by the small camera 3 (61 in FIG. 5). In step S102, the binarized video is subjected to a labeling process, and what occupies the maximum area is determined as a hand area and extracted.

  In step s103, thinning is performed using the thinning method of Hilditch (62 in FIG. 5). In step s104, a downward end point is searched based on the thinned image, and the detected portions are associated as the fingertips from the thumb to the little finger in order from the left of the image (63 in FIG. 5).

  Simultaneously with the processing of steps s103 and s104, in step s105, the contour line at the bottom of the screen in the hand region portion of the binarized image is extracted (64 in FIG. 5). In step s106, the contour line is traced along the extracted contour line from the left to the right of the screen to search for a downward arched portion. FIG. 6 is a diagram illustrating such a search for the downward upward arch portion. Specifically, for each point of interest to be tracked, a point on the outline existing within a certain distance (within 3 pixels in the example of FIG. 6) around the point of interest is displayed on the screen. It is determined whether or not it exists above, that is, a local minimum. As a result, when it is determined that the target point is a local minimum value within a certain interval, it is detected as a fingertip point and associated with each fingertip from the thumb to the little finger in order from the left of the image (65 in FIG. 5).

  The reason why the fingertips are detected by simultaneously performing the two processes of steps s103 and s104 and steps s105 and s106 is that if the fingertips overlap each other even if they are visible from a small camera, they are binarized and thinned. This is because the portions of the fingertips that overlap each other when performing processing do not become end points but are connected to form a ring. In this case, the five fingertips are not normally detected by the thinning process. Therefore, in step s107, it is determined whether or not five fingers are normally detected in the result obtained in step s104. If so, in step s108, the detection is performed by thinning processing (steps s103 and s104). If the finger overlaps and five fingertips cannot be detected, it is determined in step s109 whether the five fingers are normally detected in the result obtained in step s106. For example, the result of the contour line extraction process (steps s105 and s106) is adopted. In this way, it is possible to continue to detect the fingertip almost always. If the five fingertips cannot be detected normally in both processes, it is determined in step s111 that the fingertip cannot be detected, and the detection of the fingertip position in that frame is given up.

  Next, a process for recognizing a typing operation from a fingertip movement will be described with reference to FIG. FIG. 7 is a graph showing an example of the movement of five fingers within a certain time. The horizontal axis is the number of frames (time), and the vertical axis is the coordinate value in the screen Y-axis direction (vertical direction). Refer to this graph as an example.

  In step s201, the fingertip position is acquired for each frame by the processing described with reference to FIG. In subsequent steps, based on these fingertip positions, it is observed how each fingertip moves on the screen. At that time, if a certain fingertip operation satisfies a certain condition, it is determined that a typing operation has been performed.

  First, in step s202, after suddenly starting to move downward in the screen (the coordinate value at this moment is the highest point), moving in the same direction for a while, the moving direction is reversed or stopped in the upward direction on the screen (this It is determined whether or not there is a finger (with the instantaneous coordinate value as the lowest point). If not, the process proceeds to step s203, and it is determined that there is no key input in the current frame. If there is, the process proceeds to step s204. When a fingertip that has been stationary (not moving in the Y-axis direction) or moving upward in the screen moves within 2 frames or less, for example, 2 pixels or less within one frame from a certain time, The movement start position at this time is set as the highest point. In the example of FIG. 7, the section from the A point to the B point and the section from the C point to the D point in the middle finger graph correspond to places where it is recognized that “the movement has started suddenly downward on the screen”. The top points in are A point and C point, respectively. The lowest point is B tent E point.

  In step s204, the distance that the finger of interest (the finger determined to have moved in step s202, the middle finger in the example of FIG. 7) has moved, that is, the distance from the highest point to the lowest point, It is judged whether it is larger than the movement distance of the finger. If not, the process proceeds to step s203, and it is determined that there is no key input in the current frame. If so, the process proceeds to step s205. In the example of FIG. 7, the section from the A point to the B point of the middle finger is 2 pixels, and the section from the C point to the E point is 11 pixels. First, when comparing the interval from point A to point B with other fingers, it can be seen that the index finger and little finger are moving downward on the screen at the time when the lowest point is reached (26th frame). The index finger has moved one pixel from the Y coordinate value 80 to the time of reaching the lowest point of the middle finger in the 25th frame. The little finger has moved 2 pixels from the Y coordinate value 68 to the time when the middle finger reaches the lowest point in the 24th frame. Therefore, in this case, since the moving distance of the middle finger and the moving distance of the little finger are the same, it is determined that the condition of step s204 is not met. Next, when comparing the section from point C to point E with other fingers, it can be seen that the ring finger and little finger are moving downward in the screen at the time when the lowest point is reached (the 32nd frame). The ring finger has moved by one pixel from the Y coordinate value 85 to the time of reaching the lowest point of the middle finger in the 31st frame. The little finger moves one pixel from the Y coordinate value 68 to the time when the middle finger reaches the lowest point in the 31st frame. Therefore, in this case, since the moving distance of the middle finger is the maximum compared with the moving distance of the other fingers, it is determined that the condition of step s204 is met.

  In step s205, it is determined whether or not the moving distance from the highest point to the lowest point of the finger of interest is a certain distance or more. If the distance is less than the predetermined distance, the process proceeds to step s203, and it is determined that there is no key input in the current frame. If the distance is equal to or greater than the certain distance, the process proceeds to step s206. The threshold values of the movement distance from the highest point to the lowest point for determining whether or not the typing operation is performed are the distortion of the screen by the fisheye lens, the distance from the small camera to the fingertip (about 13 cm), the screen of each finger For example, the thumb and little finger are set to 5 pixels, and the remaining fingers are set to 4 pixels or more. This distance corresponds to about 2 cm in the real world. In the example of FIG. 7, when looking at the section from the C point to the E point of the middle finger, the moving distance from the highest point to the lowest point is 11 pixels. Therefore, since the threshold value that is regarded as the typing operation of the middle finger is exceeded, it is determined that this section meets the condition of step s205.

  In step s206, whether the finger of interest is stopped after it starts moving, or the time until the direction of movement changes again, that is, whether the time difference between the time of starting from the highest point and the time of reaching the lowest point is within a certain time. Judge whether. If the time difference exceeds a certain time, the process proceeds to step s203, and it is determined that there is no key input in the current frame. If the time difference is within a certain time, the process proceeds to step s207. For example, if the difference between the departure time of the highest point and the arrival time of the lowest point is within 0.5 seconds, it is determined that the typing operation is being performed. In the example of FIG. 8, when the section from the C point to the E point of the middle finger is seen, it can be seen that it takes 2 frames, that is, 0.2 seconds, to move from the highest point to the lowest point. Therefore, it is determined that this section meets the condition of step s206.

  In step s207, it is determined from the determination results in steps s202, s204, s205, and s206 that the finger of interest has performed a typing operation, and in step s208, key input corresponding to the finger is performed. In the example of FIG. 8, since the section from the C point to the E point of the middle finger satisfies the above four criteria, a typing operation on the middle finger is detected at the 32nd frame.

  In actual operation, for example, a typing input device as shown in FIG. 2 is attached to the wrist of the right arm and the operation is performed with only one hand. When a typing operation (an operation of swinging down a finger) is performed hollow at intervals such as when the keyboard is actually operated, a key corresponding to each finger is input. Selection of key candidates that can be typed is performed by tilting the front, rear, left and right of the wrist. Also, the keyboard input can be switched between valid / invalid by moving the wrist up and down. A hollow typing operation is accepted as a key input only when enabled, and no finger operation is accepted as a key input when disabled. In addition, it is possible to reset the tilting state of the arm serving as the home position by this key input valid / invalid switching operation. Recognize each time the arm is tilted when it is switched from disabled to enabled as the home position.

  FIG. 9 is a diagram showing an example of a presentation screen displayed on a display device of a computer connected with a typing input device according to the present invention. An image obtained by thinning and contour extraction of a hand image taken by a small camera and a virtual keyboard visualized are presented at the top of the screen. The key layout of the virtual keyboard is the same QWERTY layout (alphabet part) as the actual keyboard.

  FIG. 10 is a diagram for explaining a change in keys assigned to each finger according to the tilt of the wrist. In the example shown in this figure, the keys assigned to three of the index finger, the middle finger, and the ring finger are changed by changing the tilt of the wrist forward, backward, left and right. Special keys such as Enter, Space, and BackSpace keys that are frequently used are always assigned to the thumb and little finger. On the keyboard screen displayed on the screen, candidates for keys that can be typed at that time are displayed in blue, for example, and the typed keys are instantaneously illuminated in yellow, for example, to react.

  FIG. 11 is a diagram for explaining an operation for changing the input character type. By pressing the Ca (CapsLock) key and the Sp (special character) key, the types of characters that can be input can be changed.

  As the means for detecting the position and / or inclination of the user's hand, the inclination sensor and the triaxial sensor for detecting inclination have been described as examples. However, the present invention can be implemented by other types of sensors.

It is a figure which shows an example of a structure of the typing input device by this invention. It is a figure explaining the movement of key candidate selection. It is a flowchart explaining the process which extracts a hand area | region and detects a fingertip. It is a flowchart explaining the process which recognizes typing operation | movement from the motion of a fingertip. It is a figure which shows the image processed in the operation | movement shown in FIG. It is a figure explaining search of the downward arch upper part. It is a graph which shows an example of a movement of five fingers within a certain time. It is a figure which shows an example of the presentation screen displayed on the display apparatus of the computer which connected the typing input device by this invention. It is a figure explaining the change of the key allocated to each finger according to the inclination of a wrist. It is a figure explaining operation of input character kind change.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Typing input device 2 Tilt sensor 3 Small camera 4 Infrared irradiation light source 5 Armband

Claims (12)

  A camera attached to the user's wrist so as to photograph the user's finger, a sensor for detecting the position and / or inclination of the user's hand, and a key that can be pressed from the information detected by the sensor; Typing comprising: an information processing device that detects a fingertip position of a user for each frame from an image photographed by a camera and detects a keystroke action based on the detected hand position and fingertip position. Input device.   2. The typing input device according to claim 1, wherein the sensor is a tilt sensor.   2. The typing input device according to claim 1, wherein the sensor is a triaxial sensor.   4. The typing input device according to claim 1, wherein the camera includes an infrared transmission filter.   5. The typing input device according to claim 4, further comprising an infrared irradiation light source for irradiating a user's hand with infrared rays.   6. The typing input device according to claim 1, wherein the information processing device binarizes an image photographed by the camera, and performs contour line extraction and thinning on the binarized image. And a typing input device that detects the position of a user's fingertip based on the results.   The typing input device according to claim 6, wherein the information processing device determines a downward end point as a fingertip in the thinned image.   8. The typing input device according to claim 6 or 7, wherein the information processing device searches for a downward arched portion in the contour-extracted image, and uses the lowest point in a certain section as a fingertip. A typing input device characterized by determining.   The typing input device according to any one of claims 1 to 8, wherein the information processing device rapidly moves the fingertip downward in the screen based on the fingertip position detected for each frame. If the distance traveled is longer than the distance traveled by all other fingers, the distance traveled is equal to or greater than a certain distance, and the time from the start of movement until it stops or the direction of motion changes is within a certain time, It is determined that the typing operation has been performed.   10. The typing input device according to claim 1, wherein the information processing device changes a key candidate assigned to each finger according to a tilt of a user's arm detected by the sensor. A typing input device.   The typing input device according to any one of claims 1 to 10, wherein the information processing device displays a virtual keyboard on a display device and changes a display of typed keys. apparatus.   12. The typing input apparatus according to claim 11, wherein the information processing apparatus distinguishes and displays key candidates assigned to each finger on the virtual keyboard.

Images