WO2012056864A1 - Input device, information apparatus provided with the input device, program for causing computer to function as input device, and method for using the input device to input characters - Google Patents

Abstract

Provided is an input device which can mitigate finger and arm fatigue for the purpose of achieving high-speed key entry. The input device has: a camera configured to consecutively photograph fingers of a user of the input device; a finger position detection unit configured to detect the position of each finger of the user on the basis of an image photographed by the camera; a home position detection unit configured so that when fingers of the user make tactile contact with a desktop and come to rest, the positions of the fingers of the user at the point in time are stored as home positions upon a virtual keyboard such that the distances from the camera to the keys for which each finger is responsible differ and which has a shape unique to the user; a typing behavior detection unit configured so as to detect that a finger of the user which has been detected by the finger position detection unit has typed a key upon the virtual keyboard; and a key code generation unit configured so as to generate the code of the key upon the virtual keyboard corresponding to the position where there has been a typing operation.

Description

INPUT DEVICE, INFORMATION EQUIPMENT HAVING THE INPUT DEVICE, PROGRAM FOR MAKING COMPUTER TO FUNCTION AS INPUT DEVICE, AND METHOD FOR INPUT CHARACTER USING INPUT DEVICE

This disclosure relates to information equipment, and more specifically to a technique for optically detecting finger movement and inputting information.

Many mobile phones realize one-time key entry by using a few keys and one-handed keystrokes with few users. Many smartphones and tablet terminals display a small soft keyboard on the display screen and realize key input by a touch panel. Information devices such as personal computers often have a hard keyboard with key tops arranged in a horizontal row.

Equipment that has been put to practical use as a virtual keyboard includes equipment that realizes key input by projecting a soft keyboard onto a desk with laser light and touching the desk with fingers. Patent Document 1 (Japanese Patent Laid-Open No. 6-83512) discloses a technique for detecting the movement of a finger with a camera without projecting a soft keyboard on a desk. Patent Document 2 (Japanese Patent Laid-Open No. 2003-288156) discloses a technique for detecting a three-dimensional position of a finger by providing two photographing means. Japanese Patent Application Laid-Open No. 2004-500657 discloses a technique for detecting a three-dimensional position of a finger using a special three-dimensional sensor. Furthermore, Non-Patent Document 1 (Hafiz Adnan Habib and Muid Mafti, “Real Time Mono Vision Gesture Based Virtual Keyboard System”, IEEE Transaction on Consumer Electronics, Vol.52, Issue4, Nov. 2006) shows the position of the finger joint. A technique for detecting the position of a finger or a keystroke operation by detecting the finger is disclosed.

JP-A-6-83512 JP 2003-288156 A Japanese translation of PCT publication No. 2004-500657

Personal computers have become widespread, and it is common to input keys such as alphabets using both hands on a standard hard keyboard such as a QWERTY keyboard. Skilled users can perform touch typing, which is high-speed key input, without looking at these hard keyboards. However, in the portable terminal, there is no means for achieving both a small external size and high-speed key input.

Many mobile phones have few keys, and in many smartphones and tablet devices, the soft keyboard displayed on the display screen is smaller than the normal hard keyboard, and the key spacing is narrow, High-speed key input using all fingers is difficult. Some devices integrate a small hard keyboard. However, since the interval between the keys is narrow, input with two fingers is the limit, and high-speed key input is difficult. If a large hard keyboard is used as an external device, high-speed key input is possible. However, carrying such an external device impairs the characteristics of a handheld portable terminal. In addition, hard keyboards for information devices such as personal computers require that the keys are arranged side by side and that the fingertips must be raised to avoid double keystrokes. Many of the workers are suffering from tendonitis and stiff shoulders.

There is a device that uses a laser beam to project a virtual keyboard on a desk and detects the movement of fingers with a camera to realize key input, but it is necessary to carry the device for projection and it is difficult to say that it is wearable .

Patent Document 1 discloses a technique for detecting the movement of a finger with a camera without projecting a soft keyboard on a desk, but does not mention a technique for making it wearable.

Patent Document 2 discloses a technique for detecting a three-dimensional position of a finger with two cameras. Although it is possible with only one camera, there is no mention of specific means for realizing it.

Patent Document 3 discloses a technique for detecting a three-dimensional position of a finger using a special three-dimensional sensor. However, this sensor cannot be easily obtained, and the resolution of the distance from the sensor to the finger is about 1 cm.

Also, Non-Patent Document 1 discloses a technique for detecting the position of a finger and a keystroke action by detecting the position of a finger joint. In the case of this method, like a normal hard keyboard, it is necessary to stand on the desk with the fingertips standing almost vertically. In the case of a normal hard keyboard, the impact on the finger due to keystroke is mitigated by the springiness of the key top, but when you hit the key on the desk, the impact is transmitted directly to the joint, causing pain and fatigue in the fingers and arms to increase The burden is large in doing.

An ergonomic keyboard has been developed as a hard keyboard that is hard to get tired of fingers and arms even after long-time key input, but such a key input device has not been developed in the case of a virtual keyboard.

Therefore, there is a need for a technique that can reduce the fatigue of fingers and arms without impairing the wearability of the mobile terminal, and enables high-speed key input using an inexpensive camera. In addition, there is a need for a technique that can reduce the fatigue of fingers and arms caused by the use of a hard keyboard of an information device such as a personal computer.

An input device according to an embodiment includes a camera configured to continuously capture a front image of a finger of a user of the input device when the input device is disposed close to a desk, and a captured front surface. The finger position detection unit configured to detect the position of the user's finger from the image and the position of the user's finger at that time when the user's finger touches and rests on the desk. The home position detector configured to detect the home position on the virtual keyboard having a different distance from the camera of the key in charge and having a shape unique to the user, and the user's finger on the virtual keyboard A keystroke operation detector configured to detect that a key has been pressed, and a key code generator configured to generate a key code on the virtual keyboard corresponding to the position where the keystroke occurred. Provided with a door.

Employees who are keying in many letters every day often suffer from tendonitis or stiff shoulders. Most of the workers engaged in such work can perform touch typing without looking at the keyboard, and in one aspect, symptoms can be expected to be improved.

The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention which is to be understood in connection with the accompanying drawings.

It is explanatory drawing of the description of the three-dimensional space used by the following description. It is explanatory drawing of the row direction and column direction of the virtual keyboard used by the following description. It is a system perspective view of the present invention. It is a system perspective view at the time of holding a finger with an easy posture in Drawing 2A. It is a figure of the virtual keyboard at the time of holding a finger with a comfortable posture like FIG. 2B. FIG. 2B is a diagram of a virtual keyboard when fingers are held in a comfortable posture as shown in FIG. 2B, and is an example in which a row of key tops are diagonally arranged like a normal hard keyboard. It is explanatory drawing of this invention. It is an image figure of the front image of the finger image | photographed with the camera of this invention. The finger is ready to be placed on a normal hard keyboard. FIG. 5B is an image view in which only the right hand approaches the camera for one line of the virtual keyboard in FIG. 5A. It is an image figure of the front image of the finger image | photographed with the camera of this invention. The fingers are in an easy posture. FIG. 6A is an image diagram in which only the right hand moves away from the camera for one line of the virtual keyboard in FIG. It is a functional block diagram which shows an example of a functional structure of this invention. It is a functional block diagram which shows another example of a functional structure of this invention. It is explanatory drawing of the method of inputting a command with the finger | toe stationary at a home position. It is explanatory drawing of the finger | toe placed in the home position on a virtual ten keyboard. It is explanatory drawing of an example of the method of superimposing and displaying the present detection position of the finger arrange | positioned on the desk on the virtual keyboard displayed on a display. It is explanatory drawing of an example of the method of displaying the virtual keyboard, the image of the fingers which the camera caught on the display, and the input character string. It is explanatory drawing of an example of the method of measuring the position and magnitude | size of a finger. It is explanatory drawing of an example of the method of calculating the horizontal movement distance from the home position of the finger on a virtual keyboard. It is explanatory drawing of Formula (1). It is explanatory drawing of an example of a keystroke operation | movement detection method. It is explanatory drawing of an example of a keystroke operation | movement detection method. It is an example of a hardware block diagram of the present invention. It is a figure which shows an example of the detection method of a desk edge, and the measuring method of the area of a finger. It is an example of the method of obtaining the area of a nail | claw. State A to state G are diagrams illustrating a procedure of a method for inputting a command with fingers. It is an example of the flowchart which concerns on this Embodiment. It is an example of the flowchart which concerns on this Embodiment. This is an example of a virtual keyboard in which the positions of keys assigned to each finger overlap each other. This is an example of a virtual keyboard in which the positions of keys assigned to each finger overlap each other. It is an example of the system perspective view concerning this Embodiment. It is an example of the system perspective view concerning this Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

FIG. 1A is an explanatory diagram of a three-dimensional space notation used in the following description. Description will be made assuming that a certain position of the camera 20 is an origin, a plane on the desk is an XY plane, and a height direction from the desk is a Z axis. However, this shows only the direction of the coordinate axis. In the description of the image taken by the camera 20, the X-axis coordinate is represented by a pixel number, and the size of the finger on the image is represented by the number of pixels that it occupies. The direction of the finger photographed by the camera 20 from the positional relationship of FIG. 1A is referred to as a front image of the finger. FIG. 1B is an explanatory diagram of the definition of the row direction and the column direction of the virtual keyboard used in the following description. The direction from the camera to the perspective is the column direction.

FIG. 2A is an example of a system perspective view according to the present embodiment. The mobile terminal placed on the desk and the user's fingers are shown. A part of the image on the desk acquired by the camera 20 mounted on the side surface of the mobile terminal 10 and arranged close to the desk is displayed on the finger image display unit 33 on the display 30. The user can check whether his / her left hand 50A and right hand 51A are within the shooting range of the camera 20 by looking at the front image of his / her finger, and the straight line 21 which is the horizontal angle of view of the camera 20 can be confirmed. The fingers of both hands can be accommodated between the straight lines 22.

The light irradiation unit 40 irradiates the range of the straight line 41 and the straight line 42 so that the contours of fingers and nails can be detected more clearly. On the keyboard display unit 32, finger detection positions detected by the key input device are displayed superimposed on the virtual keyboard. The user sees this and can hit the key more accurately. The input character string display unit 31 displays a character string input by the user, which is obtained from the key codes generated by the key input device so far.

FIG. 2B is an example of a perspective view of a system in which a user holds a finger in a comfortable posture. The camera 20 is mounted on a certain surface of the display 30 of the mobile terminal 10 and can be used as a self-portrait camera. In a normal hard keyboard, the key tops are aligned in the horizontal direction. Therefore, when the user holds the user in such an easy posture, the length of each finger is different, and the user cannot correctly press the key. Further, when the user hits the key, the user presses the surrounding keys at the same time. In the present embodiment, the skill of the user of the system according to the present embodiment from the case of holding a finger as in a normal hard keyboard as shown in FIG. 2A to the case of holding in a sufficiently easy posture as shown in FIG. 2B. You can choose freely according to the degree.

FIG. 3A illustrates a virtual keyboard 60A in the case where a finger is held in a comfortable posture as in FIG. 2B (note that the virtual keyboard is only shown in FIG. I want to be) The distance from the camera of the key of each row changes according to the length of the finger. Since the user can freely set the shape of the virtual keyboard depending on the posture of the fingers, an ergonomic keyboard suitable for him / her can be configured. Although FIG. 3A shows an integrated virtual keyboard, the left and right hands can be separated from each other than when using a normal hard keyboard so that a virtual keyboard separated in the center can be obtained. The burden can be further reduced. Since the thumb basically only presses the space key, it can be input that the space key is pressed no matter where the thumb hits the virtual keyboard.

FIG. 3B illustrates a virtual keyboard 60B in which a row of key tops are arranged obliquely like a normal hard keyboard. The key top of the space key is not shown because the thumb is regarded as a space key no matter where the key is pressed. Whether the user uses the virtual keyboard of FIG. 3A or FIG. 3B is determined based on the setting that the user inputs in advance to the key input device. Alternatively, there is a method of setting by learning with an initial character string described later.

FIG. 4 is an explanatory diagram of this embodiment. FIG. 4 shows the principle that the virtual keyboard can be freely set as described above and the principle that the key input device detects the distance change between the camera and the finger (a specific calculation method of the distance change). Is described in FIG. The left hand 50B and the right hand 51B placed on the home position of the virtual keyboard 60C constitute a horizontal angle of view (viewing angle) of the camera 20 that is mounted on the side surface of the mobile terminal 10 and arranged close to the desk. It is placed in the range of the straight line 21 and the straight line 22.

FIG. 4 shows an example in which the horizontal angle of view of the camera 20 is 90 degrees, the number of pixels is VGA, and the number of horizontal pixels is 640. The pixel numbers in the horizontal direction are expressed as 1 to 640. FIG. 4 illustrates a case where the right hand 51B has moved from the home position to the key position 51E one row above. The pixel 25 that captures the left end of the four fingers of the right hand excluding the thumb in the right hand 51B at the home position is the pixel number 277, and the pixel 26 that captures the right end is the pixel number 147. The pixel 27 that captures the left end of the four fingers of the right hand excluding the thumb when moved to the key position 51E one level above is the pixel number 282, and the pixel 28 that captures the right end is the pixel number 137.

Therefore, the width of the four fingers of the right hand excluding the thumb captured by the camera is increased by 15 pixels due to the movement of the fingers. Although not shown, the width of one finger is approximately 3 pixels wider. By making the number of pixels of the camera larger than that of the VGA, more changes in the number of pixels can be obtained. FIG. 4 shows an example of finger arrangement similar to the case where the fingers are held in a straight line on a normal hard keyboard, but the fingers are held in a comfortable posture at the home position based on the detection principle of the back and forth movement described above. Even in this case, the system according to the present embodiment can detect the movement distance by measuring a change in the width of the finger when moving back and forth with reference to the width of the finger at that time.

The method for determining that the finger is on the home position is, for example, the time when the finger on the image is approximately stationary for less than one pixel on average on both the vertical and horizontal movements and the change in the width of the finger over 10 frames. There is a technique. At that time, it is not necessary to distinguish whether the fingers are held in a straight line on a normal hard keyboard or in an easy posture. The key point of the system according to the present embodiment is to determine the key on which the finger is placed by measuring the distance of the forward / backward movement based on the change in the width of the finger from the current finger width. .

• The pitch of the virtual keyboard key tops varies depending on the user. When the user's finger is placed at the home position, the average value of the pitch of the finger corresponds to the key top pitch, so the key input device can determine the left and right pitches of the key top. Also, since each key top is substantially square, it can be determined how much the finger has moved back and forth for one line of the virtual keyboard assumed by the user.

The angle θ in FIG. 4 is an angle from the pixel number 1 in the horizontal field angle to the pixel number with the finger. As can be seen from FIG. 4, when the horizontal pixel number is known, the angle θ can be determined. This relationship is used in the description of FIG. Also, the horizontal pixel numbers in FIG. 4 are allocated at equal intervals while ignoring distortion caused by camera lens aberrations. This countermeasure will also be described with reference to FIG.

FIG. 5A is an image diagram of a front image of fingers captured by the camera according to the present embodiment. FIG. 5A shows a front image of the left hand 50C and the right hand 51C at the home position. The fingers are ready to be placed on a normal hard keyboard. The four fingers other than the right thumb are surrounded by an outline 80A.

FIG. 5B is an image diagram in which only the right hand of FIG. 5A approaches the camera for one line of the virtual keyboard. FIG. 5B shows a right hand 51D having an increased external size and its outline 80B. In FIG. 4, the change in the outer size photographed by the camera by moving the finger back and forth has been explained only by the number of pixels in the horizontal direction, but as can be seen from FIGS. 5A and 5B, the change is actually It is an area change. Therefore, the use of this information makes it easier to identify the forward / backward movement distance.

FIG. 6A is also an image diagram of a front image of fingers captured by the camera according to the present embodiment. The fingers are in an easy posture. FIG. 6A shows finger images of the left hand 52C and the right hand 53C at the home position. The widths 81A, 81B, and 81C indicate the width of the middle finger extracted from the contour of the finger image.

FIG. 6B is an image diagram in which only the right hand of FIG. 6A is moved away from the camera by one line of the virtual keyboard, and the middle finger is keyed. FIG. 6B shows the right hand 53D in which the outer size of the right hand 53C in FIG. 6A is reduced, and the widths 81D, 81E, and 81F of the middle finger that has performed the keystroke operation. Thus, the change in the width of the finger can be captured at a plurality of points instead of one point, and the measurement accuracy can be improved by taking the average value.

In FIG. 6B, the middle finger of the right hand performing the keystroke operation touches the desk, and the other fingers of the right hand are away from the desk. The left hand that is not performing the keystroke operation remains touching the desk. Thus, even if the fingers on the side not performing the keystroke operation are arranged in an easy posture touched on the desk, there is no problem in detecting the keystroke operation. In addition, as can be seen from FIG. 6B, by taking a front image of a finger with the camera placed close to the desk, the difference in height between the finger that made the keystroke and the other finger from the desk is analyzed. (The details will be described with reference to FIGS. 16A and 16B).

The camera shoots by comparing the case where the fingers are held in the home position in a straight line on a normal hard keyboard as shown in FIG. 5A and the case where the fingers are held in the home position in an easy posture as shown in FIG. 6A. From the image, it can hardly be identified whether the position of the fingertip is in a straight line from side to side. By taking advantage of this fact, they are treated equally, and the relative size of the fingers is calculated from the change in size based on the size of each finger and nail image at the home position. By calculating, the position of the finger on the virtual keyboard can be detected. As a result, the personality of the user's fingers can be absorbed, and the function as an ergonomic keyboard can be realized in which fingers and arms are less tired.

Also, by widening the angle of view of the camera or using a plurality of cameras, the keyboard according to the present embodiment can be arranged with both hands wider than a general ergonomic keyboard. When a plurality of cameras are used, the front / rear position information of fingers as a 3D image can also be used.

FIG. 7 is a functional block diagram showing an example of the functional configuration of the present embodiment. The key input device 12 is configured as an external device or a built-in module of the mobile terminal 10. Hand images taken by the camera 20 placed close to the desk in the camera module 90 are preprocessed by a DSP (Digital Signal Processor) 29 in the camera module, and then the contour candidates of fingers and nails Such feature data is extracted by the feature extraction unit 70.

The key input processor 13 receives feature data such as a finger contour candidate from the DSP 29, and all functions from the finger position detecting unit 71 to the command determining unit 77 are executed by a software program. The finger position detection unit 71 determines each finger from the finger contour candidates, determines the position of the finger on the XZ plane, and calculates the size of the finger and nails. For example, when all fingers are substantially stationary on a desk for a predetermined time, the home position detection unit 72 stores the positions and sizes of the fingers at that time as home positions on the virtual keyboard. In the home position, when the virtual keyboard is a QWERTY keyboard, the keys "A", "S", "D", "F", "J", "K", "L" ”,“; ”(Semicolon). Since the thumb basically hits only the space key, the position on the virtual keyboard is not important. In addition, when changing the key assigned to each finger due to finger injury or the like, it is possible to cope with the change by pre-registering the change before using the key input device.

Based on the position on the XZ plane of each finger on the image at the home position and the size of the finger or nail as a reference, the finger position detector 71 Calculate the position of the finger on the virtual keyboard.

The keystroke detection unit 73 checks the height of the fingertip of each finger from the desk and determines whether or not there is a keystroke action. Further, even when another key is pressed with the right hand while the SHIFT key is pressed with the left hand, the key pressing operation detecting unit 73 can determine by calculating the key pressing position of both hands.

The key code generation unit 74 generates a key code corresponding to the position where the keystroke operation was performed, and sends the key code to the host processor 11 of the mobile terminal 10.

The detection results of the finger position detection unit 71 and the home position detection unit 72 and the key code generated by the key code generation unit 74 are a learning unit 75 based on an initial character string, a learning unit 76 during character input, and a command determination unit 77. And used as input information for learning and determination. The learning and determination results are fed back to the finger position detection unit 71, the home position detection unit 72, and the keystroke operation detection unit 73, and the detection parameters are adjusted. Thereby, detection accuracy improves.

The learning unit 75 based on the initial character string asks the user to input a predetermined character string or an arbitrary character string or perform a fingertip operation before using the key input device 12, and the keystroke position information In addition, the detection parameters of the finger position detection unit 71, the home position detection unit 72, and the keystroke operation detection unit 73 are adjusted based on the size information of fingers and nails. Even when the little finger is half hidden behind the ring finger in the front image of the finger, by letting the learning unit 75 by the initial character string learn by a method such as bringing the little finger into contact with the desk before and after placing the finger at the home position, The size of the little finger can be measured. The learning unit 76 during character input takes statistics of how much the coordinates of the finger with the keystroke operation and the coordinates of the center of the key top on the virtual keyboard calculated by the finger position detection unit 71 are deviated. Correct the coordinates of the keys in order. The command determination unit 77 will be described with reference to FIG.

The host processor 11 receives the output from the key code generation unit 74 at the input key reception unit 78. If the output is a valid character code, the host processor 11 converts the character corresponding to the character code into the input character string on the display 30. Is displayed on the display unit 31. The host processor 11 also receives the detection result of the finger position detection unit 71 on the desk, and displays the finger detection position display 62 superimposed on the keyboard display unit 32 on the display 30.

FIG. 8 is a functional block diagram showing another example of the functional configuration of the present embodiment. FIG. 8 illustrates a case where all processing is performed by the processor 14 of the mobile terminal 10. An image captured by the camera 20 that is mounted on the mobile terminal 10 and arranged close to the desk is captured by the processor 14 of the mobile terminal 10. The processor 14 executes all the processes from the feature extraction unit 70 to the command determination unit 77 (for example, extraction of feature data such as nail contour candidates, determination of the validity of the input command, etc.). The detection result is displayed on the display 30.

Compared with the functional configuration of FIG. 7, a mouse pointer control unit 79 and a mouse pointer display 34 are added to the functional configuration of FIG. 8. The command determination unit 77 is a command (hereinafter also referred to as “mouse command”) that moves the mouse pointer because the number of fingers of the left hand of the user placed in contact with the desk is less than the original number, and will be described in detail with reference to FIG. .) Is determined. Then, the command determination unit 77 feeds back to the hand position detection unit 71 and the home position detection unit 72 that a mouse command has been issued.

When the mouse command is issued, the finger position detection unit 71 and the home position detection unit 72 perform the following operations in addition to the operations described in the embodiment of FIG. The home position detection unit 72 stores the position and size of the right hand of the right hand as the current position (home position) of the mouse pointer when only one finger of the right hand touches the desk and stops. The finger position detection unit 71 calculates the moving direction and distance from the home position when the position or size of one finger resting on the desk on the right hand changes, and sends it to the mouse pointer control unit 79.

The mouse pointer control unit 79 holds the current position of the mouse pointer display 34 displayed superimposed on the input character string display unit 31. The mouse pointer control unit 79 moves the display position of the mouse pointer display 34 based on the movement information sent from the finger position detection unit 71.

If the above-described function for controlling the mouse pointer is used, for example, when an error is found in the input character string, the user issues a mouse command, moves the mouse pointer to an erroneous position, and clicks (keystrokes). By moving the cursor, it becomes possible to correct the error.

Also, if the function for controlling the mouse pointer is used, it can be used for, for example, inputting a button on the display screen or scrolling the screen even when no key is input.

The light illuminating unit 40 illuminates fingers on the desk, sharpens the contours of fingers and nails photographed by the camera 20, and enables key input even in a dark environment. The microphone 45 can be used, for example, when the user inputs information to the mobile terminal by voice that the finger is placed at the home position. The speaker 46 can notify the user that the mobile terminal 10 has accepted the key input by outputting a click sound of the key input.

FIG. 9 is an explanatory diagram showing an example of a method for inputting a command with a finger stationary at the home position. FIG. 5A shows a finger when the QWERTY keyboard is arranged for two-hand key input, whereas FIG. 9 shows an image of a finger arranged for key input by grasping the left hand 50D in this way. An image having a characteristic different from the above is given to the key input device to indicate that a command is being input. According to the operation shown in FIG. 9, for example, it is indicated to the key input device that the key to be input with the right hand 51C is capitalized from now on, or the virtual keyboard on which the right hand 51C is arranged is a numeric keyboard. It can be shown on a key input device.

FIG. 10 is an explanatory diagram of fingers placed at the home position on the virtual numeric keyboard. The fingers are ready to be placed on a normal hard keyboard. The right hand 51F is disposed at the home position on the virtual numeric keyboard 60D. The configuration of the virtual keyboard can be instantaneously switched from the virtual QWERTY keyboard as shown in FIG. 3A to such a virtual numeric keyboard by grasping the left hand as shown in FIG.

FIG. 11 is an explanatory diagram of a method for displaying the current detection position of the finger placed on the desk on the virtual keyboard displayed on the display. On the keyboard display 61 displayed on the keyboard display unit 32, a finger detection position display 62A and a finger center position display 63A indicating the center thereof are displayed in an overlapping manner. If a key is pressed in a state where the position of the left index finger extends over two or more key tops as in the finger detection position display 62B, an ordinary hard keyboard will be double-keyed. However, with the key input device according to the present embodiment, the F key on which the finger center position display 63B is placed is input to the key input device. As described above, the virtual keyboard has a feature that there is a margin with respect to the positional deviation of fingers compared to a normal hard keyboard, and even if there is an error in the finger detection position of the key input device, there is a feature that erroneous input is less likely to occur.

The range in which the camera 20 is shooting is indicated by shooting area displays 64A and 64B. The user can check whether the range used by the virtual keyboard is within the shooting area by looking at the shooting area displays 64A and 64B. As the user moves his / her finger away from the camera 20 and the distance between the fingers is reduced, the range used by the virtual keyboard is more easily accommodated within the horizontal angle of view of the camera 20, and the shootable range of the virtual keyboard is expanded.

FIG. 12 is an explanatory diagram of an example of a method for displaying a virtual keyboard, a finger image captured by the camera 20 and an input character string on the display 30. FIG. 12 shows an input character string display unit 31, a keyboard display unit 32, and a finger image display unit 33 that are displayed on the display 30. According to the present embodiment, since the left and right sides of the finger image are displayed in an inverted manner, the user can easily imagine the position of the finger on the virtual keyboard.

FIG. 13 is an explanatory diagram of an example of a method for measuring the position and size of a finger. FIG. 13 illustrates a portion in which only the middle finger of the right hand that is pressing the key in FIG. 6B is taken out. First, the finger position detection unit 71 extracts the contour 82 of the middle finger. For example, when the light irradiation unit 40 illuminates the user's fingers from the front, the image from the camera 20 is obtained as a dark image with the fingers bright and the others. Therefore, the finger position detection unit 71 can extract the contour of the finger by taking out the portion where the density value changes rapidly by differentiation or the like. The finger position detection unit 71 sets ZP, which is the Z coordinate value of the lowest point of the contour, as the Z coordinate value of the middle finger of the right hand (because of the keystroke, it is in contact with the desk).

The finger position detection unit 71 uses the coordinate value ZP as a starting point, obtains the horizontal width 83 of the left and right contours of the finger at n predetermined heights Z1, Z2, and Zn from the coordinate value ZP. 84 is determined. The finger position detection unit 71 obtains n middle points in this way, and obtains a regression line 85 from these n points by the least square method. The finger position detection unit 71 sets a point 86 where the straight line and the outline of the finger intersect as the X coordinate value XP of the middle finger on the image. The reason why the X coordinate value is not simply set as the lowest point of the contour line is that the fingertip with the keystroke is flat, and there may be many lowest points.

Furthermore, a straight line that passes through the n predetermined points on the Z axis and n points where the regression line 85 intersects and goes straight to the regression line 85 is drawn. A line segment 87 where this straight line is cut off by the contour 82 of the middle finger is the diameter of the middle finger at the height Zn. The finger position detection unit 71 obtains n diameters in this way, and sets the average value as the average diameter (number of pixels) of the middle finger.

With the above procedure, the finger position detection unit 71 determines the position and size of the middle finger (in this case, the average diameter). The finger position detection unit 71 can obtain other fingers in the same manner, but what is important is the coordinate value of the finger that has performed the keystroke operation. Therefore, for example, if the finger position detection unit 71 first obtains the lowest point of the contour of all fingers, and there is a finger that is determined to be performing a keystroke operation based on the positional relationship of all fingers on the Z axis. The finger position detection unit 71 can reduce the calculation time by a method in which only the finger is used to obtain the coordinates and size according to the above procedure.

The X coordinate value XP of the middle finger obtained here is a horizontal pixel number on the image. The average diameter of the finger is the number of pixels occupied by the finger on the image. Therefore, it is necessary to obtain the horizontal movement distance from the home position of the finger on the virtual keyboard based on the number of pixels.

FIG. 14 is an explanatory diagram showing an example of a method for calculating the horizontal movement distance from the home position of the finger on the virtual keyboard. FIG. 14 illustrates a state in which the left hand 52D at the home position moves on the virtual keyboard and approaches the camera diagonally to the left hand 52E (only the middle finger to be pressed is shown).

Suppose that the horizontal angle of view of the camera is the angle α and the number of pixels in the horizontal direction is the number N of pixels (not shown). The actual average diameter of the left middle finger is defined as an average diameter Dmm (not shown), and the average diameter (number of pixels) of the middle finger image is defined as an average diameter d (not shown). Let the average diameter (number of pixels) of the left-hand middle finger image at the home position P1 be the average diameter d1, and the angle from the left end of the horizontal field angle be the angle θ1. It is assumed that the middle finger of the left hand moves to the keying position P2 on the desk, the average diameter (number of pixels) at that time is the average diameter d2, and the angle from the left end of the horizontal field angle is the angle θ2.

The angle from the left end of the horizontal angle of view can correspond to the horizontal pixel number as shown in FIG. Therefore, if a conversion table between the angle and the horizontal pixel number is stored in advance in a memory (not shown) in the key input device, including correction of distortion due to aberration of the lens of the camera 20, etc., the angle from the pixel number is calculated. Is sought immediately. In this conversion table, the horizontal pixel numbers are integers corresponding to the number of pixels, and the angle corresponding to the horizontal pixel number includes a value less than the decimal point. Since the finger position is obtained by the regression line 85 as described in FIG. 13, the pixel number is obtained to the decimal point. Therefore, if the angle corresponding to the pixel number obtained by rounding down the decimal number of the pixel number and the angle corresponding to the pixel number obtained by rounding up the decimal point are obtained from the conversion table and proportionally calculated from the values after the decimal point of the pixel number, An approximation of the angle is obtained. In this way, an accurate angle can be obtained with a small conversion table for the number of pixels in the horizontal direction and with a short calculation time.

In this manner, the angles θ1 and θ2 are obtained from the horizontal pixel numbers (obtained in FIG. 13) of the fingers on the image. Therefore, if the distances L1 and L2 from the camera are obtained, the movement vector V from the home position of the middle finger of the left hand is obtained. The distances L1 and L2 can be approximately calculated from the distance L in the following expression (1) (details of the expression (1) will be described with reference to FIG. 15).

L = D / sin (α · d / N) (1)
Further, when the approximation of the trigonometric function is used, the distance L is obtained from the following equation (2).

L = D · N / (α · d) (2)
By substituting the average diameter (number of pixels) d1 and d2 of the middle finger image into this equation (2), distances L1 and L2 to the fingers are obtained, and as a result, the movement vector V is obtained. The forward / backward movement distance and the left / right movement distance of the finger from the home position on the virtual keyboard can be calculated from the movement vector V. The distance between the keys on the virtual keyboard is equal to the distance between fingers at the home position, and the keys are square (or parallel four deformations with the same height and bottom length), and there is no problem. From the above, it is possible to determine the key on the virtual keyboard at the position where the keystroke operation has occurred.

The problem here is that the average diameter of the actual fingers is expressed as average diameter D, but the value of this average diameter D cannot be measured by the camera. There are several means for determining the value of the average diameter D. One method is to have the key input device preliminarily set the key interval (mm) of the virtual keyboard assumed by the user. Since the average interval (number of pixels) of each finger at the home position corresponds to the key interval (mm), the actual finger diameter (mm) can be determined from the average diameter (number of pixels) of each finger on the image.

Next, let's calculate backward how much the average diameter (number of pixels) of the finger image changes when the finger moves on the virtual keyboard for one line using Equation (2). Assume that the actual average diameter D of the fingers is 15 mm, the horizontal field angle α of the camera is π / 2 (that is, 90 degrees), and the number N of pixels in the horizontal direction is 640. The distance L between the finger and the camera at the home position is 250 mm. By substituting these into equation (2), it can be seen that the average diameter (number of pixels) d of the finger image is about 24.5. It is assumed that the key pitch of the virtual keyboard is 19 mm, the finger approaches the camera 20 for one line from the home position, and the distance L becomes 231 mm. By substituting this into equation (2), the average diameter (number of pixels) d of the finger image is about 26.5. Therefore, moving the virtual keyboard by one line increases the average diameter (number of pixels) of the finger image by about two pixels.

Using the average value of the diameters at various heights in the Z-axis direction as the average diameter (number of pixels) of the finger image, the difference between the two pixels can be sufficiently detected even when the influence of noise is taken into consideration. Although only two images, the home position and the keystroke position, have been described, the frame frequency is increased, and information on the average diameter (number of pixels) of the finger image before and after the keystroke while the finger is moving is used for further accuracy. It can be improved.

FIG. 15 is an explanatory diagram of Expression (1). FIG. 15 shows an isosceles triangle having an actual average diameter D of fingers at positions separated by a distance L. FIG. The angle β can be approximately expressed by the following expression (3) using the horizontal view angle α, the number of pixels N in the horizontal direction, and the average diameter (number of pixels) d of the finger image.

β = α · d / N (3)
Such a representation is used because the ratio of the average diameter (number of pixels) d of the finger image in the number of pixels N in the horizontal direction approximates the ratio of β in the figure in the horizontal view angle α. Because it does. Further, the length of the line segment A perpendicular to the bottom of FIG. 15 is expressed by the following equation (4).

A = L · sin (α · d / N) (4)
If the actual average diameter D of the finger is replaced by the approximated segment A, Equation (1) is derived. In FIG. 15, the average diameter D of the fingers is greatly written. Actually, it is sufficiently smaller than the distance L, so that the error is small even if A is substituted.

As described above, the relative change of the position on the horizontal plane on the desk is calculated by the two parameters of the X coordinate position of the finger on the XZ plane obtained from the camera image and the size of the finger. Even if the mobile terminal 10 is arranged so that the camera 20 is close to the desk, the camera 20 is actually slightly away from the desk. Therefore, when the finger moves from the home position and approaches the camera 20, for example, the Z coordinate of the fingertip on the image is slightly lowered. Therefore, it is also possible to utilize the change in the Z coordinate for calculating the relative change in the position of the finger on the horizontal plane. However, in order to improve the detection accuracy of the keystroke operation described below, the mobile terminal 10 is arranged in a manner in which the camera 20 is sufficiently close to the desk so that the change of the Z coordinate on the image due to the horizontal movement of the finger is minimized. Is desirable.

FIG. 16A and FIG. 16B are explanatory diagrams of an example of a keystroke operation detection method. All the fingertips of the right hand 53C in the home position in FIG. 16A are in contact with the desk. The keystroke detection unit 73 stores the height of the fingertip of each finger at that time on the Z axis and uses it for the determination. The height of the middle finger at the home position is represented by a height ZH, and a threshold value (ZH + a) and a threshold value (ZH + b) are provided based on the height ZH. These threshold values are used when the middle finger strikes the virtual keyboard one line away from the camera. The condition that the keystroke detection unit 73 determines that the keystroke is keyed is that the lowest point of the middle finger is below the threshold value (ZH + a), and at the same time, fingers other than the middle finger are above the threshold value (ZH + b). . The right hand 53D of FIG. 16B that has moved from the home position and performed a keystroke operation satisfies this condition.

The key top of a hard keyboard has a sinking of several millimeters when a key is pressed, and users accustomed to this have a habit of lifting a finger other than the key that has been pressed. The keystroke operation detection unit 73 uses this to provide two thresholds to increase the detection accuracy of the keystroke operation. Two thresholds for detecting the keystroke are set for each finger and each line of the virtual keyboard.

Thus, in the case of the present embodiment, the portable terminal 10 is arranged so that the camera 20 is close to the desk. Therefore, even if the fingers move back and forth on the virtual keyboard, the change in the Z-axis on the image is very small, and it is easy to detect the keystroke operation based on the height of each finger on the Z-axis at the home position. is there.

FIG. 17 is an example of a hardware block diagram, and configures the functions of the functional block diagram of FIG. The functions from the feature extraction unit 70 to the mouse pointer control unit 79 in FIG. 8 are realized by a software program 18 written in a ROM (Read Only Memory) 17 of the processor 14, and are stored in the DSP 15 and the CPU (Central Processing Unit) 16. Read and execute. Information obtained during execution is stored in a RAM (Random Access Memory) 19. The ROM 17 can be externally attached.

FIG. 18 is a diagram illustrating an example of a method for detecting the desk edge 102 and a method for measuring the area of the finger. An image 100 is an image taken by the camera 20 arranged close to the desk. As in the state (A), after the user 101 activates the virtual keyboard, the user 101 once drops his hand from the desk. As a result, the desk edge 102 appears on the image without being blocked by the hand. For example, the desk edge 102 can be detected by differentiating the image in the Z-axis direction and extracting a line segment having a large absolute value of the differential value. Next, when the user 101 places the finger 103 on the desk as in the state (B), the desk edge 102 is partially invisible by the finger, and a portion that cannot be detected by the differential calculation occurs. In other words, the user's finger is placed at the pixel position where the desk edge 102 is no longer visible, and the finger position detector 71 can easily detect the finger.

State (C) is an enlarged image of the broken line rectangle in state (A) and represents the desk edge 102 and the desk top 104. The state (D) is an enlarged image of the broken line rectangle in the state (B), and is a diagram in which the user's fingers 106 are arranged. When the frame difference between the images of the state (A) and the state (B) is taken and only the area on the desk cut out by the desk edge 102 is extracted, the area 107 of the finger in the state (E) is obtained. Actually, since the shadow of the finger is reflected on the desk, the area 107 cannot be accurately obtained by simply performing a frame difference, but the area 107 is obtained from the contour of the finger and the desk edge 102. Although the method for measuring the diameter of the finger has been described with reference to FIG. 13, the area can also be measured as shown in FIG. The area can also be used to calculate the horizontal movement distance from the home position of the finger on the virtual keyboard described in FIG.

The image on the desk divided by the desk edge 102 has little change in shading. Compared with the image on the desk, the remaining image has a large variation in shading depending on the pattern of the human body and the arrangement of the background. Therefore, if the extraction threshold value for finger detection is determined for the entire image, the accuracy Does not improve. In one aspect, mobile terminal 10 according to the present embodiment detects desk edge 102. The portable terminal 10 calculates a parameter such as a finger clipping threshold only for a desk image as a partial area obtained by the detected desk edge 102. Thereby, detection accuracy can be improved. A finger image is also a relatively easy pattern recognition target by detecting only the tip of a finger arranged below the image of the desk edge 102 as in the state (E) of FIG. .

FIG. 19 shows an example of a method for obtaining the nail area. Mobile terminal 10 according to the present embodiment obtains maximum width 109 and maximum height 110 of nail 108 from the contour of the image of nail 108 of finger 106. Each of the maximum width 109 and the maximum height 110 can be used as one of the feature quantities that determine the size of the image of the finger or nail, but the product of the maximum width 109 and the maximum height 110 is used as a substitute for the nail area. Can also be used.

FIG. 20 exemplifies the procedure of a method for inputting a command with fingers. FIG. 9 illustrates an example in which a command is input by grasping a hand. FIG. 20 illustrates a method other than the method illustrated in FIG. 9, and only the front image of the finger 103 and the desk edge 102 are extracted. The state is illustrated. The desk edge 102 is shown as a straight line regardless of whether it is hidden by the finger 103. In the case of normal key input without a command, the right hand temporarily floats from the home position in the state (A) as in the state (B), and the index finger strikes the key as in the state (C). Meanwhile, the user may lift his left hand, but may remain stationary at the home position as in the state (C). In the state (D), the user moves the right hand with the thumb of the left hand floating, and presses the key as in the state (E), thereby inputting an image different from the state (C) to the key input device. These series of operations can be input to the command determination unit 77 as one command. When the user's left thumb is floating, the mobile terminal 10 can realize the same function as pressing a shift key, for example. As a result, it is possible to realize key input with high speed and few errors with less movement of the left hand as compared with normal hard keyboard operations. Note that it is also possible to input commands for modifier keys (Ctrl, Alt, etc.) other than the shift key, depending on the combination of fingers to be floated.

In state (F), the user of the mobile terminal 10 gives a command with the left thumb and index finger lifted. This example shows a command to move the mouse pointer (hereinafter also referred to as “mouse command”). Only the index finger is in contact with the desk on the right hand, and that position is the position of the mouse pointer currently displayed on the display screen. Is associated with. When the right hand is moved as in the state (G), the moving distance and direction of the index finger are calculated by the method described with reference to FIG. 13, and the position of the mouse pointer on the display screen is moved accordingly. Although not shown in the figure, a function equivalent to mouse click input can be realized by lifting the right index finger from the desk in the state (G) and returning it to the desk again. By using this function, even when key input is not performed, for example, input of buttons on the display screen and scrolling of the screen can be realized.

FIG. 21A and FIG. 21B are examples of a flowchart of the present embodiment. Referring to FIG. 21A, when the user activates the virtual keyboard mode by pressing the start button of the virtual keyboard displayed as a software key on the screen of the mobile terminal 10, the camera 20 performs continuous shooting on the desk. Start (step S0). In accordance with a predetermined usage method, the user once removes his / her hand from the desk, and the portable terminal 10 detects the desk edge 102 (step S1). When the detection is completed, the mobile terminal 10 informs the user of the fact and starts detecting the position of the finger on the desk (step S2). The position detection includes not only the position of the finger on the image but also the size of the finger, and is continuously performed until the end of the virtual keyboard program. Next, the mobile terminal 10 waits for the user's fingers to rest (step S3). When the portable terminal 10 detects that the finger is stationary (step S3: YES), it measures the position and size of each finger at the home position, and stores the position and size in the RAM 19 (step S4). ). Once the home position is stored in step 4, the mobile terminal 10 starts detecting various operations of the user. First, it is confirmed again whether or not all fingers are placed on the desk and stopped (step S5). If so (step S5: YES), the mobile terminal 10 performs re-detection of the home position (step S6). If it is not stationary (step S5: NO), the portable terminal 10 checks whether or not an end command is indicated (step S7). For example, when it is detected that the user holds both hands together, the mobile terminal 10 determines that an end command has been indicated. If the end command is indicated (step S7: YES), the mobile terminal 10 ends the position detection of the fingers on the desk, stops continuous shooting on the desk by the camera 20 (step S8), and ends the virtual keyboard. (Step 9).

Referring to FIG. 21B, next, the mobile terminal 10 checks whether the mouse command described in FIG. 20 is indicated (step S10). If the mouse command is indicated (step S10: YES), the mobile terminal 10 confirms whether one finger corresponding to the mouse pointer is stationary on the desk (step S18). If not stationary (step S18: NO), the mobile terminal 10 returns the control to step S5. If the mobile terminal 10 is stationary (step S18: YES), the mobile terminal 10 uses the position of the finger as the current position (home position) of the mouse displayed on the display screen of the display 30, and the position of the finger on the image. The size is stored in the RAM 19 (step S19). If the mouse command is released (step S20: NO), the mobile terminal 10 returns the control to step S5. If the mouse command is held (step S20: YES), the mobile terminal 10 checks whether or not the user's finger has moved (step S21). If the finger has moved (step S21: YES), the portable terminal 10 compares the position and size of the finger after movement with the position and size of the finger at the home position stored in the RAM 19, and The moving direction and the moving distance of are calculated. The mobile terminal 10 moves the mouse pointer displayed on the display screen of the display 30 in accordance with the calculated moving direction and moving distance (step S22).

If the finger operation detected by the mobile terminal 10 is not a mouse command (step S10: NO), the mobile terminal 10 confirms whether or not a keystroke operation has been performed (step S11). If there is no keystroke operation (step S11: NO), the portable terminal 10 returns control to step S5. If there is a keystroke operation (step S11: YES), the mobile terminal 10 determines the position of the finger on the image and the size of the finger and the position of the finger on the image at the home position stored in step S4 or step S6. The movement direction and distance of the finger on the desk are calculated by comparing the size of the finger and the finger, and the position of the key on the virtual keyboard where the key is hit is obtained (step S12). Next, the mobile terminal 10 confirms whether a command is indicated (step S13). If no command is indicated (step S13: NO), the mobile terminal 10 generates a key code of the virtual QWERTY keyboard corresponding to the position (step S14). If the command is indicated with the hand opposite to the key that is pressed (step S13: YES), if the indicated command is a shift command, the portable terminal 10 has the shift key pressed at the same time as the key is pressed. And a key code corresponding to the determination is generated (step S15). If the command is a numeric keypad command, the mobile terminal 10 determines that the virtual keyboard has been switched to the numeric keyboard, and generates a key code for the numeric keypad corresponding to the position where the key was pressed (step S16). After the key input, the mobile terminal 10 waits for the finger that has pressed the key to leave the desk (step S17). If the finger is released (step S17: YES), the mobile terminal 10 returns the control to step 5.

FIG. 21 does not describe error handling when a keyless position is pressed. In FIG. 21, the shift command and the numeric key command are described as examples of the key input command. However, the key input command is not limited to the shift command and the numeric key command. In addition to these commands, a control command indicating that a character key or numeric key and a control key are simultaneously pressed, or a function key command that can be operated with one hand can be added. The function key command can be configured as, for example, each key of a keyboard having the same shape as a numeric keyboard, with the F1 key to F12 key, Back Space key, Delete key, etc. of the QWERTY keyboard arranged from the home position. Distant keys can be input instantly.

FIG. 22A and FIG. 22B are examples of a virtual keyboard in which the positions of keys assigned to each finger overlap each other. The keyboard 112 in FIG. 22A is a virtual keyboard that is handled by the index finger of the right hand 111. The index finger is arranged on the key “C” which is the home position. The user rotates his / her hand with the wrist as a fulcrum, and the index finger strikes the key “E” from the key “A”. FIG. 22B is a virtual keyboard handled by the middle finger of the same right hand 111. The index finger is arranged on the key “H” which is the home position. Similarly, when the user rotates his hand, the index finger can press the J key from the F key. When the right hand 111 of FIG. 22A and the right hand 111 of FIG. 22B are overlapped, the two virtual keyboards partially overlap each other, and a plurality of keys exist at the same position on the desk.

22A and 22B, a virtual keyboard having 40 or more types of keys can be realized by having eight fingers excluding the thumb in charge of five keys each. In this case, the finger moves only to the left and right, and does not need to move back and forth. Therefore, in the present embodiment, the finger position detection unit 71 only needs to detect a change in the position of the finger in the camera image, and does not need to detect a change in the size of the finger or nail. Each finger is responsible for only one key in each column, but the length of each finger is different, so in this case too, the distance from the camera of the key responsible for each finger is different and is specific to the user This is a virtual keyboard having the following shape.

The embodiment shown in FIGS. 22A and 22B can be expanded so that the virtual keyboard assigned to each finger has a plurality of lines. In the case of a plurality of lines, it is necessary to move each finger forward and backward, and the finger position detection unit 71 needs to detect changes in the size of fingers and nails. The functional configuration of FIGS. 7 and 8 and the flowchart of FIG. 21 are also explanatory diagrams of a virtual keyboard configured by overlapping the positions of keys assigned to each finger.

As described above, the virtual keyboard according to the present embodiment is configured as a virtual keyboard configured such that the positions of the keys assigned to each finger are overlapped with each other, thereby minimizing finger movement required for key operation. it can. By doing so, it is possible to realize a virtual keyboard in which hands and arms are less likely to get tired and the required area on the desk required for operation is minimized. After the user has mastered the familiar layout of the virtual keyboard such as the QWERTY keyboard, the next step is for the user to create a virtual keyboard composed of the keys arranged by each finger overlapping each other. It is desirable to use FIG. 23A and FIG. 23B are other examples of a system perspective view of the present embodiment. Although the description has been made centering on the mobile terminal 10 so far, the virtual keyboard can be realized in other modes. The virtual keyboard according to the present embodiment has a feature that the fingers can be held in a comfortable posture, and can improve tendonitis and stiff shoulders caused by long-time key operations. Therefore, the virtual keyboard according to this embodiment is also useful for use in information equipment such as a personal computer. FIG. 23A shows an example in which the key input device 118 is configured as an external device of a personal computer. The key input device 118 is disposed on the desk, and the camera 116 captures a user's finger image from a position close to the desk. The inputted key information is sent to the personal computer 114 through the cable 117 and displayed on the monitor 115. FIG. 23B shows an example in which the function as a key input device is realized by a program of the personal computer 114. Specifically, the camera 116 is attached to the monitor 115, takes an image on the desk, and sends the image to the personal computer 114 through the cable 117. A CPU (not shown) of the personal computer 114 executes each process (FIG. 21) performed on the mobile terminal 10 and functions as a virtual keyboard.

So far, the description has been made on the assumption that nothing is attached to the fingers operating the virtual keyboard, but the embodiment is not limited to this. For example, in order to make it easier to detect which finger is in the home position or the finger that has been pressed, the user uses a device such as a finger sack with different characteristics that can be identified by the camera. You may attach to. Alternatively, the user attaches the instrument to the fingertip, and the mobile terminal 10 and other key input devices are configured to detect the position and size of the instrument instead of detecting the position and size of the finger itself. May be.

In yet another aspect, a user who is unfamiliar with the key input device can also operate while checking the position of the key by placing a sheet on which a keyboard suitable for the length of his / her finger is printed on the desk.

The mobile terminal 10 according to the present embodiment is realized in one aspect by a processor executing a computer program stored in a data recording medium. The computer program can also be executed by a computer system having a known configuration. Therefore, it can be said that the essential part of the present invention is software stored in a RAM, hard disk, CD-ROM or other data recording medium, or software downloadable via a network. Since the operation of each hardware of the computer system is well known, detailed description will not be repeated.

Data recording media are not limited to CD-ROM, FD (Flexible Disk), and hard disk, but are magnetic tape, cassette tape, optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)). ), IC (Integrated Circuit) card (including memory card), optical card, mask ROM, EPROM (Electronically Programmable Read-Only Memory), EEPROM (Electronically Erasable Programmable Read-Only Memory), semiconductor memory such as flash ROM, etc. It may be a medium that carries the program in a fixed manner.

Here, the program may include not only a program that can be directly executed by the CPU, but also a program in a source program format, a compressed program, an encrypted program, and the like.

In one aspect, according to the key input device 12 according to the present embodiment, the user can reduce fatigue of fingers and arms without sacrificing the wearability of the mobile terminal, and the virtual keyboard having the optimal position and shape for himself / herself Using, enables fast key input using both hands. A normal hard keyboard has keys in each row aligned in a horizontal direction, so it is necessary to set each finger of different length so that the fingertip is almost perpendicular to the keyboard. For example, it is possible to hold all fingers straight. In this case, the keys in each row of the virtual keyboard are not aligned in the horizontal direction, and many users have the longest middle finger. Therefore, the key corresponding to the home position of the middle finger (for example, the D key for a QWERTY keyboard) And K key) is a virtual keyboard having a shape most suitable for the user's fingers such that the keys in the row are closer to the camera than the keys in the other rows. And since the finger is stretched easily, the key is pressed with the belly of the finger (the part with the fingerprint). If it is a normal hard keyboard, it is easy to be double-keyed when the key of the finger is pressed at the same time, but since it is a virtual keyboard, it can be processed as if only the key at the fingertip position was input. The impact on the finger joint and the like is greatly reduced by hitting the key with the belly of the finger. This characteristic is effective when applied to information devices such as desktop computers that require key input other than portable terminals.

In another aspect, the key input device 12 can introduce the information about the size of the finger image in addition to the two-dimensional position information of the finger on the two-dimensional image obtained by the camera 20, and makes the input information three-dimensional. be able to. The input device can calculate the relative movement distance from the change in the size of the finger based on the size of the finger image when the finger is placed at the home position.

In one aspect, since the camera 20 is arranged close to the desk and is configured to capture a front image of the fingers on the desk, the key input device 12 has a height distance at which the fingers on the desk are separated from the desk. And the relative change distance of the two-dimensional position of the finger on the desk can be easily calculated. In addition, when the key input device 12 determines the keystroke operation, the difference in height between the finger performing the keystroke operation and the other fingers from the desk can be easily determined from the analysis of the image.

Further, according to the key input device 12 according to the present embodiment, the user can confirm the key arrangement, and can confirm the current position of the finger detected by the key input device 12. When the user slowly inputs a key, if the user presses the key after confirming that the position of the finger detected by the key input device 12 is displayed on the key to be input next, , Key input without error. In addition, the user can check whether the camera 20 can capture the entire virtual keyboard that the user is trying to input. When a part of the virtual keyboard is outside the imaging range of the camera, the user can correct the imaging range by adjusting the interval between fingers or the distance between the fingers and the camera.

According to the key input device 12 according to another embodiment, the user does not need to perform any operation on the key input device 12 and can instantly switch the keyboard type. For example, if the user holds the left hand in the shape of a goo, the key input device 12 can easily detect that the left hand is not effective.

Since the key input device 12 according to another embodiment can detect the contours of fingers and nails photographed by the camera 20 more clearly, it is possible to improve the accuracy of measuring the size of the fingers. Further, even in a dark environment, the user can use the key input device 12.

Although the invention has been described and shown in detail, it is clearly understood that this is by way of example only and should not be taken as a limitation, the scope of the invention being construed by the appended claims Will.

10 mobile terminals, 11 host processors, 12 key input devices, 13 key input processors, 14 processors, 15 DSP, 16 CPU, 17 ROM, 18 programs, 19 RAM, 20 cameras placed close to the desk, 21, 22 Range of horizontal angle of view of camera, 23, 24, 25, 26, 27, 28 VGA image horizontal pixel number specified position, 29 DSP, 30 display, 31 Displayed character string display, 32 Keyboard display 33, 34 Mouse pointer display, finger image display unit, 40 light irradiation unit, 41, 42 light horizontal irradiation range, 45 microphones, 46 speakers, 50A, 50B, 50C arranged on a virtual keyboard aligned in a straight line Left-hand image, 50D left-hand image, 5 A, 51B, 51C, 51D, 51E, 51F Right hand image arranged on a virtual keyboard aligned in a straight line, 52A, 52B, 52C, 52D, 52E Left hand arranged on the virtual keyboard in an easy posture, 53A, 53B, 53C, 53D Right hand, 60A, 60B, 60C virtual QWERTY keyboard, 60D virtual numeric keyboard, 61 keyboard display, 62, 62A, 62B finger detection position display, 63A, 63B finger center position display arranged on the virtual keyboard in an easy posture 64A, 64B shooting area display, 70 finger and nail contour feature extraction unit, 71 finger position detection unit, 72 home position detection unit, 73 keystroke detection unit, 74 key code generation unit, 75 learning unit with initial character string, 76 Learning part during character input, 77 One hand or both hand Part, 78 input key accepting part, 79 mouse pointer control part, 80A, 80B, four finger outlines, 81A, 81B, 81C, 81D, 81E, 81F finger image width, 82 middle finger outline, 83 outlines Horizontal width, 84 Midpoint of horizontal width of contour, 85 n middle points regression line, 86 Middle finger position on image, 87 Middle finger diameter, 90 camera module, 100 Camera image, 101 user, 102 desk edge, 103 user's finger placed on the desk, 104 desk, 106 user's finger, 107 finger area separated by desk edge, 108 nail image, 109 nail horizontal diameter, 110 nail vertical Diameter, 111 Right hand placed on the virtual keyboard, 112 Virtual keyboard for the right hand index finger, 113 Virtual key for the right hand middle finger From board, 114 personal computer, 115 monitor, 116 camera, 117 cable, 118 key input device, D middle finger actual diameter (mm), d1, d2 middle finger image average diameter (number of pixels), L1, L2 from middle finger camera , P1, P2 Middle finger position on virtual keyboard, V Middle finger movement vector, XP, ZP, ZH Middle finger coordinates, α Camera horizontal angle of view, θ, θ1, θ2 Horizontal finger angle from left edge of horizontal field of view The angle to the position where there is.

Claims (16)

1. An input device,
   A camera configured to continuously capture front images of the fingers of the user of the input device when the input device is placed close to the desk;
   A finger position detector configured to detect the position of the user's finger from the photographed front image;
   When the user's finger touches the desk and stops, the position of the user's finger at that time is different in the distance from the camera of the key assigned to each finger, and the shape unique to the user A home position detector configured to detect as a home position on a virtual keyboard having
   A keystroke operation detector configured to detect that the user's finger has pressed a key on the virtual keyboard;
   An input device, comprising: a key code generation unit configured to generate a code of a key on the virtual keyboard corresponding to a position where a keystroke operation has been performed.
2. The finger position detector is
   By comparing the position on the image of the finger at the home position and the size of the finger or nail with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, The input device according to claim 1, wherein the input device is configured to calculate a position of a finger on the virtual keyboard.
3. When the number of fingertips that touch and rest on the desk is less than a predetermined number for only one hand, the virtual keyboard has a function of a modifier key according to the combination of the fingertips touching the desk on one hand The input device according to claim 1, configured as described above.
4. When the number of fingertips that are in contact with the desk and are stationary is less than a predetermined number for only one hand, the input device can be operated with the other hand by a combination of fingertips that are in contact with the desk on the other hand. The input device according to claim 1, wherein the virtual keyboard is configured to automatically switch between an alphanumeric keyboard and a keyboard adapted to accept an operation with one hand.
5. An input device,
   A camera configured to continuously capture front images of the fingers of the user of the input device when the input device is placed close to the desk;
   A finger position detector configured to detect the position of the user's finger from the photographed front image;
   A monitor configured to display at least a mouse pointer in the display area;
   A home position detector configured to associate the position of the user's finger at that time with the current position of the mouse pointer on the display area when the user's finger touches the desk and is stationary;
   A mouse pointer control unit configured to move the mouse pointer on the display area according to the moving direction and distance when the user's finger moves on the desk;
   The finger position detector is
   By comparing the position of the finger on the image at the home position and the size of the finger or nail with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, An input device configured to calculate a moving direction and a distance.
6. The input device according to any one of claims 1 to 5, wherein the finger position detection unit is configured to use information on a desk edge in order to measure a position and a size of a finger.
7. An information device comprising the input device according to any one of claims 1 to 6.
8. A program for causing a computer to function as an input device, the program being
   Continuously capturing front images of the fingers of the user of the input device with a camera arranged close to the desk; and
   Detecting a position of a user's finger from the photographed front image;
   When the user's finger touches the desk and stops, the position of the user's finger at that time is different in the distance from the camera of the key assigned to each finger, and the shape unique to the user Detecting as a home position on a virtual keyboard having:
   Detecting that the user's finger has pressed a key on the virtual keyboard;
   Generating a code of a key on the virtual keyboard corresponding to a position where a keystroke operation has been performed.
9. Detecting the position of the finger;
   By comparing the position on the image of the finger at the home position and the size of the finger or nail with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, The program according to claim 8, comprising calculating a position of a finger on the virtual keyboard.
10. A program for causing a computer to function as an input device, wherein the program causes the computer to
    Continuously capturing front images of the fingers of the user of the input device with a camera placed close to the desk;
    Detecting a position of a user's finger from the photographed front image;
    Displaying a mouse pointer in the display area of the monitor;
    Associating the position of the user's finger at that time with the home position, which is the current position of the mouse pointer on the display area, when the user's finger contacts and rests on the desk;
    Moving the mouse pointer on the display screen according to the moving direction and distance when the user's finger moves on the desk;
    Detecting the position of the finger;
    By comparing the position on the image of the finger and the size of the finger or nail at the home position with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, the finger on the desk A program including a step of calculating a moving direction and a distance.
11. The program is stored in the computer.
    As a step of detecting the position of the finger,
    The program according to any one of claims 8 to 10, wherein the program uses a step of using desk edge information to measure the position and size of a finger.
12. A non-volatile computer-readable data recording medium storing the program according to any one of claims 8 to 11.
13. A method for inputting characters using an input device,
    Continuously capturing front images of the fingers of the user of the input device with a camera arranged close to the desk; and
    Detecting a position of a user's finger from the photographed front image;
    When the user's finger touches the desk and stops, the position of the user's finger at that time is different in the distance from the camera of the key assigned to each finger, and the shape unique to the user Detecting as a home position on a virtual keyboard having:
    Detecting that the user's finger has pressed a key on the virtual keyboard;
    Generating a code of a key on the virtual keyboard corresponding to a position where a keystroke operation has occurred.
14. Detecting the position of the finger;
    By comparing the position on the image of the finger at the home position and the size of the finger or nail with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, The method of claim 13, comprising calculating a position of a finger on the virtual keyboard.
15. A method for inputting characters using an input device,
    Continuously capturing front images of the fingers of the user of the input device with a camera placed close to the desk;
    Detecting a position of a user's finger from the photographed front image;
    Displaying a mouse pointer in the display area;
    Associating the position of the user's finger at that time with the home position, which is the current position of the mouse pointer on the display area, when the user's finger contacts and rests on the desk;
    Moving the mouse pointer on the display screen according to the moving direction and distance when the user's finger moves on the desk,
    Detecting the position of the finger;
    By comparing the position on the image of the finger and the size of the finger or nail at the home position with the position on the image of the finger after the finger has moved from the home position and the size of the finger or nail, the finger on the desk Calculating a moving direction and a distance of.
16. Detecting the position of the finger;
    16. A method according to any of claims 13 to 15, comprising the step of using desk edge information to measure finger position and size.

Images