JP2012089083A - Information input device and method for electronic apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To input information equal to characters, equal to a mouse, equal to a numeric key or equal to a shortcut key, without affecting another input screen with respect to a selected input screen, by selecting and switching the input screen in accordance with an arbitrarily settable finger posture or motion.SOLUTION: An information input device includes imaging means S020, extraction means S030 which extracts finger information from a video image of the imaging means, instruction finger determination means S040 which calculates two-dimensional coordinates of a finger on an input screen on the basis of the finger information, and determines a current instruction finger state from a predetermined instruction finger state table so as to facilitate operation by an operator, operation state determination means S050 which determines an operation state corresponding to the instruction finger state from an operation state determination table in which a predetermined operation procedure comprising a combination of the instruction finger state, the duration of the instruction finger state and an operation instruction is associated with the input screen, and operation procedure implementation means S060 which implements the operation instruction to select an input screen or to produce input information in accordance with the determined operation state.

Description

  The present invention relates to an information input apparatus and method for an electronic device. More specifically, the present invention relates to an information input device and method for an electronic device that can input not only mouse information but also character information, numeric keypad information, and the like in accordance with a preset operator's finger appearance and movement. .

  As an input device for electronic equipment for the purpose of improving the efficiency of input work, reducing fatigue, and saving space, in addition to the conventional mechanical input operation, a finger that further operates the input surface for one input surface There has been proposed an input device that uses, as input information, an image obtained by capturing the movement of the camera with a camera.

  One of them is applied to a notebook personal computer. In addition to the usual keyboard operation input, the camera also captures finger movements on the keyboard and substitutes the mouse function. In the normal operation that operates with the home position as the base point, it is impossible to move the index finger and the middle finger together, and video input is performed when the index finger and the middle finger move together.

  That is, when it is confirmed in the video of the finger on the keyboard that the index finger and middle finger are moving together, it is determined that the operation is to move the mouse pointer. When one of the fingers is moved up or down, it is determined that the user is performing a click operation. Using the finger image on the keyboard as input information, a mouse pointer movement operation and a click operation are proposed. (For example, refer to Patent Document 1).

  However, this method can only switch between two types of input surfaces (normal keyboard input and pointing input based on video information), and can only perform pointing operations with the index and middle fingers aligned. There was a fault that I couldn't say.

  The other is for the purpose of providing an input device that can be stored in a small package such as an electronic device, so that one input surface can be used as a plurality of input surfaces. Create a panel so that one input surface can be selected and called, and the selected input panel is called on the input surface captured by two cameras, and the input on the called input panel is input. There has been proposed a technique for generating an input signal of a type that is detected from video information photographed by a camera and created in today's mechanical input device (for example, see Patent Document 2).

  However, although a plurality of input surfaces can be selected, a selection button for selecting a selectable input surface is provided on each input surface. There were drawbacks in terms of operability and miniaturization. In addition, although input determination depends on only the distance between the user's finger and the input surface and is simple, there is a disadvantage that user-friendly input operation cannot be performed.

JP 2008-242881 A JP-T-2006-509269

The present invention solves the above-mentioned problems, improves the efficiency of input work in manual input work to electronic equipment, reduces fatigue (labor saving), saves space on the input device, and reduces user input. An object of the present invention is to provide an input device using image recognition as a machine interface.
In other words, the input surface of the input device of the present invention provided on the input operation surface of the electronic device does not affect other input surfaces, and the operator can add or change the input surface.
The shape of the finger for operating the input device of the present invention is arbitrary in consideration of the operator's request, such as a finger shape that is easy for the operator to operate and a finger shape that is meaningful to the operator. Can be changed or set.
The operation procedure of the input device according to the present invention is constructed in consideration of the operability and certainty of the input operation according to the finger shape determined by the operator and the speed of change of the finger shape for each input surface determined by the operator. Can do.

An information input device for an electronic device according to claim 1,
Imaging means for sequentially capturing images of a predetermined area of the input operation surface;
Finger information extracting means for extracting finger information from the image of the imaging means;
Based on the finger information from the finger information extraction means, the current finger position and finger position coordinates on the video are determined, and the finger position of the previous finger extraction information stored in the current finger position coordinates Determine the difference status from the coordinates, and determine the current pointing finger status from the pointing finger status table that associates the pointing finger status with the difference status between the figure of the finger and the position coordinates, which can be determined and changed in advance. And pointing finger determination means for calculating the two-dimensional coordinates of the current finger on the input surface formed at a predetermined position on the input operation surface from the current finger position coordinates;
At least one consisting of a combination of the pointing finger state, a time during which the pointing finger state continues, and an operation instruction with respect to the input surface formed at a predetermined position of the input operation surface, which can be determined and changed in advance. From an operation state determination table that associates the operation procedures of a pair or more, an operation state determination unit that determines an operation state according to the pointing finger state determined by the pointing finger determination unit,
And an operation procedure execution means for executing an operation instruction for selecting an input surface or generating input information in accordance with the operation state determined by the operation state determination means.

  According to a second aspect of the present invention, there is provided the input device according to the first aspect, wherein the imaging means is constituted by a two-dimensional image sensor.

  An input device according to a third aspect is the input device according to the first aspect, wherein the imaging means is composed of at least two one-dimensional image sensors provided laterally with respect to the input surface. These two-dimensional coordinates are calculated by a triangulation method.

  According to a fourth aspect of the present invention, there is provided the input device according to the first aspect, wherein the imaging means is composed of a combination of a two-dimensional image sensor and a one-dimensional image sensor.

  The input device according to a fifth aspect is the input device according to the first aspect, wherein the imaging means includes an irradiating means for irradiating a predetermined area of the input operation surface in the vicinity thereof.

  The input device according to claim 6 is the input device according to claim 1, wherein the pointing finger determination unit changes the appearance of the operator's finger, whether there is a gap between fingers, the number of fingers, or the number of fingers. And the width of the finger.

The input device according to claim 7 is the input device according to claim 1, wherein the operation procedure execution means is predetermined and can be changed with respect to the input surface, the input information generation instruction, and the two-dimensional coordinates. An input information table that associates the input information of
If the content of the operation instruction from the operation state determination means is an input surface selection instruction, the currently selected input surface is changed according to the input surface selection instruction,
If the content of the operation instruction from the operation state determination means is an instruction to generate input information, on the currently selected input surface,
Generate input information defined by the operator in response to the input surface and input information generation instruction, or
The input information is generated from the input information table from the input surface, the input information generation instruction, and the two-dimensional coordinates of the finger.

  The input device according to claim 8 is the input device according to claim 1, wherein the input surface formed at a predetermined position of the input operation surface includes at least an input surface of the keyboard, an input surface of numeric keypad information, and mouse information. Each of the input surfaces is switched to each other by the operating procedure execution means.

  The input device according to claim 9 is the input device according to claim 8, wherein an operation procedure is further performed between the input surface formed at a predetermined position of the input operation surface and the input surface of mouse information. An input surface for continuing to press the left click and an input surface waiting for scrolling, which are switched by means, are included.

A method for inputting information to an electronic device according to claim 10 is provided.
An imaging process for sequentially capturing images of a predetermined area of the input operation surface;
Finger information extraction process for extracting finger information from the video obtained in the imaging process;
Based on the finger information obtained in the finger information extraction process, the current finger shape and the finger position coordinates on the video are determined, and the finger information of the previous finger extraction information stored in the current finger position coordinates is stored. The current pointing finger state from the pointing finger state table in which the pointing finger state is associated with the difference between the figure of the finger and the position coordinate, which can be determined, changed and determined in advance, And a pointing finger determination process for calculating the two-dimensional coordinates of the current finger on the input surface formed at a predetermined position on the input operation surface from the current finger position coordinates;
At least one consisting of a combination of the pointing finger state, a time during which the pointing finger state continues, and an operation instruction with respect to the input surface formed at a predetermined position of the input operation surface, which can be determined and changed in advance. From an operation state determination table that associates a set of operation procedures or more, an operation state determination process for determining an operation state according to the pointing finger state determined by the pointing finger determination process,
And an operation procedure execution process for executing an operation instruction for selecting an input surface or generating input information in accordance with the operation state determined by the operation state determination process.

  An information input method according to an eleventh aspect is the input method according to the tenth aspect, wherein the imaging process is performed using a two-dimensional image sensor.

  An information input method according to a twelfth aspect is the input method according to the tenth aspect, wherein the imaging process is performed using at least two one-dimensional image sensors provided laterally with respect to the input surface. Two-dimensional coordinates on the surface are calculated by a triangulation method.

  An information input method according to a thirteenth aspect is the input method according to the tenth aspect, wherein the imaging process is performed by a combination of a two-dimensional image sensor and a one-dimensional image sensor.

  The information input method according to claim 14 is the input method according to claim 10, wherein the pointing finger determination process is performed to determine the appearance of the operator's finger, whether there is a gap between fingers, the number of fingers, It is characterized by recognizing the number and the width of the finger.

  In the input device according to the first aspect of the present invention, the finger information extracted from the video imaged by the imaging means can be expressed as a pointing finger state that is predetermined and can be changed by the operator. The operation state determination means operates when the pointing finger state matches the pointing finger state registered in the first set of input surface data currently selected in the operation state determination table determined in advance by the operator. As the start trigger of the procedure, the subsequent pointing finger state is received as the pointing finger state of the selection determination intention, and when the pointing finger state matches the pointing finger state registered in the data set as the start trigger, the operation instruction of the operation procedure is confirmed. Therefore, the operation procedure execution means can select an input surface and generate input information of the selected input surface according to the operation instruction.

  In the input device according to the second aspect, since the imaging means is composed of a two-dimensional image sensor, it is possible to accurately detect finger information.

  In the input device according to the third aspect, since the imaging means is composed of at least two one-dimensional image sensors provided in a direction transverse to the input surface, the device can be configured at low cost.

  In the input device according to the fourth aspect, since the imaging means is composed of a combination of a two-dimensional sensor and a one-dimensional sensor, it is possible to accurately detect finger information and to configure the entire device at low cost.

  In the input device according to the fifth aspect, since the image pickup means includes the irradiation means, it is possible to take an image while clearly distinguishing the finger and the background.

  In the input device according to claim 6, depending on the type of the imaging unit used, the pointing finger determination unit determines whether or not the operator's finger has a gap between fingers, the number of fingers, or the number of fingers and the width of the fingers. It can be easily recognized by a free combination.

  In the input device according to the seventh aspect, since the operation procedure execution means has an input information table in which a plurality of input information that can be arbitrarily set / changed by the operator is associated, a finger on the selected input surface Based on the two-dimensional coordinates, input information determined in the input information table can be input immediately.

  9. The input device according to claim 8, wherein the input surface formed at a predetermined position of the input operation surface includes at least an input surface of a keyboard, an input surface of numeric keypad information, and an input surface of mouse information. Can be switched to each other by the operating procedure execution means, so that all information necessary for normal computer input operations can be efficiently input while switching the input surface.

  Since the input device according to claim 9 further includes an input surface that can be switched by the operating procedure executing means and an input surface that is left-clicked continuously, and an input surface that is waiting for scrolling. All the information of the mouse can be input intensively and efficiently.

  In the input method according to the tenth aspect of the present invention, the finger information extracted from the video imaged in the imaging process can be expressed as a pointing finger state that is predetermined and can be changed by the operator. The operation state determination process is performed when the pointing finger state matches the pointing finger state registered in the first set of input surface data currently selected in the operation state determination table determined in advance by the operator. As the start trigger of the procedure, the subsequent pointing finger state is received as the pointing finger state of the selection determination intention, and when the pointing finger state matches the pointing finger state registered in the data set as the start trigger, the operation instruction of the operation procedure is confirmed. Therefore, according to the operation procedure execution process, it is possible to select the input surface and generate input information of the selected input surface according to the operation instruction.

  In the input method according to the eleventh aspect, since the imaging process is performed using a two-dimensional image sensor, finger information can be accurately detected.

  In the input method according to claim 12, the imaging process is performed by using at least two one-dimensional image sensors provided in a direction transverse to the input surface, and the two-dimensional coordinates on the input surface are calculated by the triangulation method. Therefore, finger information can be obtained at low cost.

  In the input method according to the thirteenth aspect, since the imaging process is performed by a combination of a two-dimensional image sensor and a one-dimensional image sensor, finger information can be obtained accurately and inexpensively.

  In the input method according to claim 14, in the pointing finger determination process, depending on the imaging process, the operator's finger shape can be freely determined by the presence or absence of a gap between the fingers, the number of fingers, or the number of fingers and the width of the fingers. Can be easily recognized by simple combinations.

FIG. 1 is a perspective view of the personal computer according to the first embodiment. FIG. 2 is a front view of the keyboard according to the first embodiment. FIG. 3A is a block diagram illustrating an electrical configuration of the first embodiment. FIG. 3B is a diagram showing the storage contents of the RAM in the block diagram of FIG. 3A. FIG. 3C is a diagram showing the storage contents of the HDD in the block diagram of FIG. 3A. FIG. 4 is a schematic diagram of an operation procedure according to the first embodiment. FIG. 5A is a pointing finger state diagram on the mouse information input surface according to the first embodiment. FIG. 5B is a state diagram of the pointing finger on the input screen waiting for scrolling according to the first embodiment. FIG. 5C is a state diagram of the pointing finger on the input screen for continuing the left click pressing according to the first embodiment. FIG. 5D is a diagram of the pointing finger state on the input surface of the keyboard according to the first embodiment. FIG. 5E is a state diagram of the pointing finger on the numeric keypad input surface according to the first embodiment. FIG. 5F is a diagram of the pointing finger state in the second procedure for selecting the input surface for mouse information according to the first embodiment. FIG. 6 is an operation state determination table according to the first embodiment. FIG. 7 is an input information table of the first embodiment. FIG. 8A is an operation state determination table of a modification example of the operation procedure of the first embodiment. FIG. 8B is an input information table of a modification example of the operation procedure of the first embodiment. FIG. 8C is a pointing finger state diagram in the modified example of the operation procedure of the first embodiment. FIG. 9 is a flowchart of overall processing in the first embodiment. FIG. 10 is a perspective view of the personal computer according to the second embodiment. FIG. 11 is a diagram of the personal computer according to the second embodiment viewed from the left side. FIG. 12 is a front view of the keyboard according to the second embodiment. FIG. 13A is a two-dimensional video information diagram according to the second embodiment. FIG. 13B is a one-dimensional video information diagram in FIG. 13A. FIG. 13C is a two-dimensional image information diagram according to the second embodiment. FIG. 13D is a one-dimensional video information diagram in the case of FIG. 13C. FIG. 14 is an explanatory diagram of a calculation procedure of two-dimensional coordinates according to the second embodiment. FIG. 15A is a two-dimensional image information diagram when the finger is stationary in the second embodiment. FIG. 15B is a two-dimensional image information diagram when the finger is moving in the second embodiment. FIG. 15C is a two-dimensional image information diagram when there is no finger in the second embodiment. FIG. 15D is a two-dimensional image information diagram when two fingers are stationary in the second embodiment. FIG. 15E is a two-dimensional image information diagram when two fingers are moving in the second embodiment. FIG. 15F is a two-dimensional image information diagram when three fingers are stationary in the second embodiment. FIG. 15G is a two-dimensional image information diagram when three fingers are moving in the second embodiment. FIG. 16 is an operation state determination table of the second embodiment. FIG. 17A is a two-dimensional video information diagram according to the fourth embodiment. FIG. 17B is a one-dimensional video information diagram in the case of FIG. 17A. FIG. 17C is a two-dimensional video information diagram according to the fourth embodiment. FIG. 17D is a one-dimensional video information diagram in FIG. 17C. FIG. 18A is a state diagram of fingers at different distances L and 2L from the camera according to the fourth embodiment. FIG. 18B is a one-dimensional image information diagram of a finger at a distance L in the fourth embodiment. FIG. 18C is a one-dimensional image information diagram of a finger at a distance 2L in the fourth embodiment. FIG. 19 is a perspective view of a personal computer according to the fifth embodiment. FIG. 20 is a diagram of the personal computer of the fifth embodiment viewed from the left side. FIG. 21 is a front view of the keyboard according to the fifth embodiment. FIG. 22A is a diagram of the pointing finger state according to the fifth embodiment. FIG. 22B is an excerpt of the operation state determination table of the fifth embodiment. FIG. 23 is a perspective view in which the imaging means of the seventh embodiment is a separate housing. FIG. 24A is a flowchart of processing performed by the imaging apparatus side according to the seventh embodiment. FIG. 24B is a flowchart of processing performed on the PC side according to the seventh embodiment. FIG. 25 is a perspective view in which the imaging means of Embodiment 9 is a separate housing. FIG. 26A is a perspective view in which the imaging means of Embodiment 11 is a separate housing. FIG. 26B is a diagram of the imaging device of Example 11 as viewed from the left side. FIG. 27 is a front view of the input auxiliary seat according to the eleventh embodiment. FIG. 28 is a schematic diagram of an operation procedure according to the eleventh embodiment. FIG. 29A is an excerpt of the operation state determination table of the eleventh embodiment. FIG. 29B is an excerpt of the input information table of the eleventh embodiment. FIG. 30A is an imaging diagram of a finger in a certain state according to the eleventh embodiment. FIG. 30B is a one-dimensional image information diagram of the finger in the state shown in FIG. 30A. FIG. 30C is a one-dimensional video information diagram when the state of a specific finger changes from a certain state in the eleventh embodiment. FIG. 31 is a perspective view of the thirteenth embodiment. FIG. 32 is a perspective view of the fifteenth embodiment. FIG. 33 is a perspective view of the sixteenth embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Throughout the drawings, the same reference numerals or symbols denote the same or equivalent parts throughout the specification.

  First, a notebook personal computer that functions as an input device of the invention, on the keyboard of the notebook personal computer, in addition to conventional keyboard operation, information equivalent to a mouse using image recognition, information equivalent to a numeric keypad, and shortcut keys A first embodiment capable of inputting equivalent information and information equivalent to characters will be described with reference to FIGS.

First, the physical configuration will be described with reference to FIGS.
Reference numeral 1 denotes an openable notebook personal computer (hereinafter referred to as PC1), which is a keyboard composed of a plurality of operation keys for inputting various characters, symbols, and function information (function keys, etc.) on the main body side. 4 is provided.
In addition, a display 2 as a display screen is provided in the center of the lid side, and a CCD (Charge Coupled Device) camera 3 (hereinafter referred to as a camera 3) for always photographing a finger on the keyboard 4 during operation is provided at the top. It has been. In addition, a USB (Universal Serial Bus) connector 8 is provided on the side surface on the main body side.
The dotted line 4a in FIGS. 1 and 2 is an area used when inputting information equivalent to a mouse, information equivalent to a numeric keypad, information equivalent to a shortcut key, or information equivalent to a character from the image of the camera 3. A small area of the dotted line 4b is an arrangement position of an area for inputting numerals 0 to 9 of numeric key information and symbols of four arithmetic operations.

An adjustment knob 5 is provided around the lens of the camera 3 so that the visual field of the camera 3 can be adjusted so that a finger operating the keyboard 4 can be reliably captured as an image.
In particular, since it is the left hand and right hand portions on the keyboard 4 that are desired to be imaged by the camera 3, it is preferable to set the keyboard 4 to a color that can be easily identified from the hand, such as black. In this way, since the contrast of the hand is increased, the hand part can be easily extracted. Further, by providing irradiation means for irradiating the keyboard 4 near the camera 3, and making the surface of the keyboard 4 a member having a low reflectance, the contrast of the hand can be further increased.

Next, the electrical configuration of the PC 1 will be described with reference to FIG. 3A.
The PC 1 is provided with a CPU 10 for controlling the overall functions of the PC 1 in the control unit 50. In addition to the input program, the CPU 10 stores a ROM 11 that stores various programs for operating the PC 1, various data used in the input program (see FIG. 3B described later), and various data of the PC 1. Are temporarily connected, a timer 13 for measuring the passage of time in the input program, and an I / O interface 14 are connected via a bus. The I / O interface 14 includes a video controller 15 for controlling the display 2 and the camera 3, a key controller 16 for controlling the keyboard 4, and various tables for controlling the input program (FIG. 3C described later). Are connected to a USB connector 8 and a hard disk device (HDD) 17 which is a storage device for storing the information.

Next, an outline of the operation procedure of the input device will be described with reference to FIG.
The input device inputs input information (OP-K: keyboard input surface) for inputting the operation key from the keyboard 4 and inputting information equivalent to the shortcut key and information equivalent to the character, and information equivalent to the mouse. Input surface (OP-M: mouse information input surface), input surface for inputting information equivalent to a numeric keypad (OP-T: numeric keypad information input surface), and input surface for inputting scroll information (OP-MS: input surface waiting for scrolling) and an input surface (OP-MC: input surface for continuing left click pressing) for continuing the left click pressing (operation for dragging operation). The input operation is performed after the input surface to be used in the input operation is selected.

  When the keyboard input screen (OP-K) is selected, the mouse information input screen (OP-M), the scroll waiting input screen (OP-MS), and the numeric keypad information input screen (OP-T) are input. The figure of the finger used as a trigger for selecting a face, or the figure of a finger used as a trigger for inputting input information equivalent to a shortcut key or information equivalent to a character is acquired at predetermined time intervals. It is monitored whether there is an image (K-1, K-3, K-2, and IN-SC: details will be described later with reference to FIG. 6).

If the image captured by the camera 3 is applied to a finger as a trigger for selecting the input surface, the subsequent image captured by the camera 3 determines the selection of the input surface registered in advance. It is determined whether it is a figure, and the selection of the input surface is confirmed.
In addition, if the image captured by the camera 3 is applied to the figure of a finger serving as a trigger for inputting the input information, the subsequent image captured by the camera 3 determines the input information registered in advance. It is determined whether it is a figure, and the input information is confirmed.
The figure of the finger serving as the trigger and the figure of the finger following this form an operation procedure for selecting an input surface and an operation procedure for determining input information (hereinafter referred to as an operation procedure of the input device).

When the mouse information input surface (OP-M) is selected,
Trigger to select keyboard input screen (OP-K), numeric keypad information input screen (OP-T), left-click continuation input screen (OP-MC), scroll input screen (OP-MS) The image of the finger acquired as a trigger for determining input information on the input surface of the mouse information or the image of the finger acquired at a predetermined time interval is monitored (M-9 and M-9). -10, M-7, M-8, and M-1: Details will be described later with reference to FIG.

When the numeric keypad input surface (OP-K) is selected,
The figure of a finger that triggers to select the keyboard input surface (OP-K) or mouse information input surface (OP-M), or the shape of a finger that triggers the input of numeric keypad information. It is monitored whether there is an image acquired at a predetermined time interval (T-2, T-3, and T-1: details will be described later with reference to FIG. 6).
Therefore, the keyboard input surface (OP-K), the mouse information input surface (OP-M), and the numeric keypad information input surface (OP-K) are specially meaningful finger shapes that trigger the operator. You can come and go as you go.

  In addition, when a finger shape serving as a trigger for determining the input information is found on the mouse information input surface (OP-M), the input information is determined by the subsequent finger shape. And the subsequent finger form constitute an operation procedure (M-2 to M-6) for determining input information on the input surface of mouse information.

Therefore, the operation procedure of the input device comprising the operation procedure for selecting the input surface and the operation procedure for determining the input information on the selected input surface is the figure of the finger that starts the operation procedure for each input surface on which the operation procedure can be used. And the subsequent combination of the figure of the finger.
In other words, this combination of finger shapes can give an input device an instruction to select an input surface and an instruction to generate input information on the selected input surface (hereinafter referred to as an operation instruction).

Furthermore, by setting in advance the minimum time that the finger must continue (hereinafter referred to as the pointing finger duration) for each of the fingers constituting the combination of the fingers, Whether or not the figure continues the set pointing finger duration can be set as a determination condition for issuing an operation instruction. For example, if the pointing finger duration is zero seconds, an operation instruction can be issued immediately when the finger shape changes.
If the finger changes before the instruction finger duration time is reached, the operation according to the operation procedure is not performed.

Therefore, an operation procedure can be constructed so that the operator can operate at a speed that is easier to operate. Further, the operation instruction can be determined based on the time during which the same pointing finger state is maintained.
The elapsed time to be compared with the pointing finger duration time is a duration time during which the same finger is continued, measured by the timer 13, and stored in the elapsed time 122 as needed.

Next, how to identify a finger shape that is easy for the operator to operate and a finger shape that is meaningful to the operator, as determined by the operator, will be described with reference to FIG.
Considering that the finger can be easily taken by the operator and can be easily extracted and determined from the video information of the camera 3, the finger is identified based on the following criteria. Is called the pointing finger state.
First, it is determined from the video information whether there is a gap between fingers, and the gap state is determined. Even when there is no gap between the fingertips, it is determined that there is no gap between the fingers. Next, in order to identify the finger moving with the left and right fingers, the position coordinates of the left and right fingers on the video information are obtained, and the position coordinates of the left and right fingers of the previous video information are stored in the storage area. Compared with the previous coordinate 121, the difference situation of the position coordinate is determined. Note that the position coordinates of the finger on the video information may be obtained from the position of the tip of each finger within the area of the dotted line 4a in FIG.
When the difference state between the gap state and the position coordinate is determined, the pointing finger state is determined from the pointing finger state table 173 in which the pointing finger state is associated with the difference state between the gap state and the position coordinate. The two-dimensional coordinates of the finger on the input surface are the same as the position coordinates of the finger on the video information. After the difference status is determined, the finger position coordinates are stored in the previous coordinates 121.

For example, the right hand is used so that the finger shape that triggers the selection of the mouse information input surface and the finger shape that is used when moving the mouse pointer become a meaningful finger shape for the operator. The shape was like holding a mouse. The right hand has a gap between the thumb and the index finger, a gap between the index finger and the middle finger, and a shape between the middle finger and the ring finger and a gap between the ring finger and the little finger. Then, the left hand is prevented from entering the dotted line 4a in FIG.
Hereinafter, the state in which the figure of the finger is stationary is referred to as a pointing finger state “001” (see the pointing finger state “001” in FIG. 5A).

Next, the pointing finger state determined from the image of the camera 3 is
Operation is performed according to the operation procedure, and the operation procedure ends (hereinafter referred to as W operation state). The operation procedure is narrowed down to one operation procedure and is performed according to the operation procedure (hereinafter referred to as X operation state). The operation procedure is being narrowed down (hereinafter referred to as the “Y operation state”), and the operation state is determined and determined whether the pointing finger state does not apply to the operation procedure (hereinafter referred to as the “Z operation state”). The operation state determination table 171 will be described with reference to FIG.

  From the description so far, the operation procedure is as follows: “Indicating finger state, pointing finger duration and operation instruction” which is a starting procedure for each input surface, and “Indicating finger state and instruction” which is a subsequent procedure if necessary. It can be seen that it is composed of “finger duration time and operation instruction”.

  Therefore, the input device includes an operation state determination table 171 (see FIG. 6) in which one or more sets of pointing finger states, pointing finger durations, and operation instructions are associated with the input surface. The operation state (the W, X, Y, or Z operation state) is determined by matching the pointing finger state determined from the video acquired at a predetermined time interval, the elapsed time 122, and the operation state determination table 171. I can grasp it.

  For example, if the input screen of the keyboard (OP-K in FIG. 4) is selected and the video acquired from the camera 3 is the pointing finger state “001”, the operation state determination table 171 in FIG. 6 is currently selected. Among the data in which the input surface is the input surface of the keyboard, the data with the pointing finger state of the starting procedure “001” is only (K-1) in FIG. The operation procedure indicated by the data of -1) is started with the pointing finger state “001” as a start trigger, and the X operation state is entered.

After that, if the pointing finger state of the video acquired from the camera 3 is “001” and the time for continuously acquiring the pointing finger state “001” is less than 1 second, the operation state is a predetermined input device. The X operation state continues while the operation procedure is followed.
After that, when the pointing finger state “001” of the video acquired from the camera 3 continues for 1 second or more specified by the pointing finger duration, the operation state is operated according to a predetermined operating procedure of the input device. Assuming that the user is willing to make a selection decision, the W operation state is entered. As the operation instruction, the input surface of the mouse information is selected, and the operation procedure with the pointing finger state “001” as a start trigger is completed.

Therefore, the input device can generate input information if there is an input information generation instruction on the selected input surface and operation instruction.
Also, when the input information for the selected input surface and the input information generation instruction is determined for the two-dimensional coordinates of the finger (for example, numeric keypad information), the selected input surface and the input information generation instruction And an input information table 172 (see FIG. 7) in which input information is associated with the determination conditions of the two-dimensional coordinates of the finger.
In the operation state determination table 171 of FIG. 6, the number of procedures following the start procedure is three, but may be four or more.

Next, details of the operation procedure will be described with reference to FIGS.
First, when the PC 1 startup screen is displayed, the input device immediately after startup of the PC 1 starts photographing a predetermined area on the keyboard 4 (within the dotted line 4a in FIG. 2) by the camera 3, and displays the mouse information input screen. It is monitored (K-1) whether or not the figure of the finger serving as a trigger for selection (indicating finger state "001") is present in the video acquired at a predetermined time interval. The input surface that is automatically selected when the personal computer is activated is the keyboard input surface (OP-K in FIG. 4).

From this state, “To start the mail software from the startup screen of PC 1, move the mouse pointer to the mail software start icon and double-click at that position to start the mail software. Using the operation procedure of “Close the mail software window” as an example, an explanation will be given with reference to FIG.
Procedure 1: First, from the initial state in which the keyboard input surface (OP-K in FIG. 4) is selected, the mouse information input surface (OP-M in FIG. 4) is selected (K-1 in FIG. 4). To do.
Procedure 2: Waiting for input of information equivalent to that of a mouse (M-1 in FIG. 4).
Procedure 3: After that, move the mass pointer to double click left (M-2 in FIG. 4).
Procedure 4: After that, left-clicking (M-4 in FIG. 4) starts the mail software.
Procedure 5: Then, from the menu, use the same information as the mouse to select create new mail, and then select a destination.
Procedure 6: After that, in order to input the subject and text from the keyboard 4, the keyboard input surface is selected (M-9 in FIG. 4). Note that the name and address of the text are input as input information (K-4, K-5 in FIG. 4) equivalent to characters.
Step 7: After that, when the keyboard input surface is selected, enter the same information as the shortcut key for sending mail (K-6 in Fig. 4), send the mail, and then close the mail soft window Enter the key equivalent information (K-7 in Fig. 4) and close the mail software.
It becomes.

Each operation procedure will be described in association with the operation state determination table 171 (see FIG. 6).
Procedure 1: To select an input surface for mouse information while the input surface of the keyboard is selected, hold the mouse for at least 1 second in the state of the pointing finger state “001”, which is the shape of a finger holding the mouse. do it. In this case, attention is paid to the fact that, in the pointing finger state “001”, resting for 1 second or more is an operation procedure that is impossible for an operator who inputs the operation key of the keyboard 4 to input.

It should be noted that an operation procedure of “indicating finger state“ 001 ”and stationary for 1 second or longer” can be performed when the operation key of the keyboard 4 is input and input information from the operator or the keyboard 4 and the same information as the mouse. For an operator who frequently repeats input operations, for example, the left hand has a gap between the thumb and forefinger, no gap between the index finger and the middle finger, no gap between other fingers, and the right hand is in the pointing finger state As in “001”, the mouse information input surface is selected by standing still for 1 second or more in a state where all the left and right fingers are stationary (indicated finger state “003” state, see FIG. 5F). By doing so, the operability is improved for this operator (this operation procedure is referred to as a second procedure for selecting the input surface of mouse information).
In this way, it is possible to arbitrarily set the pointing finger state that is easy for the operator to operate and the pointing finger duration that is the time during which the pointing finger state continues.

  When the input device acquires the video of the camera 3 and confirms that the pointing finger state is “001”, the pointing finger state serving as a start trigger becomes “001” while the input surface of the keyboard is selected. The operation state determination table 171 determines that only the data of K-1 in FIG. 6 is included, and the operation procedure “selects the input surface of mouse information” based on the data of K-1 in FIG. Since the operation is in accordance with the operation procedure, the X operation state is set.

Thereafter, since the pointing finger duration of the pointing finger state “001” in the data of K-1 in FIG. 6 is 1 second, the input device displays the following video as the selection decision of the input surface selection by the operator. It is monitored whether or not the pointing finger state “001” is continuously found for 1 second or more.
If it does not continue for 1 second, it means that the operator has not decided to select the input, and the procedure 1 is not operated according to the operation procedure, so the Z operation state is entered, and the keyboard input screen is displayed. The state in which (OP-K in FIG. 4) is selected continues without being changed.
If it continues for more than 1 second, the operator's input selection decision has been made. That is, since the operation state is operated according to the operation procedure, the operation state is changed to the W operation state, the input surface of the mouse information as the operation instruction is selected, and the operation procedure of the procedure 1 is completed.
As described above, the operation procedure for selecting the input surface is constructed such that the operation procedure is completed when the selection of the input surface is determined. This is because the operation procedure is determined for each input surface that can be used at present.

  Procedure 2: When an input surface for mouse information is selected, it enters a state of waiting for input of information equivalent to that of a mouse. In order to continue this state, the pointing finger state “001” may be continued.

When the input device acquires the video of the camera 3 and determines the pointing finger state to be “001”, the pointing finger state that becomes the start trigger becomes “001” in a state where the input surface of the mouse information is selected. It is determined that only the data of M-1 in FIG. 6 is present, the pointing finger duration of this pointing finger state is zero seconds, and there is no subsequent procedure, so when the pointing finger state “001” is found. Since the operation state is operated according to the operation procedure, the W operation state is set. Immediately after that, input information (a two-dimensional coordinate of the middle finger of the right hand is set as the current position of the mouse pointer) that makes it possible to input information equivalent to that of a mouse is input (see M-1 in FIG. 7). The operation procedure 2 is completed.
In this way, it can be assumed that the operator's intention to select an input is immediately determined by the pointing finger state that triggers the start.

Procedure 3: To move the mouse pointer (M-2 in FIG. 4) prior to the left double click, the entire right hand may be moved on the mouse information input screen (OP-M in FIG. 4).
Moving the entire right hand means moving the entire right hand in the pointing finger state “001” (hereinafter, this state is referred to as the pointing finger state “006”, see FIG. 5A).
When the mouse information input surface is selected in the second step of selecting the mouse information input surface, only the right hand is moved in the state of the pointing finger state “003” (hereinafter referred to as the pointing finger state “007”. (See FIG. 5F).

  When the input device acquires the video of the camera 3 and determines the pointing finger state to be “006”, the pointing finger state that is a start trigger becomes “006” in a state where the input surface of the mouse information is selected. It is determined that only the data of M-2 in FIG. 6 is present, and the pointing finger duration of this pointing finger state is zero seconds, and there is no subsequent procedure, so when the pointing finger state “006” is found. Since the operation state is operated according to the operation procedure, the W operation state is set. Immediately thereafter, input information for moving the mouse pointer is input (see M-2 in FIG. 7), and the operation procedure of the procedure 3 is completed.

Procedure 4: To perform a left double-click, only the index finger is moved while the mouse information input surface is selected (hereinafter, this state is referred to as a pointing finger state “008”, see FIG. 5A: The direction of movement may be up and down or sideways). After that, the pointing finger state “001” is changed to the pointing finger state “008” again before the pointing finger state “001” passes 0.5 seconds or more. Then, the pointing finger state “001” may be set again.
Similarly, when the mouse information input surface is selected in the second step of selecting the mouse information input surface, it is also necessary to move only the index finger of the right hand (indicating finger state “009”, see FIG. 5F).

When the input device acquires the video of the camera 3 and confirms that the pointing finger state is “008”, the pointing finger state serving as a start trigger becomes “008” while the input surface of the mouse information is selected. The data of M-3, M-4, M-6, and M-8 in FIG. 6 is determined, and only the start trigger determines which data should be used among these four data. Since it is not possible, it is not possible to determine which operation procedure is being operated.
Therefore, the operation state is the Y operation state during narrowing down of the operation procedure.
Thereafter, when the pointing finger state “001” is acquired, the data of M-3 and M-4 in FIG. 6 are narrowed down during the operation procedure, and the operation state becomes the Y operation state during the narrowing down of the operation procedure.
Thereafter, when the elapsed time of the pointing finger state “001” exceeds 0.5 seconds of the pointing finger duration, it is determined that the operator's input selection intention is determined by the data of M-3 in FIG. Is operated in accordance with the operation procedure, so that the W operation state is entered, the left click is input (see M-3 in FIG. 7), and the operation procedure of the procedure No. 4 is completed.
Further, if the pointing finger state “008” is acquired before the elapsed time of the pointing finger state “001” exceeds 0.5 seconds of the pointing finger duration, only the data of M-4 in FIG. The state is in operation according to the operation procedure, and becomes the X operation state. Thereafter, when “001” is acquired as the subsequent pointing finger state, since the operation state is operated according to the operation procedure, the W operation state is entered, a double-click is input (see M-4 in FIG. 7), and the mail software is activated. .

  Procedure 5: From the mail software menu, select "Create new mail" and select a destination. In this case, the left double-click operation procedure may be performed after the operation procedure of moving the mouse pointer is performed.

Procedure 6: In order to select a keyboard input surface for inputting characters from the keyboard 4 in the subject and body, the pointing finger state is "001" while the mouse information input surface is selected. On the other hand, the left hand may be placed in a dotted line 4a in FIG. 2 in an arbitrary form so as to be reflected in the video of the camera 3 (hereinafter referred to as pointing finger state “002”, see FIG. 5A).
When the mouse information input surface is selected in the second step of selecting the mouse information input surface, the index finger in contact with the middle finger of the left hand is moved (hereinafter referred to as the pointing finger state “004”, FIG. 5F). And the left index finger and middle finger may be stationary (hereinafter referred to as the pointing finger state “005”, see FIG. 5F).

  When the input device acquires the video of the camera 3 and determines the pointing finger state to be “002”, the pointing finger state that becomes the start trigger becomes “002” in a state where the input surface of the mouse information is selected. What is shown in FIG. 6 is data (M-9) for selecting the input surface of the keyboard (OP-K in FIG. 4) and data (M-10) for selecting the input surface of the numeric keypad information. The state is the Y operation state during the narrowing down of the operation procedure.

It can be seen that the operator's intention to select the input based on the data of M-9 in FIG. 6 is the pointing finger state “002” and the pointing finger duration is 1 second.
Therefore, if the operation procedure continues to acquire the pointing finger state “002” for 1 second or longer, the operation procedure is performed according to the operation procedure of the data of M-9, so that the W operation status is entered, and the input surface of the keyboard is selected.

After that, the subject and text are input using the keyboard 4 (IN-K in FIG. 4). However, the name and address in the text are input as input information (K-4, K-5 in FIG. 4) equivalent to characters.
Therefore, the input of the address is performed by bringing the left and right thumbs into contact with each other in a state where there are gaps between the thumb and forefinger of the left and right hands and between the index finger and the middle finger (hereinafter, pointing finger state “010”). Then, move only the index finger of the right hand (hereinafter referred to as pointing finger state “011”, see FIG. 5D). Thereafter, when the user stops in the pointing finger state “010”, the contents of the address information determined by the operator are input.

When the input device acquires the video of the camera 3 and confirms that the pointing finger state is “010”, the pointing finger serving as a start trigger in a state where the input surface of the keyboard (OP-K in FIG. 4) is selected. The state of “010” is determined to be the data of K-4, K-5, K-6, and K-7 in FIG. 6, and the operation state is set to the Y operation during the narrowing down of the operation procedure. State.
Thereafter, when the pointing finger state “011” is acquired, the data of K-4 in FIG. 6 is targeted, and the operation state is in operation according to the operation procedure of input of character 1 (address information), and becomes the X operation state. . Thereafter, when the pointing finger state “010” is acquired, the operation is performed according to the operation procedure of the data of K-4, so that the W operation state is entered, and address information is input (see K-4 in FIG. 7).
Note that the name is input by moving only the middle finger in the pointing finger state “010” (referring to the pointing finger state “012”, see FIG. 5D). This is input in the same manner by the data of K-5 in FIG. 6 and K-5 in FIG.

Procedure 7: Since the input of the subject and the text has been completed, the state of the pointing finger state “010” is next to the pointing finger state “010” in order to input a shortcut key (eg, Alt + S) for mail transmission. Then, the index finger of the left hand is moved (hereinafter referred to as the pointing finger state “013”, see FIG. 5D), and then the pointing finger state “010” is set again. This is realized by the data of K-6 in FIG. 6 and the data of K-6 in FIG.
The shortcut key (Alt + F4) for closing the mail software window is also moved only in the middle finger of the left hand in the pointing finger state “010” (hereinafter referred to as the pointing finger state “014”, see FIG. 5D). This is realized by the data of K-7 in FIG. 6 and K-7 in FIG.

  Hereinafter, an outline of an operation procedure for inputting information equivalent to that of other mice will be described.

  The right click (M-5) moves only the middle finger of the right hand in a state waiting for input of the same information as the mouse (OP-M in FIG. 4) (hereinafter referred to as the pointing finger state “015”, see FIG. 5A). Thereafter, when the subsequent pointing finger state “001” continues for 0.5 seconds, the right-click is performed. This is realized by the data of M-5 in FIG. 6 and M-5 in FIG.

  Repeated movement of the mouse pointer (M-6: When moving a long distance with the mouse, once the mouse is lifted back and then moved again), the mouse pointer does not move when the moved hand is returned. In order to achieve this, the moving hand is temporarily stopped (the pointing finger state is “001”), the index finger is moved (the pointing finger state is “008”), and the entire finger is moved immediately thereafter (the pointing finger state) (006)). This is realized by the data of M-6 in FIG. 6 and M-1 in FIG. Since the operation instruction of M-6 in FIG. 6 is the same as the operation instruction of M-1 in FIG. 6, input information is generated by M-1 in FIG.

The movement (drag) operation is performed in (C-2) with the input screen (OP-MC in FIG. 4) for continuing the left click pressed being selected.
In order to use the input screen for continuing the left click, the mouse information input screen (OP-M in FIG. 4) is selected, and only the thumb is moved (hereinafter referred to as the pointing finger state “016”). 5A), the subsequent pointing finger state is “001”, and after 0.3 seconds, the input surface for continuing the left click is selected. This is realized by the data M-7 in FIG.
When the input screen (OP-MC in FIG. 4) for continuing the left click is selected, the state for continuing the left click is continued by the pointing finger state “001”. This is realized by the data of C-1 in FIG. After that, if the pointing finger state is set to “006” in a state where the input surface (OP-MC in FIG. 4) for continuing the left click is selected, a movement (drag) operation can be performed. This is realized by the data of C-2 in FIG. 6 and C-2 in FIG.

The drop operation may be performed by releasing the left click pressing state (C-3 in FIG. 4) and selecting the mouse information input surface (OP-M in FIG. 4).
Canceling the left click pressing state is a state where the left click pressing state is continued (the pointing finger state is “001”), the thumb is moved (the pointing finger state is “016”), and then the pointing finger continues. The state may be “001”. This is realized by the data of C-3 in FIG. 6 and C-3 in FIG.

  The screen scrolling operation (S-2) selects the input screen waiting for scrolling (OP-MS in FIG. 4) (M-8) while the mouse information input screen (OP-M in FIG. 4) is selected. ) And the keyboard input screen (OP-K in FIG. 4) are selected, the input screen waiting for scrolling (OP-MS in FIG. 4) is selected (K-3). There are two operational procedures to be performed.

In the case of the former (M-8), the scroll-waiting input surface is a state where the index finger is moved to the right and there is no gap between the index finger and the middle finger (hereinafter, the pointing finger state is referred to as “017”, see FIG. 5A) And selected by standing still for 0.5 seconds or more. This is realized by the data M-8 in FIG.
Note that the two-dimensional coordinates of the middle finger when waiting for scrolling are stored in the scroll determination coordinates 125, and the right hand is moved up, down, left and right based on the scroll determination coordinates 125 (the pointing finger state is "018"). ", See FIG. 5B), and scrolls up, down, left and right.
When the input screen waiting for scrolling is selected and the pointing finger state continues to be “017” (see FIG. 5A), the screen can be scrolled. This is realized by the data of S-1 in FIG.
Since the data corresponding to the “screen scrollable state” of the operation instruction in S-1 in FIG. 6 does not exist in the input information table (see FIG. 7), the input information is not generated and the screen can be scrolled. Continue.

  In the latter case (K-3), the input screen waiting for scrolling is selected by holding the pointing finger in the state of “017” for at least one second. This is realized by the data of K-3 in FIG.

  The screen scroll (S-2) is based on the scroll determination coordinate 125 when the right hand is moved on the input screen waiting for scrolling (OP-MS in FIG. 4) (the pointing finger state is “018”, see FIG. 5B). Then, when the finger moves more than a predetermined distance, the screen is scrolled according to the moving direction of the finger (see the determination condition of the two-dimensional coordinates of the finger in S-2 in FIG. 7). This is realized by the data of S-2 of FIG. 6 and S-2 of FIG.

  To select the mouse information input surface (OP-M in FIG. 4) from the state in which the scroll waiting input surface (OP-MS in FIG. 4) is selected, the index finger of the right hand is moved to the left (indicating finger). The state is “008”), and then the pointing finger state is set to “001”, and it may be stationary for 0.5 seconds or longer. This is realized by the data of S-3 in FIG.

  In order to select the keyboard input surface (OP-K in FIG. 4) from the state in which the scroll-waiting input surface (OP-MS in FIG. 4) is selected, the pointing finger state to enter the screen scroll wait state When the pointing finger state other than “017”, the pointing finger state “018” for scrolling the screen, or the pointing finger state “008” for selecting the input surface for mouse information is selected, the input surface of the keyboard is selected. This is realized by the data of S-4 in FIG.

Next, FIG. 4 shows an operation procedure of “after inputting characters from the keyboard 4, selecting a numeric keypad information input surface, inputting numeric keypad information, then returning to the keyboard input screen and inputting characters from the keyboard 4”. To explain,
Procedure No. 1: First, the input surface for numeric key information is selected (K-2 in FIG. 4) while the keyboard input surface (OP-K in FIG. 4) is selected.
Procedure 2: After that, enter the numeric keypad information (T-1 in FIG. 4).
Procedure 3: After that, in order to input characters from the keyboard 4, the input surface of the keyboard is selected (T-2 in FIG. 4).
It becomes.
This operation procedure will be described in association with the operation state determination table 171 (see FIG. 6).

  Step 1: To select an input surface for numeric keypad information when the input surface of the keyboard is selected, input character information with the figure of a finger hitting the key on the numeric keypad, for example, with only the index finger of the right hand (Hereinafter referred to as the pointing finger state “019”, see FIG. 5A). The pointing finger state “019” may be any state of the left hand.

  When the input device acquires the video of the camera 3 and confirms that the pointing finger state is “019”, the pointing finger serving as a trigger for start is selected with the keyboard input surface (OP-K in FIG. 4) selected. The operation state determination table 171 whose state is “019” includes only the data K-2 in FIG. 6, and the operation procedure of the data K-2 in FIG. 6 is being operated according to the operation procedure. , X operation state.

Thereafter, the input device monitors whether or not the pointing finger state “019” is continuously found in the following video for at least 1 second as an intention to select the input surface by the operator (the pointing finger duration is 1 second). .
If the operation did not continue for 1 second, the operator did not intend to select the input, so procedure 1 was not operated according to the operating procedure, so the Z operation state was entered and the keyboard input surface was selected. The state that has been done will continue.
If the operation is continued for 1 second or more, the operator's intention to select the input is determined, and the operation procedure is operated according to the operation procedure, so that the operation state becomes the W operation state. Then, an input surface for ten-key information as an operation instruction is selected, and the operation procedure of procedure 1 is completed.

  Procedure 2: To enter the numeric keypad information, the user can enter at the position of the middle finger of the right hand if the finger is stationary for at least one second in the state of the pointing finger “019”.

  When the input device acquires the video of the camera 3 and confirms that the pointing finger state is “019”, the pointing finger serving as a trigger for the start is selected when the input surface of the numeric keypad (OP-T in FIG. 4) is selected. The operation state determination table 171 whose state is “019” determines that only the data of T-1 in FIG. 6 is included, and the operation procedure of the data of T-1 in FIG. 6 is being operated according to the operation procedure. Let the operation state be the X operation state.

Thereafter, the input device monitors whether or not the pointing finger state “019” is continuously found in the following video for at least 1 second as an intention to select the input surface by the operator (the pointing finger duration is 1 second). .
If the operation continues for 1 second or more, it is assumed that the operator's intention to select the input has been made, and the operation procedure is operated according to the operation procedure, so the operation state becomes the W operation state. Then, input of numeric keypad information as an operation instruction is determined, input information associated with the two-dimensional coordinates of the middle finger of the right hand (see T-1 in FIG. 7) is determined and generated as input information, and procedure 2 This completes the operation procedure.

  Procedure 3: To select a keyboard input surface (OP-K in FIG. 4) for inputting characters from the keyboard 4, the pointing finger state is "019" and the pointing finger state is "001". If the state is other than, the keyboard input surface is selected.

  When the input device acquires the video of the camera 3 and determines that the pointing finger state is “019” and the pointing finger state is a state other than “001”, the pointing finger state that triggers the start is “019”, and Since the operation state determination table 171 in which the pointing finger state is in a state other than “001” is only the data of T-2 in FIG. 6, the data operation procedure is in operation according to the operation procedure, and the operation state is The X operation state is assumed. Thereafter, when the same pointing finger state continues for 1 second or longer, the operation procedure is operated according to the operation procedure, and therefore the operation state becomes the W operation state. Then, the input surface of the keyboard is selected, and the operation procedure of procedure 3 is completed.

  In order to select the mouse information input surface from the ten key information input surface, the mouse information input surface may be stopped for one second in the state of the pointing finger state “001” (see T-3 in FIG. 6). In order to select the input surface of the numeric keypad from the state, it is only necessary to set the pointing finger state “002” and then rest for one second in the pointing finger state “019” (see M-10 in FIG. 6).

The change of the input surface, the pointing finger state, and the operating procedure can be easily performed by the operator himself by changing the contents of the operation state determination table 171, the input information table 172, and the pointing finger state table 173.
Therefore, it is possible for the operator to perform an input operation with the input surface that is most easily operated, the pointing finger state, and the operation procedure. An example of changing the input screen, the pointing finger state, and the operation procedure will be described below.

First, the change of the pointing finger state for performing the input operation will be described.
If the operator wishes to move the mouse pointer with one finger (indicating finger state “019”, see FIG. 5A) so as to operate the touch pad, the pointing finger state for performing the input operation is changed. I can do it.
For this purpose, first, the pointing finger state “001” of the data K-1 and M-1 in the operation state determination table 171 in FIG. 6 is changed to “019” (K-1 ′ and M− in FIG. 8A). 1 ′).
Then, the pointing finger state in which the pointing finger state “019” is moving is added to the pointing finger state table 173 as “029”, for example.
Then, the pointing finger state “006” of the data M-2 in the operation state determination table 171 in FIG. 6 may be changed to the pointing finger state “029” (see M-2 ′ in FIG. 8A).

Next, for the addition of the operation procedure for selecting the input surface and the input surface, for example, the input surface of the numeric keypad information of the first embodiment is an input surface for inputting single-byte numbers and symbols. An operation procedure for selecting an input surface and an operation procedure for determining input information on the selected input surface as an input surface for inputting full-width numbers and symbols will be described.
First, the input surface to be added is “tenkey information input surface 2”, and the input surface selected for the T-1 data in the input information table 172 of FIG. 7 is “tenkey information input surface 2”. Is changed to full-width and added to the input information table 172 in FIG. 7 (see T-4 in FIG. 8B).
Next, an operation procedure for selecting “10-key information input surface 2” is added to the operation state determination table 171 of FIG. First, in the pointing finger state for selecting “10-key information input surface 2”, the index finger and the middle finger are shaped like a letter V (refer to pointing finger state “039” in FIG. 8C). Then, when the finger is stationary for 1 second in this pointing finger state “039”, “10-key information input surface 2” is selected (see K-8 in FIG. 8A). Then, in the state where the input surface 2 for the numeric key information is selected, an operation procedure for determining the input information may be added (see T-5 in FIG. 8A).

Next, a change in selection decision will be described.
The operation procedure can be issued only with the figure of the starting finger, but the operator uses the pointing finger duration, the figure of the following finger, etc. After the operator makes an intention to select an input, a confirmation OK button pressing message or the like can be issued on the display 2 to reconfirm, and a more reliable input operation can be performed.

For example, if the pointing finger duration of the data of K-1 in the operation state determination table 171 in FIG. 6 is set to zero seconds, when the pointing finger state “001” is found, the mouse information input surface is immediately selected and the mouse pointer is selected. Can be moved.
Further, in the data of T-1 in the operation state determination table 171 in FIG. 6, the pointing finger state “019” is input when it continues for 1 second or more, but the pointing finger state in which the pointing finger state “019” is moving. If “029” is added and the pointing finger state “019” is found after 1 second, and the pointing finger state “029” is found, the input can be determined (see T-6 in FIG. 8A).
Further, in the data of T-1 in the operation state determination table 171 in FIG. 6, when the pointing finger state “019” continues for 1 second or longer, the input information in the input information table 172 in FIG. It is also possible to change so that an application program for issuing a message or the like and reconfirming is started (see T-7 in FIG. 8B).

In addition, various input operations can be performed on the input information by linking several subsequent finger shapes to the starting finger shape, or by transitioning from the input surface to the input surface.
For example, various input operations can be constructed with only two types of pointing finger states “019” and “029”.
In T-5 of FIG. 8A, the input on the input surface 2 of the numeric keypad information is determined when the pointing finger state “019” is held for 1 second or longer.
In T-8 of FIG. 8A, when the pointing finger state is “019” and the pointing finger state is “029” after standing still for 2 seconds or more, the address input on the personal information input surface is determined.
T-9 in FIG. 8A indicates that the pointing finger state is “029” before 1 second elapses in the pointing finger state “019”, and then the pointing finger state “019” indicates that the postal code on the personal information input surface is The input is determined.

  Next, the overall operation of the input device according to the first embodiment configured as described above will be described with reference to the flowchart of FIG. When the input device is activated, the input program stored in the ROM 11 is read into the RAM 12, and the process shown in FIG.

  First, it is determined whether or not it is a video capture timing (S010). The input device captures video from the camera 3 at a predetermined time interval (in this embodiment, at an interval of 30 milliseconds), and executes the following processing (after S020). If it is not the capture timing (S010: NO), it waits until the capture timing is reached.

  When the capture timing is reached (S010: YES), the video from the camera 3 is captured (S020).

  Next, information on the fingers of the left and right hands is extracted from the video (S030).

Next, based on the finger extraction information, the pointing finger state is determined by the following procedures (1), (2), and (3), and the two-dimensional coordinates of the finger on the input surface are determined (S040).
(1) First, in order to identify the state of the finger to be operated, it is determined whether or not there is a gap between the fingers, and the gap state is determined. Even when there is no gap between the fingertips, it is determined that there is no gap between the fingers.
(2) Next, in order to identify the finger moving with the left and right fingers, the position coordinates of the left and right fingers on the image are obtained, and the storage area in which the position coordinates of the left and right fingers of the previous image are stored Compared with the previous coordinate 121, the position coordinate difference status is determined. Note that the position coordinates are stored in the previous coordinates 121 after the position coordinate difference status is determined. At the time of initial startup, the previous coordinate 121 is an initial setting value, so the next video may be acquired (return to S010), or the difference status may be determined assuming there is a difference in position coordinates.
(3) Next, when the difference situation between the gap situation and the position coordinate is determined, the pointing finger state is determined from the pointing finger state table 173 in which the pointing finger state is associated with the gap situation and the position coordinate difference situation. Confirm.
The position coordinates of the left and right fingers are set as the two-dimensional coordinates of the finger on the selected input surface.

  Next, the determined pointing finger state is operated according to the operation procedure, and the operation procedure is completed (W operation state). The operation is performed according to the operation procedure, narrowed down to one operation procedure. The operation state determination table 171 indicates whether the operation state is (X operation state), the operation procedure is being narrowed down (Y operation state), or the pointing finger state is not applicable to the operation procedure (Z operation state). Match and confirm (S050).

To confirm the operation status,
First, when the previous operation state 127 storing the operation state of the pointing finger state of the previous image is W, Z operation state, or initial value, it is a work area for determining the operation state, and the operation state determination The search table 126 composed of the same items as the table 171 is prepared as described later (S051).
After that, the operation state is determined based on all the data in the search table 126 according to the determined pointing finger state, and when the determination is completed, the operation state can be determined. The operation state is set to the previous operation state 127 (S052).

Preparation of the search table 126 (S051) is selected by the input surface and the input device currently selected in the operation state determination table 171 based on the data registered in the operation state determination table 171. The selected input surface 124 (initial value is OP-K and the keyboard input surface is selected) storing the input surface is equal, and the pointing finger state and the operation state determination table 171 determined this time are the same. The data having the same pointing finger state in the starting procedure is extracted and stored in the search table 126 in the storage area.
If there is no data to be extracted, a blank is set in the storage area search table 126.

The determination of the operation state for each data (S052) is performed by the following method (1), (2) or (3). However, if the search table 126 is blank before the determination process, the operation state determination process for each data need not be performed, and the operation state is determined as the Z operation state.
In addition, when data whose operation state is the W operation state is found during the determination process for all the data, it is not necessary to determine the operation state for the subsequent data, the operation state determination process for each data is terminated, and the operation state Is confirmed in the W operation state.
After the determination of the operation state for all data in the search table 126 is completed, if the search table 126 is blank, the operation state is determined as the Z operation state, and one data item is stored in the search table 126. If it remains, the operation state is determined in the X operation state, and if two or more pieces of data remain in the search table 126, the operation state is determined in the Y operation state.

(1) At this time, when the search table 126 is prepared, the determined pointing finger state is set to the previous pointing finger state 123.
Thereafter, the operation state is determined by the following determination with respect to the starting procedure of the search table 126.
When the determination is completed, the timer 13 is started and the elapsed time is stored in the elapsed time 122 as needed.
(1-1) If the pointing finger duration is “zero” and there is no procedure following the start procedure, the operation state of this data is set to the W operation state.
(1-2) If the pointing finger duration is “zero” and the data has a procedure following the start procedure, the operation state of this data is set to the Y operation state.
(1-3) If the pointing finger duration is not “zero”, the operation state of this data is set to the Y operation state.

(2) When the determined pointing finger state and the previous pointing finger state 123 are not equal, the determined pointing finger state is set to the previous pointing finger state 123. Thereafter, the determination of the operation state is performed for the following procedure following the procedure of the search table 126 used in the final determination of the operation state.
When the determination is completed, the timer 13 is stopped, the timer 13 is restarted, and the elapsed time is stored in the elapsed time 122 as needed.
(2-1) When the determined pointing finger state and the pointing finger state in the subsequent procedure are not equal, this data is excluded from the operation state determination target, and is deleted from the search table 126.
(2-2) If the determined pointing finger state and the pointing finger state of the following procedure are equal, and if the pointing finger duration is “zero” and there is no procedure following the following procedure, this data The operation state is defined as the W operation state.
However, if the elapsed time 122 has not passed the pointing finger continuation time of the previous pointing finger state, this data is excluded from the operation state determination target, and is deleted from the search table 126.
(2-3) If the determined pointing finger state is the same as the pointing finger state of the following procedure, and if the pointing finger duration is “zero” and there is a procedure following the following procedure, this data The operation state is defined as the Y operation state.
However, if the elapsed time 122 has not passed the pointing finger continuation time of the previous pointing finger state, this data is excluded from the operation state determination target, and is deleted from the search table 126.
(2-4) If the determined pointing finger state is the same as the pointing finger state in the following procedure and the pointing finger duration is not “zero”, the operation state of this data is set to the Y operation state.
However, if the elapsed time 122 has not passed the pointing finger continuation time of the previous pointing finger state, this data is excluded from the operation state determination target, and is deleted from the search table 126.

(3) When the confirmed pointing finger state is equal to the previous pointing finger state 123,
The relationship between the pointing finger duration in the pointing finger state and the elapsed time 122 in the procedure of the search table 126 used for determining the final operation state is as follows:
(3-1) When the elapsed time 122 <the pointing finger duration time, the operation state of this data is set to the Y operation state.
(3-2) If the instruction finger duration is equal to or less than the elapsed time 122 and there is no procedure following the current procedure, the operation state of this data is set to the W operation state.
(3-3) If the instruction finger duration is equal to or less than the elapsed time 122 and there is a procedure that follows the current procedure, the operation state of this data is set to the Y operation state.

  Next, according to the confirmed operation state, the following (1) or (2) or (3) or (3) or (3) or (3) or (3) or (3) The method 4) is performed (S060).

(1) In the case of the W operation state, that is, when the operation procedure is completed after the operation procedure is completed (S061: W),
If the input surface is selected by the operation instruction of the data in which the operation state is determined as the W operation state, the selected input surface is set as the input screen 124 during selection of the storage area. When the selected input screen is an input screen waiting for scrolling, the two-dimensional coordinate of the middle finger of the right hand is set as the scroll determination coordinate 125 (S062).
If an instruction for generating input information is issued in response to an operation instruction for data whose operation state is determined to be the W operation state, input information is generated based on the input information table 172 (S063). Finally, the process returns to S010 to acquire the next video.
-When there is no instruction | indication by the operation instruction | indication of the data by which the operation state was decided with the W operation state, in order to acquire the next image | video, it returns to S010.

(2) In the case of the X operation state, that is, when the operation state is in operation according to the operation procedure (S061: X), and an input information generation instruction is issued by the operation instruction, the input information table 172 Based on the above, input information is generated (S063). Finally, the process returns to S010 to acquire the next video.
If no input information generation instruction is issued as an operation instruction, the process returns to S010 to acquire the next video.

  (3) In the Y operation state, that is, when the operation procedure is being narrowed down (S61: Y), the process returns to S010 in order to acquire the next video.

(4) In the case of the Z operation state, that is, when the operation state is the pointing finger state not included in the operation procedure (S061: Z), the following preparation is performed to determine the operation state next time. .
Stop timer 13,
The initial value (zero) is set to the elapsed time 122 of the storage area.
An initial value ("OP-K") is input to the input surface 124 during storage area selection.
The initial value (zero) is set to the scroll determination coordinate 125 of the storage area.
The initial value (blank) is stored in the storage area search table 126.
An initial value (blank) is set in the previous operation state 127 of the storage area (S064).
Finally, the process returns to (S010) to acquire the next video.
In the above, the setting contents of the storage area selection input surface 124 may be determined by the operator.

  Since the process of FIG. 9 is performed at a predetermined time interval, the number of times that the process flow of FIG. 9 is performed may be used instead of measuring the elapsed time with the timer 13. For example, when processing is repeatedly performed at intervals of 0.1 seconds, an elapsed time of 1 second can be handled in the same manner as when the number of processes is 10.

  The PC 1 is not limited to a notebook personal computer but may be a desktop personal computer.

In the above embodiment,
S010 to S030 in FIG. 9 correspond to the finger information extracting means and process of the present invention. Also,
S040 in FIG. 9 corresponds to the pointing finger determination means and process of the present invention. Also,
S050 in FIG. 9 corresponds to the operation state determination means and process of the present invention. Also,
S060 in FIG. 9 corresponds to the operating procedure execution means and process of the present invention.

  Next, in addition to the conventional keyboard operation, information equivalent to a mouse using image recognition, information equivalent to a numeric keypad, information equivalent to a shortcut key, and information equivalent to a character can be input on a keyboard of a notebook computer. Differences from the first embodiment will be described with reference to FIGS. 10 to 16.

First, regarding the physical configuration, differences from the first embodiment will be described with reference to FIGS.
The camera 3 has a camera 3a at the right end position on the opposite side of the keyboard 4, a camera 3b (see FIG. 10) at a position near the center on the opposite side of the keyboard 4, and the center 3a1 of the angle of view of each camera lens. It is installed so as to be parallel to the position 4c (see FIG. 11) immediately above the operation key surface of the keyboard 4 in the direction. The angle of view of each camera lens is about 90 degrees.

10 and 12 is an area used when inputting information equivalent to the mouse, information equivalent to the numeric keypad, information equivalent to the shortcut key, and information equivalent to the character, and is common to the camera 3a and the camera 3b. The camera 3a and the camera 3b are installed so that it may become a visual field.
The camera 3b assumes that the right hand inputs information equivalent to the mouse, information equivalent to the numeric keypad, information equivalent to the shortcut key, and information equivalent to the character at the home position where the keyboard 4 is operated. It is installed so as not to enter the field of view.

  In particular, since the camera 3a and the camera 3b are to be picked up by the hand part on the keyboard 4, by providing irradiation means for irradiating the hand on the keyboard 4 near the camera 3a and the camera 3b, The contrast between the hand and the background can be increased.

  The electrical configuration of the second embodiment includes a camera 3a and a camera 3b instead of the camera 3 in the block diagram of FIG. 3A of the first embodiment. The outline of the operation procedure is the same as that of the first embodiment (see FIG. 4).

  Next, a method for identifying the figure of a finger will be described with reference to FIGS. 13 to 15 for differences from the first embodiment.

In the second embodiment, the number of fingers is determined instead of the finger gap state. Then, the determination of the difference between the position coordinates uses the one-dimensional image information of the position 4c (see FIG. 11) immediately above the operation key surface of the keyboard 4 in the images of the cameras 3a and 3b. In the second embodiment, the finger position coordinates are determined for the video information for each camera.
FIG. 13A is an image of the finger camera 3a that operates the area of the dotted line 4a in FIG. 12 with one finger so as to operate the touch pad. The finger 6a moves the position 4c immediately above the operation key surface. In this state, the background 6b far from the camera 3a is black, and the finger 6a near the camera 3a is imaged white.
FIG. 13B shows one-dimensional video information at a position 4c immediately above the operation key surface in FIG. 13A. Similarly, FIG. 13C is an image of the camera 3b, and FIG. 13D is one-dimensional image information at a position 4c immediately above the operation key surface of FIG. 13C.

A method for calculating the two-dimensional coordinates of the finger on the input surface will be described with reference to FIG.
The two-dimensional coordinates of the finger 6a in FIG. 14 are based on the position coordinate near the center of the finger 6a in FIG. 13B, the straight line 7a connecting the camera 3a near the center of the finger 6a in FIG. 13B, and the base line 7 connecting the cameras 3a and 3b. The angle 7b1 between the straight line 7b connecting the vicinity of the center of the finger 6a in FIG. 13D and the camera 3b and the base line 7 connecting the camera 3a and the camera 3b can be determined from the position coordinates near the center of the finger 6a in FIG. 13D. Therefore, the two-dimensional coordinates of the finger 6a are obtained by triangulation.

  In the case of the figure of FIG. 13, when the number of fingers is one and the finger is stationary, the pointing finger state is “101” (see FIG. 15A). On the other hand, the pointing finger state in which the figure of the finger is moving is called “102” (see FIG. 15B). Even if there is finger information in the video information, if there is no finger information in the one-dimensional video information at the position 4c immediately above the operation key surface of the keyboard 4, it is determined that there is no finger (hereinafter referred to as the pointing finger state). Called “103”, see FIG. 15C).

  In this way, if the figure of the finger is identified to determine the pointing finger state and the operation state determination table of FIG. 16 is provided, the same input as in the first embodiment can be performed. For example, when the keyboard input surface is selected and the pointing finger state “101” is held for 1 second or longer, the mouse information input surface is selected (see K-1 in FIG. 16). When moved, the pointing finger state becomes “102” and the mouse pointer is moved (see M-2 in FIG. 16 and M-2 in FIG. 7).

  In the flowchart of the second embodiment, after extracting finger information from the captured video in S030 of the flowchart of FIG. 9 of the first embodiment, one-dimensional video information of the position 4c immediately above the operation key surface is further extracted. Change as follows.

  It should be noted that the area of the dotted line 4a in FIG. 12 is changed by changing the camera installation position or changing the angle of view of each camera lens (for example, a lens having an angle of view of 135 degrees). It is possible to change the operation area, change the operable area, or operate the mouse pointer with the left hand.

  The PC 1 is not limited to a notebook personal computer but may be a desktop personal computer.

  Next, with respect to the third embodiment in which the CCD camera of the second embodiment is replaced with a one-dimensional image sensor, differences from the second embodiment will be described. An input device in which the camera 3a in FIG. 10 in the second embodiment is replaced with a one-dimensional image sensor 3c and the camera 3b is replaced with a one-dimensional image sensor 3d is a third embodiment.

  The video of the camera 3a of the second embodiment is shown in FIG. 13A, but the video of the one-dimensional image sensor 3c of the third embodiment is shown in FIG. 13B. The video of the camera 3a is located at a position 4c immediately above the operation key surface of the keyboard 4. It is the same as one-dimensional video information. Similarly, the image of the one-dimensional image sensor 3d is the same as FIG. 13D.

  Therefore, the determination of the difference between the number of fingers and the position coordinates in the second embodiment uses the one-dimensional image information at the position 4c immediately above the operation key surface of the keyboard 4 in the images of the cameras 3a and 3b. The determination of the difference between the number of fingers and the position coordinates according to the third embodiment may be performed using images from the one-dimensional image sensor 3c and the one-dimensional image sensor 3d. Accordingly, in S030 of FIG. 9, since the captured video becomes one-dimensional video information, the difference status between the number of fingers and the position coordinates is determined based on the one-dimensional video information.

  Even with such a configuration, the same effects as those of the second embodiment can be obtained, and the imaging means can be a one-dimensional image sensor, so that an inexpensive input device can be provided for the CCD camera. It becomes like this. It should be noted that either one of the camera 3c and the camera 3d is used as a CCD camera, and the same operation and effect can be obtained by extracting one-dimensional video information at a position 4c immediately above the operation key surface from the video information of the CCD camera. Is obtained.

In the finger information that can be determined from the one-dimensional video information in the second and third embodiments, the finger information is one finger (see FIG. 15A), or a plurality of fingers are aligned (for example, the middle finger and the ring finger) And the shape with the little fingers aligned (see FIG. 17A).
Therefore, the finger width that can be determined from the finger information extracted from the one-dimensional video information is used depending on the distance between the camera and the finger 6a.

For example, FIG. 18A is a diagram showing a finger 6a at a distance L from the camera 3a and a finger 6a at a distance 2L (a distance twice as long as the distance L) from the camera 3a. 18B shows one-dimensional video information of the camera 3a of the finger 6a at a distance L from the camera 3a, and FIG. 18C shows one-dimensional video information of the camera 3a of the finger 6a at a distance 2L from the camera 3a. The ratio between the width of the finger 6a and the background in FIG. 18B is 1: 3, and if the imaging width is 16 cm, the width of the finger 6a is 4 cm. The ratio of the width of the finger 6a to the background in FIG. 18C is 1: 7, and if the imaging width is 16 cm, the width of the finger 6a is 2 cm.
Here, it can be seen that the product of the distance L and the width (4 cm) of the finger 6a in FIG. 18B is equal to the product of the distance 2L and the width (2 cm) of the finger 6a in FIG. 18C. Therefore, it can be seen that the product of the finger width and the distance between the camera and the finger 6a that can be determined from the finger information is constant.

  Accordingly, the determination of the pointing finger state further takes into consideration the configuration of the finger, which means whether the finger extracted from the one-dimensional video information is a single finger or a plurality of fingers are aligned. The pointing finger state is determined from the pointing finger state table 173 in which the pointing finger state is associated with the difference state of the number of fingers, the position coordinates, and the configuration state of the finger.

The configuration status of the finger is obtained as follows.
The input device has a standard reference width for determining whether the width of a finger at a predetermined distance from the right camera is one finger or a plurality of fingers aligned. For example, when “standard reference width 1 ≦ finger width ≦ standard reference width 2” is satisfied, it is determined that the finger is composed of one finger, and “standard reference width 2 <finger width”. If it satisfies the condition, it is determined that the finger is configured with three fingers aligned. If “finger width <standard reference width 1”, it is determined whether or not the finger is shown in the video information. It is determined that the operation has just started, and it is determined that there is no finger information.

  The product of the predetermined distance and the standard reference width of the predetermined distance and the product of the reference width of the arbitrary position and the distance between the finger of the arbitrary position and the camera are constant, so that the reference width of the arbitrary position is Since it is obtained, the configuration of the finger at an arbitrary position is determined.

  If a user is specified, three standard reference widths can be set, and it can be determined whether the finger configuration is one, two, or three. In addition, with one standard reference width, the same finger state information can be obtained when one finger and a plurality of fingers are aligned.

  A fifth embodiment in which the PC 1 configured in the first embodiment is further provided with a CCD camera 3a in the center on the opposite side of the keyboard 4 will be described with reference to FIGS.

First, with respect to the physical configuration, differences from FIGS. 1 and 2 of the first embodiment will be described with reference to FIGS.
The camera 3a is positioned at a position 3a2 in which the center 3a1 of the angle of view of the lens of the camera 3a is parallel to the position 4c immediately above the operation key surface of the keyboard 4 (see FIG. 20), and the keyboard 4 is equally divided into left and right. The camera 3a is set so that the shooting area includes the area 4a created by the camera 3 (see FIG. 21). The angle of view of the lens of the camera 3a is about 135 degrees.

  Next, in the electrical configuration, the difference from FIG. 3A of the first embodiment is that the camera 3 and the camera 3 a are connected to the video controller 15.

Next, the operation procedure is different from the first embodiment in the operation procedure for selecting the input surface of the numeric key information and the operation procedure for inputting the numeric key information.
Further, the operation state of the operation procedure for inputting the numeric key information is determined by using the pointing finger state determined from the images of the camera 3 and the camera 3a. Therefore, the operation state determination table 171 is constructed by the pointing finger states and the pointing finger duration time of the cameras 3 and 3a.

  Hereinafter, an operation procedure for selecting an input surface for numeric key information and an operation procedure for inputting numeric key information will be described. When the keyboard input surface (see OP-K in FIG. 4) is selected, the operation procedure for selecting the numeric keypad information input surface (see OP-T in FIG. 4) is as follows. A pointing finger state in which the right hand may be in any state with a gap and no gap between the index finger and the middle finger (hereinafter, the pointing finger state is referred to as “501”, see FIG. 22A) Is determined from the video of the camera 3, the input surface of the numeric keypad information is selected (see K-2 in the operation state determination table 171 in FIG. 22B).

  In order to input the numeric key information in a state where the input surface of the numeric key information is selected, the pointing finger state is “501”, and the pointing finger state of the right hand portion is “103” from the image of the camera 3a. This is a time when it is determined that the state has changed from "1" to "101" (see FIG. 15) (see T-1 in the operation state determination table 171 in FIG. 22B).

  When it is determined that the numeric keypad information has been input, the position coordinates on the video of the center of the finger 6a of the pointing finger state “101” of the right hand portion of the camera 3a can be determined. Since the finger that has passed the position 4c immediately above the operation key surface of the keyboard 4 can be determined from the extracted finger information, the two-dimensional coordinates of the finger that has passed are determined from the image of the camera 3, and the input information shown in FIG. Input information is determined from the table 172.

With respect to the operation of the input device according to the fifth embodiment configured as described above, differences from the first embodiment will be described with reference to FIG.
In the video capturing in S020, video from the camera 3 and the camera 3a is captured.
In the extraction process of finger information in S030, for the camera 3a, one-dimensional video information at a position 4c immediately above the operation key surface of the keyboard 4 is extracted as in the second embodiment.
In the pointing finger state determination process of S040, the pointing finger state is determined in the same manner as in the first embodiment, and further, the pointing finger state is determined in the same manner as in the second embodiment from the finger extraction information of the camera 3a.
In S063, the two-dimensional coordinates of the finger for determining the input information are specified from the position coordinates of the finger on the image of the camera 3a and the two-dimensional coordinates obtained from the figure of the finger taken by the camera 3.
Note that an operation procedure of information equivalent to that of a mouse can be constructed using the pointing finger states of the camera 3 and the camera 3a.

  Even with such a configuration, the same operational effects as those of the first embodiment can be obtained, and the operation procedure can be constructed for the pointing finger state acquired from the imaging of the camera 3 and the camera 3a. It can be a procedure.

  The sixth embodiment is characterized in that a one-dimensional image sensor 3c is used instead of the camera 3a provided in the fifth embodiment, and finger information is extracted using an image of the one-dimensional image sensor 3c.

  Even with this configuration, the same effects as those of the fifth embodiment can be obtained, and an inexpensive input device can be provided.

  A seventh embodiment in which the camera 3 of the input device configured in the first embodiment is an external imaging device 9 will be described with reference to FIGS.

First, a physical configuration and an electrical configuration will be described with reference to FIG.
The input device connects the imaging device 9 to the PC 1 via the USB connector 8, acquires the video of the camera 3 by the control circuit 9 a of the imaging device 9, extracts finger information, and then extracts the finger extraction information to the USB In this configuration, the data is sent to the PC 1 via the connector 8.

  Next, the operation of the input device of the seventh embodiment configured as described above, that is, the operations of the imaging device 9 and the PC 1 will be described with reference to the flowchart of FIG. To do.

The imaging device 9 performs processing corresponding to S010 to S030 of the first embodiment (see FIG. 24A).
First, it is determined whether or not it is a video capture timing (S710). The imaging device 9 captures video from the camera 3 at predetermined time intervals and executes the following processing. If it is not the capture timing (S710: NO), it waits until the capture timing is reached. When the capture timing is reached (S710: YES), the video from the camera 3 is captured (S720).
Next, information on the fingers of the left and right hands is extracted from the video, and the extracted data is transmitted via the USB connector 8. (S730).

The PC 1 performs processing corresponding to S040 to S060 of the first embodiment (see FIG. 24B).
First, it is determined whether or not it is the fetch timing of the transmission data transmitted from the imaging device 9 via the USB connector 8. If it is not the capture timing (S711: NO), it is determined again whether the capture timing is reached. If it is the capture timing (S711: YES), the transmission data is captured (S721).
Thereafter, the same processes from S040 to S060 as in the first embodiment are performed (S740 to S760).

  Even with such a configuration, the same effects as those of the first embodiment can be obtained, and utilization on various input operation surfaces is possible. For example, if the external imaging device 9 is installed so that the input surface is formed on the operation surface on the desk, it can be used as a mouse-dedicated device even in a usage environment with much dust and oil water.

The control circuit 9a according to the seventh embodiment includes an input program provided in the control unit 50 for causing the PC 1 to function as an input device, a timer for measuring the passage of time in the input program, and various tables for controlling the input program. The imaging device 9 of the eighth embodiment will be described.
With such a configuration, the imaging device 9 can be used as an input device of the personal computer by connecting the imaging device 9 to the USB connector 8 of the personal computer.
The operation status determination table 171, the input information table 172, and the pointing finger status table 173 are updated by providing the personal computer with a program for editing these tables and a program for transmitting these tables to the control circuit 9a. It ’s fine.

  A ninth embodiment in which the camera 3a and the camera 3b of the input device configured in the second embodiment are used as an external imaging device 9 will be described with reference to FIG.

First, a physical configuration and an electrical configuration will be described with reference to FIG.
The input device connects the imaging device 9 to the PC 1 via the USB connector 8, acquires the images of the cameras 3 a and 3 b by the control circuit 9 a of the imaging device 9, extracts finger information, and then extracts finger information. Is sent to the PC 1 via the USB connector 8.
In addition, Example 3 and Example 4 can also be made into the same structure. Further, one of the camera 3a and the camera 3b may be a one-dimensional image sensor.

  Even with such a configuration, the same operational effects as those of the second embodiment can be obtained, and use on various input operation surfaces is possible. For example, if the external imaging device 9 is installed on the operation surface on a desk, it can be used as a dedicated mouse device even in a usage environment with a lot of dust and oil.

  The control circuit 9a according to the ninth embodiment includes an input program provided in the control unit 50 for causing the PC 1 to function as an input device, a timer for measuring the passage of time in the input program, and various tables for controlling the input program. The imaging device 9 of Example 10 will be described.

With such a configuration, the imaging device 9 can be used as an input device of a personal computer by connecting the imaging device 9 to the USB connector 8.
The operation state determination table 171, the input information table 172, and the pointing finger state table 173 may be updated by providing the personal computer with a program for editing these tables and a program for transmission to the control circuit 9a. .

  Even with such a configuration, the same operational effects as those of the second embodiment can be obtained, and use on various input operation surfaces is possible. For example, if the external imaging device 9 is installed on the operation surface on a desk, it can be used as a dedicated mouse device even in a usage environment with a lot of dust and oil.

  26 to FIG. 30 regarding an eleventh embodiment in which the camera 3 and the camera 3a of the input device configured in the fifth embodiment are used as an external imaging device 9, and information equivalent to a keyboard can be input from the imaging device 9. Will be described with reference to FIG.

First, regarding the physical configuration, differences from the fifth embodiment will be described with reference to FIGS.
The input device connects the imaging device 9 to the PC 1 via the USB connector 8, acquires the video of the camera 3 and the camera 3a by the control circuit 9a of the imaging device 9, extracts finger information, and then extracts finger information. Is sent to the PC 1 via the USB connector 8.

  An input auxiliary sheet 4d (see FIGS. 26 and 27) showing an input information area arranged on an input surface used when inputting information equivalent to a keyboard or information equivalent to a numeric keypad from the image pickup device 9 is displayed on the camera 3. It is installed on the input operation surface so as to fall within the imaging area.

  The camera 3a has a position 3a2 in which the center 3a1 of the angle of view of the lens of the camera 3a is parallel to the position 4c immediately above the surface of the input auxiliary sheet 4d (see FIG. 26B) and the input auxiliary sheet 4d is equally divided into left and right. Whether the imaging area of the camera 3a includes the area 4a created by the camera 3 (see FIG. 27), and which finger of the left and right hand passes the position 4c directly above the input auxiliary sheet 4d surface Can be determined. The angle of view of the lens of the camera 3a is about 135 degrees.

Next, the outline of the operation procedure will be described with reference to FIG.
First, instead of the “keyboard input surface (OP-K)” provided in the fifth embodiment, a “QWERTY input surface (OP-Q)” for inputting the same information as the keyboard is used. Accordingly, an operation procedure for selecting a QWERTY input surface, an operation procedure for determining input information in a state where the QWERTY input surface is selected, and input information which can be input in a state where the QWERTY input surface is selected are determined. The operation procedure enables input of information equivalent to a mouse, information equivalent to a numeric keypad, information equivalent to a shortcut key, information equivalent to a character, and information equivalent to a keyboard.

  The operation procedure in the state where the QWERTY input surface or the numeric keypad information input surface is selected uses the pointing finger states obtained from the camera 3 and the camera 3a, and the mouse information input surface or waiting for scrolling is used. For the operation procedure in a state where the input screen or the input screen for continuing the left click is selected, the pointing finger state of the camera 3 is used.

Next, the details of the operation procedure will be described with reference to FIG.
Since the QWERTY input surface (OP-Q) is used instead of the keyboard input surface (OP-K), the operation instruction contents of M-9, S-4, and T-2 are changed.

  In the fifth embodiment, when inputting ten-key information, an operation instruction is issued when the pointing finger state of the right hand portion changes from “103” to “101” (T-1). However, when a plurality of fingers have passed the position 4c immediately above the input auxiliary sheet 4d surface and an input instruction is issued because a new finger has passed the position 4c, the operation state determination table 171 is entered. To register a combination of changes in the pointing finger state, a considerable load increases.

  Therefore, since the position coordinates on the image of the finger passing the position 4c immediately above the input auxiliary sheet 4d surface are known for each finger, the position coordinates of each finger on the previous image and the position coordinates of each finger of this time are obtained. In comparison, if the position coordinates are all the same except for the newly generated position coordinates, the operation state determination table 171 may be constructed with the pointing finger state of the camera 3a set to, for example, “999”.

For example, if the hand performing the input operation is seen from the camera 3a as shown in FIG. 30A, the one-dimensional video information at the position 4c immediately above the input auxiliary sheet 4d surface of the video information of the camera 3a is shown in FIG. 30B. The images are taken in the same way. From this state, when the index finger of the right hand is lowered and the position 4c immediately above the surface of the input auxiliary sheet 4d is passed, the one-dimensional image information of FIG. 30C is acquired.
Therefore, from the one-dimensional image information of FIG. 30B and FIG. 30C, the finger that has newly passed the position 4c immediately above the input auxiliary sheet 4d surface can be found, and the position coordinate of the index finger of the right hand (see FIG. 30C) can be found.

Therefore, the pointing finger state determined from the finger information extracted from the video of the camera 3 is other than “501” or “010”, and the pointing finger state determined from the finger information extracted from the video of the camera 3a is When “999”, it can be determined that information equivalent to that on the keyboard has been input (see K-8 in FIG. 29A).
The pointing finger states “001” and “017” are used for selection of the mouse information input screen and the scroll waiting input surface. If there is, it can be used as the pointing finger state of K-8.

  If it is determined that information equivalent to the keyboard has been input, the information on the input auxiliary sheet 4d surface is newly determined from the finger position coordinates with the pointing finger state set to “999” and the finger information extracted from the video of the camera 3. Since the two-dimensional coordinates of the finger over the position 4c immediately above can be determined, input information is determined from the input information table 172 in FIG. 29B.

The operation of the input device of the eleventh embodiment configured as described above, that is, the operations of the imaging device 9 and the PC 1 will be described with respect to differences from the flowchart of FIG. 24 of the seventh embodiment.
In S720, images are extracted from the camera 3 and the camera 3a.
In step S730, finger information is extracted from the captured video. For the camera 3a, one-dimensional image information at a position 4c immediately above the input auxiliary sheet 4d surface is extracted. Then, the extracted data is sent to the PC 1.
In step S740, the pointing finger state is determined by the pointing finger determination unit of the seventh embodiment, and the pointing finger state is determined for the finger information of the camera 3a.
Then, the two-dimensional coordinates of the finger for determining the input information are specified from the position coordinates of the finger on the image of the camera 3a and the two-dimensional coordinates obtained from the figure of the finger taken by the camera 3.
The camera 3a may be a one-dimensional image sensor 3c. At this time, it is not necessary to extract one-dimensional video information at the position 4c immediately above the input auxiliary sheet 4d.

The control circuit 9a according to the eleventh embodiment includes an input program provided in the control unit 50 for causing the PC 1 to function as an input device, a timer for measuring the passage of time in the input program, and various tables for controlling the input program. The imaging device 9 of Example 12 will be described. With such a configuration, the imaging device 9 can be used as an input device of a personal computer by connecting the imaging device 9 to the USB connector 8.
The operation state determination table 171, the input information table 172, and the pointing finger state table 173 may be updated by providing the personal computer with a program for editing these tables and a program transmitted to the control circuit 9a.

The thirteenth embodiment is an input device configured by combining the camera installation methods of the first and second embodiments, and will be described with reference to FIG.
The thirteenth embodiment includes a region 4a where the input surface is arranged by the camera 3 and a region 4a 'where the input surface is arranged by the camera 3a and the camera 3b.
By specifying the pointing finger state determined for each camera uniquely, the operation procedure of the first embodiment and the second embodiment can be freely selected.

  It is to be noted that either the camera 3a or the camera 3b, or both the camera 3a and the camera 3b may be a one-dimensional image sensor, as in the embodiment already described. In addition, in the input program and the input program provided in the control unit 50 for causing the imaging means and the PC 1 to function as an input device, a timer for measuring the passage of time and various tables for controlling the input program are provided separately. This may be the same as in the embodiment already described.

Since the input surface formed on the input operation surface of the input device described above is generated based on the video of the camera, the input operation surface necessarily has a mechanical configuration like the keyboard as described above. It does not need to be generated above, and can be generated on a simple plane or in a spatial domain.
The input device according to the present invention allows a personal computer by setting the input surface, the figure of the finger, the operation procedure, and the input information in advance and changing them arbitrarily as necessary, even with the same hardware and software. In addition to the keyboard device surface, various areas can be used as the input operation surface. Such an embodiment will be described below.
Therefore, first, a fourteenth embodiment in which the input surface is formed on the display screen will be described.
The imaging device 9 according to the ninth embodiment or the tenth embodiment is installed on a display device so that an input surface is formed on the display screen, and an input surface and an operation can be performed so that a pointing operation of the absolute position of the display screen can be performed. If the procedure, input information, and pointing finger state are determined and the operation state determination table 171, the input information table 172, and the pointing finger state table 173 are constructed, it can be used as a touch panel.
In addition, when installing in a large display, if the one-dimensional image sensor shown in Example 3 is used, an inexpensive touch panel can be constructed.
In addition, with an existing touch panel, an operation such as moving a page by pointing to a specified position on the screen was performed. However, an input screen for inputting function information, an operation procedure, input information, and an instruction finger are used. If the state is determined and the operation state determination table 171, the input information table 172, and the pointing finger state table 173 are constructed, an operation such as page movement can be performed at an arbitrary position on the screen.

Next, a fifteenth embodiment in which an input surface is formed on a desk will be described.
The operation procedure of the fifteenth embodiment (see FIG. 32) in which the imaging device 9 and the display of the twelfth embodiment are connected to the USB connector 8 of the main body 60 of the desktop personal computer is the same as that of the eleventh embodiment. With such a configuration, it is possible to input from the imaging device 9 instead of the key-press type keyboard device or mouse device. Therefore, it becomes easy to construct a dedicated keyboard device for business use.
Furthermore, if an input plane selection alternative means for selecting an input plane from the content indicated by the barcode from the barcode taken by the camera 3, it can also be used as a POS terminal.
The input plane selection alternative means is:
First, when an operation instruction for selecting an input surface is acquired using a barcode, the determination of the operation state in S061 in FIG. 9 is unconditionally set to the W operation state. Next, assuming that there is an input surface selection instruction, the input surface specified by the barcode is selected.

Next, Example 16 in which the input surface is formed on the desk will be described.
A sixteenth embodiment, which is a personal computer 70 having an input device using image recognition, in which the control unit 60 and the imaging means 9 of the fifteenth embodiment are integrated, will be described with reference to FIG.
The personal computer 70 includes a projection unit 20 that projects the display content 102 displayed on the display device on the wall 101 instead of the display device, and a position for selecting input information in conjunction with selection of the input surface when the input surface is selected. Input auxiliary information projecting means 21 is connected to project the input auxiliary information 4e for indicating to the desktop 100. The operation procedure is the same as that in the eleventh embodiment.
In such a form, in addition to the operational effects of the fifth embodiment, the sixth embodiment, the eleventh embodiment, and the twelfth embodiment, the personal computer can be miniaturized.
In addition, the input device in such a form includes an input surface for inputting information equivalent to an electronic piano, information equivalent to an electronic percussion instrument, control information of a mixer, input information of a game device, operation procedures, input information, and pointing finger status. And the operation state determination table 171, the input information table 172, and the pointing finger state table 173 are constructed, it can be used as an input device for an electronic musical instrument, audio equipment, or game device.

Next, Example 17 in which the input surface is formed on the desk will be described.
The imaging device 9 according to the seventh embodiment or the eighth embodiment is installed on the top so that the input surface is formed on the desk, and home appliances (for example, lighting power source, illuminance switch, various TV switches, operation of an air conditioner, etc.) The input surface, operation procedure, input information, and pointing finger state are determined for each switch), and the input surface, operation procedure, input information, and pointing finger state are determined for each game content, and the operation state determination table 171 and input information table 172 are determined. If the pointing finger state table 173 is constructed, it can be used as a device that comprehensively manages panel switches of various home appliances or an input device that comprehensively manages game inputs.

Next, Example 18 in which the input surface is formed on the desk will be described.
The imaging device 9 of Example 7 or Example 8 can be installed on a desk,
By installing the imaging device 9 of Example 9 or Example 10 on a desk,
An input surface is formed on the desk, an input surface for generating input information of absolute coordinates on the input surface, an operation procedure, input information, and a pointing finger state are determined, and an operation state determination table 171, an input information table 172, and an instruction If the finger state table 173 is constructed, it can be used as an input device for a tablet.

Next, Example 19 in which the input surface is formed on the wall will be described.
The imaging device 9 of Example 9 or Example 10 is installed on the entrance wall,
So that an input surface is formed on the wall,
For each switch of home appliances (for example, lighting power supply, illuminance switch, TV switch, air conditioner operation, etc.)
Determine the input surface, operation procedure, input information and pointing finger status for each input purpose of the immigration device (for example, interphone or PIN input)
If the operation state determination table 171, the input information table 172, and the pointing finger state table 173 are constructed, it can be used as a device that comprehensively manages panel switches of various home appliances and a device that comprehensively manages immigration devices.

Next, Example 20 in which the input surface is formed on the wall will be described.
The imaging device 9 of the eighth embodiment is connected to a personal computer connected to a projector, and an input surface for inputting information equivalent to input information on a touch panel or an electronic blackboard, an operation procedure, input information, and a pointing finger state are determined. By constructing the determination table 171, the input information table 172, and the pointing finger state table 173, the image on the wall projected by the projector can be used as a touch panel or an electronic blackboard. In addition, when using the imaging device 9 of Example 7, the resource for making a personal computer function as an input device should just be installed in a personal computer.

Next, Example 21 in which the input surface is formed on the existing device surface will be described. A form in which the input surface is formed on the keyboard device surface of the personal computer is one of these forms.
The image pickup apparatus 9 according to the eighth embodiment is installed so as to be able to take a picture on the steering wheel of the car. If the input information and the pointing finger state are determined, and the operation state determination table 171, the input information table 172, and the pointing finger state table 173 are constructed, it can be used as an input device that comprehensively manages the in-vehicle electronic device apparatus.
In addition, when using the imaging device 9 of Example 7, the resource for functioning an in-vehicle electronic device as an input device may be installed in the in-vehicle electronic device.

Next, a description will be given of a twenty-second embodiment in which the input surface is formed on the space.
If the digital camera is provided with the image pickup apparatus 9 of the eighth embodiment using the digital camera, the digital camera, for example, finds the pointing finger state “039” and operates in such a manner as to release the shutter 3 seconds later. The table 171 and the input information table 172 can be set, and the digital camera can be remotely operated by the method and timing set by the operator.
In addition, it is also possible to perform remote control by inputting input information in the same manner using a surveillance camera.

  The preferred embodiments of the present invention and specific applications thereof have been described above with reference to the drawings. However, the technical scope of the present invention is not limited to these, and departs from the scope of the claims described below. Without mentioning, some examples have been described in the present specification, but it goes without saying that various changes and modifications can be made.

  The present invention can be used as an information input device for various electronic devices such as computers, electronic musical instruments, game devices, and audio devices.

DESCRIPTION OF SYMBOLS 1 Personal computer 2 Display 3 Camera 3a Camera 3a1 Center of field angle 3a2 of a lens of a camera 3b The position which divides | segments a keyboard or an input auxiliary sheet | seat equally right and left 3b Camera 3c One-dimensional image sensor 3d One-dimensional image sensor 4 Keyboard 4a Formed range 4b Position 4c of each input information of numeric key information 4c Position 4d immediately above Input auxiliary sheet 5 Adjustment knob 6 Hand 6a Finger 6b Background 7 Base line 7a connecting camera 3a Straight line 7a1 connecting camera 3a and finger Base line 7 and straight line 7a The angle 7b formed by the camera 7b The straight line 7b1 connecting the finger to the camera 3b The angle 8 formed by the base line 7 and the straight line 7b USB connector 9 Imaging device 9a Control circuit 10 CPU
11 ROM
12 RAM
121 Previous coordinate 122 Elapsed time 123 Previous pointing finger state 124 Selected input surface 125 Scroll determination coordinate 126 Search table 127 Previous operation state 13 Timer 17 HDD
171 Operation state determination table 172 Input information table 173 Point finger state table

Claims (14)

An information input device for electronic equipment, the device comprising:
Imaging means for sequentially capturing images of a predetermined area of the input operation surface;
Finger information extracting means for extracting finger information from the image of the imaging means;
Based on the finger information from the finger information extraction means, the current finger position and finger position coordinates on the video are determined, and the finger position of the previous finger extraction information stored in the current finger position coordinates Determine the difference status from the coordinates, and determine the current pointing finger status from the pointing finger status table that associates the pointing finger status with the difference status between the figure of the finger and the position coordinates, which can be determined and changed in advance. And pointing finger determination means for calculating the two-dimensional coordinates of the current finger on the input surface formed at a predetermined position on the input operation surface from the current finger position coordinates;
At least one consisting of a combination of the pointing finger state, a time during which the pointing finger state continues, and an operation instruction with respect to the input surface formed at a predetermined position of the input operation surface, which can be determined and changed in advance. From an operation state determination table that associates the operation procedures of a pair or more, an operation state determination unit that determines an operation state according to the pointing finger state determined by the pointing finger determination unit,
According to the operation state determined by the operation state determination means, an operation procedure execution means for performing an operation instruction for selecting an input surface or generating input information;
An information input device for electronic equipment, comprising:   The information input device according to claim 1, wherein the imaging unit includes a two-dimensional image sensor.   The information input device according to claim 1, wherein the imaging unit includes at least two one-dimensional image sensors provided in a direction transverse to the input surface, and two-dimensional coordinates on the input surface are An information input device for electronic equipment, which is calculated by a triangulation method.   The information input device according to claim 1, wherein the imaging unit is configured by a combination of a two-dimensional image sensor and a one-dimensional image sensor.   The information input device according to claim 1, wherein the imaging unit includes an irradiating unit for irradiating a predetermined region of the input operation surface in the vicinity thereof.   The information input device according to claim 1, wherein the pointing finger determination unit recognizes the appearance of the operator's finger as the presence or absence of a gap between the fingers, the number of fingers, or the number of fingers and the width of the fingers. An information input device for an electronic device. The information input device according to claim 1, wherein the operation procedure execution unit associates a plurality of input information that is predetermined and changeable with respect to the input surface, the input information generation instruction, and two-dimensional coordinates. Input information table,
If the content of the operation instruction from the operation state determination means is an input surface selection instruction, change the currently selected input surface according to the input surface selection instruction,
If the content of the operation instruction from the operation state determination means is an instruction to generate input information, on the currently selected input surface,
Generating input information determined by an operator in response to the input surface and the input information generation instruction; or
The input information is generated from the input information table from the input surface, the input information generation instruction, and the two-dimensional coordinates of the finger.
An information input device for an electronic device.   The information input device according to claim 1, wherein the input surface formed at a predetermined position of the input operation surface includes at least a keyboard input surface, a numeric keypad information input surface, and a mouse information input surface. The information input device for electronic equipment is characterized in that the input surfaces are switched to each other by the operating procedure execution means.   9. The information input device according to claim 8, wherein an input surface formed at a predetermined position of the input operation surface is further left-clicked to be switched between the mouse information input surface and the operation procedure execution means. An information input device for an electronic device, comprising an input surface for continuing pressing and an input surface waiting for scrolling. A method of inputting information to an electronic device, the method comprising:
An imaging process for sequentially capturing images of a predetermined area of the input operation surface;
Finger information extraction process for extracting finger information from the video obtained in the imaging process;
Based on the finger information obtained in the finger information extraction process, the current finger shape and the finger position coordinates on the video are determined, and the finger information of the previous finger extraction information stored in the current finger position coordinates is stored. The current pointing finger state from the pointing finger state table in which the pointing finger state is associated with the difference between the figure of the finger and the position coordinate, which can be determined, changed and determined in advance, And a pointing finger determination process for calculating the two-dimensional coordinates of the current finger on the input surface formed at a predetermined position on the input operation surface from the current finger position coordinates;
At least one consisting of a combination of the pointing finger state, a time during which the pointing finger state continues, and an operation instruction with respect to the input surface formed at a predetermined position of the input operation surface, which can be determined and changed in advance. From an operation state determination table that associates a set of operation procedures or more, an operation state determination process for determining an operation state according to the pointing finger state determined by the pointing finger determination process,
In accordance with the operation state determined by the operation state determination process, an operation procedure execution process for performing an operation instruction for selecting an input surface or generating input information;
An information input method for an electronic device, comprising:   11. The information input method according to claim 10, wherein the imaging process is performed using a two-dimensional image sensor.   The information input method according to claim 10, wherein the imaging process uses at least two one-dimensional image sensors provided laterally with respect to the input surface, and the two-dimensional coordinates on the input surface are triangular. An information input method to an electronic device, characterized by being calculated by a surveying method.   11. The information input method according to claim 10, wherein the imaging process is performed by a combination of a two-dimensional image sensor and a one-dimensional image sensor.   11. The information input method according to claim 10, wherein in the pointing finger determination process, the figure of the operator's finger is recognized as the presence / absence of a gap between the fingers, the number of fingers, or the number of fingers and the width of the fingers. A method for inputting information to an electronic device.