JP4922266B2 - Input device and input method - Google Patents

Description

  The present invention relates to an input device and an input method for a user to input a command or the like to a device, and more specifically, using a body part such as a hand based on information displayed on a display or the like by a user. The present invention relates to an input device and an input method capable of inputting commands and the like.

  With the recent increase in functionality of information equipment terminals, user interfaces that are easier for users to use are required. An example of an easy-to-use user interface is a touch panel display. A touch panel display is a device in which a user inputs a command or the like to an information device terminal or the like by manually pushing a GUI (Graphical User Interface) part displayed on the display. From this, it can be said that the touch panel display is an interface that even a beginner can operate intuitively.

  However, in a touch panel display, since a device for detecting touch (hereinafter referred to as an input unit) and a display unit are integrated, there is a problem that the touch panel display cannot be operated unless the user has it.

In order to solve this problem, the technique of Patent Document 1 separates an input unit and a display unit, detects a user's hand placed on the input unit, and forms a hand shape model of computer graphics (hereinafter referred to as CG). Are superimposed on the GUI displayed on the display unit. Thus, even when the user is away from the display unit, the user can perform an operation while maintaining an intuitive operational feeling of the touch panel display.
JP 2006-72854 A

  The above-described conventional technology uses the input unit (touch panel) that can simultaneously acquire a plurality of contact positions between the user's hand and the operation surface of the input unit, and based on the detected fingertip position and palm position of the user's hand. The hand shape is restored. In the conventional technique described above, when the user's fingertip is not in contact with the operation surface of the input unit, the fingertip position is not detected, and the user's hand shape cannot be faithfully reproduced. As a result, the above-described conventional technique has a problem that the shape of the hand shape model restored by the CG is different from the shape of the user's hand, making intuitive operation difficult.

  Therefore, an object of the present invention is to provide an input device and an input method that can faithfully reproduce a user's hand shape as a hand shape model even when there is a user's fingertip that is not in contact with the operation surface of the input unit. Is to provide.

  The present invention is directed to an input device that displays a CG model of a user's hand placed on an operation surface on a display unit and gives an instruction of a user input using this display to a device. In order to achieve the above object, the input device of the present invention has an operation surface, a contact region detection unit that detects a contact region between the operation surface and a hand, and an operation surface based on the contact region. A position coordinate data creation unit that creates a plurality of fingertip position points indicating the fingertip position of the touching hand at predetermined time intervals, and a pseudo fingertip position that indicates a pseudo fingertip position point indicating a fingertip position that is not currently in contact with the operation surface Based on the position creation unit, the fingertip position point, and the pseudo fingertip position point, a model shape creation unit that creates a CG model reflecting the shape of the hand, and a GUI part holding that holds a GUI part to which an instruction given to the device is assigned A superimposed image creating unit that creates an image in which the CG model created in the model shape creating unit and the GUI part held in the GUI part holding unit are superimposed and displayed on the display unit, GUI part and CG model Based on the positional relationship between, and an operation content determining unit that determines a user's instruction to be given to devices.

  Thereby, since the input device of the present invention can estimate the position of the user's fingertip that is not in contact with the operation surface, the user's hand shape and the shape of the hand shape model drawn by CG can be matched.

  Preferably, the position coordinate data creation unit further creates a palm position point indicating the palm position of the hand currently in contact with the operation surface at predetermined time intervals based on the contact area detected by the contact area detection unit. Then, the model shape creation unit further creates a CG model reflecting the shape of the hand based on the palm position point.

  Thereby, the input device of the present invention can detect the palm position of the user's hand and reflect it in the shape of the CG model, so that a more accurate CG model of the user's hand can be created.

  In addition, the pseudo fingertip position creation unit creates a pseudo fingertip position point by setting the nearest fingertip position point among the past fingertip position points indicating the fingertip position at the time of touching the operation surface as the pseudo fingertip position point. May be.

  Thereby, the input device of the present invention can create a pseudo fingertip position point corresponding to a fingertip that is initially in contact with the operation surface and is not currently in contact.

  Preferably, the model shape creation unit calculates a motion vector of each fingertip position point, moves the pseudo fingertip position point based on an average motion vector obtained by averaging the calculated motion vectors, and reflects the shape of the hand. Create a CG model.

  As a result, the input device of the present invention can estimate the position of the moved fingertip even when the fingertip not in contact with the operation surface moves, so that an accurate CG model of the user's hand can be created.

  Further, the pseudo fingertip position creating unit may create the pseudo fingertip position points between adjacent fingertip position points whose mutual distance exceeds a predetermined value.

  Thereby, the input device of the present invention can create a pseudo fingertip position point corresponding to a fingertip that is not in contact with the operation surface from the beginning.

  In addition, the model shape creation unit may create a CG model reflecting the shape of the hand by moving the pseudo fingertip position point so that it is positioned in the middle of the fingertip position points located on both sides of the pseudo fingertip position point. .

  As a result, the input device of the present invention can estimate the position of the moved fingertip even when the fingertip not in contact with the operation surface moves, so that an accurate CG model of the user's hand can be created.

  The present invention is also directed to an input method in which a CG model of a user's hand placed on an operation surface is displayed on a display unit and a user's instruction input using this display is given to a device. In order to achieve the above object, the input method of the present invention includes a contact area detection step for detecting a contact area between the operation surface and the hand, and a fingertip of the hand currently in contact with the operation surface based on the contact area. A position coordinate data creation step for creating a plurality of fingertip position points indicating positions every predetermined time; a pseudo fingertip position creation step for creating a pseudo fingertip position point indicating a fingertip position not currently in contact with the operation surface; and a fingertip position A model shape creation step for creating a CG model reflecting the shape of the hand based on the point and the pseudo fingertip position point, a CG model created in the model shape creation step, and a GUI part to which an instruction to be given to the device is assigned Based on the superimposed image creation step of creating a superimposed image and displaying it on the display unit, and the positional relationship between the GUI part and the CG model, the user instruction to be given to the device is determined. And an operation content determining step that.

  As a result, the input method of the present invention can estimate the position of the fingertip of the user who is not in contact with the operation surface, so that the user's hand shape and the shape of the hand shape model drawn by CG can be matched.

  According to the input device and the input method of the present invention, since the position of the user's fingertip that is not in contact with the operation surface can be estimated, it is possible to match the shape of the user's hand with the shape of the hand shape model drawn by CG. it can. As a result, the user can perform an operation without feeling uncomfortable.

(One embodiment)
Hereinafter, an input device according to an embodiment of the present invention will be described with reference to the drawings. In each drawing, elements that are not particularly important for implementation of the present invention are omitted in view of visibility.

  FIG. 1 is a diagram illustrating a hardware configuration example of an input device 100 according to an embodiment of the present invention. As shown in FIG. 1, the input device 100 includes a contact area detection unit 1 and a calculation unit 2. The calculation unit 2 includes an I / O 11 that inputs and outputs data, an HDD (Hard Disk Drive) 12 that stores data, programs, and the like, a ROM 13 that stores basic programs such as BIOS, and programs and programs stored in the HDD 12. A CPU 14 that executes a program stored in the ROM 13, a RAM 15 that holds data, an image processor 17 that processes image data, a VRAM 18 that holds image data, and a bus line 16 that connects the above components. .

  The contact area detection unit 1 is connected to the input side of the I / O 11 and detects the contact area of the user's hand 19 in contact with the operation surface at multiple points. The detected contact area position data (hereinafter referred to as contact position data). ). Contact position data input from the contact area detector 1 to the input side of the I / O 11 is held in the RAM 15.

  The display unit 3 is connected to the output side of the I / O 11. The display unit 3 is a display that displays an image corresponding to the image data. Further, an operation target device 4 such as a car navigation device, an audio device, and an air conditioner operated by the input device 100 is connected to the output side of the I / O 11. The calculation unit 2 outputs an instruction to the operation target device 4 based on the user instruction detected by the contact area detection unit 1.

  The HDD 12 deforms a standard hand shape model (hereinafter referred to as a standard hand model) held in the RAM 15 based on the contact position data held in the RAM 15, and the image data of the deformed hand shape model together with the GUI. A control program that causes the CPU 14 to function so as to be displayed on the display unit 3 is stored.

  Note that the control in the input device 100 is executed by hardware or by cooperation of hardware and software.

  Hereinafter, the configuration and operation of the input device 100 will be described in more detail. FIG. 2 is a diagram illustrating an example of functional blocks of the input device 100. As illustrated in FIG. 2, the input device 100 includes a contact area detection unit 1 and a calculation unit 2. The calculation unit 2 includes a position coordinate data creation unit 21, a pseudo fingertip position creation unit 22, a model shape creation unit 23, a hand shape model holding unit 24, a GUI parts holding unit 25, a superimposed image creation unit 26, An operation content determination unit 27 and a command transmission unit 28 are included.

  The position coordinate data creation unit 21 generates a fingertip position point and a palm position point from the contact position data input from the contact area detection unit 1.

  The pseudo fingertip position creation unit 22 generates a pseudo fingertip position point based on the fingertip position point generated by the position coordinate data creation unit 21.

  The hand shape model holding unit 24 holds a standard hand model in advance.

  The model shape creation unit 23 reads the standard hand model from the hand shape model holding unit 24. Then, the model shape creating unit 23 deforms the standard hand model based on the fingertip position point generated by the position coordinate data creating unit 21 and the pseudo fingertip position point generated by the pseudo fingertip position creating unit 22, thereby generating a contact region. An image of a hand shape model reflecting the shape of the user's hand placed on the operation surface of the detection unit 1 is created.

  The GUI parts holding unit 25 holds GUI parts included in the operation image displayed on the display unit 3. The GUI parts held in the GUI parts holding unit 25 are, for example, operation buttons for operating the operation target device 4.

  The superimposed image creating unit 26 creates an image of the GUI part read from the GUI part holding unit 25, superimposes the image of the hand shape model created by the model shape creating unit 23 on the created image, and displays it as an operation image. 3 is displayed.

  The operation content determination unit 27 detects an operation (instruction) performed by the user using the operation image displayed on the display unit 3.

  The command transmission unit 28 transmits the user instruction detected by the operation content determination unit 27 to the operation target device 4.

  FIG. 3 is a flowchart illustrating an example of the control process of the input device 100. Hereinafter, the control processing of the input device 100 will be described with reference to FIG. The following processing is executed by the CPU 14 shown in FIG. 1 by expanding the control program stored in the HDD 12 in the RAM 15.

  First, in step S <b> 301, the position coordinate data creation unit 21 acquires contact position data from the contact area detection unit 1. The contact position data is data representing a region where the user's hand has touched the operation surface of the contact region detection unit 1 by a point group (hereinafter referred to as a contact point group).

  Next, in step S302, the position coordinate data creation unit 21 obtains a fingertip position point and a palm position point using the acquired contact point group. FIG. 4 is a diagram for explaining the fingertip position point and the palm position point obtained in step S302. As shown in FIG. 4A, the position coordinate data creation unit 21 designates a plurality of regions by enclosing the contact point group acquired from the contact region detection unit 1 with an ellipse. Then, for example, the position coordinate data creation unit 21 determines that the contact point group indicated by the lowest ellipse B is the palm contact point group, and selects the contact point group above the palm contact point group as the finger. Judged as a contact point group. Thereafter, the position coordinate data creation unit 21 uses, for example, the upper tip of the ellipse A as a fingertip position point (see FIG. 4B) and the center of gravity of the ellipse B as a palm position point (FIG. 4C). See). Note that the fingertip position point and the palm position point may be determined by using other methods or other positions. FIG. 4D shows the fingertip position points 41 to 45 and the palm position point 46 obtained by the method described above.

  Next, in step S303, the position coordinate data creation unit 21 assigns IDs to the obtained fingertip position point and palm position point based on how fingers are arranged (hand structure). For example, as shown in FIG. 4D, when five fingertip position points are obtained in step S302, IDs indicating the thumb, index finger, middle finger, ring finger, and little finger are respectively displayed at the fingertip position points 41 to 45. And an ID indicating a palm is assigned to the palm position point 46. Here, the above-described processing assumes that the user operates the contact area detection unit 1 with the right hand. Therefore, for example, it is preferable to notify the user that the right hand is placed on the operation surface of the contact area detection unit 1 using voice, display, or the like.

  Next, in step S304, the position coordinate data creation unit 21 acquires contact position data from the contact area detection unit 1 at predetermined time intervals, and acquires fingertip position points and palm position points at predetermined time intervals as in step S302. To do. Then, the position coordinate data creation unit 21 associates the fingertip position point and palm position point acquired at predetermined time intervals with the IDs of the thumb, index finger, middle finger, ring finger, little finger, and palm assigned in step S303, respectively. To accumulate. More specifically, the position coordinate data creation unit 21 calculates the distances between all the current fingertip position points and palm position points for each of the past fingertip position points and palm position points at predetermined time intervals. The current fingertip position point (or palm position point) of the closest distance is recognized as the fingertip position point (or palm position point) of the same ID and paired. Accordingly, the position coordinate data creation unit 21 can associate the ID with the fingertip position point and the palm position point acquired every predetermined time.

  Next, in step S305, the pseudo fingertip position creation unit 22 creates a pseudo fingertip position point. Here, the pseudo fingertip position point is a fingertip position point that represents the position of the fingertip that is not in contact with the operation surface of the contact area detection unit 1 (the position of the fingertip that is floating from the operation surface).

  FIG. 5 is a flowchart for explaining the processing in step S305. Hereinafter, the process of step S305 will be described in detail with reference to FIG.

  First, in step S501, the pseudo fingertip position creation unit 22 recognizes the fingertip position that has been lifted by using the information on the fingertip position points for each predetermined time accumulated in the RAM 15 in step S304, and whether or not there is a fingertip that has been lifted. Determine. Specifically, the pseudo fingertip position creation unit 22 recognizes that the fingertip is lifted for the past fingertip position points accumulated in the RAM 15 in step S304 that do not have the current fingertip position point to be paired. It is determined that there is a floating fingertip. If it is determined that there is a floating fingertip, the process proceeds to step S502. If it is determined that there is no floating fingertip, the process moves to step S504.

  In step S502, the pseudo fingertip position creation unit 22 sets the last past fingertip position point corresponding to the floating fingertip as the pseudo fingertip position point of the floating fingertip. FIG. 6 is a diagram for explaining the processing in step S502. FIG. 6 shows, as an example, a case where the middle finger that is in contact with the operation surface of the contact area detection unit 1 is lifted. In FIG. 6A, the five broken lines indicate the past fingertip position point locus 602, the four black circles each indicate the current fingertip position point 601, and one white circle immediately before the middle finger is not detected because the middle finger is lifted. The fingertip position point 603 is shown. The palm position point is omitted. In step S502, the pseudo fingertip position creation unit 22 sets the fingertip position point 603 immediately before it is no longer detected as a pseudo fingertip position point 604 as shown in FIG. 6B.

  Next, in step S <b> 503, the pseudo fingertip position creation unit 22 stores the pseudo fingertip position point set in step S <b> 502 in the RAM 15.

  Next, in step S504, the pseudo fingertip position creation unit 22 determines whether there are five total fingertip position points (which may include pseudo fingertip position points) accumulated in the RAM 15. That is, in step S504, the pseudo fingertip position creation unit 22 determines whether all of the five fingertips have been detected as fingertip position points or pseudo fingertip position points. If there are five points, the process moves to step S306 shown in FIG. If five points do not exist, the process moves to step S505.

  In step S505, the pseudo fingertip position creation unit 22 calculates the distance between adjacent fingertip position points. Here, some of the fingertip position points may be pseudo fingertip position points set in step S502.

  Next, in step S506, the pseudo fingertip position creation unit 22 determines whether or not each of the distances calculated in step S505 is equal to or greater than a predetermined threshold value. If there is a distance greater than or equal to the predetermined threshold, the process moves to step S507. If there is no distance equal to or greater than the predetermined threshold, the process moves to step S306 shown in FIG.

  In step S <b> 507, the pseudo fingertip position creation unit 22 sets a pseudo fingertip position point between two adjacent fingertip position points determined to be a distance equal to or greater than a predetermined threshold. Thereafter, the process proceeds to step S306 shown in FIG.

  FIG. 7 is a diagram for explaining the processing of steps S505 to S507 shown in FIG. Hereinafter, the process of steps S505 to S507 will be specifically described with reference to FIG. 7 by taking as an example a case where the middle finger is floating from the beginning.

  First, in step S505, as shown in FIG. 7A, the fingertip position points 701 to 704 (some of them may be pseudo fingertip position points set in step S502), between adjacent fingertip position points. Distances 711 to 713 are respectively calculated. More specifically, a distance 711 between the fingertip position point 701 corresponding to the thumb and the fingertip position point 702 corresponding to the index finger is calculated, and the fingertip position point 702 corresponding to the index finger and the fingertip position point 703 corresponding to the ring finger are calculated. The distance 712 between the fingertip position point 703 corresponding to the ring finger and the fingertip position point 704 corresponding to the little finger is calculated.

  Next, in step S506, each of the calculated distances 711 to 713 is compared with a predetermined threshold stored in the RAM 15 in advance. Here, the predetermined threshold values stored in advance in the RAM 15 are a1 to a4. The threshold value a1 is the maximum value that can normally be taken as the distance between the thumb and the index finger. Similarly, the threshold value a2 is the distance between the index finger and the middle finger, the threshold value a3 is the distance between the middle finger and the ring finger, and the threshold value a4 is the maximum value that is normally possible as the distance between the ring finger and the little finger. In consideration of the structure of the hand, it is preferable that the magnitude relationship between the threshold values is set so that a1> a4> a2 to a3. Specifically, the distance 711 between the fingertip position point 701 corresponding to the thumb and the fingertip position point 702 corresponding to the index finger is compared with the threshold value a1. Similarly, a distance 712 between the fingertip position point 702 corresponding to the index finger and the fingertip position point 703 corresponding to the ring finger is compared with the threshold value a2. Here, since the threshold value a2 is the maximum value that is normally possible as the distance between the index finger and the middle finger, the possibility that the distance 712 is greater than or equal to the threshold value a2 is high. If it is determined that the distance 712 is greater than or equal to the threshold value a2, it is detected that the middle finger is floating, and the process proceeds to step S507.

  Next, in step S507, as shown in FIG. 7B, between the fingertip position point 702 corresponding to the index finger and the fingertip position point 703 corresponding to the ring finger, the pseudo fingertip position point corresponding to the floating middle finger. 730 is created and stored in the RAM 15.

  If it is not determined in step S506 that the distance 712 is greater than or equal to the threshold value a2, it is not detected that the middle finger is floating. In this case, the pseudo fingertip position point corresponding to the floating middle finger is not created.

  As described above, the pseudo fingertip position point is set for the fingertip that has not floated at the beginning but has floated in the middle by the processing of steps S501 and S502. In addition, the pseudo fingertip position point is set for the fingertip that has floated from the beginning by the processing of steps S505 to S507.

  Hereinafter, returning to the flowchart of FIG. 3, the description will be continued. In step S306, the model shape creating unit 23 creates the fingertip position point and the palm position point acquired every predetermined time accumulated in the RAM 15 in step S304, and the pseudo fingertip position creating unit 22 creates in the RAM 15 in step S305. Using the stored pseudo fingertip position points, the corresponding fingertip position points (even if some of them are pseudo fingertip position points) of the standard hand model which is a model held by the hand shape model holding unit 24 from the beginning. The hand shape model is created by deforming the standard hand model so that the palm position point is good).

  FIG. 8 is a diagram for explaining an example of a method in which the model shape creation unit 23 deforms the standard hand model when a pseudo fingertip position point exists. Hereinafter, with reference to FIG. 8, a detailed description will be given of a method in which the model shape creation unit 23 creates a hand shape model by deforming the standard hand model when a pseudo fingertip position point exists in step S306.

  A case where four fingertip position points 601 move as shown in FIG. 8C will be described. Here, as already described, the fingertip position point 601 corresponds to a fingertip that is not floating (detected fingertip), and the pseudo fingertip position point 604 corresponds to a fingertip that is floating (undetected fingertip). . As shown in FIG. 8C, the model shape creation unit 23 calculates a motion vector of each fingertip position point 601 and averages the calculated motion vectors to calculate an average motion vector. Thereafter, as shown in FIG. 8D, the model shape creating unit 23 moves the pseudo fingertip position point 604 using the calculated average motion vector. By this method, the movement of the floating fingertip can be reflected in the hand shape model, so that a shape close to the actual hand shape can be reproduced by CG.

  FIG. 9 is a diagram for explaining another example of a method in which the model shape creation unit 23 deforms the standard hand model when a pseudo fingertip position point exists. FIG. 9A shows a pseudo fingertip position point 1010 corresponding to a floating middle finger and fingertip position points 1001 to 1004 as an example. Consider a case where the fingertip position point 1002 moves downward as shown in FIG. 9B when the user bends the index finger from the state shown in FIG. 9A. In this case, the model shape creation unit 23 moves the pseudo fingertip position point 1010 to the center position between the fingertip position point 1002 and the fingertip position point 1003 after the movement, as shown in FIG. That is, the model shape creating unit 23 moves the pseudo fingertip position point so as to be positioned at the center of the two fingertip position points located on both sides of the pseudo fingertip position point. By this method, the movement of the floating fingertip can be reflected in the hand shape model, so that a shape close to the actual hand shape can be reproduced by CG. Here, the above-described method uses a hand structure in which the other four fingers except the thumb move in conjunction with each other. Therefore, it is preferable to apply the above-described method when the middle finger or the ring finger is floating.

  Next, in step S307, the superimposed image creation unit 26 superimposes the hand shape model created by the model shape creation unit 23 in step S306 on the image constituted by the GUI parts read from the GUI parts holding unit 25, and the user Is displayed on the display unit 3 as an operation image used for the operation.

  Next, although omitted in the flowchart of FIG. 3, the operation content determination unit 27 detects the operation performed by the user by viewing the operation image displayed on the display unit 3 in step S <b> 307. Specifically, the operation content determination unit 27 detects, for example, a collision between a GUI part in the operation image and the fingertip of the hand shape model as a user operation. Thereafter, the command transmission unit 28 transmits the user operation detected by the operation content determination unit 27 to the operation target device 4. Through the above operation, the user can operate the operation target device 4 using the input device 100.

  Hereinafter, a method of obtaining the bending angle of the finger joint in the hand shape model created in step S306 of FIG. 3 will be described. The hand shape model created in step 306 is appropriately deformed according to the position change of the fingertip position point (some may be a pseudo fingertip position point) and the palm position point. Here, there is an inverse kinematics (hereinafter referred to as IK) technique known in the field of robot engineering or the like as a method for obtaining the bending angle of the finger joint from the displacement amount of the fingertip position point. The IK technique is used to move the tip of an arm having a plurality of movable parts to a target position. When the arm has a plurality of movable parts, there are a plurality of solutions for the bending angle of the movable part in order to move the tip of the arm to the target position using the IK technique.

  Also in the present invention, since the finger has a plurality of joints, there are a plurality of solutions for the bending angles of the finger joints. For this reason, in the present invention, as an example, a solution is obtained using a constraint condition that the palm and fingertip of the user exist on the operation surface (on the same plane) and a constraint condition that the bending angles of the finger joints are equal. Find uniquely. FIG. 10 shows a constraint condition that a fingertip position point (some may be pseudo fingertip position points) and a palm position point exist on the same plane, and a constraint condition that the bending angles of each finger joint are equal. It is a figure which shows an example of the deform | transformed hand shape model. FIG. 10A shows the displacement of the fingertip position point when the index finger is bent as an example. FIG.10 (b) is the figure which looked at the hand shape model of the index finger of the bent state shown to Fig.10 (a) from the side surface. As can be seen from FIG. 10, since the three finger joints usually bend at the same time when the finger is bent, when the hand shape model is created by deforming the standard hand model using such constraint conditions, the user can operate without any sense of incongruity. It can be carried out. Further, if the standard hand model is deformed using such a constraint condition, the amount of calculation can be greatly reduced, and therefore, deformation that responds instantaneously to the movement of the user's hand is possible. FIG. 11 is a side view of the standard hand model and the created hand shape model. The standard hand model shown in FIG. 11A can be deformed as shown in FIG. 11B based on the angle of each joint acquired from the displacement amount of the position for each finger.

  As described above, when the user's fingertip that was initially in contact with the operation surface of the contact area detection unit 1 floats, the input device 100 according to the present invention estimates the position of the floating fingertip and simulates the pseudo fingertip. Set the position point. Moreover, the input device 100 of the present invention estimates the position of the fingertip of the user who is not in contact with the operation surface of the contact area detection unit 1 from the beginning, and sets the pseudo fingertip position point. Then, the input device 100 according to the present invention estimates the movement of the set pseudo fingertip position point, and the shape of the floating finger on the hand shape model (CG model) that reproduces the user's hand shape displayed on the operation screen. Reflect. As a result, according to the input device 100 of the present invention, the shape of the user's hand and the shape of the CG model can be matched, and the user can operate without a sense of incongruity.

  In the above, the pseudo fingertip position point is set for the fingertip that has floated from the beginning by the processing of steps S505 to S507 in FIG. 5 (see FIG. 7). However, in this process, when the end finger (thumb or little finger) has floated from the beginning, it is difficult to set the pseudo fingertip position point corresponding to the end finger. Therefore, for example, the pseudo fingertip position point when the thumb or little finger has floated from the beginning may be set by the processing described below with reference to FIG.

  FIG. 12A is a diagram illustrating a case where a pseudo fingertip position point 903 corresponding to a thumb that has floated from the beginning is set. The pseudo fingertip position creation unit 22 recognizes a group of fingertip position points 901 as shown by an ellipse in FIG. This can be achieved, for example, by identifying the fingertip position point 901 whose interval from the adjacent fingertip position point 901 is equal to or smaller than a predetermined interval as one group. Here, because of the structure of the hand, the fingertips of the thumb are usually located away from the group of fingertips of other fingers. From this, when the thumb is floating, there is a high possibility that one group consisting of the four fingertip position points 901 is recognized as shown in FIG. By utilizing this fact, when one group of four fingertip position points 901 is recognized, the pseudo fingertip position creation unit 22 performs pseudo-corresponding to the thumb in the lower left direction of the fingertip position point 901 corresponding to the index finger. A fingertip position point 903 is set. As a result, the pseudo fingertip position creation unit 22 can set the pseudo fingertip position point 903 corresponding to the thumb that has floated from the beginning. Here, the distance between the pseudo fingertip position point 903 corresponding to the thumb and the fingertip position point 901 corresponding to the index finger is, for example, 2 of the distance between the fingertip position point 901 corresponding to the index finger and the fingertip position point 901 corresponding to the middle finger. Double it.

  FIG. 12B is a diagram illustrating a case where the pseudo fingertip position point 903 corresponding to the little finger that has floated from the beginning is set. The pseudo fingertip position creation unit 22 recognizes a group of fingertip position points 901 as shown by an ellipse in FIG. Here, the fingertip position point 901 corresponding to the thumb is usually separated from the fingertip position point 901 corresponding to the index finger by a predetermined distance or more. From this, as shown in FIG. 12B, the fingertip position point 901 corresponding to the thumb is not normally recognized as a group of fingertip position points 901. Using this fact, when one group of three fingertip position points 901 and one fingertip position point are recognized, the pseudo fingertip position creation unit 22 lowers the right of the fingertip position point 901 corresponding to the ring finger. A pseudo fingertip position point 903 corresponding to the little finger is set in the direction. Accordingly, the pseudo fingertip position creation unit 22 can set a pseudo fingertip position point 903 corresponding to the little finger that has floated from the beginning. Here, the distance between the pseudo fingertip position point 903 corresponding to the little finger and the fingertip position point 901 corresponding to the ring finger is equal to the distance between the fingertip position point 901 corresponding to the ring finger and the fingertip position point 901 corresponding to the middle finger, for example. Good.

  The input device of the present invention can be used as an input device for a user to input a command to a device, and is particularly useful when it is desired to realize an operational feeling while maintaining an intuitive operational feeling of a touch panel display. It is.

The figure which shows the hardware structural example of the input device 100 which concerns on one Embodiment of this invention. The figure which shows an example of the functional block of the input device 100 which concerns on one Embodiment of this invention. The flowchart which shows an example of the control processing of the input device 100 which concerns on one Embodiment of this invention. The figure for demonstrating the fingertip position point and palm position point calculated | required by step S302 of the flowchart shown in FIG. The flowchart for demonstrating the process of step S305 of the flowchart shown in FIG. The figure for demonstrating the process of step S502 of the flowchart shown in FIG. The figure for demonstrating the process of step S505-S507 of the flowchart shown in FIG. The figure for demonstrating an example of the method in which the model shape preparation part 23 deform | transforms a standard hand model when a pseudo fingertip position point exists. The figure for demonstrating another example of the method in which the model shape preparation part 23 deform | transforms a standard hand model when a pseudo fingertip position point exists. A hand shape deformed using a constraint condition that a fingertip position point (a part of the fingertip position point may be a pseudo fingertip position point) and a palm position point exist on the same plane and a constraint condition that the bending angles of each finger joint are equal. Diagram showing an example of a model View of standard hand model and created hand shape model from the side The figure which shows the case where the pseudo fingertip position point 903 corresponding to the thumb or little finger which floated from the beginning is set.

Explanation of symbols

1 Contact Area Detection Unit 2 Calculation Unit 11 I / O
12 HDD
13 ROM
14 CPU
15 RAM
16 bus line 17 image processor 18 VRAM
DESCRIPTION OF SYMBOLS 21 Position coordinate data creation part 22 Pseudo fingertip position creation part 23 Model shape creation part 24 Hand shape model holding part 25 GUI parts holding part 26 Superimposed image creation part 27 Operation content determination part 28 Command transmission part 41-45, 601, 603, 701 to 704, 720, 901, 1001 to 1004 Fingertip position point 46, 604, 730, 903, 1010 Palm position point 100 Input device

Claims (7)

An input device that displays a CG model of a user's hand placed on an operation surface on a display unit and gives an instruction of the user input using the display to the device,
A contact area detection unit that has the operation surface and detects a contact area between the operation surface and the hand;
Based on the contact area, a position coordinate data creation unit that creates a plurality of fingertip position points indicating the fingertip position of the hand that is currently in contact with the operation surface at a predetermined time;
A pseudo fingertip position creation unit that creates a pseudo fingertip position point indicating a fingertip position that is not currently in contact with the operation surface;
A model shape creation unit that creates a CG model reflecting the shape of the hand based on the fingertip position point and the pseudo fingertip position point;
A GUI part holding unit for holding a GUI part to which an instruction to be given to the device is assigned;
A superimposed image creating unit that creates an image in which the CG model created in the model shape creating unit and the GUI part held in the GUI part holding unit are superimposed and displayed on the display unit;
An input device comprising: an operation content determination unit that determines a user instruction to be given to the device based on a positional relationship between the GUI part and the CG model. The position coordinate data creation unit further creates a palm position point indicating the palm position of the hand currently in contact with the operation surface at predetermined time intervals based on the contact area detected by the contact area detection unit. ,
The input device according to claim 1, wherein the model shape creation unit further creates a CG model reflecting the shape of the hand based on the palm position point.   The pseudo fingertip position creation unit sets the nearest fingertip position point among the past fingertip position points indicating the fingertip position at the time of contact with the operation surface as the pseudo fingertip position point. Item 2. The input device according to Item 1.   The model shape creation unit calculates a motion vector of each fingertip position point, moves the pseudo fingertip position point based on an average motion vector obtained by averaging the calculated motion vectors, and reflects the shape of the hand The input device according to claim 3, wherein a CG model is created.   The input device according to claim 1, wherein the pseudo fingertip position creation unit creates the pseudo fingertip position points between adjacent fingertip position points whose mutual distance exceeds a predetermined value.   The model shape creation unit creates the CG model reflecting the shape of the hand by moving the pseudo fingertip position point so as to be positioned between the fingertip position points located on both sides of the pseudo fingertip position point. The input device according to claim 5, wherein: An input method in which a CG model of a user's hand placed on an operation surface is displayed on a display unit, and a user instruction input using the display is given to a device,
A contact area detection step of detecting a contact area between the operation surface and the hand;
Based on the contact area, a position coordinate data creating step for creating a plurality of fingertip position points indicating the fingertip position of the hand that is currently in contact with the operation surface every predetermined time;
A pseudo fingertip position creating step for creating a pseudo fingertip position point indicating a fingertip position not currently in contact with the operation surface;
A model shape creating step for creating a CG model reflecting the shape of the hand based on the fingertip position point and the pseudo fingertip position point;
A superimposed image creating step of creating an image in which the CG model created in the model shape creating step and a GUI part assigned an instruction to be given to the device are superimposed and displayed on the display unit;
An input method comprising: an operation content determination step for determining a user instruction to be given to the device based on a positional relationship between the GUI part and the CG model.

Images