JP4962467B2 - Input device and input method - Google Patents

Description

  The present invention relates to an input device and an input method for a user to input a command or the like to a device, and more specifically, using a body part such as a hand based on information displayed on a display or the like by a user. The present invention relates to an input device and an input method capable of inputting commands and the like.

  With the recent increase in functionality of information equipment terminals, user interfaces that are easier for users to use are required. An example of an easy-to-use user interface is a touch panel display. The touch panel display is a device in which a user inputs a command or the like to an information device terminal or the like by pressing a GUI (Graphical User Interface) part displayed on the display. From this, it can be said that the touch panel display is an interface that even a beginner can operate intuitively.

  However, in a touch panel display, since a device for detecting a touch (hereinafter referred to as an operation unit) and a display unit are integrated, there is a problem that the touch panel display cannot be operated unless the user has it.

In order to solve this problem, the technique of Patent Document 1 separates an operation unit and a display unit, detects an operator's hand placed on the operation unit, and forms a computer graphic (hereinafter referred to as CG) hand shape. A model (hereinafter simply referred to as a hand model) is displayed superimposed on a GUI displayed on the display unit. Thus, even when the user is away from the display unit, the user can perform an operation while maintaining an intuitive operational feeling of the touch panel display.
JP 2006-72854 A

However, since the above-described conventional technology cannot calculate the bending of the finger of the user who is an operator, when the user performs an operation by bending his / her finger, a hand model having a shape significantly different from the shape of the user's hand is used. Display on the display. As a result, there is a problem that the user feels uncomfortable during the operation.

  A first aspect of the present invention is an input device that operates a device by displaying a CG model of a hand placed on an operation surface on a display unit, and detects a plurality of contact areas between the operation surface and a hand. An area detection unit, a movement amount calculation unit for calculating the movement amount of the contact area from the initial state where the finger of the hand is extended, and movement of each fingertip position by finger bending based on a change in the area of the contact area from the initial state A finger bending movement amount calculating unit for calculating the amount, a finger bending angle calculating unit for calculating a bending angle of each finger joint of the CG model based on a moving amount by finger bending of each fingertip position, and a contact region from an initial state Based on the amount of movement and the amount of movement by finger bending of each fingertip position, based on the amount of translation of the CG model, the amount of translation of the CG model, and the bending angle of each finger joint, Model that determines the position and deformation shape of the CG model An image obtained by superimposing a shape determining unit, a GUI part holding unit that holds a GUI part to which a command for a device is assigned, a CG model whose shape is determined by the model shape determining unit, and a GUI part held in the GUI part holding unit Is generated and displayed on the display unit, a collision determination unit that determines a collision between the GUI part and the fingertip of the CG model, and when the collision is determined by the collision determination unit, the collision is determined. A command transmission unit that transmits a command assigned to the GUI part to the device.

  A second aspect of the present invention is an input device that operates a device by displaying a CG model of a hand placed on an operation surface on a display unit, and detects a plurality of contact areas between the operation surface and the hand. Based on the area detection unit, the finger bending movement amount calculation unit that calculates the movement amount of each fingertip position by finger bending based on the change in the area of the contact area from the initial state, and the movement amount of each fingertip position by finger bending The deformed shape of the CG model is determined based on the bending angle of each finger joint of the CG model and the bending angle calculation unit that calculates the bending angle of each finger joint of the CG model. The shape is determined by a model shape determining unit that determines a position to move each fingertip position to fit a plurality of contact areas, a GUI part holding unit that holds a GUI part to which a command for the device is assigned, and a model shape determining unit. CG model A superimposed image creating unit that creates an image in which a GUI part and a GUI part held in the GUI part holding unit are superimposed and displayed on the display unit, a collision determination unit that determines a collision between the GUI part and the fingertip of the CG model, When a collision is determined by the collision determination unit, a command transmission unit that transmits a command assigned to the GUI part determined to have the collision to the device is provided.

  The third aspect of the present invention is directed to an input method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit. In order to achieve the above object, the input method of the present invention includes a contact region detection step for detecting a plurality of contact regions between the operation surface and the hand, and movement of the contact region from the initial state where the finger of the hand is extended. A movement amount calculating step for calculating the amount, a finger bending movement amount calculating step for calculating a movement amount by finger bending of each fingertip position based on a change in the area of the contact area from the initial state, and by finger bending of each fingertip position Based on the finger bending angle calculating step for calculating the bending angle of each finger joint of the CG model based on the moving amount, the moving amount of the contact area from the initial state, and the moving amount by finger bending of each fingertip position, the CG model A parallel movement amount calculating step for calculating the parallel movement amount of the CG model, and a model shape determination step for determining the position and deformation shape of the CG model based on the parallel movement amount of the CG model and the bending angle of each finger joint. A superimposed image creating step of creating an image by superimposing the CG model whose shape has been determined in the model shape determining step and a GUI part held in advance, to which a command for the device is assigned, and displaying the superimposed image on a display unit; A collision determination step for determining a collision between the GUI part and the fingertip of the CG model, and a command transmission for transmitting a command assigned to the GUI part for which the collision is determined to the device when the collision is determined by the collision determination step. Steps.

The fourth aspect of the present invention is directed to an input method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit. And in order to achieve the said objective, the input method of this invention is based on the contact area detection step which detects the some contact area of an operation surface and a hand, and each change of the area of the contact area from an initial state. A finger bending movement amount calculating step for calculating a movement amount by finger bending of the fingertip position; a finger bending angle calculating step for calculating a bending angle of each finger joint of the CG model based on the movement amount by finger bending of each fingertip position; A model that determines a deformed shape of the CG model based on a bending angle of each finger joint of the CG model, and determines a position to be moved so that each fingertip position of the CG model for which the shape is determined fits a plurality of contact areas. Creates an image in which the shape determination step, the CG model whose shape has been determined in the model shape determination step, and the GUI parts that are stored in advance and assigned commands to the device are superimposed A superimposed image creation step to be displayed on the display unit, a collision determination step for determining a collision between the GUI part and the fingertip of the CG model, and when the collision is determined by the collision determination step, the GUI part for which the collision is determined is displayed. According to the input device and the input method of the first to fourth aspects of the present invention, the movement of the operator's hand can be reflected in the CG model. Furthermore, the movement of each finger of the operator can be reflected in the CG model.

  According to the input device and the input method of the present invention, even when the user operates by bending a finger, the user's hand shape can be accurately reflected in the hand model. As a result, according to the input device and the input method of the present invention, the user can perform an operation without feeling uncomfortable.

  FIG. 1 is a diagram for explaining the outline of the operation of the input device 100 according to the embodiment of the present invention. Hereinafter, the operation of the input device 100 will be briefly described with reference to FIG.

  As shown in FIG. 1, the input device 100 includes a contact area detection unit 1 and a calculation unit 2. The contact area detection unit 1 detects an area touched by a user (hereinafter referred to as an operator) placed on the operation surface of the contact area detection unit 1 and detects information on the detected contact area (hereinafter simply referred to as contact). (Referred to as region information) is output to the calculation unit 2. The contact area detection unit 1 is, for example, a touch pad that detects contact at multiple points. The contact area detection unit 1 may be a general device that uses an optical method, a capacitance method, a pressure-sensitive method, a resistance film method, or the like as long as it can detect contact at a plurality of points.

  The calculation unit 2 creates display data using the contact area information detected by the contact area detection unit 1 and outputs the display data to the display unit 3. Specifically, the calculation unit 2 creates a CG hand model using the contact area information, creates display data in which the hand model is superimposed on the GUI, and outputs the display data to the display unit 3. Further, the calculation unit 2 determines a command intended by the operator using the positional relationship between the created hand model and the GUI part, and outputs the determined command to the operation target device 4 such as an information terminal device. Note that the arithmetic unit 2 may be a general personal computer (hereinafter referred to as a PC), or may be a general-purpose dedicated machine incorporating a dedicated graphic IC or the like. The calculation unit 2 includes an interface that receives a signal input from the contact area detection unit 1, a CPU, a ROM, and a RAM, a bus that connects the modules, and an interface that outputs a calculation result as an image.

  The display unit 3 performs display using the display data input from the calculation unit 2. The display unit 3 is a display such as a liquid crystal display or a CRT, for example. The operation target device 4 executes the command input from the calculation unit 2.

  With the above configuration, the operator operates the hand model by moving the hand on the operation surface of the contact area detection unit 1 while visually observing the hand model superimposed on the GUI part displayed on the display unit 3. Thus, the operation target device 4 can be operated.

  Hereinafter, the configuration and operation of the input device 100 will be described in detail. FIG. 2 is a diagram illustrating a configuration example of the input device 100. As illustrated in FIG. 2, the input device 100 includes a contact area detection unit 1 and a calculation unit 2. The calculation unit 2 includes a movement amount calculation unit 12, a finger bending movement amount calculation unit 13, a finger bending angle calculation unit 14, a parallel movement amount calculation unit 15, a model holding unit 16, a model shape determination unit 17, A GUI parts holding unit 18, a superimposed image creation unit 19, a collision determination unit 20, and a command transmission unit 21 are included.

  FIG. 3 is a flowchart for explaining the operation of the input device 100. When the start switch is turned on by an operator or the like, the input device 100 starts operating.

  First, in step S <b> 101, the contact area detection unit 1 determines whether or not the operator's hand is placed on the operation surface of the contact area detection unit 1. When the operator's hand is not placed, the contact area detection unit 1 enters a process waiting state, and when the operator's hand is placed, the process proceeds to step S102. Note that the determination in step S101 can be made based on whether or not contact is detected.

  Next, in step S102, the contact area detection unit 1 detects an area where the operator's hand is in contact with the contact area detection unit 1 as a point group (hereinafter referred to as a contact point group), and detects the detected contact point. The group data is output to the movement amount calculation unit 12. The movement amount calculation unit 12 acquires a point indicating the position of the operator's four fingertips (hereinafter referred to as a fingertip position point) and a contact area using the input contact point group data. The method for acquiring the four fingertip position points and the contact area will be described in detail later.

  Next, in step S103, the movement amount calculation unit 12 waits until four fingertip position points are acquired, and when acquired, the process proceeds to step S104.

  In step S <b> 104, the movement amount calculation unit 12 determines whether or not the initial position point that is the fingertip position point and the reference contact area in a state where the operator has opened his hand (a state where the finger is extended) have been acquired. If the initial position point and the reference contact area have been acquired, the process proceeds to step S106. If the initial position point and the reference contact area have not been acquired, the process proceeds to step S105. Here, the determination of whether or not the initial position point and the reference contact area have been acquired is to determine whether or not the initial position point and the reference contact area have been acquired after the input device 100 has been activated this time.

  In step S <b> 105, the movement amount calculation unit 12 notifies a message prompting the operator to open the hand placed on the contact area detection unit 1 and acquires the initial position point and the reference contact area. This message is displayed on the display unit 3 and notified to the operator, for example. Thereafter, the movement amount calculation unit 12 creates a hand model (hereinafter referred to as an initial hand model) in which the operator has opened his / her hand using the acquired initial position point. The initial position point, the reference contact area, and the initial hand model data acquired in step S105 are stored in the model holding unit 16. Thereafter, the process returns to step S102. That is, the input device 100 acquires the initial position point, the reference contact area, and the initial hand model when the operator places the hand on the contact area detection unit 1 for the first time after activation. The method for acquiring the initial position point, the reference contact area, and the initial hand model will be described in detail later.

  When the operator changes or when the left and right hands of the operator are switched, the already acquired initial position point, reference contact area, and initial hand model are discarded, and the operation of step S105 is performed again. The initial position point, the reference contact area, and the initial hand model may be updated. In addition, the initial position point, the reference contact area, and the initial hand model are not acquired every time the activation is performed, and the operation of step S105 is performed in response to an instruction from the operator, thereby obtaining the initial position point, the reference contact area, and the initial hand model. You may acquire newly. If the contact area detector 1 does not detect a hand contact for a predetermined period in step S101, the already acquired initial position point, reference contact area, and initial hand model are discarded, and the operation in step S105 is performed again. A new initial position point, reference contact area, and initial hand model may be acquired.

  In step S106, the movement amount calculation unit 12 acquires the movement amount of the fingertip position point from the initial position point and the change amount of the current contact area from the reference contact area, respectively. The processing in step S106 will be described in detail later.

  Next, in step S107, the finger bending movement amount calculation unit 13 calculates the movement amount of each fingertip position point by finger bending using the contact area change amount acquired in step S106. Then, the finger bending angle calculation unit 14 calculates the bending angle of each finger joint of the hand model using the calculated movement amount of each fingertip position point due to finger bending. The processing in step S107 will be described in detail later.

  Next, in step S108, the parallel movement amount calculation unit 15 determines the movement amount from the initial position point of the fingertip position point acquired in step S106 and the movement amount of each fingertip position point by finger bending acquired in step S107. Based on this, the amount of translation of the hand model is determined. A method for determining the parallel movement amount will be described later in detail.

  Next, in step S109, the model shape determining unit 17 uses the bending angle and the parallel movement amount of each finger joint of the hand model determined in steps S107 and S108 to determine the hand model held in the model holding unit 16. Read to reflect the deformation. Here, when the initial hand model is held in the model holding unit 16, the initial hand model is read to reflect the deformation.

  The hand model held by the model holding unit 16 is preferably a CG (Computer Graphics) model having a general skeleton (hereinafter referred to as a bone structure). An example of a hand model held by the model holding unit 16 is shown in FIG. The hand model shown in FIG. 4 includes a plurality of polygons and lines indicating bone structures. The file format for holding the hand model in the model holding unit 16 may be a general format. As an example, there is a file format that stores the vertex coordinates of a polygon. The hand model may also hold texture information and amplify the realism of the hand model.

  Next, in step S110, the superimposed image creating unit 19 reads the GUI parts previously held by the GUI parts holding unit 18 and creates an operation image in which the read GUI parts are arranged. Here, the GUI part is a button or the like to which a control command displayed on the display unit 3 is assigned. The GUI parts and the arrangement method thereof may be general. Thereafter, the superimposed image creating unit 19 creates a superimposed image by superimposing the hand model reflected in the deformation in step S109 and the operation image on which the GUI parts are arranged, and displays the superimposed image on the display unit 3. .

  Next, in step S111, the collision determination unit 20 determines whether or not there is a collision between the GUI part and the fingertip position point of the hand model. Details of the collision determination will be described later. If it is determined that there is no collision, the processes in steps S101 to S111 are repeated, and the hand model in the superimposed image is deformed according to the movement of the operator's hand. If it is determined that there is a collision, the collision determination unit 20 notifies the command transmission unit 21 of the command assigned to the GUI part with the collision, and the process proceeds to step S112.

  In step S112, the command transmission unit 21 transmits the notified command to the operation target device 4, and the process returns to step S101. The operation target device 4 receives and executes the transmitted command. By repeating the above processing, the input device 100 operates the operation target device 4 by repeatedly transmitting a command. Note that the processing ends when the start switch is turned off by an operator or the like.

  By the operation described above, the operator can operate the hand model in the superimposed image by moving his / her hand while viewing the superimposed image displayed on the display unit 3. Then, the operator can operate the operation target device 4 by operating the hand model and superimposing the position of the fingertip on the GUI part.

  Below, each step demonstrated using FIG. 3 is demonstrated in detail.

  FIG. 5 is a diagram for explaining the operation of the contact area detector 1 in steps S101 and S102 of FIG. FIG. 5A shows an example of a state where the operator's hand is placed on the operation surface of the contact area detection unit 1. FIG. 5B is a diagram showing the contact point group detected by the contact area detection unit 1 as a black circle group. As shown in FIG. 5, the contact area detection unit 1 detects a contact area of the hand by detecting a contact point (contact point group) in steps S <b> 101 and S <b> 102 of FIG. 3.

  FIG. 6 is a flowchart for explaining step S102 of FIG. 3 in detail. FIG. 7 is a diagram for explaining the calculation method of the fingertip position point acquired in step S102 of FIG. Hereinafter, the process of step S102 of FIG. 3 will be described in detail with reference to FIGS.

  First, in S102-1, the movement amount calculation unit 12 acquires data indicating the contact point group described with reference to FIG.

  Next, in step S102-2, the movement amount calculation unit 12 calculates a plurality of regions indicated by ellipses in FIG. 7 based on the positional relationship between the contact points. A general method may be used as a method of calculating a plurality of regions in S102-2. For example, when there are other contact points in the vicinity of the contact point, there is a method of dividing these contact points into a plurality of regions by regarding them as the same region.

  Next, in step S102-3, the movement amount calculation unit 12 performs an ellipse fitting process on each region calculated in step S102-2. In step S102-3, the movement amount calculation unit 12 encloses the contact point group of the finger with an ellipse as indicated by an ellipse A in FIG. Thereafter, the movement amount calculation unit 12 sets, for example, the center of the ellipse A as the fingertip position point (see FIG. 7B). It should be noted that the fingertip position point may be determined by using another method or another position.

  After determining the fingertip position points, the contact area is obtained for each fingertip position point. The contact area is an area where each fingertip is in contact with the operation surface. For example, when the fingertip position point is obtained by ellipse fitting as described above, it can be obtained by calculating the area of the ellipse. Further, the number of contact points in each region can be used as the contact area.

  Next, in step S102-4, the movement amount calculation unit 12 stores the position of the fingertip position point and the contact area acquired in step S102-3 in a RAM (not shown in FIG. 2). FIG. 8 is a diagram illustrating an example of the position of the fingertip position point acquired in step S102-3 in FIG. In FIG. 8, a black circle indicates a fingertip position point 801, and a dotted ellipse 802 indicates a contact area region.

  FIG. 9 is a flowchart for explaining in detail the processing in step S105 in FIG. First, in step S105-1, the movement amount calculation unit 12 notifies the operator to open a hand on the operation surface of the contact area detection unit 1, and obtains four initial position points and a reference contact area. To do. The reference contact area is a contact area in a state where the operator's hand is open, and at this time, the finger pad is in contact with the operation surface of the contact area detection unit 1. Therefore, the contact area is maximized. The one having the maximum contact area is stored in the RAM (not shown in FIG. 2) as a reference contact area. However, if an area that is equal to or greater than a preset threshold value is detected, noise or an erroneous input is assumed and the reference contact area is acquired again. As will be described later, when the operator bends the finger, the contact portion moves from the belly of the finger toward the toe, and the contact area gradually decreases.

  Next, in step S105-2, the movement amount calculation unit 12 uses the initial position point (fingertip position point) acquired in step S105-1 and the model that is the model that the model holding unit 16 holds from the beginning. The initial model is created by deforming the model hand model so that the corresponding fingertip position point of the hand model overlaps. Here, the four initial position points correspond to the little finger, ring finger, middle finger, and index finger, respectively. If the operator's operation is limited to the left hand, use the left hand model hand model so that the initial position point overlaps with the little finger, ring finger, middle finger, and index finger in that order from the left. Deform the model to create an initial hand model.

  Here, when creating the initial hand model, it is necessary to know from which direction the operator has entered the hand with respect to the contact area detection unit 1. When the hand is entered from the opposite direction, the initial hand model is created in the opposite direction even if the same four initial position points are used. Therefore, it is assumed in advance which direction the operator enters the hand with respect to the contact area detection unit 1.

  FIG. 10 shows an example of the position where the contact area detector 1 is installed in the vehicle. FIG. 10A is a diagram showing an example when the vehicle is attached to the center console of the vehicle. FIG.10 (b) is a figure which shows an example at the time of the contact area detection part 1 being attached to the operation part of the door of a vehicle. FIG.10 (c) is a figure which shows an example at the time of the contact area detection part 1 being attached to the steering wheel of a vehicle. By attaching the contact area detection unit 1 to the position of the vehicle as shown in FIG. 10A, an operator as a driver may advance the left hand in the direction of the arrow 1001 with respect to the contact area detection unit 1 in advance. I understand. Further, by attaching the contact area detection unit 1 to the position of the vehicle as shown in FIG. 10B, the operator as a driver causes the right hand to enter the direction of the arrow 1002 with respect to the contact area detection unit 1. Is known in advance. In the case of FIG. 10C, it is known in advance that the operator who is a driver causes his / her hand to enter the contact area detection unit 1 in the direction of the arrow 1003. However, in this case, since the operator does not know whether to operate with the right hand or the left hand, it is only necessary to be able to input information on whether to operate with the right hand or the left hand according to the operator's preference in advance. .

  Further, when the operator operates with the right hand or the left hand, the little finger is usually smaller than the other fingers and the contact area with the operation surface is also small, so the region with the smallest contact area acquired in step S102 is selected. It can be determined as a little finger, and the right hand or the left hand can be specified from the positional relationship with other regions. That is, if the region specified as the little finger is located on the leftmost side from the other three regions, it is determined as the left hand, and the left hand template hand model is deformed. Conversely, if the region identified as the little finger is located on the rightmost side from the other three regions, it is determined as the right hand, and the model hand model of the right hand is deformed.

  Returning to FIG. 9, thereafter, the movement amount calculation unit 12 displays the created initial hand model on the display unit 3. By this processing, the movement amount calculation unit 12 can create an initial hand model reflecting the size of the operator's hand and display it on the display unit 3.

  Next, in step S105-3, the movement amount calculation unit 12 uses the model holding unit 16 to store the position of the initial position point, the reference contact area acquired in step S105-1, and the initial hand model created in step S105-2. And return to step S102 (see FIG. 3).

  FIG. 11 is a flowchart for explaining the process of step S106 of FIG. In FIG. 11, in step S106-1, the movement amount calculation unit 12 calculates the movement amount of the current fingertip position point from the initial position point. Here, in order to acquire the movement amount of each point, a general method may be used as a method of taking correspondence between each point after movement and each point at the initial position.

  FIG. 12 shows the change of the contact area at each fingertip position when the operator bends the finger. FIG. 12A shows the case of the initial state, and FIG. This shows a case where four fingers are bent without moving the entire hand. Here, the contact area of each fingertip position is indicated by a black ellipse. By bending the finger, the contact position between the operation surface and the finger moves from the belly portion of each finger toward the nail, so the contact area is reduced. Conversely, when the finger is extended from the bent state, the contact area increases. Here, the movement amount from the initial position point of the current fingertip position point calculated by the movement amount calculation unit 12 can be calculated from each coordinate value of the corresponding finger as shown by the arrow in FIG. it can.

  Next, in step S106-2, the movement amount calculation unit 12 calculates a change amount of the current contact area from the reference contact area. The amount of change from the reference contact area of the current contact area can be calculated from each contact area of the corresponding finger as shown in FIG. Here, since the contact area varies depending on the size of the operator's finger or the like, it is preferable to calculate the amount of change in the contact area as the rate of change.

  Next, in step S106-3, the movement amount calculator 12 stores the movement amount of the fingertip position point acquired in step S106-1 and the change amount of the contact area calculated in step S106-2 in the RAM (see FIG. 2). The process proceeds to step S107 (see FIG. 3).

  Next, the process of step S107 in FIG. 3 will be described in detail. In step S107, the finger bending movement amount calculation unit 13 determines the movement amount of each fingertip position point by bending the finger of the hand model using the contact area change amount acquired in step S106.

  FIG. 13 is a diagram illustrating an example of the rate of change of the contact area at each fingertip position when the operator gradually bends the finger from the initial state, as illustrated in FIG. 12. In FIG. 13, the lower diagram of the table shows a state where the finger placed on the operation surface in each state is viewed from the side. The figure at the left end shows the initial state in which the hand is opened and placed on the operation surface as described above. The contact area at this time is the maximum as the reference contact area. By bending the finger, the contact position between the operation surface and the finger moves from the belly portion of each finger toward the nail, so the contact area is reduced.

  The rightmost diagram in FIG. 13 shows a state in which a straight line connecting a joint (distal interphalangeal joint) closest to the fingertip by bending the finger and a fingertip position point is perpendicular to the operation surface (hereinafter, a state perpendicular to the operation surface) It represents). The middle diagram in FIG. 13 shows an intermediate state between the two. Here, assuming that the fingertip position point (initial position point) in the initial state in FIG. 13 is the zero point and the movement amount to the fingertip position point in the vertical state with respect to the operation surface is A, the movement amount to the fingertip position point in the intermediate state Is A / 2.

  Each numerical value in the table of FIG. 13 represents the rate of change of the contact area in each state based on the reference contact area in the initial state for each finger. As shown in FIG. 13, the rate of change of the contact area does not change linearly with respect to the amount of movement of the fingertip position point (initial position point is 0). If the rate of change of the contact area at the fingertip position is known by using a table as shown in FIG. 13 or an approximate function obtained therefrom, the amount of movement of the fingertip position point due to finger bending can be calculated.

  Here, since the change rate of the contact area at each fingertip position is considered to vary depending on the operator, an average value is obtained from experimental values of a plurality of persons, and is calculated in advance as a table as shown in FIG. 13 or from the average value. It can also be stored in the model holding unit 16 as an approximate function.

  Further, when the initial position point and the reference contact area are acquired in step S105 of FIG. 3, a message prompting the operator to bend the finger is notified as shown in FIG. In addition, the fingertip position point and the contact area in each state may be acquired. By doing so, the change rate of the contact area of each fingertip position according to the operator can be acquired, so that the hand shape of the user can be faithfully reflected in the hand model.

  Note that the contact area may change slightly depending on how the finger is pressed against the operation surface. Therefore, a predetermined threshold is set in advance for the change of the contact area within a certain time, and the absolute change rate of the contact area is determined. If the value is larger than a predetermined threshold value, it may be determined that the contact area has changed. For this reason, when the rate of change is equal to or less than a certain threshold value, it is conceivable that the amount of movement of the fingertip position point due to finger bending is not calculated, and processing is performed with no area change. Further, when the absolute value of the rate of change of the contact area within a preset time is smaller than a predetermined threshold but the contact area continues to increase or decrease, the operator intentionally bends the finger. The amount of movement of the fingertip position point by finger bending may be determined after a preset time.

  The finger bending angle calculation unit 14 determines the bending angle of each joint of each finger by using the movement amount of the fingertip position point due to the bending of each finger obtained as described above. Here, there is an inverse kinematics (hereinafter referred to as IK) technique known in the robot engineering field or the like as a method for obtaining the bending angle of the finger joint from the displacement of the fingertip. The IK technique is used to move the tip of an arm having a plurality of movable parts to a target position. When the arm has a plurality of movable parts, there are a plurality of solutions for the bending angle of the movable part in order to move the tip of the arm to the target position using the IK technique.

  Also in the present invention, since the finger has a plurality of joints, there are a plurality of solutions for the bending angles of the finger joints. For this reason, in the present invention, as an example, using the constraint condition that the palm and fingertip of the operator exist on the operation surface (on the same plane) and the constraint condition that the bending angles of the finger joints are equal, Is uniquely determined.

  FIG. 14A shows the displacement of the fingertip position point when the index finger is bent as an example. FIG. 14B is a side view of the hand model of the index finger 1400 in the bent state shown in FIG. As can be seen from FIG. 14, normally, when the finger is bent, the three finger joints bend at the same time. Therefore, when the hand model is deformed using the constraint that the bending angles 1401 to 1403 of each finger joint are equal, the operator feels uncomfortable. The operation can be performed without. Further, when the hand model is deformed using such a constraint condition, the amount of calculation can be greatly reduced, and therefore the hand model that responds instantaneously to the movement of the operator's hand can be deformed.

  Note that the constraint condition for calculating the bending angle of the finger joint is not limited to this, and may be a constraint condition for uniquely obtaining a solution. However, as described above, the constraint condition for calculating the bending angle of the finger joint is preferably a constraint condition that is a hand model that does not cause the operator to feel uncomfortable, and is a hand model that allows the operator to act naturally. Restraint conditions are preferred. In addition, as a method for obtaining the bending angle of the arm movable portion from the target position in the IK technique, a method of repeatedly calculating using a Jacobian matrix, a method of calculating geometrically, and the like are known. May be used.

  3 described above is performed, the finger bending movement amount calculation unit 13 calculates the movement amount of each fingertip position point by finger bending, and the finger bending angle calculation unit 14 determines the bending angle of each finger joint. calculate. Then, the finger bending movement amount calculation unit 13 and the finger bending angle calculation unit 14 store the calculated movement amount of each fingertip position point of the hand model and the bending angle of each finger joint in a RAM (not shown in FIG. 2). .

  Next, the process of step S108 in FIG. 3 will be described in detail. In step S108, the parallel movement amount calculation unit 15 calculates the movement amount from the initial position point of the fingertip position point acquired in step S106 and the movement amount of each fingertip position point by finger bending acquired in step S107. Calculate the amount of translation of the hand model. When the operator moves the entire hand without moving the finger and translates it on the operation surface, the fingertip position point changes, but the contact area at the fingertip position does not change. Therefore, the parallel movement amount calculation unit 15 reads the movement amount from the initial position point of the fingertip position point acquired in step S106 and the movement amount of each fingertip position point by finger bending acquired in step S107 from the RAM. The parallel movement amount of the hand model is calculated by subtracting the movement amount of each fingertip position point by finger bending from the movement amount of the read fingertip position point from the initial position point, and RAM (not shown in FIG. 2). ).

  Next, the process of step S109 in FIG. 3 will be described in detail. FIG. 15 is a flowchart for explaining the process of step S109 of FIG. By the processing of the flowchart of FIG. 15, the deformed shape of the hand model is determined using each value determined in steps S107 and S108. Hereinafter, with reference to FIG. 15, step S109 in FIG. 3 will be described in detail.

  First, in step S109-1, the model shape determination unit 17 reads the parallel movement amount of the hand model determined in step S108 from the RAM, and determines the position of the hand model.

  Next, in step S109-2, the model shape determination unit 17 reads the bending angle of each finger joint determined in step S107 from the RAM, and determines the deformed shape of the hand model. Thereafter, the process proceeds to step S110.

  FIG. 16 is a diagram showing a specific example of the deformation of the hand model in step S109 of FIG. FIG. 16A shows an initial position point (a fingertip position point in a state where the operator opens his hand). FIG. 16B is an initial hand model drawn using the initial position points shown in FIG. FIG. 16C shows a fingertip position point when the operator bends the index finger as an example. FIG. 16D is a hand model deformed using the fingertip position points shown in FIG. FIGS. 17A and 17B are views of the hand model shown in FIGS. 16B and 16D as viewed from the side. As shown in FIGS. 16 and 17, in step S109 of FIG. 3, the hand model is deformed realistically according to the deformation of the operator's hand.

  FIG. 18 is a diagram for explaining the process of step S110 of FIG. As shown in FIG. 18, in step S110, the superimposed image creating unit 19 creates a superimposed image 1820 by superimposing the operation image 1800 on which the GUI part 1801 is arranged and the deformed hand model 1810, and the display unit 3 To display. Note that when the superimposed image 1820 is created, the hand model may be made semi-transparent or the like in order to make it easy to visually recognize the GUI part that overlaps the hand model.

  FIG. 19 is a diagram for explaining the processing in step S111 in FIG. When the GUI parts 1901 to 1904 and the fingertip position point of the hand model 1910 overlap in the superimposed image, the collision determination unit 20 determines that the GUI part and the fingertip position point of the hand model 1910 have collided. In the case of FIG. 19, the collision determination unit 20 notifies the command transmission unit 21 of the command assigned to the GUI part 1902 for which the collision indicated by the arrow 1920 is determined.

  Thereafter, the command is transmitted to the operation target device 4 in step S112. The operation target device 4 executes the received command.

  As described above, according to the input device 100 according to the present invention, even when the operator is away from the display unit 3, the operator can perform an operation while maintaining an intuitive operational feeling of the touch panel display. Further, the input device 100 according to the present invention can display a hand model that faithfully reproduces the movement of the operator's finger on the display unit 3. As a result, the input device 100 according to the present invention can provide the operator with a comfortable operation feeling without a sense of incongruity.

In step S108 of FIG. 3 of the present embodiment, based on the movement amount of the fingertip position point acquired from step S106 from the initial position point and the movement amount of each fingertip position point by finger bending acquired in step S107. The amount of translation of the hand model was calculated. For this parallel movement, if the bending angle of each finger joint in step S107 can be calculated, the deformed shape of the hand model is determined. Therefore, the fingertip position point of the deformed hand model is detected on the current operation surface. It is also possible to move it so that it matches the fingertip position point. In this case, the movement amount calculation unit 12 does not need to calculate the movement amount of the fingertip position point from the initial state, and the model shape determination unit 17 deforms the hand model based on the bending angle of each finger joint of the hand model. The shape is determined, and the position to move the deformed hand model is determined so that each fingertip position of the hand model for which the shape is determined matches each fingertip position point that is currently detected. In step S103 in FIG. 3, four fingertip position points and contact areas are acquired, but the number is not limited to four. If the right hand or the left hand is known and the finger touching can be identified (for example, touching with the index finger), the position of the hand model can be determined if the hand approach direction is known. The fingertip position point and the contact area may be acquired.

  The present invention can be used for an input device, an input method, and the like, and is particularly useful when a CG model of an operator's hand is more realistically deformed in accordance with the movement of the operator's hand.

The figure for demonstrating the outline | summary of operation | movement of the input device 100 which concerns on embodiment of this invention. The figure which shows the structural example of the input device 100 which concerns on embodiment of this invention. The flowchart for demonstrating operation | movement of the input device 100 which concerns on embodiment of this invention. The figure which shows an example of the hand model which the model holding part 16 hold | maintains FIGS. 3A and 3B are diagrams for explaining the operation of the contact area detection unit 1 in steps S101 and S102 of FIG. 3, and FIG. 3A shows an example of a state where an operator's hand is placed on the operation surface of the contact area detection unit 1; (B) The contact point group detected by the contact area detector 1 is represented by a black circle group. Flowchart for explaining step S102 in FIG. 3 in detail FIGS. 3A and 3B are diagrams for explaining the fingertip position points acquired in step S102 of FIG. 3, and FIG. 3A shows an example in which a contact point group is subjected to ellipse fitting processing, and FIG. Figure The figure which shows an example of the position of the fingertip position point acquired by step S102-3 of FIG. Flowchart for explaining in detail the process of step S105 in FIG. It is a figure which shows the example of the installation position to the vehicle of the contact region detection part 1, (a) A figure which shows an example at the time of being attached to the center console of a vehicle, (b) When it is attached to the operation part of the door of a vehicle The figure which shows an example, (c) The figure which shows an example at the time of attaching to the steering wheel of a vehicle Flowchart for explaining the process of step S106 of FIG. It is a figure which shows the change of the contact area | region of each fingertip position when an operator bends a finger | toe, (a) The figure which shows an initial state, (b) The state which the operator bent four fingers, without moving the whole hand Figure showing The figure which shows an example of the change rate of the contact area of each fingertip position when an operator bends a finger from an initial state. FIG. 3 is a diagram for explaining a method of calculating a finger bending angle of a hand model in step S107 of FIG. 3, (a) a diagram showing displacement of a fingertip position point when the index finger 1400 is bent, and (b) a bent state. View of hand model of index finger 1400 from the side Flowchart for explaining the process of step S109 of FIG. FIGS. 3A and 3B are diagrams showing a specific example of deformation of the hand model in step S <b> 109 in FIG. 3, (a) a diagram showing an initial position point, (b) a diagram showing an initial hand model drawn using the initial position point, and (c). The figure which shows the fingertip position point when an operator bends the index finger, (d) The figure which shows the hand model deform | transformed using the fingertip position point when the index finger is bent FIG. 17 is a view of the hand model shown in FIG. 16 as viewed from the side, (a) a view of the hand model shown in FIG. 16b as viewed from the side, and (b) a view of the hand model shown in FIG. The figure for demonstrating the process of step S110 of FIG. The figure for demonstrating the process of step S111 of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Contact area | region detection part 2 Calculation part 3 Display part 4 Operation object apparatus 12 Movement amount calculation part 13 Finger bending movement amount calculation part 14 Finger bending angle calculation part 15 Parallel movement amount calculation part 16 Model holding part 17 Model shape determination part 18 GUI Parts holding unit 19 Superposed image creation unit 20 Collision determination unit 21 Command transmission unit 100 Input device

Claims (4)

An input device for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detector that detects a plurality of contact areas between the operation surface and the hand;
A movement amount calculation unit for calculating a movement amount of the contact area from an initial state in which the finger of the hand is extended;
A finger bending movement amount calculation unit for calculating a movement amount by finger bending of each fingertip position based on a change in the area of the contact region from the initial state;
A finger bending angle calculation unit that calculates a bending angle of each finger joint of the CG model based on a movement amount of the fingertip position by finger bending;
A parallel movement amount calculation unit that calculates a parallel movement amount of the CG model based on a movement amount of the contact area from the initial state and a movement amount of each fingertip position by finger bending;
A model shape determining unit that determines a position and a deformed shape of the CG model based on a parallel movement amount of the CG model and a bending angle of each finger joint;
A GUI parts holding unit for holding GUI parts to which commands for the device are assigned;
A superimposed image creating unit that creates an image in which the CG model whose shape is determined by the model shape determining unit and a GUI part held in the GUI part holding unit are superimposed and displayed on the display unit;
A collision determination unit that determines a collision between the GUI part and the fingertip of the CG model;
When a collision is determined by the collision determination unit, an input device includes a command transmission unit that transmits a command assigned to the GUI part for which the collision is determined to the device. An input device for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detector that detects a plurality of contact areas between the operation surface and the hand;
A finger bending movement amount calculation unit for calculating a movement amount by finger bending of each fingertip position based on a change in the area of the contact region from the initial state;
A finger bending angle calculation unit that calculates a bending angle of each finger joint of the CG model based on a movement amount of the fingertip position by finger bending;
Based on the bending angle of each finger joint of the CG model, a deformed shape of the CG model is determined, and a position to be moved so that each fingertip position of the CG model for which the shape is determined fits the plurality of contact areas. A model shape determining unit to determine;
A GUI parts holding unit for holding GUI parts to which commands for the device are assigned;
A superimposed image creating unit that creates an image in which the CG model whose shape is determined by the model shape determining unit and a GUI part held in the GUI part holding unit are superimposed and displayed on the display unit;
A collision determination unit that determines a collision between the GUI part and the fingertip of the CG model;
When a collision is determined by the collision determination unit, an input device includes a command transmission unit that transmits a command assigned to the GUI part for which the collision is determined to the device. An input method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detection step of detecting a plurality of contact areas between the operation surface and the hand;
A movement amount calculating step for calculating a movement amount of the contact area from an initial state in which the finger of the hand is extended;
A finger bending movement amount calculating step of calculating a movement amount by finger bending of each fingertip position based on a change in the area of the contact region from the initial state;
A finger bending angle calculating step of calculating a bending angle of each finger joint of the CG model based on a movement amount by finger bending of each fingertip position;
A parallel movement amount calculating step for calculating a parallel movement amount of the CG model based on a movement amount of the contact area from the initial state and a movement amount of each fingertip position by finger bending;
A model shape determining step for determining a position and a deformed shape of the CG model based on a parallel movement amount of the CG model and a bending angle of each finger joint;
A superimposed image creation step of creating an image in which the CG model whose shape has been determined in the model shape determination step and a GUI part that is held in advance and assigned a command to the device are superimposed and displayed on the display unit; ,
A collision determination step of determining a collision between the GUI part and the fingertip of the CG model;
When a collision is determined by the collision determination step, a command transmission step of transmitting a command assigned to the GUI part determined to have the collision to the device. An input method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detection step of detecting a plurality of contact areas between the operation surface and the hand;
A finger bending movement amount calculating step of calculating a movement amount by finger bending of each fingertip position based on a change in the area of the contact region from the initial state;
A finger bending angle calculating step of calculating a bending angle of each finger joint of the CG model based on a movement amount by finger bending of each fingertip position;
Based on the bending angle of each finger joint of the CG model, a deformed shape of the CG model is determined, and a position to be moved so that each fingertip position of the CG model for which the shape is determined fits the plurality of contact areas. A model shape determination step to be determined; and
A superimposed image creation step of creating an image in which the CG model whose shape has been determined in the model shape determination step and a GUI part that is held in advance and assigned a command to the device are superimposed and displayed on the display unit; ,
A collision determination step of determining a collision between the GUI part and the fingertip of the CG model;
When a collision is determined by the collision determination step, a command transmission step of transmitting a command assigned to the GUI part determined to have the collision to the device.

Images