JP5005585B2 - Operating device and method - Google Patents

Description

  The present invention relates to an operating device for a user to input a command or the like to a device, and more specifically, a command or the like using a body part such as a hand based on information displayed on a display or the like by a user. It is related with the operating device which can input.

  With the recent increase in functionality of information equipment terminals, user interfaces that are easier for users to use are required. An example of an easy-to-use user interface is a touch panel display. The touch panel display is a device in which a user inputs a command or the like to an information device terminal or the like by pressing a GUI (Graphical User Interface) part displayed on the display. From this, it can be said that the touch panel display is an interface that even a beginner can operate intuitively.

  However, in a touch panel display, since a device for detecting a touch (hereinafter referred to as an operation unit) and a display unit are integrated, there is a problem that the touch panel display cannot be operated unless the user has it.

In order to solve this problem, the technique of Patent Document 1 separates an operation unit and a display unit, detects an operator's hand placed on the operation unit, and forms a computer graphic (hereinafter referred to as CG) hand shape. A model (hereinafter simply referred to as a hand model) is displayed superimposed on a GUI displayed on the display unit. Thus, even when the user is away from the display unit, the user can perform an operation while maintaining an intuitive operational feeling of the touch panel display.
JP 2006-72854 A

  However, since the above-described conventional technology cannot calculate the bending of the finger of the user who is an operator, when the user performs an operation by bending his / her finger, a hand model having a shape significantly different from the shape of the user's hand is used. Display on the display. As a result, there is a problem that the user feels uncomfortable during the operation.

  The present invention is directed to an operation device for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit. And in order to achieve the said objective, the operating device of this invention contacted the operation area | region and the palm, the contact area detection part which detects the some contact area of an operation surface and a hand, and a some contact area. A movement amount calculation unit that calculates a movement amount of the palm contact area and the finger contact area from the initial state in which the finger is extended; Based on the movement amount of the palm contact area and the finger contact area of the CG model, and based on the movement amount of the palm contact area and the finger contact area from the initial state Then, a finger bending angle determination unit that determines the bending angle of each finger joint of the CG model, and a model shape determination that determines the deformation shape of the CG model according to the parallel movement amount, the rotation amount, and the bending angle of each finger joint of the CG model. Parts and devices Create an image that superimposes the GUI parts holding unit that holds the GUI parts to which the model is assigned, the CG model whose shape is determined by the model shape determining unit, and the GUI parts held in the GUI part holding unit on the display unit A superimposed image creation unit to be displayed, a collision determination unit that determines a collision between the GUI part and the fingertip of the CG model, and a command assigned to the GUI part that is determined to have the collision when the collision determination unit determines a collision And a command transmission unit for transmitting to the device.

  Thereby, the operating device of the present invention can reflect the movement of the operator's hand in the CG model, and can also reflect the movement of each finger of the operator in the CG model.

  Preferably, the movement amount calculation unit determines a palm position point indicating a predetermined position in the palm contact area and a fingertip position point indicating a position corresponding to the tip of the finger in the finger contact area, and the palm position Using the point and the fingertip position point, the palm contact area and the movement amount of the finger contact area from the initial state where the finger is extended are calculated.

  Thereby, since the operating device of the present invention can detect a more accurate hand movement, a more accurate hand movement can be reflected in the CG model.

  Preferably, the model movement amount determination unit determines the movement amount of the palm contact area from the initial state as the parallel movement amount of the CG model.

  Preferably, the model movement amount determination unit calculates the rotation amount of the finger contact area around a predetermined position in the palm contact area, and determines the average of the calculated rotation amounts as the rotation amount of the CG model. .

  Preferably, the finger bending angle determination unit calculates the distance between the palm contact area and the finger contact area, and determines the bending angle of each finger joint of the CG model based on the calculated distance.

  Preferably, the finger bending angle determination unit determines the bending angle of each finger joint of the CG model using a constraint condition that the bending angle of each finger joint of the same finger of the CG model is constant.

  As a result, the operation device of the present invention can greatly reduce the amount of calculation for determining the bending angle of each finger joint of the CG model, so that the shape of the CG model can be quickly transformed and the cost of the device can be further reduced. be able to.

  Preferably, the finger bending angle determination unit determines a bending angle of each finger joint of the CG model using a constraint condition that the palm contact area and each finger contact area exist on the same plane.

  As a result, the operation device of the present invention can greatly reduce the amount of calculation for determining the bending angle of each finger joint of the CG model, so that the shape of the CG model can be quickly transformed and the cost of the device can be further reduced. be able to.

  The finger bending angle determination unit further determines the orientation of each finger with respect to the palm based on the palm contact area and the movement amount of the finger contact area from the initial state, and the model shape determination unit further determines the finger bending angle. The deformed shape of the CG model may be determined according to the orientation of each finger with respect to the palm determined by the unit.

  Thereby, the operating device of this invention can deform | transform the shape of CG model more realistically.

The present invention is also directed to an operation method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit. In order to achieve the above object, the operation method of the present invention includes a contact area detecting step for detecting a plurality of contact areas between the operation surface and the hand, and the operation surface and the palm contacting the plurality of contact areas. A movement amount calculation step for calculating a movement amount of the palm contact area and the finger contact area from the initial state in which the finger is extended, divided into a palm contact area and a finger contact area in which the operation surface and the finger are in contact, and from the initial state A model movement amount determination step for determining a parallel movement amount and a rotation amount of the CG model based on the movement amounts of the palm contact area and the finger contact area of the palm, and based on the movement amounts of the palm contact area and the finger contact area from the initial state. Then, a finger bending angle determination step for determining the bending angle of each finger joint of the CG model, and a model shape determination for determining the deformation shape of the CG model according to the parallel movement amount, the rotation amount of the CG model, and the bending angle of each finger joint. A superimposed image creating step of creating an image by superimposing a step, a CG model whose shape is determined in the model shape determining step, and a GUI part held in advance, to which a command for the device is assigned, and displaying the superimposed image on the display unit; A collision determination step for determining a collision between the GUI part and the fingertip of the CG model, and a command transmission step for transmitting a command assigned to the GUI part for which the collision is determined to the device when the collision is determined by the collision determination step. With.

  Accordingly, the operation method of the present invention can reflect the movement of the operator's hand in the CG model, and can also reflect the movement of each finger of the operator in the CG model.

  According to the hand shape superimposed display type information terminal operating device and the method of the present invention, even when the user is away from the display unit, the user can perform an operation while maintaining an intuitive operation feeling of the touch panel display. Further, even when the user operates by bending his / her finger, the user's hand shape can be faithfully reflected in the hand model. As a result, according to the hand shape superimposed display type information terminal operating device and method of the present invention, the user can operate without feeling uncomfortable.

(Embodiment)
FIG. 1 is a diagram for explaining the outline of the operation of the hand shape superimposed display type information terminal operating device 100 according to the best mode for carrying out the operating device of the present invention. Hereinafter, the operation of the hand-shaped superimposed display type information terminal operating device 100 will be briefly described with reference to FIG.

  As shown in FIG. 1, the hand shape superimposed display type information terminal operating device 100 includes a contact area detection unit 1 and a calculation unit 2. The contact area detection unit 1 detects an area in which the hand of a user (hereinafter referred to as an operator) placed on the contact area detection unit 1 is in contact, and information on the detected contact area (hereinafter simply referred to as contact area information). ) Is output to the calculation unit 2. The contact area detection unit 1 is, for example, a touch pad that detects contact at multiple points. Note that the contact region detection unit 1 only needs to be able to detect contact at independent multiple points, and may be a general device using a capacitance method, a pressure-sensitive method, a resistance film method, or the like.

  The calculation unit 2 creates display data using the contact area information detected by the contact area detection unit 1 and outputs the display data to the display unit 3. Specifically, the calculation unit 2 creates a CG hand model using the contact area information, creates display data in which the hand model is superimposed on the GUI, and outputs the display data to the display unit 3. Further, the calculation unit 2 determines a command intended by the operator using the positional relationship between the created hand model and the GUI part, and outputs the determined command to the operation target device 4 such as an information terminal device. Note that the arithmetic unit 2 may be a general personal computer (hereinafter referred to as a PC), or may be a general-purpose dedicated machine incorporating a dedicated graphic IC or the like. The calculation unit 2 includes an interface that receives a signal input from the contact area detection unit 1, a CPU, a ROM, and a RAM, a bus that connects the modules, and an interface that outputs a calculation result as an image.

  The display unit 3 performs display using the display data input from the calculation unit 2. The display unit 3 is a display such as a liquid crystal display or a CRT, for example. The operation target device 4 executes the command input from the calculation unit 2.

  With the above configuration, the operator operates the hand model by moving the hand on the contact area detection unit 1 while viewing the hand model superimposed on the GUI part displayed on the display unit 3. The target device 4 can be operated.

  Hereinafter, the configuration and operation of the hand shape superimposed display type information terminal operating device 100 will be described in detail. FIG. 2 is a diagram illustrating a configuration example of the hand shape superimposed display type information terminal operating device 100. As shown in FIG. 2, the hand shape superimposed display type information terminal operating device 100 includes a contact area detection unit 1 and a calculation unit 2. The calculation unit 2 includes a movement amount calculation unit 12, a model movement amount determination unit 13, a finger bending angle determination unit 14, a model holding unit 15, a model shape determination unit 16, a GUI parts holding unit 17, and a superimposed image. A creation unit 18, a collision determination unit 19, and a command transmission unit 20 are included.

  FIG. 3 is a flowchart for explaining the operation of the hand shape superimposed display type information terminal operating device 100.

  When the start switch is turned on by an operator or the like, the hand shape superimposed display type information terminal operating device 100 starts operating.

  First, in step S <b> 101, the contact area detection unit 1 determines whether or not an operator's hand is placed on the contact area detection unit 1. When the operator's hand is not placed, the contact area detection unit 1 enters a process waiting state, and when the operator's hand is placed, the process proceeds to step S102. Note that the determination in step S101 can be made based on whether or not contact is detected.

  Next, in step S102, the contact area detection unit 1 detects an area where the operator's hand is in contact with the contact area detection unit 1 as a point group (hereinafter referred to as a contact point group), and detects the detected contact point. The group data is output to the movement amount calculation unit 12.

  Next, in step S103, the movement amount calculation unit 12 uses the input contact point group data to indicate a point indicating the position of the operator's five fingertips (hereinafter referred to as a fingertip position point) and one palm point. A point indicating the position (hereinafter referred to as a palm position point) is acquired. The movement amount calculation unit 12 stands by until five fingertip position points and one palm position point are acquired, and when acquired, the process proceeds to step S104. A method for acquiring five fingertip position points and one palm position point will be described later.

  In step S104, the movement amount calculation unit 12 has acquired the fingertip position point and palm position point (hereinafter, may be collectively referred to as an initial position point) in a state where the operator has opened his hand (a state where the finger is extended). It is determined whether or not. If the initial position point has been acquired, the process proceeds to step S106. If the initial position point has not been acquired, the process proceeds to step S105. Here, the determination of whether or not the initial position point has been acquired is to determine whether or not the initial position point has been acquired after the hand-shaped superimposed display type information terminal operating device 100 has been activated this time.

  In step S <b> 105, the movement amount calculation unit 12 notifies the operator of a message that prompts the operator to open the hand placed on the contact area detection unit 1 and acquires the initial position point. This message is displayed on the display unit 3 and notified to the operator, for example. Thereafter, the movement amount calculation unit 12 creates a hand model (hereinafter referred to as an initial hand model) in which the operator has opened his / her hand using the acquired initial position point. The initial position point and the initial hand model data acquired in step S105 are stored in the model holding unit 15. Thereafter, the process returns to step S103. That is, the hand shape superimposed display type information terminal operating device 100 acquires the initial position point and the initial hand model when the operator places the hand on the contact area detection unit 1 for the first time after activation. A method for acquiring the initial position point and the initial hand model will be described later.

  When the operator changes, in order to update the initial position point and the initial hand model, the initial position point and the initial hand model of the previous operator already acquired in response to the instruction of the subsequent operator are updated. May be discarded, and the initial position point and initial hand model of the subsequent operator may be acquired by performing the operation of step S105 again. In addition, the initial position point and the initial hand model may be newly acquired by performing the operation in step S105 in response to the operator's instruction without acquiring the initial position point and the initial hand model each time the apparatus is activated. If the contact area detection unit 1 does not detect a hand contact for a predetermined period in step S101, the already acquired initial position point and initial hand model are discarded, and the operation in step S105 is performed again to obtain a new initial position. Points and initial hand shapes may be acquired.

  In step S <b> 106, the movement amount calculation unit 12 acquires displacements of the fingertip position point and the palm position point from the initial position point, respectively. Here, in order to obtain the displacement of each point, a general method may be used as a method for taking correspondence between each point after displacement and each point before displacement.

  Next, in step S107, the model movement amount determination unit 13 uses the displacement from the initial position point of the fingertip position point and palm position point acquired in step S106, and translates and rotates the rotation amount (rotation angle) of the hand model. ). A method for determining the parallel movement amount and the rotation amount will be described later.

  Next, in step S108, the finger bending angle determination unit 14 determines the bending angle of each finger of the hand model using the displacement of the fingertip position point and the palm position point from the initial position point acquired in step S106. A method for determining the bending angle of each finger will be described later.

  Next, in step S109, the model shape determining unit 16 uses the parallel movement amount, the rotation amount, and the bending angle of each finger determined in steps S107 and S108 to hold the hand held in the model holding unit 15. Read the model to reflect the deformation. Here, when the initial hand model is held in the model holding unit 15, the initial hand model is read to reflect the deformation.

  Note that the hand model held in the model holding unit 15 is preferably a CG model having a general skeleton (hereinafter referred to as a bone structure). An example of a hand model held by the model holding unit 15 is shown in FIG. The hand model shown in FIG. 4 includes a plurality of polygons and lines indicating bone structures. The file format for holding the hand model in the model holding unit 15 may be a general format. As an example, there is a file format that stores the vertex coordinates of a polygon. The hand model may also hold texture information and amplify the realism of the hand model.

  Next, in step S110, the superimposed image creating unit 18 reads the GUI parts previously held by the GUI parts holding unit 17, and creates an operation image in which the read GUI parts are arranged. Here, the GUI part is a button or the like to which a control command displayed on the display unit 3 is assigned. The GUI parts and the arrangement method thereof may be general. Thereafter, the superimposed image creating unit 18 creates a superimposed image by superimposing the hand model reflected in the deformation in step S109 and the operation image on which the GUI parts are arranged, and displays the superimposed image on the display unit 3. .

Next, in step S111, the collision determination unit 19 determines whether or not there is a collision between the GUI part and the fingertip position point of the hand model. Details of the collision determination will be described later. If it is determined that there is no collision, the processes in steps S101 to S111 are repeated, and the hand model in the superimposed image is deformed according to the movement of the operator's hand. If it is determined that there is a collision, the collision determination unit 19 notifies the command transmission unit 20 of the command assigned to the GUI part that has the collision, and proceeds to step S112.

  In step S112, the command transmission unit 20 transmits the notified command to the operation target device 4, and the process returns to step S101. The operation target device 4 receives and executes the transmitted command. By repeating the above processing, the hand-shaped superimposed display information terminal operating device 100 operates the operation target device 4 by repeatedly transmitting a command. Note that the processing ends when the start switch is turned off by an operator or the like.

  By the operation described above, the operator can operate the hand model in the superimposed image by moving his / her hand while viewing the superimposed image displayed on the display unit 3. Then, the operator can operate the operation target device 4 by operating the hand model and superimposing the position of the fingertip on the GUI part.

  Below, each step demonstrated using FIG. 3 is demonstrated in detail.

  FIG. 5 is a diagram for explaining the operation of the contact area detection unit 1 in steps S101 and S102 of FIG. FIG. 5A shows an example of a state where the operator's hand is placed on the operation surface of the contact area detection unit 1. FIG. 5B is a diagram showing the contact point group detected by the contact area detection unit 1 as a black circle group. As shown in FIG. 5, the contact area detection unit 1 detects a contact area of the hand by detecting a contact point (contact point group) in steps S101 and S102 of FIG.

  FIG. 6 is a flowchart for explaining step S103 of FIG. 3 in detail. FIG. 7 is a diagram for explaining the fingertip position point and palm position point acquired in step S103 of FIG. Hereinafter, the processing in step S103 in FIG. 3 will be described in detail with reference to FIGS.

  First, in S103-1, the movement amount calculation unit 12 acquires data indicating the contact point group described with reference to FIG.

  Next, in step S103-2, the movement amount calculation unit 12 calculates a plurality of regions indicated by ellipses in FIG. 7 based on the positional relationship between the contact points. A general method may be used as a method of calculating a plurality of regions in S103-2. For example, when there are other contact points in the vicinity of 4 of the contact points, there is a method of dividing these contact points into a plurality of regions by regarding them as the same region. Since the contact area detection unit 1 detects finger contact and palm contact, the method of calculating the upper half of the operation surface of the contact area detection unit 1 for detecting finger contact and the operation for detecting palm contact. It is good also as a method different from the area | region calculation method of the lower half of a surface. For example, in the upper half of the operation surface that detects finger contact, in addition to the above-mentioned four points, the upper half of the operation surface is regarded as the same area even if they are two apart, and in the lower half of the operation surface that detects the palm, In addition to the vicinity of the four points, it may be regarded as the same region even if they are two apart in the left-right direction. As a result, misrecognition due to dents in the finger joint position, palm swell, and the like can be reduced, and detection accuracy of finger / palm contact can be improved.

  Next, in step S103-3, the movement amount calculation unit 12 performs an ellipse fitting process on each region calculated in step S103-2. FIG. 7 is a diagram for explaining a calculation method of the fingertip position point and the palm position point. In step S103-3, the movement amount calculation unit 12 surrounds the finger contact point group with an ellipse as shown by an ellipse A in FIG. 7A, and touches the palm as shown by an ellipse B in FIG. 7A. Enclose the point cloud with an ellipse. Here, the movement amount calculation unit 12 determines, for example, the lowest contact point group (see FIG. 7A) as the palm contact point group. Thereafter, the movement amount calculation unit 12 uses, for example, the upper end of the ellipse A as a fingertip position point (see FIG. 7B) and the center of gravity of the ellipse B as a palm position point (see FIG. 7C). reference). Note that the fingertip position point and the palm position point may be determined by using other methods or other positions.

  Next, in step S103-4, the movement amount calculation unit 12 holds the positions of the fingertip position point and the palm position point acquired in step S103-3. FIG. 8 is a diagram illustrating an example of the positions of the fingertip position point and the palm position point acquired in step S103-3 in FIG. In FIG. 8, black circles indicate fingertip position points, and black triangles indicate palm position points.

  Next, in step S103-5, the movement amount calculation unit 12 stands by until five fingertip position points and one palm position point are acquired, and when these are acquired, the process proceeds to step S104 (see FIG. 3).

  FIG. 9 is a flowchart for explaining in detail the processing in step S105 in FIG. First, in step S105-1, the movement amount calculation unit 12 notifies the operator to open a hand on the operation surface of the contact area detection unit 1, and obtains six initial position points.

  Next, in step S105-2, the movement amount calculation unit 12 uses the initial position point (fingertip position point and palm position point) acquired in step S105-1 and the model held by the model holding unit 15 from the beginning. The initial hand model is created by deforming the model hand model so that the corresponding fingertip position point and palm position point of the model hand model are. Thereafter, the movement amount calculation unit 12 displays the created initial hand model on the display unit 3. By this processing, the movement amount calculation unit 12 can create an initial hand model reflecting the size of the operator's hand and display it on the display unit 3. In addition, it is preferable that the movement amount calculation unit 12 also changes the size of the palm in accordance with changing the finger length of the model hand model. This makes it possible to create a more realistic initial hand model.

  Next, in step S105-3, the movement amount calculation unit 12 stores the position of the initial position point acquired in step S105-1 and the initial hand model created in step S105-2 in the model holding unit 15, and step The process returns to S103 (see FIG. 3).

  FIG. 10 is a flowchart for explaining the processing of steps S107 and S108 of FIG. The processes in steps S107 and S108 in FIG. 3 are performed as a unit. Therefore, hereinafter, the processes of steps S107 and S108 of FIG. 3 will be described simultaneously with reference to FIG. In the following, for convenience of explanation, the subject that executes each step of FIG. 10 is referred to as the model movement amount determination unit 13 or the finger bending angle determination unit 14, but this subject may be replaced depending on the step.

  First, in step S178-1, the model movement amount determination unit 13 determines the displacement of the palm position point acquired in step S106 from the initial position point as the parallel movement amount of the hand model.

  Next, in step S178-2, the finger bending angle determination unit 14 determines the initial position of one palm position point acquired in step S106 from the displacement of the five fingertip position points acquired in step S106 from the initial position point. Subtract the displacement from each point. Accordingly, the finger bending angle determination unit 14 calculates the distance between the palm position point and each fingertip position point. FIG. 11 is a diagram illustrating distances R1 to R5 from the palm position point to each fingertip position point. That is, in step S178-2, the finger bending angle determination unit 14 calculates the lengths R1 to R5 shown in FIG. Here, the amount of change in length of R1 to R5 shown in FIG. 11 is the amount of change caused by bending of the finger.

  Next, in step S178-3, the finger bending angle determination unit 14 determines the bending angle of each joint of each finger using the distance between the palm position point calculated in step S178-2 and each fingertip position point. .

  Next, in step S178-4, the model movement amount determination unit 13 calculates the rotation amount θ of each fingertip position point generated by the rotation about the palm position point. FIG. 12 is a diagram for explaining the rotation amount θ calculated in step S178-4 of FIG. In FIG. 12, as an example, the fingertip position point has moved from the initial position X to the position Y. In step S178-4, the model movement amount determination unit 13 calculates the rotation amount θ of the fingertip position points illustrated in FIG. 12 for the five fingertip position points.

Then, in step S178-5, the model movement amount determination unit 13 obtains the five rotation amount theta fingertip location points on average in step S78-4 calculates an average rotation amount theta _1. Then, the model movement amount determination unit 13 determines the calculated average rotation amount θ ₁ as the rotation amount of the hand model.

Next, in step S178-6, the finger bending angle determination unit 14 determines the rotation amounts θ of the five fingertip position points calculated in step S178-4 and the average rotation amount θ calculated in step S178-5. ₁ is used to calculate the rotation amount γ from the base of each finger. The rotation amount γ from the base of each finger is a value indicating the direction of each finger with respect to the palm. FIG. 13 is a diagram for explaining the processing in step S178-6 in FIG. FIG. 13 shows the same state as FIG. 12, and FIG. 13 shows the rotation amount θ calculated in step S178-4 and the average rotation amount θ ₁ calculated in step S178-5. . As illustrated in FIG. 13, the finger bending angle determination unit 14 calculates the rotation amount γ from the base of the finger using the rotation amount θ and the average rotation amount θ ₁ .

  By performing the series of processes described above, the model movement amount determination unit 13 and the finger bending angle determination unit 14 perform the processes of steps S107 and S108 in FIG. Determine the bending angle of the knuckles and the orientation of each finger relative to the palm.

  FIG. 14 is a flowchart for explaining the process of step S109 of FIG. By the processing of the flowchart of FIG. 14, the deformed shape of the hand model is determined using each value determined in steps S107 and S108. Hereinafter, with reference to FIG. 14, step S109 in FIG. 3 will be described in detail.

  First, in step S109-1, the model shape determination unit 16 determines the position of the hand model using the movement amount of the hand model determined in step S107.

  Next, in step S109-2, the model shape determining unit 16 determines the orientation of the hand model using the rotation amount of the hand model calculated in step S107.

  Next, in step S109-3, the model shape determining unit 16 uses the bending angle of each finger joint determined in step S108 and the orientation of the finger with respect to the palm (see FIG. 13) to determine the deformed shape of the hand model. decide. Thereafter, the process proceeds to step S110.

  FIG. 15 is a diagram showing a specific example of the deformation of the hand model in step S109 of FIG. FIG. 15A shows an initial position point (a fingertip position point and a palm position point in a state where the operator opens his hand). FIG. 15B is an initial hand model drawn using the initial position points shown in FIG. FIG. 15C shows the fingertip position point and palm position point when the operator bends the index finger as an example. FIG. 15D shows a hand model deformed using the fingertip position point and palm position point shown in FIG. FIG. 16 is a side view of the hand model shown in FIGS. 15B and 15D. As shown in FIGS. 15 and 16, in step S <b> 109 of FIG. 3, the hand model is deformed realistically according to the deformation of the operator's hand.

Here, as a method for obtaining the bending angle of the finger joint from the fingertip displacement, there is an inverse kinematics (hereinafter referred to as IK) technique well known in the field of robot engineering or the like. The IK technique is used to move the tip of an arm having a plurality of movable parts to a target position. When the arm has a plurality of movable parts, there are a plurality of solutions for the bending angle of the movable part in order to move the tip of the arm to the target position using the IK technique.

  Also in the present invention, since the finger has a plurality of joints, there are a plurality of solutions for the bending angles of the finger joints. For this reason, in the present invention, as an example, using the constraint condition that the palm and fingertip of the operator exist on the operation surface (on the same plane) and the constraint condition that the bending angles of each finger joint are equal, Is uniquely determined. That is, in step S108 of FIG. 3, the finger bending angle determination unit 14 uses the constraint condition that the fingertip position point and the palm position point exist on the same plane and the constraint condition that the bending angle of each finger joint is equal. Determine the deformed shape of the model. FIG. 17 shows an example of the hand model deformed using the constraint condition that the fingertip position point and the palm position point exist on the same plane and the constraint condition that the bending angles of the finger joints are equal in step S108 of FIG. FIG. FIG. 17A shows the displacement of the fingertip position point when the index finger is bent as an example. FIG. 17B is a side view of the bent index finger hand model shown in FIG. As can be seen from FIG. 17, normally, when the finger is bent, the three finger joints bend at the same time. Therefore, if the hand model is deformed using such a constraint condition, the operator can perform an operation without feeling uncomfortable. Further, when the hand model is deformed using such a constraint condition, the amount of calculation can be greatly reduced, and therefore the hand model that responds instantaneously to the movement of the operator's hand can be deformed.

  Note that the constraint condition for calculating the bending angle of the finger joint is not limited to this, and may be a constraint condition for uniquely obtaining a solution. However, as described above, the constraint condition for calculating the bending angle of the finger joint is preferably a constraint condition that is a hand model that does not cause the operator to feel uncomfortable, and is a hand model that allows the operator to act naturally. Restraint conditions are preferred. In addition, as a method for obtaining the bending angle of the arm movable portion from the target position in the IK technique, a method of repeatedly calculating using a Jacobian matrix, a method of calculating geometrically, and the like are known. May be used.

  FIG. 18 is a diagram for explaining the process of step S110 of FIG. In FIG. 18, A is an example of the operation image in which the read GUI parts are arranged, B is an example of the hand model deformed in step S109, and C is an example of the superimposed image in which A and B are superimposed. is there. As shown in FIG. 18, in step S <b> 110, the superimposed image creating unit 18 creates a superimposed image C by superimposing the operation image A on which the GUI parts are arranged and the deformed hand model B, and displays them on the display unit 3. To do. Note that when the superimposed image C is created, the hand model may be made semi-transparent or the like in order to make it easy to visually recognize the GUI part that overlaps the hand model.

  FIG. 19 is a diagram for explaining the processing in step S111 in FIG. When the GUI part and the fingertip position point of the hand model overlap each other in the superimposed image created in step S110, as shown by the arrow in FIG. 19A, the collision determination unit 19 performs the GUI part and the hand model. It is determined that the fingertip position point has collided. In this case, the collision determination unit 19 notifies the command transmission unit 20 of the command assigned to the GUI part for which the collision is determined.

In FIG. 19A, the collision determination in step S111 is performed by the overlap between the GUI part and the fingertip position point of the hand model. However, in step S111, for example, as shown by the arrow in FIG. 19B, the GUI part and the fingertip position point of the hand model overlap, and the finger joint angle having the overlapping fingertip position point is This may be done by bending more than a predetermined value. In this case, it is possible to reduce the operator from executing the command assigned to the GUI part by mistake. As another method, the collision determination may be performed by detecting that the GUI part and the fingertip position point of the hand model overlap and the click operation is performed at the position of the corresponding GUI part. Furthermore, instead of the click operation, the operator's pressing operation may be detected by detecting that the area of the finger contact point group has increased at the position of the GUI part, and the collision determination may be performed.

  Thereafter, the command is transmitted to the operation target device 4 in step S112. The operation target device 4 executes the received command.

  As described above, according to the hand-shaped superimposed display type information terminal operating device 100 according to the present invention, even when the user is away from the display unit 3, the operation that maintains the intuitive operation feeling of the touch panel display. It can be performed. Moreover, the hand shape superimposed display type information terminal operating device 100 according to the present invention can display a hand model that faithfully reproduces the movement of the user's finger on the display unit 3. As a result, the hand-shaped superimposed display type information terminal operating device 100 according to the present invention can provide a comfortable operation feeling without a sense of incongruity to the user.

(Modification)
In the embodiment of the present invention described above, the palm position point is detected by the same method as the fingertip position point. However, the palm position point may be detected using a wrist stand, as in a modification of the present invention described below. In the following description of the modified example of the present invention, only portions different from the embodiment of the present invention will be described.

  FIG. 20 is a diagram illustrating an example of a contact area detection unit 50 including a wrist base. FIG. 20A is a side view of the contact area detection unit 50 in a state where the operator places his hand. FIG. 20B is a view of the contact area detection unit 50 in a state where the operator places his hand, as viewed from above. FIG.20 (c) is the figure which looked at the contact area detection part 50 from the upper surface and the rear surface.

  As shown in FIG. 20, the operator performs an operation by placing a palm on the wrist stand and placing a finger on the operation surface for detecting contact. The wrist platform slides following the vertical and horizontal movements of the operator's palm, and detects the movement of the operator's palm. Thereby, the contact area detection unit 50 detects the displacement of the palm position point. In order to always place the palm of the operator at a certain position, it is preferable to provide a mark or a recess at the position where the palm of the wrist is placed. The wrist platform mechanism may be a general mechanism as long as the displacement of the operator's palm can be detected.

  FIG. 21 is a diagram illustrating an example of a contact point group detected by the operation surface of the contact area detection unit 50 and an example of a fingertip position point acquired by the movement amount calculation unit 12. FIG. 21A is an example of a contact point group detected by the operation surface of the contact area detection unit 50. FIG. 21B is an example of the fingertip position point acquired by the movement amount calculation unit 12. As shown in FIG. 21A, the operation surface of the contact area detection unit 50 detects a contact point group in the same manner as the operation surface of the contact area detection unit 1. Then, as illustrated in FIG. 21B, the movement amount calculation unit 12 acquires five fingertip position points. Here, the movement amount calculation unit 12 always acquires the palm position point, and performs processing using the displacement of the palm position point according to the displacement of the palm input from the wrist.

  As described above, the same effects as those described in the embodiment of the present invention can be obtained also by the modification of the present invention using the contact area detection unit 50 including the wrist stand.

まず、式１を用いて５つの指先位置点の重心位置を求める。そして、求めた重心位置の変位量と各指先位置点の位置の変位量との差を、それぞれ求める。或る指先位置点について、求めた差の値が閾値より大きければ、当該指先位置点に対応する指は曲がっていると判断して、手モデルの指を変形させる。また、或る指先位置点について、求めた差の値が閾値より小さければ、当該指先位置点に対応する指は曲がっておらず手全体が移動又は回転していると判断して、手モデルを移動又は回転させる。この様にして、手モデルの移動、回転及び変形を決定してもよい。 In the embodiment and the modification described above, the movement amount, the rotation amount, and the deformation shape of the hand model are determined using the method described with reference to FIGS. However, the movement amount, the rotation amount, and the deformed shape of the hand model may be determined using the method described below. Expression 1 is an expression for calculating the centroid positions (X _G , Y _G ) of five points on the xy plane.

First, the center-of-gravity positions of the five fingertip position points are obtained using Equation 1. Then, the difference between the obtained displacement amount of the center of gravity position and the displacement amount of each fingertip position point is obtained. If the calculated difference value is greater than the threshold value for a certain fingertip position point, it is determined that the finger corresponding to the fingertip position point is bent, and the finger of the hand model is deformed. If the calculated difference value is smaller than the threshold value for a certain fingertip position point, it is determined that the finger corresponding to the fingertip position point is not bent and the entire hand moves or rotates, and the hand model is Move or rotate. In this way, the movement, rotation and deformation of the hand model may be determined.

  INDUSTRIAL APPLICABILITY The present invention can be used for a hand shape superimposed display type information terminal operating device and method thereof, and particularly, when a CG model of an operator's hand is desired to be more realistically deformed in accordance with the movement of the operator's hand. Useful for.

The figure for demonstrating the outline | summary of operation | movement of the hand shape superimposition display type information terminal operating device 100 which concerns on embodiment of this invention. The figure which shows the structural example of the hand shape superimposition display type information terminal operating device 100 which concerns on embodiment of this invention. The flowchart for demonstrating operation | movement of the hand shape superimposition display type information terminal operating device 100 which concerns on embodiment of this invention. The figure which shows an example of the hand model which the model holding part 15 hold | maintains The figure for demonstrating operation | movement of the contact area detection part 1 in step S101 and S102 of FIG. Flowchart for explaining step S103 of FIG. 3 in detail The figure for demonstrating the fingertip position point and palm position point acquired by step S103 of FIG. The figure which shows an example of the position of the fingertip position point and palm position point which were acquired by step S103-3 of FIG. Flowchart for explaining in detail the process of step S105 in FIG. Flowchart for explaining the processing of steps S107 and S108 of FIG. The figure which shows distance R1-R5 from palm position point to each fingertip position point The figure for demonstrating rotation amount (theta) calculated by step S178-4 of FIG. The figure for demonstrating the process of step S178-6 of FIG. Flowchart for explaining the process of step S109 of FIG. The figure which shows the specific example of a deformation | transformation of the hand model in step S109 of FIG. The figure which looked at the hand model shown in Drawing 15 (b) and (d) from the side The figure which shows an example of the hand model deform | transformed using the constraint condition that the fingertip position point and the palm position point exist on the same plane and the constraint condition that the bending angles of each finger joint are equal in step S108 of FIG. The figure for demonstrating the process of step S110 of FIG. The figure for demonstrating the process of step S111 of FIG. The figure which shows an example of the contact area | region detection part 50 provided with a wrist stand. The figure which shows an example of the contact point group which the operation surface of the contact area detection part 50 detects, and an example of the fingertip position point acquired by the movement amount calculation part 12

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 50 Contact area | region detection part 2 Calculation part 3 Display part 4 Operation object apparatus 12 Movement amount calculation part 13 Model movement amount determination part 14 Finger bending angle determination part 15 Model holding part 16 Model shape determination part 17 GUI parts holding part 18 Superimposition Image creation unit 19 Collision determination unit 20 Command transmission unit 100 Hand shape superimposed display type information terminal operating device

Claims (9)

An operation device for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detector that detects a plurality of contact areas between the operation surface and the hand;
The plurality of contact areas are divided into a palm contact area where the operation surface and a palm are in contact with each other and a finger contact area where the operation surface and a finger are in contact with each other, and the palm contact area from an initial state where the fingers are extended. And a movement amount calculation unit for calculating a movement amount of the finger contact area;
A model movement amount determination unit that determines a parallel movement amount and a rotation amount of the CG model based on the movement amounts of the palm contact area and the finger contact area from the initial state;
A finger bending angle determination unit that determines a bending angle of each finger joint of the CG model based on the movement amount of the palm contact area and the finger contact area from the initial state;
A model shape determining unit that determines a deformed shape of the CG model according to a parallel movement amount, a rotation amount, and a bending angle of each finger joint of the CG model;
A GUI parts holding unit for holding GUI parts to which commands for the device are assigned;
A superimposed image creating unit that creates an image in which the CG model whose shape is determined by the model shape determining unit and a GUI part held in the GUI part holding unit are superimposed and displayed on the display unit;
A collision determination unit that determines a collision between the GUI part and the fingertip of the CG model;
An operation device comprising: a command transmission unit configured to transmit a command assigned to a GUI part for which a collision is determined to the device when a collision is determined by the collision determination unit.   The movement amount calculation unit determines a palm position point indicating a predetermined position in the palm contact area and a fingertip position point indicating a position corresponding to the tip of the finger in the finger contact area, and the palm position point The operation device according to claim 1, wherein a movement amount of the palm contact region and the finger contact region from an initial state where the finger is extended is calculated using the fingertip position point.   The operating device according to claim 1, wherein the model movement amount determination unit determines a movement amount of the palm contact area from the initial state as a parallel movement amount of the CG model.   The model movement amount determination unit calculates a rotation amount of the finger contact region around a predetermined position in the palm contact region, and determines an average of the calculated rotation amounts as a rotation amount of the CG model. The operating device according to claim 1, wherein:   The finger bending angle determination unit calculates a distance between the palm contact area and the finger contact area, and determines a bending angle of each finger joint of the CG model based on the calculated distance. The operating device according to claim 1.   The finger bending angle determination unit determines a bending angle of each finger joint of the CG model using a constraint condition that a bending angle of each finger joint of the same finger of the CG model is constant. The operating device according to claim 5.   The finger bending angle determination unit determines a bending angle of each finger joint of the CG model using a constraint condition that the palm contact area and each finger contact area exist on the same plane. The operating device according to claim 5. The finger bending angle determination unit further determines the direction of each finger with respect to the palm based on the movement amount of the palm contact area and the finger contact area from the initial state,
2. The operating device according to claim 1, wherein the model shape determining unit further determines a deformed shape of the CG model according to a direction of each finger with respect to the palm determined by the finger bending angle determining unit. An operation method for operating a device by displaying a CG model of a hand placed on an operation surface on a display unit,
A contact area detection step of detecting a plurality of contact areas between the operation surface and the hand;
The plurality of contact areas are divided into a palm contact area where the operation surface and a palm are in contact with each other and a finger contact area where the operation surface and a finger are in contact with each other, and the palm contact area from an initial state where the fingers are extended. And a movement amount calculating step for calculating a movement amount of the finger contact area;
A model movement amount determination step for determining a parallel movement amount and a rotation amount of the CG model based on the movement amounts of the palm contact area and the finger contact area from the initial state;
A finger bending angle determination step of determining a bending angle of each finger joint of the CG model based on the movement amount of the palm contact area and the finger contact area from the initial state;
A model shape determining step for determining a deformed shape of the CG model according to a parallel movement amount, a rotation amount of the CG model, and a bending angle of each finger joint;
A superimposed image creation step of creating an image in which the CG model whose shape has been determined in the model shape determination step and a GUI part that is held in advance and assigned a command to the device are superimposed and displayed on the display unit; ,
A collision determination step of determining a collision between the GUI part and the fingertip of the CG model;
When a collision is determined by the collision determination step, a command transmission step of transmitting a command assigned to the GUI part determined to have the collision to the device.

Images