JP5618926B2 - Multipointing device control method and program - Google Patents

Description

  The present invention relates to a control method and program for a multipointing device.

  Various man-machine interfaces are used for an operator to give appropriate instructions to an application running on a computer.

  For example, in order to make a pose (posture, shape, etc.) intended by the user take a skeleton model (skeleton model) as a three-dimensional model in the virtual space displayed on the display, the user gives an appropriate instruction. Need to give. For this purpose, various interfaces are used. Also in the field of two-dimensional graphics, a user interface is used to move or deform the two-dimensional graphics. In this case, a user interface that can easily and appropriately instruct user requests is required.

  For example, there are a mouse, a two-dimensional tablet, a multi-pointing device, and the like as an interface used by an operator for causing the above-described skeleton model to take an intended pose. These devices are also used as a user interface for two-dimensional graphics. It is also actively used as a user interface for other application programs.

  Of the above-described user interfaces, in recent years, multipointing devices that can point to a plurality of positions on a screen independently and give instructions to a computer have been used. A typical multi-pointing device includes a device (for example, a multi-touch screen) in which a touch panel is installed on the surface of a display and a plurality of locations can be pointed at the same time using a finger directly on the touch panel.

  Using this multipointing device, the operator can give various input instructions to the computer. Examples are given below. For example, “tap” corresponds to a click performed with the mouse by touching the screen and releasing it immediately. “Double tap” corresponds to a double click performed with a mouse by tapping twice continuously in a short time. “Drag” is an operation of moving a finger while touching the screen, and corresponds to dragging the mouse. “Flick” means an operation of moving a finger simultaneously with a tap, and there is no corresponding operation with a mouse. “Swipe” is an operation that sweeps the screen, and is an operation corresponding to the middle of dragging and flicking. This “swipe” is used, for example, when scrolling is continued even after a finger to be operated is released. “Pinch-in” and “pinch-out” correspond to operations to move two fingers closer and away from each other. “Touch and hold” means an operation of keeping the touch screen touched with a single finger for a certain period of time.

  Touch FLO is an interface that can input a plurality of instructions by tracing the screen with one finger and performing a specific gesture (for example, drawing a circle, drawing a straight line, etc.).

  However, even if a multipointing device or other various user interfaces are used, in order for the operator to give instructions as intended to the skeleton model, a complicated work is still required for the operator. For example, in order to make the skeleton model on the display take the pose (posture, shape, etc.) intended by the user, the instruction of the joint to be fixed, the movement of the joint to be moved, the release of the fixed joint, the fixation of another joint The user has to perform complicated and repeated operations such as instructions for moving the joint to be moved. Further, it is not possible to simply give an instruction to move a plurality of joints in an arbitrary direction.

  In view of such a situation, the method and the program according to the present embodiment can easily perform an instruction intended by the user using a user interface such as a multi-pointing device that can perform an instruction independently at a plurality of point positions. It is an object to provide a method and a program. The object of the present invention is not limited to application to the above-described skeleton model taken as an example.

  According to the present method, an object is displayed in a display area, and a pointing device capable of designating instruction information independently with respect to the display area by each of a plurality of pointers corresponds to each of the plurality of pointers. If the change of the instruction information corresponding to the step of acquiring the instruction information and the first pointer does not exceed a predetermined threshold, the instruction information corresponding to the first pointer corresponds to the first pointer Replacing the instruction information different from the instruction information to be associated with the first pointer, and changing the object displayed in the display area based on the instruction information corresponding to each of the plurality of pointers. A method is provided.

  According to another method of the present invention, a skeleton model is displayed in a display area, and the skeleton model is provided on each of a plurality of pointers by a pointing device capable of independently pointing a plurality of positions on the display area. Associating the joints of the plurality of pointers, calculating the pose of the skeletal model so that the distance between each of the plurality of pointers and the joint corresponding to each of the plurality of pointers approaches, and the calculated pose of the skeleton model A method is provided.

  By using a user interface such as a multi-pointing device that can independently give instructions at a plurality of point positions, it is possible to easily give instructions intended by the user.

It is a figure which shows the instruction | indication of the attitude | position of a skeleton model according to one Example of this invention. It is a figure which shows the instruction | indication of the attitude | position of a skeleton model according to one Example. FIG. 6 is a diagram of deforming a virtual paper object according to one embodiment. It is the figure which showed an example regarding matching with a pointer and the object controlled by a pointer using a skeleton model. It is a figure which shows the instruction | indication of the attitude | position of a three-dimensional object according to one Example. It is a figure which shows the hardware constitutions according to one Example. FIG. 3 shows a flowchart of a method according to one embodiment. FIG. 6 shows a flowchart of another method according to one embodiment. It is a figure which shows a skeleton model, a master bone, a skeleton, and a coordinate system. It is a figure which shows the control method of the skeleton model by CCD method.

  For example, there is inverse kinematics (inverse kinematics) as an example of a technique for controlling a skeleton model used when simulating postures (poses, movements, etc.) of human bodies, animals, structures, etc. in a three-dimensional virtual space. In this inverse kinematics, a skeletal model is defined by multiple bones and joints. In the case of a skeletal model based on inverse kinematics, it is necessary to control the position, angle, rotation (bending, twisting), etc. of each bone and joint as intended. For example, in the human skeleton model shown in FIG. 9A, each bone is connected by a joint, and the movement of each bone affects the movement of other bones. In inverse kinematics, there are various skeletal model control methods such as CCD (Cyclic-Coordinate-Desent). Details will be described later.

  In order to control the posture of the skeleton model, at least a joint to be fixed and a joint to be moved are designated. Note that the tip of a bone such as a fingertip is also treated and described as a joint in this specification. For the joint to be moved, it is necessary to specify the direction and amount (distance) to move. Note that joints other than the fixed joint and the joint to be moved move to appropriate positions in accordance with the skeleton model.

  In this specification, in order to avoid unnecessarily complicating the description of the embodiments, the inverse kinematics will be described only as a general principle and will not be described in detail, as will be described later. Note that the present invention is not limited to application to a specific skeleton model.

  FIG. 1 illustrates skeletal model pose control using a multipointing device, in accordance with one embodiment of the present invention. Hereinafter, the position where the operator's finger points to the multipointing device is referred to as a pointer position. That is, the pointer refers to a portion where an operator's finger or the like is touching the multi-pointing device.

  FIG. 1A shows a skeleton model X100 displayed on the screen. The skeletal model X100 has arms extended horizontally and is facing this direction, with the right hand at position a100 and the left hand at position b100. Also, the right foot tip is at position α100 and the left foot tip is at position β100. The above shows the initial position of the skeleton model X100. Here, the position refers to a position on the screen.

  The pointer of the operator's right hand R1 exists at the position b100. In addition, the pointer of the operator's left hand L1 exists at the position α100.

  The multipointing device sequentially measures the pointer position of the right hand R1 and the pointer position of the left hand L1, and the position of each pointer can be known from these measurement results. The speed of the pointer can be calculated by changing the pointer position with time. The calculated pointer speed of the right hand R1 is VR1, and the pointer speed of the left hand L1 is VL1. Further, it is assumed that a predetermined speed threshold value Vk is stored in the memory.

  In FIG. 1A, the pointer of the operator's right hand R1 has moved from the position b100 to the position b101 in the vector n100 direction. Further, the pointer of the operator's left hand L1 stops at the position α100. Therefore, the speed VR1 of the operator's right hand pointer R1 is VR1> Vk, and the speed VL1 of the left hand pointer L1 of the operator satisfies VL1 ≦ Vk.

  In this case, since the speed VL1 of the pointer of the left hand L1 does not exceed the speed threshold value Vk, the pointer of the left hand L1 does not give an instruction to move the right foot tip of the skeleton model but an instruction to fix the joint of the right foot tip to the skeleton model. give. Then, since the velocity VR1 of the pointer of the right hand R1 exceeds the threshold value Vk, the pointer of the right hand R1 gives an instruction to move the joint of the left hand tip of the skeleton model to the skeleton model.

  FIG. 1B shows a skeleton model X101 having a new posture obtained as a result of the operation shown in FIG. The joint at the right foot tip of the skeleton model X101 is fixed at a position α100 as a fixed joint. The joint on the left hand side of the skeleton model X101 has moved to the position b101. Based on this operation, the skeletal model moves other bones and joints to appropriate positions. The right hand has moved to position a101. Also, the left foot tip has moved to position β101. How to determine the positions of other joints and bones depends on the inverse kinematics calculation method and the skeletal model modeling. How to set the inverse kinematics calculation method and skeletal model modeling is a matter to be determined by those skilled in the art.Select the appropriate calculation method, appropriate model, setting value, etc. as necessary. That's fine.

  FIG. 1B further shows that the pointer of the left hand L1 is moved from the position α100 to the position α101 in the direction of the vector n101, and the pointer of the right hand R1 remains at the position b101.

  FIG. 1C shows a pose of the skeleton model X103 in a state where the above-described operation of FIG. In the skeleton model X103, the joint of the right foot tip is dragged and moved to the position α101. The joint on the left hand is stopped at the position b101. In addition, by this operation, the other joints and bones are also moved, the joint at the right hand tip is moved to the position a102, and the joint at the left foot tip is moved to the position β102.

  In the above embodiment, the positions of a plurality of pointers are independently measured by a multipointing device, and the pointer speed is compared with a predetermined speed threshold value Vk. Then, depending on the pointer speed, for the pointer whose pointer speed does not exceed the threshold value Vk, the corresponding joint movement instruction is replaced with the corresponding joint fixing instruction. On the other hand, for a pointer whose pointer speed exceeds the threshold value Vk, an instruction to move the corresponding joint is given. This movement instruction provides the skeleton model with the original instruction from the pointer as it is.

  Note that when the movement distance of the pointer exceeds a certain threshold value, the instruction to move the joint may be used without performing the process of replacing the skeleton model with the joint fixing instruction regardless of the speed of the pointer. Further, a movement distance threshold value may be used instead of the speed threshold value Vk. For example, the movement distance from the position of the pointer to the current pointer position at the time when the skeleton model was last redrawn may be measured, and this movement distance may be compared with a movement distance threshold.

  FIG. 7 is a flowchart showing this operation. First, in step 702, N pieces of independent position information are acquired from the multipointing device. In the example of FIG. 1, N = 2.

  In step 704, it is determined whether or not the moving speed of the first pointer exceeds the speed threshold value Vk. When the speed of the first pointer does not exceed the threshold value Vk (in the case of “Yes”), the process moves to step 705, and the instruction of the first pointer is set in the joint fixed point instruction information. Conversely, if the moving speed of the first pointer exceeds the speed threshold value Vk (in the case of “No”), the process moves to the determination step 706 for the second pointer.

  In step 706, the same determination as in step 704 is performed for the second pointer. Step 707 is executed as necessary.

  The same determination is performed up to the Nth pointer determination step 708. Step 709 is executed as necessary.

  In step 710, the pose of the skeleton model is calculated using the instruction information of the N pointers. If the pointer speed does not exceed the speed threshold value Vk in the above determination step, the pointer instruction information is replaced with the fixed point instruction information. Therefore, the fixed point instruction information is used for the calculation of the skeleton model. Apply. In other cases, the pointer instruction information is used as a movement instruction that is original instruction information.

  In step 712, the calculated skeleton model is displayed.

  When the display is completed, the process returns to step 702. By repeating the above steps, it is possible to easily cause the skeleton model to take a desired pose.

  If the operations in FIGS. 1A to 1C are performed using a mouse, for example, the operation is as follows. Dragging with the left mouse button is defined as an instruction to move the joint, and right clicking the mouse is defined as an instruction to fix the joint or to release the fixation of the joint.

  First, in order to designate a fixed joint, the right foot of the skeleton model is right-clicked with the mouse. This indicates that the joint at the right foot of the skeleton model is a fixed joint. Next, the mouse is moved to the joint on the left hand side of the skeleton model, and left dragged from the position b100 to the position b101. By this operation, the state of the skeleton model X101 of FIG. Next, the joint of the left hand of the skeleton model is right-clicked to instruct the joint to be fixed. Next, the mouse cursor is moved to the right foot of the skeleton model, right-clicked, and the release of the fixed joint is instructed. Then, the left mouse drag is performed from position α100 to position α101. By this operation, the state of the skeleton model X103 in FIG.

  As described above, when a pointing device having a single pointer such as a mouse is used, the control of the pose of the skeleton model shown in FIG. 1 becomes very complicated. On the other hand, in the case of the above-described embodiment using a multipointing device, the posture of the skeletal model can be simply indicated.

  Note that the operations of the embodiments shown in this specification may be combined with various operations using a multipointing device as exemplified above as long as there is no contradiction. For example, if one pointer out of a plurality of pointers is stationary and the pointer instruction is a joint fixing instruction, if the stationary time of the pointer exceeds a certain time, it is fixed even if the finger is released. The instruction may be continued. This operation can also be regarded as an operation in which the functions of the above-described embodiment and the touch and hold function are used together.

  FIG. 2 shows another embodiment. As shown in FIG. 2 (a), the movement speeds of the two pointers located at the two toes do not exceed the threshold value Vk, so the joints at the two toes of the skeleton model are respectively the two pointers of the operator's left hand L2. Instructed as fixed. Further, as shown in FIG. 2 (b), the joints of both hands of the skeleton model are instructed to move by the two pointers of the right hand R2 of the operator. That is, the pointer corresponding to the operator's right hand index finger moves from position a200 to a201 at a speed exceeding the threshold Vk, and the pointer corresponding to the operator's right thumb extends from position b200 to position b201 and exceeds the threshold Vk. Moving at speed. For this reason, the two pointers corresponding to the right hand R2 of the operator give the skeleton model a movement instruction that is the original instruction. Accordingly, the joint on the right hand side of the skeleton model moves from a200 to a201. Similarly, the joint on the left hand side of the skeleton model moves from b200 to b201. Since the joints of both foot tips of the skeletal model are designated as fixed joints, the joints of the left and right foot tips remain at position α200 and position β200, respectively.

  As described above, the deformation of the pose of the skeleton model from FIG. 2A to FIG. 2B can be realized by performing the same operation as the method shown in FIG. 1 and FIG.

  Note that the pose of the skeleton model may be changed by using the method of the modification shown in FIG. 8 without using the speed threshold Vk.

  FIG. 8 shows a processing flow of the modified method.

  In step 804, the correspondence between each of the plurality of pointers and the joint of the skeleton model is determined. For example, in the case of FIG. 2A, the operator's four fingers are touching the screen, and the four pointers exist at position a200, position b200, position α200, and position β200, respectively. Each of these pointers is associated with the joint of the right hand tip, the joint of the left hand tip, the joint of the right foot tip, and the joint of the left foot tip of the skeleton model. The total number N of pointers is N = 4. The association between the pointer and the joint will be described later with reference to FIG.

  In step 806, the variable n is initialized (n = 1) for the iterative algorithm.

  Step 810 calculates the pose of the skeletal model to move the joint corresponding to this pointer to the position of the nth pointer (or bring them closer together).

  Step 812 determines whether the variable n for the iterative algorithm is n ≧ N.

  Step 814 is executed when the determination in step 812 is “Yes”. In this case, since the processing has been completed for all N pointers, the skeleton model is displayed.

  Step 816 is executed when the determination in step 812 is “No”. In this case, since processing has not been completed for all N pointers, n is incremented and the processing returns to step 810.

  In the above-described flow, the skeleton model is calculated for each pointer. However, the position information of all pointers may be given to calculate the skeleton model at a time. Which calculation method is executed may be determined in consideration of the skeleton model to be used, the time required for the calculation, and the like.

  Step 818 updates the association between the pointer and the joint if there is a lost pointer or a newly generated pointer. When the number of pointers increases or decreases, the number N of pointers is updated appropriately. Then, the process returns to step 806.

  For example, in the case of the example in FIG. 2, the above processing is performed when the right-hand joint and the left-hand joint of the skeleton model move from position a200 to position a201 and from position b200 to position b201, respectively. The processing steps shown in FIG. 8 are repeated a plurality of times. Note that the distance threshold Lp may be set, and the skeleton model may be calculated only when the distance between the pointer and the corresponding joint is larger than the threshold Lp. In this case, the smaller the Lp is, the smoother the skeleton model is displayed again. An appropriate distance threshold Lp may be set in accordance with the complexity of the skeleton model and the calculation capability of the processing machine. The distance threshold Lp may be set by the user. Alternatively, a predetermined threshold value Lp may be automatically set according to the complexity of the skeleton model. The complexity of the skeletal model is the number of bones, the number of joints, the number of polygons to be pasted around the skeletal model, whether or not texture mapping is applied to the polygons, and whether or not the bending angle of joints in the skeletal model is limited The determination may be made based on the time required for rendering the skeleton model.

  If the pointer disappears or newly appears in the middle of the iterative process of FIG. 8, the iterative process is interrupted, and the skeleton model display in step 814 or the step 818 is forcibly performed. You may jump to.

  FIG. 3 is a diagram showing an operation of folding virtually displayed paper as another embodiment.

  As shown in FIG. 3A, paper P301 is shown on the screen. The pointer of the left hand L3 points to the position α300, and the speed does not exceed the speed threshold value Vk. The pointer of the right hand R3 points to the position β300 and moves in the vector n5 direction. The speed exceeds the speed threshold Vk.

  In this case, the pointer of the left hand L3 gives an instruction to fold the straight line m300 to the virtual paper model. In the example of FIG. 3A, the crease m300 may be determined as “a straight line passing through the point α300 and having an angle with the vector n5”. Alternatively, instead of a right angle, a predetermined angle may be set. In addition, if the paper is a quadrilateral, the direction of the crease m300 may be parallel to the diagonal line, and an appropriate value may be determined in advance by the user or the system designer.

  FIG. 3B shows the paper P302 in which the corner of the paper is moved to the position β301 by moving the pointer of the right hand R3 in the vector n5 direction.

  FIG. 3C shows a state of the paper P303 in which the paper is completely folded at the crease m300 and the paper corner reaches the position β302.

  In the embodiment shown in FIG. 3, if the pointer speed exceeds the speed threshold value Vk, the drag that is the original instruction of the pointer is applied to the deformation process on the virtual paper model as the movement instruction information. If the pointer speed does not exceed the speed threshold value Vk, an instruction to set a paper fold is applied to the virtual paper model deformation process instead of the pointer movement instruction.

FIG. 4 shows an example of association between a pointer and an object controlled by the pointer using a skeleton model. FIG. 4A shows a part of a skeletal model in which bones 401 to 403 are connected by joints 411 and 412. A pointer 400 exists near the skeleton model. Here, the pointer 400 exists at a position separated from the joint 411 by L450. Further, the pointer 400 is present at a position separated from the joint 412 by L460. And L450 <L460. In this case, the joint 411 closest to the pointer 400 may be associated with the pointer 400. It should be noted that a certain threshold value La is provided, and when the distance from the pointer 400 to the nearest joint exceeds the threshold value La, the association between the pointer 400 and the joint may not be performed.

  FIG. 4B shows a case where the distance from the pointer 400 to the joints 411 and 412 becomes L451 and L461 (L451> L461) after the pointer 400 is associated with the joint 411, respectively. In this case, the joint closest to the pointer 400 is a joint 412. However, since the pointer 400 is already associated with the joint 411 in FIG. 4A, this association may be maintained and the association may not be changed.

  Thus, by maintaining a pair of a pointer and a joint that are once associated, the skeleton model can be deformed in a more stable manner. This is because if the association between the pointer and the joint is changed in the middle, a behavior may be exhibited in which a joint not intended by the user is attracted to the pointer.

  Note that the above-described association is an example, and the present invention is not limited to this.

  FIG. 5 shows another embodiment. As an example, a virtual object of a mobile phone is displayed.

  In FIG. 5A, the pointer of the operator's left hand L5 is at a position α500 at the top of the mobile phone. The pointer of the operator's right hand R5 is present at the position β500 of the flank portion of the mobile phone. The speed of the left hand L5 pointer does not exceed the speed threshold Vk, and the right hand R5 pointer moves in the vector n500 direction at a speed exceeding the speed threshold Vk. In the case of this embodiment, the pointer of the left hand L5 passes through the position α500 on the surface of the mobile phone, and gives an instruction to set, for example, a straight line s500 passing through the center of gravity of the mobile phone as the rotation center axis of the mobile phone. Then, as the operator moves the pointer of the right hand R5 in the direction of the vector n500, the mobile phone rotates about the rotation axis s500.

  FIG. 5B shows a state where the above-described operation is performed and the pointer of the right hand R5 moves to the position β501, and the mobile phone is rotated around the rotation axis s500 accordingly.

  FIG. 5B further shows that in this state, the moving speed of the pointer of the right hand R5 does not exceed the speed threshold value Vk, and conversely, the moving speed of the pointer of the left hand L5 exceeds the speed threshold value Vk and moves toward the vector n501. It shows how it began to move. In this case, the mobile phone rotates about the straight line s501 passing through the pointer of the right hand R5 and passing through the center of gravity of the mobile phone (the straight line s501 is a straight line that is substantially perpendicular to the paper surface).

  FIG. 5C shows a state where the pointer of the left hand L5 has moved to the position α501. As shown in the figure, the mobile phone rotates about a straight line s501 as a rotation center axis.

  In the above-described embodiment, a straight line passing through the center of gravity of the mobile phone from the point on the surface of the mobile phone that is a pointer position that does not exceed the threshold speed Vk is used as the rotation center. A predetermined point may be determined. Or not rotation around the axis. Rotation with the pointer position as the center point of rotation may be performed. How to set the rotation may be determined in advance.

  In the above embodiment, it is possible to easily rotate a virtual object in a desired direction by a simple operation using two pointers.

  FIG. 6 illustrates an example hardware configuration of a system 600 that implements the various embodiments described herein. The system 600 includes a CPU 601, a display 602, a touch screen 604, a keyboard 606, a mouse 608, a ROM 610, a RAM 612, an external memory interface 614 that reads and writes external memory media 615, a communication interface 616 that communicates with a network, and each hardware. A bus 620 to be connected is included.

  The skeleton model will be described below with reference to FIGS. 9 and 10.

  In such a skeletal model used for controlling the posture (pose, movement, etc.) of a human body model, animal, structure, etc. in a three-dimensional virtual space, the position of each bone and joint shown in FIG. , Angle, rotation (bending, twisting), etc. must be controlled as intended. In the skeleton model shown in FIG. 9A, each bone is connected by a joint, and the movement of each bone affects the movement of other bones.

  As an example of a method for controlling the skeleton model, a CCD (Cyclic-Coordinate-Desent) method in inverse kinematics will be described below. The present invention is not limited to application to this method.

As shown in FIG. 10, a skeleton model having a plurality of joints J _{1 to} J ₃ and a plurality of bones B _{1 to} B ₃ will be described as an example. In the case where the position of the joint J ₃ is fixed, showing the operation of the CCD method when moving the point A of the bone B ₁ skeleton model point D which is a target point below. In the CCD method, only the angle of _one joint (J ₁ ) is moved in one calculation step to minimize the distance between point A and point D, and this operation is sequentially performed on all joints (J _{1 to} J ₃ ). Calculate about. By this operation, the distance between the point A and the point D is sequentially reduced. Repeat this calculation. A specific calculation method is shown below.
(1) As shown in FIG. 10 (b), first, the distance between point A and point D rotation angle only by operating the joints J ₁ is set to be the shortest. For this purpose, around the joint J ₁ rotates the bone B ₁ by an angle theta _1, on a straight line h ₁ of connecting the joints J ₁ and the target D, to move the point A. Thus, point A, in order to position on the straight line h _1, the distance between the points A and D are the shortest.
(2) As shown in FIG. 10 (c), the distance between the point A and the point D and then operated by the rotation angle of the joint J ₂ is set to be the shortest. For this purpose, around the joint J ₂ rotates the bone B ₂ by an angle theta _2, on a straight line h ₂ for connecting the joint J ₂ and the target D, to move the point A. This minimizes the distance between point A and point D.
(3) As shown in FIG. 10 (d), the distance between the point A and the point D and then operated by the rotation angle of the joint J ₃ is made to be the shortest. For this purpose, around the joint J ₃ rotate the bone B ₃ by the angle theta _3, on a straight line h ₃ of connecting the joint J ₃ and the target D, to move the point A. This minimizes the distance between point A and point D.
(4) Repeat the same operation from (1) above.

  The calculation may be terminated when at least one of the distance between the point A and the point D is equal to or less than the threshold value e and the number of iterations exceeds the upper limit number is satisfied.

  In the above example, the point A is an end point, but may generally be a skeletal joint.

  Although various embodiments of the present invention have been described above, the present invention is not limited to the application to the above-described multipointing device, and can be applied to various other user interfaces (for example, a single Even a pointing device having a pointer of (5) can be applied to a case where a plurality of operators are connected via a network or the like and the same object is deformed). Further, the present invention is not limited to the application to the field of computer graphics, but also applies to instructions from a user for computer applications that display using a mobile phone, a tablet computer, a projector, a television, or the like. obtain. In addition, not only an instruction by a single operator for an application program but also a plurality of independent instructions by a plurality of operators may be used.

  In general, a multipointing device brings a plurality of fingers of an operator into contact with the touch panel, and each contact position functions as a pointer. It may be a thing (for example, a pen). The fingers touching the touch panel may be fingers of both hands or a plurality of fingers of one hand. In addition, the multipointing device is not limited to a touch panel, and may be any input user interface that can give a plurality of independent instructions. Furthermore, the user interface may be an interface without contact. For example, sound, light, or the like may be used. Moreover, the instruction | indication by a some operator may be sufficient. In this case, a plurality of operators may exist at remote locations via a network.

  The pointer may be displayed on the screen like a mouse cursor, or may not be displayed. This is because the user can recognize the position of the pointer if the finger is touching the screen even if it is not displayed.

  In the above-described embodiment, an example in which the pointer can point on the two-dimensional display screen is shown. Therefore, for example, the distance between the joint and the pointer in the above-described embodiment indicates the distance on the screen between the joint and the pointer projected on the screen. However, it goes without saying that the various embodiments described above can be similarly applied to a multi-pointing device in which a pointer can move and point in a three-dimensional space. In this case, for example, the distance between the joint and the pointer indicates a distance in a three-dimensional virtual space.

  The processing steps of the above-described embodiments may be executed by changing the order as long as there is no contradiction. Further, it goes without saying that a method in which the steps of the method according to the claims are interchanged is also included in the technical scope of the claims.

Claims (7)

A method for displaying an object in a display area,
Obtaining instruction information corresponding to each of the plurality of pointers from a pointing device capable of independently specifying instruction information for the portion of the object pointed to by a plurality of pointers with respect to the display area;
Change instruction information corresponding to a predetermined pointer, it does not exceed the predetermined threshold, the instruction information corresponding to the predetermined pointer specific instructions of different nature from the instruction information corresponding to the predetermined pointer Replacing with information and associating with the predetermined pointer;
Changing an object displayed in the display area based on the instruction information corresponding to each of the plurality of pointers or the replaced specific instruction information ;
I have a,
The instruction information is a position of the part of the object pointed to, and the specific instruction information is an instruction to fix the part of the object pointed to.
Method. The threshold is a moving speed or moving distance of the position indicated by the pre-Symbol instruction information,
The method of claim 1. The object is a skeletal model in which bones and bones are connected by joints,
The instruction information is a joint position,
The specific instruction information is an instruction to fix a joint.
The method according to claim 1 or 2.   The method according to claim 1, wherein the pointing device is a touch panel having a display for displaying the object.   The method according to claim 1, wherein the pointer is invisible in a display area. A method for displaying a skeleton model in a display area,
Associating a joint of the skeleton model with each of a plurality of pointers by a pointing device capable of independently pointing a plurality of positions on a display area;
And each of the plurality of pointers, said plurality of pointers to approach the distance between the corresponding joints each, calculating the pose of the skeleton model,
Displaying the calculated pose of the skeletal model;
I have a,
The step of associating includes
If no joint is associated with the predetermined pointer, the first joint closest to the predetermined pointer is set as the joint corresponding to the predetermined pointer, and the second pointer is already set to the second pointer. When the joint is associated, the correspondence between the predetermined pointer and the second joint is maintained even when the third joint closest to the predetermined pointer exists. Step,
Including, METHODS. The program which has a command which makes a computer perform the method of any one of Claims 1 thru | or 6 .

Images