JP6091407B2 - Gesture registration device - Google Patents

Description

  The present invention relates to a gesture registration device that generates and registers a gesture definition necessary for a device to recognize a gesture performed by a human.

  It is possible to obtain time-series data of the posture of the human body by a motion capture system capable of calculating in real time the position of the body part that characterizes the structure of the human body and the relationship (skeleton) from the video image of the human body. With the advent of such a system that can acquire the movement of the human body, it has become easier to develop an application that uses body posture data as input to a computer and performs some kind of control processing. For example, by defining a series of movements of human arms and legs as gestures in advance on a computer and associating them with a certain control process, specific movements are identified from human actions during application execution and corresponding to them Control processing can be executed. Gesture recognition is performed by pattern matching between input time-series posture data and pre-registered gesture definition data.

  A gesture can be represented by a sequence of a plurality of postures (poses). The gesture definition is composed of a plurality of poses (conditions of body parts constituting posture) that are keys in motion and transition conditions between poses. By defining these in advance, when all of the conditions included in the gesture definition having an operation performed at the time of executing the application are satisfied, the gesture can be identified. The motion capture system can distinguish a plurality of characteristic human body parts (for example, a hand, an elbow, a shoulder, and a head) by a known image processing technique, and can acquire a three-dimensional coordinate of each part. One pose is defined using the three-dimensional coordinates of each body part, the angles of skeletons (joints) calculated from the three-dimensional coordinates, and the like. The pose does not necessarily need to be defined from whole body information, and may be defined from information of one or more partial body parts. A pose can be defined using the positional relationship between body parts. In order to define a state where the right arm is lowered as a pose, for example, a relationship that the position of the right hand is positioned below the position of the base of the right foot can be set as a condition. It is also possible to provide a transition condition for moving the body from one pose to the next. Examples of transition conditions include the movement direction, movement speed, and joint angle during movement of each body part. If a movement that satisfies the transition condition between poses is performed during gesture recognition, the recognition is continued.

  When using gesture recognition as an application interface, it is necessary for a gesture registrant such as an application developer to define and register an appropriate gesture corresponding to the function. As a conventional technique, a method has been proposed in which an action actually performed by a gesture registrant is registered as a gesture corresponding to a specific control process of the device. For example, in the method of Patent Document 1, when registering a gesture, first, the user input recognition unit periodically generates gesture feature information including the moving direction and moving amount of parts such as left and right arms, elbows, hands, and fingers from a captured image. And record it as a gesture definition. At the time of gesture recognition, similarly, gesture feature information is extracted from the captured video, matched with gesture feature information defined in advance, and the gesture is recognized. The developer can register the gesture by exemplifying the gesture examined according to the function in this way.

JP 2012-181809 A

  When human natural behavior is used as an input to a computer, it is important to recognize only human motions that are supposed to be performed as gestures. To achieve this, application developers need to design gesture definitions with constraints that appropriately filter their actions. In the definition of gestures using the positional relationship between body parts, parameters such as body parts and joint angles used for recognition are the constraint conditions for poses, and parameters such as body part movement speeds and joint angles are restrictions on transitions between poses. An example of the condition. Then, the developer needs to set a constraint condition of an appropriate recognition range (a parameter type set as a constraint, a parameter value range, etc.) according to the contents of the application.

  However, in practice, it is difficult to define a gesture as intended by the developer, and the definition is often modified repeatedly during evaluation. For example, even if the developer intends to design a gesture that the user wants to perform, it is actually intended because the constraints that are the scope of recognition are too wide, the scope is too strict, or there is a leak in the scope. May not recognize the street. Specifically, if the recognition constraint is too wide, even a user's movement that the developer did not assume may be recognized as a gesture. On the other hand, if the constraint condition is too narrow, the developer may not be recognized as a gesture even if the user performs a movement close to the intention. The cause of this is that it is difficult for the developer to recognize the recognition state of the gesture he designed on the computer, and conversely, the computer cannot correctly determine what parameter the developer focuses on as a feature of the gesture. It is in. There is a need to be able to define gestures with constraints within an appropriate range by bringing such developers and computers closer together.

  In this regard, in Patent Document 1, it is possible to register a gesture by using the movement direction and movement amount parameters of the body part of the gesture registrant as a constraint condition. However, it is not possible to register a gesture in which the width of the constraint condition is set appropriately.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide a gesture registration device capable of defining and registering a gesture in accordance with a developer's intention.

  The gesture registration device according to the present invention includes a motion data acquisition unit that acquires motion data of a gesture, and gesture analysis that generates pose data indicating the three-dimensional coordinates of each body feature point in each pose constituting the gesture based on the motion data. Based on the pose data, a feature analysis unit that calculates feature data indicating the constraint conditions of the parameters representing the features of the gesture at the time of each pose and at the time of transition between poses, and a human body that defines the movable range of the skeleton constituting the human body The motion model database, a candidate gesture generation unit that generates other candidate gestures within the movable range of the skeleton based on the human body motion model database, while satisfying the constraints of some parameters, and the candidate gesture to the user Present and use the candidate gesture An input / output unit that accepts an operation of the user, and the candidate gesture generation unit generates a new candidate gesture upon accepting a user operation that rejects the candidate gesture in the input / output unit, and the user's operation on the candidate gesture And a gesture definition registration unit for determining constraint conditions for the other parameters and registering gesture definition data indicating the constraint conditions of each parameter in the gesture in the gesture definition database.

  The gesture registration device according to the present invention includes a motion data acquisition unit that acquires motion data of a gesture, and gesture analysis that generates pose data indicating the three-dimensional coordinates of each body feature point in each pose constituting the gesture based on the motion data. Based on the pose data, a feature analysis unit that calculates feature data indicating the constraint conditions of the parameters representing the features of the gesture at the time of each pose and at the time of transition between poses, and a human body that defines the movable range of the skeleton constituting the human body The motion model database, a candidate gesture generation unit that generates other candidate gestures within the movable range of the skeleton based on the human body motion model database, while satisfying the constraints of some parameters, and the candidate gesture to the user Present and use the candidate gesture And an input-output unit that accepts the operation. The candidate gesture generation unit generates a new candidate gesture upon accepting a user operation for rejecting the candidate gesture at the input / output unit, and determines a constraint condition for the other parameters based on the user operation on the candidate gesture. . The gesture registration device according to the present invention further includes a gesture definition registration unit that registers gesture definition data indicating constraint conditions of each parameter in the gesture in the gesture definition database. As a result, various gestures obtained by partially deforming the gestures indicated by the user are presented to the user as candidate gestures, and the constraint conditions of each parameter are determined by the user by determining the constraint conditions of each parameter based on the user operation on them. It is possible to review without omission, and it is possible to easily register an appropriate gesture definition according to the intention.

1 is a block diagram illustrating a configuration of a gesture registration device according to a first embodiment. It is a figure which illustrates the data stored in the body structure table by Embodiment 1. FIG. It is a figure which illustrates the data stored in the body movable range table by Embodiment 1. FIG. It is a figure which illustrates the body feature point analyzed by the gesture analysis part by Embodiment 1. FIG. It is a figure which illustrates the coordinate system at the time of acquiring the three-dimensional coordinate of a body feature point in the gesture analysis part by Embodiment 1. FIG. It is a figure which illustrates the pose data which the gesture analysis part by Embodiment 1 produces | generates. It is a figure which illustrates the pose analyzed by the gesture analysis part by Embodiment 1. FIG. It is a figure which illustrates the pose analyzed by the gesture analysis part by Embodiment 1. FIG. It is a figure which illustrates the feature data which the feature analysis part by Embodiment 1 generates. 6 is a diagram illustrating a display example of candidate gestures in the input / output unit according to Embodiment 1. FIG. 6 is a flowchart illustrating an operation of a candidate gesture generation unit according to the first embodiment. It is a figure which illustrates the gesture definition data stored in the gesture definition database by Embodiment 1. FIG. It is a figure which illustrates the pose data produced | generated by the gesture analysis part by Embodiment 2. FIG. It is a figure which illustrates the pose data produced | generated by the gesture analysis part by Embodiment 2. FIG. FIG. 10 is a block diagram illustrating a configuration of a gesture registration device according to a third embodiment. 10 is a flowchart illustrating an operation of a candidate gesture generation unit according to Embodiment 3. FIG. 10 is a diagram illustrating a pose analyzed by a gesture analysis unit according to the third embodiment. FIG. 10 is a diagram illustrating a pose analyzed by a gesture analysis unit according to the third embodiment. FIG. 10 is a block diagram illustrating a configuration of a gesture registration device according to a fourth embodiment. 14 is a flowchart illustrating an operation of a candidate gesture generation unit according to the fourth embodiment.

<A. Embodiment 1>
<A-1. Configuration>
FIG. 1 is a block diagram showing an example of a configuration of a gesture registration apparatus according to Embodiment 1 of the present invention. As shown in FIG. 1, the gesture registration apparatus according to the first embodiment includes a motion input unit 1, a motion analysis unit 2 having a gesture analysis unit 21 and a feature analysis unit 22, a human body motion model database 3, and candidates. A gesture generation unit 4, a gesture presentation unit 5, an input / output unit 6, a gesture definition registration unit 7, and a gesture definition database 8 are provided.

  The motion input unit 1 accepts one moving image data obtained by photographing a human motion to be defined as a gesture using various image sensors, and inputs it to the gesture analysis unit 21 in the motion analysis unit 2. Examples of moving image data include time-series data of RGB images and distance images.

  The human body motion model database 3 stores a body structure table 31 relating to the structure of a part constituting the human body as shown in FIG. 2, and a body movable range table 32 relating to the joint movable range as shown in FIG. When a part of the body moves, related parts move simultaneously. For example, when trying to move the hand, the wrist, elbow, and shoulder also move together. Because of such interlocking of the body, the body can be roughly divided into six parts (head, trunk, right arm, left arm, right leg, left foot). The body structure table 31 includes a part 311, a body feature point 312 corresponding to the part (a more detailed part described later), a connection destination body feature point 313 indicating a connection destination of each body feature point, A parent-child relationship 314 of body feature points is registered. Specific contents of the body movable range table 32 will be described later.

  The motion analysis unit 2 includes a gesture analysis unit 21 and a feature analysis unit 22, and performs various processes while referring to the human body motion model database 3.

<A-2. Gesture analysis>
The gesture analysis unit 21 discriminates human characteristic parts (body feature points) by using an existing image processing technique for all frames of the input moving image data, and acquires the three-dimensional coordinates. For example, any method may be used, such as a method of discriminating body feature points from input RGB images or distance images using human body structure learning data, etc., and obtaining respective three-dimensional coordinates. The body feature points determined here are the same as the body feature points 312 in FIG. In the first embodiment, as an example, 16 body feature points FP1 to FP16 from the head to the foot as shown in FIG. 4 are discriminated and their three-dimensional coordinates are acquired. FIG. 4 shows an example in which body feature points are analyzed for a certain frame image of moving image data. The coordinate system of the three-dimensional coordinates is a coordinate system centered on the image sensor C1 shown in FIG. The type and number of body feature points to be distinguished are not limited to this example. In the future, the time series data of the body feature points and three-dimensional coordinates obtained by this process will be referred to as motion data MD. That is, the gesture analysis unit 21 operates as a motion data acquisition unit that acquires motion data from moving image data. In addition, although the motion input part 1 received the moving image data above, you may receive motion data. In this case, the gesture analysis unit 21 acquires motion data from the motion input unit 1.

  The gesture analysis unit 21 further analyzes and extracts poses constituting the gesture from the movement of the three-dimensional coordinates of the distinguished body feature points. A gesture is represented by taking several poses that characterize it in order. Here, some gestures consist of movements of one part, such as those that move the left arm, and some gestures involve multiple parts, such as those that move both the right arm and the left arm. In particular, when a plurality of parts move, the gesture is represented by a combination of movements (sub-gestures) of the parts. Therefore, when extracting the pose, the motion data MD is analyzed for each part. Specifically, the body structure table 31 is referred to, the movement of the body feature point for each part described in the body structure table 31 is followed, and the part where the remarkable movement is seen is analyzed.

  A gesture is composed of a start pose, an end pose, and an intermediate pose. The start pose is defined as a pose at the start of a gesture or sub-gesture, and can be detected as a pose when a part in the motion data MD starts to move. The end pose is defined as a pose at the end of the gesture or sub-gesture, and can be detected as a pose when the movement of a part in the motion data MD stops. An intermediate pose is defined as a characteristic time pose between the start and end of a gesture. The characteristic time point is a time point when the moving direction of the skeleton changes when, for example, the gesture or the sub-gesture includes a movement including a plurality of moving directions.

  The gesture analysis unit 21 creates pose data PD from the motion data MD. FIG. 6 illustrates pause data PD. The pose data PD includes a three-dimensional coordinate PDT1 of each body feature point, a frame position PDT2 in the motion data MD, and a part (gesture part) PDT3 involved in the gesture. For example, as the pose data PD1 of the start pose P1, the three-dimensional coordinates of each body feature point, the frame position, and the currently analyzed gesture part at the beginning of the movement of the gesture part are extracted. Further, as the pose data PD2 of the end pose P2, the three-dimensional coordinates of each body feature point at the end of the movement of the gesture part, the frame position, and the currently analyzed gesture part are extracted. When the pose at the time when the moving direction of the skeleton changes is an intermediate pose, the frame position of a frame with a small amount of change in three-dimensional coordinates and the three-dimensional coordinates of each body feature point in the frame are used as the pose data of the intermediate pose. Can be extracted. This is because the movement of the body part tends to decrease when the moving direction of the skeleton changes. In order to detect a frame with a small change amount of three-dimensional coordinates, for example, the change amount of each three-dimensional coordinate before and after each frame is calculated, and a representative frame (for example, an intermediate frame) among several frames with a small change amount What is necessary is just to detect.

  It should be noted that a frame at a point where the moving direction changes abruptly may be used as an intermediate pause frame. Further, the method of acquiring the pose data PD by the gesture analysis unit 21 is not limited to the above method.

  FIG. 7 shows an example of a gesture. In this gesture, the left arm in a lowered state is lifted straight through the front direction with the arm extended. FIG. 7A shows the body feature point and frame position in the start pose P1 of this gesture, and FIG. 7B shows the body feature point and frame position in the end pose P2. The gesture analysis unit 21 analyzes the motion data MD indicating the gesture and generates pose data PD1 and PD2. The pose data PD1 is the three-dimensional coordinates, frame position, and gesture part of each body feature point in the start pose P1, and the pose data PD2 is the three-dimensional coordinates, frame position, and gesture of each body feature point in the end pose P2. It is a part. As a gesture part, a part called “left arm” is acquired from the body movable range table 32 as a part that moves remarkably.

  FIG. 8 shows another gesture. This gesture is to extend the left arm in a lowered state to the front of the body and then move to the left. 8A shows body feature points and frame positions in the start pose P3, FIG. 8B shows body feature points and frame positions in the intermediate pose P4, and FIG. 8C shows body feature points in the end pose P5. Each frame position is shown. The gesture analysis unit 21 analyzes the motion data MD indicating the above gesture and generates pose data PD3 to PD5 corresponding to the poses P3 to P5.

  7 and 8 exemplify a gesture in which one part moves. However, for example, in the case of a gesture for moving the right arm and the left arm at the same time, the motion data MD is analyzed by focusing on the body feature points belonging to the right arm and the left arm, and the movement of each part starts, ends, and the posture of the posture in the middle Data PD is extracted.

<A-3. Feature analysis>
The feature analysis unit 22 acquires the motion data MD and the pose data PD from the gesture analysis unit 21, and calculates the feature data FD that characterizes the gesture based on these. An example of the feature data FD is shown in FIG. The feature data FD shown in FIG. 9 includes six items: a gesture part FD1, a main movement direction FD2, an important body feature point FD3, an arrangement feature FD4 of the important body feature point, a joint angle FD5 at the time of transition between poses, and a movement speed FD6. Have. The gesture part FD1 indicates a part of the human body involved in the gesture. A main movement direction FD2 indicates a main direction in which the gesture part moves in the gesture. The important body feature point FD3 indicates a body feature point that characterizes the gesture, and will be described in detail later. When there is pose data for a plurality of parts, the timing at which a pose for each part is generated can be one item of feature data.

  By using such feature data for candidate gesture generation processing, which will be described later, it is possible to create candidate gestures having different parameters within a range that does not greatly deviate from the features of the gesture input by the user. Note that the item of the feature data FD is not limited to this.

  Hereinafter, a method of calculating the feature data FD will be described using an example in which the pose data PD of the pose illustrated in FIG. 7 is acquired.

  The gesture part FD1 is acquired from the gesture part PDT3 of the pose data PD. The gesture part FD1 in the example of FIG. 7 is the left arm part. Hereinafter, processing is performed for every two consecutive pose data having the same gesture part FD1.

  First, a space (movement space) in which the movement of the gesture part FD1 occurs between two poses is obtained. With reference to the motion data MD between the poses, the maximum value and the minimum value of the coordinates of the body feature points constituting the gesture part are obtained for each of the x, y, and z axes. A rectangular parallelepiped region that satisfies the range is used as a movement space for body feature points, and a region that satisfies the movement section for all body feature points is used as a movement space for gesture parts.

  Next, the main movement direction FD2 of the gesture part is calculated. A body feature point having the largest movement amount is obtained from the gesture parts, and a main component (first component) of the movement direction of the body feature point is calculated. Specifically, for example, a body feature point having a large movement amount is assumed to be the largest among the movement amounts of each body feature point calculated using the motion data MD between the poses. For example, in the example of FIG. 7, the left hand is the body feature point with the largest amount of movement. However, when there are two or more body feature points with almost the same movement amount, such as in the case of an operation in which two or more parts move in parallel without bending of the joint, all of them are set as calculation targets. The movement direction is calculated based on the movement of the body feature point having the largest movement amount. Specifically, the range of each axis constituting the movement space of the body feature point and the difference between the three-dimensional coordinates of the body feature point in the first pose among the two consecutive poses are calculated. Based on the coordinates of the first pose of the two poses, the direction of the largest movement among the six positive, negative, and negative x-axis directions is calculated as the principal component of the movement direction. For example, in FIG. 7, the positive y-axis direction is the main component (first component) of the movement direction. Similarly, the second and third main directions are obtained in the same manner, the second (second component) is the negative z-axis direction, and the third (third component) is the negative x-axis direction.

  Next, a body feature point (important body feature point FD3) that is considered to be moved with particular consciousness among the gesture parts is estimated. When humans make gestures, the part that they try to move consciously moves often. For example, in a gesture involving bending of a joint (such as bending an extended arm), the end point (hand) of the part moves greatly. Therefore, when a body feature point having the largest movement amount is uniquely determined, it is set as an important body feature point FD3. In the example of FIG. 7, “left hand” is the important body feature point FD3. However, for example, in the case of a gesture that translates two body feature points in the same direction as a skeleton composed of the two points, the movement amounts of the plurality of body feature points are substantially the same. As described above, when there are two or more body feature points with a large movement amount, the body feature point closer to the direction of the main component of the movement direction is set as the important body feature point FD3.

  Next, an arrangement feature FD4 of important physical feature points is calculated for the first and last poses of two consecutive poses. Specifically, the relative positions of the important body feature point FD3 and the body feature point that is the most parent in the gesture part FD1 (the body feature point having no parent in the parent-child relationship 314 of the body structure table 31) The feature point arrangement feature is FD4. In the example of the poses P1 and P2 shown in FIG. 7, the important body feature point FD3 is the left hand, and the closest body feature point is the shoulder. Therefore, the difference between the three-dimensional coordinates of the left hand and the shoulder is taken, and the first component, the second component, and the third component of the arrangement feature FD4 of the important physical feature point are determined in the order of the largest difference among the three axes. In the example of the poses P1 and P2, the first component is the y axis, the second component is the z axis, and the third component is the x axis. In addition, the relative position directions of the first to third components between the important body feature point FD3 and the other body feature points in the gesture part FD1 are also calculated as the arrangement feature FD4 of the important body feature point. In the case of the start pose P1, the first component of the arrangement feature FD4 of the important physical feature point is the y-axis, the second component is the z-axis, and the third component is the x-axis. Calculate the difference in coordinates about the axis and the x-axis. As a result, data is obtained that the left hand is in the negative y-axis direction, negative z-axis direction, and negative x-axis direction with respect to the elbow and shoulder.

  Next, the joint angle FD5 is calculated at each pose of two consecutive poses and at the time of transition between each pose. A joint is a point where skeletons (lines connecting two body feature points arranged side by side) are connected to each other, and a joint angle is defined as an angle between skeletons. The skeleton direction is calculated from the three-dimensional coordinates of the two body feature points as direction vectors, and the joint angle is calculated from the direction vectors of the two skeleton directions. The joint angle at each pose is obtained by the above method. As for the joint angle at the time of transition between poses, the joint angle of the gesture part is calculated for all the frames in the motion data MD between poses, and the difference is calculated every two consecutive frames. Find the joint angle.

  The moving speed FD6 at the pose transition is, for example, the number of frames of the motion data MD between two consecutive poses.

  Although the calculation method of the feature data FD has been described above, the calculation method of various parameters constituting the feature data FD is not limited to this.

<A-4. From generation of candidate gestures to registration of gesture definitions>
Next, the candidate gesture generation unit 4, the gesture presentation unit 5, the input / output unit 6, the gesture definition registration unit 7, and the gesture definition database 8 will be described.

  The candidate gesture generation unit 4 uses the feature data FD calculated by the feature analysis unit 22 and retains the basic features of the gesture, but the range of the constraint condition of some parameters constituting the input motion data MD Generate definition data of candidate gestures that have been changed. At this time, with reference to the body movable range table 32 of the human body motion model database 3, a candidate gesture that can be executed by the human body is generated. The generated candidate gesture definition data is transmitted to the gesture presentation unit 5. In addition, a new candidate gesture is created by reflecting the user's reaction to the candidate gesture.

  Here, the body movable range table 32 shown in FIG. 3 will be described. In the body movable range table 32, the movable range of each joint is recorded. Specifically, a part 321, joints 322 included in each part, body feature points 323 constituting the joints, joint movement 324, and movable range 325 for each movement are registered. The movable range 325 indicates a limit angle at which the joint can be moved in the + direction and the − direction from the state of the human natural posture.

  The gesture presentation unit 5 converts the content of the candidate gesture definition data generated by the candidate gesture generation unit 4 into drawing data in a form that can be easily understood by a user such as a developer, and transmits the drawing data to the input / output unit 6. Furthermore, a display that prompts the user to react to the candidate gesture is also created and transmitted to the input / output unit 6. In addition, it accepts user input from the input / output unit 6 and transmits the contents to the candidate gesture generation unit 4.

  The input / output unit 6 displays the content of the candidate gesture to a user such as a developer, and accepts an input by the user (such as a selection operation for the candidate gesture). An example of display to the user by the input / output unit 6 is shown in FIG. In FIG. 10, a candidate gesture presentation area 62 and a user operation area 63 are provided on the output screen 61. In the candidate gesture presentation area 62, candidates for poses P1 and P2 and candidates for transition conditions between poses are displayed. FIG. 10 is an example when a candidate regarding the position of the left hand of the start pose P1 is displayed, and a left arm candidate state 621, its coordinates 622, and the left hand position 623 before the change are displayed. Regarding the display contents of the candidates, the candidates may be displayed one by one as shown in FIG. 10, or a plurality of candidates may be displayed at a time. In the user operation area 63, for example, it is assumed that a message 631 that prompts the user to input a user's reaction to the system and an input button 632 are displayed. As an example of the operation for the candidate gesture, in addition to selecting whether the candidate gesture presented as shown in FIG. 10 is acceptable, it is possible to select a range of allowable gesture parameters, and to select which one is better from a plurality of candidates. It may be a thing, but it is not this limitation.

  Thus, by repeating the flow of presenting a candidate gesture to the user at the input / output unit 6 and creating a candidate gesture with the constraint condition changed based on the user's reaction to the candidate gesture, one constraint condition for the gesture is obtained. Determine each one and generate a gesture definition.

  When the gesture definition is finally determined, the gesture presentation unit 5 transmits the gesture definition data to the gesture definition registration unit 7.

  The gesture definition registration unit 7 registers the received gesture definition data in the gesture definition database 8. The contents registered in the gesture definition database 8 can be referred to at the time of gesture recognition by a device having a gesture recognition application.

<A-5. Generating candidate gestures>
Hereinafter, generation of candidate gesture definition data by the candidate gesture generation unit 4 will be described in detail.

  Even for gestures that move in the same manner, the gesture recognition conditions change depending on the image of the developer. For example, the gesture example in Fig. 7 shows a gesture that raises the left hand straight up with little bending of the elbow, but if you don't do this, you want to prevent it from being recognized as a gesture that raises the left hand. May think. On the other hand, a gesture in which the hand is raised straight but the elbow is bent, or a gesture in which the elbow is not bent but the hand is raised diagonally above the shoulder may be recognized as a gesture of raising the hand. Sometimes the developer thinks. If the developer simply thinks that the hand should be raised above the shoulder, there are no restrictions on the elbow angle or the fine position of the hand. In the middle, the gesture may be considered on the condition that the hand should be present at a position within a certain range, and the elbow may be bent within the certain range. As described above, the constraint conditions of the parameters constituting the gesture include three levels of strength: “specific value” having the strongest constraint, “fixed range” in the middle, and “free” being the weakest. The candidate gesture generating unit 4 according to the first embodiment refers to the body movable range table 32 of the human body motion model database 3, and within the range in which the human body can move, the parameters of the poses constituting the gesture and the parameters during the transition between poses To generate a candidate gesture.

  FIG. 11 is a flowchart of candidate gesture generation processing. Hereinafter, the candidate gesture generation process will be described with reference to FIG. First, the candidate gesture generation unit 4 acquires gesture feature data FD and pose data PD as gesture data from the feature analysis unit 22 (step S1). In the following description, it is assumed that the pose data PD shown in FIG. 6 and the feature data FD shown in FIG. 9 are acquired.

  Thereafter, the candidate gesture generation unit 4 calculates a basic constraint condition BR indicating the essence of the gesture (step S2). This means that, as a result of changing the parameter, the constraint condition of the parameter that should be kept at a minimum is calculated so as not to deviate significantly from the original feature of the gesture. The basic constraint condition BR consists of two types: a constraint condition BR1 related to the position of the important body feature point in each pose, and a constraint condition BR2 related to the movement direction of the important body feature point at the time of transition between poses. In calculating this, for each of two consecutive poses, the following processing is performed using the two pose data PD and feature data FD.

  The constraint condition BR1 related to the position of the important body feature point FD3 is a first component of the arrangement feature FD4 of the important body feature point in each pose in the feature data FD. Here, the first component of the start pose P1 is the y-axis negative direction. The second component is the negative z-axis direction, and the third component is the negative x-axis direction. The first component of the end pose P2 is the positive y-axis direction. The second component is the negative z-axis direction, and the third component is the negative x-axis direction. That is, the constraint condition BR1 is that the left hand is in the negative y-axis position from the elbow and shoulder in the start pose P1, and the left hand is in the positive y-axis position from the elbow and shoulder in the end pose P2. In other words, the arrangement feature with respect to the other body feature point of the important body feature point is a fixed parameter having a constraint condition.

  The constraint condition BR2 regarding the movement direction of the important body feature point FD3 is the main movement direction FD2 (first component, second component, third component) during the transition in the feature data FD. In this example, the first component is the positive y-axis direction, the second component is the negative z-axis direction, and the third component is the negative x-axis direction. In other words, the moving direction of the body feature point having the longest moving distance at the time of transition between poses is a fixed parameter having a constraint condition.

  Therefore, in the example of the feature data FD shown in FIG. 9, the basic constraint condition BR is that the left hand moves largely in the y-axis positive direction and in the z-axis negative direction from the position of the start pose P1 in the y-axis negative direction from the elbow and shoulder. And the content is that the end pose P2 in the positive y-axis direction is reached from the elbow or shoulder. Note that the calculation method of the basic constraint condition BR is not limited to this.

  Next, the candidate gesture generation unit 4 changes the pose and the parameters at the time of transition between poses within the range satisfying the basic constraint condition BR obtained in step S2, and generates a candidate gesture (step S3). However, with reference to the movable range 325 of the body movable range table 32 of the human body motion model database 3, the parameters are changed within the range in which the human body can operate. The variation parameters to be changed here include the joint angle of the gesture part as the pose parameter, and the moving joint angle and the moving speed as the parameters during transition between poses.

  Hereinafter, the parameter changing method will be described in detail. First, for each pose parameter, the pose definition is generated by changing the joint angle of the gesture part. Here, the joint angle is determined by the coordinates of three body feature points that substantially constitute the joint. Therefore, the candidate gesture generation unit 4 changes the coordinates of the important body feature point FD3 and other body feature points for each three-dimensional coordinate axis. The change is made in order from the important body feature point to the other body feature points, and in each body feature point, the change is made in order from the axis of the first component of the arrangement feature FD4 of the important body feature point. In the example of the feature data FD in FIG. 9, the parameter in the y-axis direction of the left hand is first changed. Basically, the parameters are changed so as to satisfy the movable range 325 of the body movable range table 32. However, for the important body feature point FD3, the basic constraint condition BR obtained earlier is also satisfied at the same time.

  In order to determine the constraint condition, first, the parameter is greatly changed to the limit of the basic constraint condition BR and the movable range 325, and the change range is gradually narrowed based on the user input. In the example of the feature data FD in FIG. 9, there is a basic constraint condition BR that the left hand that is the important body feature point FD3 is in the negative y-axis direction from the left shoulder and the left elbow. Therefore, first, the y-axis component of the three-dimensional coordinates of the left hand is set to a coordinate close to the limit to the y-axis component of the left shoulder having the largest y-coordinate value in the region. The coordinates of the left elbow, which is the parent of the left hand, are obtained by an inverse kinematics technique for calculating the position of the left hand in reverse.

  In this way, the candidate gesture generating unit 4 transmits the pose data in which the coordinates of the body feature points are changed to the gesture presenting unit 5 as candidate gesture data (step S4). The gesture presentation unit 5 presents the candidate gesture data to the user through the input / output unit 6 in the manner illustrated in FIG.

  When a candidate gesture selection operation by the user is input through the input / output unit 6, the candidate gesture generating unit 4 receives the input result from the gesture presenting unit 5 (step S5). Here, as an example of the candidate gesture selection operation, the user selects whether the candidate gesture is acceptable for the user. The candidate gesture generation unit 4 determines whether or not the candidate gesture is allowed (step S6). If the input result indicates that the candidate gesture is acceptable (Yes in step S6), the process proceeds to step S7. In step S7, it is determined whether or not all parameters have been processed. Here, only the y-axis component, which is the first component of the arrangement feature FD4 of the important physical feature point, has been changed (No in step S7). Therefore, the process returns to step S3, and the second component and the third component z-axis and x-axis are changed in order.

  If the input result of the user is that the candidate gesture cannot be permitted, the process returns to step S3, and the y coordinate of the y-axis component of the left hand is set so as to approach the left hand coordinate of the conventional gesture data. For example, the coordinates are set close to the limit of the y-axis component of the left elbow. And the said pose data are transmitted to the gesture presentation part 5 (step S4). This is repeated until the user inputs that the candidate gesture is acceptable.

  If the changed coordinates are determined for all of the x, y, and z axis components, the position constraint condition (the range of possible positions) for the left hand can be obtained. Next, a process for creating a candidate gesture in which the coordinates of the left elbow connected to the left hand are changed is performed. The left elbow creates candidate pose data whose coordinates have been changed based on the movable range 325 within the range of the previously determined left hand position constraint condition, and similarly generates a position constraint condition. Then, the constraint condition (definition) of the entire pose is determined from the arrangement feature FD4 of the important physical feature points that defines the arrangement characteristics of the positions of the left hand, the left elbow, and the left shoulder. When a new constraint condition other than the movable range 325 is not generated for a certain body feature point (when all changed parameters are accepted by the user), the constraint strength of the constraint condition of the body feature point becomes “free” It is not considered as a pose definition in gesture definition. When a position constraint condition (position range) is added to a certain body feature point, the constraint strength of the constraint condition is “a certain range”, and a range that the coordinates can take is set as a pose definition in the gesture definition. On the other hand, when all the coordinates of the gesture part are the same as before the original parameter change (when the candidate pose was presented to the user but not accepted), the constraint strength of the constraint condition is the strongest “specific value”, The coordinates are set as the pose definition in the gesture definition. When setting constraint conditions (coordinates and coordinate ranges) for each body feature point as a definition, a relative position is calculated and set based on the body feature point at another nearby position (for example, in the case of the left hand of the start pose P1) Is based on the port side). However, the present invention is not limited to this, and an absolute position in a three-dimensional space may be set.

  Next, returning to step S3, similarly to the pose parameters, the joint angle and the moving speed at the time of movement, which are parameters at the time of transition between poses, are changed. The joint angle at the time of movement is changed in consideration of the main movement direction of the basic constraint condition BR. There are three possible joint angle constraints: “free” that does not consider the joint angle, “constant range” that provides a range for the joint angle, and “specific value” that maintains the joint angle in the previous and next poses during movement. It is done.

  Therefore, the candidate gesture generation unit 4 changes each joint angle one by one within the range while taking into consideration the constraint condition of the pose before and after the movement obtained above. First, it is calculated whether or not the same joint angle can be obtained in the front and rear poses. If the same joint angle cannot be obtained, the constraint condition is “constant range” or “free”. The candidate gesture generation unit 4 calculates the maximum angle and the minimum angle that each joint of the gesture part can take in consideration of the movable range 325, for example, and transmits it to the gesture presentation unit 5 as candidate gesture data (step S4).

  Based on the joint angle data obtained from the candidate gesture data, the gesture presentation unit 5 calculates the coordinates of the body feature points of the gesture part that moves between poses. The result is converted into a form that is easy for the user to understand (for example, moving image data), and is transmitted to the input / output unit 6 as a candidate for the motion during the transition. As in the case of the pose, the user selects and inputs whether or not the candidate is allowed, and the input result is received by the candidate gesture generation unit 4 from the input / output unit 6 via the gesture presentation unit 5 (step S5).

  If the candidate gesture data is allowed in the first step S6 (Yes in step S6), the joint angle is set to “free” in any range. That is, the joint angle is not defined as a parameter during transition. On the other hand, if not permitted, the process returns to step S3 to change the maximum angle or the minimum angle of the joint until the candidate gesture is permitted by the user. The maximum and minimum joint angles allowed by the user are the “constant range” and become a constraint condition during pose transition.

  When only a specific value of the joint angle is allowed by the user, the “specific value” becomes a constraint condition at the time of pose transition. In such a case, the joint angle is the same between the poses before and after the transition, and the movement is performed while the joint angle remains constant.

  Further, regarding the movement speed of the pose transition, a candidate gesture in which the speed magnification is changed is created based on the movement speed FD6 of the feature data FD (step S3). In addition, the maximum value and the minimum value of the speed magnification are registered in advance in the candidate gesture generation unit 4, and two patterns of candidate gestures in which the speed is changed at the maximum magnification and the minimum magnification are generated. The candidate gesture is transmitted from the candidate gesture generating unit 4 to the gesture presenting unit 5 (step S4) and converted into drawing data. The gesture presenting unit 5 creates a moving image in which the speed is changed to the above-mentioned magnification by using the motion data MD from which the candidate gesture is created, and displays it on the input / output unit 6.

  The user selects and inputs whether or not to allow the candidate gesture, and the input result is received by the candidate gesture generation unit 4 from the input / output unit 6 via the gesture presentation unit 5 (step S5).

  Here, the candidate gesture is obtained by changing the speed magnification between the maximum value and the minimum value. In the first step S6 regarding the movement speed of the pose transition, if the candidate gesture is permitted by the user (Yes in step S6), the movement speed is set to “free” in any range. That is, the moving angle is not defined as a parameter during transition. If the candidate gesture is not permitted (No in step S6), the process returns to step S3, and the generation of the candidate gesture whose magnification has been changed and the presentation to the user are repeated until the user allows the candidate gesture. Then, if the moving speed is allowed within a certain range, the range is defined as a “constant range” and the moving speed is defined as a constraint condition for the moving speed. On the other hand, when the magnification has converged to the moving speed of the original feature data FD, the moving speed is defined as a “specific value” as a constraint condition.

  The above-described various parameter changing methods are merely examples, and other methods may be used.

  When the above-described processing is performed for all parameters (Yes in step S7), constraint conditions for various parameters at the time of each pose and transition between poses are set, and a gesture definition is generated (step S8).

  FIG. 12 shows a gesture definition generated in response to a user input for the gesture shown in FIG. 7 (from the state in which the left arm is lowered to the gesture in which the arm is extended and then forwards in the forward direction). Data are shown. The gesture definition data includes a gesture number 81, a pose 82, a gesture part 83, and a constraint condition 84. The input content of the user may be that the position of the left hand in each pose may have a certain recognition width in the y-axis direction, the position of the left elbow is free, and the joint angle and moving speed during the pose transition are y As long as it moves in the positive axis direction and the negative z-axis direction, it is free.

<A-6. Modification>
Note that the gesture definition method described above is an example in which the user selects whether or not a candidate gesture is allowed, but the gesture may be defined by another method. For example, the user may input the parameter range directly for the candidate gesture. In addition, for each parameter, the user can input that no change is made or the parameter is not considered, and the gesture definition may be generated based on the input.

  In the above example, the parameters to be changed one by one in order are presented, but the user may freely select parameters and set parameter constraint conditions. In such a case, an input means such as a button for determining the gesture definition data to be transmitted to the gesture definition registration unit 7 after the user sets the constraint condition of the desired parameter in the user operation area 63 (see FIG. 10). It may be provided. In addition, when “specific value” is set as a constraint condition, a certain range may be provided so that it can be recognized before and after the value in consideration of ease of recognition.

  If there is pose data PD for a plurality of parts, the feature analysis unit 22 may extract the feature of the movement timing of each part from the frame position of each pose and add it to the constraint condition for the gesture. That is, the order of whether each part starts to move simultaneously or whether a part moves before or after another part is set as a constraint condition. And the candidate gesture production | generation part 4 produces | generates the gesture definition which followed the order of the pose of each site | part, or was changed.

<A-7. Effect>
The gesture registration device 101 according to the first embodiment includes a gesture analysis unit 21, a feature analysis unit 22, a human body motion model database 3, a candidate gesture generation unit 4, an input / output unit 6, and a gesture definition registration unit 7. The gesture analysis unit 21 operates as a motion data acquisition unit that acquires motion data of a gesture. In addition, the gesture analysis unit 21 generates pose data indicating the three-dimensional coordinates of each body feature point in each pose constituting the gesture based on the motion data. Based on the pose data, the feature analysis unit 22 calculates feature data indicating a constraint condition of a parameter representing the feature of the gesture at each pose and at the transition between poses. The human body motion model database 3 defines the movable range of the skeleton constituting the human body. The candidate gesture generation unit 4 generates candidate gestures by changing other parameters while satisfying the constraint conditions of some parameters. When the user's operation for rejecting the candidate gesture is accepted in the input / output unit 6, a new candidate gesture is generated, and the constraint conditions for the other parameters are determined based on the user's operation on the candidate gesture. The input / output unit 6 presents a candidate gesture to the user and accepts a user operation on the candidate gesture. The gesture definition registration unit 7 registers, in the gesture definition database 8, gesture definition data indicating constraint conditions for each parameter in the gesture. According to the above configuration, when a user (gesture registrant) inputs one piece of motion data to the gesture registration device 101, other parameters are changed while leaving the basic features of the motion data analyzed by the feature analysis unit 22. The candidate gesture is presented to the user. And since the constraint conditions of each parameter are determined based on the user's reaction to the candidate gesture, the user can study the constraint conditions of each parameter without omission. Therefore, it is possible to easily register an appropriate gesture definition according to the intention.

  Moreover, since the candidate gesture production | generation part 4 produces | generates the candidate gesture within the movable range of skeleton based on the human body movement model database 3, it can register the gesture definition which a human body can actually move.

  In addition, the candidate gesture generation unit 4 divides the parameters defining the gesture into fixed parameters and variable parameters, and generates candidate gestures by changing the variable parameters while satisfying the fixed parameter constraint condition (basic constraint condition BR). The fixed parameters at the time of each pose include the arrangement feature of the body feature point with the longest movement distance in the gesture with respect to the other body feature points, and the movement direction of the body feature point with the longest movement distance in the gesture. Therefore, it is possible to generate a candidate gesture in which other parameters are changed while leaving the basic characteristics of the motion data.

<B. Second Embodiment>
In the first embodiment, when the user inputs motion data through one moving image data, the gesture registration device 101 presents the user with a corresponding gesture, and creates a gesture definition based on the user's reaction. In the second embodiment, another gesture definition creation method of the gesture registration apparatus 101 will be described. That is, when the gesture registration device 101 receives a plurality of motion data through a plurality of moving image data, the gesture registration device 101 calculates a constraint condition for the gesture satisfied by the plurality of motion data, and presents a candidate gesture to the user based on the constraint condition. By exemplifying a plurality of possible motion data in the gesture registration device regarding the gesture definition to be registered, the user can present the intended candidate gesture and easily create the gesture definition.

<B-1. Operation>
The motion input unit 1 receives input of a plurality of moving image data or a plurality of motion data, and transmits it to the gesture analysis unit 21. In the present embodiment, description will be made assuming that two types of motion data MD1 and MD2 are input.

  FIG. 13 shows a gesture of motion data MD1, and FIG. 14 shows a gesture of motion data MD2. Like the gesture shown in FIG. 7, the gesture of the motion data MD <b> 1 relates to a gesture of moving the left arm straight down and moving it upward through the forward direction while keeping the arm extended. The motion data MD2 relates to a gesture for moving the left arm slanted downward near the starboard area to the upper left via the front direction.

  The gesture analysis unit 21 and the feature analysis unit 22 of the motion analysis unit 2 generate the feature data FD11 and FD12 by performing the same processing as in the first embodiment on the plurality of motion data, respectively. The feature data FD11 is the same as the feature data FD of the first embodiment. In the feature data FD12, the gesture part FD1 is the left arm, the first component of the main movement direction FD2 is the positive y-axis direction, the second component is the negative x-axis direction, the third component is the negative z-axis direction, and the important body feature point FD3. Becomes the left hand. Further, as the arrangement feature FD4 of the important physical feature point, the first to third components of the pose P21 are the y axis, the x axis, and the z axis, respectively, and the first to third components of the pose P22 are the y axis, the x axis, and z The joint angle FD5 at the time of the axis and transition is 170 for both the maximum and minimum values, and the moving speed FD6 is 70.

  The candidate gesture generation unit 4 receives a plurality of gesture data including feature data FD11 and FD12, and generates a constraint condition of parameters satisfying each feature therefrom.

  The candidate gesture generation processing of the candidate gesture generation unit 4 will be described below along the flowchart of FIG. First, a plurality of gesture data is received from the feature analysis unit 22 (step S1), and a unified basic constraint condition BRC that satisfies the features of the plurality of motion data is generated (step S2).

  The unified basic constraint condition BRC is the sum of the basic constraint conditions BR for each motion data. The basic constraint condition BR consists of two types: a constraint condition BR1 related to the position of the important body feature point in each pose, and a constraint condition BR2 related to the movement direction of the important body feature point at the time of transition between poses. In this example, the basic constraint condition BR of the motion data MD2 is that the left hand, which is an important body part, largely moves from the position of the starting pose P21 in the negative y-axis direction to the positive y-axis direction and in the negative x-axis direction from the elbow and shoulder. It shows the content of transition with the movement of, reaching the end pose P22 in the positive y-axis direction from the elbow or shoulder. By combining this with the basic constraint condition BR of the motion data MD1, a unified basic constraint condition BRC is obtained. The content is that the left hand, which is an important body part, is mainly in the y-axis negative direction from the elbow or shoulder and starts from the position in the range from the start pose P11 to the start pose P21, and greatly increases in the y-axis positive direction to the z-axis negative direction. And transition with a movement in the negative x-axis direction, reaching the position in the positive y-axis direction from the end pose P12 to the end pose P22 from the elbow or shoulder.

  Next, based on the unified basic constraint condition BRC obtained above, the candidate gesture generation unit 4 first generates a constraint condition of parameters at the time of transition between poses that satisfies the features of the multiple motion data as a candidate gesture (step S3). The parameter constraint conditions include the joint angle of the gesture part in each pose, the joint angle of the gesture part at the time of transition between poses, and the moving speed.

  The constraint condition regarding the joint angle of the gesture part in each pose is calculated based on the coordinates of each body feature point as in the first embodiment. For example, the constraint condition regarding the left hand is a range (constant range) of each x, y, z coordinate of each left hand in the poses P11, P21.

  The constraint condition regarding the joint angle of the gesture part at the time of transition between poses refers to the joint angle FD5 at the time of transition of each feature data, and uses the sum of the range of the maximum value and the minimum value as the constraint condition. In this example, since all are 170 degrees, the joint angle becomes a “specific value” constraint condition that the joint angle moves while maintaining 170 degrees.

  The constraint on the movement speed at the time of transition between poses is the sum of the movement speed ranges in the respective feature data FD11 and FD12. Here, since the moving speed of the feature data FD11 is 60 and the moving speed of the feature data FD12 is 70, the “constant range” of the moving speeds 60 to 70 is a constraint condition.

  The candidate gesture generation unit 4 transmits the candidate gesture definition data describing the constraint conditions thus obtained to the gesture presentation unit 5 (step S4).

  In the gesture presentation unit 5, the constraint condition of each pose and the constraint condition during transition between poses are converted into a display form that is easy for the user to understand and displayed in the candidate gesture presentation area 62 of the input / output unit 6. Here, in the user operation area 63, a display for prompting a selection operation as to whether or not the constraint condition of the current candidate gesture is acceptable for the user, or a display for selecting the parameter to be changed and inputting the change contents. Is made.

  For example, the user performs a selection operation on the input / output unit 6 as to whether or not the constraint conditions of the current candidate gesture are acceptable for the user. The candidate gesture generation unit 4 receives the input result via the gesture presentation unit 5 (step S5).

  If the user does not allow the current candidate gesture (No in step S6), the process returns to step S3, and the generation of the candidate gesture with the changed parameter and the presentation to the user are repeated until the user allows the candidate gesture. It should be noted that the generation of a new candidate gesture is performed with reference to the set constraint conditions and the movable range 325 of the body, as in the first embodiment. Alternatively, input of new motion data may be received by the motion input unit 1 and a candidate gesture having a constraint condition that also includes feature data of the new motion data may be generated.

  If the user allows the candidate gesture (step S6), the parameter constraint condition changed in the candidate gesture is determined. If the constraint condition is confirmed for all parameters of the candidate gesture (Yes in step S7), a gesture definition is generated (step S8). That is, the gesture definition registration unit 7 registers the gesture definition data in the gesture definition database 8.

  Whether to perform the processing of the first embodiment or the second embodiment may be determined according to the number of motion data MD input to the motion input unit 1 first. For example, the operation of the first embodiment may be performed if the number of input motion data MD is 1, and the operation of the second embodiment may be performed if the number is 2 or more. Alternatively, the candidate gesture generation unit 4 may have a mode for which processing is performed, and the number of motion data received may be controlled according to the setting.

<B-2. Effect>
In the gesture registration device 101 according to the second embodiment, the gesture analysis unit 21 operates as a motion data acquisition unit that acquires a plurality of gesture motion data. The gesture analysis unit 21 generates pose data for each motion data. Further, the feature analysis unit 22 calculates feature data indicating a parameter constraint condition for each pose data. Then, the candidate gesture generating unit 4 generates candidate gesture data by changing the variation parameter while satisfying the sum of the constraint conditions of the fixed parameters in all the pose data. According to the above configuration, when a user (gesture registrant) inputs a plurality of motion data to the gesture registration device 102, some features of the motion data analyzed by the feature analysis unit 22 are included. Candidate gestures with changed parameters are presented to the user. And since the constraint conditions of each parameter are determined based on the user's reaction to the candidate gesture, the user can study the constraint conditions of each parameter without omission. Therefore, it is possible to easily register an appropriate gesture definition according to the intention. In addition, by registering a plurality of motion data in advance, the user can easily register a gesture definition in accordance with the user's intention.

<C. Embodiment 3>
When a developer registers multiple gestures, it is necessary to set each gesture definition after considering that they are correctly distinguished and recognized. This is because if the gestures are defined, the gesture may or may not be recognized at an unintended timing. The former occurs when the definition (constraint condition) of the start pose or end pose of one gesture overlaps the definition (constraint condition) of the end pose or start pose of another gesture. For example, if the constraint condition overlaps between the end pose of a certain gesture and the start pose of another gesture, another gesture starts at the same time as the end of the certain gesture. In addition, when a gesture definition is included in the range of another gesture definition, the gesture may not be recognized.

  In order to solve such a problem, in the third embodiment, it is determined whether or not the gesture input by the user overlaps with the registered gesture definition, that is, the consistency of the gesture.

<C-1. Configuration>
FIG. 15 is a block diagram illustrating the configuration of the gesture registration device 102 according to the third embodiment. The gesture registration device 102 includes a consistency determination unit 9 in addition to the configuration of the gesture registration device 101 according to the first and second embodiments.

<C-2. Operation>
FIG. 16 is a flowchart of the gesture definition registration process in the gesture registration device 102. Hereinafter, the candidate gesture generation process will be described with reference to FIG. However, steps other than step S2A are the same as those in the flowchart of the gesture registration apparatus 101 shown in FIG.

  When the candidate gesture generation unit 4 generates the basic constraint condition BR from the pose data PD and the feature data FD of the motion data MD (step S2), the candidate gesture generation unit 4 receives the pose data PD, the feature data FD, and the basic constraint condition BR from the consistency determination unit 9. Send.

  The consistency determination unit 9 determines the consistency between the currently input motion data and each gesture definition while referring to each gesture definition data registered in the gesture definition database 8 (step S2A).

  Specifically, first, it is determined whether or not the gesture part of the motion data matches the gesture part 83 of the gesture definition data registered in the gesture definition database 8. Here, the gesture portion of the motion data is the gesture portion PDT3 of the pose data PD. If there is matching gesture definition data, the start pose and end pose of the motion data are compared with the constraint conditions of the start pose and end pose in the gesture definition database to determine consistency. This comparison consists of (1) motion data and gesture definition data start poses, (2) motion data and gesture definition data end poses, (3) motion data start poses and gesture definition data end poses, (4 ) It is performed in four patterns, the end pose of motion data and the start pose of gesture definition data.

  The comparison target on the gesture definition data side is the constraint condition 84, and the comparison target on the motion data side is the three-dimensional coordinate PDT1 of the body feature point included in the gesture part PDT3 in the pose data PD. Body feature points that are not described in the gesture definition constraint 84 are “free” constraints that may take any value in the three-dimensional coordinates. First, only the body feature points described as constraint conditions are compared. Specifically, it is determined whether or not the three-dimensional coordinate PDT1 on the motion data side satisfies the constraint condition 84.

  In the comparison of the patterns (3) and (4), when it is determined that the three-dimensional coordinate PDT1 satisfies the constraint condition 84, the gesture is recognized at an unintended timing continuously with another gesture. Therefore, it is determined that there is a problem with consistency.

  In the comparison of the patterns (1) and (2), when it is determined that the three-dimensional coordinate PDT1 satisfies the constraint condition 84, the number of poses excluding the start pose and the end pose is compared. If the number of poses is the same, the three-dimensional coordinate PDT1 and the constraint condition 84 are also compared for those poses. If it is determined that the three-dimensional coordinate PDT1 satisfies the constraint condition 84 in all poses including the start pose and the end pose, it is determined that there is a problem in consistency. This is because the motion data is very similar to the previously registered gesture definition and is not recognized even if it is registered.

  When it is determined that there is no consistency (No in step S2A), the consistency determination unit 9 sends a signal indicating that to the gesture presentation unit 5. Then, the gesture presenting unit 5 generates a display indicating that there is no consistency and displays it on the input / output unit 6 (step S9). The process after step S3 shown in FIG. 11 is interrupted, and no candidate gesture is generated from the motion data. Then, the motion input unit 1 can accept input of another motion data.

  If it is determined that there is no problem with the consistency (Yes in step S2A), the processing after step S3 is performed in the same manner as in the first and second embodiments, and the gesture definition is registered.

  The above process will be described by taking as an example the case where the motion data of the gesture shown in FIGS. For example, it is assumed that the gesture definition shown in FIG. 12 is registered in advance in the gesture definition database 8. Note that the gesture definition shown in FIG. 12 corresponds to the motion data shown in FIG.

  In the case of the gesture shown in FIG. 17, the x-coordinate and z-coordinate of the left hand of the start pose P1 are not different from the gesture of FIG. 7, but the y coordinate is located at the head y coordinate +30 (FIG. 17 (a)). Similarly, in the end pose P2, the x-coordinate and z-coordinate of the left hand are not different from the gesture of FIG. 7, but only the y-coordinate is located at the port part-5 (FIG. 17B). That is, at the end pose P2 of the motion data, the three-dimensional coordinates of the left hand satisfy the start pose restriction condition 84 of the gesture definition data. Further, in the start pose P1 of the motion data, the three-dimensional coordinates of the left hand satisfy the restriction condition 84 of the end pose of the gesture definition data. Therefore, it is determined that the motion data of the gesture shown in FIG. 17 has a problem with consistency.

  In the case of the gesture shown in FIG. 18, the left elbow is bent slightly in the start pose P1, and the left hand is positioned in the positive direction of the y-axis (left port part y-coordinate +10) with respect to the gesture of FIG. ). The x-coordinate and z-coordinate of the left hand are the same as the gesture in FIG. Similarly, the left hand of the end pose P2 is located in the negative y-axis direction (head y coordinate + 20) from FIG. 7 (FIG. 18B). The x-coordinate and z-coordinate of the left hand are the same as the gesture in FIG. That is, the 3D coordinate of the start pose P1 of the motion data satisfies the constraint condition of the start pose of the gesture definition data, and the 3D coordinate of the end pose P2 of the motion data satisfies the constraint condition of the end pose of the gesture definition data. Yes. In the gesture motion data shown in FIG. 18, since there is no pose other than the start pose P1 and the end pose P2, it is determined that there is a problem in consistency.

  If the consistency determining unit 9 determines that there is a problem with the consistency of the input motion data, the consistency determination is not stopped, but the generation of the candidate gesture for the motion data is not stopped. Candidate gestures with changed parameters may be generated.

  In addition to the gesture definition data, motion data, pose data, and feature data used when creating the candidate gesture may be registered in the gesture definition database 8 together. Then, when the input motion data is not consistent with the gesture, the candidate gesture generation unit 4 may change the registered gesture definition data.

<C-3. Effect>
The gesture registration device 102 according to the third embodiment includes a consistency determination unit 9 in addition to the configuration of the gesture registration device 101. The consistency determination unit 9 determines whether or not the gesture definition data registered in the gesture definition database 8 and the motion data gesture acquired by the gesture analysis unit 21 (motion data acquisition unit) overlap. Further, the candidate gesture generation unit 4 generates a candidate gesture only for the motion data that the consistency determination unit 9 determines not to overlap. Thereby, it is possible to avoid the problem that overlapping gesture definitions are registered and the gesture is recognized or not recognized at an unintended timing.

<D. Embodiment 4>
When the user registers a plurality of gestures, if the consistency with the registered gesture definition data cannot be obtained, it is necessary to re-register other gestures. At this time, as another candidate gesture, a method of correcting the parameter of the gesture to be registered may be considered, but a gesture that performs a completely different movement using the same body part or a gesture using another body part is possible. It is possible to register. In the present embodiment, a method for generating a candidate gesture such as the latter in the case where consistency between gestures cannot be obtained will be described.

<D-1. Configuration>
FIG. 19 is a block diagram illustrating the configuration of the gesture registration device 103 according to the fourth embodiment. The gesture registration device 103 includes a gesture rule database 10 in addition to the configuration of the gesture registration device 102 according to the third embodiment.

  In the gesture rule database 10, a plurality of candidate gesture generation rules and general-purpose gesture definition data are registered for each part. The candidate gesture generation rule is a rule for generating an alternative candidate gesture when the gesture to be registered is not consistent with the gesture definition data. The general-purpose gesture definition data is gesture definition data that can be generally used.

  Examples of candidate gesture generation rules include: (1) Candidate to perform the same movement using the other part when the gesture using one of the left and right parts such as arms and legs cannot be registered Generating a gesture, (2) generating a candidate gesture using both parts, and (3) generating a gesture of the same part registered in the gesture rule database 10 as a candidate gesture. In the candidate gesture generation rule, the processing contents are branched according to the characteristics of the body part, the registration status of the gesture in the gesture definition database 8, and the user's reaction when the candidate gesture is presented.

  As general-purpose gesture definition data, numbers for distinguishing gesture definitions, pose data, gesture parts, constraint conditions, and the like are registered. In addition, conditions for determining whether or not to use together with gesture definitions of other parts, gesture definition numbers that can be used together, and the like may be registered.

<D-2. Operation>
FIG. 20 is a flowchart of the gesture definition registration process in the gesture registration device 103. Hereinafter, the gesture definition registration process will be described with reference to FIG. However, steps other than steps S2A, S9, and 10 are the same as those in the flowchart of the gesture registration apparatus 101 shown in FIG.

  The consistency determination unit 9 determines the consistency between the currently input motion data and each gesture definition while referring to each gesture definition data registered in the gesture definition database 8 (step S2A). If there is consistency, the processing from step S3 is performed as in the first embodiment, and the gesture definition is registered (step S8).

  If there is no consistency in step S <b> 2 </ b> A, the consistency determination unit 9 sends a signal indicating the content to the candidate gesture generation unit 4. The candidate gesture generation unit 4 refers to the gesture rule database 10 and generates a candidate gesture based on the rule (step S9).

  The process of step S9 will be described by taking as an example a case where the motion data of a gesture in which the left hand is raised from the bottom to the top shown in FIG. 7 is input and is inconsistent with the already registered gesture definition. It is described in the gesture rule database 10 that “if there is a paired part in the gesture part and a gesture that performs the same movement using the part is not registered, the same movement is performed in the paired part”. And The candidate gesture generating unit 4 refers to the part 321 of the body movable range table 32 to obtain a paired part (right arm part), and whether the gesture of raising the right hand from the bottom to the top is registered in the gesture definition database 8. Check. If not registered, the gesture is generated as a candidate gesture (step S10) and transmitted to the gesture presenting unit 5 (step S11). Candidate gestures are presented to the user through the input / output unit 6. When a candidate gesture selection operation by the user is input through the input / output unit 6, the candidate gesture generating unit 4 receives the input result from the gesture presenting unit 5 (step S12). The candidate gesture generation unit 4 determines whether or not the candidate gesture is allowed (step S13). If the input result indicates that the candidate gesture is acceptable (Yes in step S13), the candidate gesture is registered as a gesture definition (step S8).

  When the candidate gesture is not permitted by the user in step S13, the process returns to step S10, and further refers to the gesture rule database 10 to generate a new candidate gesture until permitted by the user. For example, refer to the gesture definition using the left arm part of the general-purpose gesture definition data according to the rule, generate a candidate gesture that moves the left arm from the left to the right, or move the left arm part and the right arm part together. Generate candidate gestures.

<D-3. Effect>
In addition to the configuration of the gesture registration device 102, the gesture registration device 103 according to the fourth embodiment includes an alternative gesture generation rule that substitutes a gesture registered in the gesture definition database, and a gesture definition using each body part. Is registered, the gesture rule database 10 is provided. Then, when the consistency determination unit 9 determines that the gesture of the motion data acquired by the gesture analysis unit 21 overlaps with the registered gesture, the candidate gesture generation unit 4 is based on the gesture rule database instead of the candidate gesture. Generate alternative candidate gestures. When the acquired motion data overlaps with the registered gesture definition, an alternative candidate gesture is generated and presented to the user, so that the gesture registration device is convenient.

  It should be noted that the present invention can be freely combined with each other within the scope of the invention, and each embodiment can be appropriately modified or omitted.

  DESCRIPTION OF SYMBOLS 1 Motion input part, 2 Motion analysis part, 3 Human body movement model database, 4 Candidate gesture production | generation part, 5 Gesture presentation part, 6 Input / output part, 7 Gesture definition registration part, 8 Gesture definition database, 9 Consistency judgment part, 10 Gesture rule database, 21 gesture analysis unit, 22 feature analysis unit, 101, 102, 103 gesture registration device.

Claims (6)

A motion data acquisition unit for acquiring gesture motion data;
A gesture analysis unit that generates pose data indicating three-dimensional coordinates of each body feature point in each pose constituting the gesture based on the motion data;
Based on the pose data, a feature analysis unit that calculates feature data indicating constraint conditions of parameters representing the features of the gesture at the time of each pose and at the time of transition between the poses;
A human body motion model database that defines the movable range of the skeleton constituting the human body,
A candidate gesture generation unit that changes the other parameters while satisfying the constraints of some of the parameters, and generates a candidate gesture within a movable range of the skeleton based on the human body motion model database;
An input / output unit that presents the candidate gesture to a user and receives the user's operation on the candidate gesture;
The candidate gesture generation unit generates a new candidate gesture upon accepting the user's operation to reject the candidate gesture at the input / output unit, and based on the user's operation on the candidate gesture, Determine the constraints of the parameters;
A gesture definition registration unit that registers, in a gesture definition database, gesture definition data indicating the constraint condition of each of the parameters in the gesture;
Gesture registration device. The motion data acquisition unit acquires a plurality of the motion data of the gesture,
The gesture analysis unit generates the pose data for each motion data,
The feature analysis unit calculates the feature data indicating the constraint condition of the parameter for each pose data,
The candidate gesture generation unit generates the candidate gesture by changing the other parameters while satisfying the sum of the constraints of the partial parameters in all the pose data.
The gesture registration device according to claim 1. The candidate gesture generation unit divides the parameter into a fixed parameter and a variable parameter, generates the candidate gesture by changing the variable parameter while satisfying the constraint condition of the fixed parameter,
The fixed parameter at each pose includes an arrangement feature of the body feature point having the longest movement distance in the gesture with respect to the body feature point.
The gesture registration device according to claim 1. The candidate gesture generation unit divides the parameter into a fixed parameter and a variable parameter, generates the candidate gesture by changing the variable parameter while satisfying the constraint condition of the fixed parameter,
The fixed parameter at the time of transition between the poses includes a moving direction of the body feature point having the longest moving distance in the gesture.
The gesture registration device according to any one of claims 1 to 3. A consistency determination unit that determines whether or not the gesture definition data registered in the gesture definition database overlaps the gesture of the motion data acquired by the motion data acquisition unit;
The candidate gesture generation unit generates the candidate gesture only for the motion data determined that the consistency determination unit does not overlap.
The gesture registration device according to claim 1. A gesture rule database in which a rule for generating an alternative gesture that replaces the gesture registered in the gesture definition database and a definition of the gesture using each body part are registered;
When the consistency determination unit determines that the gesture of the motion data acquired by the motion data acquisition unit overlaps the registered gesture, the candidate gesture generation unit replaces the gesture with the gesture. Generate alternative candidate gestures based on the rule database,
The gesture registration device according to claim 5.

Images