JP2015191471A - Emotion information estimation device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately estimate emotion information of a human and the like who are acting according to action data of the human and the like.SOLUTION: A motion analysis part 1 divides action data into action fragments on a temporal axis and classifies the action fragments to identify which of the plurality of action fragments they correspond to. A feature amount calculation part 2 calculates a feature amount according to the respective action fragments. A fragment emotion estimation part 3 estimates, on the basis of the calculated feature amount, which of the plurality of emotion information the respective action fragments correspond to. An emotion integration part 4 identifies which of the plurality of emotion information the action data corresponds to by integrating the emotion information estimated for each action fragment all over the action data. In integrating the information, weight information with respect to the action kind identified for each action fragment is used.

Description

  The present invention relates to an emotion information estimation apparatus, method, and program for estimating emotion information of a person or the like who is acting from the action data of the person or the like.

  In recent years, for example, Patent Documents 1 to 3 have developed a technique for recognizing and estimating a user's emotion using multimodal such as an audio signal, facial expression, action (or gesture). However, in many cases, all of the multimodal information cannot be obtained. Especially when considering the shooting angle and shooting distance of a surveillance camera in a public space, it is difficult to capture voice and face information, and there are many personal information that may not be available.

  Therefore, a technology has been developed to estimate emotions from human behavior data that contains only body joint movement information. For example, in Patent Document 4, emotion recognition is more correctly realized by extracting feature quantities using a dimension reduction technique after reflecting motion features in more detail from time series data of whole body joints.

  As described above, a general technique for estimating emotion information from human behavior data composed only of joint motion information includes the following four procedures, as described in Non-Patent Document 1, for example.

[Procedure 1] Acquire joint time-series data (human behavior) from the sensor.
[Procedure 2] The action data is divided into a plurality of small actions on the time axis.
[Procedure 3] A feature value is extracted for each action. Reduce dimensions when necessary.
[Procedure 4] Estimate emotion information.

  Specifically, in Non-Patent Document 2, for example, human activity data is divided into a plurality of small actions on the time axis, and the divided plurality of small movements are converted into a plurality of actions with semantic content. It is classified into types (sub-activity), feature quantities are calculated for each small movement, emotion information is estimated for each action type (sub-activity) using the calculated feature quantities, and each action type ( sub-activity) emotion information is voted, and emotion information with the largest number of votes is output as an action estimation result.

JP 2007-41988 JP 2010-66844 A JP 2010-134937 JP JP 2009-037410 A

Karg, M .; Samadani, A .; Gorbet, R .; Kuhnlenz, K .; Hoey, J .; Kulic, D., "Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation," Affective Computing, IEEE Transactions on, in press. Daniel Bernhardt, Peter Robinson, "Detecting affect from non-stylised body motions," Affective Computing and Intelligent Interaction, Lecture Notes in Computer Science Volume 4738, 2007, pp 59-70.

  However, in the above-described prior art, since the voting is performed equally regardless of what the action type is, there is a possibility that the performance of emotion estimation is reduced. Generally, motion data such as gestures is classified into a plurality of motion types (sub-activity), but each motion type has a specific role, and the ease of expressing emotions is also different.

  As an example, among the gestures having three types of motions, that is, a preparation stage, a production stage, and an end stage, the action including the emotion information most is the production stage. Even if the estimation results in the preparation stage and the end stage are different from those in the production stage, it can be said that the production stage has higher reliability as information for emotion estimation. However, in the prior art, since each of the preparation stage, the production stage, and the end stage carries out voting equally, the high reliability in the production stage is not reflected in the result of emotion estimation.

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to provide an emotion information estimation apparatus, method, and program capable of performing emotion estimation with high accuracy.

  In order to achieve the above object, the present invention is an emotion information estimation device, which divides behavior data into motion segments on a time axis, and classifies each motion segment as one of a plurality of motion types. A motion analysis unit that identifies whether it corresponds, a feature amount calculation unit that calculates a feature amount from each motion unit, and each motion unit corresponds to any of a plurality of emotion information based on the calculated feature amount By integrating the estimated emotion information for each motion segment and the entire emotion data, the behavior data corresponds to which of a plurality of emotion information. An emotion integration unit that identifies the movement type, and the emotion integration unit performs the integration by using weight information for the specified action type for each movement element.

  In addition, the present invention is an emotion information estimation method, in which behavior data is divided into motion segments on a time axis, and each motion segment is classified to identify one of a plurality of motion types. A motion analysis stage, a feature quantity calculation stage for calculating a feature quantity from each motion segment, and an element for estimating which of the plurality of emotion information each motion segment corresponds to based on the calculated feature quantity A single emotion estimation stage, and emotion integration for identifying which of the plurality of emotion information the behavior data corresponds to by integrating the estimated emotion information for each motion element over the entire behavior data In the emotion integration step, the integration is performed using weight information for the specified action type for each action element.

  Furthermore, the present invention is an emotion information estimation program that causes a computer to function as the emotion information estimation device.

  According to the present invention, by the emotion integration process, the estimated emotion information is integrated over the entire behavior data by using the weight information for the specified operation type for each operation unit. Thus, it is specified which of the plurality of emotion information the behavior data corresponds to. Therefore, it is possible to estimate emotion information with high accuracy by reflecting weight information for each action type.

It is a functional block diagram of an emotion information estimation device. It is an example of the structure of the bone expressed as a tree structure. It is a functional block diagram of a motion analysis part. It is a figure for demonstrating notionally the data processed by each part of a motion analysis part. It is an example of the graph of the calculated speed. In the example of FIG. 5, it is a graph of the number of operation | movement pieces at the time of changing a threshold value. It is a figure which shows the example of the number of operation | movement types for every kind of action data registered as prior knowledge in a table format. It is a functional block diagram of a segment emotion estimation part. It is a functional block diagram of an emotion integration part. This is an example of the distribution of emotion information determined for each action type when there are four kinds of emotion information and four kinds of action types.

  FIG. 1 is a functional block diagram of an emotion information estimation apparatus according to an embodiment. The emotion information estimation device 10 includes a motion analysis unit 1, a feature amount calculation unit 2, a segment emotion estimation unit 3 and an emotion integration unit 4, receives behavior data as input, and what is the emotion corresponding to the behavior data Is output as an emotion estimation result. The outline of each part is as follows.

  The motion analysis unit 1 divides the behavior data into a plurality of motion elements in time series, and assigns a distinction to which of the plurality of motion types to each of the divided motion elements. Then, it is passed to the feature amount calculation unit 2.

  The feature amount calculation unit 2 calculates a feature amount related to the behavior represented by the data of the motion unit from each of the motion units obtained by the above division, and the result is a unit emotion estimation unit Pass to 3.

  The segment emotion estimation unit 3 estimates emotion information in each of the motion segments for which the feature amount has been calculated using a classifier built in advance, and passes the result to the emotion integration unit 4. The classifier is constructed by learning in advance by a method to be described later so as to output emotion information with the feature quantity as an input.

  The emotion integration unit 4 integrates the emotion information estimated for each motion segment with the entire behavior data as input data using weight information whose value is defined for each motion type by a method described later. Thus, an estimation result of emotion information corresponding to behavior data is obtained as a final output.

  Hereinafter, although the detail of the said each part is demonstrated, the action data which are input data are demonstrated first.

  The action data is data of a pose (and a gravity center position) of a human or the like that changes in time series. The action data cut out at a certain time is based on a human skeleton and is connected sequentially from the root via the joint, using bones and bone connection points (joints) as one root. The structure of the bone can be defined by a tree structure.

  FIG. 2 shows an example of a bone structure expressed as the tree structure. In FIG. 2, the joint 100 is the waist part and is defined as the root. Joint 101 is the elbow part of the left arm, joint 102 is the wrist part of the left arm, joint 103 is the elbow part of the right arm, joint 104 is the wrist part of the right arm, joint 105 is the knee part of the left foot, and joint 106 is the part of the left foot The ankle part, joint 107 is the right leg knee part, and joint 108 is the right leg ankle part.

  With such a structure, the behavior data can be expressed by angle information, position information, speed information, acceleration information, and the like of each joint. Here, the angle information will be described as an example.

  The angle information data represents a series of movements of a person by a sequence of a plurality of postures (poses), and represents basic pose data representing a person's basic pose and each pose of the actual movement of the person. Frame data. The basic pose data includes information such as the position of the root and the position of each joint in the basic pose, and the length of each bone. The basic pose is specified by the basic pose data. The frame data represents the amount of movement from the basic pose for each joint. Here, for example, angle information can be used as the movement amount. Each frame data identifies each pose in which each movement amount is added to the basic pose. Thereby, a series of movements of a person is specified by the continuation of each pose specified by each frame data.

  Note that the human motion can be created by a motion capture process from video captured by the camera, or can be created manually by key frame animation. In the above description, human movement has been described. However, if a joint structure is provided, behavior data can be obtained in the same manner for animals and robots.

  FIG. 3 is a functional block diagram of the motion analysis unit 1. The motion analysis unit 1 includes a physical quantity conversion unit 11, a division unit 12, a normalization unit 13, and a classification unit 14. FIG. 4 is a diagram for conceptually explaining data processed by the respective units of the motion analysis unit 1. The outline of each part is as follows.

  The physical quantity conversion unit 11 converts the action data on the time series to make frame data on the time series (frame data as a joint relative position as will be described later), and passes it to the dividing unit 12. In FIG. 4, data A1 shown in (1) is a conceptual example of frame data. Note that the physical quantity conversion unit 11 may be omitted if the behavior data is in the format of the frame data that is converted by the physical quantity conversion unit 11 in advance.

  The dividing unit 12 divides the frame data into motion elements corresponding to individual short movements sequentially arranged in time series, and passes them to the normalizing unit 13. In the example of FIG. 4, the frame data A1 in (1) is divided into 16 motion elements D1 to D16 as shown in (2).

  The normalizing unit 13 generally normalizes the time segments of the divided motion elements whose durations are different from each other and sets the normalized motion elements to equal durations. Obtained and passed to the classification unit 14. The normalization process is a preprocess for enabling the process in the classification unit 14.

  In the example of FIG. 4, the 16 motion elements D1 to D16 of (2) are normalized to become normalized motion elements N1 to N16 as shown in (3), respectively. In addition, in FIG. 4, the length in the horizontal axis direction (time direction) between (2) and (3) is different from each other due to normalization and expansion / contraction, but FIG. Since this is a typical example, such expansion and contraction is not expressed.

  The classification unit 14 obtains a classification result indicating which of the plurality of motion types each of the normalized motion elements corresponds to, and passes the result to the feature amount calculation unit 2 in FIG.

  In the example of FIG. 4, as a result of classifying the (normalized) motion elements N1 to N16 of (3), the motion elements N1 to N4 are classified into the motion type S1 as shown in (4). The motion elements N5 to N8 are classified into the action type S2, the action elements N9 to N12 are classified into the action type S3, and the action elements N13 to N16 are classified into the action type S4. In the example of (4), each of the motion types S1 to S4 is composed of continuous motion elements.In general, there is a high possibility that the motion types are not continuous. is there. That is, there is a case where division on the time axis occurs between operation elements classified into a certain common operation type. For example, instead of being classified as shown in (4) of FIG. 4, N1 to N3, N8, and N10 may be classified as the operation type S1.

  Hereinafter, details of the respective units 11 to 14 will be described.

  The physical quantity conversion unit 11 converts the behavior data into frame data as the joint relative position by calculating how much each joint moves with respect to the route in the behavior data as input. A specific calculation method of the conversion is as follows.

The physical quantity converter 11 calculates the joint position using the basic pose data and the frame data in the action data described above with reference to FIG. The basic pose data includes information for specifying the basic pose, such as the position of the root and the position of each joint in the basic pose, and the length of each bone. The frame data has information on the amount of movement from the basic pose for each joint. Here, angle information is used as the movement amount. In this case, p ^k (t) which is the position (x, y, z coordinate) of the k-th joint at time t is calculated by the following equations (1) and (2). Note that time t is the time of the frame data. In the following description, time t is simply referred to as “frame index”. That is, the values of t = 0, 1, 2,. Here, T is the number of frames of action data.

However, the 0th joint (i = 0) is the root. R _axis ^{i-1, i} (t) is a coordinate rotation matrix between the i-th joint and its parent joint ("i-1" -th joint), and is included in the basic pose data. A local coordinate system is defined for each joint, and the coordinate rotation matrix represents the correspondence of the local coordinate system between joints in a parent-child relationship. R ⁱ (t) is a rotation matrix of the i-th joint in the local coordinate system of the i-th joint, and is angle information included in the frame data. T ⁱ (t) is a transition matrix between the i-th joint and its parent joint, and is included in the basic pose data. The transition matrix represents the bone length between the i-th joint and its parent joint.

Next, the physical quantity conversion unit 11 calculates the relative position (joint relative position) p ′ ^k (t) of the k-th joint with respect to the root at time t using Equation (3).

Here, p ^root (t) is the position p ⁰ (t) of the root (0th joint) at time t.

  As described above, the physical quantity conversion unit 11 determines the relative position x (t) with respect to the root of each joint as data representing the frame at each time t as shown in the following equation (4), and converts The final frame data is passed to the dividing unit 12. K is the number of joints excluding the root.

  Based on the converted frame data, the dividing unit 12 specifies a time for which a boundary of motion in the frame data is given, and divides the frame data into a plurality of motion units by dividing the specified boundary. And split. Here, the boundary of motion is determined to exist when the speed crosses the threshold value TH. That is, (1) the moment when the speed increases and shifts from a value smaller than the threshold value TH to a value larger than the threshold value TH, and (2) conversely, the speed decreases and exceeds the threshold value TH. Two times, that is, a moment from a large value to a value smaller than the threshold value TH, are determined as the time corresponding to the boundary of motion.

  Further, the velocity v (t) is calculated by the following equation (5) as an absolute value of the differentiation of the relative position x (t) of the equation (4). FIG. 5 shows an example of a graph of the calculated speed.

  The threshold value TH can be set as follows. That is, when the value is changed little by little from the lowest value TH_low to the highest value TH_high, the number of obtained motion elements changes for each threshold value TH. A threshold value that gives the maximum number among them may be used as the optimum threshold value TH_opt.

  FIG. 6 is a graph of the number of operating pieces obtained when the threshold value TH is changed with respect to the example of the velocity graph of FIG. In the example of FIG. 6, since the maximum value 55 of the number of motion elements is obtained at the threshold TH = 0.8, the threshold TH = 0.8 may be adopted as an optimum value.

  The dividing unit 12 passes the motion element obtained by dividing by the optimum threshold TH_opt to the normalizing unit 13 as a final result.

  The normalization unit 13 performs normalization of the time axis at the relative position of each joint by interpolating with a fixed number of frames (for example, 25 frames) for each motion element, and the normalized motion element is Passed to the classification unit 14. As an interpolation method, for example, cubic spline interpolation is preferably used.

  The classification unit 14 treats each normalized motion element as one vector composed of all joint relative position data of all the frames, so that each motion element is assigned to any motion type. The corresponding classification is performed, and the result is passed to the feature amount calculation unit 2. The following embodiments are possible for the classification.

  In one embodiment, it is possible to classify which operation type corresponds to the k-means method (k-means method). However, in k-means, it is necessary to give the number of types of operation as the number of divided clusters as a fixed value. Here, as a general rule, the number of action types included in the action varies depending on the type of human action, so the number of action types for each action data type is registered as prior knowledge as shown in the table of FIG. In addition, the number of operations may be set according to the registered information.

  In the example of FIG. 7, three types of actions “Door Knock, Walking, Sitting in a Chair” are given, and the number of types of actions constituting the action is “4 types, 4 types, 3 types”, respectively. Is given as prior knowledge.

  Note that the action data as the input data in the present invention is preferably one in which the kind of action is determined as in the example of FIG. For example, if there is action data that includes all three types of Fig. 7 "Door knock and then walk and sit in a chair", the "Door knock", "Walk", and "Sit on chair" in advance What has been divided is preferably used as the input in the present invention. When determining the number k of divided clusters based on the prior knowledge, in addition to the prior knowledge as shown in FIG. 7, in order to use the prior knowledge, information on what kind of action is included in the action data is also emotion It is prepared as an input to the information estimation device 10.

  Also, in one embodiment, when prior knowledge as described above is not given, the number k of action types to be classified is determined by the Rule of thumb (rule of thumb) according to the following equation (6). Then, k-means may be applied. Note that n is the number of motion elements divided by the dividing unit 12.

  In one embodiment, when prior knowledge as described above is not given, classification may be made by a hierarchical cluster tree instead of k-means. In a hierarchical cluster tree, even if the number of types k to be classified is not given as prior knowledge, it can be naturally classified into sufficiently separated clusters. Here, since the inconsistency coefficient of the link of the cluster tree represents that the similarity between the motion elements changes suddenly, it is possible to perform natural classification by referring to the inconsistency coefficient.

  Note that the motion elements obtained by the processing of the motion analysis unit 1 are normalized as described above, and therefore the motion elements handled by the subsequent units 2, 3, and 4 are also normalized. In the following description, the description of “normalized” is omitted for simplification of the description of the operation pieces handled by the respective units 2, 3, and 4. Similarly, the description of “normalized” is also omitted for motion elements handled in the learning motion analysis unit 31 and later in FIG. 8 described later.

  The feature amount calculation unit 2 calculates a feature amount indicating the feature of the motion for each motion unit obtained from the motion analysis unit 1, and passes the result to the unit emotion estimation unit 3. Here, although there are many definitions of feature quantities, for example, four types listed in the following formulas (7-1) to (7-4) can be used. For “specific joint h”, one or more arbitrary combinations can be used in each equation. The feature amount finally passed to the segment emotion estimation unit 3 may be a combination of all the four types, or a combination of any part thereof. In addition to the following four types, other types of feature values may be added. In the equations (7-1) to (7-4), N is the number of frames belonging to the motion element (= time length of the motion element).

  FIG. 8 is a functional block diagram of the segment emotion estimation unit 3. The segment emotion estimation unit 3 includes an estimation unit 30, a learning motion analysis unit 31, a learning feature amount calculation unit 32, and a learning unit 33.

  The estimation unit 30 applies a classifier constructed by pre-learning to the feature amount for each motion unit calculated by the feature amount calculation unit 2, and each motion unit is assigned to any emotion of a predetermined type. Whether it is applicable is estimated as emotion information, and emotion information for each motion element is passed to the emotion integration unit 4.

  Here, in order to enable the estimation unit 30 to perform the above estimation, each of the units 31, 32, and 33 constructs a classifier by prior learning, and details thereof are as follows. For example, a support vector machine can be used as the classifier.

  Since the learning motion analysis unit 31 and the learning feature amount calculation unit 32 perform the same processing as the motion analysis unit 1 and the feature amount calculation unit 2, respectively, description thereof is omitted. However, the data to be processed is learning behavior data prepared for a large number of people (when the behavior data is about humans) in advance. In each of the learning behavior data, emotion information corresponding to any of a plurality of types of emotions, similar to that estimated by the estimation unit 30, is given as a label manually.

  In this way, for each of the plurality of learning behavior data, the learning motion analysis unit 31 divides the motion element into normalized motion elements and gives a distinction as to which motion type each motion element corresponds to, The learning feature amount calculation unit 32 calculates a feature amount for each motion element, and the learning unit 33 receives these results. Since the motion analysis unit 1 and the learning motion analysis unit 31 perform the same processing, the number k of operation types to be classified is set to a common value.

  The learning unit 33 obtains the data obtained by associating the feature amount in the motion unit obtained as described above with the motion unit and the emotion information given as a label to the corresponding original behavior data. A classifier is constructed by performing learning as learning data. If the classifier is a well-known (multilevel) support vector machine, its parameters are determined by learning. The classifier enables the estimation unit 30 to estimate emotion information as described above.

  FIG. 9 is a functional block diagram of the emotion integration unit 4. The emotion integration unit 4 includes an integration unit 40 and a weight estimation unit 41.

  The integration unit 40 uses the weight information estimated by the weight estimation unit 41, which will be described later, for each motion unit from which the emotion information obtained from the unit emotion estimation unit 3 is estimated, Then, by integrating all the action pieces of the action data that is the input data of the emotion information estimation device 10, a result as to which emotion information the action data corresponds to is obtained.

In the integration, weight information is used. The weight information is a weight w (i) for each action type i (i = 1, 2,..., K; k is the number of classifications in the classification unit 14) specified by the emotion information. Of the series of motion segments obtained by dividing the behavior data by the motion analysis unit 1, the emotion information estimated by the segment emotion estimation unit 3 is emotion j (j = 1, 2,…, M; M Is the number of labels given by learning data), and the statistically calculated likelihood based on the number N (i, j) of motion elements whose motion type classified by motion analysis unit 1 is motion type i Is L (i, j), emotion j = j _{[estimation result]} in the action data as the final result output by the integration unit 40 is given by the following equation (8). Here, when calculating the likelihood L (i, j) from the number N (i, j), various known calculation methods can be used.

  Therefore, in the present invention, as described above, voting for emotion j is performed in consideration of the weight w (i) for each action type i, and the one with the maximum likelihood is selected. Emotion estimation can be performed.

  The weight estimation unit 41 estimates weight information w (i) that the integration unit 40 refers to in order to perform the integration. The following first to fourth embodiments are possible for the estimation.

  In the first embodiment, the weight w (i) is set for each action type i (i = 1, 2,..., K) classified by the motion analysis unit 1 based on prior knowledge or user input. For example, a weight of “0.3, 0.4, 0.2, 0.1” is set for all four types of motion.

  In the second embodiment, the emotion information estimated by the segment emotion estimation unit 3 is statistically processed as follows for all motion segments included in the action data of the input data, and the weight w (i) is calculated. To do. FIG. 10 shows four types of emotion information (eg, “joy”, “anger”, “sorrow”, “easy”) and four types of motion (eg, “preparation stage”, “first half production stage”, “second half production stage”, “end stage”). ) Is an example of the distribution of emotion information j determined for each action type i. From such a distribution, entropy h (i) is calculated by the following equation (9).

However, p _i (j) is the probability of emotion j in action type i, and the probability is determined from the distribution as described above. Then, the weight w (i) for the operation type i is calculated by the following equation (10).

  In the third embodiment, the same calculation as in the second embodiment is performed when the learning unit 33 constructs a classifier instead of the behavior data that is an input to the emotion information estimation device 10 and is an emotion estimation target. It carries out using a plurality of behavior data as the learning data used. For this reason, as an additional process, an estimation process by the estimation unit 30 is performed on a series of motion elements obtained by dividing each of the plurality of behavior data as the learning data and given a feature amount. The emotion information in a series of motion elements in the learning data may be obtained, and thereby the distribution of the processing target may be obtained.

  That is, in the additional processing in the third embodiment, the classifier itself constructed by the learning data for the motion element in the learning data used for pre-constructing the classifier used by the estimation unit 30 Is applied by the estimation unit 30.

  In the fourth embodiment, the calculation is performed based on the learning data as in the third embodiment. However, in the third embodiment, the estimation process by the estimation unit 30 is performed on the motion element in the learning data, and the emotion information Further, the following processing is performed. That is, the emotion information of the estimation result in the learning data is compared with the emotion information previously given as the correct answer label in the learning data, and the correct answer rate c (i) is obtained for each action type i. Then, the accuracy rate is normalized by the following equation (11) to obtain the weight w (i).

  As described above, according to the present invention, it is possible to estimate emotion information with high accuracy from behavior data by using the weight w (i). The present invention can also be provided as a program that causes a computer composed of known hardware such as a CPU (Central Processing Unit), a memory, and an input / output interface to function as each part (including all) of the emotion information estimation apparatus 10 It can also be provided as an operation method of the emotion information estimation apparatus 10. When the computer functions as each part of the emotion information estimation device 10, it is sufficient that the CPU operates according to a predetermined command that realizes the function of each part, and data to be referred to at that time can be stored in a memory. That's fine.

  10 ... Emotion information estimation device, 1 ... Motion analysis unit, 2 ... Feature quantity calculation unit, 3 ... Fragment emotion estimation unit, 4 ... Emotion integration unit

Claims (9)

A motion analysis unit that divides the behavior data into motion segments on the time axis, classifies each motion segment, and identifies which of the plurality of motion types corresponds,
A feature amount calculation unit for calculating a feature amount from each motion element;
A unit emotion estimation unit that estimates which of the plurality of pieces of emotion information each motion unit corresponds to based on the calculated feature amount;
An emotion integration unit that identifies which of the plurality of emotion information the behavior data corresponds to by integrating the estimated emotion information over the entire behavior data for each motion element;
The emotion information estimation device, wherein the emotion integration unit integrates the motion information using weight information for the specified action type for each action element.   The emotion information estimation apparatus according to claim 1, wherein the weight information is predetermined by giving a predetermined value of weight for each action type.   The emotion integration unit obtains a distribution of emotion information of motion segments for each motion type in the behavior data, and determines the weight information for each motion type based on the distribution. Emotion information estimation device.   The emotion integration unit obtains the probability of emotion information in each action type from the distribution, obtains the entropy of each action type from the probability, and obtains the weight information for each action type based on the entropy. The emotion information estimation apparatus according to claim 3. The segment emotion estimation unit
A classifier constructed by performing learning based on a feature amount calculated from a series of motion pieces obtained from learning behavior data and emotion information previously given as a label to the learning behavior data. By using
The emotion information estimation apparatus according to claim 1, wherein based on the calculated feature amount, it is estimated which of the plurality of emotion information each motion element in the behavior data corresponds to. The emotion integration unit
While identifying which of a plurality of operation types a series of operation elements obtained from the learning action data,
A distribution of emotion information of motion segments obtained by applying the classifier to a series of motion segments obtained from the learning behavior data is obtained for each motion type, and for each motion type based on the distribution The emotion information estimation apparatus according to claim 5, wherein the weight information is obtained.   The emotion integration unit compares the emotion information of the motion segment for each motion type obtained by applying the classifier to the learning data with the emotion information previously given as a label to the learning behavior data The emotion information estimation apparatus according to claim 6, wherein a correct answer rate for estimating emotion information for each action type is obtained, and the weight information for each action type is obtained based on the correct answer rate. A motion analysis stage that divides the behavior data into motion segments on the time axis, classifies each motion segment, and identifies which of the plurality of motion types corresponds,
A feature amount calculation stage for calculating a feature amount from each motion element;
Based on the calculated feature quantity, a segment emotion estimation stage for estimating which of the plurality of emotion information each motion segment corresponds to,
An emotion integration stage that identifies which of the plurality of emotion information the behavior data corresponds to by integrating the estimated emotion information over the entire behavior data for each motion unit,
The emotion information estimation method, wherein in the emotion integration stage, the weight information for the specified action type is used for each action element and is integrated.   An emotion information estimation program for causing a computer to function as the emotion information estimation apparatus according to any one of claims 1 to 7.