JP3403393B2 - Kansei expression system and Kansei expression method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a Kansei expression system or Kansei expression method, and more particularly to a Kansei expression system or Kansei expression method for outputting a multimedia expression corresponding to the Kansei of a subject.

[0002]

2. Description of the Related Art An example of a conventional emotion expression system of this type is disclosed in Japanese Patent Application No. 2000-306079 filed on Oct. 5, 2000. This conventional technique is based on the temporal expression from the body expression of the dancing subject,
Three parameters representing power and spatiality are measured, and the subject's sensitivity is estimated based on the measured parameters.
Then, the multimedia expression (for example, the video content and the music content expressed by the large screen screen and the sound system placed in front of the subject) is changed according to the estimated sensitivity. The subject is
By expressing your sensibility through your own dance, you can lead the multimedia expression and interact with it.

[0003]

However, in the prior art, even if a plurality of subjects perform dance at the same time to express their respective sensibilities, it is not possible to change the multimedia expression in accordance with the respective expressed sensitivities. could not. Further, even if a plurality of test subjects can express the same kind of sensibilities in cooperation, it is not possible to detect the state in which the sensibilities are cooperative,
It was not possible to output a multimedia expression suitable for the cooperative state.

Therefore, a main object of the present invention is to provide a kansei expression system capable of outputting various multimedia expressions corresponding to sensibilities expressed by a plurality of people.

[0005]

According to a first aspect of the present invention , a feeling for each subject is calculated based on physical quantities of body expressions of a plurality of subjects .
Estimating means for estimating the sex vector , obtained from the estimating means
A kansei expression system comprising: a cooperation degree calculating means for calculating a cooperation degree based on a plurality of affective vectors ; and a changing means for changing a multimedia expression based on the cooperation degree calculated by the cooperation degree calculating means.

A second aspect of the present invention is a kansei expression method for a kansei expression system that outputs a multimedia expression corresponding to the kansei of a subject, and the kansei vector for each subject is based on the physical quantities of the body expressions of a plurality of subjects. Is calculated, the degree of cooperation is calculated based on each estimated plurality of affective vectors , and the multimedia expression is changed based on the calculated degree of cooperation.

[0007]

According to the first aspect of the invention, the sensitivity of each of a plurality of subjects
The vector is estimated by the estimation means based on the physical quantity of the body expression of each subject. Each estimated sensitivity
The vector-based degree of cooperation is calculated by the degree-of-cooperation calculating means, and the changing means changes the multimedia expression based on the calculated degree of cooperation. As a result, various multimedia expressions corresponding to the sensibilities expressed by a plurality of people are output.

In the preferred embodiment, the multimedia presentation is modified as follows. If the degree of cooperation is lower than the threshold value, a plurality of images corresponding to the number of subjects are divided and displayed on the screen by the first display means. When the degree of cooperation is equal to or better than the threshold value, one image is entirely displayed on the screen by the second display means.

The specific image showing space sharing may be displayed by the second display means when the degree of cooperation is equal to or better than the threshold for a predetermined time.

In another preferred embodiment, when the degree of cooperation is lower than the threshold value, the music corresponding to the sensitivity of any one of the subjects is output by the first output means. When the degree of cooperation is equal to or better than the threshold value, the music corresponding to the sensitivity of the cooperation state is output by the second output means.

In another preferred embodiment, there are plural kinds of sensibilities, and the estimating means estimates the main sensation of the subject. At this time, the purity of the estimated main sensitivity is calculated by the purity calculating means. The purity calculating means preferably calculates, as the purity, the ratio of the absolute value of the main affectiveness to the total sum of the absolute values of the multiple types of affectiveness.

In order to change the multimedia presentation,
You may display the person image which shows a test subject in the aspect according to the calculated purity. At this time, more preferably, one of the photographed image of the subject and the contour image subjected to the special effect is displayed according to the purity.

Furthermore, music corresponding to the main sensitivity of any one of the subjects may be output based on the calculated purity. At this time, in one embodiment, the music corresponding to the main sensitivity having the highest purity is output, and in another embodiment, the rhythm system is determined by the main sensitivity having the Mth (M: integer) highest purity and the purity is determined. The melody system is determined by the Nth (N: integer) highest primary sensitivity.

[0014] In yet another preferred embodiment, calculation detecting means, each of the similarity-sensitive vector normalized normalized sensitivity vector, closeness of the mean values of the elements forming each of the sensitive vector, and each The degree of cooperation is calculated based on the closeness of the variance values of the elements forming the affective vector.

[0015] In the second invention, sensitivity of each of each subject based on the physical quantity of physical expression of a plurality of subjects vectors
Le is estimated, based on the sensitivity vector of each estimated
And the degree of cooperation is calculated. The multimedia representation is changed based on the calculated degree of cooperation.

[0016]

EFFECTS OF THE INVENTION According to these inventions, the multimedia expression is changed based on the degree of affective cooperation estimated for each subject. Therefore, various multimedia expressions according to the subject's sensitivity are possible. Become.

The above-mentioned objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

[0018]

First Embodiment FIG. 1 shows the configuration of a collaborative emotional expression system 10 which is an embodiment of the present invention. This system 10 includes subjects A and B who dance in a studio S. It includes cameras 12a and 12b for taking images, respectively. The subjects A and B do not necessarily need to be present in the same studio S, but here, as a basic example, a case where they are present in the same studio S will be described.

First, the general operation will be described. In FIG.
Each of the cameras 12a and 12b has subjects A and B
Is photographed at a frame rate of (1/30) second, and a photographed image signal is given to the computer 14. Computer 14
Samples the captured image signal of the subject A and the captured image signal of the subject B every (1/15) seconds independently and in parallel. Then, the sensitivities of the subjects A and B are individually estimated based on the sampled photographed image signals, and the estimated sensibility of the sensations and the degree of cooperation of the sensitivities between the subjects A and B are calculated. Computer 1
Further, 4 reads out from the database 16 the video content and the music content suitable for the estimated sensitivity and the degree of sensitivity cooperation, and outputs the read video content and the music content from the monitor 18 and the speaker 20, respectively.

The sensibilities estimated by the computer 14 are "Natural", "Happy", and "Dynami" which are classified in advance.
c ”,“ Flowing ”,“ Sharp ”,“ Lonely ”and“ Sol
It is a score indicating the sensitivity of each of the seven types of "emn". Therefore, the estimated sensitivity is 7
It is represented by a 7-dimensional vector with elements. Hereinafter, the estimated sensitivity may be referred to as a sensitivity vector.

The computer 14 estimates such a sensitivity vector by the following method. First, based on the physical quantities of the body expressions of the subjects A and B, the three parameters of the subjects A and B, which are temporality, power, and spatiality, are determined independently and in parallel for each subject. Here, temporality means a parameter related to the movement or rhythm of the subject's body,
The force means a parameter related to the acceleration of the body motion of the subject, and the spatial property means a parameter related to the spatial spread of the body of the subject.

With reference to FIG. 2, the temporality is obtained by the first derivative of the barycentric coordinates (Xc, Yc) obtained from the silhouette image of the subject. Also, the force is the barycentric coordinate (Xc,
It is obtained by the second derivative of Yc). Further, the spatial property is obtained from the ratio of the area of the subject's silhouette image to the area of a rectangle (a rectangle having a width Xs and a height Ys) circumscribing the silhouette image.

The computer 14 subsequently performs a statistical analysis by multiple regression analysis on the three parameters individually determined for the subjects A and B. by this,
The affective vectors of subjects A and B are estimated. As described above, after the sensitivity vector is estimated, the video content and the music content suitable for the estimated sensitivity and the degree of affective cooperation are output.

In the statistical analysis by the multiple regression analysis, first, teacher data as a base for estimating the sensitivity vector is required. This teacher data is obtained in advance as follows. A photographed image output from the camera 12a by subjecting a certain subject, for example, a professional dancer excellent in sensibility expression, to dance the 7 kinds of dances expressing the 7 kinds of sensibilities in front of the camera 12a (or the camera 12b) for a certain period of time. Record the signal (dance video). Next, the recorded dance video is reproduced and the following two processes are performed.

In the first process, several tens of students are shown a dance image expressing each sensitivity, and they are asked to answer which of the seven types of sensitivity each dance image shows. With this, the voting rate for the sensitivity of each dance image is obtained. The voting rate consists of seven numbers corresponding to seven kinds of sensibility. These seven numerical values are actual measurement values indicating how much the dance image includes seven kinds of sensibilities, and this is 7
It is a numerical value that forms the actual measurement value of the dimension vector.

In the second process, the silhouette image of the dancer is extracted from each dance image, the barycentric coordinates and the area of the silhouette image are measured from the extracted silhouette image, and the barycentric image and the measured value of the area are measured. From this, three parameters of temporality, power and spatiality are calculated. Each calculated parameter is smoothed in the time direction. Specifically, each parameter is weighted averaged over a time period corresponding to, for example, two measures, and the weighted average value of each parameter is calculated every two measures period. In addition,
The two-bar period corresponds to 4 seconds when measured with a rhythm of 120 beats / second.

The 7-dimensional vector actual measurement value obtained for each of the seven types of dance images and the three weighted average values measured for each two-measure period for each dance image serve as teacher data.

When the teacher data is obtained, the multiple regression equation for each sensitivity is determined by the least squares method using the three weighted average values as explanatory variables and the seven-dimensional vector measured value as an objective variable. That is, by multiple regression analysis, 7
Seven multiple regression equations corresponding to each type of sensitivity are obtained. Right side of the obtained 7 multiple regression equations (explanatory variable side)
By substituting the three parameters based on the body motion separately photographed into, on the left side (objective variable side), the estimated values of the voting rates of the seven sensibilities that the body motion would have been obtained.
Since each element of this 7-dimensional vector can be regarded as a score of each sensitivity-likeness estimated from the body motion,
The 7-dimensional vector composed of the values on the left side is used as the estimated value of the affective vector representing the affectiveness expressed by the body motion. Such seven multiple regression equations are stored in the computer 14.

When the subjects A and B dance in the studio S while the cooperative emotional expression system 10 is activated, the computer 14 causes the cameras 12a and 12 to operate.
Three parameters of temporality, power, and spatiality are calculated based on the photographed image signal output from b, and the calculated parameters are substituted into the right side of the above seven multiple regression equations. As a result, on the left side, the values of the seven-dimensional affective vectors expressing the affectiveness expressed by the subjects A and B are individually obtained.

Next, the specific operation of the computer 14 will be described with reference to the flow charts shown in FIGS. The routine shown in FIG. 6 is a routine for estimating the sensitivity vector of the subject A based on the captured image signal output from the camera 12a, and the routine shown in FIG.
b is a routine for estimating the emotional vector of the subject B based on the captured image signal output from b, and the routine shown in FIGS. 8 to 11 is a monitor based on the estimated emotional vectors of the subjects A and B. 18 is a routine for controlling video content and music content (multimedia expression) output from the speaker 18 and the speaker 20. The routine of FIG. 6, the routine of FIG.
The three routines in FIG. 9 are activated at the same time and are executed in parallel with each other.

First, referring to FIG. 6, in step S1a, initialization processing including clearing the data storage area 141a for the subject A is performed, and in step S3a, it is determined whether (1/15) seconds have elapsed from system startup. To do. (1 /
15) After the lapse of seconds, the camera 12a is operated at step S5a.
The photographed image signal from is taken in, and then step S7a
At, it is determined whether the captured image signal includes a subject image other than the studio background image. This determination is made, for example, by the chroma key technique.

If the captured image signal includes only the background image, the process returns from step S7a to step S3a to prepare for the next captured image signal. On the other hand, if the captured image signal includes any subject image other than the background image, the subject image is regarded as the human image of the subject A, and the process proceeds to step S9a.

In step S9a, a person image is extracted from the captured image signal, and a silhouette image is created from the extracted person image. In the subsequent step S11a, the center of gravity coordinates (Xc, Yc) are calculated from the created silhouette image.
And the area Hs of the silhouette image are calculated, and the calculated barycentric coordinates (Xc, Yc) and the area Hs are calculated according to Equation 1. With this, three parameters of the subject A, that is, temporality, power, and spatiality are measured.

[0034]

[Equation 1]

The time property is obtained by first-order differentiating the barycentric coordinates, that is, by dividing the difference between the barycentric coordinates (1/15) seconds before and the current barycentric coordinates by (1/15) seconds. . The force is obtained by second-order differentiating the barycentric coordinates, that is, by dividing the difference between the previous first-order differential value and the current first-order differential value by (1/15) seconds. Spatiality is obtained by dividing the area of the silhouette image by the area of the rectangle circumscribing the silhouette image.

In step S13a, the measured time,
The affective vector of the subject A is estimated based on the three parameters of force and spatiality. That is, “Natura
l ”,“ Happy ”,“ Dynamic ”,“ Flowing ”,“ Shar
Substituting the three parameters of the subject A obtained in step S11a into the right side of the seven multiple regression equations corresponding to the seven types of sensibilities of "p", "Lonely", and "Solemn". Each element of the 7-dimensional affective vector appears on the left side of the 7 multiple regression equations.As described above, each element of the affective vector indicates the score of 7 kinds of affectiveness estimated from the dance video of the subject A. The estimated affective vector value is stored in the data storage area 141a.

In step S15a, the game end flag F
Determine the state of. The game end flag F is reset in step S21 (FIG. 8) described later, and the step S51
(Fig. 9). If the game end flag F is in the reset state, the process returns to step S3a to capture the photographed image signal of the subject A after the next (1/15) seconds. On the other hand, if the game end flag F is in the set state, the process proceeds to step S17a. In step S17a, it is checked whether the operator has given a system termination instruction. If no system termination instruction is given,
The process returns to step S1a to prepare for the start of the next game. On the other hand, when the system termination instruction is given, this routine is terminated.

FIG. 7 shows Sutin for processing the photographed image signal of the subject B captured from the camera 12b.
As can be seen by comparing FIG. 7 and FIG. 7, this routine is similar to the routine shown in FIG.

The difference is that the data storage area 141b for the subject B is cleared in step S1b, and the sensitivity vector value estimated from the dance image of the subject B in step S13b is the data storage area 141b.
Is the point stored in. Note that the routines shown in FIGS. 6 and 7 are independently executed without being synchronized with each other, but the latest estimated affective vector is always left in the data recording areas 141a and 141b.

Next, the processing for controlling the video content and the music content will be described with reference to FIGS. First, in step S21, initialization including clearing the data storage area 141c is performed, and then in step S23, the background image for introduction and the music for introduction are monitored 1
8 and the speaker 20. In step S25, it is determined whether (1/15) seconds have elapsed from the start,
If YES, both subject A and B are in studio S
It is determined in step S27 whether or not it exists. This determination is made based on whether or not the sensitivity vector values of both subjects A and B exist in the data storage areas 141a and 141b.

In step S25 immediately after the first (1/15) second has elapsed, there is a possibility that neither the sensitivity vector values of the subjects A and B exist. Moreover, even if one subject appears in the studio S and the sensitivity vector of this subject is obtained, the other subject does not appear in the studio S, and the sensitivity vector of this subject may not be obtained. In such a case, the process proceeds from step S27 to step S29, and only the human image of the subject existing in the studio is overlaid on the background image for introduction already output. When the overlay process is completed, the process returns to step S25.

Therefore, if neither subject A nor B has appeared in the studio S, only the background image for introduction is output from the monitor 18. When the subject A first appears in the studio S in this state, the human image of the subject A is overlaid on the right half of the introduction background image.
However, in any case, the introduction music is continuously output from the speaker 20.

The overlay process is executed as follows. The computer 14 has a graphic board 143.
And a camera 12 and a CG image signal such as a background image created by CG (Computer Graphic) processing.
The captured images captured from a and 12b (the captured image signals captured every (1/15) second in step S5a in FIG. 6 and step S5b in FIG. 7) are the graphic board 1
43. The graphic board 143 overlays the person image of the subject A on the introduction background image, makes the person image a silhouette, and gives a special effect such as expressing the silhouette portion of the person image with another pattern. be able to. If the computer 14 designates an image source for synthesizing and a synthesizing method for such a graphic board 143 and selects a corresponding output on the graphic board 143, the synthesized image signal created in the designated manner is obtained. It is output from the graphic board 143 to the monitor 18 at a frame rate of (1/30) second. However, the background image in the composite image is (1/3
Although it is a moving image that changes at a frame rate of 0) seconds, a person image becomes a moving image that changes at a frame rate of (1/15) seconds. Therefore, in the process of step S29, specifically, the image source for synthesizing and the synthesizing method are specified for the graphic board 143, and
It is a process of selecting the corresponding output on the item No. 3.

If YES is determined in the step S27, the process proceeds to a step S31, and the data storage areas 141a and 141a
Based on the latest emotional vector stored in 41b, the main emotions of the subjects A and B are determined, and the number of appearances of the determined main emotions is counted. The main sensitivity is the sensitivity corresponding to the element showing the maximum value among the seven elements forming the sensitivity vector. The number of appearances of the main sensibilities is seven counters 142a-1 to 142a-7 and 142b-, which are individually prepared for subjects A and B.
Counted by 1-142b-7. That is, the counters 142a-1 to 142a-7 are respectively assigned to the seven sensibilities expressed by the subject A, the counters 142a-1 to 142a-7 are respectively assigned to the seven sensitivities expressed by the subject B, and the determination is made. The counter corresponding to the obtained main sensitivity is incremented.

When the latest affective vector M _A of the subject _A and the latest affective vector M _B of the subject B are defined by the formula 2, if the formula 3 is established, the main affectiveness of the subject A is the sensitivity corresponding to the number 2. It is determined to be “Happy”, and the main affectiveness of the subject B is determined to be the affectiveness corresponding to the number 6, that is, “Lonely”. If there are a plurality of maximum values, the sensitivity corresponding to the smaller number is regarded as the main sensitivity.

[0046]

[Equation 2]

[0047]

[Equation 3]

In the following step S33, the latest sensitivity vector purity is obtained for each of the subjects A and B.
Purity is obtained by dividing the absolute value of the value of the element indicating the main affectiveness (score of the main affectiveness) by the length of the affective vector, and the calculated numerical value is between "0.0" and "1.0". Changes. The purity of subject A is defined as P _A, and subject B
If the purity of P is defined as P _B , P _A and P _B are calculated according to Equation 4. The calculated purity is stored in the data storage area 14
It is stored in 1c.

[0049]

[Equation 4]

In step S35, the latest sensitivity vector
Based on the following, the degree of affective cooperation between subjects A and B is
Calculate in this way. First, the latest sensitivity of subject A
Tol M_AAnd the latest sensitivity vector M of subject B_BThe regular
Normalized Kansei vector M _NAAnd M_NBAsk for. Positive
Normalized affective vector is all the elements of the estimated affective vector.
The average value for is 0.0 and the variance is 1.0
This is the modified sensitivity vector. Defined by the number 2
Sensitivity vector M_AAnd M_BAgainst, normalized sensitivity
Vector M_NAAnd M_NBIs defined by equation 5.

[0051]

[Equation 5]

M _A − and σ _MA are an average value and a variance value with respect to all elements (scores of seven sensibilities) of the sensitivity vector M _A of the subject A. I is a seven-dimensional unit vector, that is, I = (1,1,1,1,1,1,1,1). The normalized sensitivity vector M _NB of the subject B is also obtained in the same manner. Next, using the normalized affective vectors M _NA and M _NB , the affective cooperation degree H between the subjects A and B is calculated by the equation 6.

[0053]

[Equation 6]

[0054] Here, M _NAk and M _NBk each represent the k-th element of the normalized sensitivity vector M _NA and M _NB, therefore the first term, the normalization of the subject A and B-sensitive vector M _NA and M It is an amount to evaluate the degree of similarity of _NB . W ₁ is a weight for this evaluation amount. This evaluation amount is also the square of the length of the difference vector between the normalized sensitivity vectors M _NA and M _NB , and the closer the two normalized vectors are, the closer the value is to 0.

The second term is an amount for evaluating the closeness between the average value of all the elements forming the sensitivity vector M _A and the average value of all the elements forming the sensitivity vector M _B. However, the significance of the average value of all elements is that the larger the variance is, the thinner it is, and the average value is divided by each standard deviation (square root of variance value). W ₂ is a weight for this evaluation amount. Also in this evaluation amount, the closer the average value of the sensitivity vector M _{A and} the average value of the sensitivity vector M _B are in this sense, the closer to 0
Approach.

The third term is an amount for evaluating the closeness between the variance value of all the elements forming the sensitivity vector M _A and the variance value of all the elements forming the sensitivity vector M _B , and W ₃ is this evaluation amount. Is a weight for. This evaluation amount also approaches 0 as the variance value of the sensitivity vector M _{A and} the variance value of the sensitivity vector M _B are closer to each other.

The degree of affective cooperation thus obtained is also stored in the data storage area 141c. When evaluating the degree of affective cooperation between subjects A and B, the evaluation quantities of the second term and the third term are also used not only in the proximity of the normalized affective vector shown in the first term. This is because it is considered that the difference between the average value and the variance value of all the elements of the sensitivity vector before normalization is also effective.

In step S37, it is determined whether the two-bar period has elapsed. This period corresponds to 4 seconds when measured with a rhythm of 120 beats / second, for example. NO here
If so, the process returns to step S25 and the above-mentioned step S31.
~ The process of S35 is repeated. This makes counter 1
42a-1 to 142a-7 and 142b-1 to 142
b-7 is updated, and the purities and the degree of affective cooperation of the subjects A and B are accumulated in the data storage area 141c.

When 4 seconds have passed, the process proceeds to step S39,
The maximum predominant sensibility of subjects A and B is determined based on the number of appearances of each main sensibility counted in the last 4 seconds. That is, the sensitivity corresponding to the counter showing the maximum value among the counters 142a-1 to 142a-7 is determined as the maximum dominant sensitivity of the subject A, and the counters 142b-1 to 142b- are determined.
The sensitivity corresponding to the counter showing the maximum value out of 7 is determined as the maximum dominant sensitivity of the subject B. In step S41, the counters 142a-1 to 142a-7 and 142 are
Reset b-1 to 142b-7. Also, the counter for measuring the time of 4 seconds is cleared in this step.

In step 43, the purity calculated in the latest 4 seconds is smoothed in the time direction according to equation 7, and step S
At 45, the degree of cooperation calculated in the latest 4 seconds is smoothed in the time direction according to equation (8). In addition, the smoothing process by the formulas 7 and 8 is a process of obtaining a moving average.

[0061]

[Equation 7]

[0062]

[Equation 8]

In step S47, the video content and music content (multimedia expression) output from the monitor 18 and the speaker 20 are shown in FIGS.
Change according to the subroutine shown in.

In step S49, it is determined whether the time corresponding to one game has passed, or whether the emotional expressions by the subjects A and B have reached the goal. And
If none of the conditions are satisfied, the process returns to step S25, and the same processing as described above is repeated for the next 4 seconds.
On the other hand, if any of the conditions is satisfied, the game end flag F is set in step 51 and then the process proceeds to step S53 to determine whether or not a system end instruction has been issued by the operator. If the system termination instruction has not been issued, the process returns to step S21 to start the operation for the next game. On the other hand, if the system termination instruction has been issued, the processing is terminated.

Next, with reference to the subroutines shown in FIGS. 10 and 11 and the display examples shown in FIGS. 3 to 5, specific multimedia expressions and game characteristics will be described. At the time when this subroutine is reached, for example, subject A's maximum dominant sensation is "Happy", subject B's maximum dominant sensation is "Lonely", and subject A's purity is higher than subject B's purity. Furthermore, it is assumed that the degree of cooperation between the two is worse than a certain threshold. At this time, neither the sensitivity matching condition in step S61 nor the cooperation degree condition in step S63 is satisfied, and the process proceeds to step S65. It should be noted that the state of good cooperation is generally said to have a high degree of cooperation, but the degree of cooperation H according to equation 6 is such that the more the well cooperated , the numerical value H approaches 0, so avoid confusion here. Therefore, the better the degree of cooperation, the better the degree of cooperation, and the opposite state is used as the expression of poor degree of cooperation.

In step S65, the background image content corresponding to the maximum sensitivities of the subjects A and B is read from the database 16. In this case, as the background image of the subject A, an image reminiscent of the subject's maximum dominant sensation “Happy” (for example, the hexagonal pattern shown on the right side of the screen in FIG. A large image is selected. Also, as the background image of the subject B, the maximum dominant sensitivity of the subject B is “Lonely”.
(For example, an image in which the dark blue raindrop pattern shown on the left side of the screen in FIG. 3 slowly drops) is selected. Of course, not only this background image, but also add a gorgeous flower pattern on the "Happy" side, and add a black and white flower that flutters in the wind to the "Lonely" side to express even more loneliness. You may

In steps S67 to S71, the human image of the subject A is overlaid on the background image in a manner according to the purity of the subject A. Specifically, if the purity of the subject A is equal to or higher than the threshold value, the process proceeds from step S67 to step S69, the person image of the subject A is made into a silhouette, or a special effect image in which a specific pattern is displayed in the contour of the person image is displayed. screen
Overlays the background image on the right . If the purity of the subject A is lower than the threshold, the process proceeds from step S67 to S71, and the photographed image of the subject A is overlaid on the background image on the right side of the screen . For subject B, the same processing is executed in steps S73 to S77. In the display example of FIG. 3, the purities of the subjects A and B are both lower than the threshold value, and therefore, in both of them, the photographed image is overlaid on the background image.

The background image and the person image are displayed on the graphic board 14 by the above-described steps S65 to S77.
When set to 3, the current corresponding output on the graphic board 143 is selected in step S79. by this,
A divided image as shown in FIG. 3 is displayed on the monitor 18.
Since the subroutines shown in FIGS. 10 and 11 are executed only every 4 seconds, the background image contents of the subjects A and B and the pattern in the outline of the person image can be switched only every 4 seconds. However, due to the use of the graphic board 143, the real-life portrait image of each subject and the outline image thereof displayed in the overlay change at a frame rate of (1/15) seconds, and the background image changes at a frame rate of (1/30) seconds. Changes.

In step S81, the purities of the subject A and the subject B are compared with each other, and the process of step S83 or S85 is performed according to the comparison result. That is, if the purity of the subject A is higher, the music content corresponding to the maximum dominant sensitivity of the subject A is read from the database 16 and the read music content is set on the sound board 144 in step S83. On the other hand, if the subject B's maximum dominant sensation is greater, the music content corresponding to the subject B's maximum dominant sensitivity is read from the database 16 and the read music content is set in the sound board 144. If the purities are equal, the subject A's maximum predominant sensitivity is prioritized.

In step S87, the sound board 144
The corresponding output for this time is selected, and the music content set by this is output from the speaker 20. In the present case, since the purity of the subject A is higher, the bright and lively music that is associated with the maximum dominant sensitivity "Happy" of the subject A is selected. When the process of step S87 is completed,
Return to the upper level routine.

Under the circumstances as described above, the subject B is to pay attention to the sensibilities expressed by him and to raise the purity within a certain period of time to change the music to the one suited to his dance. You may compete with. Also, the subject B changes the expression content so as to match the sensitivity of the subject A at an early stage, and matches the sensitivities (maximum dominant sensitivities of each other) as a pair of the subjects A and B, and at the same time increases the degree of cooperation. May be.

After the above-described state, the maximum dominant sensitivities of the test subjects A and B coincide with each other as "Happy" and the degree of cooperation between the subjects reaches a certain threshold value when the process shifts to this subroutine several times. Then steps S61 and S
In any of 63, YES is determined, and the process proceeds to step S91 through step S89 in FIG. Step S
In 91, first, the background image contents corresponding to the maximum dominant sensation expressed by the subjects A and B are read out from the database 16 and set on the graphic board 143 as one background image that spreads over the entire screen. In this case, as shown in FIG. 4, the background image showing “Happy” is expanded to fill the screen.

In subsequent steps S93 to S103, the same processes as those in steps S67 to S77 described above are performed, and the human images of the subjects A and B are overlaid on the background image in a manner according to the purity. In the display example shown in FIG. 4, the purity of the subject A is equal to or higher than the threshold value, and the special effect image is overlaid on the background image. On the other hand, for subject B,
The purity is lower than the threshold value, and the photographed image is overlaid on the background image.

In step S105, the output corresponding to this time on the graphic board 143 is selected.
The image shown in is actually displayed on the monitor 18. As described above, the sensitivity of the pattern in the background image or the outline of the person image is 4
Although it is updated only every second, the overlayed human image of each subject and its outline image change at a frame rate of (1/15) seconds, and the background image changes at a frame rate of (1/30) seconds. .

At step S107, subjects A and B are examined.
Is read out from the database 16 and set in the sound board 144. In the following step S109, the output corresponding to this time on the sound board 144 is selected, and the music corresponding to the matching sensitivity is output from the speaker 20. In the present case, bright and lively music that reminds us of the matching sensitivity "Happy" is selected, and, for example, a melody by an orchestra is added. In other words, music that is more exciting than the music at the stage of the divided image shown in FIG. 3 is output. When the process of step S109 is completed, the process returns to the upper layer routine.

After the above situation, subjects A and B
If the sensibilities of the two match and the degree of cooperation between the two is better than the threshold value or equal to the threshold value continues for 32 seconds or more, the process proceeds from step S89 to step S111. Proceeding to this step is the ending stage of this game. In step S111, a pattern indicating the space sharing of the subjects A and B is added to the background image, and in the subsequent step S113, the characters "goal!" Are overlaid on the entire screen. Then, in step S115, the corresponding output on the graphic board 143 is selected. As a result, as shown in FIG. 5, the spiral pattern surrounding the subjects A and B becomes “Happ
Added to the background image corresponding to "y", and "Goal! ”
Is displayed on the full screen. From this screen display, the test subject and the spectator know the end stage of the game and are satisfied.

At step S117, subjects A and B are examined.
The sound board 144 reads the additional melody content by the orchestra that matches the maximum predominant sensation expressed by matching
Set to. In step S119, the sound board 1
The corresponding output on 44 is selected, whereby the music that is more exciting than the music at the display stage shown in FIG. 4 is output from the speaker 20. Second Example In the first example, the display mode of the overlaid person image is a real image when the purity is low, and a special effect image such as a silhouette image when the purity is high. By reversing this, a special effect image such as a silhouette image may be displayed when the purity is low, and a live-action image may be displayed when the purity is high. Third Embodiment Further, in the first and second embodiments, the display mode of the overlaid person image is changed according to the purity, but in addition to the person image, the background image also has the purity. It may be changed accordingly. Specifically, as the purity of “Happy” becomes higher, a number of gorgeous flower patterns may be displayed on the background image, or a number of happy mascot characters may be displayed on the background image. Further, when the degree of cooperation is lower than the threshold value, the area of the dividedly displayed image may be changed to a wipe shape according to the purity. In this case, for example, in FIG.
If the purity of “Happy” is high in the image shown in
A display mode in which the width of the image on the "appy" side is widened is conceivable. Of course, such processing may be performed in combination with the expression mode in the first or second embodiment. Fourth Embodiment In the first embodiment, when the degree of cooperation between the subjects A and B is worse than a certain threshold value, the music content corresponding to the estimated sensitivity having a higher purity is output. However, for example, the rhythm system is determined by the estimated sensitivity with the highest purity.
The melody system may be determined based on the second highest estimated sensitivity. In the fifth embodiment and the first embodiment, the display mode of the person image or the background image is changed according to the purity, but instead of the purity, the display of the person image or the background image is performed according to the level of the degree of cooperation. The mode may be changed. Furthermore, in the first to third embodiments, the music is changed according to the purity,
The rhythm system is determined simply by subject A's sensitivity, and subject B
The melody system may be determined according to the sensitivity of. Sixth Embodiment Furthermore, in the first embodiment, there are only two test subjects A and B, but the test subjects may be three or more. In that case, it is the same as the above point that the camera and the sensitivity estimation program are prepared for each subject, and the sensitivity vector is estimated, the maximum dominant sensitivity is determined, and the purity is calculated for each subject. There are several possible methods.

For example, the degree of cooperation may be calculated for any pair of a plurality of subjects according to the equation 6, and the worst degree of cooperation among the calculated plurality of degrees of cooperation may be the overall degree of cooperation at this time. In this way, the game does not reach the goal state as long as there is a pair whose degree of cooperation is lower than the threshold, and the goal is reached only when the degree of cooperation of all pairs is higher than the threshold.

Alternatively, for example, the degree of cooperation may be calculated for each pair according to the equation 6, and the sum of the calculated degrees of cooperation (overall degree of cooperation) may be compared with a threshold value. By doing this, even if there is a pair whose coordination is worse than the threshold, if the remaining pairs are good, another pair will help each other to reach the goal. .

Further, the degree of cooperation is calculated for each pair according to the equation 6, and when the pair having the best degree of cooperation among the calculated degrees of cooperation continues to be in a cooperative state for a certain period of time, the best couple is calculated. It may be a game of reaching a goal as a decision.

In the above modification, it was assumed that the participants did not form a pair in advance, but participated in one person each. However, they formed a pair in advance and participated in each pair. You may make it compete for superiority or inferiority of the degree of cooperation. Further, in any of the modified examples, after the degree of cooperation is better than the threshold value, for example, video content and music content that match the sensitivity of the pair with the best degree of cooperation may be output. Seventh Example In the first to sixth examples, the case where each subject is in the same studio has been described, but each subject does not necessarily have to participate in the same studio, and the number of participants In some cases, studios in several locations may be used. In this case, a camera is installed for each subject in each studio, and a large-screen monitor and a speaker are installed in front of each studio. There is only one computer, this computer and the camera of each studio,
The large screen monitor and the speaker are connected by a cable for transmitting video / music signals. By doing this, each subject will get 1
You can enjoy the game as if you were in one big studio. Eighth Example Each subject does not necessarily have to dance in a plurality of studios and may use a computer environment connected by a network. At this time, each test subject dances in front of his / her own personal computer equipped with a camera, and the dance image is sent to the center computer via the Internet. The computer estimates the sensitivity of each subject and calculates the degree of cooperation of any pair,
The video contents and music contents corresponding to the sensibilities of the best couple are transmitted to the personal computer of each subject. Further, if the video content and the music content are output to a large-screen monitor and a speaker installed in the center, the test subjects and viewers (spectators) participating in the center can enjoy them at the same time.

As can be seen from the above description, according to these embodiments, the multimedia expression is changed based on the sensitivity of each subject, so that each subject can enjoy the emotional expression at the same time. In addition, since it is configured to evaluate the sensitivity of each subject and the degree of affective cooperation, it is possible to output a multimedia expression according to the sensitivity of the subject with high purity and a multimedia expression indicating the cooperation state of each subject. The output can be game-like. Therefore, this embodiment may be installed in the game center as one game facility, or may be installed as one attraction facility in a place where people gather, such as an event venue. Also, using a computer network and a large screen display placed in various places,
It is also possible to hold a big event, which is a sensational cooperative dance competition, on a global scale.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is an illustrative view showing a part of the operation of the embodiment in FIG.

FIG. 3 is an illustrative view showing an image displayed on a monitor when a subject's degree of affective cooperation is poor.

FIG. 4 is an illustrative view showing an image displayed on a monitor when the degree of emotional cooperation of the subject is good.

FIG. 5 is an illustrative view showing an image displayed on a monitor when a goal is reached.

FIG. 6 is a flowchart showing the operation of the CPU when estimating the subject A's sensitivity.

FIG. 7 is a flowchart showing the operation of the CPU when estimating the sensitivity of subject B.

FIG. 8: CPU when changing multimedia expression
6 is a flowchart showing a part of the operation of FIG.

FIG. 9: CPU when changing multimedia expression
6 is a flowchart showing another part of the operation of FIG.

FIG. 10: CP when changing multimedia expression
It is a flowchart which shows a part of other operation | movement of U.

FIG. 11: CP when changing multimedia expression
It is a flowchart which shows a further another part of operation | movement of U.

[Explanation of symbols]

10 ... Collaborative Kansei expression system 12a, 12b ... cameras 14 ... Computer 16 ... Database 18 ... Monitor 20 ... speaker

Front Page Continuation (72) Inventor Ikuo Harada 2-2, Hikaridai, Seika-cho, Soraku-gun, Kyoto Prefecture 2nd, ATR Intelligent Video Communications Laboratories (72) Ryotaro Suzuki, Hikaru-cho, Seiraku-cho, Kyoto Prefecture 2-chome, Taiwan 2-Intelligent Video Communications Laboratory, Inc. (72) Naoto Okegawa, 2-chome, 2-chome, Hikaridai, Seika-cho, Soraku-gun, Kyoto Prefecture (72) Mao Makino 2-2 Kodai, Seika-cho, Soraku-gun, Kyoto Prefecture 2 TIR Intelligent Video Communications Laboratories, Inc. (72) Masayuki Inoue 2-chome, 2-chome, Seika-cho, Soraku-gun, Kyoto Prefecture Address 2 Intelligent Video Communications Laboratory, Inc. (56) References Iwadate, Research on Image Representation, IEICE Technical Report (HCS99-40-53), October 21, 1999, Vol. 99, No. 384, pp. 87-94 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T ⁷ /00-7/60 A63B 69/00 G06T 1/00 H04N 7/18 JISST file (JOIS)

Claims (13)

(57) [Claims] 1. Estimating means for estimating a sensitivity vector for each subject based on physical quantities of body expressions of a plurality of subjects, based on a plurality of sensitivity vectors obtained from the estimating means
A kansei expression system comprising: a cooperation degree calculating means for calculating a cooperation degree according to the above; and a changing means for changing the multimedia expression based on the cooperation degree calculated by the cooperation degree calculating means. 2. The first changing means for dividing and displaying a plurality of images corresponding to the number of the subjects on the screen when the degree of cooperation is worse than a threshold value, and whether the degree of cooperation is better than the threshold value. The emotional expression system according to claim 1, further comprising: second display means for displaying one image on the entire screen when it is equal to a threshold value. 3. The emotional expression according to claim 2, wherein the second display unit displays a specific image showing space sharing when the degree of cooperation is better than the threshold value or equal to the threshold value continues for a predetermined time. system. 4. The first changing means outputs the music corresponding to the sensitivity of any one of the subjects when the degree of cooperation is worse than a threshold value, and the changing means cooperates when the degree of cooperation is better than or equal to the threshold value. 4. The kansei expression system according to claim 1, further comprising a second output unit that outputs the music corresponding to the sensibility of the state. 5. A pre-Symbol estimating means estimates the primary sensitivity of the subject, further comprising the purity calculation means for calculating a purity of major sensitivity, sensitive expression system according to any one of claims 1 to 3. 6. The purity calculating means is an absolute value of the main sensitivity.
There is calculated the percentage of the sum of the double several sensitive as the purity, sensibility expression system of claim 5, wherein. 7. The emotional expression system according to claim 5, wherein the changing means includes a third display means for displaying a person image showing the subject in a manner according to the purity. 8. The emotional expression system according to claim 7, wherein the third display means displays either one of a photographed image of the subject and a contour image on which a special effect is applied according to the purity. 9. The affective expression according to claim 5, wherein the changing means includes third output means for outputting music corresponding to the main affection of any one of the subjects based on the purity. system. 10. The emotion expression system according to claim 9, wherein the third output means outputs the music corresponding to the main emotion having the highest purity. 11. The third output means has the purity of M.
The sensation expression system according to claim 9, wherein a rhythm system is determined by the (M: integer) second highest sensitivity, and a melody system is determined by the N (N: integer) highest purity. 12. Before SL cooperative calculating means, closeness of the mean values of the elements forming the similarity, each of said sensitive vector of the plurality of normalized sensitivity vector the sensitivity vector normalized subjects each , And the affective expression system according to any one of claims 1 to 11, wherein the degree of cooperation is calculated based on the closeness of the variance value of the elements forming each of the affective vectors. 13. A kansei expression method of a kansei expression system for outputting a multimedia expression corresponding to a subject's sensibility.
Then , the affective vector is estimated for each of the subjects based on the physical quantities of the body expressions of the plurality of subjects, the degree of cooperation is calculated based on the estimated plurality of affective vectors , and the calculated cooperation is calculated. A Kansei expression method for changing the multimedia expression based on the degree.