JP2008116551A - Expression attaching voice generator - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an expression attaching voice generator capable of generating voice with intensity which is visually suitable for the intensity of expression attaching of body motion. <P>SOLUTION: The expression attaching voice generator 10 includes a computer 22. Based on sensor values from sensors 161a to 164, 181 and 182 provided in a glove type sensor 14 which is mounted on a hand for operating a hand doll 12, the computer determines the intensity of expression attaching of body motion of the hand doll by referring to an interpretation table 24, and a morphing factor is determined by referring to a body motion-song voice expression table 28, so that it may become expression attaching of song voice of the intensity suitable for the intensity of expression attaching of the body motion. According to the morphing factor, morphing is performed on original song voice which is stored in a song voice data base 26 beforehand, and sound is output from a speaker 34. According to the invention, the song voice with the intensity which is visually suitable for the intensity of expression attaching of the body motion can be generated. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a facial expression voice generation device, and more particularly to a facial expression voice generation device that outputs voice expressed using a voice morphing technique (Expressive Voice).

  As described in Non-Patent Document 1, as a research on emotional speech (Emotional Speech) in conventional speech expression with emotion, a rule-based approach such as F0 (fundamental frequency) and speech speed, and Non-Patent Document 2 A corpus-based approach can be considered.

  The rule base mainly handles prosodic information, while the corpus-based method can handle voice tone even for facial expression of singing voices with constant prosodic information. The discontinuity of expression is conspicuous.

  In addition, the present inventors have developed an expression technique that can continuously change the intensity of facial expression by performing speech morphing using STRAIGHT (speech analysis conversion synthesis system) known in Non-Patent Documents 3 and 4, etc. ESVM (Expressive Singing Voice Morphing) has been proposed (Non-Patent Document 5).

  The ESVM shown in Non-Patent Document 5 enables natural expression and is expected to be used in various directions.

The inventors have used the ESVM voice morphing technique to output an expressive voice that can be expressed in response to a hand-puppet's body movement (gesture). (Non-Patent Document 6).
Schroder, M., “Emotional Speech Synthesis: A Review,” Proc. Eurospeech, volume 1, pp. 561-564, 2001 Iida, A., Iga, S., Higuchi, F., Campbell, N., Yasumura, M., “A Speech Synthesis System with Emotion for Assisting Communication”, Proc. ISCA Workshop on Speech and Emotion, pp. 167- 172, 2000 Kawahara, H., Masuda-Kasuse, L, and Cheveigne, A., “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and aninstantaneous-frequency-based FO extraction: Possible role of a reptitive structure in sounds,” Speech Communication, 27, pp. 187-207, 1999 http://www.wakayama-u.ac.jp/~kawahra.STRAIGHTadv/ (High Quality Speech Analysis Conversion Synthesis System STRAIGHT) Yonezawa Atsuko, Suzuki Noriko, Mase Kenji, Kogure Kiyoshi, “Perceptual examination of singing voice morphing with facial expression,” Acoustical Society of Japan Spring Conference (Sound lecture), pp. 809-810, 2004 Yoko Yonezawa, Noriko Suzuki, Kenji Mase, Kiyoshi Kogure, “Continuous facial expression system for singing voice using anthropomorphic gesture expression,” The Acoustical Society of Japan, Vol.62, No.3, pp.233-243, 2006.

  In Non-Patent Document 6, it is more natural and desirable that the intensity of the expression of the body movement (gesture) and the intensity of the expression of the voice are visually matched.

  Therefore, a main object of the present invention is to provide a novel facial expression sound generator.

  Another object of the present invention is to provide an expression voice generating device capable of generating a voice that is expressed with an intensity that visually matches the intensity of the expression of body movement.

  The invention according to claim 1 is a facial expression voice generating device that generates a voice with a facial expression according to a facial expression of a physical motion input by a physical motion input means for inputting a physical motion, and has at least different facial expressions. Audio signal database for storing respective audio signals of two sounds in advance, setting means for setting the intensity of voice expression that matches the intensity of expression of body motion, and two or more audio signals read from the audio database A morphing means for morphing at a morphing rate according to the intensity of expression of the voice set by the setting means, and a voice output means for outputting a sound by a voice signal resulting from morphing by the morphing means. is there.

  In the first aspect of the invention, a computer (22) is used, and an audio signal database (26) is set in the computer (22). In the audio signal database (26), for example, audio signals with no expression and audio signals with different expression “da”, “wh”, and “we” are recorded in advance. . In the computer (22), for example, based on a sensor signal from a sensor (161-182) provided in a glove (14) attached to a hand that operates the hand puppet (12), for example, a current gesture of the hand puppet (body) The morphing means, which is also the computer (22) or other circuit, morphs the original singing voice in the speech signal database at a morphing rate according to the facial expression intensity of the body motion.

  According to the first aspect of the present invention, the sound is generated with the morphing rate according to the expression intensity of the body motion, so that it is possible to generate the sound with the expression strength well suited to the expression strength of the body motion.

  The invention according to claim 2 is the expression according to claim 1, wherein the morphing means morphs two or more speech signals according to the inverse sigmoid function, and the setting means sets the intensity of voice expression according to the sigmoid function or its deformation function. It is an attached sound generator.

  As a sigmoid function used in the invention of claim 2 or a deformation function thereof, a function such as the following equations (2) to (6) can be considered.

  According to the present invention, when generating a voice with a facial expression according to the body motion, the morphing is performed at a morphing rate according to the facial expression intensity of the physical motion, so that the strength visually matches the facial expression strength of the physical motion. You can output voices with facial expressions.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  The expression voice generating apparatus 10 (FIG. 6) according to an embodiment of the present invention controls expression of a singing voice (singing voice expression), for example, using the above-described ESVM morphing. It is intended to use a gesture (posture) for intensity control. However, since the ESVM is described in detail in a co-pending patent application (Japanese Patent Laid-Open No. 2006-178052), the description is referred to as much as possible.

  Then, as an example of means for inputting such gesture, that is, a posture expression intensity, a hand puppet operated by hand is used, and the hand puppet includes the stuffed toy 12 shown in FIG. The stuffed toy 12 as a whole is formed of a flexible material such as cloth or felt, and communicates with the palm 120 into which the palm of the subject enters, the palm 120 of the subject, and the subject's thumb, index finger, and middle finger. Includes a thumb part 121, an index finger part 122, and a middle finger part 123 into which each can be inserted. As illustrated, the stuffed toy 12 of the embodiment represents a bird in which the index finger 122 is a head, and the thumb 121 and the middle finger 123 sandwiching it are wings or wings. However, naturally, the shape of the stuffed toy 12 can be arbitrarily changed.

  Thus, in order to use the stuffed toy 12 to input the facial expression intensity for controlling the expression of the singing voice, it is important to incorporate an appropriate personification expression. In the embodiment, two arms (wings) are used. The operation of the stuffed toy 12 having the head and the head is controlled by three fingers. Although it is conceivable to control the utterance timing by the movement of the mouth, in this embodiment, the “expression” is controlled by the whole body gesture of the stuffed toy 12.

  In order to utilize the appearance of the stuffed animal 12 and use it as an input device, it is important to measure the movement of the hand as the movement of the stuffed animal 12. In order to obtain motion data with sufficient accuracy as a controller for singing voice expression, the hand puppet includes a stuffed animal 12 as an anthropomorphic cover and an independent glove sensor 14 that measures hand movement.

  That is, the subject's hand is inserted into the stuffed toy 12 as shown in FIG. 2, and a glove-type sensor 14 is attached to the hand. The glove-type sensor 14 includes a palm portion 140 that receives a palm, a thumb portion 141 that is in communication with the palm portion 140 and into which the thumb, index finger, and middle finger are inserted, an index finger portion 142, and a middle finger, respectively. A portion 143 is formed. However, the finger parts for the ring finger and little finger are naturally formed, but are not mentioned here.

  Referring to FIGS. 3 and 4, glove-type sensor 14 includes the above-described palm portion 140 and finger portions 141-143. The thumb first bending sensor 161a is provided on the surface of the thumb 141 with a length that can cover at least the first joint and the second joint of the thumb 141. A thumb second bending sensor 161b is provided on the side surface of the thumb portion 141 with a length that can cover the first joint and the second joint of the thumb portion 141. The index finger first bending sensor 162 a is provided on the surface of the index finger 142 with a length that can cover at least the first joint and the second joint of the index finger 142, and the index finger second bending sensor 162 b is similarly provided on the side of the index finger 142. It is provided with a length that can cover the first joint and the second joint. Further, a middle finger first bending sensor 163a is provided on the surface of the middle finger portion 143 in such a length that can cover at least the first joint and the second joint of the middle finger portion 143, and the middle finger portion 143 is opposite to the thumb portion and the index finger portion. The middle thumb second bending sensor 163b is provided on the side surface of the middle finger portion 143 in such a length that can cover at least the first joint and the second joint. The reason why the middle finger second bending sensor 163b is opposite to the thumb second bending sensor 161b and the index finger second bending sensor 162b is to avoid interference between the index finger 142 and the middle finger second bending sensor 163b. If there is little or no interference, it may be provided on the same side as the other second bending sensors 161b and 162b.

  As shown in FIG. 5, the first thumb bending sensor 161 a and the second thumb bending sensor 161 b described above are arranged such that the former is arranged on the surface (back side of the hand) of the thumb portion 141 and the latter is placed on the side of the thumb portion 141. Are provided with an angle difference of 90 degrees. As a result, the bending angles in the two directions of the X direction and the Y direction having an angle difference of 90 degrees can be measured. The index finger first bending sensor 162a and the index finger second bending sensor 162b, and the middle finger first bending sensor 163a and the middle finger second bending sensor 163b are also in a positional relationship shifted by 90 degrees for the same reason.

  Further, each of the bending sensors 161a, 161b, 162a, 162b, 163a, and 163b is a piezo (piezoelectric) element, and outputs a different voltage depending on a bending angle in a direction perpendicular to the main surface. Therefore, by detecting this voltage, it is possible to detect or measure the bending angle of each bending sensor, that is, the finger portion in the direction.

  In addition, as shown in FIG. 4, pressure sensors 181 and 182 are provided at the fingertips of the thumb part 141 and the middle finger part 143 of the glove-type sensor 14 on the ventral side of the finger. The pressure sensors 181 and 182 are also piezoelectric elements, and output a voltage having a magnitude corresponding to the magnitude of pressure applied to the surface thereof. The two pressure sensors 181 and 182 are for detecting the state in which the tip of the thumb portion 141 and the tip of the middle finger portion 143 are aligned with each other.

  In the embodiment, the index finger 142 (the head of the hand doll) can measure not only the bending but also the warping, and the head 122 of the stuffed toy 12 with the index finger bent to the palm side to some extent in advance. The structure is facing the front. And the other bending sensor 164 shown with a dotted line in FIG. 3 is provided in the back side inner surface of the palm part 140 of the glove-type sensor 14. The wrist bending sensor 164 is also a piezo element, and detects the degree of bending of the index finger 142, that is, the degree of bending toward the back of the hand.

However, the wrist bending sensor 164 may be provided with another glove (not shown) in the glove-type sensor 14 and attached to the back of the hand (surface) of the glove.

  In addition, the longitudinal bending sensors 161a, 161b, 162a, 162b, 163a, 163b and 164 (hereinafter sometimes referred to as “161a-164”) are all included in the glove-type sensor 14 (and medium gloves). Although it adheres, the method of adhering gently with a thread | yarn is suitable for the adhesion method. If it is sewn or bonded firmly, the bending sensor will be pulled when the bending angle of the finger part of the glove, especially the palm side, is large, and the bending sensor will follow the bending of the finger part of the glove well. This is because a failure such as being impossible or being damaged occurs.

  Output voltages from the bending sensors 161a-164 and the pressure sensors 181 and 182 described above are converted into digital data by the A / D converter 20 and input to the computer 22 as shown in FIG. The computer 22 detects the hand and finger movements as gestures of the stuffed toy 12 based on the voltages from the sensors 161a-164, 181 and 182, that is, the facial expression intensity of the posture, and has a morphing rate according to the facial expression strength of the posture. Perform voice expression with voice morphing.

  An interpretation table 24 is preset in a memory (not shown) of the computer 22 in order to interpret the voltage values from the sensors 161 a-164, 181 and 182 as facial expression intensity.

  The voltage values from the sensors 161a-164, 181 and 182 are not directly proportional to the bending angle, and as shown in FIG. 7, the change is large when the bending angle is small, and changes as the bending angle increases. Each voltage value changes like a kind of saturation curve. Accordingly, if the voltage value is interpreted as the degree of operation (bending) as it is, it will be interpreted incorrectly.

  Therefore, in the interpretation table 24, a conversion table or a conversion equation is set such that the bending angle and the voltage value linearly change the change curve as shown in FIG. Therefore, the computer 22 converts the sensor value (voltage value) using the interpretation table 24, estimates each bending angle and pressure from the converted voltage value (sensor value), and thereby identifies or identifies the facial expression intensity of the posture. To do.

  FIG. 7 shows the relationship between the sensor value and the angle of one bending sensor, and the horizontal axis indicates “1.0”. The position is a position where the bending angle is 100%, and this is used as a reference. The degree of bending angle (%) can be identified. Like the degree of the bending angle, the degree of gesture is a% value less than that when the maximum change due to the gesture is 100%.

  Table 1 shows the relationship between the sensor value actually output from each sensor 161a-182 of the glove 14 with sensor, the bending angle of each finger part and wrist, and the contact pressure between the thumb part and the middle finger part. In Table 1, in the example, each sensor value is represented by 128 resolutions (0-127). For example, in the case of the thumb first bending sensor, a voltage was output when it was bent in the palm direction, and the maximum value was 110 and the minimum value was 60. However, the sensor value when the movement is largest is the minimum value. If attached to the wrist bending sensor, when the wrist is bent toward the back of the hand, a sensor value that changes from 20-30 to 50-60 is output. In the pressure sensor, the maximum value is 127 and the minimum value is less than 127, so that the pressure sensor is a touch sensor and detects contact between the thumb part and the middle finger part.

The computer 22 interprets the facial expression strength (gesture) of the posture of the stuffed animal 12 with reference to the interpretation table 24 based on the input sensor value of Table 1. In this embodiment, as shown in Table 1 and FIGS. 8 to 11, there are three characteristic left-right symmetric positions in the range of motion without hand (neu) and the range of hand operation so that expression is easily perceived. Take up posture (bak, drp, str).

  FIG. 8 shows a state of “calm (no expression)”, and FIG. 9 shows an operation called “back (bak)” in which the forefinger (head) 122, thumb 121, and middle finger 123 are moved to the back of the hand. It is an expression that deflects to the side. FIG. 10 shows a state in which the head 122 faces downward in an operation called “droop: drp”. FIG. 11 shows a posture in which the thumb part 121 and the middle finger part 123 are projected forward by an operation called “stretch: str”.

  Voice morphing using STRAIGHT (Non-Patent Document 3) was applied to obtain singing voices with different intensities continuously, and a morphing singing voice with expression was created.

  First, as a voice of a specific singer, an amateur 20's singing voice suitable for acquiring expressions differing only in voice color was recorded as a morphing singing voice at a sampling frequency of 44.1 kHz. “Da”, “wh”, “we” as examples of facial expression with independent articulation elements such as “no” with consistent expression during singing (“flat” singing voice without expression), oral cavity, nasal cavity, exhalation A total of four types (Table 2) were recorded. Adopting two measures of “Kubana Tsurushi”, which has a better balance of phonemes and rhythms than the traditional song “Furusato” in Japan, and the same accompaniment (C major scale, The speed was 3/4 time and 120 beats / minute). The speaking speed of the recorded singing voice was about 2.0 mora / second, and the Fo range was about 300 Hz to 450 Hz on average. The average length of each singing voice is about 3.0 seconds. Such an original singing voice is recorded in the singing voice database 28.

  Using this data, in order to create singing voices with different facial expression intensities, there is a total of 3 voices with no expression and singing voice with expression ({no · da, no → wh, no ← → we}). Voice morphing was performed on the “Type”. At this time, the morphing rate indicates the ratio of the feature amount when the original voice without expression “no” is 0 and the original voice with expression is 1. Assume that the expression intensity is adjusted by the morphing rate, and an example of the adjustment of the morphing rate is shown in FIG.

  In the embodiment, four types of original voices (original singing voices) “no”, “da”, “wh”, “we” shown in Table 1 are registered in advance in a singing voice database 28 in a memory (not shown), The expression of the singing voice synthesized (morphed) is performed.

Specifically, a method for determining a morphing rate when performing voice morphing between three types of voice will be described with reference to FIG. Now, it is assumed that morphing is performed between three kinds of voices A, B, and C. Consider a triangle with vertices 100, 102, and 104 corresponding to these three sounds, as shown in FIG.

  The mesh 110 in FIG. 12 can be created by dividing each side of this triangle into a predetermined number and connecting the dividing points with a line parallel to each side. Morphing speech corresponding to each point constituting the mesh 110 can be created as follows.

For example, the intermediate speech corresponding to each division point between speech A and speech B determines the morphing rate using two sigmoid functions so that the two speeches change at a constant rate. The morphing rate at this time is determined by the body motion-singing voice expression table 26. In the same manner, morphing between the voice A and the voice C and between the voice B and the voice C can be performed, respectively. Further, the intermediate sound at each intersection (for example, intersection 112) of the mesh 110 is obtained by converting the intermediate sound at both ends (for example, points 114 and 116) of an arbitrary line passing through the intersection to the ratio of the distance from the both ends to the intersection. It can be created by morphing at a corresponding morphing rate. Accordingly, it is possible to create all intermediate-level sounds corresponding to the respective points on the mesh 110.

  In this way, this method is not limited to the case where the original voices are the three types shown in FIG. 12, but the original voices are the four types “no”, “da”, “wh”, “ Even if there is “we” (“normal”, “dark”, “whisper”, “wet”) or more, it can be similarly applied by repeating the determination of the morphing rate between two voices.

  The determination of such a morphing rate using the sigmoid function described above is detailed in Japanese Patent Application Laid-Open No. 2006-178052 mentioned above.

  When examining cross modalities (cross modalities), it is desirable that the combination of the basic representation types be naturally accepted. Assuming that a combination of common and characteristic metaphors is desirable as a mapping of singing voice expressions to body movement expressions, the metaphors (metaphors or metaphors) shown in Table 3 are typical examples of situations in which singing voices are uttered. Was preset. Based on this metaphor, we combined singing voice and posture expression types.

In the computer 22, a physical motion (gesture) -singing voice expression table 28 is preset in a memory (not shown). The body motion-singing voice expression table 28 is a table for mapping the interpreted gesture, that is, the expression intensity of the posture, to the expression of the singing voice as described later. That is, the body motion / singing voice expression table 28 determines at what morphing rate the four original singing voices are morphed according to the expression intensity of the posture of the stuffed toy 12. In this embodiment, a sigmoid function f (x) and its deformation shown in FIG. 26, which will be described later, are set in this table 28, and the gesture interpreted according to the interpretation table 24 and its intensity, that is, the facial expression intensity of the body movement. Determine the corresponding morphing rate.

  In the embodiment shown in FIG. 6, in order to perform morphing at the morphing rate determined based on the body motion-singing voice expression table 28, the speech synthesizer 30 that performs speech morphing using STRAIGHT described above is provided. . The speech synthesizer 30 may be formed as a dedicated circuit (for example, ASIC) separate from the computer 22 in hardware, and may be executed as a function of the computer 22 if the computer 22 has sufficient capability. Good. In this speech synthesizer 30, at least two (four in the embodiment) original voices (original singing voices) registered or stored in advance in the singing voice database 28 are morphed at a morphing rate according to FIG.

  In the embodiment of FIG. 6, the computer 22 reads the voltage (sensor value) output from each of the sensors 161a-164, 181 and 182 from the A / D converter 20 in the first step S11 of FIG.

  Then, in step S13 in FIG. 13, the computer 22 refers to the interpretation table 24 and identifies the gesture intensity of the stuffed animal 12 at that time, that is, the facial expression intensity of the body movement, as shown in Table 1 above.

  Thereafter, in step S15, the computer 22 refers to the body motion-singing voice expression table 28 shown in FIG. 6 and based on the facial expression strength of the posture (body motion) as described above with reference to FIG. Determine the morphing rate.

  In step S17, the computer 22 or the speech synthesizer 30 morphs each original singing voice (voice) signal read from the singing voice database 26 according to the morphing rate determined in step S15.

  In step S19, the computer 22 outputs a volume setting signal corresponding to the warping (the sensor value of the wrist bending sensor) so that the morphing sound is output from the speaker 34 at a volume corresponding to the degree of the warping gesture. The A converter 32 is given.

  In this way, the computer 22 changes, for example, four types of morphing source voice “no” by changing the gesture of the stuffed toy 12, that is, the facial expression of the stuffed animal 12, by the subject or the user with his / her hand (wearing the glove-type sensor 14). , “Da”, “wh”, “we” can be morphed (speech synthesized) to generate morphed speech.

  When examining the cross-modality to which the present invention is directed, it is desirable that the combination of the basic expression types is naturally accepted as described above. Therefore, the inventors pay attention to the cross-modality characteristic of the expression intensity in the anthropomorphic expression.

In the cross modality with anthropomorphic expressions, we assume that there is an appropriate combination with high facial expression strength and suitability,
1. As a basic characteristic, the relationship between the expressed facial expression intensity and the actually perceived intensity-Perceptual interpolation tendency of facial expression intensity of body movement-Perceptual interpolation tendency of facial expression intensity of compound expression of singing voice and physical movement
2. Perceptual evaluation of cross modalities with different facial expression intensities—perception of facial expression intensity—perception of suitability was evaluated using the system 10 of FIG. Experiment 1 confirms the basic perception characteristics of facial expression strength, Experiment 2 conducts perception experiments of suitability and facial expression strength in combined expressions with different facial expression strengths between modalities, and Experiment 3 performs facial expression strength over time. It was confirmed that the same result as in Experiment 2 could be obtained even when the temperature changed.

In the evaluation experiment, it is necessary to ensure reproducibility when showing the facial expression strength of the posture among all the body movements using the stuffed toy. However, when moving the stuffed animal using an internal mechanism such as a robot, it is difficult to ensure complete reproducibility with the current control technology. Therefore, we introduced a perceptual evaluation by audiovisual presentation using a video showing the physical movement of a hand puppet by a performer who has no problem with hand movement ability.
Experiment 1 (Perceptual interpolation effect of facial expression intensity )
It has already been confirmed that the interpolated facial expression intensity is perceptually interpolated for voices with different expression levels of singing voices. Experiment 1 is an experiment conducted to confirm that perceptual interpolation is performed as a basic perceptual characteristic even in the expression of the body motion in the single expression / complex expression of the body motion and singing voice.

In conducting Experiment 1, the following hypotheses I and II were made.
I Stepwise facial expression intensity of body movement (posture) is perceptually interpolated.
In the combined expression of II singing voice and body movement, the expression intensity is perceptually interpolated.

Subjects: 20 men (20s-30s), 11 men and 9 women
Experimental environment: Subjects wore headphones in a soundproof room and executed the experimental program in front of a large display. The experiment program is written in Tcl / TK (trade name), and accepts the input of the subject's perceptual evaluation and displays the video by QuickTimeTcl 3.1 (trade name). The video image of 320 × 240 pixels was displayed on a large display so that the stuffed toy of the video image was projected in full size.
Experimental method:
Experiment 1-I: Shows a stuffed toy with different facial expression intensity. Listen to a silent video (g), add 3 types of facial expression {bak (Experiment 1-Ia), drp (Experiment 1-Ib), str (Experiment For 1-Ic)}, the facial expression strength is evaluated in 7 stages.
Experiment 1-II: Watch a video v + g with the expression voice singing voice (v) corresponding to the posture added to g, and add 3 types of expression {da: bak (A, Experiment 1-II-a ), Wh: drp (B, Experiment 1-II-b), we: str (C, Experiment 1-II-c)} (Table 3). The facial expression strength is subjectively evaluated according to the 7 levels shown in Table 4 below.
[Table 4]
7: Very applicable 6: Very applicable 5: A little applicable 4: Neither applicable 3: Not applicable 2: Almost not applicable 1: Not applicable at all 1 And show the subject a video without facial expression (see below for teaching).

  Experimental conditions (stimulus video): Create a video with a combination of various kinds and strengths of stuffed animal posture and singing voice expression. When it was necessary to state the name of the expression during the evaluation experiment, the label of {A, B, C} based on Table 3 was used so as not to give a specific impression such as an adjective. In the combination of the sound of the stimulating video and the moving image, the subject is given an impression that the sound and the moving image are simultaneously reproduced by combining a moving image having a length of 0.2 seconds before and after the sound of about 3 seconds. did.

In order to evaluate the continuity of perceptual interpolation, a stimulus video was created with 7 levels of facial expression intensity.
g: For each facial expression, a video was created by using a static posture image with seven levels of facial expression intensity with 0 as neu and 1 as a posture with a facial expression {bak, drp, str} (7 levels x 3 types of facial expression = 21 Type, no sound). -Experiment 1-I
v + g: A singing voice having 7 levels of expression intensity was combined with a still image having an expression intensity corresponding to g (7 × 3 types (A to C) = 21 types). -Experiment 1-II
The image of the posture corresponds to 7 steps {0, 0.17, 0.33, 0.5, 0.67, 0.83, 1} by calculating (interpreting) the expression intensity from the sensor value recording the movement. The image was cut out from a moving image in which the movement of the stuffed animal 12 was recorded. Further, the singing voice part is created by using the above-described morphing, and the singing voice with seven expression levels (morphing rate {0, 0.17, 0.33, 0.5 , 0.67, 0.83, 1}).

Instructions: As an example, in Experiment 1-Ia
1. Before the experiment, watch the stimulation video g neu in advance and confirm that it is flat, then watch the stimulation video g bak, confirm that it is a facial expression A, and use it as an evaluation standard.
2. I watched an experimental video and taught me how to evaluate the degree of expression of the stuffed toy 12's posture in 7 steps. Experiment 1-Ib, c and Experiment 1-II also taught starting the experiment after viewing the reference stimulus video.

Experimental results: The average values and standard deviations obtained by collecting the experimental results for each facial expression intensity are shown in FIGS. 14, 15 and 16. FIG. Further, Table ² shows R ² values in the linear approximation.

According to the table 5, g is drawing a graph soaring, it confirmed that from that higher R ² value, hypotheses I and II are satisfied.

In order to investigate the difference between modality and facial expression intensity and their interaction, the R ² value is generally higher for v + g compared to g, 2 (g, v + g) * 7 (Facial expression intensity) = 14 Conditional two-way analysis of variance (significance level α = 0.05, degree of freedom φ = 1, 6, 38) was performed. Table 6 shows the results of the test. According to this, the difference due to facial expression strength is F = 3.46, p =. No significant difference was observed due to the number of modalities {g, v + g} or interaction.

Experiment 2 (Cross modality perception effect by changing facial expression strength)
In Experiment 2, when the expression strengths of the singing voice and the posture are different from each other in the anthropomorphic expression by the stuffed toy 12, whether or not there is an interaction between the singing voice and the posture in the perception of the suitability and the expression strength. In conducting Experiment 2, the following hypotheses I and II were made.
I In the singing voice and posture, it is perceived as being compatible with the expression of strength taught in association with each other
II The perceived facial expression intensity is the average of singing voice and posture facial expression intensity.
Subject: Subject experiment environment identical to Experiment 1: Environment experiment method identical to Experiment 1:
Experiment 2-I: 4 stages of singing voices × 4 stages of posture = A total of 16 types of videos are viewed in random order, and a 7-level evaluation is performed for suitability. This is performed for each combination of A (referred to as Experiment 2-Ia), B (Experiment 2-Ib), and C (Experiment 2-Ic). Before the experiment, both the facial expressionless video and the video with the facial expression are shown as the evaluation standard videos as in Experiment 1-II.

  Experiment 2-II: After the evaluation of Experiment 2-I above, set the evaluation item to facial expression strength and obtain a seven-level evaluation. This is performed for each combination of A (referred to as Experiment 2-II-a), B (Experiment 2-II-b), and C (Experiment 2-II-c).

  Experimental conditions (stimulus video): Considering the minimum number of stages required when examining the interaction between the facial expression intensities of two elements, the expression intensity for each modality of singing voice and posture is given in four stages and combined for each intensity. . 4 levels of singing voices x 4 levels of posture, a total of 16 types of facial expressions, with a total of 16 different facial expressions for 3 types of facial expressions A (da: bak), B (wh: drp), and C (we: str) Prepared (FIG. 17). This experiment 2 was performed by setting the lower limit and the upper limit of the expression intensity in the movable range of body movement and the recorded voice of the amateur singer as 0.0 and 1.0, respectively.

  For the singing voice part, the singing voice with morphing rate {0, 0.33, 0.67, 1} was prepared using the singing voice synthesis (morphing) in which the facial expression intensity was explained in steps. . For the posture image, facial expression intensity was determined from the sensor value recording the movement, and images corresponding to the four steps {0, 0.33, 0.67, 1} were cut out from the moving image.

Teaching contents:
In Experiment 2-Ia and Experiment 2-II-a, the following teaching was performed.
1. Before the experiment, watch the singing voice and the video of the stuffed toy 12 (no: neu, no facial expression) and confirm that it is flat, then watch (da: bak, facial expression A) and make the facial expression A. To check the
2. Watch the experiment video and evaluate the posture of the stuffed toy 12 according to the following 7 items shown in Table 4.
I. Singing voice with expression-compatibility between postures (Experiment 2-Ia)
II. Degree of facial expression being A (Experiment 2-II-a)
Similarly, in Experiments b and c, it was taught to start the experiment after watching the video as the evaluation standard.

  Conformity was “whether the combination of expression between singing voice and posture is natural”.

  Experimental result I [Perception of suitability]: Average values of perceptual evaluation in Experiments 2-Iac are shown in FIGS. In these FIGS. 18-20, the same applies to FIGS. 21-23 described later, but the vertical axis indicates evaluation (MOS of Appropriateness), and the left side of the horizontal axis indicates the strength of expression of the singing voice (Strength of Singing Voice Expression). ), And the right side shows the strength of expression of body movement (gesture). According to this, when the facial expression strength of the singing voice is {0, 0.33}, the compatibility becomes low when the facial expression strength of the posture is strong, and conversely when the facial expression strength of the singing voice is {0.67, 1}. It can be seen that the adaptability increases as the facial expression intensity increases. Although it is considered that the evaluation value of suitability has changed due to the action of the cross modality, the suitability is not necessarily high at the strength corresponding to the singing voice and the posture. For example, the facial expression strength of the singing voice is 0.67, whereas the posture strength determined to be the highest suitability is 1. That is, hypothesis I does not always hold.

  In response to these results, the null hypothesis is that “there is no difference due to facial expression intensity in posture, difference due to singing voice facial expression intensity, or difference due to interaction in perception of suitability”. = 0.05, φ = 3, 3, 76). The results are shown in Table 7.

As a result of the test, the test of the average difference according to the expression intensity of “bak” in 2-Ia is F =. 12, p =. The test of the average difference according to the expression intensity of “we” in 2-Ic is F = 2.03, p =. A significant difference was observed in all tests except for 12.

  As a result, it was confirmed in common that the interaction between the singing voice and the expression intensity of the posture affects the compatibility. In addition, it was confirmed that the expression of the singing voice and the posture itself also affected the suitability in each of the three types of expression.

  Experimental result II [Perception of facial expression intensity]: Average values of perceptual evaluation in Experiment 2-II-ac are shown in FIGS. According to this, it is clear that the expression intensity of the posture has a strong influence on the perceived expression intensity. For this result, the null hypothesis is that there is no difference due to facial expression intensity, difference due to facial expression intensity of singing voice, or difference due to interaction in the perception of facial expression strength. 0.05, φ = 3, 3, 76).

  The results are shown in Table 8.

In the test of the average difference according to the facial expression intensity of posture, F = 394.38 at 2-II-a (bak intensity), F = 575.99 at 2-II-b (drp intensity), 2-II- In c (str intensity), F = 403.85, and a significant difference was observed in all the facial expressions. On the other hand, there was no significant difference in the expression difference between singing voices and the average difference due to interaction. As a result, it was confirmed that the strength of the posture had an influence on the expression intensity, but neither the expression of the singing voice nor the interaction was confirmed. In other words, the result is significantly different from Hypothesis II, which is expected to affect the expression intensity of both singing voice and posture, and the impression of the expression intensity of the combined expression is more influenced by the expression of the posture than the influence of the expression of the singing voice. It was clearly strong.
Experiment 3 (Change in facial expression intensity and perception of fitness)
Considering that the facial expression strength in actual expression changes from moment to moment, the same result was obtained when the facial expression strength changed with time for the conformity confirmed in Experiment 2-II. Experiment 3 was performed to confirm whether or not The following hypothesis was established for conducting Experiment 3.

  Hypothesis: In the combined expression of singing voice and posture, the compatibility is high when the expression intensity of the other changes in response to the change of the expression intensity of one expression, and conforms when the change in the expression intensity is opposite The nature is low.

Subject: Same subject as in Experiment 2 Experiment environment: Same environment as in Experiment 2 Experiment method: Prepare combinations of singing voice postures in facial expression changes in forward and reverse order, listen to stimuli in random order, and suit each Are evaluated in 7 stages (Table 4).

  Experimental conditions (stimulation video): As shown in Table 9, when the expression intensity of the singing voice changes from 0 to 1, a combination in which the expression intensity of the posture changes in the forward order (from 0 to 1), and in the reverse order (from 1 to 0) A video of changing combinations was prepared. Considering the order effect, singing voices (1 to 0) accompanied by changes in facial expression intensity in reverse order were also prepared. Stimulation videos were similarly created for the three types of facial expressions. This can be said to be a change in facial expression intensity connecting the diagonal lines of FIGS.

Contents of teaching: The conformity evaluation criteria were the same as those in Experiment 2, and it was taught that the conformity was evaluated in seven stages as in Experiment 2.

  Experimental results: The results are shown in FIG. In the figure, “-” indicates a combination in which the singing voice and posture change in time series are not appropriate. As a result of testing the null hypothesis as “there is no average difference between normal and reverse order” (α = 0.05, φ = 19), p < . The hypothesis was rejected as 01. Therefore, it was shown that when the facial expression intensity changes, the compatibility of the singing voice and the facial expression intensity in the normal order is higher than the reverse order. As a result, it was found that cross-modality has an effect on the perception of fitness even in facial expression intensity with dynamic changes.

A composite analysis will be conducted while organizing what has been clarified from the analysis results of Experiment 1, Experiment 2 and Experiment 3, and the cross modality between singing voice movements (postures) in anthropomorphic expression will be considered.
Balance between expression intensity perception modalities First, the results of Experiment 1-1 confirmed that expression intensity was perceived in order with respect to the expression intensity in the body motion posture presentation. In addition, the expression strength is perceived while maintaining the order for the expression of the expression strength, even for compound expressions with body movements using voice morphing, which is a technique that is already known to interpolate the expression strength of the singing voice. (Experiment 1-II). In other words, it was confirmed that the expression intensity of each modality or compound expression was perceived as a premise for studying the cross modality.

Next, consider the effects of singing voice and body movement. The R ² value of linear approximation in the perception curve of Experiment 1 was higher in v + g than in g (Table 5). Although the difference in perception due to the difference in modality was not recognized in the test of Experiment 1, the system 10 in FIG. 6 considers that the expression intensity in the perception of singing voice is approximated to a sigmoid function as shown in FIG. Then, the perception curve of the expression intensity depending on the posture has an arc-like shape that swells upward compared to the composite expression, and thus the possibility that the expression intensity of the singing voice has influenced cannot be denied.

  However, looking at the tendency of facial expression intensity in Experiment 2-II, the influence of facial expression intensity on singing voice and the effect of cross-modality are not significant, and the influence of facial expression intensity on the posture is large. In other words, in the video presentation stimulus of anthropomorphic expression simplified and set in the example, the effect of visual expression intensity on perception is different from auditory expression intensity in the composition and expression used in the experiment. It was observed to be large. Therefore, it can be said that the balance of expression intensity was different between modalities. This may be because the perceived sensitivity of the audiovisual expression intensity in anthropomorphic media is essentially different, or the balance of facial expression intensity between the singer's voice color and the stuffed toy posture treated in the above experiment may be unique. Guessed.

On the other hand, in order to balance the effects of visual facial expression intensity and auditory facial expression intensity, the width of facial expression intensity is adjusted on the premise of the effect of cross-modality to balance the facial expression intensity of each modality. It is considered necessary to consider this.
While the perception of balance perception between the modalities of fitness perception is strongly influenced by posture, there is a significant difference in the interaction between singing voice and posture in the test of Experiment 2-I. It became clear that the combination of facial expression intensity affects the perception of fitness. In Experiment 3, it was shown that the combination of the singing voice and the facial expression intensity of the posture affects the compatibility even when time-series changes are involved. In other words, the facial expression intensity in the video presentation stimulus of the anthropomorphic expression set in the embodiment perceives the facial expression of the body motion preferentially and is not related to the facial expression intensity of the singing voice. It becomes important and it turns out that the influence of the body movement of the expression strength in anthropomorphic-singing voice question cross modality singing voice exists.

  However, in Experiment 2-I, it is not necessarily said that the compatibility is high in all combinations of facial expression strengths corresponding to the singing voice-posture, and when the facial expression strength of the singing voice is intermediate (when it is not 0 or 1), The result was that the posture matched with 0 or 1 whichever was closer. The expression intensity of the singing voice is approximated to a sigmoid function, but from the above results, there is a possibility that an intermediate expression intensity such as 0.33 or 0.67 is emphasized and perceived.

  On the other hand, in this configuration, correcting the expression intensity curve of the singing voice, such as setting the morphing interval by an inverse sigmoid function, is effective for improving the compatibility with the expression intensity corresponding to the modalities.

  In addition, when considering the adjustment of the change range of the facial expression intensity of the singing voice shown above, instead of simply reducing the scale linearly, using the correction of the facial expression intensity curve described above, The balance between singing voice and body movement in both can be achieved, and a harmonious natural expression can be realized.

  Based on the knowledge obtained from such experimental results, here, in order to set the combination of facial expression intensity of singing voice and facial expression intensity of gesture (posture, body movement), Define a function f (x). Then, using this function f (x), (1) the calculation for determining the expression intensity of speech is performed using the inverse function f ′ (x) defined by the combination, or (2) body movement In order to determine the expression intensity of the output, the output intensity is determined using the function f (x) defined by the combination. In the following expressions, a, b, c, d, and f are constants, and x is a variable, except that e is a natural logarithm.

In the relationship mapping setting in the body movement-singing voice expression table 28 of the system 10 of FIG. 6, first, assuming that the ideal state of the fitness is high when the singing voice and the posture have the same facial expression strength, the following formula (1 ) Is considered.
[Equation 1]
a-sqrt (power (v, g)) (1)
Here, a is an initial value, for example, 7, v is the expression intensity of singing voice, and g = expression intensity of gesture (body motion).

  However, as can be seen from the previous experiment 2-I, at present, the body motions of 0 and 1 are considered to be the most suitable in the expression intensity of the singing voice of medium (0.33, 0.67) respectively. Met. That is, the expression of a medium singing voice is perceived as being emphasized compared to physical movement. Therefore, in order to adapt the expression intensity of singing voice and the expression intensity of body movement in the system 10 of FIG. 6 in which the expression intensity (morphing rate) of the singing voice is corrected by an inverse sigmoid function, a moderate singing voice expression change is required. It is only necessary that the change in expression that is weak and close to 0 or 1 is transmitted violently. In the experiment using the embodiment of FIG. 6, the influence of the singing voice on the perception of the expression intensity was clearly less than the influence of the expression intensity of the body movement. Therefore, by narrowing the change width of the body motion or widening the change width of the singing voice, it is considered that the expression intensity balance between the singing voice and the body motion can be balanced, and the expression of both can be matched well.

Therefore, instead of the 1: 1 mapping indicated by line A in FIG. 26, mapping is performed by a sigmoid function indicated by line B in FIG. However, in FIG. 26, the horizontal axis represents the expression intensity of singing voice, and the vertical axis represents the expression intensity of body movement. And the sigmoid function of the line B of FIG. 26 becomes following Formula (2).
[Equation 2]
f (x) = (1/1 + power (e, ax + b)) (2)
However, e is a natural logarithm, a and b are constants, for example, a = −8 and b = 4.

Furthermore, the sigmoid function of line B in FIG. 26 may be multiplied by c (0 <c <1) as shown in FIG. In this case, the expression intensity of the singing voice can be further suppressed with respect to the expression intensity of the body movement. Line C in FIG. 26 is given by the following equation (3).
[Equation 3]
f (x) = (1/1 + power (e, ax + b)) * c (3)
The multiple c is set to c = 0.4 as an example.

Furthermore, in order to adapt the expression intensity of the singing voice and the expression intensity of the body movement (posture), the change start position of the expression intensity may be adjusted. For example, when there is no singing voice with no expression at all, or when such a singing voice cannot be synthesized, the singing voice is already expressed with a morphing rate of 0. Therefore, the body motion corresponding to the morphing rate (expression intensity) is started from a predetermined constant value d (d> 0, for example, 0.2) (line D in FIG. 26). The line D is represented by the following formula (4).
[Equation 4]
f (x) = (1/1 + power (e, ax + b)) * c + d ... (4)
On the other hand, if the body motion cannot be expressed because the body motion without facial expression is limited to the movable range, the variable f is further incorporated and the line of FIG. 26 given by the following equation (5) is obtained. It is also conceivable to use a function f (x) as shown in E.
[Equation 5]
f (x) = (1/1 + power (e, (af) x + b + f) * c ... (5)
As an example, when a facial expression intensity of 0 is brought to a position of a voice expression intensity of 0.2, for example, f = about 2 (this f is obtained by calculating back by 0.2 or the above constant). For example, the starting point of the voice expression strength can be raised.

  It should be noted that which of these formulas (2) to (5) is applied may be fixedly set by the system 10, or an applied formula, that is, a function is changed according to the passage of time. You may do it.

  In this manner, the coefficients c, d, and f for matching the facial expression intensity of the body motion with the expression intensity of the singing voice are calculated. The case where the coefficients c, d, and f are 0 is the same as in the equation (2).

Furthermore, specifically, in the case of a body movement in which the expression intensity of the body movement is perceived linearly, in order to match the expression of the singing voice (speech), any of the above formulas is added to the intensity value to be set. By applying the inverse function f ′ (x) of the function f (x) set in the above, the morphing rate for singing voice expression can be determined. For example, in the case of equation (5), the inverse function of the following equation (6) is applicable. If the inverse function of the equation (6) is shown in FIG. 26 for reference, the line F in FIG. 26 corresponds to this.
[Equation 6]
f '(x) = (log (c / x-1)-(b + i)) / (ai) (6)
In the case where the expression intensity of body motion is not perceptually linear, that is, when the body motion has a specific meaning, for example, meanings such as a thousand years, the perceived intensity is “log (m * x + n)”. If the logarithmic function (such as m = 1.5, n = 1 as an example) is used, there is a problem that the body movement does not move smoothly when the movement is changed. It will be necessary to adjust. Specifically, the perceptual intensity formula of this body motion is g (x). For example, if the set matching function is formula (4), the matching formula is f (x), and f ′ (g (x)) In this way, perceptually adapted voice expression can be made possible.

  In this way, in order to visually match the facial expression intensity of the body motion and the expression intensity of the singing voice (voice), the morphing rate is set according to any of the matching functions f (x) in the above formulas (2) to (6). This matching function may be set in advance in the body movement-singing voice expression table 28 shown in FIG. 6 and used in step S15 in FIG.

  In the above-described embodiment, the stuffed toy 12 and the glove-type sensor 14 are used as the body motion input means. In this embodiment, for example, the performer teaches children what kind of physical expression is accompanied by musical expression or singing voice expression, or becomes a different person like a puppet show, such as body movement expression and voice. It is considered effective when performing expressions simultaneously.

Assuming the situation as shown in FIG. 27, i) A plays a singing voice using the hand puppet interface 12 (14), and feels that the voice expression changes due to the stuffed toy gesture by the movement of his hand. ii) B not only feels the change of the singing voice in the performance, but also confirms that the expression of the singing voice changes with the exchange such as touching the stuffed part and changing its shape from the outside. iii) As with A and B, C can be heard as an audience that the singing voice is naturally played with continuous facial expression changes as the movement of the stuffed animal gradually changes. iv) Based on ii), A can feel the touch of B from the inside of the stuffed toy, and at the same time change the facial expression of the singing voice.

  Although such various advantages can be obtained in the above-described embodiment, the means for inputting the body movement is not limited to the hand doll or the stuffed toy 12.

  Only the glove-type sensor 14 may be used without using the stuffed toy 12. However, in this case, for example, an effect such as healing cannot be expected from the stuffed toy.

  In another embodiment of the present invention shown in FIG. 28, cameras 361, 362 and 363 are used as body motion input means. The cameras 361 to 363 are used to three-dimensionally image the whole body of the subject or the user 38 from the front, the side, and the upper side. The camera signals from these cameras 361 to 363 are converted into image or video data by the A / D converter 40 and input to the computer 22. The computer 22 refers to the interpretation table 24A to determine the intensity of the facial motion expression performed by the subject (user) 38 at that time mainly using a pattern recognition technique. Then, based on the intensity of expression of the body motion, the morphing rate is determined according to FIG. 12 with reference to the body motion-singing voice expression table 28A. However, in the same way as in the previous embodiment, this matching function is preset in the body action-singing voice expression table 28A shown in FIG. 28 in order to visually match the expression intensity of the body action and the expression intensity of the singing voice (voice). Aside from that, step S15 in FIG.

  In FIG. 28, the voice morphing can be executed by the body motion using the whole body of the user as described above. Therefore, for example, the expression voice generating apparatus 10 of this embodiment can be used in relation to dance and music.

  In still another embodiment of the present invention shown in FIG. 29, one camera 36A is used as the body motion input means. The camera 36A and the face 38A of the subject or user are photographed two-dimensionally from the front. The camera signal from the camera 36 </ b> A is converted into image data by the A / D converter 40 and input to the computer 22. The computer 22 refers to the interpretation table 24B and identifies the facial expression of the face (A) of the subject (user) at that time as a gesture mainly using a pattern recognition technique. That is, in this embodiment, the facial expression of the user's face 38A can be used as a physical action. Then, based on the body motion (facial expression), the body motion-singing voice expression table 28B is referred to, and in step S15 of FIG. 13, the morphing rate of FIG. 12 is determined according to the matching function of the above equations (2)-(6). To decide.

  In FIG. 29, the voice morphing can be executed by the body motion using the user's face as described above. Therefore, for example, the facial expression voice generator 10 of this embodiment can be used effectively for a sick person sleeping in a bed.

  In another embodiment of the present invention shown in FIG. 30, a robot 42 is used. In such a robot 42, various emotions (anger, sadness, etc.) can be expressed by raising or bending the arm, and also by the face. For such emotional expression, emotion information is given to the connector computer 22 from an external computer (not shown), for example. Based on this emotion information, the output table 44 provides a control signal for each controller (actuator) of the robot 42. By rotating each actuator according to the control signal, the robot 42 can express the emotion as a whole.

  In the embodiment of FIG. 30, the emotion information (control signal) given from the outside is used as the gesture signal as described above, the gesture is identified based on the emotion information, and the physical action-singing voice expression table 28C is referred to. In step S15 of FIG. 13, the morphing rate of FIG. 12 is determined according to the matching function of the above equations (2)-(6) according to the emotion information, that is, the body motion.

  In FIG. 30, voice morphing can be performed in this manner with the emotion of the robot 42 or the physical movement indicated by its location. Therefore, for example, in a robot for communication with a human, such as a communication robot, the expression voice generating apparatus 10 of the embodiment of FIG. 30 can be used.

  In the embodiment of FIG. 30, emotion information does not need to be input to the computer 22 from the outside, and the computer 22 uses the control signal created inside itself to control the robot 42 as it is or after being transformed. It may be.

FIG. 1 is an illustrative view showing one example of a stuffed toy used for inputting a gesture (posture expression intensity) in one embodiment of the present invention. FIG. 2 is an illustrative view showing a state in which a hand (gloves) is inserted into the stuffed toy. FIG. 3 is an illustrative view showing one example of the back side of the hand of the glove-type sensor. FIG. 4 is an illustrative view showing one example of a palm side of a glove-type sensor. FIG. 5 is an illustrative view showing a positional relationship between the thumb part of the glove-type sensor and the thumb first bending sensor and the thumb second bending sensor. FIG. 6 is a block diagram showing an embodiment of the present invention. FIG. 7 is an illustrative view illustrating a part of the interpretation table of FIG. 6 embodiment. FIG. 8 is an illustrative view showing a state when the hand doll (stuffed toy) is “neu (no expression)”. FIG. 9 is an illustrative view showing a state when a “puppet” operation is performed with a hand doll. FIG. 10 is an illustrative view showing a state when a “drp” (unadura) operation is performed with a hand puppet. FIG. 11 is an illustrative view showing a state when the hand puppet is operated as “str” (front stretching). FIG. 12 is an illustrative view showing a morphing rate setting method when morphing three original voices. FIG. 13 is a flowchart showing an example of the operation of the embodiment in FIG. FIG. 14 shows an average of the results of experiment 1-II (perceptual interpolation effect of facial expression intensity), especially the expression of “da; dak” using the embodiment of FIG. It is a graph which shows a value and a standard deviation. FIG. 15 is an average of the results of experiment 1-II (perceptual interpolation effect on expression intensity), especially “wh; drp” expression for each expression intensity, using the embodiment of FIG. It is a graph which shows a value and a standard deviation. FIG. 16 shows an average of the results of experiment 1-II (perceptual interpolation effect of facial expression intensity) using the embodiment of FIG. It is a graph which shows a value and a standard deviation. FIG. 17 is a graph showing experimental conditions for conducting Experiment 2-I (perceptual effect of cross modality with different expression intensities) using the embodiment of FIG. FIG. 18 is a graph showing an average value obtained by summing up the results of the experiment 2-I, particularly the expression of “da; dak”, using the embodiment of FIG. FIG. 19 is a graph showing an average value obtained by summing up the results of the experiment 2-I, in particular, the “wh; drp” facial expression experiment using the embodiment of FIG. FIG. 20 is a graph showing an average value obtained by summing up the results of the experiment 2-I, especially the expression of “we; str” using the embodiment of FIG. FIG. 21 shows an average value obtained by summing up the results of experiment 2-II (perception of expression intensity), particularly “da; dak” expression expression for each expression intensity using the embodiment of FIG. It is a graph to show. FIG. 22 shows an average value obtained by summing up the results of experiment 2-II (perception of facial expression intensity) using the embodiment of FIG. 6, especially the expression of “wh; drp” for each facial expression intensity. It is a graph to show. FIG. 23 shows an average value obtained by summing up the results of the experiment 2-II (perception of expression intensity), particularly the expression of “we; str” expression for each expression intensity using the embodiment of FIG. It is a graph to show. FIG. 24 is a graph showing the suitability of a combination of changing expression intensities when Experiment 3 (change in expression intensity and perception of suitability) is performed using the embodiment of FIG. FIG. 25 is a graph showing that voice morphing is performed according to the inverse sigmoid function in the embodiment of FIG. FIG. 26 is a graph showing an example of a function f (x) set in the body movement-singing voice table of FIG. 6 according to the result of the experiment conducted using the embodiment of FIG. FIG. 27 is an illustrative view for explaining the effect or advantage when the hand doll (stuffed animal) shown in FIG. 1 is used as an interface. FIG. 28 is a block diagram showing another embodiment of the present invention. FIG. 29 is a block diagram showing still another embodiment of the present invention. FIG. 30 is a block diagram showing another embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Expression voice generating apparatus 12 ... Stuffed toy 14 ... Glove type sensor 161a ... Thumb 1st bending sensor 161b ... Thumb 2nd bending sensor 162a ... Index finger 1st bending sensor 162b ... Index finger 2nd bending sensor 163a ... Middle finger 1st bending sensor 163b ... Middle second bending sensor 164 ... Wrist bending sensor 181 and 182 ... Pressure sensor 22 ... Computer 24, 24A, 24B, ... Interpretation table 26 ... Singing voice database 28, 28A, 28B, 28C ... Body motion-singing voice expression table (mapping) table)
30 ... Speech synthesis unit 361-363, 36A ... Camera 38 ... Subject 42 ... Robot 44 ... Output table

Claims (2)

A facial expression voice generating device that generates a voice with a facial expression according to a facial expression of a physical motion input by a physical motion input means for inputting a physical motion,
An audio signal database for storing in advance each audio signal of at least two sounds with different facial expressions;
Setting means for setting the intensity of expression of the voice that matches the intensity of expression of the body movement;
Morphing means for morphing two or more voice signals read from the voice database at a morphing rate according to the intensity of expression of the voice set by the setting means, and voice by a voice signal resulting from morphing by the morphing means A facial expression sound generator comprising a voice output means for outputting. The morphing means morphs the two or more audio signals according to an inverse sigmoid function,
2. The expression voice generating apparatus according to claim 1, wherein the setting means sets the intensity of the voice expression according to a sigmoid function or a deformation function thereof.