JP5531654B2 - Control information generating apparatus and shape control apparatus - Google Patents

Description

  The present invention relates to a technique for controlling the shape of a mouth in a device having a mouth-like part in accordance with the pronunciation of a voice.

  In humanoid robots and CG (Computer Graphics) that can talk, there is a technology that makes it appear as if you are really talking by linking voice pronunciation and mouth movement (lip sync). As an example of applying lip sync to CG, Patent Document 1 stores in advance image data indicating an image having a mouth shape corresponding to the type of vowel of a syllable, and images corresponding to syllable pronunciation are stored in advance. The technology to display is described. Further, as an example in which lip sync is applied to a robot, there is a technique disclosed in Non-Patent Document 1.

Japanese Patent Laid-Open No. 11-66345

Shinichiro Nakaoka, Fumio Kinhiro, Reiko Miura, Mitsuharu Morisawa, Kiyoji Fujiwara, Kenji Kaneko, Shuji Kajita, Hirohisa Hiragawa: Development of Cybernetic Human HRP-4C-Proceedings of the 27th Annual Conference of the Robotics Society of Japan , 2009.

According to the above-mentioned prior art documents, the mouth shape can be changed corresponding to the syllable vowel, but in reality, the mouth shape changes variously, and the syllable syllable vowel changes to change the mouth. The shape changes. This change has a great influence on the natural appearance. However, depending on the above-described prior art documents, it has not been possible to actually perform fine control.
In the technique described in Patent Document 1, an image corresponding to a syllable to be pronounced is switched and displayed simultaneously with pronunciation. For this reason, the image displayed simultaneously with the pronunciation is suddenly switched, and the viewer looks unnatural.
In the technique described in Non-Patent Document 1, a change mode corresponding to the syllable can be set in advance so that the shape of the mouth is changed by natural movement. However, this setter needs to adjust in advance the temporal relationship between the syllable to be pronounced and the change in the shape of the mouth, which is complicated.

  The present invention has been made in view of the above circumstances, while performing control to change the shape of the mouth in a device having a mouth-like part based on the pronunciation information that defines the pronunciation content of the syllable, The purpose is to make the mouth shape change natural.

In order to solve the above-described problem, the present invention changes the shape of the portion simulating the mouth corresponding to the pronunciation content of the pronunciation device that produces sound based on the pronunciation information that defines the pronunciation timing of each syllable and each syllable. A control information generating device for generating control information used to control the shape of the mouth of the device controlled to perform the pronunciation information acquisition means for acquiring the pronunciation information and the shape of the mouth Control information having shape information, timing information indicating the timing when the mouth shape is changed to the shape indicated by the shape information, and control information including transition information indicating a transition time from before the change to when the shape is changed Generation means for generating based on the pronunciation information acquired by the information acquisition means, determining the shape information according to the vowel of the syllable indicated by the pronunciation information, and the syllable And determines the timing information in accordance with the tone generation timing, by determining the transition information in accordance with a predetermined algorithm, and a generation means for generating the control information, the generating means, one said shape If the shape indicated by the information is the same as the shape indicated by the next shape information and the time between timings indicated by the timing information related to each shape information is longer than a predetermined time, the shape is There is provided a control information generation device characterized by determining shape information indicating different mouth shapes and timing information related to the shape information indicating any timing between the timings .

  Moreover, in another preferable aspect, the transition information includes information indicating a timing for starting the change in the shape of the mouth, and the transition time indicated by the transition information is after the change from the timing indicated by the information. It is characterized by showing the timing up to the shape.

  In another preferable aspect, when the consonant of the syllable indicated by the pronunciation information is a specific consonant, the generation unit includes shape information indicating the shape of the mouth according to the consonant, and timing related to the shape information. Control information that further includes information is generated, and the timing information is determined as a timing that is a predetermined time before the timing indicated by the shape information corresponding to the vowel of the syllable.

Further, the present invention is a shape control device for controlling the shape of the mouth in a device having a part simulating the mouth, and is based on pronunciation information defining a plurality of syllables to be pronounced and a pronunciation timing. Sound generation control means for controlling the sound content, and when the sound generation control is performed based on the sound generation information by the sound generation control means, shape information indicating the shape of the mouth is output based on the sound generation information Output means for determining the shape information according to the vowel of the syllable indicated by the pronunciation information, and outputting the shape information before a transition time determined in advance from the pronunciation timing of the syllable; and the output When shape information is output from the means, shape control means for controlling the device so as to change the shape of the mouth to the shape indicated by the shape information over the transition time. In the output means, the shape indicated by one of the shape information and the shape indicated by the next shape information are the same, and the time between sound generation timings related to each shape information is longer than a predetermined time. If determines the shape information indicating the shape of said different port from that of the shape, the shape control device one of the timing between the sound generation timing and determines the sounding timing related to the shape information provide.

  According to the present invention, while performing control to change the shape of the mouth in a device having a mouth-like part based on the pronunciation information that defines the content of pronunciation of the syllable, the change in the shape of the mouth is made natural. can do.

It is a block diagram which shows the structure of the shape control system 1 which concerns on 1st Embodiment. It is a figure explaining an example of the pronunciation information which concerns on 1st Embodiment. It is a figure explaining the shape corresponding | compatible information which concerns on 1st Embodiment. It is a figure explaining the shape information which concerns on 1st Embodiment. It is a block diagram which shows the structure of the shape control function which concerns on 1st Embodiment. It is a figure explaining the control information produced | generated based on the pronunciation information shown in FIG. It is a figure explaining operation | movement of the sounding device and shape change apparatus at the time of using the sounding information and control information which concern on 1st Embodiment. It is a figure explaining the other example of the pronunciation information which concerns on 1st Embodiment. It is a figure explaining the other example of the control information produced | generated based on the pronunciation information shown in FIG. It is a figure explaining operation | movement of the sounding device and shape change apparatus at the time of using the example of the sounding information shown in FIG. 8, and the control information shown in FIG. It is a block diagram which shows the structure of the shape control function which concerns on 2nd Embodiment. It is a figure explaining an example of the content output from an output part based on the pronunciation information shown in FIG. It is a figure explaining the operation | movement of a sounding device and a shape change apparatus at the time of using the sounding information shown in FIG. 2, and the output content shown in FIG. It is a figure explaining an example of the control information concerning the modification 1. It is a figure explaining the operation | movement of a sounding device and a shape change apparatus at the time of using the sounding information shown in FIG. 2, and the control information shown in FIG. It is a figure explaining an example of the pronunciation information which concerns on the modification 2. FIG. It is a figure explaining the eyelid control information produced | generated based on the pronunciation information shown in FIG. It is a figure explaining operation | movement of the sounding apparatus 20 and the shape change apparatus 30 at the time of using the sounding information shown in FIG. 16, and the eyelid control information shown in FIG. It is a figure explaining the shape information which concerns on the modification 2.

<First Embodiment>
[overall structure]
FIG. 1 is a block diagram showing a configuration of a shape control system 1 according to the first embodiment of the present invention. The shape control system 1 controls the sound generation device 20 that pronounces words, the shape change device 30 that changes the shape of the mouth-like portion, and the sound content of the sound device 20 and the shape change content in the shape change device 30. A shape control device 10 is included. The shape control device 10 performs control so that the shape of the mouth in the shape change device 30 changes in accordance with the sound production content from the sound production device 20.

  The sound generation device 20 includes a sound source and a DSP (Digital Signal Processor) that generate a speech waveform signal according to control from the shape control device 10, and a speaker that emits the generated speech waveform signal. In this example, the shape changing device 30 is a humanoid robot, and has a part imitating a mouth (hereinafter simply referred to as “mouth”) on the face part. The shape changing device 30 changes the shape of the mouth under the control of the shape control device 10. Next, each structure of the shape control apparatus 10 is demonstrated.

[Configuration of Shape Control Device 10]
The shape control device 10 includes a control unit 11, a storage unit 12, an operation unit 13, a display unit 14, and an interface 15. Each of these elements is connected to each other via a bus.
The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The CPU loads various control programs stored in the ROM into the RAM and executes them, thereby controlling each part of the shape control device 10 via the bus and realizing various functions. Moreover, the control part 11 implement | achieves the shape control function for controlling the sound generator 20 and the shape change apparatus 30 by running the shape control program memorize | stored in the memory | storage part 12, ROM, etc. FIG. The RAM also functions as a work area when the CPU processes each data.

  The storage unit 12 is, for example, a storage unit such as a nonvolatile memory or a hard disk, and stores sound generation information, shape correspondence information, and control information generated by the shape control function. The control program described above may be stored instead of the ROM. The storage unit 12 may be a storage unit such as an external non-volatile memory connected via a connection interface or the like. The pronunciation information and shape correspondence information stored in the storage unit 12 will be described.

  FIG. 2 is a diagram illustrating an example of pronunciation information according to the first embodiment. The pronunciation information is information that defines the content to be generated by the sound generation device 20. As shown in FIG. 2, the pronunciation information is divided into syllable words, and “timing”, “time” (sec.), “Pitch” (MIDI (Musical Instrument Digital) Interface) corresponds to the note number in the data etc.). The sound generation timing is indicated by the elapsed time (sec.) From the sound generation start instruction. In the pronunciation information shown in FIG. 2, the “character” corresponding to the syllable and the phoneme of the character (“consonant”, “vowel”) are also described, but these are easy to understand the contents of each syllable. It is described. Therefore, the pronunciation information does not need to have all of these pieces of information corresponding to syllables, and may have information that can specify a syllable.

  FIG. 3 is a diagram illustrating shape correspondence information according to the first embodiment. The shape correspondence information is information in which phonemes are associated with shape information indicating the shape of the mouth of the shape changing device 30. As phonemes associated with the shape information, each vowel (“a” “i” “u” “e” “o”), a part of consonant (“m” “p” “b”), etc. "(N)" corresponding to the character "n". In order to clarify that “(n)” is different from the consonant “n” corresponding to “na row”, it is treated as a vowel for convenience in FIG. In this example, six types of shape information (A, I, U, E, O, N) are set. The shape information “A”, “I”, “U”, “E”, “O” corresponds to the vowels “a”, “i”, “u”, “e”, and “o”. The shape information N corresponds to the lip sound and “(n)”.

  FIG. 4 is a diagram for explaining shape information according to the first embodiment. As shown in FIG. 4, the shape information “A”, “I”, “U”, “E”, “O”, “N” are “a”, “i”, “u”, “e”, respectively. , “O”, “N” are information indicating the shape of the mouth.

Returning to FIG. 1, the description will be continued. The operation unit 13 is an operation unit such as a keyboard, a mouse, or a touch sensor provided on the surface portion of the display screen of the display unit 14. The operation unit 13 outputs operation data indicating the operation content to the control unit 11 by a user operation. The operation content includes, for example, an instruction to start sound generation and an instruction to generate control information.
The display unit 14 is a display unit such as a liquid crystal display having a display screen for displaying an image under the control of the control unit 11.

The interface 15 is, for example, a connection terminal for wired connection with an external device, a wireless connection unit for wireless connection, a communication unit for connection via a base station, a network, or the like, and transmits and receives various data to and from the connected external device. . In this example, the sound generator 20 and the shape changing device 30 are connected to the interface 15. Thereby, the shape control device 10 is connected to the sounding device 20 and the shape changing device 30 and controls each of them.
The above is the description of the configuration of each part of the shape control device 10.

[Configuration of shape control function]
Next, the shape control function realized by the control unit 11 executing the shape control program will be described with reference to FIG. In addition, about each structure which implement | achieves the shape control function demonstrated below, you may implement | achieve part or all by hardware.

FIG. 5 is a block diagram showing the configuration of the shape control function according to the first embodiment. The control unit 11 executes the shape control program to configure the pronunciation information acquisition unit 111, the pronunciation control unit 112, the generation unit 113, and the shape control unit 114, thereby realizing a shape control function.
The pronunciation information acquisition unit 111 acquires the pronunciation information from the storage unit 12 in response to an instruction by a user operation or the like. The pronunciation information acquisition unit 111 outputs the acquired pronunciation information to the generation unit 113 when the instruction includes an instruction to generate control information. When the instruction includes an instruction to start pronunciation, the pronunciation control unit To 112.

  The sound generation control unit 112 controls the sound generation device 20 via the interface 15 so as to generate sound with the sound generation content corresponding to the sound generation information acquired from the sound generation information acquisition unit 111. To generate sound with the content corresponding to the pronunciation information, the syllable indicated by the pronunciation information is converted into a speech waveform signal using speech synthesis technology, and the speech waveform signal is emitted at the timing, time, and pitch corresponding to the syllable. It means to make it. Various known techniques can be used for this speech synthesis technique. In this example, the technique described in Japanese Patent No. 3879402 is used. In this technique, the sound generation timing indicates the sound generation timing of the vowel. For this reason, in the syllable having consonants as phonemes, the sound generation control unit 112 emits a sound waveform signal corresponding to the consonant before a predetermined time (in this example, 0.01 seconds). Thus, the sound generation device 20 is controlled.

  Further, the sound generation control unit 112 outputs synchronization information indicating which part of the sound generation information is sounded to the shape control unit 114. Thereby, the control in the sound generation control unit 112 and the control in the shape control unit 114 can be performed in synchronization. Note that the sound generation control unit 112 or the shape control unit 114 may correct the synchronization information in consideration of the control delay in each.

  The generation unit 113 generates control information based on the pronunciation information acquired from the pronunciation information acquisition unit 111 and the shape correspondence information stored in the storage unit 12 and stores the control information in the storage unit 12. This control information is information defining the shape information, the timing of the mouth shape indicated by the shape information, and the transition time from before the change to after the change when the mouth shape changes. Each of these pieces of information is determined by the generation unit 113 based on the pronunciation information and the shape correspondence information. This timing indicates the elapsed time from the sound generation start instruction as with the sound generation information.

  When generating the control information, the generation unit 113 first associates the shape information N with the timing “0.00” as an initial state. Then, the generation unit 113 refers to the pronunciation information, determines shape information corresponding to each syllable, and sets the timing corresponding to the syllable as the timing to change the shape. At this time, when the syllable has a consonant, the generation unit 113 corrects the timing of changing the shape to a timing that is 0.01 seconds earlier. As described above, this timing correction is because, in the control in the sound generation control unit 112, when there is a consonant, a sound waveform signal corresponding to the consonant is emitted a predetermined time before. Done.

  In addition, when there is a period in which the syllable is not pronounced from the timing and time specified in the pronunciation information, the generation unit 113 sets the start timing of the period to make the mouth closed. The shape information N is associated with the control information and added to the control information.

The generation unit 113 further determines a transition time according to a predetermined algorithm corresponding to each shape information other than the initial value. In this example, the generation unit 113 sets the transition time to 0.01 seconds when the mouth is closed as a period during which no syllable is pronounced or when the mouth is closed from the closed state. When changing the shape of the other mouth, the transition time is determined as 0.02 seconds.
It should be noted that this algorithm can be various. For example, when the shape of the mouth changes, the generation unit 113 may determine all transition times as the same transition time, or may randomly determine transition times within a predetermined range. When all the transition times are the same, it may be set in advance in the shape control unit 114 or the shape change device 30, and information indicating the transition time may not be included in the control information. In addition, the generation unit 113 may determine the transition time so as to change according to the time in the pronunciation information, the type of syllable, or the time specified by the user's operation as the transition time. Also good.

  FIG. 6 is a diagram for explaining control information generated based on the pronunciation information shown in FIG. The control information shown in FIG. 6 is generated based on the pronunciation information shown in FIG. 2 and the shape correspondence information shown in FIG. Among the syllables indicated by the pronunciation information shown in FIG. 2, the characters “sa” and “ta” have consonants, and “i” has no consonants. Therefore, for example, the shape information “A” corresponding to “sa” is determined not to be “0.2” but “0.19”, which is 0.01 seconds earlier. The transition time is determined according to the algorithm described above, and the shape information “A” at the timing “0.19” when the previous shape information is “N” and the shape information “N” at the timing “2.00”. “0.01” and other values are determined as “0.02”.

Returning to FIG. The shape control unit 114 controls the shape changing device 30 via the interface 15 so as to change the shape of the mouth based on the control information stored in the storage unit 12. This control information is control information generated by the generation unit 113 based on the pronunciation information used by the sound generation control unit 112 for control.
At this time, the shape control unit 114 controls the shape change device 30 in synchronization with the control of the sound generation device 20 performed by the sound generation control unit 112 based on the sound generation information by the synchronization information output from the sound generation control unit 112. . This synchronization means that the control based on the pronunciation information in the sound generation control unit 112 and the control based on the control information in the shape control unit 114 are processed along the same time axis. That is, the process in which the timing indicated by the pronunciation information and the timing indicated by the control information have the same value is performed at the same timing.

  When the shape control unit 114 changes the shape of the mouth, the transition time from the shape of the mouth before the change to the shape after the change is the transition time corresponding to the shape information after the change indicated in the control information. The shape changing device 30 is controlled so as to be. For example, in the control information shown in FIG. 6, the shape information “I” at timing “0.8” has a transition time “0.02”. Therefore, the transition time from the shape of the mouth of the shape information “A” to the shape of the mouth of the shape information “I” is “0.02 seconds”, and the timing at which the shape change is started Becomes “0.78 seconds” before “0.02 seconds” before “0.8 seconds”. That is, the “transition time” indicated in the control information indicates not only the transition time for changing to the corresponding mouth shape but also the timing for starting the change in combination with the “timing” indicated in the control information. .

In addition, the shape control unit 114 performs control so that the shape of the mouth gradually changes during the transition period from the shape of the mouth before the change to the shape of the mouth after the change. Here, when the shape of the mouth is gradually changed, the shape control unit 114 interpolates and determines the shape of the mouth at each timing during the transition period from the shape of the mouth before and after the change. When the shape change device 30 has such an interpolation function, the generation unit 113 generates control information used for control for changing the shape in the shape change device 30, and the shape control unit 114. Does not have to perform the process of gradually changing.
This completes the description of the shape control function. Next, operation examples of the sound generation device 20 and the shape change device 30 controlled by the shape control function will be described with reference to FIG.

[Example of operation]
FIG. 7 is a diagram for explaining operations of the sound producing device 20 and the shape changing device 30 when the sound producing information and the control information according to the first embodiment are used. In this example, it is assumed that the pronunciation information has the contents shown in FIG. 2, and the control information generated from the pronunciation information has the contents shown in FIG.
The diagram showing the relationship between the pitch and the time shown in the upper part of FIG. 7 shows the contents defined by the pronunciation information. The “phoneme timing” shown below represents the content of the sound generated by the sound generator 20 in phonemes. As described above, for a syllable having a consonant, a speech waveform signal indicating a vowel is emitted at a timing indicated by pronunciation information. The speech waveform signal indicating the consonant is emitted before a predetermined time (0.01 seconds) from the timing.

Regarding the “mouth shape” shown under the phoneme timing, the shape of the mouth of the shape changing device 30 is indicated by the shape information. For example, the period represented by “N” is a period in which the mouth is controlled to have a closed shape corresponding to the shape information N. A hatched portion indicates a transition period in which the shape of the mouth changes. Here, in FIG. 7, transition XY indicates a period of transition from the mouth shape indicated by the shape information X to the mouth shape indicated by the shape information Y. For example, the transition NA indicates a period of transition from the mouth shape indicated by the shape information N to the mouth shape indicated by the shape information A.
Further, the shape information shown under “Mouth shape” indicates the relationship between the shape information indicated by the control information and the timing.

As described above, in the shape control system 1 according to the embodiment of the present invention, the shape control device 10 is controlled by using the shape control device 10 having the sounding device 20 and the mouth-like part. The shape control device 10 controls the sound generation device 20 based on the sound generation information, generates control information from the sound generation information, and controls the shape changing device 30 using this control information. When generating the control information from the pronunciation information, the shape control device 10 determines transition information indicating a transition time from before the change when the mouth shape changes to after the change according to a predetermined algorithm. Then, when the shape of the mouth changes, the shape control device 10 controls the shape change device 30 so that the shape gradually changes with the transition time.
Thereby, the shape of the mouth can be changed so as to look natural according to the content of pronunciation. Further, the shape control device 10 can also generate control information used for control for changing the shape in this way.

[Operation example 2]
Next, an operation example of the sounding device 20 and the shape changing device 30 controlled by the shape control function when the content of the sounding information is different from that shown in FIG. 2 will be described.

  FIG. 8 is a diagram for explaining another example of the pronunciation information according to the first embodiment. FIG. 9 is a diagram for explaining control information generated based on the pronunciation information shown in FIG. The phonetic information shown in FIG. 8 differs greatly from the phonetic information shown in FIG. 2 in that there is a syllable with a lip sound and a syllable with a long pronunciation time. The generation unit 113 generates the control information shown in FIG. 9 based on the pronunciation information shown in FIG.

When generating the control information based on such pronunciation information, the generation unit 113 performs processing described below in addition to the processing described above. The generation unit 113 has a syllable whose pronunciation time exceeds a predetermined time Ta (in this example, “0.8 seconds”), and the shape information corresponding to this syllable and the next syllable is the same. In some cases, the shape of the mouth indicated by the shape information from the timing Ts (in this example, “0.3 seconds”) before the timing corresponding to the shape information corresponding to the next syllable. The control information is generated so as to have a shape in which the opening is small. In this example, the shape information indicating the shape of the mouth with a small opening is represented by adding “(s)” to the original shape information. For example, when the shape information is “A”, the shape information indicating the shape of the mouth whose opening is reduced is represented as “A (s)”.
In addition, the shape information “A (s)”, “I (s)”, “U (s)”, “E (s)”, “O (s)” indicating the shape of the mouth with a small opening is: Instead of being generated by the generation unit 113, it may be stored in the storage unit 12. In this case, the generation unit 113 may acquire the corresponding shape information from the storage unit 12. In addition, the shape information “A (s)”, “I (s)”, “U (s)”, “E (s)”, and “O (s)” is the shape of the mouth with a small opening. There is no need to be, as long as it is different from the mouth shape indicated by the shape information “A”, “I”, “U”, “E”, “O”.

In this case, the generation unit 113 also sets the transition time (in this example, “0.01 seconds”) according to a predetermined algorithm corresponding to the shape information indicating the shape of the mouth whose opening is reduced. decide.
Note that the pronunciation time of the character “TA” is equal to or longer than the time Ta, but the shape information “A” corresponding to the character “TA” is different from the shape information “N” that follows, so that the mouth shape is opened slightly. No processing is performed.

In addition, when there is a syllable indicated by the pronunciation information having a lip sound, the generation unit 113 determines not the shape information corresponding to the vowel of the syllable as described above but the shape information “N” corresponding to the lip sound. Control information is generated. That is, the timing corresponding to the shape information “N” is determined as the timing at which the speech waveform signal corresponding to the consonant of the syllable is emitted.
Then, the generation unit 113 generates control information using the timing corresponding to the syllable, that is, the timing at which the speech waveform signal corresponding to the vowel is emitted as the timing of setting the mouth shape corresponding to the vowel. The transition time from the shape information “N” for the lip to the shape information for the vowel is set to “0.01 seconds” in this example. The generation unit 113 may determine the timing of the mouth shape corresponding to the vowel as the timing before the timing at which the speech waveform signal corresponding to the vowel is emitted. In this case, the generation unit 113 determines to shorten the corresponding transition time.

  FIG. 10 is a diagram for explaining operations of the sounding device 20 and the shape changing device 30 when the sounding information shown in FIG. 8 and the example of the control information shown in FIG. 9 are used. The detailed description of FIG. 10 is omitted because it is the same as in FIG. Note that the shape information “N” corresponding to the character “MA” immediately starts a transition to the shape information “A”. Therefore, in the description of FIG. 10, the transition NA immediately follows the transition AN. .

  In this way, when controlling the shape changing device 30 so that the same mouth shape continues, the shape control device 10 changes the shape of the mouth so that the opening is reduced, and thus according to the pronunciation content. It is possible to make the change in the shape of the mouth look more natural. In addition, when the syllable to be pronounced has a lip sound, it is possible to make the change in the shape of the mouth look more natural by changing it so that the mouth is once closed.

Second Embodiment
In the first embodiment, the shape control device 10 controls the shape of the mouth of the shape change device 30 using the control information generated in advance by the generation unit 113. In the second embodiment, the shape control device 10 controls the shape of the mouth of the shape change device 30 in real time while controlling the sound content of the sound device 20 based on the sound information.
The configuration of the shape control function in the second embodiment will be described.

[Configuration of shape control function]
FIG. 11 is a block diagram illustrating a configuration of a shape control function according to the second embodiment. By executing the shape control program, the control unit 11 configures the pronunciation information acquisition unit 111A, the sound generation control unit 112A, the shape control unit 114A, and the output unit 115 to realize the shape control function.

The pronunciation information acquisition unit 111A acquires the pronunciation information from the storage unit 12 in response to an instruction by a user operation or the like. The pronunciation information acquisition unit 111A outputs the acquired pronunciation information to the output unit 115 and the pronunciation control unit 112A.
The sound generation control unit 112A performs the same process as the sound generation control unit 112 in the first embodiment, but outputs to the output unit 115 synchronization information indicating which part of the sound generation information is sounded. Thereby, the control in the sound generation control unit 112A and the output from the output unit 115 can be performed in synchronization.
When the shape control unit 114A acquires the shape information output from the output unit 115 and the information indicating the transition time, the shape control unit 114A starts the change so that the shape of the mouth of the shape change device 30 becomes the shape of the mouth indicated by the shape information. The shape changing device 30 is controlled so that the shape of the mouth is gradually changed and, when the transition time elapses, the shape of the mouth indicated by the shape information is obtained.

The output unit 115 indicates the shape information and transition time based on the pronunciation information acquired from the pronunciation information acquisition unit 111A, the shape correspondence information stored in the storage unit 12, and the synchronization information output from the pronunciation control unit 112A. Information is output to the shape controller 114A.
When the sound generation control by the sound generation control unit 112A is started, the output unit 115 first outputs the shape information “N” as an initial value. Then, the output unit 115 outputs the shape information corresponding to the syllable and information indicating the transition time to the shape control unit 114A at a timing earlier than the timing at which the syllable is generated from the sound generation device 20 by the transition time. To do.
The output unit 115 determines the transition time according to a predetermined algorithm. This algorithm can have various aspects as described in the first embodiment. When the transition time does not change according to the shape information and is set to a predetermined time, if the transition time is set in the shape control unit 114A in advance, the output unit 115 does not output information indicating the transition time. May be.

FIG. 12 is a diagram for explaining an example of contents output from the output unit based on the pronunciation information shown in FIG. The output content shown in FIG. 12 shows the content output by the output unit 115 based on the pronunciation information shown in FIG. 2 and the shape correspondence information shown in FIG. For example, the output unit 115 outputs the shape information “A” and the transition time “0.01” at the timing “0.18”.
The above is the description of the shape control function in the second embodiment. Next, operation examples of the sound generation device 20 and the shape change device 30 controlled by the shape control function in the second embodiment will be described with reference to FIG.

[Example of operation]
FIG. 13 is a diagram for explaining the operation of the sound producing device 20 and the shape changing device 30 when the sound production information shown in FIG. 2 and the output contents shown in FIG. 12 are used. The detailed description of FIG. 13 is the same as that in FIG. The shape information shown in the lower part of FIG. 13 indicates the shape information output from the output unit 115 at the timing indicated by the arrow.
As described above, in the shape control system 1 according to the second embodiment, the shape control apparatus 10 outputs the shape information in accordance with the transition start timing, so that the output information is output to the pronunciation information without generating control information. The sound generation device 20 and the shape changing device 30 can be controlled synchronously.

<Modification>
As mentioned above, although embodiment of this invention was described, this invention can be implemented in various aspects as follows.
[Modification 1]
In the first embodiment described above, the control information generated by the generation unit 113 includes information indicating the transition time, but the transition time may be specified in another manner. Of the modes for specifying the transition time, an example of specifying the transition time using the shape information will be described with reference to FIG.

  FIG. 14 is a diagram illustrating an example of control information according to the first modification. The control information according to the first modification illustrated in FIG. 14 is the same as the control information according to the first embodiment illustrated in FIG. 6 described above. The generation unit 113 according to the first modification generates control information having shape information indicating the shape before the change in correspondence with the timing of starting the previous transition of the shape information indicating the shape after the change. That is, the control information is always defined as two sets of shape information. Thus, the timing corresponding to the second shape information is the timing at which the transition of the mouth shape change is started. The transition time is from the transition start timing to the timing corresponding to the next shape information (transition end timing). As described above, the control information may be included in any manner as long as it includes information indicating the transition time.

  FIG. 15 is a diagram for explaining the operation of the sound producing device 20 and the shape changing device 30 when the sound producing information shown in FIG. 2 and the control information shown in FIG. 14 are used. The detailed description of FIG. 15 is omitted because it is the same as in FIG. As shown in FIG. 15, the start timing and end timing of the transition period are defined in the control information so as to be sandwiched between the shape information indicating the shape before the change and the shape information indicating the shape after the change.

[Modification 2]
In the embodiment described above, the shape control device 10 may also control shapes other than the mouth shape in the shape change device 30. For example, if the shape changing device 30 has a portion simulating eyes (hereinafter simply referred to as “eyes”) and is configured to be able to open and close the eyes, the shape control device 10 controls the opening and closing. May be. The configuration of the shape control device 10 in this case will be described.

In the second modification, the storage unit 12 stores shape information indicating the shape of the eyes. In this example, the shape information “OPEN” and “CLOSE” indicate the shapes of the eyes in the open state and the closed state, respectively.
In the second modification, the generation unit 113 generates eyelid control information based on the pronunciation information and stores it in the storage unit 12. This eyelid control information is information defining the shape information, the timing of the eye shape indicated by the shape information, and the transition time from before the change to after the change when the eye shape changes. Each of these pieces of information is determined by the generation unit 113 based on the pronunciation information and the shape correspondence information. This timing indicates the elapsed time from the sound generation start instruction as with the sound generation information.

  When generating the eyelid control information, the generation unit 113 determines the shape information to be “OPEN” or “CLOSE” based on the pronunciation time of each syllable indicated by the pronunciation information in this example. The generation unit 113 determines shape information according to a predetermined algorithm. In this example, the generation unit 113 randomly determines the time Tb for each syllable from a specific range of time, and when the pronunciation time of the syllable exceeds the time Tb, the shape information corresponding to the syllable is “ “CLOSE”. Then, the generation unit 113 determines the shape information corresponding to the syllable that does not satisfy the condition that the shape information is “CLOSE” as “OPEN”.

Note that the generation unit 113 is not limited to such determination of shape information, and can determine shape information in various modes. For example, the generation unit 113 may determine the shape information based on the types and combinations of phonemes (consonants, vowels) constituting the syllable, the pitch, and the connection between syllables (phrase flow). The generation unit 113 may determine the shape information to be “CLOSE” with a certain probability when the syllable sounding time exceeds a certain time Tc.
By giving randomness to the determination of the shape information in the generation unit 113, it is possible to avoid closing eyes at the same part even when the same phrase continues.
The generation unit 113 determines the transition time according to a predetermined algorithm in the same manner as the transition time in the control information in the first embodiment.

In the second modification, the shape control unit 114 controls the shape changing device 30 via the interface 15 so as to change the opening and closing of the eyes based on the eyelid control information stored in the storage unit 12. This eyelid control information is the eyelid control information generated by the generation unit 113 based on the sound generation information used for control by the sound generation control unit 112. The control based on the eyelid control information in the shape control unit 114 is only different from the control based on the control information in the first embodiment. Therefore, detailed description of the content of control based on the eyelid control information in the shape control unit 114 is omitted.
Next, the operation of the sounding device 20 and the shape changing device 30 in the modification 2 will be described by exemplifying the sounding information.

  FIG. 16 is a diagram illustrating an example of pronunciation information according to the second modification. FIG. 17 is a diagram for explaining eyelid control information generated based on the pronunciation information shown in FIG. 18 is a diagram for explaining operations of the sound generation device 20 and the shape change device 30 when the sound generation information shown in FIG. 16 and the eyelid control information shown in FIG. 17 are used. Regarding the “eye shape” shown in FIG. 18, the shape of the eye of the shape changing device 30 is indicated by shape information.

FIG. 19 is a diagram illustrating shape information according to the second modification. In the example illustrated in FIG. 19, the shape of the eye and the shape of the mouth are illustrated using a combination of shape information “I” and “OPEN” and a combination of shape information “I” and “CLOSE” as examples.
In this way, the shape control device 10 generates control information for controlling the shapes of various parts based on the pronunciation information, and not only the shape of the mouth of the shape changing device 30 but also the shape of the eyes and other parts. The shape may also be controlled.

[Modification 3]
In the embodiment described above, the shape change device 30 is a humanoid robot, but it may have a display screen and display a human face or the like on the display screen by CG. In other words, the shape changing device 30 may be a device having a portion that simulates a mouth. The mouth is not limited to a person's mouth, but may be an animal's mouth, or may be a mouth that is artificially provided in a plant or structure that does not have a mouth.
Moreover, when displaying on a display screen, such as CG, you may make it the display part 14 display instead of the shape change apparatus 30 displaying.

[Modification 4]
In the first embodiment described above, the shape control device 10 includes the generation unit 113 and the shape control unit 114 that generate control information. However, the processing may be performed by different devices. . That is, the shape control device 10 executes a generation program by the CPU, thereby realizing a control information generation function having a pronunciation information acquisition unit 111 and a generation unit 113, and by executing the control program by the CPU. You may make it comprise with the apparatus which implement | achieves the control function which has the shape control part 114. FIG. The shape control unit 114 may control the shape change device 30 based on the control information generated by the generation unit 113.

[Modification 5]
In the first embodiment described above, the generation unit 113 sets the transition start timing for changing the shape information corresponding to the syllable having consonant to the shape indicated by the shape information as the phoneme timing of the consonant, and sets the transition end timing as the vowel. Control information may be generated so as to be the phoneme timing. In the second embodiment, the shape information and the information indicating the transition time may be output so that the output unit 115 has the transition start timing and the end timing.

[Modification 6]
In the embodiment described above, the shape control device 10, when there is a lip sound in the syllable indicated by the pronunciation information, before the mouth shape corresponding to the vowel is made, the mouth indicated by the shape information “N”. The shape changing device 30 is controlled so as to have a shape that closes. Similarly, if the shape of a person's mouth changes when a syllable is pronounced even if it is not a lip sound, for example, if it is a syllable with a specific consonant such as `` wa '', `` sa '', The memory | storage part 12 may memorize | store the shape information which shows the shape of the mouth corresponding to this. Then, the generation unit 113 and the output unit 115 may perform processing using this shape information as in the case of using the shape information “N” corresponding to the lip sound.

[Modification 7]
In the embodiment described above, the shape control device 10 uses shape information corresponding to the vowels “a”, “i”, “u”, “e”, and “o”. You may use the shape information which shows the shape of this.

[Modification 8]
The shape control program in the above-described embodiment is stored in a computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, or a semiconductor memory. Can be provided. Further, the shape control apparatus 10 may download the shape control program via a network.

DESCRIPTION OF SYMBOLS 1 ... Shape control system, 10 ... Shape control apparatus, 11 ... Control part, 12 ... Memory | storage part, 13 ... Operation part, 14 ... Display part, 15 ... Interface, 111, 111A ... Sound generation information acquisition part, 112, 112A ... Sound generation Control unit, 113 ... generating unit, 114, 114A ... shape control unit, 115 ... output unit, 20 ... sound generator, 30 ... shape change device

Claims (4)

The shape of the mouth of the device controlled so that the shape of the portion imitating the mouth changes corresponding to the pronunciation content of the sounding device that is sounded based on the sound information defining the sounding timing of each syllable and the syllable A control information generation device for generating control information used for controlling
Pronunciation information acquisition means for acquiring the pronunciation information;
It has shape information indicating the shape of the mouth, timing information indicating the timing at which the shape of the mouth is changed to the shape indicated by the shape information, and transition information indicating the transition time from before change to after change when the shape changes. Generation means for generating control information based on the pronunciation information acquired by the pronunciation information acquisition means, determining the shape information according to the vowel of the syllable indicated by the pronunciation information, and generating the syllable pronunciation timing And generating the control information by determining the transition information according to a predetermined algorithm, and determining the timing information accordingly.
In the generation unit, the shape indicated by one of the shape information and the shape indicated by the next shape information are the same, and the time between the timings indicated by the timing information related to the respective shape information is based on a predetermined time. Control information generation characterized by determining shape information indicating the shape of the mouth different from the shape and timing information related to the shape information indicating any timing between the timings when the length is long apparatus. The transition information includes information indicating timing for starting a change in the shape of the mouth,
The control information generation device according to claim 1, wherein the transition time indicated by the transition information indicates a timing from the timing indicated by the information to the timing of forming the shape after the change. When the consonant of the syllable indicated by the pronunciation information is a specific consonant, the generation unit generates control information further including shape information indicating the shape of the mouth according to the consonant and timing information related to the shape information 3. The control according to claim 1, wherein the timing information is determined as a timing that is a predetermined time before a timing at which the shape information indicated by the shape information corresponding to the syllable of the syllable is set. Information generator. A shape control device for controlling the shape of the mouth in a device having a part simulating a mouth,
Pronunciation control means for controlling the content of pronunciation for the sound generation device based on the sound information defining the plurality of syllables and sound generation timing;
Output means for outputting shape information indicating the shape of the mouth based on the pronunciation information when the pronunciation control is performed based on the pronunciation information by the pronunciation control means, the syllable indicated by the pronunciation information; Output means for determining the shape information in accordance with the vowel of, and outputting the shape information before a transition time determined in advance from the sounding timing of the syllable;
When the shape information is output from the output means, the shape control means for controlling the device so as to change the shape of the mouth to the shape indicated by the shape information over the transition time, and
When the shape indicated by one of the shape information is the same as the shape indicated by the next shape information, and the time between sound generation timings related to each shape information is longer than a predetermined time, the output means determines the shape information indicating the shape of said different port from that of the shape, the shape control device one of the timing between the sound generation timing and determines the sounding timing related to the shape information.

Images