JP2009216723A - Similar speech selection device, speech creation device, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately select speech which is similar to a target speech from a plurality of sample speeches. <P>SOLUTION: A similar speech selection device includes: a voice actor speech database (DB) for storing the plurality of sample speeches; a distance calculation processing section (Step 902 to 910) for calculating the distance between the target speech and each of the plurality of sample speeches by a weighted linear sum of a distance scale between two pieces of speech, which is calculated for each sound feature amount of one category or more for speech; and a speech selection section (Step 912) for selecting speech in which the distance calculated by the distance calculation processing section is the smallest, is selected from the plurality of sample speeches as the speech similar to the target speech. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a speech production apparatus for multimedia productions in which a performer utters according to a scenario, such as a movie or animation, and in particular, it can efficiently record and reproduce speech according to a predetermined scenario. The present invention relates to a speech sound generating device.

  With the development of computer technology, particularly video and audio processing technology, systems capable of producing multimedia products with users in a very short time are being put into practical use. For example, by introducing such a system as an attraction for an exposition, and creating an on-site movie with the participants of the exposition as characters, it is possible to realize more attractions. The effect that attracts

  Such an attraction system is disclosed in Patent Document 1. The system disclosed in Patent Document 1 includes a plurality of three-dimensional scanners and image processing personal computers (hereinafter simply referred to as “PCs”) that capture a three-dimensional face image of a participant, and a movie scenario prepared in advance. A scenario storage server for storing together with an image of a character, a background image, and the like, and a face of a movie character stored in the scenario storage server based on the 3D face images of the participants taken by the 3D scanner Is replaced with a face image of the participant, thereby including an attraction video generation device for generating a movie in which the participant appears as a character, and a video transmission device for projecting the generated movie.

Each of the plurality of participants can appear in the movie as the character by designating a desired character in the movie.
JP 2005-115740 A

  However, in the system described above, even if the face image of the character can be replaced with the face image of the participant, the voice cannot be replaced. This is because, in the case of voice, such a face image can be used in any scene by capturing the participant's face image with a 3D scanner. It is because it cannot be done.

  In the case of audio, it is necessary to read the dialogue in accordance with the scenario and the video. Such work is not only difficult, but has a problem that it is a work having a long time. It was not possible to force such a participant to an attraction participant who had a limited time, and as a result, the voice of the participant could not be used in the above-described attraction system.

  This can happen not only in movies but also in other situations. For example, when creating something like a radio drama that uses only audio, it is difficult to create a long drama using the voice of the participant if the time available for the participant is short. is there. The same problem also occurs when the animation is dubbed or when the dubbing is performed to apply a human voice to a live-action animal.

  In addition, such a problem may occur not only when using the voice of a temporarily visiting person such as an attraction participant, but also when using a voice-over profession such as a so-called voice actor. When dubbing all scenarios of a certain length with the voice of the voice actor, the minimum required time is determined, and when the available time is very limited, it is impossible to completely dubb Even cases can occur.

  If many dialogues have already been recorded in the voice of another person, it may be possible to use a voice of a person who is very similar to the voice of the participant. However, in order to do so, it is necessary to record the voices of as many people as possible in consideration of gender, age, and voice quality, which is very difficult. In order to solve these problems, it is desirable to be able to easily replace the voice of the character with the user's voice in a short time in order to produce a multimedia production in which the character's dialogue is known. . For this purpose, for example, it is desirable that a sound close to the target sound can be generated with high accuracy from a sample sound prepared in advance. For that purpose, it is further desirable to be able to select a sound close to the target sound from the sample sounds prepared in advance.

  Therefore, an object of the present invention is to provide a voice selection device capable of accurately selecting a voice similar to a target voice from a plurality of sample voices.

  Another object of the present invention is to select a sound similar to the target sound from a plurality of sample sounds with high accuracy and generate a sound very similar to the target sound from this sound. Is to provide a generator

  The similar voice selection device according to the first aspect of the present invention is a similar voice selection device for selecting a voice similar to a target voice from a plurality of sample voices, and stores the plurality of sample voices. And a target speech and each of a plurality of sample speeches by a weighted linear sum of distance measures between the two speeches calculated for each of the one or more acoustic feature quantities for the speech A distance calculating means for calculating the distance between the two and a sound for selecting a sound having the smallest distance calculated by the distance calculating means from among a plurality of sample sounds as a sound similar to the target sound Selecting means.

  There is a case where it is desired to utter a content of a speaker that is not recorded by the speaker. This similar voice selection device is effective in such a case. Specifically, the distance calculation means calculates a distance measure between two sounds calculated for each of one or more types of acoustic feature quantities for the sound, and calculates a target sound by a weighted linear sum. And the distance between each of the sample sounds. The voice selection means selects the one having the smallest distance.

  By using a linear sum of distance measures calculated for one or more types of acoustic feature quantities between two voices, it is closer to the target voice with higher accuracy than when only a single acoustic feature quantity is used. Voice can be defined.

  Preferably, the distance measure is a distance measure calculated by dynamic time warping (DTW) between the same acoustic features of two sounds.

  More preferably, the distance measure is a distance measure calculated by a Gaussian Mixture Model between the same acoustic features of two sounds.

  As described above, it can be experimentally confirmed that by using the linear sum of the distance scales calculated by using DTW or GMM, the accuracy becomes higher than when only a single acoustic feature amount is used.

  More preferably, the combination coefficient of the weighted linear sums of the distance scales is a result of ranking the similarity between each of the plurality of sounds and each of the plurality of target sounds by a human and a result obtained by the distance calculating unit. Is determined in advance so as to maximize the correlation.

  The combination coefficient of the weighted linear sum is determined in advance so as to obtain a result having a high correlation with the ranking given by human judgment. Since a linear sum of distance scales for one or more types of acoustic features is used, this correlation can be increased. As a result, it is possible to obtain a voice selection device that can obtain a result close to the selection result of a similar voice by a human.

  When the computer program according to the second aspect of the present invention is executed by a computer, the computer is calculated for each of means for storing a plurality of sample sounds and one or more acoustic feature quantities for the sounds. The distance calculation means for calculating the distance between the target sound and each of the plurality of sample sounds, and the distance calculation means from among the plurality of sample sounds The selected one having the smallest distance is made to function as a voice selection means for selecting a voice similar to the target voice.

  An audio generation device according to a third aspect of the present invention is an audio generation device for generating audio having a predetermined content with audio similar to a target audio, and having a predetermined content, By means of means for storing a plurality of sample sounds and a weighted linear sum of distance measures between two sounds calculated for each of one or more types of acoustic features for the sounds, A distance calculation means for calculating the distance between each of the sample sounds, and a sound similar to the target sound that has the smallest distance calculated by the distance calculation means among the plurality of sample sounds Voice selection means for selecting as a voice, and voice generation means for generating a voice having a predetermined content using the voice selected by the voice selection means.

  In this voice generation device, the target voice is selected from a plurality of sample voices with the same configuration as that of the voice selection device according to the first aspect, and a voice of utterance having a predetermined content is generated by the selected voice. Is done. Therefore, similarly to the voice selection device according to the first aspect, it is possible to accurately define a voice close to the target voice and generate a voice of utterance having a predetermined content by the voice.

  Preferably, the sound selection means includes means for selecting a plurality of samples having the smallest distance calculated by the distance calculation means from among a plurality of sample sounds as sounds similar to the target sound. The generation means includes voice morphing means for generating a new voice by performing morphing on the plurality of voices selected by the voice selection means.

  A sound that is most similar to the target sound is selected from the plurality of sample sounds, and the sound is further synthesized by morphing them. Morphing can generate speech that is closer to the target speech than the selected sample speech.

  More preferably, the speech generation apparatus further includes a predetermined feature vector of morphed speech obtained by morphing speech of a plurality of speakers selected by the means for selecting by speech morphing means, a target, Optimization means for optimizing the morphing ratio of a plurality of speakers at the time of morphing by the voice morphing means so that the distance between the voice and the feature quantity vector corresponding to the predetermined feature quantity is minimized. Including.

  Since the morphing ratio is optimized, the voice obtained by morphing is closer to the target voice.

  When the computer program according to the fourth aspect of the present invention is executed by a computer, the computer stores means for storing a plurality of sample sounds having predetermined contents, and one or more kinds of sounds for the sounds. Distance calculating means for calculating a distance between the target sound and each of the plurality of sample sounds by a weighted linear sum of distance measures between the two sounds calculated for each feature amount; , A voice selection unit for selecting, as a voice similar to the target voice, a voice having a smallest distance calculated by the distance calculation unit from among a plurality of sample voices, and a voice selected by the voice selection unit. It is made to function as an audio | voice production | generation means for producing | generating the audio | voice of the predetermined content.

  Hereinafter, a multimedia production system that performs similar speech selection and speech generation according to an embodiment of the speech speech creation device of the present invention will be described. In the following description and drawings, the same parts are denoted by the same names and reference numerals. Their functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 shows a block diagram of a multimedia production system 50 according to an embodiment of the present invention. Referring to FIG. 1, a multimedia production system 50 includes a three-dimensional scanner group 60 including a plurality of three-dimensional scanners similar to those described in Patent Document 1, and participants photographed by the three-dimensional scanner group 60. An image processing PC 62 for creating a three-dimensional model of the face image of the user, and a scenario storage server (not shown) for storing the movie scenario together with the face image of the character (hereinafter referred to as “character”) and other images. Then, using the face image of the participant generated by the image processing PC 62, the face image of the character stored in the scenario storage server is replaced to generate a video in which a person having the face of the participant appears, and the video data 66 A video generation device 64 for outputting the video data and a video data storage device for storing the video data 66.

  The multimedia production system 50 further includes a video material DB (database) 70 for storing the video material for creating the final video data 66, and a dubbing target by a participant among the characters of this movie. A dialogue information storage unit 72 for storing dialogue information related to the dialogue of the character, a standard voice storage unit 74 for storing standard speech in which the dialogue of the character in the movie is spoken with standard speech, and a movie And a cut information storage unit 76 for storing cut information indicating in what scene each dialogue is uttered and thereby what kind of acoustic effect should be added to the speech of the dialogue.

  The multimedia production system 50 further stores the video stored in the video material DB 70, the speech information stored in the speech information storage unit 72, the speech utterance data of the standard speech stored in the standard speech storage unit 74, and the cut information storage. Using the cut information stored in the section 76, the voice of the participant (user) is recorded, and the voice of the dialogue of a specific character of the movie is replaced with the voice of the user based on the voice (so-called “dubbing”) Similar processing) is performed, and speech audio data 86 composed of speech uttered by speech of the user, speech start time of speech in speech speech data 86, speech time, corresponding audio file name, etc. as a table A dialogue voice data creation unit 90 for outputting a dialogue voice table 88 for storage.

  Similar to the three-dimensional scanner group 60, the dialogue voice data creation unit 90 is configured to be able to process voices of a plurality of users. As will be described later, each user is distinguished by an identifier (ID), and the same user is the same in the video processing system including the three-dimensional scanner group 60, the image processing PC 62, and the video generation device 64, and in the speech audio data creation unit 90. ID is assigned and managed. By doing so, it becomes possible to replace the faces and sounds of a plurality of characters in the movie with the faces and sounds of a specific user at the same time.

  The multimedia production system 50 further includes a method list table 78 that stores information indicating which method is used for each dialogue when the speech speech data creation unit 90 creates speech speech of the character from the user's speech. For the speech that could not be recorded by the user, the speech audio data creation unit 90 used to create speech audio data instead of the user's speech, and uttered each speech of this movie in advance with the voices of various voice actors A voice actor voice DB 80 storing sample voice data and a speech segment used when the speech voice data creation unit 90 generates speech voice data having a voice quality similar to the user's voice by voice synthesis instead of the user's speech. Of the voice actor voices stored in the voice actor voice DB 80, as will be described later, the segment DB 82 stored together with the feature amount data. In order to store a linear combination coefficient for calculating a composite acoustic feature amount obtained by linearly combining a plurality of acoustic feature amounts used when selecting a voice quality similar to the user's voice And a linear combination coefficient storage unit 94. In this embodiment, as will be described later, a composite acoustic feature amount is calculated by linear combination of eight types of acoustic feature amounts. Therefore, eight coefficients are stored in the linear combination coefficient storage unit 94.

  The multimedia production system 50 further reproduces the video data 66 output from the video generation device 64 and the speech audio data 86 output from the speech audio data creation unit 90 using the speech audio table 88 in synchronization with each other. Thus, a video / audio reproduction device 92 for performing a multi-character product in which a part of the character's face image and sound is replaced with the user's face image and sound is included.

  As described above, the speech sound data creation unit 90 has a function of recording speech of a plurality of users and generating speech speech of different characters based on them. For this purpose, the speech sound data creation unit 90 has a plurality of pieces of user information for receiving input of user information including identification information, sex, name, age, information for specifying a character to be dubbed, etc. Based on the user information received by the input units 100, 100A,..., 100N and the user information input units 100, 100A,. , 102N, and a plurality of character voice generation units 102, 102A,..., 102N for generating and outputting the speech of the corresponding character with the voice quality of the user by the method of The speech of various characters replaced by the voice quality of the user speech output by So as to form a sound one multimedia productions on the basis of the speech information by integrating according to the numerical order of the words, and a voice integration section 104 for outputting as speech audio data 86 and the speech sound table 88.

  The user information input by the user information input units 100, 100A,..., 100N is also given to the image processing PC 62, and is used for managing the user's face image.

  The plurality of character voice creation units 102, 102A,..., 102N have the same configuration. Therefore, hereinafter, the configuration of the character voice creation unit 102 will be described as a representative.

  FIG. 2 is a functional block diagram of the character voice creation unit 102. Referring to FIG. 2, the character voice creation unit 102 receives user information, the video material stored in the video material DB 70, the dialogue information stored in the dialogue information storage unit 72, and the standard voice storage unit 74. A speech recording unit 114 for causing the user to utter speech of the character to be dubbed by the user and recording the uttered speech in the user speech DB 120, using speech of standard speech stored in The input / output device 112 is used for the attendant to operate the voice recording unit 114 to control the recording of the utterance in the recording unit 114 and to assist the user in speaking.

  By the way, in general, there are many dialogues constituting one movie, and it is predicted that it takes a considerable time for the user to record the speech of the dialogue even if the dialogue is limited to the dialogue of a certain character. When it comes to the utterance of a movie sound, it is necessary to utter according to the movement of the character, and this recording is likely to take more time. In particular, attraction and the like, there are many cases where it is difficult to record all speech sounds due to time restrictions. Even if recording is possible, the utterance time is often too short or too long, and the recorded speech cannot often be used as it is. Therefore, in the character voice creation unit 102 according to the present embodiment, a predetermined voice generation method is used for both the portion of the speech of a character that has been recorded by the user and the portion that has not been recorded. The goal is to generate speech with a voice quality that is as close to that of the user as possible. The method list table 78 stores a method list for each dialogue indicating which method is used for each dialogue and in what priority order, and the character voice creation unit 102 performs this method list table for voice generation. 78 is used.

  The character voice creation unit 102 further refers to the user voice stored in the user voice DB 120 by the voice recording unit 114 with reference to the method list table 78 and uses a method that matches the conditions for each speech of the character to be dubbed by the user. Further, with reference to the linear combination coefficient from the linear combination coefficient storage unit 94 shown in FIG. 1, among the voice actor voices stored in the voice actor voice DB 80, the voices of three voice actors similar to the user voice are determined. Then, a morphing rate determination vector 116 for calculating a morphing rate vector when generating a voice similar to the user voice by morphing these three voice actor voices, and the three names determined by the synthesis technique deciding unit 116 A similar voice actor storage unit 130 for storing the identifiers of voice actor voices, and a morph determined by the synthesis method determination unit 116 Using the method determined by the morphing rate storage unit 132 for storing the singing rate vector and the synthesis method determining unit 116, the speech of the character is created in accordance with the voice quality of the user, and the speech file 110 is output for each speech. And an audio creation unit 118.

  The voice creating unit 118 corresponds to the voice stored in the voice actor voice DB 80, the user voice DB 120, the segment DB 82, the standard voice storage unit 74, and the identifier stored in the similar voice actor storage unit 130 according to the method at the time of synthesis. A voice actor voice or a morphing rate vector stored in the morphing rate storage unit 132 is appropriately used. In addition, the voice creation unit 118 adds a sound effect determined based on the cut information stored in the cut information storage unit 76 to the generated speech voice of the line, and outputs a final voice file 110.

  Details of selection processing of voice actor speech similar to the user speech, details of morphing using them, and details of a method of calculating a morphing rate vector for morphing performed by the synthesis method determination unit 116 will be described later.

  The character voice creation unit 102 further includes a voice DB update unit 122 that performs processing for registering a user voice stored in the user voice DB 120 as a new voice actor voice in the voice actor voice DB 80, and a user voice stored in the user voice DB 120. A unit DB update unit 124 is included for disassembling into phonemes (units) and adding them to the unit DB 82 together with their predetermined acoustic data, phoneme labels, and user IDs. In the segmentation of speech into segments by the segment DB update unit 124, speech recognition technology is used to subdivide the user's speech stored in the user speech DB 120 in accordance with the speech information stored in the speech information storage unit 72. Perform segmentation.

  FIG. 3 shows the configuration of a dialogue information table stored in the dialogue information storage unit 72. Referring to FIG. 3, the dialogue information storage unit 72 is for managing all dialogues of a movie to be created with serial numbers (No). Each line information includes a serial number of the line (hereinafter referred to as “line number”), a character ID for identifying a movie character that utters the line, text data that is the content of the line, and the line as a standard voice. The file name of the audio file in the standard audio storage unit 74 in which the utterance is recorded, the start time indicating when the dialogue is started during the progress of the movie, and the duration of the utterance And the utterance time shown. Since the dialogue information table of the dialogue information storage unit 72 has such a configuration, it is possible to list all dialogues of a certain character by extracting dialogues with the same character ID. Further, when a user's voice cannot be used for a certain line, the corresponding standard voice can be obtained from the voice file indicated by the voice file name.

  FIG. 4 shows some possible cases as a recording situation by the user in the multimedia production system 50 according to the present embodiment. For example, referring to FIG. 4A, it is assumed that the entire utterance to be recorded for a certain user forms utterance set 140. The utterance set 140 is necessary for speech synthesis, voice quality conversion, and the like, and includes an essential utterance portion 142 including utterances that should be recorded and a dialogue portion 144 including the entire dialogue of the corresponding character. Depending on the user's recording time, skill of the user's utterance, etc., the speech part 144 can be recorded in all three ways, when it can be recorded, when only part of it can be recorded, or when it cannot be recorded at all. possible. In FIG. 4, these cases are divided, and the recorded portions are hatched, and the portions that could not be recorded are illustrated as white.

  For example, FIG. 4A shows a case where the entire utterance set 140 has been recorded. FIG. 4B shows a case where only the essential utterance portion 142 and some dialogue portions 146 can be recorded, and the remaining portion 148 cannot be recorded. FIG. 4C shows a case where only the essential utterance portion 142 can be recorded and the other dialogue portion 150 cannot be recorded at all.

  In the case shown in FIG. 4A, the speech can be created basically using only the user's voice. However, even in this case, there is a case where it is necessary to change the speech speed or adjust the speech level by the skill of the user. They are different for each line.

  In the case shown in FIG. 4B, the recorded speech portion 146 can be processed in the same manner as in the case shown in FIG. 4A, but the speech portion 148 that could not be recorded is used by the user by some method. It is necessary to generate speech similar to the user's voice from other than the above voice.

  In the case illustrated in FIG. 4C, it is necessary to generate speech for all of the speech parts 150. In that case, for example, a feature amount representing the voice quality of the user is extracted from the essential utterance portion 142, and speech voices of voice actors of similar voice quality are extracted from the voice actor voice DB, or the voice quality of the standard voice is made close to the voice quality of the user. It is necessary to perform processing (voice quality conversion) for conversion.

  The technique list table 78 shown in FIG. 2 shows in what priority order such techniques are used for each dialogue. In the present embodiment, speech speech is generated using nine types of techniques. Details of these methods will be described later.

  5 and 6 are flowcharts of a computer program that realizes the sound recording processing performed by the sound recording unit 114 shown in FIG. 2 on computer hardware. As already mentioned, it is difficult to dubb a movie character line. For example, it is necessary to speak clearly in a certain time for a dialogue. Usually, if the utterance time is too long or too short, it may be inappropriate as a dubbing. In addition, it is difficult to make sure that a user who is not a voice actor performs speech dub. Therefore, in the present embodiment, various measures are taken so that desired speech can be recorded as accurately as possible. For example, as shown in FIG. 7, a video 246 when speech is spoken and a text 240 of speech to be spoken are displayed on the screen of the input / output device 112 presented to the user at the time of speech recording. A progress bar 242 that expands with the progress is displayed, or the color of the part 244 in the dialogue text 240 where the speech should end is displayed in a color different from the color of the part that should be spoken. Adopt the method.

  Referring to FIG. 5, this program receives user information from user information input unit 100 and saves it in a predetermined storage area. On the other hand, in step 172 for assigning the designated character, and following step 172, the dialogue information storage shown in FIG. 2 shows the common practice dialogue and the corresponding standard voice, the dialogue of the character assigned in step 172 and the corresponding standard voice. Step 174 extracted from the unit 72 and the standard voice storage unit 74, and following step 174, a table for managing the user's speech is created, which is called a user speech table, and all dialogues are initialized to an unrecorded state. Step 176.

  The user voice table constitutes a part of the user voice DB 120 shown in FIG. Referring to FIG. 8, the user voice DB 120 manages a voice file stored in the user voice storage unit 262 and a user voice storage unit 262 that stores a voice file in which a user's speech is recorded for each line. User voice table 260.

  The user voice table 260 is for managing the dialogue of the character that the user performs dubbing and the corresponding user voice. The user ID is attached to the head, and for each of the dialogue of this character, Stores the line number of the extracted line, the recording flag indicating whether or not the recording of the utterance of the line by the user has been completed, the name of the audio file storing the recorded utterance audio data, and the utterance time Is for. The recording flag indicates that the utterance voice has been recorded when it is 1, and indicates that it has not been recorded when it is 0. Actually, the utterance start time, utterance time, etc. need to be managed in units smaller than 1 second. However, in the following explanation and drawings, these times are managed in units of seconds for easy understanding. And

  Referring to FIG. 5 again, in step 176, the above-described user voice table 260 is newly created, and the serial number attached to the extracted dialogue is set as the dialogue number, all the recording flags are 0, and the voice file. Blanks are assigned to the names, and 0s are assigned to the utterance times.

  This program is further selected in step 176 following step 176, starting step 178 for measuring the time required for recording, step 180 selecting the first dialogue in the user voice table 260, and the immediately preceding step. Step 182 for displaying the text of the dialogue displayed on the monitor placed in front of the user, and Step 184 for retrieving the standard speech corresponding to this speech from the standard speech storage unit 74 and reproducing it. Also in steps 182 and 184, the display as shown in FIG. 7 is performed.

  In step 184, the program further includes step 186 provided as a user's speech practice time, and the attendant determines whether to return to step 182 to practice again or proceed to the next step. And step 188 for branching the flow of control according to the determination result to be input. When the determination result at step 188 indicates that the practice should be performed again, control returns to step 182.

  The program is further executed in response to the input indicating that the practice may be terminated in step 188, and again displays step 190 and the normal speech of the selected dialogue. And a step 192 of starting to display a progress bar that changes according to the speed.

  Next, referring to FIG. 6, this program is arranged next to step 192, and step 194 for recording the speech spoken by the user, step 196 for playing the speech recorded in step 194, and step Based on the utterance time of the speech spoken in 196, the clarity and naturalness of the speech, etc., the attendant determines whether or not to complete the recording of the speech, and the control flow branches according to the result input by the attendant Step 198 is executed in response to the input indicating completion of the recording of the dialogue in Step 198, and the voice recorded in Step 194 is stored in the user voice storage unit 262 as a voice file. Then, record the voice file name in the voice file name column of the dialogue in the user voice table 260 and record it in the utterance time column. The duration of the voice, includes a step 200 to assign each, and a step 201 to assign "1" to the recording flag.

  The program further, after step 200, attempts to select the next line of the target character in step 202, and whether or not the next line tried to be selected in step 202 exists, that is, the line of the target character. It is determined whether or not all processing has been performed, and in response to the determination in step 204 that the flow of control is branched in accordance with the determination result, and in step 204, it is determined that speech is still remaining, the timer is referred to, and recording is performed. Determining whether or not a predetermined time has elapsed from the start, and branching the control flow according to the determination result. If it is determined in step 212 that the predetermined time has not yet elapsed, control returns to step 182 in FIG.

  This program is further executed when it is determined in step 204 that the recording has been completed for all dialogues of the target character, and in response to the determination that the predetermined time has elapsed in step 212, and all recorded voices are recorded. Are segmented on the basis of the corresponding dialogue text and decomposed into speech segments, and for each of the segments generated in step 206, a predetermined acoustic feature quantity such as F0, spectral distribution, etc. is calculated. 208 and the step 210 where the segment created in step 206 is added to the segment DB 82 together with the acoustic feature amount calculated in step 208, the corresponding phoneme label, and the speaker ID. Including.

  The program is further executed in step 198 in response to the input made by the attendant indicating that the recording is to be redone, and disposed after step 214 and discarding the audio data recorded in step 194. It is determined whether or not a predetermined time has elapsed with reference to the time of the timer, and it is determined in step 216 that the control flow branches according to the determination result, and that the predetermined time has not yet elapsed in step 216 Steps for branching the control flow to step 190 when resuming from speech speech recording and to step 182 when resuming from speech practice according to an attendant input that determines when to resume processing, 220, and the predetermined time has already passed in step 216 Be performed in response to the determination that there, in step 220 if the speech is an integral part of the current recording, the step 206 and otherwise, and a step 218 for branching each control.

  FIG. 9 shows a more detailed block diagram of the voice creation unit 118 shown in FIG. Referring to FIG. 9, the speech creation unit 118 includes first to ninth speech generation units 300, 302, 304, 306, 308, for generating speech speech by the first to ninth methods, respectively. 310, 312, 314, and 316, and the first to ninth speech generation units 300, 302, 304, 306, 308, 310, 312, 314, and 316 according to the method determined by the synthesis method determination unit 116. One of the above, and a voice generation selected by the branching unit 280 according to a method determined by the synthesis method determining unit 116 and a branching unit 280 that generates a voice by a method specified by giving a user voice The merging unit 292 that selects the dialogue voice data that is the output of the part and outputs it to a common output, and the dialogue voice data that is output by the merging unit 292 is cut And a sound signal processing unit 320 to output the added sound effects to be designated in accordance with the cut information stored in the broadcast storage unit 76.

  The first sound generation unit 300 is a technique when a user's line sound can be recorded for a certain line. In this case, in principle, the recorded voice is used as it is.

  The second sound generation unit 302 is also a technique when the user's line sound can be recorded for a certain line. However, in this method, speech is generated by adjusting the utterance speed of the recorded speech.

  The third sound generation unit 304 is an effective technique even when the user's speech is not recorded for some of the speech. In this method, the speech that has been recorded is converted to the speech speed of the user's speech, and speech is generated. For speech that could not be recorded, speech of the user's voice (sex, age, etc.) that matches the user information out of the standard speech stored in the standard speech storage unit 74 is used without using the user's speech.

  The fourth sound generation unit 306 is also an effective technique even when the user's speech is not recorded for some of the speech. In this method, the speech that has been recorded is converted to the speech speed of the user's speech, and speech is generated. For speech that could not be recorded, speech speech of a voice actor having a voice quality closest to the user's speech among speech speech by voice actors stored in the voice actor speech DB 80 is employed. To determine the voice actor voice at this time, the linearity of the distance calculated with the voice to be compared for the eight types of feature quantities (basic frequency, spectrum distribution, etc.) of the user voice obtained from the practice dialogue. A distance comparison between voices using sums is used. At this time, the linear combination coefficient stored in the linear combination coefficient storage unit 94 shown in FIG. 1 is used as the coefficient of the linear sum of these feature quantities. However, in the present embodiment, this comparison is performed by the synthesis method determination unit 116 shown in FIG. 2, and the resulting voice actor speech identifier is stored in the similar voice actor storage unit 130. The fourth voice generation unit 306 uses this information. The same applies to a fifth sound generation unit 308, a seventh sound generation unit 312 and an eighth sound generation unit 314 which will be described later. In the determination of the similar voice actor voice, the voice of the essential utterance portion and the corresponding speech voice of the voice actor voice among the user's utterances are used.

  The fifth voice generation unit 308 is also an effective technique when it is not possible to record the user voice for some of the lines. In this method, the speech that has been recorded is converted to the speech speed of the user's speech, and speech is generated. For the dialogue that could not be recorded, among the speech voices by voice actors stored in the voice actor voice DB 80, the three voice voices that are most similar to the user voice are identified, and the voice quality of the user is further added to the speech voice. The reflected voice quality is converted into speech. To identify similar voice actor voices, the voice actor voice identifiers stored in the similar voice actor storage unit 130 are used as described above. The voice quality conversion uses the morphing rate vector determined by the synthesis method determination unit 116 stored in the morphing rate storage unit 132 for the three voice actor voices corresponding to the identifiers stored in the similar voice actor storage unit 130. This is performed by morphing by a speech analysis conversion synthesis system STRIGHT (http://www.wakayama-u.ac.jp/~kawahara/STRAIGHTadv/).

  The sixth sound generation unit 310 is also an effective technique when the user's speech is not recorded for some of the speech. In this method, the speech that has been recorded is converted to the speech speed of the user's speech, and speech is generated. For dialogue that could not be recorded, among the speech segments generated from the recorded user speech, among the speech units of the vowels and the speech units of all consonants stored in the segment DB 82, the user Speech synthesis is performed using speech segments having feature quantities similar to the speech. Since the personal characteristics of the utterance appear mainly in the vowels, by performing such speech synthesis, it is possible to generate synthesized speech with a voice quality much like the user's speech.

  The seventh voice generation unit 312 is a method employed when no speech other than the essential utterance part can be recorded. In this method, a voice whose voice quality is most similar to the user voice among voice actor voices stored in the voice actor voice DB 80 is used as a speech voice.

  The eighth voice generation unit 314 is also an effective technique when no speech other than the essential utterance part can be recorded. In this method, speech that is most similar to the voice of the user voice among voice actor voices stored in the voice actor voice DB 80 is used, and voice quality conversion using the voice quality of the user voice is further performed on the voice actor voice to generate speech voice. To do.

  The ninth voice generation unit 316 is also an effective technique when no speech other than the essential utterance part can be recorded. In this method, among speech units generated from user speech recorded for essential utterance parts, vowel speech units and consonant speech units stored in the unit DB 82 are similar to the user's speech. Speech synthesis is performed using speech segments having feature quantities. As described above, by performing such speech synthesis, synthesized speech having a voice quality much similar to that of the user can be generated.

  As is clear from the description of each method described above, the dialogue information stored in the dialogue information storage unit 72 is the first to ninth speech generation units 300, 302, 304, 306, 308, 310, 312, 314, and Referenced by all of 316. The standard voice stored in the standard voice storage unit 74 is referred to by the third voice generation unit 304. The voice actor voice stored in the voice actor voice DB 80 is referred to by the fourth voice generator 306, the fifth voice generator 308, the seventh voice generator 312, and the eighth voice generator 314. The element DB 82 is referred to by the sixth sound generation unit 310 and the ninth sound generation unit 316. The similar voice actor storage unit 130 is referred to by the fourth voice generation unit 306, the fifth voice generation unit 308, the seventh voice generation unit 312, and the eighth voice generation unit 314. The morphing rate storage unit 132 is referred to by the fifth sound generation unit 308 and the eighth sound generation unit 314.

  FIG. 10 is a flowchart of a computer program that implements, on computer hardware, the speech generation method determination process performed by the synthesis method determination unit 116 shown in FIG. Referring to FIG. 10, this program has the same composite acoustic feature obtained from a composite acoustic feature amount composed of a linear sum of eight types of acoustic feature amounts obtained from user speech and each voice actor speech stored in voice actor speech DB 80. The three voice actor voices most similar to the user voice are identified by the distance comparison using the quantity, and the identifiers of these voice actor voices are stored in the similar voice actor storage unit 130 shown in FIG. A step 332 of calculating a morphing rate vector when synthesizing a voice similar to the voice quality of the user voice by morphing using the three voice actor voices and storing it in the morphing rate storage unit 132 shown in FIG. . Details of step 330 and step 332 will be described later.

  The program further includes step 340 for substituting 0 for variable i for controlling the following iteration, step 342 for adding 1 to variable i, and whether or not the value of variable i exceeds the number of lines MAX. If it is determined that the value of the variable i is determined to be less than or equal to MAX in step 344, the process is terminated. And a step 346 of reading out a method list corresponding to “line (i)” from the method list table 78 and storing it in the work list variable WLIST.

  Details of the technique list table 78 are shown in FIG. Referring to FIG. 11, method list table 78 includes a method list that lists identifiers of available methods for each line number. Normally, the method list table 78 is created in advance so that dialogue can be processed without fail using any of the methods listed in this method list. However, a default method of outputting, for example, standard speech speech is prepared in advance, including cases where usable methods are not included in the method list.

  Referring again to FIG. 10, this program is further arranged after step 346, substituting the number of elements of the list variable WLIST into the variable CMAX, step 348, and step 348 to the variable j that controls the following iterations And step 350 for substituting 0. In general, the index of the element of the list variable starts from 0.

  In step 350, the program further determines whether j + 1 exceeds the value of CMAX, and branches the control flow according to the determination result. In step 352, it is determined that j + 1 is equal to or less than CMAX. It is executed in response to this, and it is determined whether or not the technique indicated by the list element WLIST [j] among the list variables WLIST can be realized by a given user voice, and the control is branched according to the determination result. Step 354. Whether or not each of these methods can be adopted depends on the recording situation of the speech to be processed. Basically, the first method and the second method cannot be used unless the corresponding speech is recorded, but other methods can be used even if the corresponding speech is not recorded. Is possible. The reason will be clear from the explanation of each method.

  The program is further executed in response to determining in step 354 that the technique indicated by list element WLIST [j] is not available, adding 1 to the value of variable j, and returning control to step 352. Executed in response to determining that the technique indicated by list element WLIST [j] is available in step 356 and step 354, using the technique indicated by WLIST [j] Processing and returning control to step 342, executed in response to determining that the value of j + 1 is greater than CMAX in step 352, processing line (i) in a default manner, and controlling And step 360 of returning to 342.

  FIG. 12 shows the configuration of the cut information storage unit 76 shown in FIG. Referring to FIG. 12, the cut information storage unit 76 stores, for each dialogue, a dialogue number and an acoustic effect list listing the acoustic effects to be applied to the dialogue. When an acoustic effect is to be applied to a certain dialogue, the audio signal processing unit 320 examines the acoustic effect list corresponding to the dialogue number to be processed in the cut information storage unit 76, and sequentially executes them from the top.

  FIG. 13 is a flowchart of a program that implements the first sound generation unit 300 shown in FIG. Referring to FIG. 13, this program includes a step 380 of reading the speech voice (i) from the user voice DB. Step 380 ends the process. The read speech (i) is given to the audio signal processing unit 320 and processed. Details of the processing of the audio signal processing unit 320 will be described later with reference to FIG.

  This first technique is a technique when the user's voice can be recorded for the target dialogue, and the user's voice is used as it is as the dialogue voice.

  FIG. 14 is a flowchart showing a control structure of a program for realizing the second sound generation unit 302 shown in FIG. Referring to FIG. 14, this program reads the user's speech voice (i) and its speech time from user speech DB 120, and continues to step 410, and the speech time of speech (i) from the speech information table. Using the utterance time ti read at step 410 and the utterance time Ti read at step 410 and the utterance time Ti read at step 412, the speech speed conversion is performed so that the utterance time of the user's speech (i) is changed from ti to Ti. And 414 to end the process.

  FIG. 15 is a flowchart of a program for realizing the third sound generation unit 304 shown in FIG. Referring to FIG. 15, this program reads out recording flag (i) from user voice table 260 of user voice DB 120, and determines whether the value of the recording flag read out in step 440 is 1 or not. And step 442 for branching the control flow according to the result.

  This program is further executed in response to the determination that the recording flag is not 1 in step 442 (that is, the user's voice cannot be recorded for this line), and the line (i) of the line (i) is read from the standard voice storage unit 74. The standard voice is read and output as a dialogue voice (i), and the process is terminated in response to the determination that the recording flag is 1 in step 444 and the processing, and the dialogue voice ( i) and the utterance time ti are read in step 446, the utterance time Ti of the line (i) is read from the line information table stored in the line information storage unit 72, and read in steps 446 and 448, respectively. The speech speed conversion is performed so that the speech time of the user's speech (i) becomes Ti using the spoken time ti and Ti. It is output, and a step 450 to end the process.

  FIG. 16 is a flowchart showing a control structure of a program for realizing the fourth sound generation unit 306 shown in FIG. Referring to FIG. 16, this program reads out recording flag (i) for the i-th speech voice from user voice table 260 of user voice DB 120, and recording flag (i) read in step 470. The control flow is executed depending on whether the value of the recording flag (i) is not 1 (that is, 0) in step 472 where the flow of control is branched depending on whether the value is 1 or not. Among the voice actor voices of the line (i) stored in the voice DB 80, the voice voice having the most similar voice quality to the user voice is read and output as the line voice (i), and the process is terminated 474. As described above, the voice actor voice identifier stored in the similar voice actor storage unit 130 (FIG. 9) is used to read voice actor voices having similar voice qualities. The same applies to step 504 in FIG. 17, step 560 in FIG. 19, and step 580 in FIG. 20 described below. Therefore, the above description will not be repeated for those portions.

  This program is further executed in response to the determination that the recording flag is 1 in step 472, and reads the speech (i) and the speech time ti from the user voice DB 120, and in step 476 Subsequently, step 478 for reading the speech time Ti of the speech (i) from the speech information table of the speech information storage unit 72, and the speech of the user speech (i) read at step 476 using the speech times ti and Ti. Step 480 which performs speech speed conversion so as to change the time from ti to Ti and outputs as speech (i) and ends the process.

  FIG. 17 is a flowchart showing a control structure of a program that realizes the fifth sound generation unit 308 shown in FIG. Referring to FIG. 5, this program reads step 500 for recording flag (i) from user voice DB 120, determines whether or not the value of the recorded recording flag is 1, and performs control according to the determination result. Of the voice actor voices in the line (i), which is executed in response to the branching of the flow 502 and the fact that the value of the recording flag is determined not to be 1 in step 502 and stored in the voice actor voice DB 80, the user Step 504 that identifies the voice quality most similar to the voice quality of the voice of the line (i) identified in Step 504 is voice-converted using the characteristics of the user voice, and is output as the line voice (i) for processing. And ending step 506.

  This program is also executed in response to determining that the recording flag is 1 in step 502, reading the speech (i) and speech time ti from the user speech DB 120, and speech information storage. In step 510, the speech time Ti of the speech (i) is read from the speech information table of the section 72, and the speech speed is converted so that the speech time of the user speech speech (i) is changed from ti to Ti. and step 512 for outputting as i) and ending the processing.

  FIG. 18 is a flowchart of a program for realizing the sixth sound generation unit 310 shown in FIG. Referring to FIG. 18, this program reads step 530 of recording flag (i) from user voice DB 120, determines whether the value of this recording flag is 1, and branches the control flow according to the determination result. Step 532, and in response to the determination that the value of the recording flag is not 1 in step 532, the line (i), the feature amount of the user speech, the speech unit of the user vowel, and the speech unit DB 82 And step 534 for generating and outputting speech speech (i) by performing speech synthesis using consonant speech units.

  This program is further executed in response to the determination that the recording flag = 1 in Step 532, and reads the speech (i) and the speech time ti from the user speech DB 120, and the speech information storage. Step 538 for reading the speech time Ti of the speech (i) from the speech information table of the section 72, and the speech speed of the user speech speech (i) so that the speech time of the user speech speech (i) is changed from ti to Ti. Step 540 of performing conversion and outputting as speech (i).

  FIG. 19 is a flowchart showing a control structure of a program that implements the seventh sound generation unit 312 shown in FIG. Referring to FIG. 19, this program reads the speech most similar to the voice quality of the user speech from speech (i) in voice actor speech DB 80, outputs it as speech speech (i), and ends the processing. Step 560 is included.

  FIG. 20 is a flowchart showing a control structure of a program that realizes the eighth sound generation unit 314 shown in FIG. Referring to FIG. 20, this program specifies and reads out the voice most similar to the voice quality of the user voice among voice actor voices of dialogue (i) stored in voice actor voice DB 80, Step of generating and outputting the user's speech (i) by converting the voice (i) read in 580 into a voice quality close to the user's voice quality using the voice characteristics of the user's utterance essential part 582.

  FIG. 21 is a flowchart of a program for realizing the ninth sound generation unit 316 shown in FIG. Referring to FIG. 21, this program uses the dialogue (i), the feature amount of the user voice, the speech unit of the user's vowel, and the speech unit of all consonants stored in the unit DB 82. ) Is synthesized and output as speech (i), and the process ends.

  FIG. 22 is a flowchart of a program that implements the audio signal processing unit 320 shown in FIG. The audio signal processing unit 320 performs the following processing on the speech (i) output from the merging unit 292. That is, the program reads the sound effect list ELIST of the line (i) from the cut information storage unit 76, and after step 382, substitutes the number of elements of the sound effect list ELIST into the variable EMAX, step 384. Subsequent to step 384, step 386 is performed following step 386, in which 0 is substituted for variable k for controlling subsequent iterations. Following step 388, it is determined whether the value of k + 1 is greater than EMAX. Step 390 for branching control according to the determination result, and executed in response to the determination that the value of k + 1 is equal to or less than EMAX in Step 390, and the acoustic effect of ELIST [k] is added to the speech speech (i). After adding step 392 and step 392, 1 is added to the value of variable k and step 390 is performed. And a step 388 to return the control.

  The program is further executed in response to determining that the value of k + 1 is greater than EMAX in step 390, and writing the speech speech (i) to a speech file; after step 394, the speech speech table 88 And 396 to update the speech file name of the line (i) with a new file name and end the process.

  The structure of the speech table 88 updated in step 396 is shown in FIG. Referring to FIG. 24, the speech sound table 88 includes speech number 86, speech playback start time, speech playback (utterance) time, and speech speech (speech speech) stored therein. File name and playback flag. The dialogue reproduction start time is a time determined so that the beginning of a movie to be created is a predetermined time, and reproduction of the dialogue is started based on that time. Playback time refers to the duration of dialogue playback. The playback file name is the file name of the file storing the speech in the speech audio data 86 as described above. If the playback flag is 0, it indicates that audio is played back when the movie is played back, and if it is 1, it indicates that playback is not performed. As will be described later, this reproduction flag is used to realize a voice overlap (two or more characters speak at the same time). The method will be described later.

  FIG. 23 is a block diagram of a playback system for playing back a movie created by the multimedia production system 50 according to the present embodiment. Referring to FIG. 23, this playback system is played back by video signal playback unit 620 for outputting a video signal, a video / synchronization signal, and a sound effect sound signal from video data 66, and video signal playback unit 620. A display device 622 for reproducing the reproduced video signal and displaying the video, a sound effect output device 624 for converting the sound signal of the sound effect output from the video signal reproducing unit 620 into sound, and outputting the sound. Prior to playback, speech speech data 86 and speech speech table 88 are received as input, and speech that should be spoken at the same time based on the speech start time and speech time of each speech stored in speech speech table 88. Is detected, and the voices of those voice files are synthesized to create a new voice file, which is replaced with one voice file in the line where the overlap is detected. Further, by updating the speech flag of the other speech voices to “1”, the speech speech data 86 and the speech speech table 88 for updating the speech speech data 86 and the speech speech table 88 so that speech speech spoken at the same time is integrated. 632.

  The reproduction system further receives a synchronization signal from the video signal reproduction unit 620 at the time of reproduction, refers to the dialogue sound table 88, and is a speech sound at an utterance start time coinciding with the time indicated by the synchronization signal, and corresponds. A synchronous reproduction unit 638 for detecting a reproduction flag of “0”, reading out from the speech audio data 86, reproducing and outputting the audio signal, and converting the audio signal output from the synchronous reproduction unit 638 into audio And a speech output device 640 for outputting.

  In other words, this playback system generates sound effects and line speech completely separately, and plays the speech in order according to the order of the utterance start times. Therefore, it is possible to play a movie in which the sound of the characters and the face image are replaced with those of the user while utilizing the sound effects.

  FIG. 24 shows the structure of the speech table 88 as described above. FIG. 25 shows the configuration of the speech table 88 after the speech (utterances 1, 2 and 3) having the same speech time in the speech table 88 shown in FIG. .

  Referring to FIG. 25, the structure itself of dialogue speech table 88 is the same as that before the update. The difference is that the playback time of dialogue 1 has increased from 7 seconds to 11 seconds, the playback file name of dialogue 1 has been changed from “wave0001.wav” to “comb0001.wav”, and This means that the playback flags of lines 2 and 3 have been changed from “0” to “1”. This is due to the following reason.

  In the speech table 88 shown in FIG. 24, the playback start time of speech 1 is 0: 0: 3 and the playback time is 7 seconds, so the playback end time is 0: 0: 10. On the other hand, the playback start time of dialogue 2 is 0: 0: 8, and the playback time is 5 seconds, so the playback end time is 0: 0: 13. Then, the speech time of line 1 and the speech time of line 2 partially overlap each other. In the present embodiment, for dialogues in which utterance times overlap with each other in this way, those speeches are integrated into a new speech file, and one speech file (usually the one with the earliest playback start time) ) And the utterance time is updated with the utterance time of the new audio file. The playback flag is set to 1 for the other speech file.

  In the example shown in FIG. 24, since the playback times of dialogues 1, 2, and 3 overlap, these are integrated, and finally, the reproduction time of dialogue 1 is 11 seconds, dialogue 2 and dialogue, as shown in FIG. 3 is 1 (that is, no reproduction is performed).

  FIG. 26 is a flowchart of a program for realizing the simultaneous audio integration processing unit 632. Referring to FIG. 26, this program substitutes 0 as an initial value for variable X representing the line number of the line being processed, step 662 for adding 1 to this variable X, and processing of step 662 Receiving the result, determining whether or not the speech of the Xth dialogue (X) exists (that is, whether or not all dialogue speech has been processed) and branching the control according to the judgment result; including. In step 664, if all the speech sounds have been processed, the process is terminated.

  This program is further executed in response to the determination that the speech line (X) is present in step 664, and determines whether or not the value of the playback flag of the speech line (X) in the speech line table 88 is 0. And step 666 of branching the control according to the determination result. If it is determined in step 666 that the playback flag is not 0, it is not necessary to play the speech (X). Therefore, in this case, the control returns to step 662 and proceeds to the next dialogue speech processing.

  This program is further executed in response to the determination that the speech flag value of the speech line (X) is 0 in step 666, and determines whether or not the speech line (X) and the speech overlap. Step 668 for substituting the value of X into the variable Y indicating the line number of the line speech, step 670 for adding 1 to the value of this variable Y after step 668, and the processing result of step 670, Y) is present, that is, it is determined whether or not the processing for checking the overlap with the speech (X) for all speech is completed, and the flow of control 672 is branched according to the determination result. Including. If it is determined in step 672 that the Yth line does not exist, control returns to step 662.

  This program is further executed in response to the determination that the Yth speech line is present in step 672, determines whether the value of the playback flag of the speech line (Y) is 0, and depends on the determination result. Step 674 for branching the control flow. When it is determined in step 674 that the value of the playback flag of the speech line (Y) is not 0, the control returns to step 670, and the process shifts to a process for examining the overlap of the next speech line with the speech line (X).

  This program is further executed in response to determining that the value of the playback flag of the speech line (Y) is 0 in step 674, and the speech start times of both speech lines stored in the speech line table 88. And a step 676 of determining whether or not at least a part of the speech time of the dialogue (X) and the dialogue (Y) overlaps based on the value of the speech time and branching the control according to the judgment result. If it is determined in step 676 that the speech times do not overlap, control returns to step 670.

  This program is further executed in response to the determination that at least a part of the speech time of the dialogue (X) and dialogue (Y) overlaps in step 676, and dialogue speech (X) and dialogue speech ( Y) is mixed to create a new speech, and the speech speech data 86 is updated as speech speech (X), and the speech time t of this new speech speech (X) is set to the value before duplication correction. It is calculated as follows between the speech time tx of the speech speech (X) and the speech time ty of the speech speech (Y), and this is calculated as the speech time tx of the new speech speech (X). Following step 680, the table 88 is updated, and following step 680, the value of the speech flag (Y) playback flag in the speech table 88 is updated to "1", and control returns to step 670.

  FIG. 27 is a flowchart of a program for realizing the synchronous playback unit 638 shown in FIG. Referring to FIG. 27, in this program, step 700 for reading the synchronization signal provided from video signal reproduction unit 620 shown in FIG. 23 and the time indicated by the synchronization signal read in step 700 are the speech sound table 88. And step 702 for determining whether or not the speech start time has been reached for any of the dialogues stored in and whose playback flag is 0, and branching the control flow according to the determination result. If it is determined in step 702 that the time indicated by the synchronization signal is not the playback start time of any speech sound, control returns to step 700 and the synchronization signal is read again.

  This program is further executed in response to the determination in step 702 that the time indicated by the synchronization signal has reached the speech start time of any speech, starts playback of that speech, and performs control. And step 704 which returns to step 700.

  FIG. 28 is a diagram for explaining the contents of speech speed conversion and volume normalization processing among the sound effect processing executed by the audio signal processing unit 320.

  Referring to FIG. 28 (A), the speech speed conversion process is the recording time of recorded voice 722 in FIG. 28 (A) compared to the utterance time of reference voice 720, which is the standard of speech time of dialogue. This is a process of generating a corrected speech 724 having a speech time equal to the speech time of the reference speech 720 by converting the speech speed of the recorded speech 722 when it is too short or too long as shown. . The existing speech speed conversion technology can be used for the speech speed conversion.

  FIG. 28B shows volume normalization. Compared with the average level L0 of the reference voice 740, when the average level L1 of the recorded voice 742 is too low or too high as shown in FIG. In the sound volume normalization process, the average level L3 which is substantially equal to the average level L0 is set. In such volume normalization processing, the volume of audio recorded by multiple users must not vary, and conversely, depending on the scene, it is necessary for the user to make a difference in the volume of audio. To be done. For this volume normalization, the existing technology can be used.

[Realization by computer]
FIG. 29 is an external view of the hardware of the speech voice data creation unit 90 for recording the user's voice in the multimedia production system 50. Referring to FIG. 29, the speech sound data creation unit 90 is substantially composed of a computer system 830. FIG. 30 shows the internal configuration of the computer system 830.

  Referring to FIG. 29, a computer system 830 includes a computer 840 having a memory port 852 for removable memory and a DVD (Digital Versatile Disc) drive 850, a keyboard 846 for inputting character information and command operations, and pointing. The device includes a mouse 848, two monitors 842 and 844, two microphones 868 and 870, and two sets of speakers 872 and 874. Among these, the monitor 844, the speaker set 874, and the microphone 868 are separated from the main body of the computer system 830 and are installed in the user's recording booth as shown in FIG. Used as an input / output interface with the user.

  Referring to FIG. 30, a computer 840 includes a memory port 852, a DVD drive 850, microphones 868 and 870, speaker sets 872 and 874, a CPU (central processing unit) 856, a CPU 856, and a memory port. A bus 866 connected to the 852 and the DVD drive 850, a read only memory (ROM) 358 for storing a boot-up program and the like, and a random access connected to the bus 866 for storing a program command, a system program, work data and the like. A memory (RAM) 860 and a sound board 884 connected to a bus 866, microphones 868 and 870, and speaker sets 872 and 874 are included.

  Computer 840 further includes a network interface card (NIC) 878 that provides a connection to a local area network (LAN) 876 for communicating with other computers.

  The various computer programs described above for causing the computer system 830 to operate as the multimedia production system 50 are stored in the DVD 862 or the removable memory 864 inserted into the DVD drive 850 or the memory port 852, and further stored in the hard disk 854. Transferred. Alternatively, the program may be transmitted to the computer 840 through a network (not shown) and stored in the hard disk 854. The program is loaded into the RAM 860 when executed. The program may be loaded into the RAM 860 directly from the DVD 862, from the removable memory 864, or via the network 876.

  These programs include a plurality of instructions that cause the computer 840 to operate as the multimedia production system 50 of this embodiment. Some of the basic functions required to perform this operation are an operating system (OS) or a third party program running on the computer 840, or various tools for voice processing and statistical model processing installed on the computer 840. Provided by module of kit. Therefore, this program does not necessarily include all functions necessary for realizing the system and method of this embodiment. This program calls out the appropriate “tools” in a so-called “tool kit” which is a collection of computer programs with appropriate functions or prepared in advance in a controlled manner to obtain the desired result. Therefore, it is only necessary to include an instruction for executing the operation as the above-described speech sound creation device. The operation of computer system 830 is well known and will not be repeated here.

  In the system shown in FIG. 1, the dialogue voice data creation unit 90 realizes a plurality of computer systems having the same configuration as the computer system 830 and a voice integration unit 104 for recording for each user. One computer system for The computer system that implements the voice integration unit 104 has the same hardware configuration as the computer system 830, but does not require a microphone, a speaker, or the like.

  Further, in the present embodiment, in the video / audio playback device 92 shown in FIG. 23, the video signal playback unit 620 is realized by one computer system, and the simultaneous audio integration processing unit 632 and the synchronous playback unit 638 are different. This is realized by one computer system.

  All of the computer systems used in this system communicate with each other via the network 876, and finally create the video data 66, the speech audio data 86, and the speech audio table 88 on the hard disk of the video / audio reproduction device 92. And play from there.

[Operation]
The multimedia production system 50 whose configuration has been described above operates as follows. The operation of the multimedia production system 50 when selecting the voice actor voice most similar to the user voice and determining the morphing rate will be described in detail later. Further, calculation of a linear combination coefficient when calculating a composite acoustic feature amount by linearly combining eight types of acoustic feature amounts will be described later. In the following description, it is assumed that the linear combination coefficient is already calculated and stored in the linear combination coefficient storage unit 94.

  Referring to FIG. 1, it is assumed that a plurality of users use the multimedia production system 50, and identification information is assigned to each user in advance. Further, it is assumed that each user is determined as to whom a character in the movie is to be replaced.

  In the multimedia production system 50, video material is stored in advance in the video material DB 70, dialogue information is stored in the dialogue information storage unit 72, and standard speech corresponding to men, women, and ages for each dialogue in the standard audio storage unit 74. However, the sound effect information is stored in the cut information storage unit 76, respectively. Also, it is assumed that the voice actor voice DB 80 stores each speech uttered by a plurality of voice actors for each speech and each voice actor. The voice of each voice actor is subjected to acoustic analysis in advance, and eight types of acoustic feature quantities representing the respective voice qualities are calculated and stored in the voice actor voice DB 80. The unit DB 82 stores speech units created by segmenting standard speech and voice actor speech. Each speech segment is assigned a phoneme label of the corresponding phoneme, an acoustic feature, an original speech identifier, and a speaker identifier.

  User information of each user is input by the user information input units 100, 100A,..., 100N, and sent to the corresponding one of the image processing PC 62 and the plurality of character voice generation units 102, 102A,.

  The three-dimensional scanner group 60 scans each user's face and sends three-dimensional scan data to the image processing PC 62. Thereafter, the image processing PC 62 creates the user's three-dimensional face model using the user's three-dimensional scan data, further creates a three-dimensional face image from an arbitrary angle, and gives it to the video generation device 64. The video generation device 64 replaces the character's face image with the user's face image created by the image processing PC 62, and outputs it as video data 66. Note that the video data 66 includes synchronization signal reproduction data for synchronizing with audio.

  On the other hand, each of the plurality of character voice generation units 102, 102A,..., 102N records the corresponding user's speech as follows, and based on the recorded voice, the first voice generation unit 300˜ The ninth sound generation unit 316 is used to create and output movie sound data utilizing the user's voice. The processes of the first sound generation unit 300 to the ninth sound generation unit 316 at this time are the same. Hereinafter, the operation of the character voice creation unit 102 will be described.

  With reference to FIG. 2, the sound recording unit 114 receives user information from the user information input unit 100 (step 170 in FIG. 5), and uses this user information in the subsequent processing. Subsequently, information relating to the character assigned to the user is input (step 172 in FIG. 5). The voice recording unit 114 reads the dialogue information related to the input dialogue of the character from the dialogue information storage unit 72, reads the corresponding standard voice from the standard voice storage unit 74, and if there is a corresponding video from the video material DB 70, respectively. (Step 174 in FIG. 5). The voice recording unit 114 further creates a user voice table 260 and initializes the recording flags of all dialogue information to zero.

  Next, the audio recording unit 114 starts a timer (step 178) and starts recording lines. In the recording of dialogue, the dialogue target speech is selected (step 180), video and dialogue information are displayed (step 182), and reproduction of standard audio is started at the same time. As a result, the display as shown in FIG. 7 is performed on the screen of the input / output device 112 (the screen of the monitor 844). Thereafter, the user imitates the standard voice and utters it as practice (step 186).

  The attendant listening to the user's utterance while operating the computer system 830 determines whether or not to practice the utterance (step 188), and if further practice is needed (step 188). In NO, the same operation is repeated for the utterance. If it is determined that the practice can be finished (YES in step 188), the selected dialogue and the corresponding video are displayed (step 190), and the progress bar is displayed (step 192). Audio is recorded (step 194).

  If the recorded voice is correct, the utterance is clear, and the utterance time is within an allowable range, the attendant saves the recorded voice as a voice file in the user voice storage unit 262, and the user whose configuration is shown in FIG. The speech file name stored in the user speech storage unit 262 is substituted into the speech file name column of the speech line being processed in the speech table 260, and the actual speech time (ti) of the user speech speech in the speech time column. Is substituted (step 200). Furthermore, the audio recording unit 114 updates the recording flag of the line to 1 (step 201), and selects the next line (step 202). If the recording of the user's speech for all lines has been completed (YES in step 204), all the recorded user's utterances are segmented into phonemes (step 206). Is calculated (step 208) and added to the segment DB 82.

  If it is determined in step 204 that the recording of speech for all the lines has not been completed yet, in step 212, a timer is referred to and it is determined whether or not a predetermined time has been exceeded. To do. If it has exceeded, the process proceeds to step 206, and thereafter the same processing as when recording is completed for all lines is performed. If the predetermined time has not yet been reached, the process returns to step 182 in FIG. 5, and the above-described processing is repeated for the next line of the character corresponding to this user.

  In step 198, the attendant determines that the recorded voice is not preferable (for example, the content is significantly different from the original utterance text, the utterance is unclear, or the utterance time is outside the allowable range). If so, the recorded sound is discarded in step 214. Subsequently, the timer is checked to determine whether or not the time for recording has been exceeded (step 216). If the time has not exceeded, the attendant determines whether to start again from the speech of the speech being processed (step 182) or simply from the recording of the utterance by the user (step 190), and instruct according to the determination result. input. The audio recording unit 114 branches the control according to the instruction (step 220), and as a result, the processing is resumed from step 182 or step 190.

  On the other hand, if it is determined in step 216 that the time required for recording has already exceeded the predetermined time, it is determined in step 218 whether the currently recorded dialogue is an essential portion of dialogue. If it is an indispensable part, this recording must be performed. Therefore, the control proceeds to step 220 and the recording is resumed according to the determination of the attendant. If it is not an essential part, the recording operation should be terminated, and control proceeds to step 206. Thereafter, the same operations as those performed when the recording of all dialogues is completed are executed in steps 206, 208 and 210.

  In this way, the voice recording unit 114 stores the voice file of the user's speech for a certain character's speech in the user speech storage unit 262 shown in FIG. 8, and the user speech table 260 can record each speech. The recording flag indicating whether or not, the name of the corresponding voice file in the user voice storage unit 262, and the utterance time by the user are recorded.

  As a result of each of the plurality of character voice creation units 102, 102A,..., 102N executing the above-described processing, the speech of each character is stored in the user voice DB 120 (user voice table 260 and user voice storage unit 262). Is output in the form. The voice integration unit 104 integrates the speech voices of the users of these various characters so that they can be read in a predetermined order based on the dialogue information stored in the dialogue information storage unit 72, and the dialogue voice data 86 and the dialogue voice table 88 are integrated. Output. In this way, when the sound recording for the target user is completed, the sound recording unit 114 instructs the synthesis method determination unit 116 to start generating speech.

  In response to this instruction, the corresponding synthesis method determination unit 116 executes the following processing. Referring to FIG. 10, in step 330, three voice actor voices most similar to the user voice are selected from voice actor voices stored in voice actor voice DB 80, and their identifiers are assigned to similar voice actor storage unit 130 (FIG. 2). ). In step 332, a morphing rate vector r for generating a voice similar to the voice quality of the user voice from these three voice actor voices by morphing is estimated and stored in the morphing rate storage unit 132 (FIG. 2). Details of the processing of step 330 and step 332 will be described later together with the processing performed by other processing units. Further, at step 340 to step 344, the first dialogue among the dialogues to be processed is selected. Then, the method list table 78 is searched using the line number of the line as a key, and the method list WLIST for the line is obtained.

  Subsequently, by the processing in steps 348 to 354, the methods described in the method list WLIST are checked in order from the top, and the first method found using the available methods is used, and the processing target dialogue is processed by that method. Is determined, and information for specifying the method is given to the voice creating unit 118 for processing. The method list is created so that there is always a method list that can be used, but even if there is no method list, it is possible to generate speech using the default method.

  In this way, the first line in the processing target corresponds to the method selected in the first to ninth sound generation units 300 to 316 of the sound generation unit 118 based on the user's recorded sound. Is instructed to generate speech. At this time, the synthesis method determining unit 116 controls the branching unit 280 to give the user voice to the selected voice generation unit, and to control the merging unit 292 so as to select and output the output speech. Thus, when generation of speech is started for the first dialogue, the synthesis method determination unit 116 restarts the process from step 342 again, determines the speech generation method for the next dialogue, and sends the dialogue to the corresponding speech generation unit. Generate sound. When the generation of speech for all the dialogues of the target character is completed in this way, the processing of the synthesis method determination unit 116 is finished.

  With reference to FIG. 9, the voice creation unit 118 operates as follows. The branching unit 280 activates the designated voice generation unit according to an instruction from the synthesis method determination unit 116, and gives a user voice. Among the first voice generation unit 300 to the ninth voice generation unit 316, the activated one generates a speech line using each method based on the given user voice. The output speech is selected by the merging unit 292 and given to the audio signal processing unit 320.

  Here, when the first method is selected, the first voice generation unit 300 shown in FIG. 9 reads the line voice (i) from the user voice DB 120 (step 380). Step 380 ends the process.

  When the second method is selected, the second sound generation unit 302 shown in FIG. 9 operates as follows. Referring to FIG. 14, first, second speech generation unit 302 reads the user's speech speech (i) and its utterance time ti from user speech DB 120 (step 410). Next, the second speech generation unit 302 reads the speech time Ti of the speech information table speech (i) (step 412). Further, the second voice generation unit 302 uses the utterance time ti read at step 410 and the utterance time Ti read at step 412 so that the utterance time of the user's speech (i) is changed from ti to Ti. Speech speed conversion is performed (step 414).

  When the third method is selected, the third sound generation unit 304 shown in FIG. 9 operates as follows. Referring to FIG. 15, first, the third voice generation unit 304 reads the recording flag (i) from the user voice table 260 of the user voice DB 120 (step 440). Next, the third sound generation unit 304 determines whether or not the value of the read recording flag is 1. When the recording flag is not 1, the third sound generation unit 304 reads the standard sound of the line (i) from the standard sound storage unit 74, and the line The voice (i) is output, and the process ends (step 444). When it is determined in step 442 that the recording flag is 1, the speech information (i) and the speech time ti are read from the user speech DB 120 (step 446), and the speech information table stored in the speech information storage unit 72 is stored. The speech time Ti of the line (i) is read from (step 448). Then, using the utterance times ti and Ti read in steps 446 and 448, respectively, the speech speed is converted and output so that the utterance time of the user's speech (i) becomes Ti (step 450).

  When the fourth method is selected, the fourth sound generation unit 306 shown in FIG. 9 operates as follows. Referring to FIG. 16, fourth voice generation unit 306 reads a recording flag (i) for the i-th speech voice from user voice table 260 of user voice DB 120 (step 470). Next, when the value of the recording flag (i) read out in step 470 is not 1, the voice actor voice of the line (i) stored in the voice actor voice DB 80 is the one having the most similar voice quality to the user voice. Read out and output as speech (i) (step 474). In this case, the voice actor voice stored in the similar voice actor storage unit 130 is used for selection of the voice actor voice. Since this is the same in the later processing, the details will not be repeated in the later description. If it is determined in step 472 that the recording flag is 1, the third speech generation unit 304 reads the speech speech (i) and the speech time ti from the user speech DB 120 (step 476). Next, the speech time Ti of the speech (i) is read from the speech information table of the speech information storage unit 72 (step 478), and the speech time of the speech speech (i) of the read user is read using the speech times ti and Ti. The speech speed is converted so as to change from ti to Ti and output as speech (i) (step 480).

  When the fifth method is selected, the fifth sound generation unit 308 illustrated in FIG. 9 operates as follows. Referring to FIG. 17, fifth voice generation unit 308 reads recording flag (i) from user voice DB 120 (step 500). When the value of the read recording flag is not 1, the voice actor voice of the line (i) stored in the voice actor voice DB 80 is identified most closely to the voice quality of the user (step 504), and the voice actor voice is determined. Is converted to voice quality using the characteristics of the user voice and output as speech voice (i), and the process is terminated (step 506). The operation of the voice generation unit 308 at the time of selecting a voice actor voice at step 504 and at the time of voice quality conversion at step 506 will be described later. If the recording flag is 1 in step 502, the fifth speech generation unit 308 reads the speech speech (i) and the speech time ti from the user speech DB 120 (step 508). Next, the speech time Ti of the dialogue (i) is read from the dialogue information table of the dialogue information storage unit 72 (step 510). Finally, speech speed conversion is performed so that the speech time of the user's speech (i) is changed from ti to Ti, and the speech is output as speech (i), and the process is terminated (step 512).

  When the sixth method is selected, the sixth sound generation unit 310 illustrated in FIG. 9 operates as follows. Referring to FIG. 18, the sixth sound generation unit 310 reads the recording flag (i) from the user sound DB 120 (step 530). If the value of this recording flag is not 1, speech synthesis is performed by using speech (i), user speech features, user vowel speech units, and consonant speech units of the unit DB 82 to perform speech synthesis. (I) is generated and output (step 534). If the recording flag = 1, the sixth voice generation unit 310 reads the line voice (i) and the utterance time ti from the user voice DB 120 (step 536). Next, the speech time Ti of the dialogue (i) is read from the dialogue information table of the dialogue information storage unit 72 (step 538). Finally, the speech speed of the user's speech (i) is converted so that the speech time of the user's speech (i) is changed from ti to Ti and output as speech (i) (step 540).

  When the seventh method is selected, the seventh sound generation unit 312 illustrated in FIG. 9 operates as follows. Referring to FIG. 19, the seventh voice generation unit 312 reads the voice most similar to the voice quality of the user voice among the voices of the voice (i) in the voice actor voice DB 80 and outputs the voice as the voice (i). The process is terminated (step 560).

  When the eighth method is selected, the eighth sound generation unit 314 operates as follows. Referring to FIG. 20, the eighth voice generation unit 314 identifies and reads the voice most similar to the voice quality of the user voice among the voice actor voices of line (i) stored in the voice actor voice DB 80. (Step 580). Next, the speech (i) read in step 580 is converted to a voice quality close to the user's voice quality using the voice characteristics of the user's utterance essential part, thereby generating the user's speech voice (i). And output (step 582). The operation of the voice generation unit 314 at the time of voice quality conversion is the same as that in step 506 of the fifth method.

  When the ninth method is selected, the ninth sound generation unit 316 shown in FIG. 9 operates as follows. Referring to FIG. 21, the ninth speech generation unit 316 uses the speech (i), the feature amount of the user speech, the speech unit of the user's vowel, and the consonant segment stored in the segment DB 82. The speech (i) is synthesized, output as speech (i), and the process ends (step 600).

  The speech effect output from the merging unit 292 is added with an acoustic effect specified by the cut information storage unit 76 as follows by the audio signal processing unit 320 shown in FIG. That is, referring to FIG. 22, audio signal processing unit 320 reads out acoustic effect list ELIST of dialogue (i) from cut information storage unit 76 for dialogue speech (i) output from merging unit 292 (step 382). The sound signal processing unit 320 further examines the elements of the sound effect list ELIST in order, adds all the sound effects specified by these elements to the line sound (i), and then the line sound (i ) Is written to the audio file (step 394). At this time, processing such as volume normalization processing (FIG. 28B) is also executed simultaneously. Thereafter, the sound signal processing unit 320 updates the sound file name of the line (i) in the line sound table 88 with a new file name, and ends the process (step 396).

  The speech signal processing unit 320 generates a speech speech table 88 as shown in FIG. 24 and speech speech data 86 to which an acoustic effect is added.

  In this way, speech is generated for all dialogues of all characters, and when speech speech data 86 and speech speech table 88 corresponding to them are created, the video / audio reproduction device is combined with the video data 66. The movie can be played back by 92. At this time, the video signal reproduction unit 620, the simultaneous audio integration processing unit 632, and the synchronous reproduction unit 638 shown in FIG. 23 operate as follows.

  First, the simultaneous speech integration processing unit 632 integrates speeches that overlap each other into a single file by a program having a control structure as shown in FIG. 26, and the speech file names in the speech speech table 88 accordingly. Is updated, and the process of setting the reproduction flag corresponding to the audio file that is no longer necessary due to the integration to 1 is executed. Through this process, speech sound data 86 and speech sound table 88 that can be finally reproduced are generated. At this time, the time to start playback of each speech is recorded in the playback start time of each speech in the speech speech table 88.

  When the reproduction of the movie is started, the video signal reproduction unit 620 reproduces an audio signal indicating a sound effect such as a video signal and a background sound, and supplies the sound signal to the display device 622 and the sound effect output device 624. The display device 622 reproduces this video signal and displays the video. The sound effect output device 624 converts the sound signal of the sound effect into sound. The face image of the character in this movie has been replaced with the face image of the user.

  On the other hand, the video signal reproduction unit 620 generates a synchronization signal based on the synchronization data recorded in the video data in synchronization with the reproduction of the video signal and supplies the synchronization signal to the synchronization reproduction unit 638.

  The synchronized playback unit 638 always monitors the synchronized signal, and when the time represented by the synchronized signal matches the playback start time of the speech stored in the speech audio table 88, the speech is reproduced and the speech audio output is performed. To device 640. The dialogue voice output device 640 reproduces this voice. Dialogue speech is speech reproduced or synthesized according to any of the above-described methods. This voice is basically the voice of each user as it is, the voice speed converted, or the voice selected or synthesized so as to be as similar as possible to the voice quality of the user's voice. Of course, some standard voices may be reproduced as they are. However, when the entire dialogue is viewed, the voice of each character feels similar to the voice quality of the corresponding user.

  In the description of the above embodiment, the user's voice is segmented and added to the segment DB 82. However, the present invention is not limited to such an embodiment. For example, speech speech that has been recorded with high quality among user speech may be registered in the voice actor speech DB 80. By doing so, it becomes possible to add the voices of many users to the voice actor voice DB 80, and it is possible to efficiently collect various voices.

  Therefore, the multimedia production system 50 can create and screen a movie as if not only the face images of the characters of the movie prepared in advance but also the speech of the user was replaced with the user's voice. As a result, it is possible to easily replace the voice of the character with the voice of the user in a short time in a multimedia product in which the line of the character is known. In addition, the voice of the character can be easily replaced with a voice quality close to that of the user in a short time. Furthermore, it is possible to collect a large number of user voices and use them in voice replacement so that the voices of the characters can be easily replaced in a short time with a voice quality close to that of the user. .

[Configuration for selecting similar voices and converting voice quality]
Hereinafter, a process of determining a voice actor voice similar to the user voice executed in step 330 of FIG. 10 and a process of converting voice quality (voice quality conversion process) executed in step 506 of FIG. 17 and step 582 of FIG. ). Variables and constants (I, i, J, j, M, N, S, s, etc.) used in the following description are all used as local variables in the program, and appear in other figures. Are separate.

First, an outline of the principle for determining similar speech will be described. Details will be described later. In this embodiment, when determining a voice actor voice similar to the user's voice, a standard that is as close to human perception as possible is adopted instead of simply using a difference in specific acoustic feature quantities as a reference. For this purpose, a plurality of types of acoustic feature amounts are calculated in advance for the sound, and the similarity of the sound (this is referred to as “perceptual distance” in this specification) by linear combination of distance measures calculated by the two sounds. Will be expressed). The linear combination coefficient α _i is optimized in advance so that a similar speech can be selected in a form close to human perception based on this perceptual distance. Processing for realizing this optimization will be described later.

  For the voice actor voice, each of the plurality of acoustic feature quantities described above is calculated in advance and stored in association with the voice actor voice identifier. When the user's voice is obtained, a plurality of acoustic feature quantities are calculated in the same manner from the voice. A perceptual distance is calculated between the plurality of acoustic feature quantities and the plurality of acoustic feature quantities calculated for each voice actor, and sounds similar to the user's voice in order from the smallest perceptual distance. And

In the embodiment described below, the voice actor voice whose voice quality is similar to the user voice is determined for each utterance, but actually, when the recording of the practice utterance is finished, the recorded utterance is used. A voice actor voice similar to the user voice is determined. In this way, a morphing rate (described later) used for voice quality conversion can also be calculated in advance. In the present embodiment, three voice actor voices are used as voices similar to user voices in the process of determining similar voices in relation to the voice quality conversion process described later. Also, eight types of acoustic feature quantities are used. That is, in the present embodiment, α _{1 to} α ₈ are used as the linear combination coefficient αi.

The acoustic feature quantities used in the present embodiment are as follows. As described above, in the present embodiment, since the speech analysis conversion / synthesis system STRAIGHT is used for morphing, acoustic features specific to STRAIGHT are included.
(1) MFCC (Mel Frequency Cepstrum Coefficient) 12 dimensions + ΔMFCC 12 dimensions + Δ power 1 dimension in total 25 dimensions feature vector,
(2) 35th-order or higher order STRAIGHT cepstrum (CepH) representing the frequency characteristics of the vocal cord sound source,
(3) the first order (Cep1) of the STRAIGHT cepstrum representing the slope of the frequency characteristics of the vocal cord sound source;
(4) Logarithmic spectrum (spectrum) of 2.6 kHz or more,
(5) 2.6 kHz or less (Ap) of an aperiodicity index which is an analysis parameter of STRIGHT
(6) Fundamental frequency (F0),
(7) First to fourth formant frequencies (Formant) important for voice quality expression, and (8) Spectral slope of 0 to 3 kHz (SpecSlope) of the logarithmic spectrum.

  As a distance measure calculated for each of these acoustic features between two speeches, a DTW (Dynamic Time Warping) distance or a GMM (Gaussian Mixture Model) likelihood (mixing number = 16) is used. When any of these is used to combine the distances of the acoustic feature amounts, a more preferable result is obtained than when the acoustic feature amounts are used alone. Above all, DTW gave better results than using GMM. Therefore, DTW is used in this embodiment.

  Next, the principle of voice quality conversion processing will be described. The voice quality conversion performed in the present embodiment is performed by mixing (morphing) a plurality of voice actor voices selected as voices having a short perceptual distance from voice actor voices, thereby further reproducing voices closer to the voice quality of the user. Is a process of synthesizing. Conceptually, it can be expressed as follows.

  As shown in FIG. 37, consider the case where there are three selected voice actor voices (first to third voice actor voices) as in the present embodiment. In order to simplify the description, a case will be described in which the number of acoustic feature values used for calculating the perceptual distance is three instead of eight. In this case, it can be said that the selected three voice actor voices correspond to the three points 990, 992, and 994 in the three-dimensional acoustic feature space (speaker space). A sound obtained by mixing (morphing) these sounds becomes a point in a triangular plane having three points 990, 992, and 994 as vertices. In this embodiment, among the morphed voices, the voice corresponding to the point 996 where the distance 997 to the point 998 corresponding to the voice of the user (target speaker) is the smallest is adopted.

  Through such processing, speech speech close to the voice quality of the user can be synthesized from speech speech previously recorded by a voice actor.

  The number of voice actor voices selected here is not limited to three, but may be two or four or more. When two voice actor voices are selected, the voice corresponding to the point closest to the target speaker at the point on the line connecting the two voice actor voices is synthesized by morphing. When there are four or more selected voice actor voices, the point within the tetrahedron defined by the points corresponding to the four voice actor voices instead of the plane is located closest to the target speaker voice. A certain voice is synthesized by morphing. The same applies to the case of five or more.

  Further, the acoustic feature amount need not be plural, and may be one or more. In this case, the linear combination coefficient is not a vector format but a single number. Even in this case, since the linear combination coefficient is selected to have a high correlation with the similarity order of speech based on human perception, it is possible to select speech having the same voice quality as that selected by the human.

  Hereinafter, a procedure for calculating a linear combination coefficient of the distance scale of the acoustic feature amount used for the above-described determination of similar speech prior to the operation of the system will be described.

  FIG. 32 shows, in a flowchart form, a procedure for creating data indicating the similar order of speech perceived by humans as basic data for calculating these linear combination coefficients. Referring to FIG. 32, this procedure prepares M speaker voices recorded in advance and calculates their acoustic feature value 920, and for each of these M speaker voices, Step 922 of performing the process 924 of FIG.

  The process 924 sets a rank in which the speech to be processed is assumed to be similar to the target voice by the human perception with the target voice as the target voice, and the similarity rank at this time is set as the step 926. And storing 928 in association with the target speech.

  By executing this rank data creation process, one set of data indicating ranks 1 to M is created for each target voice. Since M speakers are selected as target speakers, M sets of rank data are created as a whole.

  In the present embodiment, the rank data creation process described above is performed at least twice in order to increase the reliability of the linear combination coefficient. That is, finally, two sets of rank data each indicating the ranks of Nos. 1 to M are created for one target voice, and 2M sets in total are created.

  When rank data is created in this way, a linear combination coefficient can be calculated. FIG. 33 shows a flowchart of a program that realizes linear combination coefficient calculation processing.

  Referring to FIG. 33, the program prepares the acoustic feature quantity data of M speakers prepared in the rank data creation process in a storage device in a computer-readable format, and 930 of these M people. For each speaker, select that speaker as the target speaker and perform a process 934 that performs process 934 described below.

  The process 934 selects N speakers (N ≦ M) from M speaker voices, and among the rank data created by the rank data creation process, the process 934 applies to the target speaker being processed. The two created rank data sets are read, and based on the respective rank data, the relative ranks perceived that the N speakers selected in step 935 are similar to the target speaker are calculated. Step 936. Since there are two sets of rank data sets, two sets of data sets of relative rank at this time are also created.

  The process 934 further includes a step 938 for determining whether or not the two sets of relative ranks calculated in the step 936 match and branching the control flow according to the determination result. If the determination result in step 938 is NO, it means that the reliability of the rank data for these N speakers is low, and no further processing is performed on the set of these N speakers. Jump to the end of 934. On the other hand, if the decision result in the step 938 is YES, the following process is executed. Similarity ranking by human perception is affected by the subjectivity of the similarity ranking setter, so by increasing the objectivity as much as possible, the reliability of the finally obtained linear combination coefficient can be increased. .

  The process 934 is further executed when the determination result in step 938 is YES, and the combination of this target speaker, the N speakers selected in step 935, and the ranking of the N speakers is stored in the RAM. Step 940 of storing and ending this process 934 is included.

  The program further includes a step 942 for determining an initial value α = (a 1, a 2,..., A 8) prepared with appropriate values as linear combination coefficients, and optimizing the linear combination coefficient α using Newton's method. Step 944 for obtaining the highest correlation value between the order by perceptual distance and the relative relative order of similar speech determined by humans, and the linear combination coefficient α optimized by step 944 as the linear combination coefficient And step 946 for saving the processing in the storage unit 94 and ending the processing. Since the perceptual distance is based on an acoustic feature, this is referred to as acoustic similarity in this specification. On the other hand, since the relative order of similarities by human beings is based on human perception, this is called perceptual similarity in this specification.

  FIG. 34 is a flowchart of a program for optimizing the linear combination coefficient α = (a1, a2,..., A8) by the Newton method, which is executed in step 944 of FIG. Referring to FIG. 34, the program includes a step 1000 of preparing a peripheral value of α according to the Newton method. In the present embodiment, 16 are used as the peripheral value of α. Specifically, with respect to the element a1 of α = (a1, a2,..., A8), while maintaining the values of the other elements, the first peripheral value α1 obtained by adding only the value of the element a1 to the element a1 , The second peripheral value α2 obtained by subtracting only the value of the element a1 while maintaining the values of the other elements is included. Thereafter, similar processing is performed for each of the other elements a2 to a8, and peripheral values α3 to α16 are set. That is, 16 values of α1 to α16 are obtained as peripheral values of the linear combination coefficient α by the process of step 1000.

  This program is further executed following step 1000, and in step 1002 for reading all combinations of N speakers stored in step 940 of FIG. 33, and for all of the combinations of N speakers read out, step 1002 is executed. On the other hand, it includes step 1004 for performing a process 1006 described later.

  Process 1006 includes step 1008 of executing step 1010 for each of the linear combination coefficient α and its peripheral values α1 to α16 for each speaker included in the combination of N speakers to be processed. . In step 1010, the perceptual distance between the target speaker and the speaker to be processed is calculated by the following equation using these as linear combination coefficients when calculating the perceptual distance.

ただし、Ｌ（Ｋ）は、線形結合係数αによりＫ番目の話者に対して算出された知覚的距離、Ｌ^ｊ（Ｋ）は、ｊ番目の周辺値αｊによりＫ番目の話者に対して算出された知覚的距離、Ｌ_ｉはｉ番目の音響特徴量に対してターゲット話者と距離計算対象の話者との間で算出された距離、ａ_ｉは線形結合係数αのｉ番目の要素、a_ｉｊはｊ番目の周辺値αｊのｉ番目の要素である。

Where L (K) is the perceptual distance calculated for the Kth speaker by the linear combination coefficient α, and L ^j (K) is for the Kth speaker by the jth peripheral value αj. The calculated perceptual distance, L _i is the distance calculated between the target speaker and the distance calculation target speaker for the i th acoustic feature, and a _i is the i th element of the linear combination coefficient α. , A _ij is the i-th element of the j-th peripheral value αj.

  By executing step 1010, 17 different perceptual distances are calculated for each of the N speakers.

  The program further calculates a combination of 17 types of ranks for the N speakers to be processed based on the 17 types of perceptual distance (acoustic similarity) calculated in step 1008. 1012 and Spearman's rank correlation coefficient ρ between each of the 17 kinds of rank combinations calculated in step 1012 and the similar relative order (perceptual similarity) determined by a human being: And step 1014 for calculating and storing.

ここで、ａ、ｂはそれぞれ知覚的類似度による順位、及び音響的類似度による順位である。

Here, a and b are ranks based on perceptual similarity and ranks based on acoustic similarity, respectively.

By performing the above process 1006 for each of the combinations of N speakers, up to 17 × _M C _N Spearman rank correlation coefficients ρ are calculated.

This program is further executed after the process 1006 is completed, and the common linear combination coefficient α or its peripheral value among the maximum 17 × _M C _N Spearman rank correlation coefficients ρ stored in step 1014. Steps 1016 for reading out the values calculated using αj (1 ≦ j ≦ 16) and calculating the average value for each linear combination coefficient α or its peripheral value αj, and the rank correlation calculated in step 1016 Among the average values of the numbers, it is determined whether or not the average value calculated for the linear combination coefficient α is greater than all the average values calculated for each of the other peripheral values αj, and control is performed according to the determination result. Step 1018 for branching the flow and executed when the determination result in Step 1018 is NO, the linear combination coefficient α is set to the peripheral value αx (x = 1 to 16). Of, updated with an average value of rank correlation coefficient was highest, and a step 1020 returning control to step 1000. If the decision result in the step 1018 is YES, the process of this program ends, and the process returns to the step 946 in FIG. Accordingly, the linear combination coefficient α at this time is stored in the linear combination coefficient storage unit 94.

  The linear combination coefficient obtained in this way uses data for ranking similarities by one or two people, but the number of target speakers is M and plural, and M Since there are generally many ways to select N people from among them, the reliability is high.

  Hereinafter, how the voice actor speech similar to the user is selected using the linear combination coefficient determined and stored in this manner will be described. This selection is performed by a similar voice determination program described below.

Referring to FIG. 31, the similar speech determination program reads step 900 for reading linear combination coefficient αi (i = 1 to 8) calculated and stored in advance from the storage device, and controls the following repetitions Subsequent to step 902 for initializing variable J by substituting 0 for variable J, step 904 for adding 1 to variable J, and step 904, the value of variable J is the voice actor in voice actor voice DB 80. determining whether greater than the number J _MAX of, and a step 906 for branching control flow depending on the result of determination.

  The program is further executed in response to the determination result of step 906 being NO, and reads out the Jth voice actor voice data from the voice actor voice DB 80; and the Jth voice actor voice data; And a step 910 of calculating a perceptual distance L (J) with the user's voice data by the following equation.

なおＬ_ｉはi番めの音響特徴量について算出された距離である。この後、処理はステップ９０４へ戻される。

Note that L _i is a distance calculated for the i-th acoustic feature amount. Thereafter, the process returns to step 904.

  This program is further executed in response to the determination of YES in step 906, and the perceptual distances calculated by the number stored in the voice actor voice DB 80 are sorted in ascending order, and the perceptual distance is the smallest. A predetermined number of voice actor voices (in this embodiment, this number is generally S. S = 3 in this embodiment) are selected as the voices most similar to this user's voice, and the process is terminated. Step 912.

Referring to FIG. 35, the voice quality conversion program has been calculated and stored in advance for voices of S voice actors selected as having voice quality similar to the voice quality of the user (S = 3 in the present embodiment). Step 950 for reading the above-described eight types of acoustic feature quantities and previously stored linear combination coefficients from the storage device, and the time series of these acoustic feature quantities so that the time information and formant position of the phoneme during morphing coincide with each other. Step 952 for assigning feature points (start time, end time, intermediate time, etc., and first to fourth formant positions) to the voice feature, and the acoustic feature quantity of the selected voice actor voice and the acoustic feature quantity of the user voice are used. Step 953 for estimating the morphing rate (mixing rate) of each voice actor voice for mixing voice actor voices and synthesizing voice similar to the voice quality of the user voice; Based on the rate (mixing rate), step 954 for determining the position after the morphing of the feature point described above, and the time axis and the frequency by piecewise linearly interpolating the acoustic feature quantity of the point other than the feature point between the feature points A step 956 for expanding and contracting the axis and a step 958 for calculating the morphing sound X _M by linearly combining the acoustic feature values at each time point based on the morphing rate given to each voice actor voice. A method for determining the morphing rate will be described later. In the present embodiment, the feature points are assigned in step 952 by an attendant.

In step 958, morphing audio _{X M} is calculated by the following equation.

ｒ＝［ｒ_１ ｒ_２ … ｒ_ｓ］^Ｔをモーフィング率ベクトルと呼ぶ。

r = [r ₁ r ₂ ... r _s ] ^T is called a morphing rate vector.

  With reference to FIG. 36, a program for realizing the processing of step 953 in FIG. 35 for calculating the morphing rate used in step 954 and step 958 in FIG. 35 will be described. Since the equation for estimating the morphing rate is difficult to solve analytically as will be described later, an optimal solution is obtained by sequential processing using the Newton method. Hereinafter, this sequential processing will be described.

  In the morphing of a plurality of speaker voices (voice actor voices similar to user voices in the present embodiment), various voice qualities can be realized by morphing rates. For this reason, the morphing rate is estimated such that the voice quality of the target speaker (user in the present embodiment) is achieved. In the present embodiment, the morphing rate is determined so as to satisfy the following formula (1).

この手法では、この式（２）を満足するモーフィング率ベクトル＾ｒは、以下の式（３）を最小とするように推定される。

In this method, the morphing rate vector {circumflex over (r)} r satisfying the equation (2) is estimated so as to minimize the following equation (3).

本実施の形態の音声モーフィングには、前述のように音声分析変換合成システムＳＴＲＡＩＧＨＴをモーフィングに用いる。この音声モーフィングでは、特徴量およびモーフィング後の時間周波数平面を同一のモーフィング率で制御するため、式（３）における勾配は非線形であり、解析的に解くことが困難である。そこで、次の式（５）（６）により、ニュートン法でモーフィング率を逐次的に求める。

As described above, the speech analysis conversion synthesis system STRIGHT is used for morphing in the speech morphing of the present embodiment. In this voice morphing, the feature amount and the time-frequency plane after morphing are controlled with the same morphing rate, so the gradient in equation (3) is non-linear and difficult to solve analytically. Therefore, the morphing rate is sequentially obtained by the Newton method using the following equations (5) and (6).

ここで、ε’（＾ｒ_ｎ）は、式（３）に対して、時間成分及び周波数成分にかかるモーフィング率を定数として近似したものを表す。βは適切な定数であり、例えば０．１〜１の範囲の任意の数が選ばれる。また、

Here, ε ′ (＾ r _n ) represents the approximation of the morphing rate for the time component and the frequency component as a constant with respect to Equation (3). β is an appropriate constant, and for example, an arbitrary number in the range of 0.1 to 1 is selected. Also,

はｎ回目の更新で得られたモーフィング率＾ｒ_ｎにより時間周波数伸縮を行なった特徴量であり、

Is a feature quantity of performing a time-frequency expansion by n-th obtained morphing rate update ^ r _n,

である。

It is.

  Referring to FIG. 36, the morphing rate estimation processing program first stores the maximum value that can be stored in the above-described variable storing ε (r), and continues to step 970. Step 972 for setting an appropriate initial value as the rate vector r, Step 974 for calculating ε (r) using Equation (3) using the set morphing rate vector r, and Step 974 Determining whether or not the value of ε (r) has converged, and branching the flow of control according to the determination result 976. In the determination of step 976, it is determined whether or not the absolute value of the difference between the value of ε (r) calculated before and the value of ε (r) calculated in step 974 is smaller than a predetermined threshold value. The standard.

  This program is further executed when the determination result of step 976 is NO, and using step 978 for calculating En by the equation (6) and the currently set morphing rate vector r, a new And a step 980 of calculating a correct morphing rate vector r using equation (5) and returning control to step 974.

  The program further includes a step 982 that is executed when the determination result in step 976 is NO, and that the morphing rate vector r at that time is a value to be obtained and ends the process.

  The algorithm shown in FIG. 36 is an example for realizing the Newton method, and other optimization algorithms may be used. In many cases, there are program kits for numerical calculation that realize an optimization algorithm such as Newton's method, and this calculation can be easily realized by using them.

[Operation of Multimedia Production System 50 when Selecting Similar Voices and Converting Voice Quality]
In selecting similar voices and converting voice quality, the multimedia production system 50 operates as follows. First, the processing for selecting similar speech will be described, and then the processing for converting voice quality will be described. Prior to this, it is assumed that the linear combination coefficients for the eight kinds of acoustic feature amounts for evaluating the similar speech are obtained and stored in the storage device as described above.

  First, among the voice actor voices of the line (i) stored in the voice actor voice DB 80, the voice voice having the most similar voice quality to the user voice is determined and read as described below, and output as the voice voice (i) ( Step 474, Step 504, Step 560 and Step 580). Here, this process is executed by the fourth sound generation unit 306, the fifth sound generation unit 308, the seventh sound generation unit 312 and the eighth sound generation unit 314. The voice generation unit is not distinguished and is simply referred to as a voice generation unit.

  The voice generation unit operates as follows in the similar voice determination process. Referring to FIG. 31, the voice generation unit reads linear combination coefficient αi (i = 1 to 8) obtained and stored in advance from the storage device (step 900). The perceptual distance from the user's voice is calculated for all voice actor voices stored in the voice actor voice DB 80 by the processing of steps 902 to 910 (step 910). When calculation of the perceptual distance is completed for all voice actor voices stored in the voice actor voice DB 80, a predetermined number of voice actor voices having the smallest calculated perceptual distance is selected as having the most similar voice quality to the user voice. (Step 912).

  If only one voice actor voice having the most similar voice quality is determined, the voice actor voice having the smallest perceptual distance is selected in step 912. When a plurality of voice actor voices having the most similar voice qualities are determined, in step 912, the number of voice actor voices is selected in order from the smallest perceptual distance.

  Referring to FIG. 33, when voice quality conversion is performed, multimedia production system 50 operates as follows. Here, three types of voice actor voices similar to the user are selected. When these voice actor voices are selected, the voice actor voice of the speech to be processed is converted into voice quality using the characteristics of the user voice (steps 506 and 582). This process is executed by the fifth sound generation unit 308 and the eighth sound generation unit 314. In the following description, these sound generation units are not distinguished from each other and are simply referred to as a sound generation unit.

  The voice generation unit operates as follows in the voice quality conversion process. Note that the similar voice determination process shown in FIG. 31 and the morphing rate estimation process shown in FIG. 36 are performed when recording of the practice voice is finished. Further, it is assumed that feature points for morphing are assigned in advance to all voice actor voices.

  Referring to FIG. 35, the voice generation unit reads the acoustic feature quantities (eight types) of the selected three voice actor voices from the storage device (step 950). A feature point is given to the user voice by an attendant (step 952). Subsequently, the position of the feature point after morphing is determined based on the morphing rate calculated based on the acoustic feature amount of the three voice actor voices and the acoustic feature amount of the user voice (step 954). The time and frequency axes are expanded / contracted by piecewise linear interpolation of the acoustic features between the points other than the points (step 956), and then the acoustic features of the voice actor voice are linearly combined using the estimated morphing rate. (Step 958).

  Therefore, the multimedia production system 50 can select the voice of the voice actor that is most similar to the voice of the user, as perceived by humans, and morphs the voices of the multiple speakers to resemble the voice quality of the user voice. Voice can be generated. As a result, in multimedia productions where the dialogue of the characters is known, even if the user's voice is low, the voice of the characters should be accurately replaced with a voice that humans feel as being closest to the user's voice. Is possible.

  In the embodiment described above, the present invention is applied to a multimedia production system for producing a movie. The system to which the present invention can be applied is not limited to this, for example, a television program, a radio drama, etc. In general, the timing at which speech is uttered, its length, etc., proceed according to a scenario determined for each speaker. It can be applied to anything.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are included. Including.

It is a functional block diagram of the multimedia production system 50 which concerns on one embodiment of this invention. 3 is a functional block diagram of a character voice creation unit 102. FIG. It is a figure which shows the structure of the dialog information table memorize | stored in the dialog information storage part. It is a figure which shows the example of the recording state of the line sound at the time of the end of recording. It is a flowchart of the first half part of the computer program which implement | achieves the audio | voice recording process performed by the audio | voice recording part 114 shown in FIG. 2 on computer hardware. It is a flowchart of the latter half part of the computer program which implement | achieves the audio | voice recording process performed by the audio | voice recording part 114 shown in FIG. 2 on computer hardware. It is a figure which shows an example of the display which the user information input part 100 displays on the screen of the input / output device 112 at the time of a user's audio | voice recording. It is a block diagram which shows the structure of user audio | voice DB120. It is a functional block diagram which shows the structure of the audio | voice preparation part 118. It is a flowchart of the computer program which implement | achieves the determination process of the audio | voice production | generation method performed in the synthetic | combination method determination part 116 shown in FIG. 2 on computer hardware. It is a figure which shows the structure of the method list table. It is a schematic diagram which shows the structure of the acoustic effect list table memorize | stored in the cut information storage part. It is a flowchart of the program which implement | achieves the 1st audio | voice production | generation part 300 shown in FIG. It is a flowchart which shows the control structure of the program for implement | achieving the 2nd audio | voice production | generation part 302 shown in FIG. It is a flowchart of the program for implement | achieving the 3rd audio | voice production | generation part 304 shown in FIG. It is a flowchart which shows the control structure of the program for implement | achieving the 4th audio | voice production | generation part 306 shown in FIG. 10 is a flowchart showing a control structure of a program that realizes the fifth sound generation unit 308 shown in FIG. 9. It is a flowchart of the program for implement | achieving the 6th audio | voice production | generation part 310 shown in FIG. FIG. 10 is a flowchart illustrating a control structure of a program that implements a seventh sound generation unit 312 illustrated in FIG. 9. FIG. It is a flowchart which shows the control structure of the program which implement | achieves the 8th audio | voice production | generation part 314 shown in FIG. It is a flowchart of the program for implement | achieving the 9th audio | voice production | generation part 316 shown in FIG. It is a flowchart of the program which implement | achieves the audio | voice signal processing part 320 shown in FIG. 2 is a block diagram of a playback system for playing back a movie created by the multimedia production system 50. FIG. It is a figure which shows typically the example structure of the speech sound table 88. FIG. FIG. 24 is a diagram illustrating an exemplary configuration of a speech speech table 88 after the simultaneous speech integration processing by the simultaneous speech integration processing unit 632 illustrated in FIG. 23. 10 is a flowchart of a program for realizing a simultaneous voice integration processing unit 632. It is a flowchart of the program for implement | achieving the synchronous reproduction | regeneration part 638 shown in FIG. It is a figure for demonstrating the content of speech speed conversion and a volume normalization process among the acoustic effect processes which the audio | voice signal process part 320 performs. It is an external view of the hardware constitutions of the computer 840 which implement | achieves the speech audio | voice data preparation part 90 for recording a user's audio | voice in the multimedia production system 50. FIG. 25 is a block diagram showing an internal configuration of a computer 840. It is a flowchart which shows the control structure of the similar audio | voice determination program performed with an audio | voice production | generation part. It is a flowchart which shows the control structure of the program which implement | achieves the order data creation process of similar speech used in the linear combination coefficient calculation process performed prior to a similar speech determination process. It is a flowchart which shows the control structure of the linear combination coefficient calculation program which calculates the linear combination coefficient of the acoustic feature-value used in order to obtain a result very similar to human perception. It is a flowchart of the program for the optimization of the linear combination coefficient using the Newton method performed by step 944 of FIG. It is a flowchart which shows the control structure of the voice quality conversion program performed with an audio | voice production | generation part. It is a flowchart which shows the control structure of the morphing rate estimation program performed with an audio | voice production | generation part. It is a figure which shows notionally the speaker space for demonstrating estimation of the morphing rate.

Explanation of symbols

50 Multimedia production system 60 Three-dimensional scanner group 62 Image processing PC
64 Video generation device 66 Video data 70 Video material DB
72 Dialog information storage unit 74 Standard voice storage unit 76 Cut information storage unit 78 Method list table 80 Voice actor voice DB
82 Segment DB
86 speech sound data 88 speech sound table 90 speech sound data creation unit 92 video / audio reproduction device 94 linear combination coefficient storage unit 100 to 100N user information input unit 102 to 102N character speech creation unit 104 speech integration unit 112 input / output device 114 speech Recording unit 116 Synthesis method determination unit 118 Voice creation unit 120 User voice DB
122 audio DB update unit 124 segment DB update unit 130 similar voice actor storage unit 132 morphing rate storage unit 280 branching unit 292 merge unit 300 to 316 first audio generation unit to ninth audio generation unit 320 audio signal processing unit 620 video Signal reproduction unit 622 Display device 624 Sound effect output device 632 Simultaneous audio integration processing unit 638 Synchronous reproduction unit 640 Line sound output device

Claims (9)

A similar voice selection device for selecting a voice similar to a target voice from a plurality of sample voices,
Means for storing the plurality of sample sounds;
The distance between the target speech and each of the plurality of sample speeches by a weighted linear sum of distance measures between two speeches calculated for each of one or more acoustic feature quantities for speech A distance calculating means for calculating
A similar voice selection device comprising: a voice selection means for selecting a voice having a smallest distance calculated by the distance calculation means from among the plurality of sample voices as a voice similar to the target voice. The similar speech selection apparatus according to claim 1, wherein the distance measure is a distance measure calculated by dynamic time axis expansion / contraction between the same acoustic features of two sounds. The similar distance selecting device according to claim 1, wherein the distance scale is a distance scale calculated by a Gaussian mixture distribution model between the same acoustic feature quantities of two sounds. The combination coefficient of the weighted linear sum of the distance scale is a correlation between the result of ranking the similarity between each of the plurality of voices by the human and the result of the distance calculation unit. The similar speech selection device according to any one of claims 1 to 3, wherein the similar speech selection device is determined in advance so as to be maximized. When executed by a computer, the computer is
Means for storing a plurality of sample sounds;
Distance calculating means for calculating a distance between a target voice and each of the plurality of sample voices by a weighted linear sum of distance scales calculated for each of one or more types of acoustic feature quantities for the voice When,
A computer program for causing a sample having the smallest distance calculated by the distance calculating means to function as a sound selecting means for selecting a sound similar to the target sound from the plurality of sample sounds. A voice generation device for generating a voice having a predetermined content with a voice similar to a target voice,
Means for storing a plurality of sample sounds of the predetermined content;
The distance between the target speech and each of the plurality of sample speeches by a weighted linear sum of distance measures between two speeches calculated for each of one or more acoustic feature quantities for speech A distance calculating means for calculating
A voice selection means for selecting a voice having the smallest distance calculated by the distance calculation means from among the plurality of sample voices as a voice similar to the target voice;
A voice generation device comprising: voice generation means for generating voice of the predetermined content using the voice selected by the voice selection means. The voice selection means includes means for selecting, from the plurality of sample voices, a plurality of voices having the smallest distance calculated by the distance calculation means as voices similar to the target voice,
The voice generation device according to claim 6, wherein the voice generation unit includes a voice morphing unit for generating a new voice by performing morphing on the plurality of voices selected by the voice selection unit. Further, a predetermined feature vector of post-morphed speech obtained by morphing speech of the plurality of speakers selected by the selecting means by the speech morphing means, and the target speech, Optimization means for optimizing a morphing ratio of the plurality of speakers at the time of morphing by the voice morphing means so that a distance between the feature quantity vector corresponding to the predetermined feature quantity is minimized; The voice generation device according to claim 7. When executed by a computer, the computer is
Means for storing a plurality of sample sounds of predetermined content;
The distance between the target speech and each of the plurality of sample speeches is obtained by a weighted linear sum of distance measures between the two speeches calculated for each of one or more acoustic feature quantities for the speech. A distance calculating means for calculating;
A voice selection means for selecting a voice having the smallest distance calculated by the distance calculation means from among the plurality of sample voices as a voice similar to the target voice;
A computer program that functions as a voice generation unit for generating a voice having the predetermined content using a voice selected by the voice selection unit.

Images