JP2010237307A - Speech learning/synthesis system and speech learning/synthesis method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an easy-to-use speech learning/synthesis system and method that can reduce facility costs and the burden of the costs on a user, and execute speech learning processing even during use of a portable terminal. <P>SOLUTION: The speech learning/synthesis system includes a user terminal 100 to which speech data and a text are input, and a server 200 connected to the user terminal 100 through a network. The user terminal 100 includes a feature quantity analyzer 110 which analyzes/extracts a feature quantity from speech data, and a waveform generator 140 which generates a synthesized speech from intermediate information, and the server 200 includes a DB generator 210 which generates a sound source DB using feature quantities, and an intermediate information generator 220 which generates the intermediate information from the text. The feature quantity and text are transmitted from the user terminal 100 to the server 200, and the intermediate information is transmitted from the server 200 to the user terminal 100. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention learns input speech data, and then generates and outputs a synthesized speech having the properties of speech obtained by learning when text is input, and speech learning / synthesis Regarding the method.

  In recent years, statistical speech synthesis technology for generating synthesized speech from an arbitrary text by statistically automatically learning and modeling features of a given speech corpus has been developed (for example, Non-Patent Document 1). reference).

  With such a speech synthesis technique, it has become possible in principle to generate synthesized speech that reproduces the characteristics of a speech corpus. Specifically, static and dynamic features related to the speech spectrum, fundamental frequency (F0), and phoneme duration are analyzed and extracted from the speech corpus in advance, and a hidden Markov model (HMM model) using the EM algorithm. To learn. When synthesizing speech from text, an appropriate HMM model is selected and connected from the input text, and then a feature quantity sequence of spectrum, fundamental frequency, and phoneme duration is generated and synthesized from this feature quantity sequence. Generate audio.

Yoshimura et al., “Simultaneous Modeling of Spectrum, Pitch, and State Duration in Speech Synthesis Based on HMM”, IEICE Theory, J83-D-II, No.11, pp.2099-2107, November 2000

By the way, there are various problems in speech learning and synthesis as described above.
First, the first problem is the amount of processing required for speech learning. In general, the amount of processing involved in learning is large, and the amount of memory required is also large. As one of the fields of use of speech synthesis, it can be applied to voice services on terminals such as telephones. For example, even the latest mobile terminals have the computational power and computational resources necessary for such learning. However, it is difficult to complete learning and synthesis processing on the terminal side.

  The second problem is equipment cost. As a method for solving the first problem, a method of performing processing on the server side instead of the terminal side can be considered easily. However, in this case, since the processing is concentrated on the server side, the scale is large in proportion to the number of terminals. It is necessary to prepare equipment, which is a big problem in terms of cost.

  Further, as a third problem, there are problems relating to delay associated with NW (network) transmission and user convenience in terms of packet unit price. When processing is performed by the server, it is necessary to transmit the learning speech and synthesized speech between the terminal and the server using the network. However, there is a delay and packet loss due to network transmission, so there is no interruption on the terminal side. Processing such as buffering is necessary to reproduce audio. In this case, naturally a waiting time is required. In addition, in a cellular phone network or the like in which a pay-per-use packet unit price is set, there is a problem that a large burden is placed on the user for distributing a large amount of audio data.

  An object of the present invention is to provide a distributed speech learning / synthesis system and method capable of solving such various problems.

  According to the first aspect of the present invention, a speech learning / synthesis system that learns input speech data and generates synthesized speech for the input text based on the learning, receives speech data and text. A user terminal and a server connected to the user terminal via a network. The user terminal analyzes and extracts a feature value from speech data, and a waveform generation unit generates a synthesized speech from intermediate information. The server includes a DB generation unit that generates a sound source DB using the feature amount, and an intermediate information generation unit that generates the intermediate information from text, and the feature amount and text are transmitted from the user terminal to the server. The intermediate information is transmitted from the server to the user terminal.

  According to a second aspect of the present invention, in the first aspect of the invention, the user terminal includes a DB request unit that requests the server to transmit the sound source DB, and the server transmits the sound source DB to the user terminal based on the request. Is done.

  According to the invention of claim 3, a speech learning / synthesis method for learning input speech data and generating a synthesized speech for the input text based on the learning is a user connected via a network. The terminal includes a terminal and a server, and learning includes a process in which a user terminal analyzes and extracts a feature value from input voice data, a process in which the user terminal transmits the feature value to the server, and a feature value received by the server. And a process of generating a sound source DB. Compositing is a process in which the user terminal transmits the input text to the server, a process in which the server generates intermediate information from the received text, a process in which the server transmits the intermediate information to the user terminal, and the user terminal receives And a process of generating synthesized speech from the intermediate information.

  According to a fourth aspect of the present invention, the method according to the third aspect of the present invention includes a process in which the user terminal requests the server to transmit the sound source DB, and a process in which the server transmits the sound source DB to the user terminal based on the request.

  According to the present invention, the processing is shared between the user terminal and the server, the learning processing or the like having a large calculation amount or necessary memory amount is executed on the server side, and only the processing having a small calculation amount or memory amount is executed on the user terminal side. Therefore, even when a mobile terminal is used as a user terminal, for example, the voice learning process can be executed, and the first problem described above can be solved.

  Moreover, it is possible to reduce the amount of processing required on the server side by executing a part of the processing on the user terminal side instead of executing all the processing only on the server. Therefore, it is possible to reduce the number of servers to be prepared or to use a server with low cost and low computing capacity, so that the equipment cost can be reduced and the second problem described above can be solved.

  Furthermore, 1) only intermediate information is transmitted at the time of synthesis, and the user terminal side performs synthesis processing from the intermediate information, so that the amount of transmission data can be reduced compared with the case of transmitting synthesized speech from the server. Can reduce the waiting time.

  2) When a network delay occurs at random, if voice is transmitted, it will be inevitable that the voice will be interrupted due to buffering, making it difficult to hear. By transmitting the intermediate information in a unit of unit and converting it into voice on the user terminal side, even if a delay occurs, it becomes a unit of an accent phrase or the like, so it is possible to reduce difficulty in hearing.

3) At the time of learning, analysis processing is executed on the user terminal side, and the feature amount is transmitted, so that the amount of transmission data can be reduced compared to the case of transmitting the voice as it is, and only intermediate information is transmitted to the user terminal at the time of synthesis In addition, the amount of data transmitted and received between the user terminal and the server through the network can be reduced. Thereby, a user's expense burden can be reduced.
Therefore, the above-described third problem can be solved by the effects 1) to 3).

  In addition, the feature value analysis processing during learning and the speech synthesis processing from the intermediate information are theoretically reverse processes, and there are many common parts in the calculation algorithm. By installing the feature amount analysis unit and the waveform generation unit that performs synthesis processing in the user terminal, there are many parts that can be used in common in terms of program code, and the program code size that should be installed on the user terminal side is simply the learning unit. This can be reduced as compared with the case where the synthesis unit is mounted. Therefore, it is possible to obtain an effect that the development cost of the program can be reduced and application to a terminal with a small amount of installed memory is possible.

The figure which shows the whole structure of one Example of the speech learning and the synthesis | combination system by this invention. The figure which shows the structure of the user terminal in FIG. The figure which shows the structure of the server in FIG. The sequence diagram for demonstrating the process sequence of one Example of the speech learning and the synthesis | combination system by this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows the overall configuration of an embodiment of a speech learning / synthesizing system according to the present invention. A user terminal 100 and a server 200 are connected to each other via a network 10, and in this example, these users are connected. The terminal 100 and the server 200 constitute a speech learning / synthesis system.

FIG. 2 shows the configuration of the user terminal 100. First, the configuration of the user terminal 100 will be described with reference to FIG.
In this example, the user terminal 100 includes a feature amount analysis unit 110, a text preprocessing unit 120, a DB request unit 130, a waveform generation unit 140, an input unit 150, an output unit 160, a network interface 170, and a control unit 180. The voice data and text for learning are input from the input unit 150.

The feature amount analysis unit 110 analyzes and extracts a feature amount (speech feature amount) from the input speech data for learning. The feature amount includes, for example, a spectrum, a fundamental frequency (F0), a phoneme duration, and the like.
The text preprocessing unit 120 performs text preprocessing according to the type of the input text, and performs processing for converting character codes and removing tags and headers that are not subject to speech synthesis from emails or HTML texts.

  The waveform generation unit 140 generates synthesized speech from the intermediate information received from the server 200. The intermediate information is, for example, a fundamental frequency (F0), a phoneme duration, a model index, or the like. A sound source DB (database) is used if necessary when generating synthesized speech from the intermediate information. Whether or not the sound source DB is necessary depends on the contents of the intermediate information. This point will be described in detail later.

The DB requesting unit 130 requests the server 200 to transmit the sound source DB if the sound source DB is necessary when the waveform generation unit 140 generates the synthesized speech. In FIG. 2, the sound source DB 190 that has been requested and received in this manner is illustrated by a broken line.
The synthesized speech generated by the waveform generation unit 140 is output from the output unit 160. Note that the control unit 180 controls the operation of the user terminal 100 as a whole, and the network interface 170 is connected to the network 10 and enables communication with the server 200.

Next, the configuration of the server 200 will be described with reference to FIG.
The server 200 includes a DB generation unit 210, an intermediate information generation unit 220, a network interface 230, and a control unit 240. The network interface 230 communicates with the user terminal 100. The control unit 240 controls the operation of the server 200 as a whole.

The DB generation unit 210 generates a sound source DB necessary for speech synthesis using the feature amount transmitted from the user terminal 100. Since the sound source DB basically needs to have speaker characteristics, a different one is generated for each speaker. In FIG. 3, the sound source DB generated for each speaker is indicated by 250 ₁ to 250 _N.

The intermediate information generation unit 220 generates intermediate information from the text transmitted from the user terminal 100.
FIG. 4 shows a processing procedure of the speech learning / synthesis system including the user terminal 100 and the server 200 having the above-described configuration, and the processing procedure and details of each processing will be described below.

<Learning>
Learning voice data is input to the user terminal 100 (step S11). The user terminal 100 analyzes and extracts the feature amount from the input voice data (step S12). Examples of the feature amount include a spectrum, a fundamental frequency (F0), and a phoneme duration.
There are various spectrum analysis methods. For example, classically, there are a frequency analysis by FFT and a spectrum estimation method by LPC analysis. Also, estimation method based on sine wave superposition model (Kameoka et al., “Derivation of parameter optimization algorithm for sine wave superposition model”, IEICE Technical Report, Vol.106, EA2000-97, pp.49-54, 2006), STRAIGHT analysis (H. Kawahara et al, “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a reptitive structure in sounds”, Speech Communication, Vol. 27, No. 3-4, pp. 187-207, 1999) have been proposed.

Various F0 analysis methods have been proposed. For example, the above-described STRAIGHT analysis method includes the F0 estimation method, and is one of the representative methods of F0 estimation.
Phoneme segmentation using HMM (Ljolje A., and Riley MD, “Automatic Segmentation and Labeling of Speech”, Proc. Of ICASSP'91, pp.473-476, 1991) Is a typical method. However, what kind of feature value is analyzed depends on the synthesis method, and is determined. If the data of F0 and phoneme duration length is not required at the time of synthesis, F0 and phoneme duration need not be analyzed.

The analyzed and extracted feature amount is transmitted to the server 200 (step S13). The server 200 generates a sound source DB using the received feature amount (step S14).
There are various methods for generating the sound source DB. For example, in the case of a segment base, the closed loop method (Takashima et al., “Analytical generation of an optimal speech segment based on closed loop learning”, theory of theory, J83 -D-II, No.6, pp.1405-1411, 2000), a form like a segment database is typical, and when it is based on a statistical model such as HMM, a speaker based on HMM A model creation method (see Non-Patent Document 1 described above) is a typical method.

<Synthesis>
A text is input to the user terminal 100 (step S21). The user terminal 100 preprocesses the input text (step S22), and transmits the preprocessed text to the server 200 (step S23). The server 200 analyzes the received text (step S24) and determines prosody information such as reading information and accent.

Text analysis processing mainly consists of morphological analysis processing and reading / accenting processing, but there are various conventional methods for these processing methods. For example, the text analysis processing is based on the methods described in Documents 1 and 2 below. Can also be processed.
Reference 1: Japanese Patent No. 3379643 “Morphological analysis method and program
Recorded recording medium "
Reference 2: Japanese Patent No. 3518340 “Reading Prosody Information Setting Method and Apparatus and Reading Prosody Information”
Storage medium storing information setting program "

The text analysis is performed by the intermediate information generation unit 220. After the text analysis, the intermediate information generation unit 220 generates intermediate information (step S25). The generated intermediate information is transmitted from the server 200 to the user terminal 100 (step S26).
Intermediate information generation is roughly divided into a prosodic parameter generation step and a synthesis parameter generation step.

a) Prosodic Parameter Generation Step Various prosodic parameters are obtained based on morphological information, readings, and prosodic information. Here, there are F0, phoneme duration, power, etc. as prosodic parameters, but there are conventional methods for obtaining them, for example, based on F0 data included in the sound source DB by the method described in the following document 3. Thus, the pitch (basic frequency) can be obtained, and the phoneme duration can also be obtained based on the duration length data included in the sound source DB by the method described in Reference 4 below.
Reference 3: Japanese Patent No. 3240691 “Voice Recognition Method”
Reference 4: MDRiley, “Tree-based modeling for speech synthesis” In G. Bailly, C. Benoit, and TRSawallis, editors, Talking Machines: Theories, Models, and Designs, pp.265-273, Elsevier, 1992
Note that the sound source DB is not required when using a method in which F0 and phoneme duration are determined completely by rule, such as a classic point pitch model or a duration model with equal characteristics of beats.

b) Synthesis Parameter Generation Step Information necessary for synthesis is generated using the prosodic parameters described above. The specific information to be generated depends on the synthesis method.
b-1) In the case of unit connection type A unit sequence that is an optimal combination of units matching the reading information and prosodic parameters obtained as described above is determined based on the sound source DB. For example, the series of synthesis units can be determined by the method described in Document 5 below.
Document 5: Japanese Patent No. 3515406 “Speech Synthesis Method and Device”

  The subsequent processing (intermediate information generation and intermediate information transmission) has two forms. One is a mode in which only segment index information indicating the position of the segment data in the sound source DB is transmitted as intermediate information as a synthesis parameter. Another mode is a mode in which segment data is sequentially read from the sound source DB based on the index information, the segment data is combined to generate a sequence of spectrum feature values, and then the spectrum feature values are transmitted as intermediate information. .

b-2) Statistical Model Type For example, as described in the above-mentioned paper by Yoshimura et al. (Non-Patent Document 1), an optimum context-dependent HMM model is selected from the above reading information and prosodic parameters using a decision tree. .

  Subsequent processing (intermediate information generation and intermediate information transmission) has two forms as in the unit connection type. One is a mode in which only model index information indicating which model is included in the sound source DB is transmitted as intermediate information as a synthesis parameter. In another form, model data is sequentially read out from the sound source DB based on the index information, and as described in the above-mentioned paper by Yoshimura et al., A sequence of spectral feature values is generated from the model, and then the spectral feature values are transmitted as intermediate information. It is a form.

  When receiving the intermediate information, the user terminal 100 generates synthesized speech from the received intermediate information (step S27). In both cases b-1) and b-2), when a spectral feature quantity sequence is received as intermediate information, the sound source DB is unnecessary, and a speech waveform is simply generated from the spectral feature quantity sequence. . The generation method of the speech waveform depends on the analysis method of the spectral feature amount. For example, if the feature amount is analyzed by the above-described STRAIGHT analysis method, it may be synthesized by the STRAIGHT synthesis method.

  On the other hand, in the case of b-1), when receiving segment index information as intermediate information, a sound source DB is required on the user terminal 100 side, and the sound source DB is received from the server 200 before speech synthesis. There is a need. The user terminal 100 requests the server 200 to transmit the sound source DB (step S31), and the server 200 transmits the sound source DB to the user terminal 100 based on the request (step S32).

  The waveform generation unit 140 of the user terminal 100 sequentially reads the segment data from the sound source DB based on the segment index information, combines the segment data to generate a spectrum feature amount, and then uses the spectrum feature amount as described above. Generate a speech waveform.

In the case of b-2), when the model index information is received as intermediate information, the sound source DB is necessary on the user terminal 100 side, and the user terminal 100 requests the server 200 to transmit the sound source DB (step S31). The server 200 transmits the sound source DB to the user terminal 100 based on the request (step S32).
The waveform generation unit 140 of the user terminal 100 sequentially reads out the model data from the sound source DB based on the index information, and generates a sequence of spectral feature values from the model as described in the above-mentioned Yoshimura et al. Generate a speech waveform from the quantity.

  As described above, in this example, among the processes related to speech learning and synthesis, processes close to the interface to the user, such as processes immediately after speech input and processes immediately before speech output, are executed on the user terminal 100 side. In other words, in the calculation process of the feature amount analysis and the voice synthesis process, a process having many common points is executed on the user terminal 100 side.

  Note that the user terminal 100 is not limited to a portable terminal such as a telephone, but may be a PC (personal computer), for example. In the case of a PC, there is a form in which a plurality of users (speakers) are shared. In this case, the method of speech synthesis for each speaker may be different, that is, the content of the intermediate information generated in the server 200 may be different for each speaker. Since whether or not the sound source DB is necessary in the user terminal 100 depends on the speech synthesis method, whether or not the user terminal 100 holds the sound source DB is determined for each speaker, that is, whether or not the sound source DB is different depending on the speaker. Occurs.

Claims (4)

A speech learning / synthesizing system that learns input speech data and generates synthesized speech for input text based on the learning,
The user terminal to which the voice data and the text are input, and a server connected to the user terminal via a network,
The user terminal includes a feature amount analysis unit that analyzes and extracts a feature amount from the speech data, and a waveform generation unit that generates the synthesized speech from intermediate information,
The server includes a DB generation unit that generates a sound source DB using the feature amount, and an intermediate information generation unit that generates the intermediate information from the text,
The feature amount and the text are transmitted from the user terminal to the server,
A speech learning / synthesis system characterized in that the intermediate information is transmitted from the server to the user terminal. The speech learning and synthesis system according to claim 1,
The user terminal includes a DB request unit that requests the server to transmit the sound source DB,
The speech learning / synthesis system, wherein the server is configured to transmit the sound source DB to the user terminal based on the request. A speech learning / synthesizing method that learns input speech data and generates synthesized speech for the input text based on the learning,
A user terminal and a server connected via a network;
The above learning
A process of analyzing and extracting feature values from the input voice data by the user terminal;
A process in which the user terminal transmits the feature amount to the server;
The process includes generating a sound source DB using the feature received by the server,
The above synthesis is
A process in which the user terminal transmits the input text to the server;
Generating intermediate information from the text received by the server;
A process in which the server transmits the intermediate information to the user terminal;
A speech learning / synthesizing method comprising a step of generating synthesized speech from intermediate information received by the user terminal. The speech learning / synthesis method according to claim 3,
A process in which the user terminal requests the server to transmit the sound source DB;
A speech learning / synthesizing method comprising: a step in which the server transmits the sound source DB to the user terminal based on the request.

Images