KR20210109222A - Device, method and computer program for synthesizing voice - Google Patents

Abstract

A device for synthesizing voice based on a tree-structured corpus database may include: a collection part for collecting sample text data and example recording data corresponding to the example text data; a derivation part for deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase, based on the collected example text data and example recording data; a database generation part that generates the tree-structured corpus database based on the derived example sentence, example phrase, and phoneme sequence information, and a synthesis part for synthesizing synthesized voice data corresponding to a synthesis request based on the tree-structured corpus database.

Description

DEVICE, METHOD AND COMPUTER PROGRAM FOR SYNTHESIZING VOICE

The present invention relates to an apparatus, method and computer program for synthesizing speech.

The corpus-based speech synthesis method can generate a synthesized sound of high sound quality by selecting a unit necessary for synthesis from a database composed of a voice in the form of a synthesis unit, and connecting them appropriately to generate a synthesized sound. Such a corpus-based speech synthesis method can generate a synthesized sound at a high speed because it takes less time to search for a unit required for speech synthesis, and the construction cost is low because a separate GPU module is not installed. However, the corpus-based voice synthesis method requires a large amount of voice data, and since the sound quality of the synthesized sound is determined according to the data quality of the voice corpus (that is, the labeling accuracy of the recorded data), it takes a lot of time to tune the synthesized sound. .

Deep learning-based speech synthesis technology, which is being actively developed recently, is a technology that reads text as a specific person's voice and provides a more natural, high-quality synthesized speech. However, the deep learning-based speech synthesis technology requires the use of a GPU because of the large amount of deep learning computations, and thus the cost of hardware is high. In addition, the deep learning-based speech synthesis technology has disadvantages in that relatively many calculations are required not only in the training process of speech synthesis but also in the synthesis process, the GPU is essential in the speech synthesis process, and the synthesis sound generation speed is slow.

Korean Patent Publication No. 10-0769033 (registered on October 16, 2007)

The present invention is to solve the problems of the prior art described above, and example sentences derived from collected example text data and example recorded data, at least one example phrase included in the example sentences, and at least one example phrase included in the example phrase We want to create a tree-structured corpus database based on one phoneme sequence information. In addition, the present invention intends to synthesize synthesized speech data corresponding to a synthesis request based on a tree-structured corpus database.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the apparatus for synthesizing speech based on a tree-structured corpus database according to the first aspect of the present invention includes example text data and example recording corresponding to the example text data. a collection unit for collecting data; a derivation unit for deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data; a database generator for generating a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information; and a synthesizer for synthesizing synthesized speech data corresponding to a synthesis request based on the tree-structured corpus database.

According to a second aspect of the present invention, a method for synthesizing speech based on a tree-structured corpus database through a speech synthesis apparatus includes: collecting sample text data and example recording data corresponding to the example text data; deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data; generating a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information; and synthesizing synthesized speech data corresponding to the synthesis request based on the tree-structured corpus database.

A computer program stored in a medium including a sequence of instructions for synthesizing a voice based on a tree-structured corpus database according to the third aspect of the present invention, when executed by a computing device, corresponds to the example text data and the example text data. collecting sample recording data, and deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data, , a sequence of commands for generating a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information, and synthesizing synthesized speech data corresponding to a synthesis request based on the tree-structured corpus database. can do.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention includes example sentences derived from the collected example text data and example recorded data, and at least one example phrase and example phrases included in the example sentences. A tree-structured corpus database may be generated based on at least one piece of phoneme sequence information. In addition, the present invention can synthesize synthesized speech data corresponding to a synthesis request based on a tree-structured corpus database.

Through this, the present invention searches for example sentences or example phrases corresponding to a synthesis request in a tree-structured corpus database, and generates synthesized speech data using the searched example sentences and example phrases, so that the synthesis speed can be increased.

In addition, since the present invention generates synthesized speech data using continuous phoneme sequence information related to the searched example sentence (or example phrase), an additional smoothing operation is not required, so that the synthesis speed can be increased, and continuous phoneme sequence information Because it is used, the sound of the synthesized voice data is natural and there is no distortion or noise.

1 is a block diagram of a speech synthesis system according to an embodiment of the present invention.
2 is a block diagram of the speech synthesis apparatus shown in FIG. 1 according to an embodiment of the present invention.
3 is a diagram for explaining a method of labeling an example sentence through phoneme analysis and voice recognition according to an embodiment of the present invention.
4 is a diagram for explaining an existing corpus database.
5A to 5C are diagrams for explaining a corpus database having a tree structure, according to an embodiment of the present invention.
6 is a flowchart illustrating a method of generating a tree-structured corpus database according to an embodiment of the present invention.
7 is a flowchart illustrating a method of generating a tree-structured corpus database according to another embodiment of the present invention.
8A to 8C are diagrams for explaining a method of synthesizing speech using a tree-structured corpus database, according to an embodiment of the present invention.
9 is a flowchart illustrating a method of synthesizing speech using a tree-structured corpus database according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in the present specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, detailed contents for carrying out the present invention will be described with reference to the accompanying configuration diagram or process flow diagram.

1 is a block diagram of a speech synthesis system according to an embodiment of the present invention.

Referring to FIG. 1 , a speech synthesis system may include a speech synthesis apparatus 100 and a user terminal 110 . However, since the speech synthesis system of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 , and may be configured differently from FIG. 1 according to various embodiments of the present invention.

In general, each component of the speech synthesis system of FIG. 1 is connected through a network (not shown). A network refers to a connection structure that enables information exchange between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The speech synthesis apparatus 100 may collect example text data and example recording data corresponding to the example text data.

The speech synthesis apparatus 100 may derive an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example sentence based on the collected example text data and example recording data. can

The speech synthesis apparatus 100 may generate a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information.

The user terminal 110 may transmit a request phrase for speech synthesis and a synthesis request including an emotion tag corresponding to the request phrase to the speech synthesis apparatus 100 .

The speech synthesis apparatus 100 may synthesize synthesized voice data corresponding to the synthesis request of the user terminal 110 using a tree-structured corpus database, and may provide the synthesized synthesized voice data to the user terminal 110 .

Hereinafter, the operation of each component of the speech synthesis system of FIG. 1 will be described in more detail.

FIG. 2 is a block diagram of the speech synthesis apparatus 100 shown in FIG. 1 according to an embodiment of the present invention.

Referring to FIG. 2 , the speech synthesis apparatus 100 may include a collection unit 200 , a derivation unit 210 , a database generation unit 220 , and a synthesis unit 230 . However, the speech synthesis apparatus 100 shown in FIG. 2 is only one embodiment of the present invention, and various modifications are possible based on the components shown in FIG. 2 .

The collection unit 200 may collect a plurality of example data including example text data and example recording data corresponding to the example text data.

Also, the collection unit 200 may collect emotion tag information for each example sentence of example text data included in a plurality of example data.

The pre-processing unit (not shown) may pre-process the sample recording data corresponding to the collected example text data according to a preset format. For example, the preprocessor (not shown) may perform preprocessing to adjust the length of silence between example sentences constituting the example recorded data. Also, the pre-processing unit (not shown) may perform pre-processing to adjust the volume level of the example recording data to a preset volume level. In addition, the pre-processing unit (not shown) may perform pre-processing to correct the example recorded data including a pronunciation error during recording by comparing the collected example text data and example recording data corresponding to the example text data.

The derivation unit 210 derives an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information (triphone information) included in the example phrase based on the collected example text data and example recording data. can do.

The derivation unit 210 may separate the example sentences into at least one example phrase through analysis of morphemes and dependency structures with respect to example sentences constituting the collected example text data. In addition, the derivation unit 210 may perform a voice activity detection (VAD) operation on example sentences constituting the collected example recording data to designate a section having a long silence as a separation position of the example phrase.

The derivation unit 210 may separate the example sentence from the example phrase based on the analysis result of the morpheme and the dependency structure for the example sentence and the separation position of the example phrase by the VAD operation.

The database generator 220 may generate a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information.

A comparison will be made between the existing corpus database and the tree-structured corpus database of the present invention with reference to FIGS. 4 to 5C .

Referring to FIG. 4 , the existing corpus database provides phoneme sequence information for a plurality of phoneme sequences constituting each example sentence (eg, phoneme start and end position information, phoneme section and intensity, standard deviation for phoneme sections, phonemes) It consists only of a phoneme sequence table including the standard deviation of the intensity of the section).

Referring to FIG. 5A , the tree-structured corpus database is a database configured in a tree format with information on example sentences and example phrases related to phoneme sequences, and consists of example sentence tables, example phrase tables, and phoneme sequence tables related to each phoneme sequence. do. Because the tree-structured corpus database contains information on example sentences and example phrases, it is possible to perform a faster search for example sentences or example phrases for speech synthesis, and Information on related phoneme sequences can be provided.

Referring to FIG. 5B , the database generator 220 may generate an example sentence table based on example sentences, emotion tag information corresponding to the example sentences, and management information for at least one example phrase included in the example sentences. . Here, the example sentence table contains example sentences, sentence numbers of example sentences, sentence text length, sentence time information and sentence types (declarative sentences, interrogative sentences, etc.), main vowel length and distribution information of example sentences, and example sentences included in example sentences. management information (eg, the number and sequence number of example phrases included in the example sentence) and emotion tag information corresponding to the example sentence may be included.

The database generator 220 may generate an example phrase table based on example phrases, order information of the example phrases, and management information on at least one phoneme sequence information included in the example phrases. Here, the example phrase table contains example phrases, sequence numbers of example phrases, phrase text lengths of example phrases, phrase times and phrase positions, major vowel length and distribution information of example phrases, number of phoneme sequences included in example phrases, and phoneme sequence information. It may include management information (eg, a sequence number of a phoneme sequence, information on start and end positions of a phoneme sequence).

The database generator 220 may generate a phoneme sequence table based on phoneme sequence information and order information of the phoneme sequence information. Here, the phoneme sequence table includes a phoneme sequence (triphone), a sequence number of the phoneme sequence, a phoneme interval (position information or phoneme length at the beginning and end of a phoneme), phoneme strength, standard deviation of phoneme interval, and standard deviation of phoneme strength phoneme sequence information and order information of the phoneme sequence information.

The database generator 220 may create a tree-based corpus database by mapping the example sentence table, the example phrase table, and the phoneme sequence table to each other.

The database generator 220 may map the example sentence table and the example phrase table based on the management information included in the example sentence table and the order information included in the example sentence table.

The database generator 220 may map the example phrase table and the phoneme sequence table based on management information included in the example phrase table and order information included in the phoneme sequence table.

As such, the reason for configuring the corpus database in a tree structure will be described with reference to FIG. 5C.

Referring to FIG. 5C , the derivation unit 210 converts an example sentence including 'If you want to listen to USB music, tell me to play USB music', a first example phrase including 'If you want to listen to USB music' and 'USB can be separated into a second example phrase containing 'Tell me to play music'. In fact, the example text data provided to the voice actor who records the sample recording data is provided with spaces marked. At this time, through the analysis of the morpheme and dependency structure of the example text data, the meaning is divided (e.g., in a sentence, an adjective phrase, a phrase with a change in meaning (e.g., before a negative word), individual information (number, etc.)) Alternatively, a short space to read may be specified. Phoneme sequence information is not connected to phonemes before and after the long spaced reading part, and phoneme sequence information is connected to phonemes in the short spaced reading part.

The derivation unit 210 may separate the example phrases from the long spaced reading part of the example recorded data. In addition, the derivation unit 210 divides a portion above a certain volume in the example recording data into a VAD section, divides a portion corresponding to a certain volume or less in the example recording data into a silence section, and the length for each silence section is more constant than the average The part exceeding the ratio (that is, the part that reads with a long space) can be designated as the dividing point of the example phrase.

The calculator (not shown) may calculate a minimum value of the cost of the example sentences during learning or speech synthesis. In addition, the calculator (not shown) may calculate the cost of each phrase constituting the example sentence. Here, the calculation cost may be calculated through [Equation 1], and the calculated cost may be used to generate a synthesized sound. As a result, when speech synthesis is performed using continuous speech sequence information, a natural synthesized sound can be generated.

[Equation 1]

cost of sentence = w1*TC + w2*JC

Here, TC (Target Cost) is calculated through a specified formula such as MSE (Mean Squared Error), etc., and JC (Join Cost) is higher as there is more voice sequence information in the same phrase. do.

The derivation unit 210 may derive phoneme sequence information (eg, Tri-Phone) constituting each example phrase with respect to each separated first example phrase and second example phrase. For example, , in the case of 'Play', first phoneme sequence information consisting of 'ㅡㄹ', second phoneme sequence information consisting of 'ㅡㄹㅓ', third phoneme sequence information consisting of 'ㄹㅓㅓ', and 'ㅓㅓㅝ' The fourth phoneme sequence information composed of can be derived, and the approximate location of phoneme sequence information can be derived from example phrases through deep learning in connection with a voice recognition system and a phoneme analysis system.

Hereinafter, a method of constructing a tree-structured corpus database will be described.

The database generator 220 divides the example text data and the example recording data into the first example data and the second example data, and includes it in the second example data using a corpus database of a tree structure generated based on the first example data. Any one of the example sentences (synthesized voice data) can be synthesized. For example, the database generator 220 classifies the example data of a first ratio (eg, 50%) of the plurality of example data including the collected example text data and example recording data into the first example data, and the remaining Example data of 1 ratio may be classified as second example data.

The database generator 220 may optimize the tree-structured corpus database in a direction to reduce the difference by comparing the synthesized example sentences with the example recorded data included in the second example data.

In another embodiment, the database generating unit 220 classifies the example data of a second ratio (eg, 40%) of the plurality of example data including the collected example text data and example recording data into the first example data, Example data of 1 ratio may be classified as second example data, and example data of a third ratio (eg, 20%) may be classified as third example data. Here, the first example data and the second example data may be data to be used in a training process of speech synthesis, and the third example data may be data to be used in a final test of speech synthesis.

The database generator 220 may perform phoneme analysis and voice recognition on the example recorded data included in the first example data to select an initial position of the phoneme constituting the example recorded data. In this case, the database generator 220 may select an initial position of a phoneme constituting the example recording data included in the first example data in connection with the voice recognition system and the phoneme analysis system.

Here, the voice recognition system provides phoneme and time information for the recognized sentence, but because it uses a unit different from the phoneme unit used in the voice synthesis apparatus 100 due to its characteristics, the pronunciation position of the phoneme required by the voice synthesis apparatus 100 is determined. does not provide accurate In the case of a phoneme analysis system, a frequency section in which the result data of phoneme analysis shows a pattern similar to a specific vowel is determined as a vowel, and a section in which the corresponding frequency component is maintained is set as a vowel section. However, although the phoneme analysis system provides section information of vowels, it is difficult to identify consonants.

For this reason, it is necessary to use both a voice recognition system and a phoneme analysis system to determine the initial position of a phoneme. Referring to FIG. 3 for a moment, the database generator 220 provides phoneme and time information for the example recording data 30 including 'If you want to listen to USB music, please tell me to play USB music' through the voice recognition system. , and may receive vowel generation time information of the example recording data 30 through the phoneme analysis system. In addition, the database generating unit 220 uses the phoneme and time information for the example recorded data 30 and the vowel generation time information of the example recorded data 30 to the initial position 301 of the phoneme of the example recorded data 30 . can be selected.

The database generator 220 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the first example data. Specifically, the database generator 220 may extract the length of each phoneme section including the initial position of the phoneme, and the pitch and stress of the phoneme section from the example recording data included in the first example data.

The database generator 220 generates a tree structure based on example text data included in the first example data and example sentences derived from example recording data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. 1 You can create a corpus database. In addition, the database generator 220 is configured to construct a tree structure based on information on the initial position of phonemes, the length of each phoneme section, and the pitch and stress of the phoneme section derived from the example recording data included in the first example data. A first corpus database may be constructed.

The database generator 220 may generate the first synthesized speech data corresponding to any one example sentence included in the second example data by using the tree-structured corpus database generated based on the first example data.

The database generator 220 may compare the first synthesized voice data with the example recorded data included in the second example data to optimize the first corpus database having a tree structure in a direction to reduce a difference value. That is, the database generator 220 may optimize the first corpus database by optimizing the length (start and end positions of phonemes) of at least one phoneme section constituting the first corpus database.

The database generator 220 may perform phoneme analysis and voice recognition on the example recorded data included in the second example data to select an initial position of the phoneme constituting the example recorded data.

The database generator 220 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the second example data. Specifically, the database generator 220 may extract the length of each phoneme section including the initial position of the phoneme, the pitch of the phoneme section, and the stress from the example recording data included in the second example data.

The database generator 220 generates a tree structure based on example text data included in the second example data and example sentences derived from example recording data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. 2 You can create a corpus database. In addition, the database generator 220 is configured to construct a tree structure based on information on the initial position of phonemes, the length of each phoneme section, and the pitch and stress of the phoneme section derived from the example recording data included in the second example data. A second corpus database may be constructed.

The database generator 220 may generate second synthesized speech data corresponding to any one example sentence included in the first example data by using the second corpus database having a tree structure generated based on the second example data. have.

The database generator 220 may compare the second synthesized voice data with the example recorded data included in the first example data to optimize the tree-structured second corpus database in a direction to reduce a difference value.

The database generator 220 may integrate the optimized first corpus database and the second corpus database to generate the tree-structured integrated corpus database.

The database generator 220 may generate third synthesized speech data corresponding to any one example sentence included in the third example data using the integrated corpus database.

The database generator 220 may optimize the tree-structured integrated corpus database in a direction to reduce the difference by comparing the third synthesized voice data with the example recorded data included in the third example data.

When the synthesis request is received from the user terminal 110 , the synthesizer 230 may synthesize synthesized voice data corresponding to the synthesis request based on the tree-structured corpus database. Here, the synthesis request may include a request phrase and an emotion tag corresponding to the request phrase. In addition, the synthesis request may further include information on the user terminal 110 (eg, an authentication value and speed of the user terminal 110 ), height and volume information on the synthesized sound desired by the user terminal 110 , and the like.

For example, the synthesizing unit 230 may include a request phrase including 'If you want to listen to USB music, tell me to play USB music' from the user terminal 110 , and an emotion tag corresponding to a desired dialogue body for the request phrase. (For example, when a synthesis request including an emotion tag corresponding to a 'friendly conversational body' or an emotion tag corresponding to a 'calm news reading body' is received), the request phrase included in the synthesis request in the tree-structured corpus database and An example sentence based on the emotion tag may be extracted, and synthesized voice data may be synthesized using the example sentence.

The synthesizer 230 may extract an example sentence from the tree-structured corpus database based on the request phrase and the emotion tag, and synthesize synthesized speech data based on the extracted example sentence. Referring to FIG. 8A , the synthesizing unit 230 has an emotion tag of the request phrase for a request phrase including 'If you want to listen to USB music, tell me to play USB music', 'If you want to listen to USB music' Extract an example sentence including a first example phrase (801) including , and a second example phrase (803) including 'Tell me to play USB music' from the example sentence table of the corpus database in a tree structure, and , synthesized speech data may be generated using phoneme sequence information (TP) related to the extracted example sentence.

Conventionally, the minimum value of the target value and the join cost for each phoneme sequence according to the analysis of language and prosody of the requested phrase is obtained and synthesized through a smoothing process. , the present invention retrieves the same example sentence (or example phrase) as the emotion tag and the request phrase received from the user terminal from the corpus database of a tree structure without having to calculate the target value and the join value for the request phrase, and the retrieved example sentence (or Since the synthesized voice data is generated using continuous phoneme sequence information related to the example sphere), the generation time of the synthesized sound can be shortened. In addition, since the present invention does not require an additional smoothing operation, the synthesis speed can be increased, and since continuous phoneme sequence information is used, the sound of the synthesized voice data is natural and distortion or noise does not occur.

If the synthesis unit 230 cannot extract an example sentence corresponding to the request phrase, it extracts an example phrase from the corpus database of a tree structure based on at least one request phrase included in the request phrase, and Synthetic voice data may be synthesized based on the synthesized speech data.

When the synthesis unit 230 cannot extract the same example phrase as the at least one request phrase included in the request phrase, at least one request phrase included in the request phrase includes at least two example phrases included in the tree-structured corpus database. It can be determined whether a combination can be derived. The synthesizer 230 may extract two example phrases determined to be derivable from the tree-structured corpus database, and synthesize synthesized speech data based on the extracted example phrases.

8B and 8C together, an example sentence corresponding to a request phrase including 'If you want to listen to USB music, tell me to play USB music' does not exist in the example sentence table of the corpus database of the tree structure. In this case, the synthesizing unit 230 tree example phrases corresponding to each of the first request phrase including 'If you want to listen to USB music' and the second request phrase including 'Tell me to play USB music' among the request phrases. You can search for example phrase tables in the corpus database of structures. If an example phrase 803 identical to the second request phrase including 'Tell me to play USB music' exists in the example phrase table, the second request phrase using phoneme sequence information related to the example phrase 803 Since the synthesized voice data can be synthesized, there is no need to obtain a separate candidate for sequence information, thereby increasing the synthesis speed of the synthesized voice data.

The synthesis unit 230 includes an example phrase including 'If you want to listen to USB music' and an example including 'If you want to listen to Genie music', in which a part of the first request phrase including 'If you want to listen to USB music' is the same from the example phrase table. The phrase is extracted, and the same part as the part of the first requested phrase ('USB music' and 'If you want to listen to music') is extracted from the extracted example phrase, and the phoneme sequence information related to the extracted part is used for the first requested phrase. Synthetic voice data can be synthesized.

Meanwhile, those skilled in the art will fully understand that the collecting unit 200, the derivation unit 210, the database generating unit 220, and the synthesizing unit 230 may be implemented separately, or one or more of them may be integrated and implemented. will be.

6 is a flowchart illustrating a method of generating a tree-structured corpus database according to an embodiment of the present invention.

Referring to FIG. 6 , in step S601, the voice synthesis apparatus 100 may perform phoneme analysis and voice recognition on the example recorded data included in the first example data to select an initial position of the phoneme constituting the example recorded data. have.

In operation S603, the speech synthesis apparatus 100 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the first example data.

In step S605, the speech synthesis apparatus 100 generates a tree based on example text data included in the first example data and example sentences derived from example recorded data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. A first corpus database of structures may be created. Also, the speech synthesis apparatus 100 may generate a first corpus database having a tree structure based on the extracted prosody information.

In step S607, the speech synthesis apparatus 100 may generate the first synthesized speech data corresponding to the example sentence included in the second example data by using the first corpus database having a tree structure generated based on the first example data. have.

In operation S609 , the speech synthesis apparatus 100 may optimize the first corpus database in a direction of reducing a difference between the first synthesized speech data and the example recorded data included in the second example data.

In step S611, the speech synthesis apparatus 100 may perform phoneme analysis and voice recognition on the example recorded data included in the second example data to select an initial position of the phoneme constituting the example recorded data.

In operation S613, the speech synthesis apparatus 100 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the second example data.

In step S615, the speech synthesis apparatus 100 generates a tree based on example text data included in the second example data and example sentences derived from example recorded data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. A second corpus database of structures may be created. Also, the speech synthesis apparatus 100 may generate a second corpus database having a tree structure based on the extracted prosody information.

In step S617, the speech synthesis apparatus 100 may generate second synthesized speech data corresponding to the example sentences included in the first example data by using the second corpus database having a tree structure generated based on the second example data. have.

In operation S619 , the speech synthesis apparatus 100 may optimize the second corpus database in a direction to reduce a difference between the second synthesized speech data and the example recorded data included in the first example data.

In operation S621, the speech synthesis apparatus 100 may generate an integrated corpus database in which the first corpus database and the second corpus database are integrated.

In operation S623, the speech synthesis apparatus 100 may generate the third synthesized voice data corresponding to the example sentence included in the third example data using the integrated corpus database.

In operation S625 , the speech synthesis apparatus 100 may optimize the integrated database in a direction to reduce a difference between the third synthesized speech data and the example recorded data included in the third example data.

In the above description, steps S601 to S625 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

7 is a flowchart illustrating a method of generating a tree-structured corpus database according to another embodiment of the present invention.

Referring to FIG. 7 , in step S701, the speech synthesis apparatus 100 may perform phoneme analysis and voice recognition on the example recorded data included in the first example data to select an initial position of the phoneme constituting the example recorded data. have.

In operation S703, the speech synthesis apparatus 100 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the first example data.

In step S705, the speech synthesis apparatus 100 generates a tree based on example text data included in the first example data and example sentences derived from example recorded data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. A first corpus database of structures may be created. Also, the speech synthesis apparatus 100 may generate a first corpus database having a tree structure based on the extracted prosody information.

In step S707, the speech synthesis apparatus 100 may generate the first synthesized speech data corresponding to the example sentence included in the second example data by using the first corpus database having a tree structure generated based on the first example data. have.

In operation S709 , the speech synthesizing apparatus 100 may optimize the first corpus database in a direction of reducing a difference value by comparing the first synthesized speech data and the example recorded data included in the second example data.

In step S711 , the speech synthesis apparatus 100 may perform phoneme analysis and voice recognition on the example recorded data included in the second example data to select an initial position of a phoneme constituting the example recorded data.

In operation S713 , the speech synthesis apparatus 100 may extract prosody information from the example recorded data based on the initial position of the phoneme with respect to the example recorded data included in the second example data.

In step S715, the speech synthesis apparatus 100 generates a tree based on example text data included in the second example data and example sentences derived from example recorded data, example phrases included in the example sentences, and phoneme sequence information included in the example phrases. A second corpus database of structures may be created. Also, the speech synthesis apparatus 100 may generate a second corpus database having a tree structure based on the extracted prosody information.

In step S717, the speech synthesis apparatus 100 may generate second synthesized speech data corresponding to the example sentences included in the first example data by using the second corpus database having a tree structure generated based on the second example data. have.

In operation S719 , the speech synthesis apparatus 100 may optimize the second corpus database in a direction to reduce a difference between the second synthesized speech data and the example recorded data included in the first example data.

In operation S721, the speech synthesis apparatus 100 may generate an integrated corpus database in which the first corpus database and the second corpus database are integrated.

In step S723, the speech synthesis apparatus 100 may generate third synthesized speech data corresponding to the deep learning example sentences included in the deep learning example data generated by the deep learning model using the integrated corpus database.

In step S725 , the speech synthesis apparatus 100 may optimize the integrated database in a direction to reduce a difference by comparing the third synthesized speech data and the deep learning example voice data included in the deep learning example data.

9 is a flowchart illustrating a method for synthesizing speech using a tree-structured corpus database according to an embodiment of the present invention.

Referring to FIG. 9 , in step S901 , the speech synthesis apparatus 100 may collect example text data and example recording data corresponding to the example text data.

In step S903, the speech synthesis apparatus 100 may derive an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data. can

In operation S905, the speech synthesis apparatus 100 may generate a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information.

In operation S907, the speech synthesis apparatus 100 may synthesize synthesized voice data corresponding to the synthesis request based on the tree-structured corpus database. Hereinafter, the synthesis request may include a request phrase and an emotion tag corresponding to the request phrase.

In the above description, steps S901 to S907 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: speech synthesizer
110: user terminal
200: collection unit
210: derivation part
220: database generator
230: synthesis unit

Claims (17)

An apparatus for synthesizing speech based on a tree-structured corpus database, the apparatus comprising:
a collection unit for collecting example text data and example recording data corresponding to the example text data;
a derivation unit for deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data;
a database generator for generating a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information; and
A synthesizer for synthesizing synthesized speech data corresponding to a synthesis request based on the tree-structured corpus database
That comprising a, speech synthesis device.
The method of claim 1,
The database generator generates an example sentence table based on the example sentence, emotion tag information corresponding to the example sentence, and management information on at least one example phrase included in the example sentence.
3. The method of claim 2,
The database generator generates an example phrase table based on the example phrase, order information of the example phrase, and management information on at least one phoneme sequence information included in the example phrase,
and mapping the example sentence table and the example phrase table based on management information included in the example sentence table and order information included in the example phrase table.
4. The method of claim 3,
The database generator generates a phoneme sequence table based on the phoneme sequence information and order information of the phoneme sequence information,
and mapping the example phrase table and the phoneme sequence table based on management information included in the example phrase table and order information included in the phoneme sequence table.
The method of claim 1,
The database creation unit,
dividing the example text data and the example recording data into first example data and second example data;
Any one example sentence included in the second example data is synthesized using a tree-structured corpus database generated based on the first example data,
Comparing the synthesized example sentences with the example recording data included in the second example data and optimizing the tree-structured corpus database in a direction to reduce the difference value, the speech synthesis apparatus.
The method of claim 1,
The synthesis request includes a request phrase and an emotion tag corresponding to the request phrase,
wherein the synthesizer extracts an example sentence from the tree-structured corpus database based on the request phrase and the emotion tag, and synthesizes the synthesized voice data based on the extracted example sentence.
7. The method of claim 6,
If the synthesis unit cannot extract an example sentence corresponding to the request phrase, extract an example phrase from the corpus database of the tree structure based on at least one request phrase included in the request phrase, and the extracted example phrase and synthesizing the synthesized speech data based on
8. The method of claim 7,
The synthesis unit determines whether at least one request phrase included in the request phrase can be derived from a combination of at least two example phrases included in the tree-structured corpus database, and the two example phrases determined to be derivable is extracted from the tree-structured corpus database, and the synthesized speech data is synthesized based on the extracted example phrase.
A method for synthesizing speech based on a corpus database of a tree structure through a speech synthesis device, the method comprising:
collecting sample text data and example recording data corresponding to the example text data;
deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data;
generating a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information; and
and synthesizing synthesized voice data corresponding to a synthesis request based on the tree-structured corpus database.
10. The method of claim 9,
The step of creating the corpus database
and generating an example sentence table based on the example sentence, emotion tag information corresponding to the example sentence, and management information on at least one example phrase included in the example sentence.
11. The method of claim 10,
The step of creating the corpus database
generating an example phrase table based on the example phrase, order information of the example phrase, and management information on at least one phoneme sequence information included in the example phrase; and
and mapping the example sentence table and the example phrase table based on management information included in the example sentence table and order information included in the example phrase table.
12. The method of claim 11,
The step of creating the corpus database
generating a phoneme sequence table based on the phoneme sequence information and order information of the phoneme sequence information;
and mapping the example phrase table and the phoneme sequence table based on management information included in the example phrase table and order information included in the phoneme sequence table.
10. The method of claim 9,
The step of creating the corpus database
dividing the example text data and the example recording data into first example data and second example data;
synthesizing any one example sentence included in the second example data using a tree-structured corpus database generated based on the first example data; and
and optimizing the tree-structured corpus database in a direction to reduce a difference value by comparing the synthesized example sentences with example recorded data included in the second example data.
10. The method of claim 9,
The synthesis request includes a request phrase and an emotion tag corresponding to the request phrase,
The step of synthesizing the synthesized voice data is
and extracting an example sentence from the tree-structured corpus database based on the request phrase and the emotion tag, and synthesizing the synthesized voice data based on the extracted example sentence.
15. The method of claim 14,
The step of synthesizing the synthesized voice data is
When an example sentence corresponding to the request phrase cannot be extracted, an example phrase is extracted from the corpus database of the tree structure based on at least one request phrase included in the request phrase, and based on the extracted example phrase and synthesizing the synthesized speech data.
16. The method of claim 15,
The step of synthesizing the synthesized voice data is
determining whether at least one request phrase included in the request phrase can be derived from a combination of at least two example phrases included in the tree-structured corpus database; and
and extracting two example phrases determined to be derivable from the tree-structured corpus database, and synthesizing the synthesized speech data based on the extracted example phrases.
A computer program stored in a medium including a sequence of instructions for synthesizing speech based on a tree-structured corpus database, the computer program comprising:
When the computer program is executed by a computing device,
collecting sample text data and example recording data corresponding to the example text data;
Deriving an example sentence, at least one example phrase included in the example sentence, and at least one phoneme sequence information included in the example phrase based on the collected example text data and example recording data,
Create a tree-structured corpus database based on the derived example sentences, example phrases, and phoneme sequence information,
and a sequence of instructions for synthesizing synthesized speech data corresponding to a synthesis request based on the tree-structured corpus database.

Images