KR20210032235A - Device, method and computer program for synthesizing emotional voice - Google Patents

Abstract

A device for synthesizing an emotional voice comprises: a synthesis model generation unit which generates a synthesis model based on original voice data and a sentence corresponding to the original voice data; an emotion model generation unit which generates an emotion model based on emotional voice data by emotion and synthesis voice data generated by the synthesis model; a user synthesis model generation unit which inputs user voice data of a user and a sentence corresponding to the user voice data into the generated synthesis model to generate a user synthesis model of the user; a user synthesis model training unit which uses the generated user synthesis model to generate user synthesis voice data, inputs the generated user synthesis voice data into the generated emotion model to generate user emotion voice data, and inputs the generated user emotion voice data into the user synthesis model to train the user synthesis model; and a synthesis emotional voice data generation unit which inputs a request sentence and emotional information into the trained user synthesis model to generate user synthesis emotional voice data.

Description

Apparatus, method, and computer program for synthesizing emotional voices TECHNICAL FIELD

The present invention relates to an apparatus, a method, and a computer program for synthesizing emotional voices.

As the demand for kids services using artificial intelligence increases, many parents need a service that reads fairy tales to their children with their own voice.

However, in a fairy tale service using the voices of parents, since parents are not professional voice actors, the amount of voice recording of the parents is small, and the emotional expression is poor, so it is difficult to extract emotional data from the recorded voices of the parents.

Meanwhile, in order to separately perform emotional synthesis using a specific voice, a large amount of emotional data for the corresponding voice is required. In addition, the method of adding an emotion filter to a voice has a disadvantage in that it requires emotion data for the corresponding voice and time for additional work is consumed.

Korean Patent Publication No. 2008-0060909 (published on Feb. 2008)

The present invention is to solve the above-described problems of the prior art, and input a user's request sentence and emotion information to a user synthesis model learned based on a synthesis model for generating synthesized voice data and an emotion model for generating emotional voice data. Thus, it is intended to generate user synthesized emotion voice data. However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the apparatus for synthesizing emotional speech according to the first aspect of the present invention generates a synthesis model for generating a synthesis model based on original speech data and a sentence corresponding to the original speech data. part; An emotion model generator configured to generate an emotion model based on emotion voice data for each emotion and synthesized voice data generated by the synthesis model; A user synthesis model generating unit for generating a user synthesis model of the user by inputting user voice data of the user and a sentence corresponding to the user voice data to the generated synthesis model; And generating user synthesized voice data using the generated user synthesis model, inputting the generated user synthesized voice data to the generated emotion model to generate user emotion voice data, and generating the generated user synthesized voice data in the user synthesis model. A user synthesis model learning unit that inputs and trains user emotion voice data; And a synthesized emotion voice data generator configured to generate user synthesized emotion voice data by inputting a request sentence and emotion information into the learned user synthesis model.

A method for synthesizing an emotional voice according to a second aspect of the present invention includes: generating a synthesis model based on original voice data and a sentence corresponding to the original voice data; Generating an emotion model based on emotion voice data for each emotion and synthesized voice data generated by the synthesis model; And generating user synthesized emotion voice data by inputting a request sentence and emotion information to the synthesis model and the user synthesis model learned based on the emotion model.

A computer program stored in a medium including a sequence of instructions for synthesizing an emotional voice according to the third aspect of the present invention, when executed by a computing device, generates a synthesis model based on the original voice data and the sentence corresponding to the original voice data. And generate an emotion model based on the emotion voice data for each emotion and the synthesized voice data generated by the synthesis model, and add the user’s user voice data and the sentence corresponding to the user voice data to the generated synthesis model. Input to generate a user synthesis model of the user, generate user synthesis speech data using the generated user synthesis model, and input the generated user synthesis speech data to the generated emotion model to provide user emotion speech data. It includes a sequence of commands for generating, inputting and learning the generated user emotion voice data into the user synthesis model, and inputting a request sentence and emotion information into the learned user synthesis model to generate user synthesized emotion voice data can do.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, the present invention constructs a synthetic model for generating synthetic speech data and an emotion model for generating emotional speech data, and is trained based on the constructed synthetic model and emotion model. By inputting the user's request sentence and emotion information into the user synthesis model, the user synthesized emotion voice data reflecting the emotion suitable for the corresponding request sentence may be generated.

Compared to the existing emotion application method in which a base sound source is created for expressing emotions and additional sound sources are applied or adjusted, the user synthesis model can be trained using the emotion model for each emotion generated in advance, without affecting the synthesis time. , You can apply your emotions.

1 is a block diagram of an apparatus for synthesizing emotional speech according to an embodiment of the present invention.
2A to 2E are diagrams for explaining a method for synthesizing emotional speech according to an embodiment of the present invention.
3A to 3I are diagrams for explaining a method of generating an emotion model according to an embodiment of the present invention.
4 is a flowchart illustrating a method for synthesizing emotional speech according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

In the present specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized by using two or more hardware, or two or more units may be realized by one piece of hardware.

In this specification, some of the operations or functions described as being performed by the terminal or device may be performed instead in a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal or device connected to the server.

Hereinafter, with reference to the accompanying configuration diagram or processing flow chart, it will be described in detail for the implementation of the present invention.

1 is a block diagram of an emotional speech synthesis apparatus 10 according to an embodiment of the present invention.

Referring to FIG. 1, the emotion-voice synthesis apparatus 10 includes a synthesis model generation unit 100, an emotion model generation unit 110, a user synthesis model generation unit 120, a user synthesis model learning unit 130, and a synthesis emotion. A voice data generator 140 may be included. However, the emotion-voice synthesis apparatus 10 shown in FIG. 1 is only an example of implementation of the present invention, and various modifications are possible based on the components shown in FIG. 1.

Hereinafter, FIG. 1 will be described with reference to FIGS. 2A to 3I together.

The synthesis model generator 100 may generate and train the synthesis model 20 based on the original speech data 201 and the sentence 203 corresponding to the original speech data 201. Here, the original voice data 201 may be voice data in which voice actors' voices are recorded. The data pair input for the generation of the synthetic model may consist of a pair of {original speech data, sentence}.

The sentence 203 input to the synthesis model may be a value embedded in phoneme units. For example, if it is embedded like'a: 2, b:3, c: 4', the phoneme unit of'fairy tale' is [c ㅗ ㅐ ㅎ ㅘ], so [5, 29, 13, 20, 30] Can be embedded.

The original voice data 201 input to the synthesis model may be a value obtained by separating frequencies for each unit through a short-time fourier transform (STFT) algorithm after being transformed in a mel spectrogram method. In this case, the size of the unit may be, for example, a size in which 80 frequencies are separated by a mel-scale value for each time unit of 50 ms.

For example, the synthesis model generation unit 100 learns in which time zone each phoneme of the sentence 203 is located in the original voice data 201, and the frequency position at which each phoneme is located in the original voice data 201 and The synthesis model 20 may be trained by learning the intensity of the frequency.

The emotion model generator 110 may generate the emotion model 22 for each emotion based on the emotion voice data 205 for each emotion and the synthesized speech data 207 generated by the synthesis model 20. Here, each of the emotional voice data 205 and the synthesized voice data 207 is voice data for the same sentence. Here, the emotional voice data 205 may be voice data in which voices containing voice actors' emotions are recorded. The data pair input to generate the emotion model 220 may be a pair of {synthetic voice data and emotion voice data}.

The emotion model generator 110 converts the emotion voice data 205 and the synthesized voice data 207 for the same sentence into a spectrogram image form, and analyzes the emotion voice data and synthesized voice data converted to the spectrogram image form. It can be converted into a speech matrix and a synthesized speech matrix.

Referring to FIG. 3A, the emotion model generator 110 may convert synthesized speech data for a'high' sentence 301 into a spectrogram image 303. The horizontal axis of the spectrogram image 303 corresponding to the synthesized speech data for the'high' sentence 301 corresponds to the time axis (sample axis), and the vertical axis corresponds to the frequency axis. The value of a pixel included in the spectrogram image 303 represents the intensity (dB) of sound.

The emotion model generator 110 may convert the spectrogram image 303 corresponding to the synthesized voice data for the'high' sentence 301 into the synthesized voice matrix 305. In this case, the row of the synthesized speech matrix 305 is frequency, the column is time, and the value of the matrix is the intensity of sound.

이고, 시간 단위가 1.25ms이면, 한 시간 단위에 포함되는 샘플의 개수(k)는 [수학식 1]을 통해 도출될 수 있다. The emotion model generator 110 may calculate the number of samples included in a time unit for a plurality of samples included in the synthesized speech matrix 305 based on a preset sample rate. Here, the preset sample rate is

And, if the time unit is 1.25 ms, the number of samples (k) included in one time unit may be derived through [Equation 1].

[Equation 1]

k =

)에 16khz을 대입하면, 1.25ms 시간 단위로 20개의 샘플이 산출될 수 있다. For example, the sample rate preset in [Equation 1] (

If 16khz is substituted for ), 20 samples can be calculated in units of 1.25ms time.

)인 16khz에 따라 1.25ms 시간 단위로 20개의 샘플이 산출되면, 감정 모델 생성부(110)는 합성 음성 행렬(305)의 복수의 열을 20칸(307)씩 분할할 수 있다. 이 때, 합성 음성 행렬(305)이 20칸(307)씩 분할되면,

크기의 합성 음성 행렬(305)이

크기의 합성 음성 행렬(305)로 줄어들게 된다. The emotion model generator 110 may divide the synthesized speech matrix 305 so that samples are included in a preset time unit as many as the number of samples calculated based on the preset sample rate. For example, a preset sample rate (

When 20 samples are calculated in units of 1.25ms time according to 16khz of ), the emotion model generator 110 may divide a plurality of columns of the synthesized speech matrix 305 by 20 spaces 307. At this time, if the synthesized speech matrix 305 is divided by 20 spaces 307,

Synthetic speech matrix of size 305 is

It is reduced to the size of the synthesized speech matrix 305.

The emotion model generator 110 may calculate an average value of the divided synthesized speech matrix (hereinafter referred to as a sub-synthetic speech matrix) for each time unit. For example, the emotion model generation unit 110 may average (313, 315) a plurality of matrix values for each frequency unit for each of the divided plurality of sub-synthetic speech matrices 309 and 311.

Thereafter, when the averaging of the matrix values for each of the sub-synthetic speech matrices 309 and 311 is completed, the emotion model generation unit 110 may generate a normalized synthetic speech matrix 317.

라고 하면, 합성 음성 행렬(317)은 (8000,

)의 크기를 가지게 된다. 3B, the number of time units of the normalized synthesized speech matrix 317

If so, the synthesized speech matrix 317 is (8000,

).

The emotion model generator 110 may derive each distance matrix by comparing components constituting the normalized synthesized speech matrix 317. The emotion model generator 110 is based on the difference between the component (matrix value) of the Nth column of the synthesized speech matrix 317 and the component (matrix value) of the N+1th column, the distance matrix for the synthetic speech matrix 317 ( 321) can be derived. The distance matrix can be derived based on [Equation 2].

[Equation 2]

는 (1, 8000) 크기의 행렬이고,

는

크기의 행렬이고,

는 스칼라 값이고,

는

크기의 행렬이다.here

Is a matrix of size (1, 8000),

Is

Is a matrix of size,

Is a scalar value,

Is

It is a matrix of size.

가 되고, 모든 주파수 값에 대해 i번째 시간 단위와 i+1번째 시간 단위에서의 세기 차를 구하면, 거리 행렬에 대한 값이 도출될 수 있다. Here, the frequency matrix in the ith unit of time is

Is, and if the intensity difference between the i-th time unit and the i+1-th time unit is calculated for all frequency values, a value for the distance matrix can be derived.

-1)의 크기를 갖는다. The emotion model generation unit 110 sums the matrix 319 including the result value according to the absolute value of the difference between the components of the Nth column and the components of the N+1th column of the synthesized speech matrix 317 for each time unit to generate a synthesized speech. The distance matrix 321 for the matrix 317 can be derived. Here, the distance matrix 321 for the synthesized speech matrix 317 is (8000,

It has a size of -1).

3B to 3C, the emotion model generator 110 is a distance matrix 321 of the synthesized voice matrix 317 based on the number of phonemes constituting the synthesized voice data for the'high' sentence 301. The distance value can be extracted from.

The emotion model generator 110 may extract a distance value having an upper value by one less than the number of phonemes constituting the synthesized voice data from the distance matrix 321 of the derived synthesized voice matrix 317.

If the number of phonemes constituting the synthesized speech data is N, the emotion model generation unit 110 extracts a distance value belonging to an upper level corresponding to N-1 among a plurality of distance values included in the distance matrix 321. I can.

Here, the'high' sentence 301 may be divided into phonemes of [ㅎㅏㅣ], and the number of phonemes constituting the sentence 301 is three. Since the number of phonemes for the'high' sentence 301 is three, the emotion model generator 110 may extract the two upper distance values 323 and 325 from the distance matrix 321.

The emotion model generator 110 may check the index value 327 of the extracted distance values 323 and 325.

3C to 3D, the emotion model generator 110 may derive a synthesized speech cluster by distributing the synthesized speech matrix 317 by the number of phonemes according to the identified index value 327. For example, based on the index 2 (the third index in order) and the index 4 (the fifth index in order) included in the index value 327, the emotion model generation unit 110 includes the synthesized speech matrix 317 Synthesized speech clusters 329, 331, and 333 may be derived by dividing s according to positions in the order of the identified index values. That is, the first to third columns corresponding to the index 2 identified in the synthesized speech matrix 317 are divided into the synthesis cluster 329, and the fourth corresponding to the index 4 identified in the synthesized speech matrix 317 From the column to the 5th column, the composite cluster 331 may be divided.

값(인덱스 값(327))으로 수행하며, 나누는 축은 시간 단위로 한다. 즉,

일 때,

,

, ...,

으로 분할될 수 있다. 여기서, Y는 감정 음성 행렬(355)이고,

,

...,

은 합성 음성 군집(319, 321, 323)이다. 예를 들어 X

이고

이라고 하면,

,

,

이 된다. If the synthesized speech matrix 317 is X, the criterion for dividing the synthesized speech matrix 317 is

It is performed by value (index value (327)), and the division axis is by time unit. In other words,

when,

,

, ...,

Can be divided into Here, Y is the emotion speech matrix 355,

,

...,

Is a synthesized speech cluster 319, 321, 323. For example X

ego

Speaking of,

,

,

Becomes.

Referring to FIG. 3E, the emotion model generator 110 may convert emotion voice data for a'high' sentence 301 into a spectrogram image 341. The horizontal axis of the spectrogram image 341 corresponding to the emotion voice data for the'high' sentence 301 corresponds to the time axis (sample axis), and the vertical axis corresponds to the frequency axis. A pixel value included in the spectrogram image 341 represents the intensity (dB) of sound.

The emotion model generator 110 may convert the spectrogram image 341 corresponding to the emotion voice data for the'high' sentence 301 into the emotion voice matrix 343. In this case, the row of the emotion speech matrix 343 is frequency, the column is time, and the value of the matrix indicates the intensity of sound.

The emotion model generator 110 may calculate the number of samples included in a time unit for a plurality of samples included in the emotion speech matrix 343 based on a preset sample rate. Here, the number of samples can be derived through [Equation 1] mentioned above.

)에 16khz을 대입하면, 1.25ms 시간 단위로 20개의 샘플이 산출되고, 이에 따라, 감정 모델 생성부(110)는 감정 음성 행렬(343)의 복수의 열을 20칸(345)씩 분할할 수 있다.The emotion model generator 110 may divide the emotion speech matrix 343 so that samples are included in a preset time unit as many as the number of samples calculated based on the preset sample rate. For example, the preset sample rate of [Equation 1] (

If 16khz is substituted for ), 20 samples are calculated in units of 1.25ms time, and accordingly, the emotion model generator 110 may divide a plurality of columns of the emotion speech matrix 343 by 20 spaces (345). have.

The emotion model generator 110 may calculate an average value of the divided emotion speech matrix (hereinafter referred to as a sub emotion speech matrix) for each time unit. For example, the emotion model generator 110 may average (351, 353) a plurality of matrix values for each frequency unit for each of the divided plurality of sub-emotional voice matrices 347 and 349.

Thereafter, the emotion model generator 110 may generate a normalized emotion speech matrix 355 when the averaging of the matrix values for each of the sub emotion speech matrices 347 and 349 is completed.

라고 하면, 감정 음성 행렬(355)은 (8000,

)의 크기를 가지게 된다. 3F, the number of time units of the normalized emotion speech matrix 355

If so, the emotion voice matrix 355 is (8000,

).

The emotion model generator 110 may derive each distance matrix by comparing components constituting the normalized emotion speech matrix 355. The emotion model generator 110 is a distance matrix for the emotion speech matrix 355 based on a difference between a component (matrix value) of the Nth column of the emotion speech matrix 355 and a component (matrix value) of the N+1th column. 359) can be derived. Here, the distance matrix may be derived based on [Equation 2] described above.

-1)의 크기를 갖는다. The emotion model generator 110 adds the matrix 357 including the result value according to the absolute value of the difference between the components in the Nth column and the components in the N+1th column of the emotion speech matrix 355 for each time unit. A distance matrix 359 for matrix 355 can be derived. Here, the distance matrix 359 for the emotion speech matrix 355 is (8000,

It has a size of -1).

3F to 3G, the emotion model generator 110 is a distance matrix 359 of the emotion speech matrix 355 based on the number of phonemes constituting the emotion speech data for the'high' sentence 301. The distance value can be extracted from.

The emotion model generator 110 may extract a distance value having an upper value by one less than the number of phonemes constituting the emotion speech data from the distance matrix 359 of the derived emotion speech matrix 355.

If the number of phonemes constituting the emotion voice data is N, the emotion model generation unit 110 extracts a distance value belonging to an upper level corresponding to N-1 among a plurality of distance values included in the distance matrix 359. I can.

Here, the'high' sentence 301 may be divided into phonemes of [ㅎㅏㅣ], and the number of phonemes constituting the sentence 301 is three. Since the number of phonemes for the'high' sentence 301 is three, the emotion model generator 110 may extract the two upper distance values 361 and 363 from the distance matrix 359.

The emotion model generator 110 may check the index value 365 of the extracted distance values 361 and 363.

3G to 3H, the emotion model generator 110 may derive an emotion speech cluster by distributing the emotion speech matrix 355 by the number of phonemes according to the identified index value 365. For example, based on the index 2 (the third index in order) and the index 5 (the sixth index in order) included in the index value 365, the emotion model generation unit 110 includes the emotion speech matrix 355 Emotional voice clusters 367, 369, and 371 may be derived by dividing s according to the positions in the order of the identified index values.

값(인덱스 값(365))으로 수행하며, 나누는 축은 시간 단위로 한다. 즉,

일 때,

,

, ...,

으로 분할될 수 있다. 여기서, Y는 감정 음성 행렬(355)이고,

,

...,

은 감정 음성 군집(367, 369, 371)이다. 예를 들어

이고

이라고 하면,

,

,

이 된다. If the emotion speech matrix 355 is Y, the criterion for dividing the emotion speech matrix 355 is

It is performed by a value (index value (365)), and the axis to be divided is by a unit of time. In other words,

when,

,

, ...,

Can be divided into Here, Y is the emotion speech matrix 355,

,

...,

Is an emotional voice cluster (367, 369, 371). E.g

ego

Speaking of,

,

,

Becomes.

,

에 기초하여 합성 음성 군집(329, 331, 333) 각각에 대응하는 행렬

,

에 대한 군집 길이를 다음과 같이 조정할 수 있다. Referring to FIGS. 3D, 3H and 3I together, the emotion model generation unit 110 adjusts the length of the synthesized speech clusters 329, 331, and 333 derived based on the derived emotion speech clusters 367, 369, and 341. I can. Specifically, a matrix corresponding to each of the emotional voice clusters 367, 369, and 371

,

Matrix corresponding to each of the synthesized speech clusters (329, 331, 333) based on

,

The cluster length for can be adjusted as follows.

에 대해

,

, 즉

는 k번째 군집의 시간 단위 길이를 나타낼 때,

인 경우(즉, 합성 음성 군집(323)의 길이가 감정 음성 군집(341)의 길이보다 긴 경우),

행렬에서 가장 작은 값을 가진 인덱스(단위 시간 기준)를

개 뽑아낸 후,

행렬)에서 추출된 인덱스에 해당하는 열(40)을 삭제한다. 삭제 시, 한 개의 열 삭제 후 삭제된 열을 기준으로 앞뒤에 있는 시간 단위 열에 대한 거리 계산을 다시 수행하여 열 삭제를 반복한다.random

About

,

, In other words

Is the length of the k-th cluster in time units,

If (i.e., when the length of the synthesized speech cluster 323 is longer than the length of the emotion speech cluster 341),

The index with the smallest value in the matrix (based on unit time)

After pulling out the dog,

The column 40 corresponding to the index extracted from the matrix) is deleted. When deleting, delete one column and repeat the column deletion by re-calculating the distance for the preceding and following time unit columns based on the deleted column.

인 경우(즉, 합성 음성 군집(331)의 길이가 감정 음성 군집(369)의 길이보다 짧은 경우),

행렬에서 가장 작은 값을 가진 인덱스를

개 뽑아낸 후,

행렬에서 추출된 인덱스에 해당하는 열(373)을 복제한다.

If (i.e., when the length of the synthesized speech cluster 331 is shorter than the length of the emotion speech cluster 369),

The index with the smallest value in the matrix

After pulling out the dog,

The column 373 corresponding to the index extracted from the matrix is duplicated.

The emotion model generator 110 may derive a synthesized emotional voice sequence based on the synthesized voice cluster 375 whose length is adjusted, a phoneme information vector, and a phoneme position vector.

The emotion model generation unit 110 determines whether the derived synthetic emotion speech sequence, the actual emotion speech sequence for the emotion speech data, the synthesized emotion speech sequence derived based on the phoneme information vector and the phoneme position vector is a speech containing emotion. Then, the emotion model may be trained using the determination result.

The emotion model can be composed of a transducer and a discriminator. Here, the converter may perform a function of converting a basic voice into an emotional voice. The discriminator may perform a function of determining whether or not the converted emotional voice is emotion.

)와 음소 정보를 담고 있는 벡터(c), 음소 위치 정보를 담고 있는 벡터(

)는 변환기(F)의 입력값으로 입력될 수 있다. 변환기의 출력값은

으로 나타낼 수 있다. 여기서,

는 합성 음성 데이터가 변환기를 거쳐 감정이 담긴 음성 벡터(행렬)로 변환된 합성 감정 음성 시퀀스일 수 있다. Synthetic speech data with adjusted length (

) And a vector containing phoneme information ( c ), a vector containing phoneme location information (

) May be input as an input value of the converter F. The output of the converter is

It can be represented by here,

May be a synthesized emotional voice sequence in which the synthesized voice data is converted into a voice vector (matrix) containing emotions through a converter.

), 음소 정보를 담고 있는 벡터(c) 및 음소 위치 정보를 담고 있는 벡터(

)는 판별기(D)의 입력값으로 입력될 수 있다. 판별기(D)의 출력값은

으로 나타낼 수 있다. 여기서, d는 확률 분포이고, d가 0에 가까울 수록 감정이 실리지 않은 음성이고, d가 1에 가까울수록 감정이 실린 음성을 의미할 수 있다. The actual emotional voice (Y) for the emotional voice data, the synthesized emotional voice sequence derived by the converter (

), a vector containing phoneme information ( c ), and a vector containing phoneme location information (

) May be input as an input value of the discriminator D. The output value of the discriminator (D) is

It can be represented by Here, d is a probability distribution, the closer d is to 0, the voice is not loaded with emotion, and the closer d is 1, the voice is loaded with emotion.

Each of the transducers (F) and discriminators (D) included in the emotion model is subjected to parameter adjustment through training, and the entropy function for this is as follows.

이며, 판별기는 감정 데이터를 정확하게 1로 출력하고 비감정 데이터는 정확하게 0을 출력할 수 있도록 설정될 수 있다. The discriminator's entropy function is

And the discriminator may be set to accurately output the emotion data as 1 and the non-emotional data to accurately output 0.

이며, 변환기는 변환한 감정 음성이 판별기를 통과하는 것을 목적으로 한다. The converter's entropy function is

And the transducer aims for the converted emotional voice to pass through the discriminator.

이며 결과적으로

가 도출되도록 파라미터를 조정할 수 있다. So the final entropy

And as a result

The parameters can be adjusted so that is derived.

Next, an encoding method performed by the converter will be described.

이 된다. 이후,

에 대해 컨벌루션이 수행된다. 여기서, 컨벌루션은 커널(kernel) 사이즈만큼 합성 음성 데이터를 분할한 후, 분할된 합성 음성 데이터 각각과 커널 가중치를 곱해서 최종 값을 출력하는 구조다. 이 때, 커널의 크기는 임의로 가능하나, 주파수의 성분이 많이 압축될 수 있도록 설정될 수 있다.Synthetic speech data, which is an input value of the converter, is converted into a spectrogram image (2D image) and input, and the spectrogram image corresponding to the synthesized speech data has a variable time length and must be cut to a certain length. At this time, the cutting length can be arbitrarily adjusted. However, if the length is too long, since it causes serious loss of the synthesized voice data, it is cut in units of 12.5ms time. If the time unit time is set to 1.25ms, the synthesized speech data is cut into 10 units. therefore,

Becomes. after,

Convolution is performed on. Here, convolution is a structure in which synthesized speech data is divided by a kernel size, and then a final value is output by multiplying each of the divided synthesized speech data by a kernel weight. In this case, the size of the kernel is arbitrarily possible, but it can be set so that a lot of frequency components can be compressed.

Thereafter, batch normalization may be performed on the convolved result. Through this, learning destabilization for convoluted data is reduced. Batch normalization can be performed based on the average and standard deviation of input values for each channel of the convolution result based on [Equation 3].

[Equation 3]

는 컨벌루션 결과물의 평균이고,

는 표준편차이고,

은 화이트 노이즈이고,

와

는 학습을 통해 계산된다. here,

Is the average of the convolutional results,

Is the standard deviation,

Is white noise,

Wow

Is calculated through learning.

Thereafter, when a deep convolutional network is applied to the batch normalization result, a hidden layer such as [Equation 4] is determined.

[Equation 4]

는 두 행렬에 대한 원소 각각의 곱이 되고, C는 음소 정보 시퀀스이고, L은 음소 위치 시퀀스를 나타낸다. Here, the W vector and the b vector are parameters for learning, and S(.) is a sigmoid function,

Is the product of each element of the two matrices, C is a phoneme information sequence, and L is a phoneme position sequence.

Next, a decoding method performed in the converter will be described.

Transposed convolution for decoding is performed. When a deep-convolutional network is applied to the transposed convolutional result, a hidden layer such as [Equation 5] is determined, and the final result can be expressed as [Equation 6].

[Equation 5]

[Equation 6]

On the other hand, the discriminator performs downsampling while increasing the channel by repeatedly performing the encoding process of the converter (i.e., convolution, batch normalization, deep-convolutional network) on the input data, and performs a softmax function. Thus, we can calculate the probability value.

Training can be performed by inputting synthesized speech data, emotional speech data, phoneme information sequence, and phoneme position sequence to each of the converter and the discriminator.

2C to 2E, the user synthesis model generation unit 120 inputs a user's user voice data 209 and a sentence 211 corresponding to the user voice data 209 into the generated synthesis model 20 A user synthesis model 24 of a user may be generated, and user synthesized speech data 213 may be generated using the generated user synthesis model 24. Here, the user voice data 209 may be voice data in which a user's voice is recorded.

The user synthesis model learning unit 130 may generate user emotion speech data 215 by inputting the generated user synthesis speech data 213 into the generated emotion model 22. Here, the emotion model 22 does not change, and is the same as the emotion model for each emotion previously generated.

The user synthesis model learning unit 130 may input and train the user emotion voice data 215 generated in the user synthesis model 24.

The synthesized emotional voice data generation unit 140 may generate the user synthesized emotional voice data 221 by inputting the request sentence 217 and the emotion information 219 to the learned user synthesis model 24. In this case, the emotion information 219 may include tag information on an emotion selected from a plurality of emotions (eg, joy, sadness, surprise, anger, etc.).

On the other hand, for those skilled in the art, each of the synthetic model generation unit 100, the emotion model generation unit 110, the user synthesis model generation unit 120, the user synthesis model learning unit 130, and the synthetic emotion voice data generation unit 140 It will be fully understood that it may be implemented separately, or one or more of these may be implemented in an integrated manner.

4 is a flowchart illustrating a method for synthesizing emotional speech according to an embodiment of the present invention.

Referring to FIG. 4, in step S401, the emotional speech synthesis apparatus 10 may generate a synthesis model based on original speech data and a sentence corresponding to the original speech data.

In step S403, the emotion-voice synthesis apparatus 10 may generate an emotion model based on the emotion-voice data for each emotion and the synthesized speech data generated by the synthesis model.

In step S405, the emotion-voice synthesis apparatus 10 may generate user-synthesized emotion-voice data by inputting a request sentence and emotion information to the user-synthesis model learned based on the synthesis model and the emotion model.

In the above description, steps S401 to S405 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, or the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

10: emotional speech synthesis device
100: synthetic model generation unit
110: emotion model generation unit
120: User synthetic model generation unit
130: User synthetic model learning unit
140: synthesized emotion voice data generation unit

Claims (15)

In the device for synthesizing emotional voices,
A synthesis model generator that generates a synthesis model based on the original speech data and a sentence corresponding to the original speech data;
An emotion model generator configured to generate an emotion model based on emotion voice data for each emotion and synthesized voice data generated by the synthesis model;
A user synthesis model generation unit for generating a user synthesis model of the user by inputting a user's user voice data and a sentence corresponding to the user voice data into the generated synthesis model;
Generate user synthesized voice data using the generated user synthesis model, input the generated user synthesized voice data to the generated emotion model to generate user emotion voice data, and the generated user in the user synthesis model A user synthesis model learning unit that inputs and trains emotion voice data; And
Synthetic emotion voice data generation unit for generating user synthesized emotion voice data by inputting the requested sentence and emotion information into the learned user synthesis model
That includes, emotional speech synthesis device.
The method of claim 1,
The emotion model generator converts the emotion voice data and the synthesized voice data into a spectrogram image form, and converts the emotion voice data and the synthesized voice data converted into the spectrogram image form into an emotion voice matrix and a synthesized voice matrix. That is, the emotional speech synthesis device.
The method of claim 2,
The emotion model generator divides the emotion speech matrix and the synthesized speech matrix so that samples are included in a preset time unit as many as the number of samples calculated based on a preset sample rate, and the divided emotion speech matrix and the divided To calculate the average value of the synthesized speech matrix for each time unit, the emotional speech synthesis apparatus.
The method of claim 3,
The emotion model generation unit derives each distance matrix by comparing components constituting the emotion speech matrix and the synthesized speech matrix,
A distance value is extracted from a distance matrix of the derived emotional speech matrix and a distance matrix of the derived synthetic speech matrix based on the number of phonemes constituting the emotional speech data and the synthesized speech data, and the extracted distance value is To check the index value, emotional speech synthesis device.
The method of claim 4,
The emotion model generation unit extracts a distance value having an upper value by one less than the number of phonemes from the distance matrix of the derived emotion speech matrix and the distance matrix of the derived synthesized speech matrix. Device.
The method of claim 4,
The emotion model generator derives an emotion speech cluster and a synthesized speech cluster by distributing the emotion speech matrix and the synthesized speech matrix by the number of phonemes according to the identified index value, and based on the derived emotion speech cluster, the To adjust the length of the derived synthesized speech cluster, emotional speech synthesis apparatus.
The method of claim 6,
The emotion model generator derives a synthesized emotional voice sequence based on the synthesized voice cluster, a phoneme information vector, and a phoneme position vector whose length is adjusted,
Based on the derived synthetic emotional voice sequence, the actual emotional voice sequence for the emotional voice data, the phoneme information vector, and the phoneme position vector, it is determined whether or not the derived synthetic emotional voice sequence is a voice bearing emotion, and the determination To learn the emotion model by using the result, the emotional speech synthesis device.
In the method of synthesizing the emotional voice,
Generating a synthesis model based on original speech data and sentences corresponding to the original speech data;
Generating an emotion model based on emotion voice data for each emotion and synthesized voice data generated by the synthesis model; And
And generating user synthesized emotional speech data by inputting a request sentence and emotion information to the synthesis model and the user synthesis model learned based on the emotion model.
The method of claim 8,
Generating the emotion model,
Converting the emotional voice data and the synthesized voice data into a spectrogram image form, and converting the emotional voice data and the synthesized voice data converted into the spectrogram image form into an emotional voice matrix and a synthesized voice matrix
That includes, emotional speech synthesis method.
The method of claim 9,
Generating the emotion model,
The emotional voice matrix and the synthesized voice matrix are divided so that samples are included in a preset time unit by the number of samples calculated based on a preset sample rate, and the divided emotional voice matrix and the divided synthesized voice matrix are selected from the Step of calculating the average value for each time unit
It will further include, emotional speech synthesis method.
The method of claim 10,
Generating the emotion model,
Comparing components constituting the emotion speech matrix and the synthesized speech matrix to derive respective distance matrices; And
A distance value is extracted from a distance matrix of the derived emotional speech matrix and a distance matrix of the derived synthetic speech matrix based on the number of phonemes constituting the emotional speech data and the synthesized speech data, and the extracted distance value is Steps to check the index value
It will further include, emotional speech synthesis method.
The method of claim 11,
Generating the emotion model,
Extracting a distance value having an upper value by one less than the number of phonemes from the derived distance matrix of the emotion speech matrix and the derived distance matrix of the synthesized speech matrix
It will further include, emotional speech synthesis method.
The method of claim 11,
Generating the emotion model,
Distributing the emotion speech matrix and the synthesized speech matrix by the number of phonemes according to the identified index value to derive an emotion speech cluster and a synthesized speech cluster; And
Adjusting the length of the derived synthetic speech cluster based on the derived emotional speech cluster
It will further include, emotional speech synthesis method.
The method of claim 13,
Generating the emotion model,
Deriving a synthesized emotional voice sequence based on the length-adjusted synthesized voice cluster, a phoneme information vector, and a phoneme position vector;
Based on the derived synthetic emotional voice sequence, the actual emotional voice sequence for the emotional voice data, the phoneme information vector, and the phoneme position vector, it is determined whether or not the derived synthetic emotional voice sequence is a voice bearing emotion, and the determination Learning the emotion model using the result
It will further include, emotional speech synthesis method.
In a computer program stored in a medium containing a sequence of instructions for synthesizing an emotional voice,
When the computer program is executed by a computing device,
Generate a synthesis model based on the original voice data and the sentence corresponding to the original voice data,
Generating an emotion model based on emotion voice data for each emotion and synthesized voice data generated by the synthesis model,
Generating a user synthesis model of the user by inputting a user's user voice data and a sentence corresponding to the user voice data into the generated synthesis model,
Generating user synthesized speech data using the generated user synthesis model,
Generating user emotion voice data by inputting the generated user synthesized voice data into the generated emotion model,
Inputting and learning the generated user emotion voice data into the user synthesis model,
A computer program stored in a medium comprising a sequence of instructions for generating user synthesized emotion voice data by inputting a request sentence and emotion information to the learned user synthesis model.

Images