KR102363955B1 - Method and system for evaluating the quality of recordingas - Google Patents

Abstract

Provided is a method for evaluating a quality of a recording. The method of the present invention receives the recording data wherein a speaker performs a recording based on a recording script corresponding to a specific text, generates the first spectrograms and a speaker embedding vector based on the recording data, and allows the second spectrograms corresponding to the recording script to be generated based on the speaker embedding vector and the first spectrograms. In addition, a score of attention alignment corresponding to the second spectrograms may be calculated, and the quality of the recording data may be evaluated based on the score. Therefore, the present invention is capable of providing the method for evaluating whether the speaker performed the recording in accordance with the recording script.

Description

METHOD AND SYSTEM FOR EVALUATING THE QUALITY OF RECORDINGAS

The present disclosure relates to a method and system for evaluating the quality of a recording.

Recently, with the development of artificial intelligence technology, interfaces using voice signals are becoming common. Accordingly, research on a speech synthesis technology capable of uttering a synthesized voice according to a given situation is being actively conducted.

Speech synthesis technology is applied to many fields such as virtual assistants, audiobooks, automatic interpretation and translation, and virtual voice actors by combining speech recognition technology based on artificial intelligence.

As a conventional speech synthesis method, there are various methods such as unit selection synthesis (USS) and statistic-based parameter synthesis (HMM-based speech synthesis, HTS). The USS method cuts out speech data into phoneme units and stores them, and finds a sound piece suitable for utterance during speech synthesis. The HTS method creates a statistical model by extracting parameters corresponding to speech characteristics and converts text into speech based on the statistical model. as a way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing a natural voice reflecting the speaker's speech style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network is attracting attention. On the other hand, there is a need for a technology capable of evaluating whether recorded data for learning of a speech synthesis system based on an artificial neural network corresponds to data recorded in accordance with a recording script.

The goal is to provide an artificial intelligence-based speech synthesis technology that can implement natural speech as if a real speaker is speaking.

An object of the present invention is to provide a technology capable of evaluating whether recorded data for training of a speech synthesis system corresponds to data recorded in accordance with a recording script.

The technical problem to be solved is not limited to the technical problems as described above, and other technical problems may be inferred.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure is a method for evaluating the quality of a recording, wherein the recording data is received by a speaker based on a recording script corresponding to a specific text. to do; generating first spectrograms and a speaker embedding vector based on the recorded data; generating second spectrograms corresponding to a specific text based on the speaker embedding vector and the first spectrograms; calculating a score of attention alignment corresponding to the second spectrograms; and evaluating the quality of the recorded data based on the score.

The generating of the second spectrograms may include: inputting the first spectrograms at each time step of a decoder included in a synthesizer for generating the second spectrograms; and generating the second spectrograms as a result of inferring each phoneme corresponding to the specific text based on the first spectrograms.

In addition, the attention alignment includes a first axis corresponding to time steps of a decoder included in a synthesizer for generating second spectrograms, and a second axis corresponding to time steps of an encoder included in the synthesizer. Represented based on an axis, generating the score may include: deriving a first largest value and a second largest value from among values corresponding to a time step of a particular decoder; and calculating the score using a difference value between a first index value indicating a time step of an encoder corresponding to the first value and a second index value indicating a time step of an encoder corresponding to the second value; may include

In addition, the attention alignment is based on a first axis corresponding to a time step of a decoder included in a synthesizer for generating second spectrograms and a second axis corresponding to a time step of an encoder included in the synthesizer. and generating the score may include: deriving a first maximum value from among values corresponding to a first time step among time steps of the decoder; deriving a second maximum value from among values corresponding to a second time step corresponding to a next step of the first time step; comparing a first index value indicating a time step of an encoder corresponding to the first maximum value and a second index value indicating a time step of an encoder corresponding to the second maximum value; and calculating the score based on a difference between the first index value and the second index value when the first index value is greater than the second index value.

In addition, the evaluating may include: comparing the score with a preset value; and evaluating the quality of the recorded data indicating whether the speaker performed the recording in accordance with the recording script as a result of the comparison.

The generating of the speaker embedding vector may include: generating the first spectrograms by performing short-time Fourier transform (STFT) on the recorded data; and generating the speaker embedding vector by inputting the first spectrograms to a learned artificial neural network model, wherein the learned artificial neural network model receives the first spectrograms and receives the first spectrograms as input and performs the recording data and the recorded data in a vector space. An embedding vector of the most similar speech data may be output as the speaker embedding vector.

The present invention may provide a method for evaluating whether the speaker performed the recording in accordance with the recording script.

1 is a diagram schematically illustrating an operation of a speech synthesis system.
2 is a diagram illustrating an embodiment of a speech synthesis system.
3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system.
4 is a diagram illustrating an embodiment of a vector space for generating an embedding vector in a speaker encoder.
5 is a diagram illustrating an embodiment of evaluating the quality of a recording using a speech synthesis system.
6 is a diagram illustrating an embodiment in which a synthesizer generates second spectrograms based on first spectrograms.
7A and 7B are diagrams for explaining the quality of attention alignment corresponding to second spectrograms.
8 is a diagram for explaining an embodiment in which a synthesizer calculates an encoder score.
9 is a diagram for explaining an embodiment in which a synthesizer calculates a decoder score.
10 is a diagram for explaining an embodiment in which a synthesizer calculates a concentration score.
11 is a diagram for explaining an embodiment in which a synthesizer calculates a step score.
12 is a flowchart illustrating one embodiment of a method for evaluating the quality of a recording.

The terms used in the present embodiments are selected as currently widely used general terms as possible while considering the functions in the present embodiments, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, etc. there is. In addition, in certain cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents throughout the present embodiments, rather than the simple name of the term.

Since the present embodiments may have various changes and may have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosed form, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present embodiments. The terms used herein are used only for description of the embodiments, and are not intended to limit the present embodiments.

Unless otherwise defined, terms used in the present embodiments have the same meanings as commonly understood by those of ordinary skill in the art to which the present embodiments belong. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, have an ideal or excessively formal meaning. should not be interpreted.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [0010] Reference is made to the accompanying drawings, which show by way of illustration specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the present invention. In addition, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention should be taken as encompassing the scope of the claims and all equivalents thereto. In the drawings, like reference numerals refer to the same or similar elements throughout the various aspects.

On the other hand, in the present specification, technical features that are individually described within one drawing may be implemented individually or may be implemented at the same time.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily practice the present invention.

1 is a diagram schematically illustrating an operation of a speech synthesis system.

A speech synthesis system is a system that converts text into human speech.

For example, the speech synthesis system 100 of FIG. 1 may be an artificial neural network-based speech synthesis system. An artificial neural network refers to an overall model that has problem-solving ability by changing the strength of synaptic bonding through learning in which artificial neurons that form a network through synaptic bonding.

The voice synthesis system 100 may be implemented in various types of devices such as a personal computer (PC), a server device, a mobile device, and an embedded device, and as a specific example, a smartphone or tablet that performs voice synthesis using an artificial neural network It may correspond to a device, an Augmented Reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, robotics, a medical device, an e-book terminal, and a navigation device, but is not limited thereto.

Furthermore, the speech synthesis system 100 may correspond to a dedicated hardware accelerator mounted on the above device. Alternatively, the speech synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which is a dedicated module for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1 , the speech synthesis system 100 may receive a text input and specific speaker information. For example, the speech synthesis system 100 may receive “Have a good day!” as shown in FIG. 1 as a text input, and may receive “speaker 1” as a speaker information input.

“Speaker 1” may correspond to a voice signal or a voice sample indicating a preset speech characteristic of speaker 1. For example, the speaker information may be received from an external device through a communication unit included in the speech synthesis system 100 . Alternatively, the speaker information may be input from a user through the user interface of the speech synthesis system 100 and may be selected from among various speaker information pre-stored in the database of the speech synthesis system 100, but is limited thereto. not.

The speech synthesis system 100 may output a speech based on a text input received as an input and specific speaker information. For example, the speech synthesis system 100 may say “Have a good day!” and “Speaker 1” may be received as inputs, and a voice for “Have a good day!” in which the speech characteristics of speaker 1 are reflected may be output. The speech characteristics of the speaker 1 may include at least one of various factors such as the speaker 1's voice, a rhyme, a pitch, and an emotion. That is, the output voice may be a voice as if the speaker 1 naturally pronounces “Have a good day!”.

2 is a diagram illustrating an embodiment of a speech synthesis system. The speech synthesis system 200 of FIG. 2 may be the same as the speech synthesis system 100 of FIG. 1 .

Referring to FIG. 2 , the speech synthesis system 200 may include a speaker encoder 210 , a synthesizer 220 , and a vocoder 230 . Meanwhile, in the speech synthesis system 200 illustrated in FIG. 2 , only components related to an embodiment are illustrated. Accordingly, it is apparent to those skilled in the art that the speech synthesis system 200 may include other general-purpose components in addition to the components shown in FIG. 2 .

The speech synthesis system 200 of FIG. 2 may receive speaker information and text as inputs and output a speech.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as an input and generate a speaker embedding vector. The speaker information may correspond to a speaker's voice signal or a voice sample. The speaker encoder 210 may receive the speaker's voice signal or voice sample, extract the speaker's utterance characteristics, and represent this as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, tone, rhyme, intonation, or emotion. That is, the speaker encoder 210 may represent the discontinuous data values included in the speaker information as a vector composed of continuous numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a BRDNN ( A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models, such as a Bidirectional Recurrent Deep Neural Network).

For example, the synthesizer 220 of the speech synthesis system 200 may receive text and an embedding vector representing the speaker's utterance characteristics as inputs, and may output a spectrogram.

3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system. The synthesizer 300 of FIG. 3 may be the same as the synthesizer 220 of FIG. 2 .

Referring to FIG. 3 , the synthesizer 300 of the speech synthesis system 200 may include a text encoder and a decoder. Meanwhile, it is apparent to those skilled in the art that the synthesizer 300 may further include other general-purpose components in addition to the components shown in FIG. 3 .

The embedding vector representing the speech characteristic of the speaker may be generated from the speaker encoder 210 as described above, and the encoder or decoder of the synthesizer 300 receives the embedding vector representing the speech characteristic of the speaker from the speaker encoder 210. can

For example, the speaker encoder 210 may input the speaker's voice signal or voice sample to the trained artificial neural network model, and output an embedding vector of voice data most similar to the speaker's voice signal or voice sample.

4 is a diagram illustrating an embodiment of a vector space for generating an embedding vector in a speaker encoder.

According to an embodiment, the speaker encoder 210 may generate first spectrograms by performing short-time Fourier transform (STFT) on the speaker's voice signal or voice sample. The speaker encoder 210 may generate a speaker embedding vector by inputting the first spectrograms to the learned artificial neural network model.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time and the y-axis represents frequency, and the value of each time frequency can be expressed as a color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given speech signal.

STFT is a method in which a voice signal is divided into sections of a certain length and a Fourier transform is applied to each section. At this time, since the result of performing STFT on the speech signal is a complex value, it is possible to take an absolute value to the complex value to lose phase information and to generate a spectrogram including only magnitude information.

On the other hand, the Mel spectrogram is a readjustment of the frequency interval of the spectrogram to the Mel scale. The human auditory organ is more sensitive in the low frequency band than in the high frequency band, and the Mel Scale expresses the relationship between the physical frequency and the frequency perceived by humans by reflecting these characteristics. The Mel spectrogram may be generated by applying a filter bank based on the Mel scale to the spectrogram.

The speaker encoder 210 may display spectrograms corresponding to various voice data and embedding vectors corresponding thereto on a vector space. The speaker encoder 210 may input spectrograms generated from the speaker's voice signal or voice sample to the learned artificial neural network model. The speaker encoder 510 may output an embedding vector of speech data most similar to the speaker's speech signal or speech sample in the vector space from the learned artificial neural network model as the speaker embedding vector. That is, the trained artificial neural network model may receive spectrograms and generate an embedding vector matching a specific point in the vector space.

3 , the text encoder of the synthesizer 300 may receive text as input and generate a text embedding vector. The text may include a sequence of characters in a particular natural language. For example, the sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The text encoder may divide the inputted text into a unit of a alphabet, a unit of a letter, or a unit of a phoneme, and may input the separated text into the artificial neural network model. For example, the text encoder may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN.

Alternatively, the text encoder may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

The decoder of the synthesizer 300 may receive a speaker embedding vector and a text embedding vector from the speaker encoder 210 as inputs. Alternatively, the decoder of the synthesizer 300 may receive a speaker embedding vector from the speaker encoder 210 as an input and receive a text embedding vector from the text encoder as an input.

The decoder may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, the decoder may generate a spectrogram of the input text in which the speaker's speech characteristics are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

Meanwhile, although not shown in FIG. 3 , the synthesizer 300 may further include an attention module for generating the attention alignment. The attention module is a module that learns which output of a specific time-step of the decoder is most related to the output of all the time-steps of the encoder. A higher quality spectrogram or Mel spectrogram can be output by using the attention module.

Returning to FIG. 2 again, the vocoder 230 of the speech synthesis system 200 may generate the spectrogram output from the synthesizer 220 as actual speech. The spectrogram output as described above may be a Mel spectrogram.

In an embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual speech signal by using Inverse Short-Time Fourier Transform (ISFT). Since the spectrogram or Mel spectrogram does not include phase information, when generating a voice signal using ISFT, the phase information of the spectrogram or Mel spectrogram is not considered.

In another embodiment, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual speech signal using a Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm for estimating phase information from magnitude information of a spectrogram or Mel spectrogram.

Alternatively, the vocoder 230 may generate the spectrogram output from the synthesizer 220 as an actual voice signal, for example, based on a neural vocoder.

A neural vocoder is an artificial neural network model that generates a voice signal by receiving a spectrogram or a Mel spectrogram as an input. A neural vocoder can learn the relationship between a spectrogram or a Mel spectrogram and a voice signal from a large amount of data, and can generate a high-quality real voice signal through this.

A neural vocoder may correspond to a vocoder based on an artificial neural network model such as, but not limited to, WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN.

For example, the WaveNet vocoder consists of several dilated causal convolution layers and is an autoregressive model that uses sequential features between speech samples. WaveRNN vocoder is an autoregressive model that replaces the dilated causal convolution layer of multiple layers of WaveNet with Gated Recurrent Unit (GRU). The WaveGlow vocoder can learn to obtain a simple distribution such as a Gaussian distribution from the spectrogram dataset (x) using an invertible transform function. After learning, the WaveGlow vocoder can output a voice signal from a Gaussian distribution sample by using the inverse function of the transform function.

On the other hand, it is important to be able to generate a voice for the input text in which the speech characteristics of the speaker are reflected, even when a voice sample of a certain speaker is input to the speech synthesis system 200 . In order to output the embedding vector of the voice data most similar to the speaker's voice sample as the speaker embedding vector even if the unlearned speaker's voice sample is input, the artificial neural network model of the speaker encoder 210 needs to be trained with the voice data of various speakers. there is

For example, the training data for training the artificial neural network model of the speaker encoder 210 may correspond to recording data in which the speaker performs recording based on a recording script corresponding to a specific text.

Accordingly, there is a need to evaluate the quality of the recorded data for training the artificial neural network model of the speaker encoder 210 . For example, the quality of the recorded data may be evaluated in relation to whether the speaker performed the recording in accordance with the recording script. Using the speech synthesis system 200, it may be evaluated whether the speaker performed the recording in accordance with the recording script.

5 is a diagram illustrating an embodiment of evaluating the quality of a recording using a speech synthesis system.

The speech synthesis system 500 of FIG. 5 may be the same as the speech synthesis system 100 of FIG. 1 or the speech synthesis system 200 of FIG. 2 . The speaker encoder 510 of FIG. 5 may be identical to the speaker encoder 210 of FIG. 2 , and the synthesizer 520 of FIG. 5 may be identical to the synthesizer 220 of FIG. 2 or the synthesizer 300 of FIG. 3 .

According to an embodiment, the speech synthesis system 500 may receive recorded data in which the speaker recorded the text based on the recording script corresponding to the specific text. Also, the speech synthesis system 500 may generate the first spectrograms and the speaker embedding vector based on the recorded data. The speech synthesis system 500 generates second spectrograms corresponding to the recording script based on the speaker embedding vector and the first spectrograms, and calculates a score of attention alignment corresponding to the second spectrograms. can do. Finally, the speech synthesis system 500 may evaluate the quality of the recorded data indicating whether the speaker performed the recording in accordance with the recording script based on the score.

Referring to FIG. 5 , the speaker encoder 510 of the speech synthesis system 500 may receive recorded data recorded by the speaker based on a recording script corresponding to a specific text. The speaker encoder 510 may generate first spectrograms by performing STFT on the recorded data. For example, the recording script may correspond to 'Turn on the set-top box and say it again', and the speaker may generate recorded data uttering specific text corresponding to the recording script. At this time, the recorded data may be voice data that correctly uttered 'Turn on the set-top box and say it again' in accordance with the recording script, but it may be the voice data that uttered 'Turn on or off the set-top box or speak again' that does not match the recording script. It may correspond to voice data.

The speaker encoder 510 may receive the first spectrograms to the learned artificial neural network model and output a speaker embedding vector having a numerical value close to that of the voice data most similar to the recorded data.

The synthesizer 520 of the speech synthesis system 500 may receive text corresponding to the recording script. For example, the synthesizer 520 may receive the text 'Turn on the set-top box and say it again'. Also, the synthesizer 520 may receive the first spectrograms and the speaker embedding vector from the speaker encoder 510 . The synthesizer 520 may generate the first spectrograms and second spectrograms corresponding to the text received in the speaker embedding vector. Finally, the synthesizer 520 may generate an attention alignment corresponding to the second spectrograms, calculate a score of the attention alignment, and evaluate whether the speaker performed the recording in accordance with the recording script.

6 is a diagram illustrating an embodiment in which a synthesizer generates second spectrograms based on first spectrograms.

Specifically, FIG. 6 shows an embodiment in which the decoder included in the synthesizer 520 generates second spectrograms based on the first spectrograms.

According to an embodiment, the synthesizer 520 may input the first spectrograms generated by the speaker encoder 510 to each time step of a decoder included in the synthesizer 520 that generates the second spectrograms. there is. The synthesizer 520 may generate second spectrograms as a result of inferring each phoneme corresponding to the recording script based on the first spectrograms.

For example, while the decoder of the synthesizer 520 infers each phoneme corresponding to the input recording script, the first spectrogram corresponding to each time step may be input as the target spectrogram or the correct answer spectrogram. That is, the synthesizer 520 does not input the value predicted by the t-1 th decoder cell as an input to the t th decoder cell, but a teacher-forcing method that inputs a target spectrogram or a correct answer spectrogram for each decoder step. Each phoneme corresponding to the inputted recording script can be inferred.

As such, when the teacher-forcing method is used, even if an erroneous result is predicted in the t-1 th decoder cell, since the target spectrogram or the correct spectrogram exists, accurate prediction may be possible in the t th decoder cell.

7A and 7B are diagrams for explaining the quality of attention alignment corresponding to second spectrograms.

7A and 7B exemplarily illustrate the attention alignment generated by the synthesizer 520 in response to the second spectrograms.

For example, attention alignment can be displayed on two-dimensional coordinates, the horizontal axis of the two-dimensional coordinates is the decoder timesteps included in the synthesizer 520, and the vertical axis is the time steps included in the synthesizer 520. It means the encoder timestep. That is, the two-dimensional coordinates in which the attention alignment is expressed mean which part should be focused on when the synthesizer 520 generates the spectrogram.

The decoder time step means the time invested by the synthesizer 520 to utter each of the phonemes corresponding to the recording script. The decoder time steps are arranged at time intervals corresponding to a single hop size, and the single hop size may correspond to, for example, 1/80 second, but is not limited thereto.

Encoder time steps correspond to phonemes included in the recording script. For example, if the input text is 'Turn on the set-top box and say it again', the encoder time steps are 'ㅅ', 'ㅔ', 't', 'ㅗ', 'b', 'b', 'a' , 'a', 'ㅅ', 'ㅡ', 'ㄹ', 'ㅡ', 'ㄹ', ' ', 'ㅋ', 'ㅕ'… (hereinafter abbreviated) may be configured.

In addition, each of the points constituting the attention alignment is expressed in a specific color. Here, the color may be matched with a specific value corresponding thereto. For example, each of the colors constituting the attention alignment is a value representing a probability distribution, and may be a value between 0 and 1.

For example, if the line indicating the attention alignment is thick and the noise is low, it may be interpreted that the synthesizer 520 has confidently performed inference at every moment when the spectrogram is generated. That is, in the case of the above-described example, it may be determined that the synthesizer 520 has generated a high-quality Mel spectrogram. Accordingly, the quality of the attention alignment (eg, the degree of darkness of the attention alignment, the degree of clarity of the outline of the attention alignment, etc.) may be used as a very important index in estimating the inference quality of the synthesizer 520 .

Referring to FIG. 7A , since the attention alignment 700 has thick lines and low noise, it can be interpreted that the synthesizer 520 infers confidently at every moment when the spectrogram is generated. For example, the attention alignment 700 of FIG. 7A is a recording in which the speaker uttered 'Turn on the set-top box and say it again' in accordance with the recording script based on the recording script of 'Turn on the set-top box and say it again' relatively accurately. When data is input, it may correspond to the attention alignment generated from the recording script 'Turn on the set-top box and say it again'.

On the other hand, referring to FIG. 7B , the line in the middle part of the attention alignment 710 is not clear, and the middle part 720 includes an unclear part, so it is interpreted that the quality of the Mel spectrogram is not very high. can be For example, in the attention alignment 700 of FIG. 7A , the speaker utters 'Turn on or off the set-top box or say it again' inconsistent with the recording script based on the recording script of 'Turn on the set-top box and say it again'. When recording data is input, it may correspond to the attention alignment generated from the recording script 'Turn on the set-top box and say it again'.

In other words, the alignment is well drawn because it is a common text between the recorded data and the input text until 'turn on the set-top box', but after that, the parts that do not match between the recorded data and the input script are aligned well. may not be drawn. This is because, after 'on', the spectrogram corresponding to 'go' should be input to the decoder cell, but as the spectrogram with the pronunciation of 'geo' is input, the synthesizer 520 concentrates on the wrong part.

As described above, when recorded data recorded by the speaker not matching the recording script is input, the quality of the output attention alignment may be poor. The quality of the attention alignment may be evaluated based on the score of the attention alignment. When it is determined that the quality of the attention alignment is not good, it may be determined that the recording was performed inconsistent with the recording script.

For example, in order to evaluate the quality of attention alignment, the synthesizer 520 calculates an encoder score, a decoder score, a concentration score, or a step score of the attention alignment. can

The synthesizer 520 may output any one of an encoder score, a decoder score, a concentration score, and a step score as a final score for evaluating the quality of attention alignment.

Alternatively, the synthesizer 520 may output a value obtained by combining at least one of an encoder score, a decoder score, a concentration score, and a step score as a final score for evaluating the quality of attention alignment.

8 is a diagram for explaining an embodiment in which a synthesizer calculates an encoder score.

Referring to FIG. 8 , values 810 corresponding to decoder time step '50' in attention alignment are displayed. Since the attention alignment is transposed by recording each softmax result value, it is 1 if all values corresponding to a single step constituting the decoder time step are added up. That is, if all the values 810 of FIG. 8 are added up, 1 is obtained.

Meanwhile, referring to the upper a values 820 among the values 810 of FIG. 8 , it is determined which phoneme the synthesizer 520 is focusing on at the point corresponding to '50' of the decoder time step to generate the spectrogram. can judge Accordingly, the synthesizer 520 may check whether the spectrogram appropriately represents the input text (ie, the quality of the spectrogram) by calculating the encoder score for each step constituting the decoder time step.

For example, the synthesizer 520 may calculate the encoder score in the s-th step based on the decoder time step based on Equation 1 below.

는 어텐션 얼라인먼트에서 디코더 타임 스텝을 기준으로 s 번째 스텝의 i번째 상위 값을 나타낸다(s 및 i는 1 이상의 자연수).In Equation 1,

denotes the i-th upper value of the s-th step based on the decoder time step in attention alignment (s and i are natural numbers greater than or equal to 1).

That is, the synthesizer 520 extracts n values from values in the s-th step of the decoder type step (n is a natural number equal to or greater than 2). Here, the n values may mean the top n values in the s-th step.

)를 연산한다. 예를 들어, 합성기(520)는 추출된 n 개의 값들을 더하여 제 s 스코어(

)를 연산할 수 있다.Then, the synthesizer 520 uses the extracted n values to obtain an s-th score (

) is calculated. For example, the synthesizer 520 adds the extracted n values to the s-th score (

) can be calculated.

)는 어텐션 얼라인먼트의 모든 디코더 타임 스텝 각각에 대하여 연산한 인코더 스코어들에 기초하여 하기 수학식 2와 같이 연산될 수 있다. final encoder score (

) may be calculated as in Equation 2 below based on encoder scores calculated for each of all decoder time steps of attention alignment.

은 스펙트로그램의 x축 길이(frame length)에 당하고, s는 디코더 타임 스텝의 인덱스에 해당한다. 수학식 2를 구성하는 다른 변수들은 수학식 1에서 설명한 바와 동일하다.In Equation 2 above,

corresponds to the x-axis frame length of the spectrogram, and s corresponds to the index of the decoder time step. Other variables constituting Equation 2 are the same as described in Equation 1.

9 is a diagram for explaining an embodiment in which a synthesizer calculates a decoder score.

Referring to FIG. 9 , values 910 corresponding to encoder time step '40' in attention alignment are displayed. Also, the top b values 920 among the values 910 are indicated.

As described above with reference to FIG. 8 , the encoder score is calculated with values in each step constituting the decoder time step. On the other hand, the decoder score is computed with values at each step constituting the encoder time step. Encoder scores and decoder scores have different purposes. Specifically, the encoder score is an index for determining whether the attention module has properly determined the phoneme to be focused on every time. On the other hand, the decoder score is an index for determining whether the attention module concentrates well on a specific phoneme constituting the input text without omitting time allocation.

For example, the synthesizer 520 may calculate the decoder score in the s-th step based on the encoder time step based on Equation 3 below.

는 어텐션 얼라인먼트에서 인코더 타임 스텝을 기준으로 s번째 스텝의 i번째 상위 값을 나타낸다(s 및 i는 1 이상의 자연수).In Equation 3 above,

denotes the i-th upper value of the s-th step based on the encoder time step in attention alignment (s and i are natural numbers greater than or equal to 1).

That is, the synthesizer 520 extracts m values from values in the s-th step of the encoder type step (m is a natural number equal to or greater than 2). Here, the m values may mean the m uppermost values in the s-th step.

)를 연산한다. 예를 들어, 합성기(520)는 추출된 m 개의 값들을 더하여 제 s 스코어(

)를 연산할 수 있다.Then, the synthesizer 520 uses the extracted m values to obtain an s-th score in the s-th step (

) is calculated. For example, the synthesizer 520 adds the extracted m values to the s-th score (

) can be calculated.

)는 어텐션 얼라인먼트의 모든 인코더 타임 스텝 각각에 대하여 연산한 디코더 스코어들에 기초하여 하기 수학식 4와 같이 연산될 수 있다. final decoder score (

) may be calculated as in Equation 4 below based on decoder scores calculated for each of all encoder time steps of attention alignment.

은 집합 x를 구성하는 값들 중에서 y번째로 작은 값(즉, 하위 y 번째 값)을 의미하고,

은 인코더 타임 스텝을 의미한다.

은 디코더 스코어의 길이를 의미하여, 하위

번째의 값까지 모두 더한 값이 된다.In Equation 4 above,

is the y-th smallest value (that is, the lower y-th value) among the values constituting the set x,

is the encoder time step.

is the length of the decoder score,

It becomes the sum of all values up to the second value.

10 is a diagram for explaining an embodiment in which a synthesizer calculates a concentration score.

According to an embodiment, the synthesizer 520 may derive a first largest value and a second largest value among values corresponding to the first time step among the time steps of the decoder. The synthesizer 520 may calculate the concentration score by using a difference value between a first index value indicating an encoder time step corresponding to the first value and a second index value indicating an encoder time step corresponding to the second value.

When determining the quality of attention alignment, if the synthesizer 520 erroneously concentrates on a certain phoneme, a difference may occur between the erroneously focused portion and the returned portion when returning to focus on the correct phoneme again. Accordingly, in attention alignment, the greater the difference between the index indicating the encoder time step corresponding to the first largest value among the values corresponding to the specific decoder time step and the index indicating the encoder time step corresponding to the second largest value, the greater the speaker's recording script It is highly probable that the recording was performed inconsistent with the text corresponding to .

For example, the synthesizer 520 may calculate the concentration score at the s-th step based on the decoder time step based on Equation 5 below.

는 디코더 타임 스텝을 기준으로 s 번째 스텝에 해당하는 값들 중 첫번째로 큰 제 1 값에 대응하는 인코더 타임 스텝을 나타내는 제 1 인덱스 값과 두번째로 큰 제 2 값에 대응하는 인코더 타임 스텝을 나타내는 제 2 인덱스 값의 차이값에 해당할 수 있다. 예를 들어, 제 1 인덱스와 제 2 인덱스의 차이가 1이면, 집중 스코어의 값은 0이 된다. 그러나, 제 1 인덱스와 제 2 인덱스의 차이가 2 이상이면 집중 스코어의 값은 음의 값을 가지게 된다. 따라서, 집중 스코어의 값이 클수록 화자가 녹음 대본에 해당하는 텍스트와 일치하게 녹음을 수행하였다고 볼 수 있다.In Equation 5, s corresponds to the index of the decoder time step,

is a first index value indicating an encoder time step corresponding to a first largest value among values corresponding to an s-th step based on a decoder time step, and a second value indicating an encoder time step corresponding to a second largest value of the decoder time step. It may correspond to the difference value of the index value. For example, if the difference between the first index and the second index is 1, the value of the concentration score becomes 0. However, if the difference between the first index and the second index is 2 or more, the value of the concentration score has a negative value. Therefore, as the concentration score increases, it can be seen that the speaker performed the recording in accordance with the text corresponding to the recording script.

For example, referring to FIG. 10 , values 1010 corresponding to '50' of the decoder time step are indicated. The index of the encoder time step corresponding to the first largest value among the values 1010 corresponding to '50' of the decoder time step is 4, and the index of the encoder time step corresponding to the second largest value is 5. Therefore, the concentration score at '50' of the decoder time step is zero. On the other hand, the index of the encoder time step corresponding to the first largest value among the values 1020 corresponding to '110' of the decoder time step is 0, and the index of the encoder time step corresponding to the second largest value is 6. Therefore, the concentration score at '110' of the decoder time step is -25. It can be seen that, unlike in '50' of the decoder time step, the attention alignment at '110' of the decoder time step includes an unclear part.

)는 어텐션 얼라인먼트의 모든 디코더 타임 스텝 각각에 대하여 연산한 집중 스코어들에 기초하여 하기 수학식 6과 같이 연산될 수 있다. Final Concentration Score (

) may be calculated as in Equation 6 below based on concentration scores calculated for each of all decoder time steps of attention alignment.

은 스펙트로그램의 x축 길이(frame length)에 당하고, 수학식 6을 구성하는 다른 변수들은 수학식 5에서 설명한 바와 동일하다.In Equation 6 above,

corresponds to the x-axis frame length of the spectrogram, and other variables constituting Equation 6 are the same as described in Equation 5.

11 is a diagram for explaining an embodiment in which a synthesizer calculates a step score.

According to an embodiment, the synthesizer 520 derives a first maximum value from among values corresponding to a first time step among decoder time steps, and a second time step corresponding to a next step of the first time step. A second maximum value may be derived from among the values. The synthesizer 520 may compare the first index value indicating the encoder time step corresponding to the first maximum value and the second index value indicating the encoder time step corresponding to the second maximum value. When the first index value is greater than the second index value, the synthesizer 520 may calculate a step score based on a difference between the first index value and the second index value.

For example, even if the synthesizer 520 mistake a specific spectrogram for a phoneme other than the correct answer, since the correct answer spectrogram is input by the teacher-forcing method, the synthesizer 520 can focus on the correct phoneme again. In this case, the index value indicating the encoder time step corresponding to the maximum value among the values corresponding to the specific decoder time step in the attention alignment is the encoder time step corresponding to the maximum value among the values corresponding to the next time step of the specific decoder time step. It may show a reverse run that becomes larger than the index value representing .

Accordingly, in attention alignment, an index indicating an encoder time step corresponding to a maximum value among values corresponding to a specific decoder time step and an index indicating an encoder time step corresponding to a maximum value among values corresponding to a next step of a specific decoder time step. The greater the difference between the two, the greater the possibility that the speaker performed the recording inconsistent with the text corresponding to the recording script.

For example, the synthesizer 520 may calculate the step score in the s-th step based on the decoder time step based on Equation 7 below.

는 디코더 타임 스텝을 기준으로 s-1 번째 스텝에 해당하는 값들 중 최대값에 대응하는 인코더 타임 스텝을 나타내는 제 1 인덱스 값과 s 번째 스텝에 해당하는 값들 중 최대값에 대응하는 인코더 타임 스텝을 나타내는 제 2 인덱스 값을 비교한 결과, 제 1 인덱스 값이 제 2 인덱스 값보다 큰 경우 제 1 인덱스 갑과 제 2 인덱스 값의 차이값에 해당할 수 있다. 제 1 인덱스 값이 제 2 인덱스 값 이하인 경우에는,

는 0에 해당할 수 있다. 따라서, 스텝 스코어의 값이 클수록 화자가 녹음 대본과 일치하게 녹음을 수행하였다고 볼 수 있다.In Equation 7, s corresponds to the index of the decoder time step,

is a first index value indicating the encoder time step corresponding to the maximum value among the values corresponding to the s-1 th step based on the decoder time step and the encoder time step corresponding to the maximum value among the values corresponding to the s th step As a result of comparing the second index values, when the first index value is greater than the second index value, it may correspond to a difference value between the first index value and the second index value. When the first index value is less than or equal to the second index value,

may correspond to 0. Therefore, it can be said that the speaker performed the recording in accordance with the recording script as the value of the step score was larger.

For example, referring to FIG. 11 , indices 1110 indicating an encoder time step corresponding to a maximum value among values corresponding to respective decoder time steps in attention alignment are indicated. As the index of the decoder time step increases, the value of the index of the encoder time step corresponding to the maximum value also generally increases. However, the reverse running section 1120 in which the index value indicating the encoder time step corresponding to the maximum value among the values corresponding to the specific decoder time step becomes greater than the index value of the encoder time step corresponding to the maximum value among the values corresponding to the next time step. It can be seen that this exists. In a section other than the reverse running section 1120 , the value of the step score is 0 , but in the reverse running section 1120 , the step score has a negative value.

)는 어텐션 얼라인먼트의 모든 디코더 타임 스텝 각각에 대하여 연산한 스텝 스코어들에 기초하여 하기 수학식 8과 같이 연산될 수 있다. final step score (

) may be calculated as in Equation 8 below based on the step scores calculated for each of all decoder time steps of attention alignment.

은 스펙트로그램의 x축 길이(frame length)에 당하고, 수학식 6을 구성하는 다른 변수들은 수학식 5에서 설명한 바와 동일하다.In Equation 8 above,

corresponds to the x-axis frame length of the spectrogram, and other variables constituting Equation 6 are the same as described in Equation 5.

In summary, the synthesizer 520 may output any one of the encoder score, decoder score, concentration score, and step score described above with reference to FIGS. 8 to 11 as a final score for evaluating the quality of attention alignment. The synthesizer 520 compares the final score with a preset value (threshold value), and when the final score is less than the threshold value, the synthesizer 520 may evaluate that the speaker performed the recording inconsistent with the text corresponding to the recording script.

는 하기 수학식 9와 같이 연산될 수 있다. Alternatively, the synthesizer 520 may output a value obtained by combining at least one of the encoder score, decoder score, concentration score, and step score described above in FIGS. 8 to 11 as a final score for evaluating the quality of attention alignment. there is. For example, the final score for evaluating the quality of attention alignment is

can be calculated as in Equation 9 below.

)는 상술한 수학식 2에 따라 연산될 수 있고, 디코더 스코어(

)는 상술한 수학식 4에 따라 연산될 수 있다. 또한, 집중 스코어(

)는 상술한 수학식 6에 따라 연산될 수 있고, 스텝 스코어(

)는 상술한 수학식 8에 따라 연산될 수 있다. 또한,

,

,

및

은 각각 임의의 양의 실수에 해당할 수 있다. In Equation 9, the encoder score (

) can be calculated according to Equation 2 described above, and the decoder score (

) can be calculated according to Equation 4 above. Also, the concentration score (

) can be calculated according to Equation 6 described above, and the step score (

) can be calculated according to Equation 8 described above. also,

,

,

and

may each correspond to any positive real number.

Similarly, the synthesizer 520 may compare the final score with a preset value (threshold value), and when the final score is less than the threshold value, it may be evaluated that the speaker performed the recording inconsistent with the recording script.

12 is a flowchart illustrating one embodiment of a method for evaluating the quality of a recording.

Referring to FIG. 12 , in operation 1210 , the synthesizer may receive recorded data recorded by a speaker based on a recording script corresponding to a specific text.

In operation 1220, the synthesizer may generate first spectrograms and a speaker embedding vector based on the recorded data.

According to an embodiment, the synthesizer inputs the first spectrograms to each time step of a decoder included in the synthesizer for generating second spectrograms, and each corresponding to the recording script based on the first spectrograms As a result of inferring the phonemes, second spectrograms may be generated.

In operation 1230, the synthesizer may generate second spectrograms corresponding to the specific text based on the speaker embedding vector and the first spectrograms.

In step 1240, the synthesizer may calculate a score of attention alignment corresponding to the second spectrograms.

According to an embodiment, the attention alignment corresponds to a first axis corresponding to time steps of a decoder included in the synthesizer for generating the second spectrograms and corresponding to time steps of an encoder included in the synthesizer. It may be expressed based on the second axis.

Further, according to an embodiment, the synthesizing unit derives a first largest value and a second largest second value among values corresponding to the first time step among the time steps of the decoder, and the encoder time step corresponding to the first value A score may be calculated using a difference value between a first index value indicating , and a second index value indicating an encoder time step corresponding to the second value.

Also, according to an embodiment, the synthesizing unit derives a first maximum value from among the values corresponding to the first time step among the decoder time steps, and from among the values corresponding to the second time step corresponding to the next step of the first time step, A second maximum can be derived. The synthesizing unit compares a first index value indicating an encoder time step corresponding to the first maximum value and a second index value indicating an encoder time step corresponding to the second maximum value, wherein the first index value is greater than the second index value In this case, a score may be calculated based on a difference value between the first index value and the second index value.

In operation 1250, the synthesizer may evaluate the quality of the recorded data based on the score.

According to an embodiment, the synthesizer may compare the score with a preset value, and evaluate the quality of the recorded data indicating whether the speaker performed the recording in accordance with the recording script as a result of the comparison.

Various embodiments of the present disclosure may be implemented as software (eg, a program) including one or more instructions stored in a storage medium readable by a machine. For example, the processor of the device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the device to be operated to perform at least one function according to at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

In this specification, “unit” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present specification described above is for illustration, and those of ordinary skill in the art to which the content of this specification belongs will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be able Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present embodiment is indicated by the claims to be described later rather than the detailed description, and it should be construed as including all changes or modifications derived from the meaning and scope of the claims and their equivalents.

Claims (5)

A method for evaluating the quality of a recording, comprising:
Receiving recording data in which a speaker performs recording based on a recording script corresponding to a specific text;
generating first spectrograms and a speaker embedding vector based on the recorded data;
generating second spectrograms corresponding to the recording script based on the speaker embedding vector and the first spectrograms;
calculating a score of attention alignment corresponding to the second spectrograms; and
Including; evaluating the quality of the recorded data based on the score;
The attention alignment includes a first axis corresponding to time steps of a decoder included in the synthesizer generating the second spectrograms and a second axis corresponding to time steps of an encoder included in the synthesizer. is expressed based on
The step of calculating the score is
deriving a first largest value and a second largest value among values corresponding to a first one of the time steps of the decoder; and
calculating the score using a difference value between a first index value indicating a time step of an encoder corresponding to the first value and a second index value indicating a time step of an encoder corresponding to the second value;
A method comprising The method of claim 1,
The generating of the second spectrograms comprises:
inputting the first spectrograms at each time step of a decoder included in a synthesizer for generating second spectrograms; and
generating the second spectrograms as a result of inferring each phoneme corresponding to the recording script based on the first spectrograms;
How to include. delete delete The method of claim 1,
The evaluation step is
comparing the score with a preset value; and
evaluating the quality of the recorded data indicating whether the speaker performed the recording in accordance with the recording script as a result of the comparison;
A method comprising

Images