KR20220170330A - Electronic device and method for controlling thereof - Google Patents

Abstract

Disclosed are an electronic device and a method for controlling the same. A method for controlling an electronic device, according to the present disclosure, comprises the steps of: obtaining text; inputting the text in a first neural network model so as to obtain acoustic feature information corresponding to the text and alignment information formed by matching each frame of the acoustic feature information with each phoneme included in the text; identifying an utterance speed of the acoustic feature information on the basis of the obtained alignment information; identifying a reference utterance speed for each phoneme included in the acoustic feature information on the basis of the text and acoustic feature information; obtaining utterance speed adjustment information on the basis of the utterance speed of the acoustic feature information and the reference utterance speech; and inputting the acoustic feature information in a second neural network model set on the basis of the obtained utterance speed adjustment information so as to obtain voice data corresponding to the text. According to the present invention, it is possible to obtain voice data with an improved utterance speed.

Description

Electronic device and its control method {ELECTRONIC DEVICE AND METHOD FOR CONTROLLING THEREOF}

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method for performing voice synthesis using an artificial intelligence model.

Thanks to the development of electronic technology, various types of devices are being developed and spread, and in particular, devices that perform voice synthesis are becoming common.

Speech synthesis is a technology for realizing a human voice from text, and is also called text to speech (TTS). Recently, neural TTS using a neural network model has been developed.

Neural TTS may include, for example, a prosody neural network model and a neural vocoder neural network model. The prosody neural network model may receive text and output acoustic feature information, and the neural vocoder neural network model may receive acoustic feature information and output voice data (waveform).

Meanwhile, in the TTS model, the prosody neural network model has the speaker's voice characteristics used for learning. That is, the output of the prosody neural network model may be sound characteristic information including voice characteristics of a specific speaker and speech speed characteristics of a specific speaker.

Conventionally, with the development of artificial intelligence models, personalized TTS models that output voice data having voice characteristics of electronic device users have been proposed. The personalized TTS model is a TTS model that is learned based on speech data of an individual user and outputs speech data including the user's voice characteristics and speech speed characteristics used for learning.

On the other hand, since the sound quality of the individual user's speech data used for learning the personalized TTS model is generally lower than the sound quality of the data used for learning the general TTS model, the speech speed of the speech data output from the personalized TTS model Problems may arise.

The present disclosure has been made to solve the above problems, and an object of the present disclosure is to provide an adaptive speech rate control method for a TTS model.

According to an embodiment of the present disclosure, a control method of an electronic device may include obtaining text; By inputting the text to the first neural network model, acoustic feature information corresponding to the text and alignment information matching each frame of the acoustic feature information with each phoneme included in the text obtaining; identifying an utterance rate of the acoustic characteristic information based on the acquired alignment information; identifying a reference speech speed for each phoneme included in the sound characteristic information, based on the text and the sound characteristic information; obtaining speech speed control information based on the speech speed of the sound characteristic information and the reference speech speed; and obtaining voice data corresponding to the text by inputting the acoustic characteristic information to a second neural network model established based on the acquired speech rate control information.

The identifying speech speed of the acoustic characteristic information may include identifying a speech speed corresponding to a first phoneme included in the acoustic characteristic information, based on the acquired alignment information, and the reference speech speed. The identifying may include: identifying the first phoneme included in the acoustic characteristic information based on the acoustic characteristic information; and identifying a reference speech rate corresponding to the first phoneme based on the text.

The identifying of the reference speech rate may include obtaining a first reference speech rate corresponding to the first phoneme based on the text and sample data used for learning of the first neural network. .

The identifying of the reference speech rate may include obtaining evaluation information on sample data used for learning of the first neural network model; and identifying a second reference speech rate corresponding to the first phoneme based on the first reference speech rate and the evaluation information, wherein the evaluation information is obtained by a user of the electronic device that can be characterized.

The method may further include identifying a reference speech speed corresponding to the first phoneme based on one of the first reference speech speed and the second reference speech speed.

The identifying of the speech speed corresponding to the first phoneme may include the speech speed corresponding to the first phoneme and the speech speed corresponding to each of at least one phoneme prior to the first phoneme in the acoustic characteristic information. and identifying an average speech rate corresponding to the first phoneme with , wherein the acquiring the speech rate control information comprises: the average speech rate corresponding to the first phoneme and the average speech rate corresponding to the first phoneme. Acquiring speech rate control information corresponding to the first phoneme based on the reference speech rate;

The second neural network model includes an encoder receiving the acoustic characteristic information and a decoder receiving vector information output from the encoder, and the obtaining of the voice data includes the first phoneme among the acoustic characteristic information. identifying the number of loops of a decoder included in a second neural network model based on speech rate control information corresponding to the first phoneme while at least one frame corresponding to is input to the second neural network model; ; and acquiring a number of first voice data corresponding to the at least one frame and the identified number of loops, based on the at least one frame corresponding to the first phoneme being input to the second neural network model. It may further include, and the first voice data may be voice data corresponding to the first phoneme.

And, when one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, second voice data corresponding to the number of loops may be obtained. there is.

And, the decoder is characterized in that the shift size (Shift size) acquires the voice data of the first frequency (khz) based on the sound characteristic information of the first time interval (sec), and the value of the speech speed control information When is the reference value, one frame included in the acoustic characteristic information is input to the second neural network model, and a number of voice data corresponding to the product of the first time interval and the first frequency is obtained. can do.

The speech rate control information may be information on a ratio value between the speech rate of the acoustic characteristic information and the reference speech rate.

Meanwhile, according to an embodiment of the present disclosure, an electronic device includes a memory for storing at least one instruction; and a processor configured to control the electronic device by executing at least one instruction stored in the memory, wherein the processor obtains text and inputs the text to a first neural network model, thereby generating sound corresponding to the text. Acoustic feature information and alignment information matching each frame of the acoustic feature information and each phoneme included in the text are obtained, and based on the obtained alignment information, the acoustic feature information Identify the speech rate of, identify the reference speech speed for each phoneme included in the sound characteristic information based on the text and the sound characteristic information, and based on the speech speed and the reference speech speed of the sound characteristic information, Speech rate control information is obtained, and voice data corresponding to the text is obtained by inputting the acoustic characteristic information to a second neural network model established based on the obtained speech rate control information.

Further, the processor identifies a speech speed corresponding to a first phoneme included in the acoustic characteristic information based on the acquired alignment information, and based on the acoustic characteristic information, the speech speed included in the acoustic characteristic information. A first phoneme may be identified, and a reference speech speed corresponding to the first phoneme may be identified based on the text.

The processor may obtain a first reference speech rate corresponding to the first phoneme based on the text and sample data used for learning of the first neural network.

The processor obtains evaluation information on sample data used to learn the first neural network model, and based on the first reference speech rate and the evaluation information, a second criterion corresponding to the first phoneme. Speech speed may be identified, and the evaluation information may be obtained by a user of the electronic device.

The processor may identify a reference speech speed corresponding to the first phoneme based on one of the first reference speech speed and the second reference speech speed.

The processor may perform an average speech speed corresponding to the first phoneme based on a speech speed corresponding to the first phoneme and a speech speed corresponding to each of at least one phoneme prior to the first phoneme in the acoustic characteristic information. may be identified, and speech rate control information corresponding to the first phoneme may be obtained based on an average speech rate corresponding to the first phoneme and a reference speech rate corresponding to the first phoneme.

The second neural network model includes an encoder receiving the acoustic characteristic information and a decoder receiving vector information output from the encoder, and the processor includes at least one of the acoustic characteristic information corresponding to the first phoneme. While the frame of is input to the second neural network model, the number of loops of the decoder included in the second neural network model is identified based on the speech rate control information corresponding to the first phoneme, and the second neural network model On the basis that at least one frame corresponding to the first phoneme is input to the first phoneme, a number of first voice data corresponding to the at least one frame and the identified number of loops is obtained, and the first voice data is It may be characterized in that it is voice data corresponding to the first phoneme.

And, when one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, second voice data corresponding to the number of loops may be obtained. there is.

And, the decoder is characterized in that the shift size (Shift size) acquires the voice data of the first frequency (khz) based on the sound characteristic information of the first time interval (sec), and the value of the speech speed control information When is the reference value, one frame included in the acoustic characteristic information is input to the second neural network model, and a number of voice data corresponding to the product of the first time interval and the first frequency is obtained. can do.

The speech rate control information may be information on a ratio value between the speech rate of the acoustic characteristic information and the reference speech rate.

According to various embodiments as described above, the electronic device can adjust the speech speed for each phoneme corresponding to the acoustic characteristic information input to the neural vocoder neural network model of the TTS model, thereby obtaining voice data with improved speech speed can do.

1 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.
2 is a block diagram for explaining the configuration of a TTS model according to an embodiment of the present disclosure.
3 is a block diagram for explaining the configuration of a second neural network model (eg, a neural vocoder neural network model) in a TTS model according to an embodiment of the present disclosure.
4 is a diagram for explaining a method of obtaining voice data with improved speech speed according to an embodiment of the present disclosure.
5 is a diagram for explaining alignment information obtained by matching each frame of sound characteristic information with each phoneme included in text, according to an embodiment of the present disclosure.
6 is a diagram for explaining a method of identifying a reference speech rate for each phoneme included in acoustic characteristic information according to a first embodiment of the present disclosure.
7 is a diagram for explaining a method of identifying a reference speech rate for each phoneme included in acoustic characteristic information according to a second embodiment of the present disclosure.
8 is a diagram for explaining a method for identifying a reference speech rate according to an embodiment of the present disclosure.
9 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.
10 is a block diagram for explaining a configuration of an electronic device according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a block diagram for explaining the configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1 , an electronic device 100 may include a memory 110 and a processor 120 . According to the present disclosure, the electronic device 100 includes smart phones, AR glasses, tablet PCs, mobile phones, video phones, e-book readers, TVs, desktop PCs, laptop PCs, netbook computers, workstations, cameras, smart watches, and servers. It may be implemented in various types of electronic devices such as the like.

The memory 110 may store at least one instruction or data related to at least one other component of the electronic device 100 . In particular, the memory 110 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard-disk drive (HDD), or a solid-state drive (SDD). . The memory 110 is accessed by the processor 120, and data can be read/written/modified/deleted/updated by the processor 120.

In the present disclosure, the term memory refers to the memory 110, a ROM (not shown) in the processor 120, a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, micro SD). card, memory stick).

As described above, the memory 110 may store at least one instruction. Here, the instruction may be for controlling the electronic device 100 . For example, an instruction related to a function for changing an operation mode according to a user's conversation situation may be stored in the memory 110 . Specifically, the memory 110 may include a plurality of components (or modules) for changing an operation mode according to a user's conversation situation according to the present disclosure, which will be described later.

The memory 110 may store data that is information in units of bits or bytes capable of representing characters, numbers, images, and the like. For example, the first neural network model 10 and the second neural network model 20 may be stored in the memory 110 . Here, the first neural network model may be a prosody neural network model, and the second neural network model may be a neural vocoder neural network model.

The processor 120 may be electrically connected to the memory 110 to control overall operations and functions of the electronic device 100 . The processor 120 may be electrically connected to the memory 110 to control overall operations and functions of the electronic device 100 .

According to an embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, it is not limited thereto, and the central processing unit ( central processing unit (CPU)), micro controller unit (MCU), micro processing unit (MPU), controller, application processor (AP), or communication processor (CP), ARM processor In addition, the processor 132 may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or an FPGA ( It may be implemented in the form of a field programmable gate array).

One or more processors control input data to be processed according to predefined operating rules or artificial intelligence models stored in the memory 110 . A predefined action rule or an artificial intelligence model is characterized in that it is created through learning. Here, being created through learning means that a predefined operation rule or an artificial intelligence model having desired characteristics is created by applying a learning algorithm to a plurality of learning data. Such learning may be performed in the device itself in which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.

An artificial intelligence model may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and the layer operation is performed through the operation result of the previous layer and the plurality of weight values. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), and Deep Neural Network (BRDNN). There are Q-networks (Deep Q-Networks), and the neural network in the present disclosure is not limited to the above-described examples except for explicitly stated cases.

The processor 120 may control hardware or software components connected to the processor 120 by driving an operating system or an application program, and may perform various data processing and operations. Also, the processor 120 may load and process commands or data received from at least one of the other components into a volatile memory, and store various data in a non-volatile memory.

In particular, the processor 120 may provide an adaptive speech rate control function when synthesizing voice data. As shown in FIG. 1 , the adaptive speech rate control function according to the present disclosure includes a text acquisition module 121, a sound characteristic information acquisition module 122, a speech rate acquisition module 123, and a reference speech rate acquisition module 124. ), a speech rate control information acquisition module 125 and a voice data acquisition module 126, and each module may be stored in the memory 110. For example, the adaptive speech speed control function may adjust the speech speed by adjusting the number of loops of the second neural network model 20 included in the text to speech (TTS) model 200 shown in FIG. 2 .

2 is a block diagram for explaining the configuration of a TTS model according to an embodiment of the present disclosure.

The TTS model 200 shown in FIG. 2 may include a first neural network model 10 and a second neural network model 20 .

The first neural network model 10 may be configured to receive text 210 and output sound characteristic information 220 corresponding to the text 210 . For example, the first neural network model 10 may be implemented as a prosody neural network model.

The prosody neural network model may be a neural network model obtained by learning relationships between a plurality of sample texts and a plurality of sample sound characteristic information corresponding to each of the plurality of sample texts. Specifically, the prosody neural network model learns the relationship between one sample text and sample sound characteristic information obtained from sample voice data corresponding to the one sample text, and performs this process on a plurality of sample texts, thereby prosody Learning may be performed on the neural network model. In addition, the prosody neural network model may include, for example, a language processing unit for the purpose of performance improvement, and the language processing unit may include a text normalization module, a grapheme-to-phoneme (G2P) module, and the like. there is. Meanwhile, the acoustic characteristic information 220 output from the first neural network model 10 may include characteristics of a speaker's voice used for learning of the first neural network model 10 . That is, the acoustic characteristic information 220 output from the first neural network model 10 may have voice characteristics of a specific speaker (speaker corresponding to data used for learning of the first neural network model).

The second neural network model 20 is a neural network model for converting the acoustic characteristic information 220 into voice data 230, and may be implemented as, for example, a neural vocoder neural network model. According to the present disclosure, the neural vocoder neural network model may receive the acoustic characteristic information 220 output from the first neural network model 10 and output voice data 230 corresponding to the acoustic characteristic information 220. Specifically, the second neural network model 20 may be a neural network model obtained by learning a relationship between a plurality of sample acoustic characteristic information and sample voice data corresponding to each of the plurality of sample acoustic characteristic information.

Then, the second neural network model 20 receives the encoder 20-1 receiving the acoustic characteristic information 220 and the vector information output from the encoder 20-1 as shown in FIG. 3 to generate voice data 230. It may include a decoder 20 - 2 for outputting, and the second neural network model 20 will be described later with reference to FIG. 3 .

Referring back to FIG. 1 , a plurality of modules 121 to 126 may be loaded into a memory (eg, a volatile memory) included in the processor 120 in order to perform an adaptive ignition rate control function. That is, in order to perform the adaptive firing speed control function, the processor 120 may load the plurality of modules 121 to 126 from non-volatile memory to volatile memory to execute respective functions of the plurality of modules 121 to 126. there is. Loading refers to an operation of loading and storing data stored in a non-volatile memory into a volatile memory so that the processor 120 can access the data.

As an example according to the present disclosure, as shown in FIG. 1 , an adaptive speech speed control function may be implemented through a plurality of modules 121 to 126 stored in the memory 110, but is not limited thereto and adaptive speech A speed control function may be implemented through an external device connected to the electronic device 100 .

Each of the plurality of modules 121 to 126 according to the present disclosure may be implemented as software, but is not limited thereto and some modules may be implemented as a combination of hardware and software. As another example, the plurality of modules 121 to 126 may be implemented as one software. Also, some modules may be implemented within the electronic device 100 and other modules may be implemented in an external device.

The text acquisition module 121 is a module for obtaining text to be converted into voice data. For example, the text acquired by the text acquisition module 121 may be text corresponding to a response to a user's voice command. For example, the text may be text being displayed on the display of the electronic device 100 . For example, the text may be text input by a user of the electronic device 100 . As an example, the text may be text provided by a voice recognition system (eg, Bixby). For example, the text may be text received from an external server. That is, according to the present disclosure, text may be various texts to be converted into voice data.

The acoustic feature information acquisition module 122 is a component for acquiring acoustic feature information corresponding to the text acquired by the text acquisition module 121 .

The sound characteristic information acquisition module 122 may input the text obtained by the text acquisition module 121 to the first neural network model 10 and output sound characteristic information corresponding to the input text.

Acoustic characteristic information according to the present disclosure may be information including information about voice characteristics (eg, pitch information, prosody information, and speech speed information) of a specific speaker. Since such sound characteristic information is input to the second neural network model 20 to be described later, voice data corresponding to text may be output.

Here, the acoustic characteristic information means a static characteristic within a short interval (frame) of voice data, and after short-time analysis of the voice data, the acoustic characteristic information can be obtained for each interval. The frame of the sound characteristic information may be set to 10 to 20 msec, but may be set to any other time interval. Examples of sound characteristic information include spectrum, mel-spectrum, cepstrum, pitch lag, pitch correlation, and the like, and one or a combination thereof may be used.

For example, the sound characteristic information may be set in a manner such as a 257th order spectrum, an 80th order Mel-spectrum, or a Cepstrum (20th order) + pitch lag (1st order) + pitch correlation (1st order). More specifically, for example, when the shift size is 10 msec and the 80th order Mel-spenctrum is used as acoustic characteristic information, [100, 80] dimension acoustic characteristic information can be obtained from 1 second of voice data, where [ T, D] includes the following meanings.

[T, D]: T frames, D-dimensional sound characteristic information

In addition, the sound characteristic information acquisition module 122 acquires alignment information obtained by matching each frame of the sound characteristic information output from the first neural network model 10 with each phoneme included in the input text. can Specifically, the acoustic characteristic information acquisition module 122 acquires acoustic characteristic information corresponding to the text by inputting text into the first neural network model 10, and furthermore, each frame of the acoustic characteristic information and the first neural network model ( Alignment information matching each phoneme included in the text input in 10) may be obtained.

According to the present disclosure, alignment information may be matrix information for alignment between input/output sequences in a sequence-to-sequence model. Specifically, through alignment information, it is possible to know information about which input each time-step of the output sequence was predicted from. In addition, according to the present disclosure, the alignment information obtained from the first neural network model 10 includes 'phonemes' corresponding to text input to the first neural network model 10 and 'sound sounds' output from the first neural network model 10. It may be alignment information that matches the 'frame' of the characteristic information, and the alignment information will be described later with reference to FIG. 5 .

The speech rate acquisition module 123 is a component for identifying the speech speed of the acoustic characteristic information acquired by the acoustic characteristic information acquisition module 122 based on the alignment information acquired by the acoustic characteristic information acquisition module 122 .

The speech rate acquiring module 123 calculates the speech speed corresponding to each phoneme included in the acoustic characteristic information acquired by the acoustic characteristic information acquisition module 122 based on the alignment information acquired by the acoustic characteristic information obtaining module 122. can be identified.

Specifically, the speech rate acquiring module 123 uses the acoustic characteristic information acquisition module 122 to obtain utterances for each phoneme included in the acoustic characteristic information acquired by the acoustic characteristic information acquisition module 122 based on the alignment information acquired by the acoustic characteristic information acquisition module 122. speed can be discerned. According to the present disclosure, the alignment information is alignment information obtained by matching 'phonemes' corresponding to text input to the first neural network model 10 with 'frames of sound characteristic information' output from the first neural network model 10. It can be seen that the first phoneme is uttered slowly as the number of frames of the acoustic characteristic information corresponding to the first phoneme among the phonemes included in the alignment information increases. For example, if three frames of sound characteristic information corresponding to the first phoneme are identified based on the alignment information and five frames of sound characteristic information corresponding to the second phoneme are identified, the first phoneme is uttered. It can be seen that the speed is relatively faster than the speech speed of the second phoneme.

Then, when the speech rate for each phoneme included in the text is obtained, the speech rate acquisition module 123 considers the speech rate corresponding to the specific phoneme included in the text and at least one phoneme prior to the corresponding phoneme, and determines the specific phoneme. The average firing rate of can be obtained. For example, the speech rate acquisition module 123 may perform an average speech corresponding to the first phoneme based on a speech speed corresponding to the first phoneme included in the text and a speech speed corresponding to each of at least one phoneme prior to the first phoneme. speed can be discerned.

However, since the speech speed for one phoneme is for a short section, when the speech speed is predicted for an extremely short section, the difference in length between phonemes is reduced, resulting in unnatural results. In addition, when the speech rate is predicted for a very short period, an unnatural result may occur because the prediction value of the speech rate changes too quickly on the time axis. Accordingly, according to the present disclosure, an average speech speed corresponding to a corresponding phoneme in consideration of speech speeds of previous phonemes may be identified, and the identified average speech speed may be used as the speech speed of the corresponding phoneme.

However, when estimating the average speech speed for an excessively long section when uttering speed is predicted, it may be difficult to reflect when there are both slow and fast utterances in the text. In addition, in the streaming structure, since the identified speech speed is a speed prediction for speech that has already been output, a delay in speech speed control may occur. Therefore, a method for measuring the average speech speed for an appropriate section is required.

The average firing speed may be identified by a simple moving average method or an exponential moving average (EMA) method according to an embodiment, and details thereof will be described later with reference to FIGS. 6 and 7 .

The reference speech rate acquisition module 124 is a component for identifying a reference speech rate for each phoneme included in the sound characteristic information. According to the present disclosure, the reference speech speed may mean an optimal speech speed that is felt as an appropriate speed for each phoneme included in the acoustic characteristic information.

As a first embodiment, based on sample data (eg, sample text and sample voice data) used for learning of the first neural network model 10, the reference speech rate acquisition module 124 may perform a first step included in sound characteristic information. A first reference speech rate corresponding to one phoneme may be obtained.

For example, when there are many vowels in a phoneme column including the first phoneme, the first reference speech speed corresponding to the first phoneme may be relatively slow. Also, when there are many consonants in a phoneme column including the first phoneme, the first reference speech speed corresponding to the first phoneme may be relatively fast. In addition, when a word including the first phoneme is a word to be emphasized, the first reference speech speed corresponding to the first phoneme may be relatively slow because the word is to be uttered slowly.

For example, the reference speech speed acquisition module 124 may obtain a first reference speech speed corresponding to the first phoneme by using a third neural network model for estimating the reference speech speed. Specifically, the reference speech rate acquisition module 124 may identify the first phoneme from the alignment information acquired by the acoustic characteristic information acquisition module 122 . Also, the reference speech rate obtaining module 124 inputs the information on the identified first phoneme and the text obtained by the text acquisition module 121 to the third neural network model, and the first reference speech rate corresponding to the first phoneme. can be obtained.

For example, the third neural network model may be trained based on sample data (eg, sample text and sample voice data) used for learning the first neural network model 10 . That is, the third neural network model may be trained to estimate the section average speech speed of the sample acoustic characteristic information based on the sample acoustic characteristic information and the sample text corresponding to the sample acoustic characteristic information. Here, the third neural network model may be implemented as a statistical model such as a Hidden Markov Model (HMM) and a Deep Neural Network (DNN) model capable of estimating a section average firing rate. Data used for learning the third neural network model will be described later with reference to FIG. 8 .

In the above-described embodiment, it has been described that the first reference speech rate corresponding to the first phoneme is obtained using the third neural network model, but the present disclosure is not limited thereto. That is, the reference speech rate acquisition module 124 may obtain the first reference speech rate corresponding to the first phoneme by using a rule-based prediction method or a decision-based prediction method in addition to the third neural network.

As a second embodiment, the reference speech rate acquisition module 124 may acquire a second reference speech rate, which is a speech rate subjectively determined by a user listening to voice data. Specifically, the reference speech rate acquisition module 124 may obtain evaluation information on sample data used for learning of the first neural network model 10 . For example, the reference speech rate acquisition module 124 may obtain user evaluation information on sample speech data used for learning of the first neural network model 10 . Here, the evaluation information may be evaluation information on speed subjectively felt by a user who has listened to the sample voice data. For example, evaluation information may be obtained by receiving a user input through a UI displayed on a display of the electronic device 100 .

For example, when a user who has listened to the sample voice data thinks that the speech rate of the sample voice data is slightly slow, the reference speech rate obtaining module 124 provides first evaluation information for quickly setting the speech rate of the sample voice data from the user. (eg, 1.1 times). For example, when a user who has listened to the sample voice data thinks that the speech rate of the sample voice data is slightly fast, the reference speech rate obtaining module 124 provides second evaluation information for setting the speech rate of the sample voice data to be slow from the user. (eg, 0.95 times).

Also, the reference speech rate acquisition module 124 may obtain a second reference speech rate obtained by applying the evaluation information to the first reference speech rate corresponding to the first phoneme. For example, when the first evaluation information is acquired, the reference speech rate acquisition module 124 sets the speech rate corresponding to 1.1 times the first reference speech rate corresponding to the first phoneme to the second reference speech rate corresponding to the first phoneme. can be identified by speed. For example, when the second evaluation information is obtained, the reference speech rate acquisition module 124 sets the speech rate corresponding to 0.95 times the first reference speech rate corresponding to the first phoneme to the second reference speech rate corresponding to the first phoneme. can be identified by speed.

As a third embodiment. The reference speech rate acquisition module 124 may obtain a third reference speech rate based on the evaluation information on the reference sample data. Here, the reference sample data may include a plurality of sample texts and a plurality of sample voice data in which a reference speaker utters each of the plurality of sample texts. For example, the first reference sample data may include a plurality of sample voice data in which a specific voice actor utters each of a plurality of sample texts, and the second reference sample data may include a plurality of sample voice data in which another voice actor utters each of a plurality of sample texts. of sample voice data. Also, the reference speech rate acquisition module 124 may obtain a third reference speech rate based on the user's evaluation information on the reference sample data. For example, when the first evaluation information is obtained for the first reference sample data, the reference speech rate obtaining module 124 converts the speech rate of the first phoneme to 1.1 times the speech rate of the first phoneme corresponding to the first reference sample data. It can be identified as a third reference speech rate corresponding to . For example, when the second evaluation information is acquired for the first reference sample data, the reference speech rate acquisition module 124 converts the speech rate of the first phoneme to 0.95 times the speech rate of the first phoneme corresponding to the first reference sample data. It can be identified as a third reference speech rate corresponding to .

Further, the reference speech rate obtaining module 124 determines one of a first reference speech rate corresponding to the first phoneme, a second reference speech rate corresponding to the first phoneme, and a third reference speech rate corresponding to the first phoneme. It can be identified as a standard speech rate corresponding to 1 phoneme.

The speech rate control information obtaining module 125 includes the speech rate corresponding to the first phoneme obtained through the speech rate acquisition module 123 and the speech rate corresponding to the first phoneme obtained through the reference speech rate acquisition module 124. Based on, it is a configuration for obtaining speech rate control information.

Specifically, if the speech rate corresponding to the n-th phoneme acquired through the speech rate acquisition module 123 is Xn and the reference speech rate corresponding to the n-th phoneme acquired through the reference speech rate acquisition module 124 is Xrefn, , Speech rate control information Sn corresponding to the n-th phoneme may be defined as (Xrefn / Xn). For example, when the currently predicted speech rate X1 corresponding to the first phoneme is 20 (phoneme / sec) and the reference speech rate Xref1 corresponding to the first phoneme is 18 (phoneme / sec), the The firing rate control information S1 may be 0.9.

The voice data acquisition module 126 is a component for acquiring voice data corresponding to text.

Specifically, the voice data obtaining module 126 inputs the sound characteristic information corresponding to the text obtained by the sound characteristic information obtaining module 122 to the second neural network model 20 set based on the speech speed control information, Voice data corresponding to text may be acquired.

While at least one frame corresponding to the first phoneme among the sound characteristic information 220 is input to the second neural network model 20, the voice data obtaining module 126 based on the speech speed control information corresponding to the first phoneme , the number of loops of the decoder 20-2 in the second neural network model 20 can be identified. Further, while at least one frame corresponding to the first phoneme is input to the second neural network model 20, the voice data acquisition module 126 receives a plurality of first voice data corresponding to the number of loops from the decoder 20-2. can be obtained.

When one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model 20, a plurality of second voice sample data corresponding to the number of loops may be obtained. Also, a set of second voice sample data obtained by inputting each of at least one frame corresponding to the first phoneme to the second neural network model 20 may be the first voice data. Here, the plurality of first voice data may be voice data corresponding to the first phoneme.

That is, by adjusting the number of loops of the decoder 20-2, the number of samples of output audio data can be adjusted. Therefore, the speech rate of voice data can be adjusted by adjusting the number of loops of the decoder 20-2. A method for adjusting the firing rate through the second neural network model 20 will be described later with reference to FIG. 3 .

Further, the voice data acquisition module 126 is included in the sound characteristic information in the second neural network model 20 in which the number of loops of the decoder 20-2 is set based on the speech rate control information corresponding to each of a plurality of phonemes. Voice data corresponding to text may be obtained by inputting each of a plurality of phonemes.

3 is a block diagram for explaining the configuration of a second neural network model (eg, a neural vocoder neural network model) in the TTS model 200 according to an embodiment of the present disclosure.

Referring to FIG. 3 , the encoder 20-1 of the second neural network model 20 may receive acoustic characteristic information 220 and output vector information 225 corresponding to the acoustic characteristic information 220. . Here, the vector information 225 is data output from a hidden layer when viewed from the viewpoint of the second neural network model 20, and thus may be referred to as a hidden representation.

While at least one frame corresponding to the first phoneme among the sound characteristic information 220 is input to the second neural network model 20, the voice data obtaining module 126 based on the speech speed control information corresponding to the first phoneme The number of loops of the decoder 20-2 can be identified. Further, while the at least one frame corresponding to the first phoneme is input to the second neural network model 20, the voice data acquisition module 126 generates a plurality of first phonemes corresponding to the number of loops identified from the decoder 20-2. Voice data can be obtained.

That is, when one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model 20, a plurality of second voice data corresponding to the number of loops may be obtained. For example, when one of the at least one frame corresponding to the first phoneme among the sound characteristic information 220 is input to the encoder 20-1 of the second neural network model 20, vector information corresponding thereto may be output. there is. Then, the vector information is input to the decoder 20-2, and the decoder 20-2 operates with the number of loops N times, that is, with the number of loops N times per frame of the sound characteristic information 220, Audio data can be output.

Also, a set of second voice data obtained by inputting each of at least one frame corresponding to the first phoneme to the second neural network model 20 may be the first voice data. Here, the plurality of first voice data may be voice data corresponding to the first phoneme.

In an embodiment in which the voice data of the first frequency (khz) is obtained by the decoder 20-2 based on the sound characteristic information of the first time interval (sec), the value of the speech speed control information When this reference value (eg, 1), one frame included in the acoustic characteristic information is input to the second neural network model 20, and the number of loops corresponding to (first time interval X first frequency) is decoded ( 20-2) is operated so that the number of voice data corresponding to the number of loops can be obtained. For example, when voice data of 24 khz is obtained from the decoder 20-2 based on sound characteristic information having a shift size of 10 msec, and the value of the speech speed control information is a reference value (eg, 1), the sound characteristic information One frame included in is input to the second neural network model 20, and the decoder 20-2 operates with 240 loops, so that 240 pieces of voice data can be obtained.

In addition, in an embodiment in which voice data of a first frequency (khz) is acquired by the decoder 20-2 based on the acoustic characteristic information of the first time interval (sec), the shift size is the acoustic characteristic information One included frame is input to the second neural network model 20, and the decoder 20-2 operates with the number of loops corresponding to (first time interval X first frequency X firing speed control information) and the corresponding loop number A number of voice data corresponding to may be obtained. For example, when voice data of 24 khz is obtained from the decoder 20-2 based on sound characteristic information having a shift size of 10 msec, and the value of the speech speed control information is 1.1, one frame included in the sound characteristic information This is input to the second neural network model 20, and the decoder 20-2 operates with 264 loops, so that 264 pieces of voice data can be obtained.

Here, the number of voice data (eg, 264) obtained when the value of the speech rate control information is 1.1 may be greater than the number of voice data (eg, 240) obtained when the value of the speech rate control information is the reference value. there is. That is, when the value of the speech rate control information is adjusted to 1.1, since voice data corresponding to the existing 10 msec is output at 11 mec, the speech rate can be adjusted slower than when the value of the speech rate control information is the reference value.

That is, when the reference value of the speech rate control information is 1 and the value of the speech rate control information is S, the number of loops N' of the decoder 20-2 may be expressed as Equation 1 below.

은, 발화 속도 조절을 위한 디코더(20-2)의 제n 음소에서의 루프 횟수를 의미하며,

은 디코더(20-2)의 기본 루프 횟수를 의미할 수 있다. 그리고, 제n 음소에서의

은 발화 속도 조절 정보의 값으로,

이 1.1인 경우 10% 빠르게 발화되는 음성 데이터를 획득할 수 있다. In Equation 1,

denotes the number of loops in the n-th phoneme of the decoder 20-2 for speech speed control,

may mean the number of basic loops of the decoder 20-2. And, at the nth phoneme

is the value of the ignition rate control information,

When is 1.1, voice data that is uttered 10% faster can be obtained.

Also, as shown in Equation 1, it can be seen that the speech rate control information is set differently for each phoneme included in the acoustic characteristic information 220 input to the second neural network model 20. That is, according to the present disclosure, based on Equation 1, by using an adaptive speech speed control method that differently adjusts speech speed for each phoneme included in the sound characteristic information 220, voice data whose speech speed is adjusted in real time is obtained. can be obtained

4 is a diagram for explaining a method for obtaining, by an electronic device, voice data with improved speech speed, according to an embodiment of the present disclosure.

Referring to FIG. 4 , the electronic device 100 may acquire text 210 . Here, the text 210 is text to be converted into voice data, and there is no limitation on how to acquire it. That is, the text 210 includes various texts such as text input from the user of the electronic device 100, text provided by the voice recognition system (eg, Bixby) of the electronic device 100, and text received from an external server. can do.

In addition, the electronic device 100 may input text 210 to the first neural network model 10 to obtain acoustic characteristic information 220 and alignment information 400 . Here, the sound characteristic information 220 may be information including voice characteristics and speech speed characteristics corresponding to the text 210 of a specific speaker (specific speaker corresponding to the first neural network model). Further, the alignment information 400 may be alignment information obtained by matching the phonemes included in the text 210 with each frame of the sound characteristic information 220 .

In addition, the electronic device 100 may acquire the speech speed 410 corresponding to the sound characteristic information 220 based on the alignment information 400 through the speech speed obtaining module 123 . Here, the speech speed 410 may be information about an actual speech speed when the sound characteristic information 220 is converted into voice data 230 . Also, the speech speed 410 may include speech speed information for each phoneme included in the sound characteristic information 220 .

In addition, the electronic device 100 may obtain the reference speech speed 420 based on the text 210 and the alignment information 400 through the speech speed control information obtaining module 125 . Here, the reference speech speed 420 may mean an optimal speech speed for phonemes included in the text 210 . Also, the reference speech speed 420 may include reference speech speed information for each phoneme included in the sound characteristic information 220 .

Further, the electronic device 100 may obtain the speech rate control information 430 based on the speech speed 410 and the reference speech speed 420 through the speech speed control information obtaining module 125 . Here, the speech speed control information 430 may be information for adjusting the speech speed of each phoneme included in the sound characteristic information 220 . For example, when the speech rate 410 for the m-th phoneme is 20 (phoneme/sec) and the reference speech rate 420 for the m-th phoneme is 18 (phoneme/sec), the speech rate for the m-th phoneme is Adjustment information 430 can be identified (18 / 20) as 0.9.

In addition, the electronic device 100 inputs the sound characteristic information 220 to the second neural network model 20 set based on the speech rate control information 430, so that voice data 230 corresponding to the text 210 is generated. can be obtained.

For example, while at least one frame corresponding to the m-th phoneme of the sound characteristic information 220 is input to the encoder 20-1 of the second neural network model 20, the electronic device 100 selects the m-th phoneme. The number of loops of the decoder 20 - 2 of the second neural network model 20 may be identified based on the speech rate control information 430 corresponding to . For example, if the speech rate control information 430 for the m-th phoneme is 0.9, the decoder 20-2 while a frame corresponding to the m-th phoneme among the sound characteristic information 220 is input to the encoder 20-1 ) may be (basic loop count/speech speed control information corresponding to the mth phoneme). That is, if the basic number of loops is 240, the number of loops of the decoder 20-2 while the frame corresponding to the mth phoneme of the sound characteristic information 220 is input to the encoder 20-1 may be 264. .

If the number of loops is identified, the electronic device 100 determines the number of loops corresponding to the m-th phoneme while the frame corresponding to the m-th phoneme is input to the decoder 20-2 of the sound characteristic information 220. By operating step 2), voice data corresponding to the number of loops corresponding to the m-th phoneme per one frame of the sound characteristic information 220 may be obtained. In addition, the electronic device 100 may obtain voice data 230 corresponding to the text 210 by performing this process on all phonemes included in the text 210 .

5 is a diagram for explaining alignment information obtained by matching each frame of sound characteristic information with each phoneme included in text, according to an embodiment of the present disclosure.

Referring to FIG. 5 , alignment information obtained by matching each frame of sound characteristic information with each phoneme included in text may have a size of (N, T). Here, N may represent the total number of phonemes included in the text 210, and T may represent the number of frames of the sound characteristic information 220 corresponding to the text 210.

를 음향 특성 정보(220)에서의 n번째 음소, t 번째 프레임에서의 웨이트라고 정의하면,

일 수 있다. And,

If is defined as the weight in the n-th phoneme and t-th frame in the acoustic characteristic information 220,

can be

는 하기의 수학식 2와 같을 수 있다.And, the phoneme mapped to the t-th frame in the alignment information

May be the same as Equation 2 below.

는 t 번째 프레임에 대응되는

의 값이 가장 큰 음소일 수 있다.That is, referring to Equation 2, the phoneme mapped to the t-th frame

corresponds to the tth frame

The value of may be the largest phoneme.

인 프레임 사이에서

에 해당되는 음소의 길이를 식별할 수 있다. 즉, n 번째 음소의 길이를

이라고 정의하면, n번째의 음소의 길이는 수학식 3과 같을 수 있다.And,

between in-frame

The length of the phoneme corresponding to can be identified. That is, the length of the nth phoneme

When defined as , the length of the n-th phoneme may be equal to Equation 3.

은 2이며,

는 3일 수 있다.That is, when referring to Equation 2, the alignment information of FIG.

is 2,

may be 3.

Meanwhile, there may be phonemes that are not mapped to max values, such as the square box area of FIG. 5 . For example, in the TTS model using the first neural network model 10, special symbols may be used as phonemes. In this case, special symbols may create a pause, but affect only the prosody before and after and may not actually be uttered. can In this case, there may be phonemes that are not mapped to frames, such as the square box area of FIG. 5 .

을 수학식 4와 같이 할당할 수 있다. 즉,

인 프레임 사이에서, n 부터

번째 음소의 길이는 수학식 4와 같을 수 있다. 여기서,

는 1보다 큰 값일 수 있다.In this case, the length of the unmapped phoneme

Can be assigned as shown in Equation 4. in other words,

Between in-frames, from n

The length of the th phoneme may be equal to Equation 4. here,

may be a value greater than 1.

은 0.5일 수 있으며,

은 0.5일 수 있다. Referring to Equation 4, the alignment information of FIG. 5

may be 0.5,

may be 0.5.

As described above, the length of the phoneme included in the acoustic characteristic information 220 can be identified through the alignment information, and the speech speed of each phoneme can be identified through the length of the phoneme.

은 수학식 5와 같을 수 있다.Specifically, the rate of speech at the n-th phoneme included in the sound characteristic information 220

may be the same as Equation 5.

는 50,

는 33.3일 수 있다. In Equation 4, r may be a reduction factor of the first neural network model 10. As an example, when r is 1 and frame-length is 10 ms,

is 50,

may be 33.3.

However, since the speech speed for one phoneme is for a short section, when the speech speed is predicted for an extremely short section, the difference in length between phonemes is reduced, resulting in unnatural results. In addition, when the speech rate is predicted for a too short period, an unnatural result may occur because the prediction value of the speech rate changes too quickly on the time axis. In addition, when estimating the average speech speed for an excessively long section when uttering speed is predicted, it may be difficult to reflect when both slow and fast utterances are present in the text. In addition, in the streaming structure, since the identified speech speed is a speed prediction for speech that has already been output, a delay in speech speed control may occur. Therefore, a method for measuring the average speech speed for an appropriate section is required, This will be described later with reference to FIGS. 6 and 7 .

6 is a diagram for explaining a method of identifying an average speech rate for each phoneme included in acoustic characteristic information according to a first embodiment of the present disclosure.

Referring to embodiment 610 of FIG. 6 , the electronic device 100 may calculate an average of speech speeds for M recent phonemes included in the acoustic characteristic information 220 . For example, when n<M, the average ignition rate may be calculated by averaging only the corresponding element.

은

,

및

의 평균 값으로 계산될 수 있다. 또한, 제5 음소에 대한 평균 발화 속도

는

내지

의 평균 값으로 계산될 수 있다. Also, as an example, when M=5, the average speech rate for the third phoneme as in the 620 embodiment of FIG. 6

silver

,

and

can be calculated as the average value of Also, the average rate of speech for the fifth phoneme

Is

pay

can be calculated as the average value of

The method of calculating the average utterance speed for each phoneme through the embodiments 610 and 620 of FIG. 6 may be referred to as a simple moving average method.

7 is a diagram for explaining a method of identifying an average speech rate for each phoneme included in acoustic characteristic information according to a second embodiment of the present disclosure.

7 is a formula for explaining an embodiment in which an average utterance rate for each phoneme is identified by an Exponential Moving Average (EMA) method according to an embodiment of the present disclosure.

That is, according to the EMA method as shown in the equation of FIG. 7 , the average length of an appropriate section can be calculated as the weight is exponentially reduced as the utterance rate for a phoneme farther from the current phoneme increases.

Here, as the value of α in FIG. 7 increases, the average firing rate for a shorter section can be calculated, and as the value of α decreases, the average firing rate for a longer section can be calculated. Accordingly, the electronic device 100 may calculate the current average speech rate in real time by selecting an appropriate value of α according to the situation.

8 is a diagram for explaining a method for identifying a reference speech rate according to an embodiment of the present disclosure.

8 is a diagram for explaining a method of learning a third neural network model for obtaining a reference speech rate corresponding to each phoneme included in the sound characteristic information 220 according to an embodiment of the present disclosure.

For example, the third neural network model may be trained based on sample data (eg, sample text and sample voice data). For example, the sample data may be sample data used for learning of the first neural network model 10 .

Then, based on the sample voice data, acoustic characteristic information corresponding to the sample voice data is extracted, and the speech rate for each phoneme included in the sample voice data may be identified as shown in FIG. 8 . In addition, a third neural network model may be learned based on the speech rate for each phoneme included in the sample text and sample voice data.

That is, the third neural network model may be trained to estimate the section average speech speed of the sample acoustic characteristic information based on the sample acoustic characteristic information and the sample text corresponding to the sample acoustic characteristic information. Here, the third neural network model may be implemented as a statistical model such as a Hidden Markov Model (HMM) and a Deep Neural Network (DNN) model capable of estimating a section average firing rate.

In addition, the electronic device 100 may identify the reference speech rate for each phoneme included in the sound characteristic information 220 using the learned third neural network model, the text 210, and the alignment information 400.

9 is a flowchart illustrating an operation of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 9 , the electronic device 100 may acquire text (S910). Here, the text may include various types of text, such as text input from a user of the electronic device 100, text provided by a voice recognition system (eg, Bixby) of the electronic device, and text received from an external server.

In addition, the electronic device 100 inputs text into the first neural network model to obtain acoustic characteristic information corresponding to the text and alignment information obtained by matching each frame of the acoustic characteristic information with each phoneme included in the text ( S920). As an example, the alignment information may be matrix information having a size of (N, T) as described in FIG. 5 .

Then, the electronic device 100 may identify the speaking speed of the acoustic characteristic information based on the acquired alignment information (S930). Specifically, the electronic device 100 may identify the speech rate for each phoneme included in the acoustic characteristic information based on the acquired alignment information. Here, the speech speed for each phoneme may be a speech speed corresponding to one phoneme, but is not limited thereto. That is, the speech speed for each phoneme may be an average speech speed in consideration of speech speeds corresponding to each of at least one phoneme prior to the corresponding phoneme.

Then, the electronic device 100 may identify a reference speech speed for each phoneme included in the sound characteristic information based on the text and sound characteristic information (S940). Here, the reference speech rate may be identified by various methods as described in FIG. 1 .

For example, the electronic device 100 may obtain a first reference speech rate for each phoneme included in the sound characteristic information based on the obtained text and sample data used for learning of the first neural network.

For example, the electronic device 100 may obtain evaluation information on sample data used for learning the first neural network model. For example, the electronic device 100 may provide voice data among sample data to the user, and then receive evaluation information for feedback thereto. Also, the electronic device 100 may obtain a second reference speech speed for each phoneme included in the sound characteristic information based on the first reference speech speed and evaluation information.

Then, the electronic device 100 may identify a reference speech speed for each phoneme included in the acoustic characteristic information based on at least one of the identified first reference speech speed and the second reference speech speed.

Then, the electronic device 100 may acquire speech speed control information based on the speech speed of the sound characteristic information and the reference speech speed (S950). Specifically, if the speech rate corresponding to the n-th phoneme is Xn and the reference speech rate corresponding to the n-th phoneme is Xrefn, the speech rate control information Sn corresponding to the n-th phoneme may be defined as (Xrefn / Xn). .

Then, the electronic device 100 may obtain voice data corresponding to the text by inputting the acoustic characteristic information to the second neural network model set based on the acquired speech rate control information (S960).

Specifically, the second neural network model may include an encoder receiving sound characteristic information and a decoder receiving vector information output from the encoder and outputting voice data. And, while at least one frame corresponding to a specific phoneme included in the sound characteristic information is input to the second neural network model, the electronic device 100 uses the second neural network model based on the speech rate control information corresponding to the phoneme. The number of loops of the decoder included in can be identified. Further, the electronic device 100 acquires first voice data corresponding to the number of loops by operating the decoder with the number of loops identified based on the input of at least one frame corresponding to the corresponding phoneme to the second neural network model. can do.

Specifically, when one of at least one frame corresponding to a specific phoneme among sound characteristic information is input to the second neural network model, second voice data corresponding to the number of identified loops may be obtained. Also, a set of a plurality of second voice data acquired through at least one frame corresponding to a specific phoneme among sound characteristic information may be first voice data corresponding to a specific phoneme. That is, the second voice data may be voice data corresponding to one frame of sound characteristic information, and the first voice data may be voice data corresponding to one specific phoneme.

For example, the decoder is characterized in that the shift size (Shift size) acquires voice data of a first frequency (khz) based on sound characteristic information of a first time interval (sec), and the value of the speech speed control information is In the case of the reference value, one frame included in the acoustic characteristic information may be input to the second neural network model, and second voice data corresponding to the product of the first time interval and the first frequency may be obtained.

10 is a block diagram for explaining a configuration of an electronic device according to an embodiment of the present disclosure. Referring to FIG. 10 , the electronic device 100 includes a memory 110, a processor 120, a microphone 130, a display 140, a speaker 150, a communication interface 160, and a user interface 170. can do. Meanwhile, since the memory 110 and the processor 120 shown in FIG. 10 overlap with the memory 110 and the processor 120 described in FIG. 1, overlapping descriptions are omitted. Also, it goes without saying that some of the components of FIG. 10 may be removed or other components may be added according to the implementation example of the electronic device 100 .

The microphone 130 is a component through which the electronic device 100 receives a voice signal. Specifically, the microphone 130 may receive an external voice signal using a microphone and process it as electrical voice data. In this case, the microphone 130 may transfer the processed voice data to the processor 120 .

The display 140 is a component for the electronic device 100 to visually provide information. The electronic device 100 may include one or more displays 140 and may display text for conversion into voice data, a UI for acquiring evaluation information from a user, and the like through the display 140 . In this case, the display 140 may be implemented as a liquid crystal display (LCD), a plasma display panel (PDP), organic light emitting diodes (OLED), a transparent OLED (TOLED), or a micro LED. Also, the display 140 may be implemented in the form of a touch screen capable of detecting a user's touch manipulation, or may be implemented as a flexible display capable of being folded or bent. In particular, the display 140 may visually provide a response corresponding to a command included in the voice signal.

The speaker 150 is a component for the electronic device 100 to provide information aurally. The electronic device 100 may include one or more speakers 150 and may output voice data obtained according to the present disclosure as audio signals through the speakers 150 . Meanwhile, although a configuration for outputting an audio signal may be implemented as the speaker 150, this is merely an example and may be implemented as an output terminal, of course.

The communication interface 160 is a component capable of communicating with an external device. Meanwhile, connecting the communication interface 160 to an external device may include communication through a third device (eg, a repeater, a hub, an access point, a server, or a gateway). Wireless communication is, for example, LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications), etc. may include cellular communication using at least one. According to one embodiment, wireless communication, for example, WiFi (wireless fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee (Zigbee), near field communication (NFC), magnetic secure transmission (Magnetic Secure Transmission), radio It may include at least one of a frequency (RF) and a body area network (BAN). Wired communication may include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), power line communication, or plain old telephone service (POTS). there is. A network in which wireless communication or wired communication is performed may include at least one of a telecommunication network, eg, a computer network (eg, LAN or WAN), the Internet, or a telephone network.

In particular, the communication interface 160 can provide a voice recognition function in the electronic device 100 by communicating with an external server. However, the present disclosure is not limited thereto, and the electronic device 100 may provide a voice recognition function within the electronic device 100 without communication with an external server.

The user interface 170 is a component for receiving a user command for controlling the electronic device 100 . In particular, the user interface 170 may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or may be implemented as a touch screen capable of simultaneously performing the above-described display function and manipulation input function. Here, the buttons may be various types of buttons such as mechanical buttons, touch pads, wheels, etc. formed on an arbitrary area such as the front, side, or rear surface of the main body of the electronic device 100 .

It should be understood to include various modifications, equivalents, and/or alternatives of the embodiments herein. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has," "may have," "includes," or "may include" indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B. Expressions such as "first," "second," "first," or "second," used in this document may modify various elements, regardless of order and/or importance, and refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, a third component) do not exist between the other components.

As used in this document, the expression "configured to" means "suitable for," "having the capacity to," depending on the circumstances. ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a coprocessor configured (or configured) to perform A, B, and C" may include a dedicated processor (e.g., embedded processor) to perform those operations, or one or more software programs stored in a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

On the other hand, the term "unit" or "module" used in the present disclosure includes units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits, for example. can A “unit” or “module” may be an integrated component or a minimum unit or part thereof that performs one or more functions. For example, the module may be composed of an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including commands stored in a storage medium readable by a machine (eg, a computer). The device may receive instructions stored from the storage medium. A device capable of calling and operating according to the called command may include a stacked display device according to the disclosed embodiments When the command is executed by a processor, the processor directly or other components under the control of the processor A function corresponding to the command may be performed using a command A command may include a code generated or executed by a compiler or an interpreter A storage medium readable by a device may include non-transitory storage It can be provided in the form of a medium Here, 'non-temporary' means that the storage medium does not contain signals and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium. .

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or online through an application store (eg Play Store™). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

Each component (eg, module or program) according to various embodiments may be composed of a single object or a plurality of entities, and some of the sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by modules, programs, or other components may be executed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added. can

100: electronic device
110: memory
120: processor

Claims (20)

In the control method of an electronic device,
obtaining text;
By inputting the text to the first neural network model, acoustic feature information corresponding to the text and alignment information matching each frame of the acoustic feature information with each phoneme included in the text obtaining;
identifying an utterance rate of the acoustic characteristic information based on the acquired alignment information;
identifying a reference speech speed for each phoneme included in the sound characteristic information, based on the text and the sound characteristic information;
obtaining speech speed control information based on the speech speed of the sound characteristic information and the reference speech speed; and
and obtaining voice data corresponding to the text by inputting the acoustic characteristic information to a second neural network model set based on the acquired speech rate control information.
According to claim 1,
Identifying the speech speed of the acoustic characteristic information,
Identifying a speech rate corresponding to a first phoneme included in the acoustic characteristic information based on the acquired alignment information;
The step of identifying the reference speech rate,
identifying the first phoneme included in the acoustic characteristic information based on the acoustic characteristic information; and
and identifying a reference speech rate corresponding to the first phoneme based on the text.
According to claim 2,
The step of identifying the reference speech rate,
Acquiring a first reference speech rate corresponding to the first phoneme based on the text and sample data used for learning of the first neural network;
According to claim 3,
The step of identifying the reference speech rate,
obtaining evaluation information on sample data used for learning the first neural network model; and
Further comprising identifying a second reference speech rate corresponding to the first phoneme based on the first reference speech rate and the evaluation information;
The control method, characterized in that the evaluation information is obtained by a user of the electronic device.
According to claim 4,
The method further includes identifying a standard speech rate corresponding to the first phoneme based on one of the first reference speech rate and the second reference speech rate.
According to claim 2,
Identifying the speech rate corresponding to the first phoneme,
identifying an average speech speed corresponding to the first phoneme based on a speech speed corresponding to the first phoneme and a speech speed corresponding to each of at least one phoneme prior to the first phoneme in the sound characteristic information; contain more,
The step of obtaining the speech rate control information,
Acquiring speech rate adjustment information corresponding to the first phoneme based on the average speech rate corresponding to the first phoneme and the reference speech rate corresponding to the first phoneme.
According to claim 2,
The second neural network model includes an encoder receiving the acoustic characteristic information and a decoder receiving vector information output from the encoder,
Obtaining the voice data,
While at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, the decoder included in the second neural network model based on the speech rate control information corresponding to the first phoneme identifying the number of loops; and
obtaining a number of first voice data corresponding to the at least one frame and the identified number of loops based on the input of at least one frame corresponding to the first phoneme to the second neural network model; Including more
The first voice data is voice data corresponding to the first phoneme.
According to claim 7,
And if one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, second voice data corresponding to the number of loops is obtained.
According to claim 7,
The decoder is characterized in that the shift size (Shift size) acquires the voice data of the first frequency (khz) based on the sound characteristic information of the first time interval (sec),
When the value of the speech rate control information is a reference value, one frame included in the sound characteristic information is input to the second neural network model, and the number of voices corresponding to the product of the first time interval and the first frequency A control method characterized in that data is acquired.
According to claim 1,
The control method, characterized in that the speech rate control information is information on a ratio value of the speech rate of the acoustic characteristic information and the reference speech rate.
In electronic devices,
a memory storing at least one instruction; and
A processor configured to control the electronic device by executing at least one instruction stored in the memory;
the processor,
get the text
By inputting the text to the first neural network model, acoustic feature information corresponding to the text and alignment information matching each frame of the acoustic feature information with each phoneme included in the text to obtain,
Based on the obtained alignment information, identifying the speech rate of the acoustic characteristic information;
Based on the text and the sound characteristic information, a reference speech rate for each phoneme included in the sound characteristic information is identified;
Obtaining speech speed control information based on the speech speed of the acoustic characteristic information and the reference speech speed;
An electronic device that obtains voice data corresponding to the text by inputting the acoustic characteristic information to a second neural network model set based on the acquired speech rate control information.
According to claim 11,
the processor,
Identifying a speech rate corresponding to a first phoneme included in the acoustic characteristic information based on the obtained alignment information;
Based on the acoustic characteristic information, identifying the first phoneme included in the acoustic characteristic information;
An electronic device that identifies a reference speech rate corresponding to the first phoneme based on the text.
According to claim 12,
the processor,
An electronic device that obtains a first reference speech rate corresponding to the first phoneme based on the text and sample data used for learning of the first neural network.
According to claim 13,
the processor,
Acquiring evaluation information on sample data used for learning the first neural network model;
Identifying a second reference speech rate corresponding to the first phoneme based on the first reference speech rate and the evaluation information;
The electronic device, characterized in that the evaluation information is obtained by a user of the electronic device.
According to claim 14,
the processor,
An electronic device that identifies a reference speech speed corresponding to the first phoneme based on one of the first reference speech speed and the second reference speech speed.
According to claim 12,
the processor,
Identifying an average speech speed corresponding to the first phoneme based on a speech speed corresponding to the first phoneme and a speech speed corresponding to each of at least one phoneme prior to the first phoneme in the acoustic characteristic information;
The electronic device for acquiring speech rate control information corresponding to the first phoneme based on the average speech rate corresponding to the first phoneme and the reference speech rate corresponding to the first phoneme.
According to claim 12,
The second neural network model includes an encoder receiving the acoustic characteristic information and a decoder receiving vector information output from the encoder,
the processor,
While at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, the decoder included in the second neural network model based on the speech speed control information corresponding to the first phoneme identify the number of loops;
Based on at least one frame corresponding to the first phoneme being input to the second neural network model, first voice data corresponding to the at least one frame and the number of loops identified is obtained;
The electronic device, characterized in that the first voice data is voice data corresponding to the first phoneme.
According to claim 17,
The electronic device, characterized in that, when one of the at least one frame corresponding to the first phoneme among the sound characteristic information is input to the second neural network model, second voice data corresponding to the number of loops is obtained.
According to claim 17,
The decoder is characterized in that the shift size (Shift size) acquires the voice data of the first frequency (khz) based on the sound characteristic information of the first time interval (sec),
When the value of the speech rate control information is a reference value, one frame included in the sound characteristic information is input to the second neural network model, and the number of voices corresponding to the product of the first time interval and the first frequency An electronic device characterized in that data is obtained.
According to claim 11,
The electronic device, characterized in that the speech rate control information is information on a ratio value between the speech rate of the acoustic characteristic information and the reference speech rate.

Images