KR20230101167A - Method and apparatus for generating audio based on text in korean culture education - Google Patents

Abstract

The present invention provides a method and apparatus for generating text-based speech in a Korean cultural education platform, comprising the steps of receiving a request for synthetic sound data for text from a client device; generating synthetic sound data for text using a pre-learned speech synthesis model; and providing synthesized sound data to the client device.

Description

Text-based voice generation method and apparatus in Korean culture education platform {METHOD AND APPARATUS FOR GENERATING AUDIO BASED ON TEXT IN KOREAN CULTURE EDUCATION}

The present invention relates to a text-based voice generation method and apparatus in a Korean cultural education platform.

In general, TTS (Text to Speech) is a technology for converting text into human voice, and is used in various fields such as unmanned automatic response systems (ARS), announcement broadcasting, and audio caption services.

As artificial intelligence-based natural language processing technology develops, the number of users who want to use human-friendly audio content having a voice closer to that of a human than audio content using existing machine sounds is increasing. Accordingly, in recent years, various TTS services have been provided that provide synthetic sound data as if various speakers read the input text when text is input.

However, the artificial intelligence-based TTS model used for the TTS service has a problem in that the training performance deteriorates, such as the sound quality of synthesized speech data degrading and pronunciation blurring as the diversity of training data increases. In particular, there is a problem in that the quality of TTS results for single syllables, two syllables, and/or words is poor.

The background description of the invention has been prepared to facilitate understanding of the present invention. It should not be construed as an admission that matters described in the background art of the invention exist as prior art.

Accordingly, a method for providing synthesized sound data of improved sound quality and improved pronunciation is desired.

As a result, the inventors of the present invention tried to develop a text-based voice generation method and apparatus in a Korean cultural education platform.

The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, a text-based voice generation method and apparatus are provided in a Korean culture education platform according to an embodiment of the present invention. The method includes receiving a request for synthesized speech data for text from a client device; generating synthetic sound data for text using a pre-learned speech synthesis model; and providing synthesized sound data to the client device.

In the speech synthesis model according to an embodiment of the present invention, learning parameters of a first model pre-trained based on the first training data are reflected through transfer-learning, and the first training data It may be a second model learned based on second training data different from .

The first training data according to an embodiment of the present invention may include a sentence of a reading language, and the second training data may include at least one of single syllables, two syllables, and words.

Other embodiment specifics are included in the detailed description and drawings.

The present invention can provide synthesized sound data with improved sound quality and pronunciation through a speech synthesis model learned using training data having a large variety.

The present invention can provide synthesized sound data with improved pronunciation for single syllables, double syllables and/or words.

Effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present invention.

1 is a schematic diagram of a text-based speech generation system in a Korean cultural education platform according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a client device according to an embodiment of the present invention.
3 is a block diagram showing the configuration of a service providing server according to an embodiment of the present invention.
4 is an exemplary diagram for explaining an artificial intelligence-based speech synthesis model according to an embodiment of the present invention.
5 is a flowchart illustrating a text-based voice generation method in the Korean culture education platform according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various forms different from each other, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has," "may have," "includes," or "may include" indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the presence of additional features.

In this document, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document may modify various elements, regardless of order and/or importance, and refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this document, a first element may be named a second element, and similarly, the second element may also be renamed to the first element.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, third components) do not exist between the other components.

As used in this document, the expression "configured to" means "suitable for," "having the capacity to," depending on the circumstances. ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (e.g., embedded processor) to perform those operations, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, an ideal or excessively formal meaning. not be interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude the embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or entirely combined or combined with each other, and as those skilled in the art can fully understand, various interlocking and driving operations are possible, and each embodiment can be implemented independently of each other. It may be possible to implement together in an association relationship.

1 is a schematic diagram of a text-based speech generation system in a Korean cultural education platform according to an embodiment of the present invention.

Referring to FIG. 1 , a text-based voice generation 10 in a Korean cultural education platform includes a client device 100 and a service providing server 300 .

In the Korean culture education platform, the text-based voice generation system 10 trains the AI-based basic model built using low-variability data with high-variability data to generate text-based voice data with improved predictive stability for various input data. It is a system for creating

The client device 100 is an electronic device capable of acquiring text, requesting synthetic sound data for the obtained text from the service providing server 300, receiving and providing synthesized sound data from the service providing server in response to the request, It may include a smartphone, a tablet PC, a PC, a laptop, and the like.

The service providing server 300 receives a request for synthesized voice data for text from the client device 100, generates synthesized voice data for the received text using a pre-learned artificial intelligence-based voice synthesis model, and generates synthesized voice It may be a general-purpose computer, laptop, web server, and/or cloud server that performs various operations to provide data to the client device 100 .

In particular, a speech synthesis model is a pre-trained model trained with low diversity data (eg, a common corpus) as an input through transfer-learning for monosyllables, two syllables, or This may mean an advanced model trained on short data (reinforced corpus) such as words.

2 is a block diagram showing the configuration of a client device according to an embodiment of the present invention.

Referring to FIG. 2 , the client device 100 may include a memory interface 110 , one or more processors 120 and a peripheral interface 130 . The various components within client device 100 may be connected by one or more communication buses or signal lines.

The memory interface 110 may be connected to the memory 150 and transfer various data to the processor 120 . Here, the memory 150 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage storage, cloud , It may include at least one type of storage medium among blockchain databases.

In various embodiments, memory 150 includes operating system 151 , communication module 152 , graphical user interface module (GUI) 153 , sensor processing module 154 , telephony module 155 , and application module 156 . ) At least one or more of them may be stored. Specifically, the operating system 151 may include instructions for processing basic system services and instructions for performing hardware tasks. The communication module 152 may communicate with at least one of one or more other devices, computers, and servers. A graphical user interface module (GUI) 153 may process a graphical user interface. The sensor processing module 154 may process sensor-related functions (eg, process voice input received using one or more microphones 192). The phone module 155 may process phone-related functions. The application module 156 may perform various functions of a user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and other processing functions. In addition, the client device 100 may store one or more software applications 156 - 1 and 156 - 2 associated with one type of service in the memory 150 .

In various embodiments, the memory 150 may store a digital assistant client module 157 (hereinafter referred to as a DA client module), thereby storing instructions and various user data 158 for performing client-side functions of the digital assistant. (eg, user-customized vocabulary data, preference data, and other data such as the user's electronic address book, to-do list, shopping list, etc.).

Meanwhile, the DA client module 157 receives a user's voice input, text input, touch input, and/or gesture input through various user interfaces (eg, the I/O subsystem 140) provided in the client device 100. can be obtained

In addition, the DA client module 157 may output audio-visual and tactile data. For example, the DA client module 157 may output data consisting of a combination of at least two of voice, sound, notification, text message, menu, graphic, video, animation, and vibration. In addition, the DA client module 157 may communicate with a digital assistant server (not shown) using the communication subsystem 180 .

In various embodiments, the DA client module 157 may collect additional information about the surrounding environment of the client device 100 from various sensors, subsystems, and peripheral devices in order to construct a context associated with the user input. . For example, the DA client module 157 may infer the user's intention by providing context information together with the user's input to the digital assistant server. Here, the situational information that may accompany the user input may include sensor information, eg, lighting, ambient noise, ambient temperature, image of the surrounding environment, video, and the like. For another example, the contextual information may include the physical state of the client device 100 (eg, device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). For another example, the contextual information is information related to the state of the software of the client device 100 (eg, processes running on the client device 100, installed programs, past and present network activity, background services, error logs, resource usage). etc.) may be included.

In various embodiments, the memory 150 may include added or deleted commands, and the client device 100 may also include additional components other than the configuration shown in FIG. 2 or may exclude some components.

The processor 120 may control the overall operation of the client device 100, acquires text by driving an application or program stored in the memory 150, and transfers synthetic sound data for the obtained text to the service providing server 300. You can perform various commands to request with .

The processor 120 may correspond to an arithmetic device such as a central processing unit (CPU) or an application processor (AP). In addition, the processor 120 may be implemented in the form of an integrated chip (IC) such as a System on Chip (SoC) in which various computing devices such as a Neural Processing Unit (NPU) are integrated.

The peripheral interface 130 may be connected to various sensors, subsystems, and peripheral devices to provide data so that the client device 100 can perform various functions. Here, the fact that the client device 100 performs a certain function may be understood as being performed by the processor 120 .

Peripheral interface 130 may receive data from motion sensor 160, light sensor (light sensor) 161, and proximity sensor 162, through which client device 100 may receive orientation, light, and proximity. sensing function, etc. As another example, peripheral interface 130 may receive data from other sensors 163 (positioning system-GPS receiver, temperature sensor, biometric sensor), through which the client device 100 may receive data from other sensors. It can perform functions related to (163).

In various embodiments, the client device 100 may include a camera subsystem 170 coupled to the peripheral interface 130 and an optical sensor 171 coupled thereto, through which the client device 100 may take pictures and video Various shooting functions such as clip recording can be performed.

In various embodiments, client device 100 may include communication subsystem 180 coupled with peripheral interface 130 . The communication subsystem 180 is composed of one or more wired/wireless networks, and may include various communication ports, radio frequency transceivers, and optical transceivers.

In various embodiments, client device 100 includes an audio subsystem 190 coupled to peripheral interface 130, which audio subsystem 190 includes one or more speakers 191 and one or more microphones 192. By including, the client device 100 can perform voice-activated functions, such as voice recognition, voice duplication, digital recording, and telephony functions.

In various embodiments, client device 100 may include I/O subsystem 140 coupled with peripheral interface 130 . For example, the I/O subsystem 140 may control the touch screen 143 included in the client device 100 through the touch screen controller 141 . As an example, the touch screen controller 141 uses any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. and cessation of movement. For another example, I/O subsystem 140 may control other input/control devices 144 included in client device 100 via other input controller(s) 142 . As an example, other input controller(s) 142 may control one or more buttons, rocker switches, thumb-wheels, infrared ports, USB ports, and pointer devices such as styluses and the like.

3 is a block diagram showing the configuration of a service providing server according to an embodiment of the present invention.

Referring to FIG. 3, the service providing server 300 may include a communication interface 310, a memory 320, an I/O interface 330, and a processor 340, each of which includes one or more communication buses or They can communicate with each other through signal lines.

The communication interface 310 may be connected to the client device 100 through a wired/wireless communication network to exchange data.

On the other hand, the communication interface 310 enabling transmission and reception of such data includes a communication pod 311 and a wireless circuit 312, where the wired communication port 311 is one or more wired interfaces, for example, Ethernet, This may include Universal Serial Bus (USB), FireWire, and the like. Also, the wireless circuit 312 may transmit/receive data with an external device through an RF signal or an optical signal. In addition, wireless communication may use at least one of a plurality of communication standards, protocols and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

The memory 320 may store various data used in the service providing server 300 .

In various embodiments, the memory 320 may include a volatile or non-volatile recording medium capable of storing various data, commands, and information. For example, the memory 320 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage storage , Cloud, and a blockchain database may include at least one type of storage medium.

In various embodiments, the memory 320 may store a configuration of at least one of the operating system 321 , the communication module 322 , the user interface module 323 , and one or more applications 324 .

Operating system 321 (e.g. embedded operating systems such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) is a variety of software for controlling and managing general system tasks (e.g. memory management, storage device control, power management, etc.) components and drivers, and may support communication between various hardware, firmware, and software components.

The communication module 323 may support communication with other devices through the communication interface 310 . The communication module 320 may include various software components for processing data received by the wired communication port 311 or the wireless circuit 312 of the communication interface 310 .

The user interface module 323 may receive a user's request or input from a keyboard, touch screen, microphone, etc. through the I/O interface 330 and provide a user interface on a display.

Applications 324 may include programs or modules configured to be executed by one or more processors 330 .

The I/O interface 330 may connect at least one of an input/output device (not shown) of the service providing server 300, for example, a display, a keyboard, a touch screen, and a microphone, to the user interface module 323. The I/O interface 330 may receive user input (eg, voice input, keyboard input, touch input, etc.) together with the user interface module 323 and process a command according to the received input.

The processor 340 is connected to the communication interface 310, the memory 320, and the I/O interface 330 to control the overall operation of the service providing server 300, and an application or program stored in the memory 320. Receives a request for synthesized voice data for text from the client device 100 through, generates synthesized voice data using the advanced model using the text as an input, and provides various synthesized voice data to the client device 100 through commands can be executed.

Specifically, the processor 340 reflects the learning parameters of the first model, which have been learned in advance based on sentences with a relatively low diversity, to the second model through transfer learning, and the learning parameters of the first model An advanced model may be generated by learning data with a large variety (eg, short sentences, word lists, and/or various sentences excluding reading texts) in the second model in which . The processor 340 may use the text received from the client device 100 as an input to generate synthesized sound data using the advanced model, and transmit the generated synthesized voice data to the client device 100 .

In various embodiments, various sentences other than short sentences, words, and/or texts may be used as training data according to different ratios.

The processor 340 may correspond to an arithmetic device such as a central processing unit (CPU) or an application processor (AP). In addition, the processor 340 may be implemented in the form of an integrated chip (IC) such as a System on Chip (SoC) in which various computing devices are integrated. Alternatively, the processor 340 may include a module for calculating an artificial neural network model, such as a Neural Processing Unit (NPU).

4 is an exemplary diagram for explaining an artificial intelligence-based speech synthesis model according to an embodiment of the present invention. In the presented embodiment, a method performed by the processor 340 of the service providing server 300 will be described.

Referring to FIG. 4, the processor 340 uses a common corpus 400 corresponding to a text-oriented sentence with less variety to learn a first model to generate a pre-trained model 410, and pre-trained After reflecting the learning parameters (eg, filter (weight), layer type, number of channels, stride, etc.) of the model 410 in the second model, the second model is built using short data (reinforced corpus) 420 with high diversity. An advanced model 430 may be generated by training.

5 is a flowchart illustrating a text-based voice generation method in the Korean culture education platform according to an embodiment of the present invention. In the presented embodiment, a method performed by the processor 340 of the service providing server 300 will be described.

Referring to FIG. 5 , the processor 340 receives a request for synthesized voice data for text from the client device 100 (S500), and according to the request, synthesized voice data for the text using a pre-learned voice synthesis model. It generates (S510), and provides the generated synthesized sound data to the client device 100 (S520).

Here, the pre-learned speech synthesis model is a model generated by learning short data with high diversity in a second model reflecting the learning parameters of the first model pre-learned as an input of text with a low diversity as an input as described above. means

Through this, the present invention can improve the sound quality of synthetic sound data of an artificial intelligence-based speech synthesis model learned using training data having a large variety.

Although one embodiment of the present invention has been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be variously modified and implemented without departing from the technical spirit of the present invention. there is. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

10: Text-based voice generation system in Korean cultural education platform
100: client device
110: memory interface 120: processor
130 Peripheral Interface 140 I/O Subsystem
141: touch screen controller 142: other input controller
143: touch screen
144: other input control devices
150: memory 151: operating system
152: communication module 153: GUI module
154: sensor processing module 155: phone module
156: applications
156-1, 156-2: application
157: digital assistant client module
158: user data
160: motion sensor 161: light sensor
162 Proximity sensor 163 Other sensors
170: camera subsystem 171: optical sensor
180: communication subsystem
190: audio subsystem
191: speaker 192: microphone
300: service providing server
310: communication interface
311 wired communication port 312 wireless circuit
320: memory
321: operating system 322: communication module
323: user interface module 324: application
330: I/O interface 340: processor

Claims (2)

In the text-based voice generation method in the Korean culture education platform according to the processor,
receiving a request for synthetic sound data for text from a client device;
generating synthetic sound data for the text using a pre-learned speech synthesis model; and
Providing the synthesized sound data to the client device;
The speech synthesis model,
Learning parameters of a first model pre-trained based on the first training data are reflected through transfer-learning, and learned based on second training data different from the first training data. The second model, a text-based speech generation method in a Korean cultural education platform. According to claim 1,
The first training data includes a sentence in a reading language,
The second training data includes at least one of monosyllables, two syllables and words, text-based speech generation method in the Korean culture education platform.

Images