KR20210112825A - System and method for providing text-to-speech service based on celebrities' voice - Google Patents

Abstract

A method of providing a voice synthesis service according to an embodiment of the present disclosure includes the steps of: allowing a voice analysis part to analyze voice data of a specific person, and building a voice DB based on the analysis result; allowing a communication part to receive content generated for the specific person selected by a user from a user terminal; allowing the communication part to receive a text-to-speech request including the selected specific person information and the generated content; allowing a voice synthesis part to extract voice data of the specific person selected from the voice DB in response to a text-to-speech conversion request, converts text information included in the content into voice synthesis data based on the extracted voice data, and provides the voice synthesis data to the user terminal; and allowing the communication part to share content including the voice synthesis data.

Description

SYSTEM AND METHOD FOR PROVIDING TEXT-TO-SPEECH SERVICE BASED ON CELEBRITIES' VOICE

The present disclosure relates to a method and system for providing a speech synthesis service based on the voice of a famous person, and more particularly, to a method and system for providing a speech synthesis service based on the voice of a famous person, and more particularly, text information included in content about a specific person preferred by a famous person and a user using the voice of the specific person to provide voice information It relates to a method and system for providing a speech synthesis service that can be converted into .

In general, speech synthesis technology called text-to-speech (TTS) is used in applications that require human voice, such as announcements, navigation, and artificial intelligence assistants, without having to record the actual voice in advance. It is a technology used to reproduce voice. A typical method of speech synthesis is the concatenative TTS, in which speech is cut into very short units such as phonemes and stored in advance, and the phonemes constituting the sentence to be synthesized are combined to synthesize speech, and speech characteristics are expressed as parameters. There is a parametric TTS method in which parameters representing speech features constituting a sentence to be synthesized are synthesized into a voice corresponding to the sentence using a vocoder.

Meanwhile, with the recent increase in demand for electronic products that use voice data, such as artificial intelligence speakers, the demand for using voice-related electronic products through the voice of a specific person has also increased. To meet this demand, 'Personalized-Text To Speech (P-TTS)' is being developed, which uses the voice of a specific person, such as a popular celebrity or character, for speech synthesis technology. Personalized speech synthesis technology is a technology that synthesizes the voice of a specific person using voice data collected through speech synthesis deep learning (artificial neural network) technology. can do.

The embodiments disclosed herein convert text information included in content shared among users who prefer a specific person, such as a celebrity, into voice synthesis data using the specific person's voice data and reproduce it, thereby providing a lively and realistic experience. A method and system for providing a speech synthesis service capable of providing a speech synthesis service are provided.

A method of providing a voice synthesis service according to an embodiment of the present disclosure includes the steps of analyzing, by a voice analysis unit, voice data of a specific person, and building a voice data DB based on the analysis result; Receiving content generated for the selected specific person, receiving, by the communication unit, a text-to-speech request including information about the selected specific person and the generated content, by the speech synthesizing unit, according to the text-to-speech request, voice data Extracting the voice data of a specific person selected from the DB, converting text information included in the content into voice synthesized data based on the extracted voice data, and providing the voice synthesized data to the user terminal, and by the communication unit, the voice synthesized data and sharing content including

According to an embodiment, a method of providing a speech synthesis service includes, by the fee management unit, calculating the amount of speech synthesis data converted by the speech synthesis unit or the reproduction time of the speech synthesis data based on the text information, and by the fee management unit. , determining a fee based on at least one of the calculated amount of the synthesized voice data and the reproduction time of the synthesized voice data.

According to an embodiment, the method for providing a voice synthesis service further includes determining, by the rank management unit, a voice synthesis rank of a plurality of specific persons based on at least one of the determined rate and the amount of voice synthesis data.

According to an embodiment, a method of providing a voice synthesis service includes, by a voice analysis unit, determining a voice range of a specific person's voice based on the voice data of a specific person, and having a voice range similar to the voice range determined from among a plurality of specific people by the communication unit The method further includes transmitting the at least one specific person information to the user terminal.

According to an embodiment, the method of providing a voice synthesis service includes, by the crowdfunding unit, generating crowdfunding information for securing voice data of a specific person not included in the voice data DB, and providing the crowdfunding information to a user terminal. The method further includes the step of sharing, collecting investment information corresponding to the crowdfunding information from the user terminal, and providing a reward to the user terminal based on the investment information when the crowdfunding is successful.

According to an embodiment of the present disclosure, a voice synthesis service providing system analyzes voice data of a specific person and generates a voice analyzer configured to build a voice data DB based on the analysis result for a specific person selected by a user from a user terminal In accordance with the text-to-speech request and the communication unit configured to receive the selected content and receive the text-to-speech request including the selected specific person information and the generated content, the voice data of the selected specific person is extracted from the voice data DB and included in the content. and a speech synthesis unit configured to convert text information into speech synthesis data and provide the converted speech synthesis data to the user terminal, wherein the communication unit is further configured to share content including the speech synthesis data.

According to an embodiment, the system for providing a speech synthesis service calculates an amount of speech synthesis data converted by the speech synthesizer and a speech reproduction time based on text information, and at least among the calculated amount of speech synthesis data and a speech reproduction time and a fee management unit configured to determine the fee based on the one.

According to an embodiment, the voice synthesis service providing system further includes a rank management unit configured to determine the voice synthesis rank of a plurality of specific persons based on at least one of the determined rate and the amount of voice synthesis data.

According to an embodiment, the voice analysis unit is further configured to determine a voice range of a specific person's voice based on the specific person's voice data, and the communication unit, at least one specific person information having a sound range similar to the determined voice range among a plurality of specific person information to the user and further configured to transmit to the terminal.

According to an embodiment, the voice synthesis service providing system generates crowdfunding information for securing voice data of a specific person not included in the voice data DB, provides and shares the crowdfunding information to a user terminal, and from the user terminal It further includes a crowdfunding unit configured to collect investment information corresponding to the crowdfunding information, and to provide a reward to the user terminal based on the investment information when the crowdfunding is successful.

According to various embodiments of the present disclosure, by converting text information included in content shared among users who prefer a specific person, such as a celebrity, into speech synthesis data using the voice data of the specific person, users directly It is possible to provide a more lively and realistic voice synthesis service by providing content for a specific person as a sound source of the specific person.

According to various embodiments of the present disclosure, by receiving content shared by other users about a specific person, such as a celebrity, with the voice of the specific person, the user does not need to continuously invest time to obtain information about the specific person. , it is possible to intuitively understand information about the specific person through voice.

In addition, according to various embodiments of the present disclosure, users may be charged based on voice synthesis data converted from text of content shared with respect to a specific person such as a celebrity, and based on the charging result, You can decide the popularity or ranking. Accordingly, a new method of ranking celebrities according to the usage or usage of the speech synthesis service is provided. According to this method, users can further activate a fan community in which content about celebrities is shared.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numerals denote like elements, but are not limited thereto.
1 is a schematic diagram illustrating a system for providing a voice synthesis service using a voice synthesis service providing server according to an embodiment of the present disclosure.
2 is a block diagram illustrating an internal configuration of a voice synthesis service providing server according to an embodiment of the present disclosure.
3 is a structural diagram illustrating an artificial neural network model according to an embodiment of the present disclosure.
4 is a diagram illustrating an example of generating a text-to-speech request according to an embodiment of the present disclosure.
5 is a diagram illustrating an example of a screen for payment according to an embodiment of the present disclosure.
6 is a diagram illustrating an example of displaying a voice synthesis ranking on a screen of a user terminal according to an embodiment of the present disclosure.
7 is a diagram illustrating an example of displaying a crowdfunding service on a screen according to an embodiment of the present disclosure.
8 is a flowchart illustrating a method of providing a voice synthesis service according to an embodiment of the present disclosure.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the subject matter of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if descriptions regarding components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and those of ordinary skill in the art to which the present disclosure pertains. It is only provided to fully inform the person of the scope of the invention.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are selected as currently widely used general terms as possible while considering the functions in the present disclosure, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates that the singular is singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, 'module' or 'unit' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, may include at least one of procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or variables. Components and 'modules' or 'units' are the functions provided therein that are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configurations. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, a 'system' may include at least one of a server (or a server device) and a user terminal connected to the server through a network, but is not limited thereto. For example, the system may include one or more user terminals, one or more server devices, or a combination thereof.

In the present disclosure, 'voice data' is, for example, data representing a voice uttered by a person who has a user's preferred voice, such as a specific person or a famous person (eg, voice waveform signal data, voice extracted from voice waveform signal) characteristics, etc.). In addition, in the present disclosure, 'celebrities' may refer to celebrities preferred or widely known by the general public (eg, people engaged in entertainment such as singers and actors), academics, politicians, and the like.

1 is a schematic diagram illustrating the configuration of a system for providing a voice synthesis service using a voice synthesis service providing server 130 according to an embodiment of the present disclosure. In the illustrated system, the voice synthesis service providing server 130 may provide a voice synthesis service to the plurality of user terminals 112 , 114 , and 116 through the network 120 . According to an embodiment, the voice synthesis service providing server 130 may store, provide, and execute computer-executable programs (eg, downloadable applications) and data related to the voice synthesis service, and/or one or more server devices and/or It may include a database, or one or more distributed computing devices and/or distributed databases based on cloud computing services. Here, the voice synthesis service provided by the voice synthesis service providing server 130 may be provided to the user through a dedicated application or a web browser installed in the plurality of user terminals 112 , 114 , 116 . Here, the user may be a celebrity in the entertainment field, such as a singer or actor, or a fan who likes a celebrity in another field such as a politician or a scholar, but is not limited thereto.

The plurality of user terminals 112 , 114 , and 116 may communicate with the voice synthesis service providing server 130 through the network 120 . The network 120 may be configured to enable communication between the plurality of user terminals 112 , 114 , and 116 and the voice synthesis service providing server 130 . Network 120 according to the installation environment, for example, Ethernet (Ethernet), wired home network (Power Line Communication), telephone line communication device and wired networks such as RS-serial communication, mobile communication network, WLAN (Wireless LAN), It may consist of wireless networks such as Wi-Fi, Bluetooth and ZigBee, or a combination thereof.

Although the mobile phone terminal 112, the tablet terminal 114, and the PC terminal 116 are illustrated as examples of the user terminal in FIG. 1, the present invention is not limited thereto, and the user terminal is capable of wired and/or wireless communication and communicates with celebrities from the user. It may be any computing device having a user interface capable of receiving selection information from such a specific person, content information related to a celebrity, and a text-to-speech (or speech synthesis) request. For example, a user terminal is a smart phone, a mobile phone, a mobile phone, a navigation system, a computer, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, a game console (game console) , a wearable device, an Internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, and the like. In addition, in FIG. 1 , three user terminals 112 , 114 , and 116 are illustrated as communicating with the voice synthesis service providing server 130 through the network 120 , but the present invention is not limited thereto. It may be configured to communicate with the voice synthesis service providing server 130 through the network 120 .

The communication method between the user terminals 112, 114, 116 and the voice synthesis service providing server 130 is not limited, and the network 120 may include a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, Broadcast network, satellite network, etc.) as well as a communication method using a short-range wireless communication between user terminals may be included. For example, the network 120 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 120 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

The voice synthesis service providing server 130 receives the voice data of a plurality of specific people from an external device (eg, a voice data server that manages and stores voice data of a specific person such as a celebrity) through the network 120, The received voice data can be stored in a database. In the present disclosure, voice data may include voice data uttered by a specific person or voice characteristic data extracted therefrom, and voice parameter data (eg, voice pitch, voice length, tone, sound strength, etc.).

Also, the voice synthesis service providing server 130 may receive text-to-speech conversion requests from the plurality of user terminals 112 , 114 , and 116 through the network 120 , and store the received information in a database. Here, the text-to-speech request may include specific person information, such as a celebrity selected by the user, and content generated for the specific person selected by the user (eg, multimedia content such as messages, news, images or videos). The content generated for the specific person selected by the user may be at least one of a plurality of contents generated by the user (eg, fans of a celebrity, etc.).

In one embodiment, the voice synthesis service providing server 130 may be configured to receive and share content generated for a specific person, such as a celebrity selected by the user, via a social network. For example, the user A sends a plurality of contents about a specific celebrity (eg, a specific celebrity's article or schedule, a fan's message, etc.) (eg, fan page, etc.).

The voice synthesis service providing server 130 receives content generated for a celebrity uploaded through a social network, and shares the received content through, for example, a bulletin board or a fan page for a specific celebrity provided from the voice synthesis service. can do. Alternatively, user A may upload a plurality of contents related to a specific celebrity to a bulletin board (eg, a bulletin board for a specific celebrity, etc.) provided from the speech synthesis service and share it with other users.

User B may select at least one of the contents for a specific celebrity shared through the speech synthesis service providing server 130 and use the speech synthesis service for converting the text of the selected contents into the voice of the specific person. Alternatively, the user B creates content about a celebrity and shares it through a bulletin board provided from a social network service and/or speech synthesis service, selects the shared content, and converts the text of the selected content into the voice of a specific person. A voice synthesis service may also be used.

The text-to-speech request described above is not limited to the specific person information selected by the user and the content for the specific person, and is not limited to user information, that is, text-to-speech information such as information about a fan of a specific person who has requested text-to-speech conversion, text information for voice conversion, etc. Various information related to the conversion request may be further included.

The voice synthesis service providing server 130 may generate and provide voice synthesis data to the user terminal based on text information included in the content generated for the celebrity selected by the user, and determine a fee for the generated voice synthesis data. Thus, it can be provided to the user terminal. Additionally, the voice synthesis service providing server 130 may provide a wake-up call and/or birthday message service using voice synthesis data of a specific person selected by the user.

As such, the speech synthesis service server 130 converts text information included in content shared among fans of a specific celebrity into speech synthesis data using the celebrity's voice data, thereby generating content about the celebrity directly created by fans. It is possible to provide a more lively and realistic voice synthesis service by playing and providing the voice of the famous person.

In addition, by receiving content shared by other users about a specific celebrity by playing it with the celebrity (or specific person)'s voice, users do not need to continuously invest time to obtain information about a specific celebrity. It is possible to intuitively obtain information about a specific celebrity through

A process in which the voice synthesis service providing server 130 provides the voice synthesis service in response to the text-to-speech request received will be described in detail below with reference to FIGS. 2 to 8 .

2 is a block diagram illustrating an internal configuration of a voice synthesis service providing server 200 according to an embodiment of the present disclosure. As shown in FIG. 2 , the voice synthesis service providing server 200 may include a communication unit 210 , a processor 230 , and a database 220 . The processor 230 may include a voice analysis unit 232 , a voice synthesis unit 234 , a rate management unit 236 , a rank management unit 238 , and a crowdfunding unit 240 . Also, the database 220 may include a user DB 222 and a voice data DB 224 . In FIG. 2 , the processor 230 includes a plurality of sub-modules and the database 220 includes a plurality of sub-DBs, but the present invention is not limited thereto. may be configured.

The communication unit 210 may be configured to enable the voice synthesis service providing server 200 to transmit and receive arbitrary information or data to and from a user terminal and/or an external device (eg, a voice data server, etc.) through a network. . The processor 230 may be configured to receive and process arbitrary information through the communication unit 210 . The database 220 may be configured to store information/data received from a user terminal and/or an external device, and to store information generated by the processor 230 .

In an embodiment, the communication unit 210 may receive voice data of a specific person, such as a celebrity, from a voice data server and transmit the received voice data to the processor 230 . Here, the voice data may include various information related to the voice of a celebrity received from the voice data server. For example, the voice data may include a celebrity's name - "Kim xx", a field - "actor", voice data including voice characteristics of a celebrity, and the like. In addition, the voice data may include voice spectrum data representing information related to sequential prosody features, speech characteristics of a speaker, and the like.

Also, the communication unit 210 may receive a text-to-speech request from a user terminal using a speech synthesis service (eg, a user terminal of a celebrity fan) and transmit the text-to-speech request to the processor 230 . Here, the text-to-speech request may include specific person information such as a celebrity selected by the user and content generated for the specific person selected by the user. As described above in FIG. 1 , the content generated for a specific person selected by the user may be at least one of a plurality of contents created by the user (eg, fans of the corresponding celebrity), but is not limited thereto.

The text-to-speech request is not limited to information about a specific person, such as a celebrity, or content generated for a specific person. According to an embodiment, the text-to-speech request may further include user information. Here, the user information may include, for example, various information related to the user, such as the user's personal information, gender, the user's residence or origin region, nationality, and the like.

The processor 230 may store information and/or data received from the communication unit 210 in the database 220 . According to an embodiment, the database 220 may be converted into a standardized notation and stored in order to efficiently manage the received information and/or data.

The database 220 may include a user DB 222 in which user-related information is classified and stored for each user. According to an embodiment, the user DB 222 may be constructed as an arbitrary data structure in which a user name, gender, residence region, nationality, text-to-speech request requested by a user, and the like are stored in association with each other.

Also, the database 220 may further include a voice data DB 224 in which specific person information such as a celebrity and voice data of a specific person are classified and stored for each specific person. According to an embodiment, the voice data DB 224 includes a specific person's name, a specific person's field of activity (eg, a singer, an actor, a politician, a scholar, etc.), a specific person's activity name, the specific person's voice data, the specific person's voice analysis result, etc. It can be built as any data structure that is stored in association with each other.

In FIG. 2 , the database 220 is illustrated to be included in the voice synthesis service providing server 130 , but the present invention is not limited thereto. can be saved.

The voice analysis unit 232 of the processor 230 may be configured to analyze the received voice data of a specific person. According to an embodiment, the voice analyzer 232 may extract sequential prosody features and/or the speaker's vocalization features based on voice data of a specific person received from an external device. Here, the received voice data may include voice spectrum data representing information related to sequential prosody features, for example, a melody and voice features of a specific speaker. The sequential prosody characteristic may include prosody information of each time unit according to a predetermined time unit, and the speaker's vocalization characteristic not only imitates the speaker's voice, but also the style, rhyme, and emotion that can compose the vocalization. , tone, pitch, and the like may include at least one of various factors.

The voice analyzer 232 may build the voice data DB 224 based on the analysis result. According to an embodiment, the voice analyzer 232 may build and manage the voice data DB 224 by storing sequential prosody characteristics, the speaker's vocalization characteristics, the name of a specific person, and the field of activity of the specific person in association with each other.

Additionally or alternatively, the voice analyzer 232 may be configured to analyze voice data of a celebrity received from an external device using a known voice data analysis technique.

The voice analyzer 232 may be configured to recommend another specific person (eg, another celebrity) having voice characteristics similar to the specific person selected by the user. According to an embodiment, the voice analyzer 232 may determine the sound range of the specific person's voice based on the specific person's voice data. For example, the voice analyzer 232 may receive the voice data of the specific person selected by the user from the voice data DB to identify the frequency of the voice data of the specific person, and use the identified frequency to identify the voice of the specific person selected by the user. can determine the range of The determined specific person's voice range may be matched with the specific person's voice data stored in the voice data DB 224 and stored.

The voice analyzer 232 may recommend another specific person (or celebrity) having voice characteristics similar to the specific person preferred by the user by providing at least one of the specific person having the same or similar sound range to the determined sound range to the user terminal. For example, when the sound range of a specific person selected by the user is determined to be 90 to 120 Hz, the voice analysis unit 232 selects at least one of the other specific persons whose sound range is determined to be 90 to 120 Hz from the voice data DB 224 as a specific person having a similar voice. can be recommended as

The speech synthesis unit 234 may be configured to generate speech synthesis data based on text information included in content. The speech synthesis unit 234 is configured to perform articulatory synthesis, formant synthesis, concatenative synthesis, statistical parametric speech synthesis, and deep learning (or artificial neural network) based. Text information may be converted into speech synthesis data using various speech synthesis methods including speech synthesis.

According to an embodiment, the voice synthesizer 234 may analyze prosody information from text information. Also, the voice synthesizer 234 may generate voice synthesis data by synthesizing voice data in the form of a phoneme-based voice waveform extracted from the sound source DB 224 based on the prosody information.

According to another embodiment, the speech synthesis unit 234 inputs text information, sequential prosody characteristics, and speaker's vocalization characteristics into a text-to-speech (TTS) model based on an artificial neural network to obtain speech synthesis data. can create The speech synthesis data may include speech data for a text in which sequential prosody features and speaker's vocal features are reflected.

The voice synthesizer 234 may provide the generated voice synthesis data to the user terminal.

The fee management unit 236 may be configured to determine a fee for the speech synthesis data generated by the speech synthesis unit 234 . According to an embodiment, the charge management unit 236 calculates the amount of speech synthesis data and the audio reproduction time generated by the speech synthesis unit 234 , and sets the amount to at least one of the calculated amount of speech synthesis data and the audio reproduction time. Fees can be determined based on For example, the fee management unit 236 may calculate a fee in proportion to the amount of speech synthesis data converted from text included in the content selected by the user. Additionally or alternatively, the fee management unit 236 may calculate a fee in proportion to the calculated voice reproduction time.

According to another embodiment, the fee management unit 236 may calculate the fee by applying a weight to each of the calculated amount of speech synthesis data and the speech reproduction time. Additionally, the fee management unit 236 determines the fee by calculating the final fee by applying a weight to each of the copyright fee for the voice data of a specific person providing the voice data and other expenses related to the specific person (eg, entertainer's appearance fee). can

When the fee determined by the fee management unit 236 according to the above-described process is paid by the user, a portion of the paid amount may be paid to the corresponding celebrity.

The rank management unit 238 may be configured to determine the voice synthesis rank of a specific person. According to an embodiment, the rank management unit 238 may determine the voice synthesis ranking based on at least one of the rate determined by the rate management unit 236 and the amount of voice synthesis data. For example, the rank management unit 238 may determine a higher voice synthesis ranking as the cumulative sum of the fees paid by the user is determined by the rate management unit 236 . Additionally or alternatively, the rank management unit 238 may determine a higher voice synthesis ranking as the accumulated amount of voice synthesis data generated from the text-to-speech conversion request increases.

According to another embodiment, the rank management unit 238 may determine the rank of the specific person by applying a weight to each of the accumulated charges for each specific person and the accumulated amount of speech synthesis data.

The crowdfunding unit 240 may be configured to provide a crowdfunding service for voice data of a specific person not included in the voice data DB 224 . According to an embodiment, when a specific person not included in the voice data DB 224 exists, the crowdfunding unit 240 generates funding information related to the voice data of a celebrity not included in the voice data DB 224 and , can be shared by providing the generated funding information to the user terminal. After the funding information is provided to the user terminal, investment information corresponding to the funding information may be collected from the user terminal, and when crowdfunding is successful, a reward may be provided to the user terminal based on the investment information. According to an embodiment, the crowdfunding unit 240 may determine a reward in proportion to the investment amount and provide the determined reward to the user terminal participating in the corresponding crowdfunding. A process in which the crowdfunding service is provided by the crowdfunding unit 240 will be described in detail below with reference to FIG. 7 .

According to an embodiment, the processor 230 may be configured to further include a statistic unit (not shown). The statistics unit may be configured to generate statistical information based on the text-to-speech request. In this case, the text-to-speech request may include, for example, user information such as a user's name, gender, region of residence or origin, nationality, etc., information on a specific person selected by the user, and content generated for the specific person.

According to an embodiment, the statistic unit may generate statistical information based on user information. For example, when a text-to-speech conversion request is made, the statistics unit generates statistical information based on the residential area or origin of users who select the same specific person as the user, and transmits the generated statistical information to the user terminal in the form of a pie graph to intuitively grasp the generated statistical information. You can print it out and provide it to users.

According to an embodiment, the statistic unit may generate statistical information based on the text-to-speech request. For example, the statistics unit may generate statistical information about celebrities selected by other users, and provide the generated statistical information to the user terminal. In this case, other users may refer to a user who has selected the same celebrity as the user when requesting text-to-speech.

3 is a structural diagram illustrating an artificial neural network model 300 according to an embodiment of the present disclosure. According to an embodiment, the system for providing a voice synthesis service may further include a learning unit. The learning unit may be configured to generate a recommendation model through a machine learning algorithm based on a plurality of training data extracted from voice data of a specific person, such as a celebrity, and to train the generated recommendation model. Here, the specific person's voice data may further include the specific person's voice range determined through the voice analysis unit 232 . The sound range for a specific person's voice may be identified based on the specific person's voice information received from an external device (eg, a voice data server, etc.) through the voice analysis unit, and the frequency of the specific person's voice may be identified and determined through the identified frequency. have.

According to an embodiment, the voice analysis unit may be configured to input the voice data of a specific person related to the content selected by the user into the learned recommendation model, and output information of another specific person having voice characteristics similar to the specific person selected by the user.

The learning and inference process of the artificial neural network executed by the learning unit and/or the voice analysis unit will be described in more detail below.

For example, the recommendation model may be an artificial neural network model 300 generated and trained through a machine learning algorithm. The artificial neural network model 300 is a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm according to a machine learning technique. According to an embodiment, the artificial neural network model 300 is an artificial neuron that forms a network by combining synapses, as in a biological neural network, by repeatedly adjusting the weights of synapses, so that the By learning to reduce the error between the output and the inferred output, it is possible to represent a machine learning model with problem-solving ability. For example, the artificial neural network model 300 may include arbitrary probabilistic models, neural network models, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

The artificial neural network model 300 is implemented as a multilayer perceptron (MLP) consisting of multilayer nodes and connections between them. The artificial neural network model 300 according to the present embodiment may be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 3 , the artificial neural network model 300 includes an input layer 320 that receives an input signal or data 310 from the outside, and an output layer that outputs an output signal or data 350 corresponding to the input data. 340 , located between the input layer 320 and the output layer 340 , receiving a signal from the input layer 320 , extracting characteristics, and transferring the characteristics to the output layer 340 , where n is a positive integer. It is composed of hidden layers 330_1 to 330_n. Here, the output layer 340 receives signals from the hidden layers 330_1 to 330_n and outputs them to the outside.

The learning method of the artificial neural network model 300 includes a supervised learning method that learns to be optimized to solve a problem by input of a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. ) is a way. The learning unit analyzes a plurality of learning data extracted from the voice data of the specific person through supervised learning in order to recommend another specific person having similar characteristics using the voice data of the specific person selected by the user, The artificial neural network model 300 may be trained so that a plurality of training data for other specific information corresponding to a plurality of training data extracted from the data may be extracted. The artificial neural network model 300 learned in this way may be provided or accessed to the voice analyzer, and may output information about other specific persons having similar voice characteristics in response to a text-to-speech request received from the user terminal.

According to an embodiment, as shown in FIG. 3 , the input variable of the artificial neural network model 300 from which other specific information having similar voice characteristics can be extracted may be voice data of a specific person selected by the user. For example, the input variable input to the input layer 320 of the artificial neural network model 300 is an input in which voice data (eg, a specific person's name, activity field, sound range, etc.) is composed of one vector data element. It may be a vector 310 . In response to the input from the voice data, the output variable output from the output layer 340 of the artificial neural network model 300 may be a vector 350 representing information about another person having a similar voice characteristic. In addition, the output layer 340 of the artificial neural network model 300 may be configured to output a vector indicating the reliability or accuracy of the outputted other specific person information having similar speech characteristics. For example, a vector representing confidence or accuracy for such an output may be interpreted or displayed as a score. In the present disclosure, the output variables of the artificial neural network model 300 are not limited to the types described above, and may be expressed in various forms related to similar specific person information and similar specific person information reliability. Here, weight values between nodes included in the input layer 320 , the hidden layers 330_1 to 330_n , and the output layer 340 may be adjusted so that a correct output corresponding to a specific input may be extracted.

As described above, the input layer 320 and the output layer 340 of the artificial neural network model 300 are matched with a plurality of output variables corresponding to the plurality of input variables, respectively, and the input layer 320, the hidden layers 330_1 to 330_n, and the output layer By adjusting the synaptic value between the nodes included in 340, it can be learned so that a correct output corresponding to a specific input can be extracted. Through this learning process, characteristics hidden in the input variable of the artificial neural network model 300 can be identified, and the error between the output variable calculated based on the input variable and the target output is reduced. You can adjust the synapse value (or weight) between them. Using the artificial neural network model 300 learned in this way, in response to the input specific person information, similar specific person information may be output.

4 is a diagram illustrating an example of generating a text-to-speech request according to an embodiment of the present disclosure. The user may select at least one of the shared contents. According to an embodiment, the content may be shared with users using a bulletin board provided from a voice synthesis service.

For example, the user may access a bulletin board related to a specific person (eg, a famous person such as a singer, actor, etc.) by entering a search keyword "Kim XX" in a user search interface (not shown). As shown in FIG. 4 , the content shared for the actor “Kim XX” may be displayed on the screen and provided to the user. When the user selects one of the contents displayed on the screen through a touch input or the like, detailed contents of the corresponding contents may be displayed on the screen. The user may select the text-to-speech button 420 through a touch input or the like to generate a text-to-speech request for the corresponding content, and may move to a screen for payment (see FIG. 5 ).

Meanwhile, the user may not display the displayed content on the screen by selecting the view other text button 410 through a touch input or the like, or may display content other than the selected content on the screen.

According to another embodiment, the user may create and share content for a specific person desired through the user interface displayed on the user terminal through the voice synthesis service, and may select the created content.

5 is a diagram illustrating an example of a screen for payment according to an embodiment of the present disclosure. Information on a fee determined based on text information included in content selected by the user may be displayed on the user terminal. For example, information on the voice synthesis data generated by the sound source synthesizer, the reproduction time of the generated voice synthesis data, the amount of data, and the fee may be displayed. As shown in FIG. 5 , the information on the voice synthesis data generated by the sound source synthesizer includes the name of the specific person and the field of activity, “actor Kim XX”, the playback time is “50 seconds”, and the amount of data is “800 KB”. , the fee may be displayed as "1,000 won". The user may cancel the payment by selecting the cancel button 510 through a touch input or the like, or may select the payment button 520 to complete the payment.

When payment is completed, the voice synthesized data generated by the voice synthesizer may be provided to the user. According to an embodiment, the synthesized voice data may be automatically reproduced through a speaker of the user terminal, and text included in the content may be narrated by the voice of a specific person.

According to an embodiment, when the payment is completed, information (not shown) on another specific person having characteristics similar to the specific person selected by the user may be displayed on the user terminal to recommend a similar specific person to the user. Alternatively or additionally, statistical information (not shown) on the specific person used by other users who have selected the same specific person as the user may be displayed.

6 is a diagram illustrating an example of displaying a voice synthesis ranking on a screen of a user terminal according to an embodiment of the present disclosure. The voice synthesis ranking determined for each specific person, such as a celebrity, may be displayed on the screen of the user terminal. According to an embodiment, the determined voice synthesis ranking may be provided in the form of a chart.

For example, the voice synthesis rankings may be arranged in ascending order and displayed on the screen of the user terminal. As shown, the determined ranking of the specific person, the profile picture of the specific person, the specific person's name, and activity field information may be displayed together. In this case, the increase/decrease of the rank may be displayed together in the area where the rank is displayed. In addition, the determined voice synthesis ranking may be displayed on the screen of the user terminal by dividing it according to the activity field of a specific person, such as, for example, an actor, a model, an idol, and the like. Additionally, it may be displayed on the screen divided according to the gender of a specific person.

On the other hand, it is also possible to provide the user with information about a specific person whose sound source ranking is rising in real time. According to an embodiment, the rank management unit may display information on a specific person whose voice synthesis rank has risen the most during a predetermined range of time on the screen and provide it to the user.

7 is a diagram illustrating an example of displaying a crowdfunding service on a screen according to an embodiment of the present disclosure. According to an embodiment, when text-to-speech conversion is requested for a specific person whose voice data is not included in the voice data DB, the screen may be moved to a screen for crowdfunding. The screen for crowdfunding may display ongoing funding information as shown. Here, the funding information may include the name of a specific person who is the funding target, the funding progress, the current amount remaining until the funding is successful, the remaining time until the funding is completed, and information on the rewards provided to users who invest in the corresponding crowdfunding. It may include. The user may select the crowdfunding participation button 710 through a touch input or the like to invest in the corresponding crowdfunding.

The user may request crowdfunding for a specific person whose sound source information is not included in the voice data DB by selecting the crowdfunding request button 720 through a touch input or the like. When the user selects the crowdfunding request button 720, the screen of the user terminal may be moved to a screen for generating a crowdfunding request, and an interface (not shown) for generating a crowdfunding request may be displayed. A user may generate crowdfunding for a specific person by inputting funding information through an interface (not shown). In this case, the crowdfunding target amount and the crowdfunding deadline may be automatically determined by the crowdfunding unit. The generated crowdfunding may be provided and shared with other user terminals so that the target amount can be achieved.

8 is a flowchart illustrating a method 800 for providing a voice synthesis service according to an embodiment of the present disclosure. The voice synthesis service providing method 800 may be performed by the voice synthesis service providing servers 130 and 200 of FIGS. 1 and 2 . As shown, the method 800 for providing a voice synthesis service may be initiated by the voice analyzer analyzing voice data of a specific person and building a voice data DB based on the analysis result ( S810 ).

According to an embodiment, the voice analyzer may extract sequential prosody features and/or the speaker's vocalization features based on voice data of a specific person received from an external device. The voice analyzer may build the voice data DB based on the analysis result (eg, sequential prosody characteristics, speaker's vocal characteristics, etc.). Here, the received voice data may include voice spectrum data representing information related to sequential prosody features, for example, a melody and voice features of a specific speaker. The sequential prosody characteristic may include prosody information of each time unit according to a predetermined time unit, and the speaker's vocalization characteristic not only imitates the speaker's voice, but also the style, rhyme, and emotion that can compose the vocalization. , tone, pitch, and the like may include at least one of various factors.

According to an embodiment, the voice analyzer determines the voice range of the specific person's voice based on the received voice data of the specific person, and provides at least one information of the specific person having the same or similar voice range to the determined voice range to the user terminal. .

Next, content generated for a specific person selected by the user may be received from the user terminal by the communication unit (S820). In an embodiment, the communication unit may receive content generated for a specific person selected by the user through a network. For example, the communication unit may receive content generated for a specific person uploaded through a social network.

Next, a text-to-speech request including information on a specific person selected from the user and the generated content may be received by the communication unit (S830). Here, the text-to-speech request may include information about the specific person selected by the user and content generated for the specific person selected by the user.

When a text-to-speech conversion is requested for a specific person whose voice data is not included in the voice data DB, the crowdfunding unit generates funding information related to the voice data of a specific person not included in the voice data DB and transmits the generated funding information to the user terminal It can be provided and shared, and investment information corresponding to the funding information can be collected from the user terminal. When the crowdfunding is successful by achieving the funding target amount, the crowdfunding unit may provide a reward to the user terminal based on the investment information.

Then, the voice synthesizer extracts voice data of a specific person selected from the voice data DB according to the text-to-speech conversion request, and converts text information included in the content into voice synthesis data based on the extracted voice data, The synthesized data may be provided to the user terminal (S840). According to an embodiment, the speech synthesis unit may generate speech synthesis data by inputting text information, sequential prosody characteristics, and speaker vocalization characteristics into an artificial neural network text-to-speech synthesis model. The speech synthesis data may include speech data for a text in which sequential prosody features and speaker's vocal features are reflected.

A fee for the converted speech synthesis data may be determined by the fee management unit. According to an embodiment, the charge management unit calculates the amount of speech synthesis data converted by the speech synthesis unit or the reproduction time of the speech synthesis data based on the text information, and reproduces the calculated amount of speech synthesis data and the speech synthesis data. The rate may be determined based on at least one of time.

Finally, content including voice synthesis data may be shared by the communication unit (S850). According to an embodiment, the communication unit may share content including voice synthesis data through a social network through, for example, a bulletin board for a specific celebrity or a fan page provided from a voice synthesis service.

The above-described method for providing a speech synthesis service may be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium is distributed in a computer system connected through a network, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the above-described embodiments can be easily inferred by programmers in the art to which the present invention pertains.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques may include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer-readable media such as programmable read-only memory, erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

If implemented in software, the techniques may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer readable medium may contain RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or desired program code in the form of instructions or data structures. may include any other medium that can be used for transport or storage to a computer and can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and microwave, the coaxial cable , fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disk and disk include CD, laser disk, optical disk, digital versatile disc (DVD), floppy disk, and Blu-ray disk, where disks are usually magnetic. Data is reproduced optically, while discs reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from, or write information to, the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may reside within the ASIC. The ASIC may exist in the user terminal. Alternatively, the processor and the storage medium may exist as separate components in the user terminal.

Although the embodiments described above have been described utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not so limited and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

112, 114, 116: user terminal
120: network
130, 200: Speech synthesis service providing server
210: communication unit
220: database
222: User DB
224: voice data DB
230: processor
232: voice analysis unit
234: speech synthesis unit
236: fee management department
238: Rank Management Department
240: crowdfunding unit
300: artificial neural network model
310: input signal or data
320: input layer
330_1 to 330_n: hidden layer
340: output layer
350: output signal or data

Claims (10)

A method for providing a speech synthesis service, the method comprising:
analyzing, by the voice analyzer, voice data of a specific person, and constructing a voice data DB based on the analysis result;
receiving, by the communication unit, content generated for a specific person selected by the user from the user terminal;
Receiving, by the communication unit, a text-to-speech request including the selected specific person information and the generated content
The voice synthesizer extracts voice data of the selected specific person from the voice data DB according to the text-to-speech conversion request, and converts text information included in the content into voice synthesis data based on the extracted voice data providing the synthesized speech data to the user terminal; and
sharing, by the communication unit, content including the speech synthesis data;
A method of providing a speech synthesis service, comprising:
According to claim 1,
calculating, by the rate management unit, the amount of the speech synthesis data converted by the speech synthesis unit or a reproduction time of the speech synthesis data based on the text information; and
and determining, by the fee management unit, a fee based on at least one of the calculated amount of the speech synthesis data and a reproduction time of the speech synthesis data.
3. The method of claim 2,
The method further comprising the step of determining, by the rank management unit, the voice synthesis rank of a plurality of specific persons based on at least one of the determined rate and the amount of the voice synthesis data.
According to claim 1,
determining, by the voice analyzer, a sound range of the specific person's voice based on the specific person's voice data; and
The method further comprising the step of transmitting, by the communication unit, information on at least one specific person having a sound range similar to the determined sound range among a plurality of specific persons to the user terminal.
According to claim 1,
generating, by the crowdfunding unit, crowdfunding information for securing voice data of a specific person not included in the voice data DB;
providing and sharing the crowdfunding information to the user terminal;
collecting investment information corresponding to the crowdfunding information from the user terminal; and
When the crowdfunding is successful, the method further comprising the step of providing a reward to the user terminal based on the investment information.
A voice synthesis service providing system, comprising:
a voice analyzer configured to analyze the voice data of a specific person and build a voice data DB based on the analysis result;
a communication unit configured to receive content generated for a specific person selected by the user from the user terminal, and receive a text-to-speech request including information on the selected specific person and the generated content; and
According to the text-to-speech conversion request, the voice data of the selected specific person is extracted from the voice data DB, text information included in the content is converted into voice synthesis data, and the converted voice synthesis data is provided to the user terminal A voice synthesizer configured to
The communication unit is further configured to share content including the speech synthesis data.
7. The method of claim 6,
calculate an amount of speech synthesis data and a speech reproduction time converted by the speech synthesis unit based on the text information, and determine a fee based on at least one of the calculated amount of speech synthesis data and the speech reproduction time; A voice synthesis service providing system further comprising a configured fee management unit.
8. The method of claim 7,
and a ranking management unit configured to determine the voice synthesis rankings of a plurality of specific persons based on at least one of the determined rate and the amount of the voice synthesis data.
7. The method of claim 6,
The voice analysis unit is further configured to determine a sound range of the specific person's voice based on the specific person's voice data,
The communication unit is further configured to transmit, to the user terminal, information on at least one specific person having a sound range similar to the determined sound range among a plurality of specific persons.
7. The method of claim 6,
generating crowdfunding information for securing voice data of a specific person not included in the voice data DB;
Sharing the crowdfunding information by providing it to the user terminal,
Collecting investment information corresponding to the crowdfunding information from the user terminal,
When the crowdfunding is successful, the speech synthesis service providing system further comprises a crowdfunding unit configured to provide a reward to the user terminal based on the investment information.

Images