KR102418465B1 - Server, method and computer program for providing voice reading service of story book - Google Patents

Abstract

The server providing the moving picture reading service selects any one moving picture content to be provided as the moving picture reading service from among a registration unit for registering user voice data uttered by the user from the user terminal, and a plurality of moving picture contents previously registered from the user terminal. A receiving selection unit, a moving image content analysis unit that analyzes a plurality of sentences included in the selected moving image content, and tags onomatopoeic information for at least one onomatopoeia included in the analyzed plurality of sentences; An onomatopoeic sound source selection unit that selects any one onomatopoeic sound source among a plurality of onomatopoeic sound sources stored in advance based on one onomatopoeic word, a fairy tale reading sound source generating a fairy tale reading sound source by synthesizing the selected onomatopoeic sound source and the registered user voice data and a providing unit for providing the generated moving picture reading sound source to the user terminal, wherein the moving picture reading sound source is generated such that the at least one onomatopoeic word is expressed based on the user's voice.

Description

Server, method and computer program providing fairy tale reading service {SERVER, METHOD AND COMPUTER PROGRAM FOR PROVIDING VOICE READING SERVICE OF STORY BOOK}

The present invention relates to a server, a method and a computer program for providing a fairy tale reading service.

An intelligent personal assistant is a software agent that processes tasks requested by users and provides specialized services to users. The intelligent personal assistant collects and provides customized information to the user based on artificial intelligence (AI) engine and voice recognition, and performs various functions such as scheduling, sending e-mail, and restaurant reservation according to the user's voice command. It has the advantage of improving convenience.

These intelligent personal assistants are mainly provided in the form of customized personal services on smartphones, and representatively include Apple's Siri, Google's Now, and Samsung's Bixby. In this regard, Korean Patent Laid-Open No. 2016-0071111, which is a prior art, discloses a method of providing a personal assistant service in an electronic device.

Recently, it has become possible to receive a fairy tale reading service based on the user's voice through an intelligent personal assistant. However, when a fairy tale reading service is provided based on the user's voice, it has a disadvantage that the onomatopoeia of various animals is not well expressed.

In order to express the onomatopoeia included in the fairy tale content based on the user's voice, onomatopoeia information is tagged for at least one onomatopoeia included in the fairy tale content, an onomatopoeic sound source is selected based on the tagged onomatopoeia information, and the selected onomatopoeic sound source and user voice An object of the present invention is to provide a server, a method, and a computer program that provide a fairy tale reading service for synthesizing data.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention provides a registration unit for registering user voice data uttered by a user from a user terminal, and reading the moving picture among a plurality of moving image contents previously registered from the user terminal. A moving picture that selects any one moving image content to be provided as a service is selected, analyzes a plurality of sentences included in the selected moving image content, and tags onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences A content analysis unit, an onomatopoeic sound source selection unit that selects any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on at least one onomatopoeia tagged with the onomatopoeic information, the selected onomatopoeia sound source and the registered user voice data are synthesized A moving picture reading sound source generating unit for generating a moving picture reading sound source and a providing unit for providing the generated moving picture reading sound source to the user terminal, wherein the moving picture reading sound source includes the at least one onomatopoeia based on the user's voice. It is possible to provide a fairy tale reading service providing server that is generated to be expressed.

Another embodiment of the present invention includes the steps of registering user voice data uttered by a user from a user terminal, and selecting any one moving image content to be provided as the moving image reading service from among a plurality of moving image contents previously registered from the user terminal. receiving, analyzing a plurality of sentences included in the selected moving image content, tagging onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences, at least one onomatopoeia in which the onomatopoeia information is tagged Selecting any one of the onomatopoeic sound sources among a plurality of pre-stored onomatopoeia sound sources based on It is possible to provide a method for providing a moving picture reading service, including providing the sound source to a user terminal, wherein the at least one onomatopoeia is generated to be expressed based on the user's voice.

In another embodiment of the present invention, when a computer program is executed by a computing device, user voice data uttered by a user is registered from a user terminal, and the moving picture reading service among a plurality of moving picture contents previously registered from the user terminal receiving a selection of any one moving image content to be provided as , analyzing a plurality of sentences included in the selected moving image content, tagging onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences, and the onomatopoeia information Selects any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on the at least one onomatopoeia tagged, synthesizes the selected onomatopoeia sound source and the registered user voice data to generate a moving picture reading sound source, and the generated A moving picture reading sound source is provided to the user terminal, wherein the moving picture reading sound source is a computer program stored in a medium including a sequence of instructions to generate the at least one onomatopoeia to be expressed based on the user's voice. have.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, conventionally, the user imitates animal onomatopoeia, thereby having a disadvantage in that there is awkwardness in expressing animal onomatopoeia. It is possible to provide a server, a method, and a computer program that provide a realistic fairy tale reading service by synthesizing the onomatopoeic part of .

A fairy tale reading service that allows users to record and register their own voice and onomatopoeia of their dog, and receive a fairy tale reading service through a fairy tale reading sound source in which their own voice and onomatopoeia of their dog are synthesized. Servers, methods and computer programs for providing may be provided.

1 is a block diagram of a moving picture reading service providing server according to an embodiment of the present invention.
2A and 2B are exemplary views for explaining a process of registering user voice data and animal onomatopoeia sound sources according to an embodiment of the present invention.
3A and 3B are exemplary views for explaining a process of learning a moving picture content and a user's voice according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a process of tagging onomatopoeic information by analyzing a plurality of sentences included in a moving picture content according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a selected onomatopoeic sound source according to an embodiment of the present invention.
6A and 6B are exemplary views for explaining a process of extracting a similar onomatopoeic sound source when an onomatopoeic sound source associated with an object type corresponding to an onomatopoeic word is not extracted according to an embodiment of the present invention.
7 is an exemplary view for explaining a process of synthesizing an onomatopoeic sound source and user voice data according to an embodiment of the present invention.
8 is an exemplary diagram illustrating a spectrum for each of an existing model, an animal real animal sound, and an onomatopoeia based on a user's voice according to an embodiment of the present invention.
9 is a flowchart of a method for providing a moving picture reading service in a server according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a moving picture reading service providing server according to an embodiment of the present invention. Referring to FIG. 1 , the moving picture reading service providing server 100 includes a registration unit 110 , a selection unit 120 , a learning unit 130 , a moving picture content analysis unit 140 , a sound source selection unit 150 , and a moving picture reading sound source. It may include a generating unit 160 and a providing unit 170 .

The registration unit 110 may receive user voice data uttered by the user registered by the user terminal. In addition, the registration unit 110 may receive the animal onomatopoeia sound source recorded from the user terminal. The process of registering user voice data and animal onomatopoeia sound sources will be described in detail with reference to FIGS. 2A and 2B .

2A and 2B are exemplary views for explaining a process of registering user voice data and animal onomatopoeia sound sources according to an embodiment of the present invention.

Referring to FIG. 2A , the registration unit 110 may register the user's own voice who wants to read a fairy tale from the user terminal through the voice registration section 200 . For example, the registration unit 110 may include the first user voice data 201 (SPK1) uttered in a recitative form by the user based on a preset script in which the aloud utterance and the dialogue utterance are specified from the user terminal and the second uttered utterance in a dialogue form. At least one of the user voice data 202 and SPK2 may be registered 204 and stored in the database. For another example, the registration unit 110 may register 204 other user's voice data in which another user's voice is recorded through the add button 203 and store it in the database.

The registration unit 110 may receive an animal onomatopoeia sound source in which onomatopoeia are recorded from the user terminal through the onomatopoeia registration section 210 . For example, if the user wants to imitate the onomatopoeic of the cat 211, the registration unit 110 may register 214 the cat onomatopoeia sound source in which the cry of the cat 211 is recorded from the user terminal and store it in the database. . For another example, if the user is raising a dog 212, the registration unit 110 receives the registration 215 of the dog onomatopoeic sound source in which the barking sound of the dog 212 raised by the user is recorded from the user terminal and can be stored in the database. have. In addition, the registration unit 110 may receive the onomatopoeic sound source of various animals from the user terminal through the add button 213 and store it in the database.

Referring to FIG. 2B , the registration unit 110 may receive a selection of user voice data to be applied to a moving picture reading service from among a plurality of voice data stored in the database from the user terminal through the voice selection section 220 . For example, the registration unit 110 receives the selection of the first user voice data 201 and SPK1 uttered in a read out of a plurality of user voice data stored in the database for the narration item 221 from the user terminal, and the conversation item 222 ) from among a plurality of user voice data stored in the database, the second user voice data 202 (SPK2) uttered in a conversational language may be selected. For another example, the registration unit 110 receives, from the user terminal, its own voice data uttered in a read out of a plurality of voice data stored in the database for the narration item 221 , and is stored in the database for the conversation item 222 . Another user's voice data uttered in a conversational language may be selected from among the plurality of voice data.

The registration unit 110 may receive any one onomatopoeic sound source registered by the user terminal among a plurality of animal onomatopoeia sound sources through the onomatopoeia selection section 230 . For example, when the registration unit 110 receives the selection 231 of the cat 211 through the onomatopoeic selection section 230 from the user terminal, the user voice based on the onomatopoeic sound source of the cat 211 registered by the user Onomatopoeia of the cat 211 of can be provided. As another example, when the registration unit 110 does not receive the selection 232 of the dog 212 through the onomatopoeic selection section 230 from the user terminal, the user voice based on the onomatopoeic sound source of the dog 212 provided by default Onomatopoeia of the base dog 212 may be provided.

Returning to FIG. 1 , the selection unit 120 may receive a selection of any one moving picture content to be provided as a moving picture reading service among a plurality of previously registered moving picture contents from the user terminal. For example, the selection unit 120 may receive a selection of “cat in boots” from among a plurality of previously registered moving picture contents from the user terminal.

The learning unit 130 may learn a phoneme based on each text constituting the moving picture content, and learn a learning model that adjusts the learned phoneme to match at least one registered user voice. The process of learning the learning model will be described in detail with reference to FIGS. 3A and 3B .

3A and 3B are exemplary views for explaining a process of learning a moving picture content and a user's voice according to an embodiment of the present invention.

Referring to FIG. 3A , the learning unit 130 extracts respective texts constituting the moving picture content selected by the selection unit 120 from the text database 300 , and based on the extracted respective texts, encodes the phonemes. , extracts a voice file matching a phoneme from the voice database 310 through the decoding part, and adjusts the extracted voice file through an attention layer and a decode layer.

Referring to FIG. 3B , the learning unit 130 extracts user voice data from the user voice database 320 , and replaces an attention layer and a decode layer with the extracted user voice data to tacotron (tacotron) may be trained, and the trained result may be stored in the user voice database 320 . The tacotron can be trained through encoders, decoders, and waveform transformation using the Griffin-Lim algorithm.

In this process, a trained tacotron may be used so that the user's voice used for reading a moving picture can be synthesized with the text of the moving picture content.

Returning to FIG. 1 , the moving image content analysis unit 140 may analyze a plurality of sentences included in the selected moving image content. For example, the moving picture content analysis unit 140 extracts from a plurality of sentences analyzed based on a preset identification symbol (eg, dialogue-double quotation marks (" "), self-talk-single quotation marks (' ')). Tagging dialogue information for dialogue sentences, tagging onomatopoeia information for text extracted from a plurality of sentences analyzed based on onomatopoeia registered in the Korean dictionary, and tagging narration information for the remaining sentences of the analyzed plurality of sentences. can The process of analyzing the moving picture content and tagging each information will be described in detail with reference to FIG. 4 .

4 is an exemplary diagram for explaining a process of tagging onomatopoeic information by analyzing a plurality of sentences included in a moving picture content according to an embodiment of the present invention. Referring to FIG. 4 , the moving image content analyzing unit 140 may analyze a plurality of sentences included in the moving image content, and tag onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences.

For example, when the sentence included in the fairy tale content is “The cat made a meow sound” ( 400 ), the fairy tale content analysis unit 140 provides onomatopoeia information to 'meow' corresponding to the onomatopoeia of the cat registered in the Korean dictionary. 420 may be tagged, and narration information 410 may be tagged for the remaining sentences such as 'cat', 'sound', and 'made out'.

Returning to FIG. 1 again, the sound source selection unit 150 may select any one onomatopoeic sound source among a plurality of pre-stored onomatopoeic sound sources based on at least one onomatopoeia tagged with onomatopoeia information.

To this end, the sound source selection unit 150 may determine the type of object corresponding to the onomatopoeia based on at least one onomatopoeia tagged with the onomatopoeia. For example, if the onomatopoeia tagged as onomatopoeic information for the sentence analyzed by the moving picture content analysis unit 140 is 'meow', the sound source selection unit 150 determines the type of object corresponding to the onomatopoeia as 'cat'. can

The sound source selector 150 may extract and select an onomatopoeic sound source associated with the determined object type from among a plurality of pre-stored onomatopoeia sound sources. For example, the sound source selector 150 may extract a cat onomatopoeic sound source from among a plurality of pre-stored onomatopoeic sound sources. At this time, the sound source selection unit 150 preferentially selects the second cat onomatopoeia sound source registered by the user terminal when the first cat onomatopoeia sound source provided by the service provider and the second cat onomatopoeia sound source registered by the user terminal are extracted. can be selected as A process of selecting an onomatopoeic sound source related to an object type will be described in detail with reference to FIG. 5 .

5 is an exemplary diagram illustrating a selected onomatopoeic sound source according to an embodiment of the present invention. Referring to FIG. 5 , the sound source selector 150 may configure a table of onomatopoeic sound sources selected to provide a moving picture reading service. The table may be composed of, for example, onomatopoeic text 500 , onomatopoeic object type 501 , sound source provider 502 , and onomatopoeic file 503 .

For example, when the sound source selection unit 150 is selected as the onomatopoeic sound source registered by the user terminal as the animal onomatopoeic sound source, the tagged onomatopoeic text 500 - 'Meow', the onomatopoeic object type 501 - 'cat', Onomatopoeia sound source provider 502 - user, onomatopoeia file 503 - You can create a table like 'Cat001.wav'.

For another example, when the sound source selection unit 150 selects the animal onomatopoeia sound source registered by the service provider as the onomatopoeic sound source, the tagged onomatopoeic text 500-'Growl', onomatopoeic object type 501-'lion, Onomatopoeia sound source provider 502-service provider, onomatopoeia file 503-'Tiger100.wav' can create a table.

Returning to FIG. 1 again, when the onomatopoeic sound source associated with the determined object type is not extracted, the sound source selector 150 selects a predetermined similar onomatopoeic sound source based on each of the phoneme similarity and phoneme similarity for the text tagged with onomatopoeia information. can be extracted.

This is because, when the tagged onomatopoeia corresponds to an onomatopoeia for which onomatopoeia sound sources are not provided by the service provider, the onomatopoeia corresponds to a sound that mimics the sounds of people, things, and animals, so that the consonants and vowels of the onomatopoeia are simply compared; By calculating the degree of similarity with the comparison of the sound value when pronouncing the onomatopoeia, it can be replaced with the most similar onomatopoeic sound source. The process of extracting the similar onomatopoeic sound source will be described in detail with reference to FIGS. 6A and 6B .

6A and 6B are exemplary views for explaining a process of extracting a similar onomatopoeic sound source when the onomatopoeic sound source associated with the object type corresponding to the onomatopoeic word is not extracted according to an embodiment of the present invention.

The sound source selector 150 may select each similar onomatopoeic related to a phoneme and a phoneme in order to determine the similarity with the word corresponding to the onomatopoeia.

For example, the sound source selector 150 divides the consonants and vowels of onomatopoeic words into a leading, middle, and final consonant in relation to a phoneme, and calculates the number of cases where the positions of the consonants and vowels match and the degree of similarity between the syllable vectors to determine the degree of agreement. You can choose 5 high onomatopoeias. The total number of initial, middle, and final consonants in Hangeul, which is composed of syllables, is 11,172, which has the disadvantage of having to calculate too many numbers. Therefore, the sound source selection unit 150 maps both the hard sounds and the harsh sounds for the initial consonant to the regular sounds (ex. }, {ㅗ, ㅛ}, {TT, ㅠ}, {ㅐ, ㅔ, ㅖ, ㅒ}, {ㅙ, ㅚ} to reduce the number of syllables and apply the same method to the initial syllable. can By going through this mapping process, it can be reduced to 10 initial consonants, 10 consonants, and 10 final consonants. In addition, by reducing the number of letters by mapping the support with two or more consonants to the representative pronunciation (eg, ㄵ->b), letters without support (10*10=100) and letters with support (100*10=) 1000) syllables can be created, which can be reduced to a total of 1100 syllables.

In this way, the reduced number of syllables can be used as an input for calculating the phoneme similarity of the hashed syllable recurrent neural network part in the form of one-hot encoding, which is a format that enables data to be expressed without duplication.

For another example, the sound source selector 150 may select five onomatopoeic words having the most similar pronunciation using a similar phoneme unit based on a Korean pronunciation dictionary in relation to a phonetic value.

Referring to FIG. 6A , the sound source selector 150 selects a phoneme-based syllable by using a similar phoneme unit of Korean including a similar phoneme 600 and a symbol 601 of the similar phoneme 600. configurable. Here, the number of elements of the initial consonant used is 23, the number of the middle vowel is 26, and the number of the final consonant is 22, so there are a total of 10,626 input units. Consonants use the same phoneme-like pronunciation as voiced consonants, so reduce the number of vowels and use the same method as used in phonemes for neutrals: {a, ㅑ}, {ㅓ, ㅕ}, {ㅗ, ㅛ}, By mapping to {TT, ㅠ}, {ㅐ, ㅔ, ㅖ, ㅒ}, {ㅙ, ㅚ}, as the number of neutrals is reduced to 10, a total of 5,060 syllables can be reduced.

In this way, the reduced number of syllables can be used as an input for calculating the similarity of sound values of the Hashed Syllable Recurrent Neural Network part in the form of one-hot encoding, which is a format that allows data to be expressed without duplication.

Referring to FIG. 6B , the hashed syllable bi-recurrent neural network uses t(U)611, which is text-based mapped for onomatopoeia 610, and p(U)612, which is mapped based on speech, to obtain phoneme similarity. An input sequence 620 of , and an input sequence 621 of sound similarity may be used as inputs. To this end, the sound source selector 150 may define a loss function using Equation 1 below.

Here, 'T' means the number of syllables, and the hashed syllable bi-recurrent neural network may be trained in a direction in which 'T' is minimized. 'N' is the number of onomatopoeias that the service provider has, 'm' is a marginal parameter, 'dis' is the cosine similarity distance, and 'x _i ⁺ , y _i ⁺ ' is It means pairs from the same onomatopoeia, and 'x _i ⁺ , y _i ^- ' means pairs from non-identical onomatopoeias.

The sound source selector 150 may define a function called w(U _i , U _i ^- ) to approximate a distance in the case of having similar phonemes using Equation 2 below.

The sound source selector 150 may register an output vector for an input sequence 620 of phoneme similarity to pre-registered onomatopoeia and an input sequence 621 of phoneme similarity to the previously registered onomatopoeia by using the network trained in this way.

The sound source selector 150 may extract a vector based on phoneme similarity and phoneme similarity derived through a hashed syllable bi-recurrent neural network with respect to text tagged with onomatopoeic information. The sound source selector 150 may determine the final onomatopoeic sound source by reflecting the extracted vector as a weight to the extracted predetermined similar onomatopoeic sound source. The sound source selector 150 may determine the final onomatopoeic sound source by using Equation 3 below.

Referring to Equation 3, the sound source selection unit 150 reflects the weight for each upper rank among the 5-best similar onomatopoeic sound sources and selects the onomatopoeic word having the highest value in the onomatopoeic word and the phonetic value. It can be replaced with the selected onomatopoeic sound source.

The sound source selector 150 includes a frequency converter 155 , and the frequency converter 155 may perform frequency conversion on the determined final onomatopoeic sound source to be similar to the user's voice. In general, a sound source is refined into a desired sound source through phoneme extracting through a phoneme recognizer previously trained with human voice. The phoneme recognizer extracts features from a voice, models a phoneme using a model such as GMM-HMM for a voice used for training, and may represent the highest result value among the result values as a phoneme recognition result.

However, phoneme recognizers trained with human voices have a disadvantage in that it is difficult to recognize onomatopoeic sounds. Accordingly, in the present invention, by performing frequency conversion so that the final determined onomatopoeic sound source is similar to the user's voice, the onomatopoeic sound source can be directly applied to the voice recognizer.

The frequency converter 155 converts the sound of onomatopoeia to a human voice similarly to a human voice through frequency conversion such as a phase vocoder before going through a phoneme recognizer because, unlike voice, onomatopoeic words have a variety of frequency ranges, such as animal voices. By converting it, it is possible to increase the phoneme recognition rate of the phoneme recognizer. Here, since the basic frequency of a general male voice is 100 to 150 Hz, and the basic frequency of a female voice is 200 to 250 Hz, the frequency converter 155 converts the final onomatopoeic sound source to a female voice, and the final onomatopoeic sound source has a low frequency range. frequency conversion can be performed with a male voice.

When the result of the phoneme recognizer is derived using the frequency-converted final onomatopoeic sound source, the sound source selector 150 provides the result of the phoneme recognizer in units of frames (eg, 0.01 seconds). The result of the phoneme recognizer can be output in the same way as "No yaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".

Since the sound source selector 150 can be replaced with a short onomatopoeic expression rather than a long onomatopoeic expression in the case of a fairy tale, the onomatopoeic expression in the implemented fairy tale does not utilize the results of all phoneme recognizers, and only the necessary parts may be briefly extracted and used. Here, as a part to be briefly extracted, the cat phoneme recognition result recognized based on the number of syllables of the onomatopoeic word may be reduced in proportion to the number of phoneme recognition frames. For example, in the case of "Nyaong", the number of syllables is 2, and the result of phoneme recognition (total 26, b-4, ya-2, ah-9, o-2, ong-8) ), if it is reduced to be proportional to half, the result of phoneme recognition is 13 in total, making it possible to express the same onomatopoeia.

The sound source selector 150 may edit the sound source with the frame of the edited phoneme recognition result to be used as the final onomatopoeic sound source. The final onomatopoeic sound source may be edited using Equation 4 below.

Referring to Equation 4, the sound source selector 150 may search for frames having phonemes for all frames and select them in proportion to the number of onomatopoeic syllables. Here, the proportionality may be γ, the converted function may mean frequency conversion, O _t may be a feature vector of a t frame, and λ _i may mean a phoneme model.

Returning to FIG. 1 again, the moving picture reading sound source generating unit 160 synthesizes the first user's voice data uttered in the reading body registered based on the learning model based on at least one sentence tagged with narration information, and the registered dialogue body The second user's voice data uttered may be synthesized based on at least one sentence in which conversation information is tagged. Here, as the speech synthesis method, for example, both a corpus-based TTS method and a deep learning-based speech synthesis method may be used.

The moving picture reading sound source generator 160 may generate a moving picture reading sound source by synthesizing the selected onomatopoeic sound source and registered user voice data. Here, the sound source for reading a fairy tale may be generated so that at least one onomatopoeic word is expressed based on the user's voice. The process of generating a sound source for reading a fairy tale will be described in detail with reference to FIG. 7 .

7 is an exemplary view for explaining a process of synthesizing an onomatopoeic sound source and user voice data according to an embodiment of the present invention. Referring to FIG. 7 , the fairy tale reading sound source generating unit 160 receives the onomatopoeic text 700 derived as a voice text for the tacotron trained with the user voice learned by the learning unit 130 as an input, and uses the voice of the user. After obtaining the Mel-based spectrum 701 that utters the text, the selected onomatopoeic sound source 710 may be inserted to obtain the Mel-based spectrum 711 corresponding to the onomatopoeia.

The fairy tale reading sound source generator 160 uses a layer of CNN in the VF block 720 to obtain a voice that makes the sound of the onomatopoeia so that the sound of the onomatopoeia and the user's voice can be naturally synthesized in the VF block 720 ) can be trained. In this case, the moving picture reading sound source generator 160 may train the VF block 720 by further using the loss function derived through Equation 5 below.

Referring to Equation 5, α may indicate a reflection rate of a spoken speaker and phoneme, and β may indicate a reflection rate of a style with respect to onomatopoeia.

As such, when the L _total function of the trained VF block 720 falls below the threshold TH ( 730 ), the VF block 720 is a linear scale spectrogram 740 ( liear-scale ) based on the trained parameter. spectorgram) can be generated. Thereafter, the moving picture reading sound source generator 160 may synthesize ( 760 ) the user voice data and the onomatopoeic sound source through a griffin-lim reconstruction ( 750 ). In this case, the threshold value TH may be set differently for each onomatopoeic word by automatically detecting a portion where saturation of the loss function starts.

8 is an exemplary diagram illustrating a spectrum for each of an existing model, an animal real animal sound, and an onomatopoeia based on a user's voice according to an embodiment of the present invention. Referring to FIG. 8 , if the onomatopoeia is a dog onomatopoeia, the spectrum of the existing model 800 in which the existing user imitates the dog onomatopoeia using his/her own voice, the spectrum of the real animal sound 810, and the present invention The spectrum of the onomatopoeia 820 expressed based on the implemented user voice is shown, respectively.

The spectrum of the existing model 800 has a disadvantage that it cannot effectively express animal onomatopoeia because it has no rhyme and tone for the animal voice as a monotonous voice, but the spectrum of the user voice-based onomatopoeia 820 proposed in the present invention has the advantage of being able to express natural animal sounds by synthesizing the user's voice and the animal's voice.

Table 1 below is a table for quantitatively evaluating the existing model 800 of FIG. 8 , the real animal sound 810 , and the onomatopoeia 820 based on the user's voice proposed through the present invention.

method original model Invention model correlation distance 0.75 0.058

In order to quantitatively evaluate each voice data, the waveform similarity between the voice data was compared respectively. Waveform similarity was analyzed in units of frames of speech, and the similarity of waveforms was measured by analyzing 512 frequencies for each frame. The similarity measurement was based on the similarity distance, which is a representative method. Here, the similarity distance may be determined to be similar to an actual animal sound as the distance is closer to 0.

Referring to Table 1, when each onomatopoeic is quantitatively evaluated, it can be seen that the similarity of the user voice-based onomatopoeia proposed in the present invention is significantly improved compared to the similarity of the existing model through the cosine correlation distance. have.

Returning to FIG. 1 again, the providing unit 170 may provide the generated sound source for reading a moving picture to the user terminal.

The moving picture reading service providing server 100 may be executed by a computer program stored in a medium including a sequence of instructions for providing a moving picture reading service. When the computer program is executed by the computing device, user voice data uttered by the user is registered from the user terminal, and any one moving picture content to be provided as a moving picture reading service among a plurality of previously registered moving picture contents is selected from the user terminal. , analyzes a plurality of sentences included in the selected moving image content, tags onomatopoeia information for at least one onomatopoeic included in the analyzed plurality of sentences, and a plurality of pre-stored onomatopoeia based on the at least one onomatopoeia tagged with onomatopoeia information. Select any one of the onomatopoeic sound sources among the onomatopoeic sound sources, synthesize the selected onomatopoeic sound source and the registered user voice data to generate a fairy tale reading sound source, and provide the generated fairy tale reading sound source to the user terminal, wherein the fairy tale reading sound source is at least one The onomatopoeia may include a sequence of commands to be generated to be expressed based on the user's voice.

9 is a flowchart of a method for providing a moving picture reading service in the moving picture reading service providing server according to an embodiment of the present invention. The method of providing a moving picture reading service in the moving picture reading service providing server 100 shown in FIG. 9 is a step of time-series processing by the moving picture reading service providing server 100 according to the embodiment shown in FIGS. 1 to 8 include those Therefore, even if omitted below, it is also applied to the moving picture reading service providing server 100 according to the embodiment shown in FIGS. 1 to 8 .

In step S910, the moving picture reading service providing server 100 may receive user voice data uttered by the user from the user terminal registered.

In step S920 , the moving picture reading service providing server 100 may receive a selection of any moving picture content to be provided as a moving picture reading service from among a plurality of previously registered moving picture contents from the user terminal.

In step S930, the moving picture reading service providing server 100 may analyze a plurality of sentences included in the selected moving picture content.

In operation S940, the moving picture reading service providing server 100 may tag onomatopoeia information for at least one onomatopoeic included in the plurality of analyzed sentences.

In step S950, the moving picture reading service providing server 100 may select any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on at least one onomatopoeia tagged with onomatopoeia information.

In step S960, the moving picture reading service providing server 100 may generate a moving picture reading sound source by synthesizing the selected onomatopoeic sound source and the registered user voice data.

In step S970, the moving picture reading service providing server 100 may provide the generated moving picture reading sound source to the user terminal.

In the above description, steps S910 to S950 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as needed, and the order between the steps may be switched.

The method of providing a moving picture reading service in the moving picture reading service providing server through FIGS. 1 to 9 can also be implemented in the form of a recording medium including a computer program stored in a medium executed by a computer or instructions executable by a computer. have. In addition, the method of providing a moving picture reading service in the moving picture reading service providing server through FIGS. 1 to 9 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: fairy tale reading service providing server
110: register
120: selection unit
130: study unit
140: fairy tale content analysis unit
150: sound source selection unit
155: frequency converter
160: fairy tale reading sound source generator
170: provider

Claims (20)

In the server providing a fairy tale reading service,
a registration unit for registering user voice data uttered by the user from the user terminal;
a selection unit for receiving, from the user terminal, selecting any one moving picture content to be provided as the moving picture reading service from among a plurality of previously registered moving picture contents;
a moving image content analysis unit that analyzes a plurality of sentences included in the selected moving image content and tags onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences;
an onomatopoeic sound source selection unit that selects any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on the at least one onomatopoeia tagged with the onomatopoeia information;
a moving picture reading sound source generating unit for generating a moving picture reading sound source by synthesizing the selected onomatopoeic sound source and the registered user voice data; and
A providing unit that provides the generated sound source for reading a fairy tale to the user terminal
including,
The fairy tale reading sound source is generated so that the at least one onomatopoeic word is expressed based on the user's voice, the fairy tale reading service providing server.
The method of claim 1,
Wherein the registration unit receives at least one of first user voice data uttered in a recited form by a user and second user voice data uttered in a conversational form by a user based on a preset script from the user terminal.
The method of claim 1,
The registration unit will further register the animal onomatopoeia sound source recorded with animal onomatopoeia from the user terminal, a fairy tale reading service providing server.
3. The method of claim 2,
The moving image content analysis unit tags the dialogue information for dialogue sentences extracted from the analyzed plurality of sentences based on a preset identification symbol, and adds to the text extracted from the analyzed sentences based on onomatopoeia registered in the Korean dictionary. For tagging the onomatopoeia information, and tagging the narration information for the remaining sentences of the analyzed plurality of sentences, a fairy tale reading service providing server.
5. The method of claim 4,
The moving picture reading further comprising a learning unit for learning a phoneme based on each text constituting the moving picture content and learning a learning model for adjusting the learned phoneme to match the registered at least one user voice Serving Server.
6. The method of claim 5,
The moving picture reading sound source generator synthesizes the registered first user voice data based on the learning model based on at least one sentence tagged with the narration information, and combines the registered second user voice data with the conversation information. A server for providing a fairy tale reading service, which is synthesized based on at least one tagged sentence.
The method of claim 1,
The sound source selection unit determines the type of object corresponding to the onomatopoeia based on at least one onomatopoeia tagged with the onomatopoeia, and extracts the onomatopoeia sound source associated with the determined object type among the plurality of pre-stored onomatopoeia sound sources, A server that provides fairy tale reading service.
8. The method of claim 7,
When the onomatopoeic sound source associated with the determined object type is not extracted, the sound source selection unit extracts a predetermined similar onomatopoeic sound source based on each of the phoneme similarity and the phonetic similarity for the text tagged with the onomatopoeia information. Serving Server.
9. The method of claim 8,
The sound source selection unit extracts a vector based on the phoneme similarity and phoneme similarity for the text tagged with the onomatopoeic information, and reflects the extracted vector as a weight to the extracted predetermined onomatopoeic sound source to determine the final onomatopoeic sound source A server that provides a fairy tale reading service.
10. The method of claim 9,
The sound source selection unit will include a frequency conversion unit for performing frequency conversion to be similar to the user's voice with respect to the determined final onomatopoeic sound source, moving picture reading service providing server.
A method for providing a fairy tale reading service in a server,
receiving user voice data uttered by the user from the user terminal;
receiving, from the user terminal, a selection of any one moving picture content to be provided as the moving picture reading service from among a plurality of previously registered moving picture contents;
analyzing a plurality of sentences included in the selected moving image content;
tagging onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences;
selecting any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on the at least one onomatopoeia tagged with the onomatopoeia information;
generating a moving picture reading sound source by synthesizing the selected onomatopoeia sound source and the registered user voice data; and
Comprising the step of providing the generated fairy tale reading sound source to the user terminal,
The moving picture reading sound source is a method for providing a moving picture reading service, wherein the at least one onomatopoeic word is generated to be expressed based on the user's voice.
12. The method of claim 11,
The step of receiving the user voice data registration,
The method for providing a children's story reading service, comprising the step of registering at least one of first user voice data uttered by a user in a read form and second user voice data uttered in a conversational form by a user based on a preset script from the user terminal; .
13. The method of claim 12,
The step of tagging the onomatopoeic information is,
tagging dialogue information for dialogue sentences extracted from the plurality of analyzed sentences based on a preset identification symbol;
tagging the onomatopoeic information with respect to the text extracted from the analyzed plurality of sentences based on the onomatopoeia registered in the Korean dictionary; and
The method for providing a moving picture reading service comprising the step of tagging narration information for the remaining sentences of the analyzed plurality of sentences.
14. The method of claim 13,
learning phonemes based on each text constituting the moving picture content; and
The method further comprising the step of learning a learning model for adjusting the learned phoneme to match the registered at least one user voice.
15. The method of claim 14,
The step of generating the sound source for reading the fairy tale is,
synthesizing the registered first user voice data based on the learning model based on at least one sentence tagged with the narration information; and
and synthesizing the registered second user voice data based on at least one sentence in which the conversation information is tagged.
12. The method of claim 11,
The step of selecting the onomatopoeic sound source is,
determining an object type corresponding to the onomatopoeia based on at least one onomatopoeia tagged with the onomatopoeia; and
The method for providing a moving picture reading service comprising extracting an onomatopoeic sound source associated with the determined object type from among the plurality of pre-stored onomatopoeia sound sources.
17. The method of claim 16,
The step of selecting the onomatopoeic sound source is,
If the onomatopoeic sound source associated with the determined object type is not extracted, extracting a predetermined similar onomatopoeic sound source based on each of the phoneme similarity and phoneme similarity with respect to the text tagged with the onomatopoeia information, which comprises the steps of: HOW TO PROVIDE SERVICES.
18. The method of claim 17,
The step of selecting the onomatopoeic sound source is,
extracting a vector based on the phoneme similarity and phoneme similarity for the text tagged with the onomatopoeic information; and
and determining a final onomatopoeic sound source by reflecting the extracted vector as a weight to the extracted predetermined similar onomatopoeic sound source.
19. The method of claim 18,
The step of selecting the onomatopoeic sound source is,
The method for providing a moving picture reading service comprising the step of performing frequency conversion on the determined final onomatopoeic sound source to be similar to the user's voice.
A computer program stored in a computer readable recording medium including a sequence of instructions for providing a story reading service,
When the computer program is executed by a computing device,
Registering the user voice data uttered by the user from the user terminal,
receiving selected one of the moving picture contents to be provided as the moving picture reading service from among the plurality of previously registered moving picture contents from the user terminal;
analyzing a plurality of sentences included in the selected moving image content, and tagging onomatopoeia information for at least one onomatopoeia included in the analyzed plurality of sentences,
Selecting any one onomatopoeic sound source among a plurality of pre-stored onomatopoeia sound sources based on the at least one onomatopoeia to which the onomatopoeic information is tagged,
Synthesizing the selected onomatopoeic sound source and the registered user voice data to create a sound source for reading a moving picture,
Provide the generated fairy tale reading sound source to the user terminal,
A computer program stored in a computer-readable recording medium including a sequence of instructions for generating the at least one onomatopoeic word to be expressed based on the user's voice, the sound source for reading the moving picture.

Images