KR20220154655A - Device, method and computer program for generating voice data based on family relationship - Google Patents

Abstract

A device for generating speech data comprises: a family relationship model generating part that generates a plurality of family relationship models corresponding to a family relationship of a family member, based on the speech data of the family member; an input part that receives the speech data of the family member related to a user who wants to generate the speech data; a selection part that selects one family relationship model among the plurality of family relationship models based on the family relationship between the user and the family member; and a generating part that generates the speech data of the user by inputting the speech data of the family member to the selected family relationship model.

Description

DEVICE, METHOD AND COMPUTER PROGRAM FOR GENERATING VOICE DATA BASED ON FAMILY RELATIONSHIP

The present invention relates to an apparatus, method and computer program for generating voice data.

TTS (Text-To-Speech) is a technology that automatically generates speech sound waves by a machine. After recording the speech sound of one person selected as a model, dividing it into certain speech units, and then attaching codes to the synthesizer It refers to a technology that artificially creates speech sounds by recombining only necessary phonetic units according to instructions after input.

Recently, it is possible to provide a natural personalized voice service by learning a desired voice using voice synthesis technology and learning pitch, stress, and pronunciation according to context.

Regarding technology for providing such a personalized voice service, Korean Patent Publication No. 2020-0016516, which is a prior art, discloses a personalized virtual voice synthesis device and method.

However, when a specific voice is to be restored using conventional voice synthesis technology, speech data of a specific user to be restored must be collected for a sufficient period of time. However, in the case of a deaf person with a congenital disability, it is difficult to restore the voice as the voice of the deaf person cannot be sufficiently secured.

An apparatus, method, and computer program for generating a plurality of family relationship models corresponding to family relationships of family members based on voice data of a plurality of family members are provided.

Receive voice data of a family member related to a user whose voice data is to be restored, select one family relationship model from among a plurality of family relationship models based on family relationships between the user and family members, and select the family member in the selected family relationship model. It is intended to provide a device, method, and computer program for generating user's voice data by inputting voice data of the user.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is a family relationship model for generating a plurality of family relationship models corresponding to family relationships of the family members based on voice data of a plurality of family members. A generation unit, an input unit that receives voice data of a family member related to a user who wants to generate voice data, and a selection for selecting one family relationship model from among the plurality of family relationship models based on a family relationship between the user and the family member. and a generating unit generating voice data of the user by inputting voice data of the family member into the selected family relationship model.

Another embodiment of the present invention includes registering deaf-mute information including identifier information of a user who will use a voice generation service, performing an authentication process for the voice generation service based on the registered deaf-mute information, and performing the authentication process. Performing a call service with another user's terminal based on a result of the process, and receiving text information or voice while the call service is being performed, wherein the call service is performed in a deaf mode according to the result of the authentication process. , the text information is converted into voice data based on a family relationship model corresponding to the registered deaf information, and the converted voice data is transmitted to the other user terminal. can provide

According to another embodiment of the present invention, a registration unit for registering deaf information including user identifier information to use a voice generation service, and an authentication process performer performing an authentication process for the voice generation service based on the registered deaf information. , Based on the result of the authentication process, a call service performing unit that performs a call service with another user terminal and an input unit that receives text information or voice while the call service is being performed, and according to the result of the authentication process When the call service is provided in the deaf mode, the text information is converted into voice data based on a family relationship model corresponding to the registered deaf information, and the converted voice data is transmitted to the other user terminal. A user terminal may be provided.

The above-described means for solving the problems is only illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, in the past, it was possible to synthesize a desired voice using data of a specific voice by securing a lot of data to be used for voice synthesis, but the present invention provides voice data of a specific voice. It is possible to provide a device, method, and computer program for generating user data using voice data of family members even without the present invention.

It is possible to provide a device, method, and computer program for restoring a user's voice based on family relationships between the user and family members so that a user with a congenital disability can have his or her own voice.

It is possible to provide a device, method, and computer program that provide a service that provides a voice data synthesized as one's own voice to the other party through a call to create a conversational situation with the other party.

1 is a block diagram of an apparatus for generating voice data according to an embodiment of the present invention.
2A and 2B are exemplary diagrams for explaining a preprocessing process of classifying a plurality of family relationships based on gender information of family members included in the plurality of family relationships according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a process of classifying a family relationship based on a pitch distance value, a pitch distribution, and an age difference value between a plurality of family members according to an embodiment of the present invention.
4A to 4E are exemplary diagrams for explaining a process of generating a family relationship model according to an embodiment of the present invention.
5A to 5C are exemplary diagrams for explaining a process of generating voice data of a user by using a family relationship model selected based on voice data of family members related to the user according to an embodiment of the present invention.
6A to 6D are exemplary diagrams for explaining a process of performing a call service through an authentication process in a user terminal according to an embodiment of the present invention.
7A to 7C are exemplary diagrams for explaining a process of receiving a user's voice data service through an app in a user terminal according to an embodiment of the present invention.
8A and 8B are exemplary diagrams for explaining a process of generating a user's voice data and providing a call service according to an embodiment of the present invention.
9 is a flowchart of a method of generating voice data in the voice data generating apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may be performed instead by a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for generating voice data according to an embodiment of the present invention. Referring to FIG. 1 , the voice data generating apparatus 100 may include a family relationship model generating unit 110 , an input unit 120 , a selection unit 130 and a generation unit 140 .

The family relationship model generation unit 110 may generate a plurality of family relationship models corresponding to family relationships of family members based on voice data of a plurality of family members. Here, the plurality of family relationship models may be generated based on, for example, gender information, a pitch distance value, a pitch distribution value, an age difference value, and the like. For example, the family relationship model generation unit 110 may generate a total of 24 family relationship models based on gender information, a pitch distance value, a pitch distribution value, and an age difference value. A process of generating a family relationship model will be described in detail with reference to FIGS. 2A to 3 .

2A and 2B are exemplary diagrams for explaining a preprocessing process of classifying a plurality of family relationships based on gender information of family members included in the plurality of family relationships according to an embodiment of the present invention.

The family relationship model generation unit 110 may classify a plurality of family relationships based on gender information of family members included in the plurality of family relationships.

For example, suppose that the male 200 and the female 210 utter the same character. Referring to FIGS. 2A and 2B , it can be confirmed that the frequency-specific characteristics of a male 200 and a female 210 appear very different under the same condition.

This is because the pitch, which is a unique voice that each person has, varies depending on the length, size, and tension of the vocal cords, and the difference in fundamental frequency according to gender is mainly caused by the difference in size and length of the vocal cords. For example, there is almost no difference in pitch between the male 200 and the female 210 before puberty, but after puberty, the male's larynx grows and the vocal cords become longer, and the frequency of the male 200 changes to that of the female 210. lower than the frequency.

Accordingly, the family relationship model generation unit 110 may classify the plurality of family relationships into a mother-daughter relationship, father-son relationship, sister-sister relationship, brother-sister relationship, and the like, based on gender information of family members included in the plurality of family relationships.

3 is an exemplary diagram for explaining a process of classifying a family relationship based on a pitch distance value, a pitch distribution, and an age difference value between a plurality of family members according to an embodiment of the present invention. Referring to FIG. 3 , the family relationship model generation unit 110 may classify family relationships into a plurality of family relationship groups based on pitch distance values between voice data of a plurality of family members and generate a plurality of family relationship group models. have. Here, the pitch indicates a unique voice that each person has, and is different depending on the length, size, and tension of the vocal cords.

For example, when the family relationship model generation unit 110 classifies family relationships as sister relationships 300 based on gender information of a plurality of family members, the pitch distance values between voice data of sisters are close according to whether or not they are close. Family relationships can be further classified.

In general, the biggest index representing a person's timbre is the tone of a voice, and the tone of a voice is also an indicator for distinguishing voices into female, male, child, and adult voices. However, in the case of a family relationship, for example, the voices between pairs of members included in the sisterhood relationship 300 are similar, but the degree of similarity may be very small or large. Accordingly, the family relationship model generation unit 110 may further classify the family relationship based on the degree of voice similarity between pairs of members based on the pitch distance value. The family relationship model generating unit 110 may calculate the pitch distance value using, for example, Equation 1 below.

Referring to Equation 1, for example, the family relationship model generation unit 110 determines that the first voice data and the second voice data are non-independent data (when covariance exists) because they have a family relationship with each other, and , the pitch distance value can be calculated using the Mahalanobis distance.

For example, the family relationship model generation unit 110 classifies family relationships into a case where the pitch distance value exceeds the threshold value 310 and a case where the pitch distance value is less than the threshold value 320 based on the pitch distance value. can be classified. Here, the case where the pitch distance value is greater than the threshold value (310) may mean a case where the pitch distance value is far, and the case where the pitch distance value is less than the threshold value (320) may mean a case where the pitch distance value is close.

At this time, the family relationship model generation unit 110 may further classify family relationships classified based on the pitch distance value into family relationship groups based on the pitch distribution value. This is because there is a distinct difference in energy distribution when the spectrum of voice data is analyzed.

For example, when the pitch distance value is less than the threshold value 320, the family relation model generation unit 110 uses a low frequency band 322 based on a frequency band in which pitch distance values of two similar voice data exist. ) or a high frequency band 321, the family relationship group may be further classified.

This is because the human ear recognizes it on a log scale, so even two data having a difference by the same frequency change the data existing in the low frequency band 322 and the data existing in the high frequency band 321 to the same value. In this case, the difference felt when listening to the changed data may be different. In general, the human ear is more sensitive in the low frequency band 322. If the value of the high frequency band 321 is changed by '2' for voice synthesis and the value of the low frequency band 322 is changed by '2', the value of the low frequency band 322 is You can think of it as a bigger change. Accordingly, the family relationship model generating unit 110 may generate a family relationship group model to change data more sensitively in the low frequency band 322 .

The family relationship model generation unit 110 may classify family relationships into a plurality of family relationship groups based on age differences between a plurality of family members and generate a plurality of family relationship models. Since age is also a factor influencing the voice, the age difference must be taken into account when converting or synthesizing the voice even if the human voice tone is similar.

For example, the family relationship model generation unit 110 determines the family relationship as a first group 311 (two sibling relationships 300) based on the age difference value when the pitch distance value exceeds the threshold value 310. When the age difference between family members exceeds a threshold value) or the second group 312 (when the age difference value between two family members is less than a threshold value), it may be classified.

For another example, when the pitch distance value for the sister relationship 300 is classified into the high frequency band 321, the family relationship model generation unit 110 generates a third group 330 or a fourth group based on the age difference value. Group 331 can be classified. In contrast, when the pitch distance value for the sister relationship 300 is classified as the low frequency band 322, the family relationship model generation unit 110 classifies the fifth group 340 or the sixth group (340) based on the age difference value. 341) can be classified as:

In this way, the family relationship model generation unit 110 further classifies the family relationship based on the pitch distance value, the pitch distribution, the age difference value, etc. for the sister relationship 300 classified based on the gender information of the family member, The family relationship group of can be classified into a total of 6 family relationship groups.

4A to 4D are exemplary diagrams for explaining a process of generating a family relationship model according to an embodiment of the present invention.

4A is an exemplary diagram illustrating an autoencoder according to an embodiment of the present invention. Referring to FIG. 4A, an autoencoder includes an encoder 410 and a decoder 411, and approximates values of an input 400 and an output 401 using the encoder 410 and the decoder 411. As a technique for doing this, it has the feature that the dimensions of the input 400 and the output 401 are the same.

The main feature of autoencoders is that each layer of a neural network is learned step by step, and the final output reproduces the initial input. The dimensions (number of nodes) of the input 400 and output 401 layers are the same, but the hidden layer has a lower dimension than the input and output layers, so the neural network extracts features by compressing the input data, Based on the extracted features, output data that reproduces the input 400 as much as possible can be derived.

이면, 히든 레이어를 통해 보틀넥 히든 레이어의 코드로서

로 표현될 수 있다. The autoencoder is implemented by stacking multiple layers of hidden layers, so that more meaningful features can be extracted. , For example, a code part capable of obtaining the smallest compressed feature may be a bottleneck hidden layer. Here, the input vector of the autoencoder is

As the code of the bottleneck hidden layer through the hidden layer

can be expressed as

* This process is a kind of compression process called deterministic mapping, and means a process of extracting meaningful features from the input 400. The deterministic mapping can be described, for example, through Equation 2 below.

Referring to Equation 2, θ=W, denotes a parameter of b, W is a weight matrix of d*D, and b may denote a bias.

으로 매핑될 수 있다. 이 때, 매핑은, 예를 들어, 다음의 수학식 3을 통해 이루어질 수 있으며, 손실함수는, 예를 들어, 다음의 수학식 4을 통해 도출될 수 있다. y, the code value calculated in the hidden layer, is the restored vector

can be mapped to In this case, the mapping may be performed through, for example, Equation 3 below, and the loss function may be derived through Equation 4 below, for example.

Through this learning process, the family relationship model generation unit 110, for example, when receiving voice data in a wav file format, can convert the input voice data into a compressed feature vector to express the voice well.

4B and 4C are exemplary diagrams for explaining a process of performing code conversion from voice data corresponding to a sister relationship according to an embodiment of the present invention.

The family relationship model generator 110 derives a first feature vector from voice data of a first family member among a plurality of family members, derives a second feature vector from voice data of a second family member, and and based on the second feature vector, a family relationship model corresponding to the family relationship between the first family member and the second family member may be generated.

The family relationship model generation unit 110 may generate a family relationship model by performing voice similarity modeling through the first feature vector and the second feature vector.

Referring to FIG. 4B , the family relationship model generation unit 110 may train a voice model of an older sister 420 among sister relationships using an autoencoder. For example, when the family relation model generation unit 110 receives the voice data of the older sister, it can train the bottleneck layer to extract effective features through an auto-encoding process.

In the encoding part 421, the first feature vector of the bottleneck layer can be extracted through unsupervised learning of the user's voice and the feature that can most effectively express the sister's spoken voice. have. At this time, the extracted older sister's code may be 'c1'.

Referring to FIG. 4C , the family relationship model generation unit 110 may train a voice model of a younger sister 430 among sister relationships using an autoencoder. For example, when receiving voice data of a younger sister, the family relation model generation unit 110 may repeat learning until an output is approximated to an input.

The family relationship model generation unit 110 may extract a second feature vector in which the younger sister's voice is well reflected through the bottleneck layer through the encoding part 431 . At this time, the code of the extracted younger sister may be 'c2'.

Referring to FIG. 4D , the family relationship model generator 110 converts 'c1' (440), a code derived from the voice model of the older sister 420, to 'c2' (code derived from the voice model of the younger sister 430). 450), modeling may be performed. This is because the code pairs of data in sister relationship have similarity, so modeling is possible. You can create relationship models.

Referring to FIG. 4E , the family relationship model generation unit 110 may connect the encoder part 421 of the older sister 420 and the decoder part 431 of the younger sister 430 when modeling of the sister-sister relationship is performed. have. For example, when the family relationship model generation unit 110 inputs input data to the encoder part 421 of the older sister 420 and extracts features through the bottleneck layer, the sister relationship model infers a similar voice using code2code. can At this time, after conversion into a code of the corresponding voice, the voice may be output through the decoding part 431 of the younger sister 430.

The input unit 120 may receive voice data of a family member related to a user who wants to generate voice data. For example, the input unit 120 may receive voice data of a family member (eg, mother, older sister, etc.) related to a deaf person with a congenital disability.

The selection unit 130 may select one family relationship group model from among a plurality of family relationship group models based on family relationships between the user and family members. For example, when a family relationship between a user and a family member is a 'sister relationship', the selector 130 may select a family relationship group model corresponding to the sister relationship.

The selection unit 130 may select one family relationship group model from among a plurality of family relationship group models further based on gender information between the user and family members. For example, when both the user and the family members are male, the selection unit 130 may select a family relationship model corresponding to a sibling relationship consisting only of males or a family relationship group model corresponding to a parent-child relationship.

The selector 130 may select one family relationship group model from among a plurality of family relationship group models further based on the age difference between the user and the family members. For example, if the family relationship between the user and the family member is a 'sister relationship' and the age difference between the user and the family member is greater than or less than a threshold value, the selector 130 selects a family relationship group model corresponding to the sister relationship. Among them, a family relation group model in which the age difference exceeds the threshold value or a family relation group model in which the age difference value is less than the threshold value may be selected.

According to an embodiment, the selector 130 selects one family relation group model from among a plurality of family relation group models based on the user's voice data and the family member's voice data, when some voice data of the user exists. can Here, the user's voice data may be data of a degree that can be used as voice among voices uttered through resonating of the user's vocal cords.

The selector 130 may select one family relation group model from among a plurality of family relation group models further based on pitch distance values between the user's voice data and the family member's voice data. For example, when the family relationship between a user and a family member is a 'sister relationship', the sister relationship is determined based on the pitch distance value between the 'f0' value of the user's voice and the 'f0' value of the family member's voice to be restored. Among the corresponding family relationship models, a family relationship group model whose pitch distance value is less than the threshold value or a family relationship group model whose pitch distance value exceeds the threshold value may be selected.

The selector 130 may select one family relationship group model from among a plurality of family relationship group models further based on a pitch distribution value between voice data of the user and voice data of family members. For example, when the family relationship between the user and the family member is a 'sister relationship' and the pitch distribution value of the user and the family member is a low frequency band or a high frequency band, the selector 130 selects a family relationship group corresponding to the sister relationship. Among the models, a family relation group model corresponding to a low frequency or a family relation group model corresponding to a high frequency may be selected.

The generation unit 140 may generate the user's voice data by inputting the family member's voice data into the family relationship model selected based on the user's voice data and the family member's voice data. Here, the user's voice data may be provided during a call service between the user and the other party.

According to another embodiment, when voice data of the user does not exist, the selector 130 may select one family relationship group model from among a plurality of family relationship group models based on voice data of family members related to the user. . A process of generating user voice data by selecting a family relation group model when user voice data does not exist will be described in detail with reference to FIGS. 5A to 5C .

5A to 5C are exemplary diagrams for explaining a process of generating voice data of a user by using a family relation group model selected based on voice data of family members related to the user according to an embodiment of the present invention. In the following description, it is assumed that the deaf person is the older sister and the hearing person is the younger sister.

Referring to FIG. 5A , when there is no voice data of a deaf older sister available as a voice, the selection unit 130 corresponds to a sister relationship 500 among a plurality of family relationship group models based on relationships between older sisters and younger sisters. You can choose a family relationship group model that does.

Thereafter, the selection unit 130 compares the ages between the deaf sister and the hearing sister among the family relation group models corresponding to the selected sister relationship 500, and if the age difference exceeds a threshold value or is less than a threshold value, the selection unit Step 130 selects a model corresponding to the first group 510 or the second group 511 based on whether the age difference exceeds or falls below a threshold among the family relationship group models corresponding to the sister relationship 500. can

Finally, the selector 130 finally selects the third group model 520 to the sixth group ( 520 ) based on the similarity between the selected first group model 510 or the second group model 511 and the voice data of the younger sister. 523) can be selected.

Referring to FIG. 5B , the generation unit 140 may extract a feature vector from the younger sister's voice data using an autoencoder or a Convolution Neural Network (CNN). For example, a 40-dimensional (N-dimensional) Mel Spectrogram can be extracted from the voice data of the younger sister in units of 10 ms. At this time, the generation unit 140 generates a matrix of M (time length of speech data) x 40 dimensions, extracts features of the sister's voice through CNN pooling from the generated matrix, and then features vector ( 1x256) can be extracted. Here, CNN pooling serves to downsample the feature map, and can serve to reduce the amount of input variables.

Thereafter, the selector 130 uses the following Equation 5 to train A corresponding to the voice characteristic of the younger sister and one group finally selected from the third group 520 to the sixth group 523. The most similar speaker can be selected through comparison between B, which is the family data.

Thereafter, the generation unit 140 may analyze the similarity between feature vectors using a deep learning model configured in a feed forward network structure. Here, the feed forward network may have a structure in which data is input to an input layer and an output value is exported to a final output layer through one or more hidden layers.

For example, the generation unit 140 may analyze the degree of similarity using a fully connected network. Here, the Fully Connected Network is the layer that determines the classification at the end of deep learning, and serves to connect the layers converted to one-dimensional vectors into one vector.

For example, a feature vector derived from the younger sister's voice data may be mapped with X 540 . In this case, Y 541 may be an index of a speaker used for training. Indexes such as 'Speaker No. 1: 1' and 'Speaker No. 100: 100' may be given according to the order of speakers. The generation unit 140 may generate voice data of the user by inputting voice data of the family member into a family relationship model selected based on the voice data of the family member related to the user. Here, the user's voice data may be provided during a call service between the user and the other party.

The voice data generating apparatus 100 may be executed by a computer program stored in a medium including a sequence of instructions for generating voice data. When the computer program is executed by a computing device, the voice data generating device 100 generates a plurality of family relationship models corresponding to the family relationships of the family members based on the voice data of the plurality of family members, and generates the voice data. Receive voice data of family members related to the user, select one family relationship model from among a plurality of family relationship models based on family relationships between the user and family members, and input voice data of family members to the selected family relationship model. and a sequence of instructions for generating user voice data.

6A to 6D are exemplary diagrams for explaining a process of performing a call service through an authentication process in a user terminal according to an embodiment of the present invention.

The user terminal may register deaf information including user identifier information to use the voice generation service. For example, after the user terminal executes a voice generation service providing app, the user terminal may receive registration of deaf information including a name and phone number necessary for performing an authentication process.

The user terminal may perform a call service with other user terminals based on the result of the authentication process. For example, registration information of the deaf-mute person may be pre-registered in the deaf-mute person database in order to perform the authentication process.

For example, by submitting a proof document proving that a deaf user is deaf in advance to an offline store of a telecommunications company, the submitted certification document may be reviewed by a service manager and registration information of the deaf person may be registered in a deaf person database.

For another example, by scanning and submitting a proof document proving that a deaf user is a deaf person through an app of a telecommunications company that can be linked with a voice generation service providing app, the proof document submitted by the service manager is reviewed and the registration information of the deaf person is confirmed. can be registered in the database.

The user terminal may perform an authentication process based on whether the registered deaf-mute information and the deaf-mute registration information registered in the deaf-mute database match. For example, the user terminal may perform an authentication process based on whether the phone number of the user terminal included in the information about the deaf-mute-mute person matches the phone number of the deaf-mute person included in the registration information of the deaf-mute person registered in the database for the deaf-mute person.

The user terminal may receive text information or voice input while a call service is being performed. For example, when authentication is successful through an authentication process, a user terminal may perform a call service with another user terminal (not shown) through a 'deaf mode'. For example, in order to perform a call service with another user terminal (not shown), information related to other users may be pre-registered as hearing registration information in a database for hearing persons.

Here, other user terminals may register hearing registration information by receiving an invitation message from the user terminal. For example, in order to perform a call service, the user terminal receives a selection of at least one other user from among a plurality of other users (hearing persons) stored in the address book of the user terminal through a voice generation service providing app from the user (deaf person), and the selected other user is selected. An invitation message including a link of a voice generation service providing app may be transmitted to the user's terminal (not shown). At this time, through transmission of the invitation message, the hearing registration information (eg, phone number) of the hearing person may be registered in the hearing database.

Thereafter, another user (hearing person) who has received the invitation message can install the voice generation service providing app through the link and authenticate the user identification information (eg, phone number) of the other user (hearing person) through the installed app. have. Here, as a result of the authentication process, the contact information of other users (hearing persons) may be displayed in the contact list of the user terminal (deaf persons), and other users (hearing persons) may perform a call service through the 'hearing mode'.

The user terminal may receive information on at least one other user (hearing person) from the hearing database and perform a call service based on the received information on the other user (hearing person). For example, the user terminal displays the contact information of at least one other user (hearing person) who will perform a call service in the contact list of the user (deaf person), and selects any one of the displayed contact information of the at least one other user (hearing person). By selecting the contact information of the user (hearing person), a call service can be performed with the selected other user (hearing person).

Hereinafter, description will be made using the UI of the voice generating service providing app of the user terminal.

6A is an exemplary diagram for explaining a process of inviting another user terminal from a user terminal according to an embodiment of the present invention. Referring to FIG. 6A , when the authentication process for the voice generation service is completed based on the deaf information, the user terminal 600 selects at least one other person through a 'friend invitation icon' 611 in the contact menu 610. A user terminal (not shown) may be invited.

For example, when a 'friend invitation icon' 611 is selected by the user, the user terminal 600 may display a contact list 612 including at least one other user's contact information. The user terminal can register in the contact management menu 615 by receiving selection of an 'additional icon' 614 for another user to be invited from among at least one other user. At this time, the user terminal uses the 'search button' 613 to more easily search for other users to invite, and then selects the 'add icon' 614 for the other users to invite, so that the contact management menu ( 615) can also be registered.

The user terminal 600 receives an 'invite button' 616 for a specific other user from the user among at least one other user registered in the contact management menu 615 and selects a specific other user to use the call service with. . At this time, if the number of other users registered in the contact management menu 615 is more than a predetermined number (eg, 4 people), the user terminal 600 performs additional registration after deleting through the 'friend delete button' 617. can be done For example, when the user terminal receives a selection of the invitation button 616 included in the name field of 'mom' from the user, an invitation message including an installation link of a voice generation service providing app is transmitted to the user terminal of 'mom'. It can be. At this time, through the transmission of the invitation message, the hearing registration information related to the hearing mother may be registered in the hearing database.

If the installation of the voice generation service providing app in the user terminal of 'mom' is completed, a call service between the user terminal and the user terminal of 'mom' may be performed.

The user terminal 600 may display a list of other users on the contact menu 610 for which the voice generation service providing app has been installed, and perform a call service by selecting any other user from the displayed list of other users. At this time, the user terminal may change the order of other users included in the other user list through the edit button 618, and may register nicknames for each other user.

6B is an exemplary diagram for explaining a process of performing a voice call service in a user terminal according to an embodiment of the present invention. Referring to FIG. 6B , the user terminal 600 may receive a user's selection of another user to perform a voice call from among at least one other user registered in the contact menu 620 . At this time, when the user terminal 600 receives a name of another user selected by the user, the user terminal 600 may display call history with the selected other user.

For example, when the user wants to make a voice call with 'Hong Gil-dong' among at least one other user registered in the contact menu 620, the user terminal 600 sends a voice call from the user to 'Hong Gil-dong' located within the contact area. A button 621 may be input. Thereafter, the user terminal 600 may display a call preparation screen 622 and receive a 'call button' 623 from the user. At this time, if the other user accepts the call, the user terminal can perform a voice call service between the user and the other user.

At this time, when performing the voice call service, the user terminal may operate in the 'deaf mode' and the other user terminal 600 may operate in the 'hearing mode'.

Hereinafter, a process of performing a voice call service in each of the user terminal 600 and the other user terminal 630 will be described.

After the user terminal 600 receives text from the user through the input window 624 and then presses the send button 625, the input text is converted into user voice data and transmitted to the other user terminal 630. It can be.

The other user's terminal 630 may activate the speaker mode 632 to output the user's voice data.

The other user's terminal 630 may output whether the microphone mode is activated (633).

The other user's terminal 630 activates the microphone mode 631 to receive a voice spoken by the other user in response to the user's voice data. At this time, the spoken voice may be converted into text and transmitted to the user terminal.

The other user's terminal 630 may receive a video call button 634 from the other user while performing the voice call service, and convert the voice call service to the video call service.

When the other user's terminal 630 wants to end the voice call service, it can receive a voice call end button 635 from the other user.

6C is an exemplary diagram for explaining a process of performing a video call service in a user terminal according to an embodiment of the present invention. Referring to FIG. 6C , the user terminal 600 may display the user's face and another user's face together through a video call service. The user terminal 600 operates in the 'deaf mode', receives text from the user through the input window 641 and then presses the send button 642, the input text is converted into the user's voice data and other It may be transmitted to the user terminal 630.

The other user's terminal 630 may display the other user's face and the user's face together through the video call service. The other user's terminal 630 is operated in a 'hearing mode', and when the other user activates the microphone mode 644 and utters a voice, the uttered voice can be converted into text and transmitted to the user terminal.

The other user's terminal 630 may receive a voice call button 643 from the other user while performing the video call service, and convert the video call service to the voice call service.

When the other user's terminal 630 wants to end the video call service, it may receive a video call end button 645 from the other user.

The user terminal 600 may receive a scroll input 648 on the screen from the user while the video call service is being performed. At this time, as the screen is scrolled up/down through the scroll input 648, the entire conversation content can be expanded or reduced.

While performing a video call service, the user terminal 600 may receive a selected image from the album library 649 or a camera and transmit the selected image to the other user terminal 630 . Here, images can be transmitted up to a preset number of sheets (for example, 5 pictures), and can be transmitted not only during a video call service but also during a voice call service.

6D is an exemplary diagram for explaining a process of converting into user's voice data according to an embodiment of the present invention. Referring to FIG. 6D , the user terminal 600 may receive a selection of a conversation storage box menu 650 from the user. A 'real-time conversation' service or a 'create my voice' service may be provided through the conversation storage box menu 650 .

For example, the user terminal 600 may provide a voice conversation service function through a 'real-time conversation' service when the user is with another user.

When the user terminal 600 receives a real-time conversation icon 660 selected by the user, the user terminal 600 activates the 'deaf mode' 661, receives text 662 from the user, and presses the speak button When 663 is input, the input text may be converted into user voice data. At this time, the user terminal 600 activates the 'hearing mode' 664, receives a voice uttered from another user through the microphone, and when the speech completion button 665 is input, the input utterance is displayed in real time. can be converted to text and displayed.

For another example, the user terminal 600 may generate and store a sentence composed of text frequently used by the user through the 'create my voice' icon 670 to be converted into voice data. For example, when the user terminal 600 receives frequently used text 671 from the user and then presses the 'speak' button 672, the user terminal 600 converts the input text 671 into the user's voice data. can At this time, the user terminal 600 may receive a 'save button' 673 from the user and store the input sentence and voice data corresponding to the sentence. Thereafter, the user terminal can easily output the corresponding sentence as voice data by clicking the play button 674 for the stored sentence.

7A to 7C are exemplary diagrams for explaining a process of receiving a user's voice data service through an app in a user terminal according to an embodiment of the present invention. Here, the voice data generating device 100 manages access rights for each user and generates the user's voice data. It plays a role of checking and may reflect various service logics for providing generated voice data. Also, the voice data generating apparatus 100 may play a role of relaying the user terminal 700 and other user terminals (not shown). For example, the voice data generating apparatus 100 may relay text and synthesized voice files between the user terminal 700 and another user terminal (not shown), and the user terminal 700 and another user terminal ( (not shown) may control a request for video and audio calls between users.

The user terminal 700 may receive a service using the user's voice data generated by the voice data generating device 100 .

Referring to FIG. 7A , the user terminal 700 may execute a voice data service providing app and select a partner 701 to have a conversation with through the launched app. Here, the user may be a deaf person with a congenital disability.

After selecting a partner 701 to have a conversation with, the user terminal 700 may select one of 'my voice conversation' 702, 'video conversation' 703, and 'text'.

Referring to FIGS. 7A and 7B , when the user terminal 700 receives a 'My Voice Conversation' 702 selection from the user, the user terminal 700 wants to have a conversation through the voice data generating device 100. Messages can be exchanged with the counterpart 701 . At this time, the message may be displayed as a text, and may be output as a user's voice generated through the 'My Voice Conversation' 711 . Alternatively, when the video call 712 button is input from the user, the user terminal 700 outputs a call screen 713 for conducting a video call, and A video call 714 may be performed with the counterpart 701 'husband'.

Referring to FIGS. 7A and 7C , when the user terminal 700 selects 'video chat' 703 from the user, the user terminal 700 proceeds with a video call with the 'husband' 721, a conversation partner. A video call 722 may be performed with the 'husband' 721, a conversation partner, through the voice data generating device 100. Thereafter, when the user terminal 700 receives the 'My Voice Conversation' 723 button from the user, the user terminal 700 switches from the video call screen to the chatting screen where 'My Voice Conversation' is possible, and generates voice data. Messages can be exchanged with the husband through the device 100 . At this time, the message may be displayed as a text and simultaneously output as a user's voice generated by the voice data generating apparatus 100 through the 'My Voice Conversation' 724 .

8A and 8B are exemplary diagrams for explaining a process of generating a user's voice data and providing a call service according to an embodiment of the present invention.

Here, the voice data generating device 100 manages access rights for each user, generates user voice data, and when the user's voice data is generated, the voice data generating device 100 displays a result of the generated voice data It plays a role of checking and may reflect various service logics for providing generated voice data. Also, the voice data generating apparatus 100 may play a role of relaying the first user terminals 800 and 820 and the second user terminals 801 and 821 . For example, the voice data generating apparatus 100 may relay text and synthesized voice files between the first user terminals 800 and 820 and the second user terminals 801 and 821, and the first user terminal Requests for video and audio calls between (800, 820) and the second user terminal (801, 821) may be controlled.

8A is an exemplary diagram for explaining a process of providing a call service between a deaf-mute user and a hearing family member according to an embodiment of the present invention. Referring to FIG. 8A , a first user terminal 800 may be a terminal of a daughter who is hearing, and a second user terminal 801 may be a terminal of a mother who is deaf.

The first user terminal 800 may execute the app 802 by receiving an execution input of the voice data service providing app 802 from the hearing daughter. Thereafter, the first user terminal 800 may select a mother 803 as a person to have a conversation with through the app 802 executed by the daughter. For example, the first user terminal 800 selects 'mom' 803 as a target to make a call with the daughter, and selects 'my voice conversation' 804, 'video conversation', or 'text'. Either one can be chosen.

For example, when the first user terminal 800 receives a selection of 'My voice conversation' 804 from the daughter, the first user terminal 800 receives a voice input from the daughter and transmits it to the voice data generating device 100. And, the text converted from the voice input by the voice data generating device 100 may be displayed as a message 805 . For example, when the daughter utters “Mom, school is over,” the first user terminal 800 may display a text converted from the uttered voice as a message 805 .

The second user terminal 801 may receive the converted message 805 from the voice data generating device 100 and display the received message 806 through a chatting window. Thereafter, the second user terminal 801 may receive the contents of the answer to the message 806 from the mother who is deaf and display the contents as a message 807 . For example, the second user terminal 801 may receive an input of an answer to the message 806, such as "Yes, my eldest daughter. My mother is at home."

In the first user terminal 800, the answer content from the second user terminal 801 is converted into text-to-speech (P-TTS) by the voice data generating device 100, and the deaf mother synthesizes it into her own voice. A message 808 including text along with the voiced voice may be received, and the received message 808 may be displayed.

When the first user terminal 800 receives a voice input saying "Can I play with a friend for an hour? Let's play at the playground in front of my house" from a daughter who is hearing, the first user terminal 800 converts the input voice to a voice data generating device. 100, and the text converted from voice by the voice data generating device 100 may be displayed as a message 809.

The second user terminal 801 may receive the converted message 809 from the voice data generating device 100 and display the received message 810 through a chatting window. Thereafter, the second user terminal 801 may receive the content of the answer to the message 810 from the mother who is deaf and display it as a message 811 . For example, the second user terminal 801 may receive a response to the message 810, such as "Okay. Come home by 4:00". At this time, the input answer content is transmitted to the voice data generating apparatus 100 and generated as voice data, so that the deaf mother can listen to the synthesized voice data with her own voice.

The first user terminal 800 receives the text in which the answer content is converted to P-TTS (Text-to-Speech) from the voice data generating device 100 and the voice synthesized by the deaf mother's own voice, and The text along with may be displayed as a message 812.

8B is an exemplary diagram for explaining a process of providing a call service between a first user who is deaf and a second user according to an embodiment of the present invention. Referring to FIG. 8B , a first user terminal 820 may be a terminal of a daughter who is deaf, and a second user terminal 821 may be a terminal of a mother who is deaf.

The first user terminal 820 may execute the app 830 by receiving an execution input of the voice data service providing app 830 from the daughter who is deaf. Thereafter, the first user terminal 820 may select a mother 831 as a person to have a conversation with through the app 830 executed by the daughter. For example, the first user terminal 820 selects 'mom' 831 as a target to have a conversation with the daughter, and then selects 'my voice conversation' 832, 'video conversation', or 'text'. either one can be chosen.

For example, when the first user terminal 820 receives the 'My Voice Conversation' 832 selected from the daughter, the first user terminal 821 may receive a text input from the daughter. For example, the first user terminal 820 may receive the text “Mom, school is over” and display the text as a message 833 .

When the second user terminal 821 receives the message 833 input in the first user terminal 820 from the voice data generating device 100, it may display a message arrival notification 834.

Thereafter, the second user terminal 821 may display the message 835 received through the chatting window, receive an answer to the message 835 from the mother, and display the message 836 . For example, the second user terminal 821 may receive an answer to the message 835, such as "Yes, my eldest daughter. My mom is at home", and transmit the inputted answer to the voice data generating device 100. have. At this time, when the second user terminal 821 receives the message 835 selected from the mother, the second user terminal 821 outputs the selected message 835 as voice data synthesized by the voice data generating apparatus 100 to the voice of the deaf daughter. can

The first user terminal 820 may receive the message 836 input in the second user terminal 821 from the voice data generating device 100 and display the received message 837 . At this time, when the first user terminal 820 receives the message 837 selected from the deaf-mute daughter, the selected message 836 is converted into voice data synthesized by the voice data generating apparatus 100 into the voice of the deaf-mute mother. can be printed out.

Thereafter, the first user terminal 820 may receive input of the text "Can I play with my friend for an hour? Let's play at the playground in front of my house."

The second user terminal 821 displays the message 838 input by the first user terminal 820 from the voice data generating device 100, receives an answer to the message 838, and generates a message 839. can be displayed as

9 is a flowchart of a method of generating voice data in the voice data generating apparatus according to an embodiment of the present invention. The method of generating voice data in the voice data generating apparatus 100 shown in FIG. 9 includes steps that are time-sequentially processed according to the embodiments shown in FIGS. 1 to 8B. Therefore, even if the content is omitted below, it is also applied to the method of generating voice data in the voice data generating apparatus 100 according to the embodiments shown in FIGS. 1 to 8B.

In step S910, the voice data generating apparatus 100 may generate a plurality of family relationship models corresponding to the family relationships of the family members based on the voice data of the plurality of family members.

In step S920, the voice data generating apparatus 100 may receive voice data of a family member related to a user to generate voice data.

In step S930, the voice data generating apparatus 100 may select one family relationship model from among a plurality of family relationship models based on family relationships between the user and family members.

In step S940, the voice data generating device 100 may generate user voice data by inputting voice data of a family member to the selected family relationship model.

In the above description, steps S910 to S940 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted as needed, and the order of steps may be switched.

The method of generating voice data in the voice data generating apparatus described with reference to FIGS. 1 to 9 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. have. Also, the method for generating voice data in the voice data generating apparatus described with reference to FIGS. 1 to 9 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: voice data generating device
110: family relationship model generating unit
120: input unit
130: selection unit
140: generating unit

Claims (10)

A method for providing a voice generation service through a user terminal,
registering deaf information including identifier information of a user who will use the voice generation service;
performing an authentication process for the voice generation service based on the registered deaf-mute information;
performing a call service with another user terminal based on the result of the authentication process; and
Receiving text information or voice while the call service is being performed;
When the call service is provided in the deaf mode according to the result of the authentication process, the text information is converted into voice data based on a family relation model corresponding to the registered deaf person information;
The converted voice data is transmitted to the other user terminal, voice generation service providing method.
According to claim 1,
A first feature vector is derived from voice data of a first family member among the plurality of family members, and a second feature vector is derived from voice data of a second family member;
wherein the family relationship model is generated to correspond to a family relationship between the first family member and the second family member based on the first feature vector and the second feature vector.
According to claim 2,
The family relationship model
The voice generation service providing method is generated by performing voice similarity modeling through the first feature vector and the second feature vector.
According to claim 2,
classifying the family relationships into a plurality of family relationship groups based on any one of age difference values of the plurality of family members, pitch distance values between voice data, and pitch distribution values;
generating a plurality of family relationship group models for the plurality of classified family relationship groups;
Further comprising, a voice generation service providing method.
According to claim 4,
The family relationship model,
wherein one family relationship group model is selected from among the plurality of family relationship group models based on the deaf person information.
In a user terminal providing a voice generation service,
a registration unit for registering deaf information including user identifier information to use the voice generation service;
an authentication process performer performing an authentication process for the voice generation service based on the registered deaf-mute information;
a call service performing unit that performs a call service with other user terminals based on the result of the authentication process; and
An input unit for receiving text information or voice while the call service is being performed;
When the call service is provided in the deaf mode according to the result of the authentication process, the text information is converted into voice data based on a family relation model corresponding to the registered deaf person information;
The converted voice data is transmitted to the other user terminal, the user terminal.
According to claim 6,
A first feature vector is derived from voice data of a first family member among the plurality of family members, and a second feature vector is derived from voice data of a second family member;
The family relationship model is generated based on the first feature vector and the second feature vector to correspond to the family relationship of the first family member and the second family member.
According to claim 7,
The family relationship model
The user terminal, which is generated by performing voice similarity modeling through the first feature vector and the second feature vector.
According to claim 7,
The family relationship is classified into a plurality of family relationship groups based on any one of the age difference value of the plurality of family members, a pitch distance value between voice data, and a pitch distribution value, and a plurality of families for the plurality of family relationship groups. Family relationship model generation unit that creates a relationship group model
To further include, the user terminal.
According to claim 9,
The family relationship model,
The user terminal of claim 1 , wherein one family relationship group model is selected from among the plurality of family relationship group models based on the deaf person information.

Images