KR20230021395A - Simultaenous interpretation service device and method for generating simultaenous interpretation results being applied with user needs - Google Patents

Abstract

Embodiments relate to a simultaneous interpretation device and method. The method includes the steps of: receiving content in the first language; generating a corresponding control command by receiving a user request to be reflected in content in the second language; and reflecting the user request corresponding to the control command to generate audio of content in the second language obtained by simultaneous interpretation of content in the first language.

Description

SIMULTAENOUS INTERPRETATION SERVICE DEVICE AND METHOD FOR GENERATING SIMULTAENOUS INTERPRETATION RESULTS BEING APPLIED WITH USER NEEDS}

Embodiments of the present application relate to a technology for generating a result of simultaneous interpretation of content in a first language with content in a second language, and more particularly, to an apparatus and method for generating and providing a result of simultaneous interpretation reflecting a user's request. related

As is well known, an automatic interpreting device is a device for enabling people who speak different languages to communicate in their native language, and performs voice recognition by receiving a voice signal of a first language and provides the result. After automatic translation into two languages, it is configured to output the result again as voice.

Recently, due to the development of artificial intelligence technology, attempts to apply artificial intelligence technology to automatic interpretation devices are increasing.

However, the conventional simultaneous interpretation service only provides the translation result as a voice or machine sound of a specific voice actor uttered collectively.

Therefore, there is a demand for reflecting various user needs in providing simultaneous interpretation services.

Patent Publication No. 10-2014-0120560 (2014.10.14.)

According to embodiments of the present invention, it is intended to provide an apparatus and method for providing a service by generating a simultaneous interpretation result reflecting a user's request.

A simultaneous interpretation method for generating a simultaneous interpretation result reflecting a user's demand according to an aspect of the present application is performed by a simultaneous interpretation apparatus including a translation unit and a synthesis unit. The simultaneous interpretation method may include: receiving a voice of a speaker who utters content in a first language; converting the received speaker's voice of the content in the first language into text in the first language; extracting a speaker's voice characteristics from the speaker's voice; generating a corresponding control command by receiving a user request to be reflected in a result of simultaneous interpretation; and reflecting a user request corresponding to the control command to generate audio of content in a second language obtained by simultaneous interpretation of content in the first language. The step of generating audio of the simultaneously interpreted content in the second language may include: translating the content in the first language into content in the second language; and generating audio of the content in the second language.

In an embodiment, the receiving the user request and generating the corresponding control command includes: receiving the situation request as the user request and generating a first control command and sending it to the translation unit; and receiving a speaker's voice request, a specific person's voice request, or a reference voice request as a user request, and generating and transmitting a second control command to a synthesis unit. The first control command is a command for the translation unit to translate the content of the first language into the content of the second language under a content situation, and the second control command is a command for the synthesizer to translate the content of the second language into a speaker's voice, characteristics A command to synthesize a human voice or a sample machine voice associated with a reference.

In one embodiment, the step of generating the voice of the content in the second language may include a second-language synthesizing unit synthesizing the speaker's voice with the content of the second language translated by the translation unit when a speaker's voice request is received. Generating 1 control command and transmitting the 2-1 control command to the synthesis unit - the 2-1 control command includes speaker voice characteristic information; and generating a synthesized speaker voice synthesized by synthesizing the speaker's voice with the content of the second language translated by the translation unit based on the speaker's voice characteristic of the 2-1 control command and the content of the second language. there is.

In one embodiment, the step of generating the voice of the content in the second language may include, when receiving a request for a specific voice, the synthesizing unit synthesizes the specific voice with the content in the second language translated by the translation unit. 2 Generating a control command and transmitting the 2-2 control command to the synthesis unit - the 2-2 control command includes pre-stored specific voice characteristic information; and generating a specific synthesized voice obtained by synthesizing a previously stored specific voice with the second language content translated by a translation unit based on the specific voice characteristic of the 2-2 control command and the second language content. may be

In one embodiment, the synthesis unit may generate synthesized speech specific to content of the second language using a specific speech model including a basic speech model and a second tuning model. The basic voice model is a model learned to utter a text in a second language as a basic voice according to a pronunciation rule of the second language, and the second tuning model reflects pre-stored specific voice characteristics to the basic voice to produce a specific synthesized voice. is a model trained to generate

In one embodiment, the step of generating the audio of the content in the second language may further include providing an interface screen for selecting one of one or more pieces of stored specific voice characteristic information. The synthesis unit is configured to use one or more second tuning models for each of one or more specific persons, and synthesizes a specific person by reflecting the voice characteristics of the corresponding specific person to the basic voice of the content in the second language using the second tuning model for the selected specific person. generate voice

In one embodiment, the step of generating the voice of the content in the second language may include, upon receipt of a request for a reference voice, a sample machine voice associated with the reference of the request for a reference voice, and the content in the second language translated by the translation unit. generating a 2-3 control command for synthesizing and transmitting the 2-3 control command to the synthesis unit, wherein the 2-3 control command includes pre-stored sample machine voice characteristic information; and generating a reference synthesized voice synthesized by synthesizing the pre-stored sample machine voice with the content of the second language translated by the translation unit based on the specification of the sample machine voice of the 2-3 control command and the content of the second language. may also include

In one embodiment, the synthesis unit may generate a reference synthesized speech of the content of the second language using a reference speech model including a basic speech model and a third tuning model. The basic voice model is a model learned to utter a text in a second language as a basic voice according to a pronunciation rule of the second language, and the third tuning model reflects pre-stored sample machine voice characteristics to the basic voice for reference. A model trained to generate synthetic speech.

In an embodiment, the generating of the content of the second language may further include providing an interface screen for selecting one of one or more references associated with one or more sample machine voice characteristic information stored. may be The synthesis unit is configured to use one or more third tuning models for each of the sample machine voices associated with the one or more references, wherein the corresponding sample machine speech characteristics are determined by using the third tuning model for the sample machine speeches associated with the selected reference. It is also possible to generate a reference synthesized voice reflected in the basic voice of the 2 languages.

In one embodiment, the voice characteristics are extracted by analyzing an input voice signal in a time domain or a frequency domain, and may include one or more of timbre, voice range, tone, and gender.

In one embodiment, the content context may include one or more of a business context, an academic context, and a daily life context.

In one embodiment, the simultaneous interpretation method characterized in that the sample machine voice characteristic information is stored as a reference template for each associated reference.

In one embodiment, the step of generating the audio of the content in the second language may further include modulating the audio of the content in the second language. The modulated audio of the content in the second language may be generated by adjusting the voice characteristic value of the generated audio of the content in the second language to the input modulation value.

A simultaneous interpretation method for generating a simultaneous interpretation result reflecting a user's demand according to another aspect of the present application is performed by a simultaneous interpretation apparatus including a translation unit and a synthesis unit. The simultaneous interpretation method includes: receiving text of content in a first language; receiving a speaker's voice of a different expression from the contents of the first language; extracting speaker voice characteristics from the speaker's voice; generating a corresponding control command by receiving a user request to be reflected in a result of simultaneous interpretation; and reflecting a user request corresponding to the control command to generate audio of content in a second language obtained by simultaneous interpretation of content in the first language. The step of generating audio of the simultaneously interpreted content in the second language may include: translating the content in the first language into content in the second language; and generating audio of the content in the second language.

A computer readable recording medium according to another aspect of the present application may record a program for performing the simultaneous interpretation method according to the above-described embodiments.

A simultaneous interpretation apparatus according to another aspect of the present application includes: a first input unit acquiring information in a voice form; a second input unit for acquiring information in non-speech form; a text generation unit that converts the audio signal into text when the audio signal is input to the second input unit; a feature extractor extracting voice features from the voice signal; When a user request is input, a control module generating a control command for generating a simultaneous interpretation result reflecting the user request; and a simultaneous interpretation module configured to reflect a user request corresponding to a control command and generate audio of content in a second language obtained by simultaneous interpretation of content in the first language. The simultaneous interpretation module may include: a translation unit that translates content in a first language acquired by the first input unit or the second input unit into content in a second language; and a synthesis unit generating audio of the content in the second language.

An apparatus according to an aspect of the present invention may provide a customized simultaneous interpretation service for each user by controlling the simultaneous interpretation service to reflect user characteristics.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS To describe the technical solutions of the embodiments of the present invention or the prior art more clearly, drawings required in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for the purpose of explaining the embodiments of the present specification and not for the purpose of limitation. In addition, for clarity of explanation, some elements applied with various modifications, such as exaggeration and omission, may be shown in the drawings below.
1 is a schematic block diagram of a simultaneous interpretation device, according to one aspect of the present application.
2A and 2B are schematic diagrams of a process of generating content in a second language in which a content situation is reflected, according to an embodiment of the present application.
3 is a schematic diagram of a process of generating content in a second language in which a speaker's voice request is reflected, according to an embodiment of the present application.
4 is a schematic diagram of a process of generating a speaker voice model according to an embodiment of the present application.
5 is a schematic diagram of a process of generating content in a second language in which a specific person's voice request is reflected, according to an embodiment of the present application.
6 is a schematic diagram of a process of generating a specific voice model according to an embodiment of the present application.
7 is a schematic diagram of a process of generating content in a second language in which a reference voice request is reflected, according to an embodiment of the present application.
8 is a schematic diagram of a process of generating a reference speech model according to an embodiment of the present application.
9 is a flowchart of a method for simultaneous interpretation, according to one aspect of the present application.
10 is a flowchart of a method for simultaneous interpretation when content in a first language is obtained as voice, according to an embodiment of the present application.
11 is a flowchart of a simultaneous interpretation method when content in a second language is acquired as non-voice, according to an embodiment of the present application.
12 is a flowchart of a simultaneous interpretation method when content in a second language is acquired as non-voice, according to another embodiment of the present application.

Hereinafter, with reference to the drawings, look at the embodiments of the present invention in detail.

However, it should be understood that this disclosure is not intended to limit the disclosure to the specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the disclosure. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this specification, expressions such as “has,” “can have,” “includes,” or “can include” refer to corresponding characteristics (eg, numerical values, functions, operations, steps, parts, elements, and/or components). elements), and does not preclude the presence or addition of additional features.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Expressions such as “first,” “second,” “first,” or “second,” used in various embodiments may modify various components regardless of order and/or importance, and the corresponding components do not limit them The above expressions may be used to distinguish one component from another. For example, the first element and the second element may represent different elements regardless of order or importance.

As used herein, the expression “configured to (or configured to)” means, depending on the situation, for example, “suitable for”, “having the capacity to” ,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured (or set) to” may not necessarily mean only “specifically designed to” hardware. Instead, in some contexts, the expression "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or set) to perform A, B, and C" may include a dedicated processor (e.g., embedded processor) to perform those operations, or one or more software programs stored in a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

1 is a schematic block diagram of a simultaneous interpretation device, according to one aspect of the present application.

Referring to FIG. 1 , a device for providing a simultaneous interpretation service reflecting a user's request (hereinafter referred to as “simultaneous interpretation device”, (1)) includes a first input unit 11; text generator 20; feature extraction unit 30; control module 40; simultaneous interpretation module 50; and an output unit 60. The simultaneous interpretation module 50 includes a translation unit 51; and a synthesis unit 55 . In some embodiments, the simultaneous interpretation device 1 may further include a second input unit 12 .

The simultaneous interpretation apparatus 1 according to embodiments may be entirely hardware, entirely software, or partially hardware and partially software. For example, the simultaneous interpretation device 1 may collectively refer to hardware capable of processing data and operating software for driving the same. In this specification, terms such as "unit", "module", "device", or "system" are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing computing device including a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), or another processor. Also, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The first input unit 11 is an input device for obtaining information or user input in the form of voice. The genital first input unit 11 receives information or user input as a voice signal. The input unit 12 may include a microphone that receives sound wave signals and converts them into electrical signals.

The second input unit 12 is an input device for obtaining information or user input in non-speech form. The second input unit 12 may include a touch panel, a keyboard, a button, a mouse, a dial, a switch, and/or a stick.

The simultaneous interpretation apparatus 1 acquires information of a simultaneous interpretation target through the first input unit 11 and/or the second input unit 12 . The information of the simultaneous interpretation target may be referred to as contents.

The content is information expressed in a first language, and may include, for example, conversations, advertisements, and technical literature. The content includes a set of text consisting of words, sentences and/or paragraphs. For example, the content may be a sentence.

The simultaneous interpretation device 1 may obtain audio of content in the first language through the first input unit 11 . The simultaneous interpretation device 1 supplies the acquired voice of the content in the first language to the text generator 20 .

The simultaneous interpretation device 1 may acquire text of content in the first language through the second input unit 12 . When content in the first language is input by the second input unit 12 through a non-voice medium such as typing or pointing, the second input unit 12 generates an electrical signal corresponding to the input information on the non-voice medium. In this way, the text of the content in the first language may be generated. For example, when the user inputs content in the first language through a keyboard, text data of the content in the first language is generated based on an electric signal for each keyboard. The simultaneous interpretation device 1 supplies text of content in the first language from the second input unit 12 to the translation unit 51 .

Also, the simultaneous interpretation apparatus 1 receives user input through the first input unit 11 and/or the second input unit 12 to generate a simultaneous interpretation result reflecting a user request. A process of generating simultaneous interpretation results according to such user input will be described in detail with reference to FIGS. 2 to 8 below.

The text generation unit 20 generates text data of content from the voice signal acquired by the first input unit 11 . The text generation unit 20 performs a process of automatically recognizing and converting a voice signal from a speaker's mouth into a character string.

In certain embodiments, the text generator 20 may include an Automatic Speech Recognition (ASR) or Speech-to-Text (STT) program. When the contents of the first language are obtained by the first input unit 11 in the form of voice information through an audio medium, the text generator 20 converts the audio signal of the contents of the first language through the STT program and outputs Creates text data of content in one language.

The text generation unit 20 supplies text data of the first language to the simultaneous interpretation module 50 .

When a voice signal is input to the first input unit 11, the feature extraction unit 30 extracts voice characteristics from the input voice signal. The voice characteristic is extracted from the voice of the content of the first language to be simultaneously interpreted or the voice of the content different from the content of the first language. When the content of the first language is input as the voice of a speaker of the content of the first language to be simultaneously interpreted by the first input unit 11, the feature extractor 30 may extract the voice characteristics of the speaker.

Also, the feature extractor 30 may extract voice features from a voice signal of a specific person other than the speaker, such as a celebrity or an actor.

In one embodiment, the voice characteristics may include timbre, vocal range, tone and/or gender.

The feature extraction unit 30 extracts voice characteristics of the input voice signal by analyzing the input voice signal in a time domain or a frequency domain. Here, the signal information related to the input voice signal may include a waveform, frequency, period, and/or intensity, but is not limited thereto.

The feature extractor 30 supplies the extracted voice features to the control module 40 .

The control module 40 is configured to perform overall control operations for generating simultaneous interpretation results reflecting user requests.

In one embodiment, the control module 40 may generate a control command for generating a simultaneous interpretation result in which the user's request, that is, the user's request, is reflected. When a user request is input to the simultaneous interpretation device 1, the control module 40 generates a control command to reflect the input user request.

When the simultaneous interpretation device 1 is configured to reflect one or more user requests, the control module 40 may generate one or more control commands.

The control module 40 supplies these control commands to the simultaneous interpretation module 50. In certain embodiments, the control module 40 may supply a control command to the translation unit 51 or synthesis unit 55 .

Control commands of the control module 40 will be described in more detail with reference to FIGS. 2 to 8 below.

The simultaneous interpretation module 50 translates the content in the first language into the content in the second language and generates the translated voice of the content in the second language as a result of simultaneous interpretation.

The translation unit 51 of the simultaneous interpretation module 50 translates content in a first language into content in a second language. Content in the second language has the same context as content in the first language. The text of the first language is translated into the text of the second language by the translation unit 51 .

The translation unit 51 translates the text of the first language into the text of the second language using various translation methods. The translation method may include a rule-based method and/or an artificial intelligence-based method.

The rule-based method is a method of interpreting using language rules established by linguists. In this case, the simultaneous interpretation module may include an interpretation program programmed based on language rules established by linguists.

The artificial intelligence-based method is a method of interpreting using a neural network model. The neural network model is configured to convert text data of a first language into text data of a second language having the same meaning. The neural network model may be implemented in any one structure of a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), a Generative Adversarial Network (GAN), and a combination thereof. not limited

The translation result of the translation unit 51, the content in the second language is supplied to the synthesis unit 55.

The synthesis unit 55 generates audio data for content in the second language. The voice data is voice data in which a user request is reflected.

User needs include contextual needs and/or audio-related needs. The contextual needs are those to improve translation accuracy. The voice-side request is a request to receive a simultaneous interpretation result in a desired voice.

The synthesis unit 55 reflects the contextual requirements and synthesizes the translated second language content with machine voice to generate a synthesized voice of the second language content, and/or voices the translated second language content. A synthesized voice of content in the second language may be generated by synthesizing the voice reflecting the request of the side.

In one embodiment, the context-wise request may include a request to reflect the context of the content.

A single expression in a first language is translatable into multiple expressions in a second language. Whether a particular expression in a first language is appropriate to be translated into which expression in a second language is related to the context of the content. For the same word or sentence, different translation results may be caused depending on the context of the content, such as the atmosphere of conversation and the purpose of the speaker. When a content situation to be translated is selected from among various predefined situations, the simultaneous interpretation device 1 translates the first language content into the second language content under the selected content situation.

The content situation is related to content purpose, content type, speaker's intention, and utterance background. The content context may include, but is not limited to, a business context, an academic context, and/or a daily life context.

2A and 2B are schematic diagrams of a process of generating content in a second language in which a content situation is reflected, according to an embodiment of the present application.

Referring to FIGS. 2A and 2B , the simultaneous interpretation apparatus 1 is configured to receive, as a user request, a content situation to be reflected in translation for simultaneous interpretation. For clarity of explanation, this user request is referred to as a contextual request and described in more detail.

When a contextual request is input, the simultaneous interpretation device 1 obtains contextual information about content in the first language. Then, the control module 40 designates the content status of the first language as the input content status.

When a situation request is input, the control module 40 generates a first control command to cause the translation unit 51 to translate the first language content into the second language content under a designated content situation. The first control command is supplied to the translation unit 51.

In one embodiment, the first control command may include content context information on content in the first language. The translation unit 51 obtains content context information of the user request input through the first control command. Then, the translation unit 51 translates the content in the first language into the content in the second language under the context of the content of the first control command.

A single text may be interpreted in different contexts depending on each situation. The translation unit 51 is configured to translate a single text in a first language into one of texts in a second language for each of a plurality of situations. The translation unit 51 uses the second language text for the corresponding content context as the translated text based on the content context information in the first control command.

For example, as shown in FIG. 2A , when a user request indicating that the content situation in the first language is a business situation is input to the simultaneous interpretation device 1, the control module 40 controls the content in the first language. Designate the content context for the business context. Then, the control module 40 generates a first control command including content status information about the content in the first language and supplies it to the translation unit 51 . The translation unit 51 translates content in a second language in the same context as the content in the first language under a business situation based on a first control command.

Alternatively, as shown in FIG. 2B, when a user request indicating that the content situation in the first language is an academic situation is input to the simultaneous interpretation device 1, the control module 40 controls the content for the content in the first language. Designate the situation as an academic situation. Then, the control module 40 generates a first control command including content status information about the content in the first language and supplies it to the translation unit 51 . Based on the first control command, the translation unit 51 translates the contents of the second language in the same context as the contents of the first language under an academic context.

The simultaneous interpretation device 1 may convert text data of content in the second language into voice data and provide the same to the user. That is, simultaneous interpretation results reflecting only the needs of the context can be provided. This will be described in more detail with reference to FIG. 10 below.

In one embodiment, voice-side requests may include speaker voice requests, specific voice requests, and/or reference voice requests.

The speaker's voice request is a user's request to provide a simultaneous interpretation result in the speaker's voice. When a speaker's voice request is input, the simultaneous interpretation apparatus 1 provides the content of the second language as the voice of the speaker who utters the content of the first language.

The specific person's voice request is a user's request to provide a simultaneous interpretation result in the voice of a specific person. Here, the specific person may be a specific person different from the speaker who utters the simultaneous interpretation target, for example, a celebrity such as a celebrity or an actor. When a specific person's voice request is input as a user request, the simultaneous interpretation device 1 provides the second language content in the specific person's voice.

The reference voice request is a user request to provide a simultaneous interpretation result with a sample machine voice desired by the user. In the reference voice request, a reference indicates a direction, taste, or pattern for a voice desired by the user. The references may be classified by voice characteristics. In this case, the reference refers to a value or group of a specific voice for each voice characteristic, and may be a label.

When voice characteristics include range, tone, timbre and/or gender, the reference may include a reference to range, a reference to tone, and a reference to gender.

The reference for each voice characteristic may include one or more sub-references. The voice may include the first to nth reference voices for each sound band or the first to nth reference voices for each tone. For example, the simultaneous interpretation apparatus 1 may include sample machine voice characteristic information corresponding to three references of "high", "middle", and "low" for the voice characteristics of the voice range.

The reference may be determined averagely or statistically based on the results of responses of people who speak the first language or the second language.

When a user request of a specific reference is input as a reference voice request, the simultaneous interpretation apparatus 1 provides a corresponding sample machine voice corresponding to the reference.

The characteristic information of the machine sample voice corresponding to the reference is stored as a reference template. When a request for a reference voice is input, the simultaneous interpretation apparatus 1 generates a synthesized voice using voice characteristic information of a corresponding reference template.

The sample machine voice may be an average or statistically determined specific voice or group of voices. A sample machine voice may be a representative voice for a group.

For example, it may be calculated as a characteristic value of a sample machine voice corresponding to the reference of “low pitch” by statistically or averagely analyzing the frequency band values of a plurality of voice samples evaluated as low pitch. The simultaneous interpretation apparatus 1 generates and stores a template for “bass” based on the calculated characteristic value of the machine voice.

In some embodiments, the reference may consist of multiple sub-references to the same specific voice characteristic. For example, the simultaneous interpretation apparatus 1 includes a bass reference as a reference for voice characteristics of a voice range, and as sub-references of the same bass reference, a first bass, . . . ., sample machine voice characteristics respectively corresponding to the sub-references of the nth bass.

3 is a schematic diagram of a process of generating content in a second language in which a speaker's voice request is reflected, according to an embodiment of the present application.

Referring to FIG. 3 , when the simultaneous interpretation apparatus 1 receives a speaker's voice request, the control module 40 controls the synthesis unit 55 to convert the speaker's voice into the second language translated by the translation unit 51. A 2-1 control command to be combined with the content is generated. The 2-1 control command is supplied to the synthesis unit 55.

In one embodiment, the 2-1 control command may include the voice characteristic information of the speaker extracted by the characteristic extraction unit 30 . The synthesis unit 55 obtains information related to the speaker's voice through the 2-1 control command.

Then, the synthesis unit 55 generates a synthesized speaker voice reflecting the voice characteristics of the speaker of the 2-1 control command. This synthesized speaker voice is obtained by synthesizing the content of the second language with the voice of the speaker, and is voice data in which the characteristics of the speaker's voice are reflected in the voice of the content of the second language.

In one embodiment, the synthesizer 55 may generate a synthesized speaker voice of the content of the second language using the speaker voice model 540 .

The simultaneous interpretation device 1 inputs the content of the second language generated by the translation unit 51 and the speaker's voice characteristics extracted from the feature extractor 30 to the speaker's voice model 540 to obtain the second language. A speaker synthesized voice of the content may be generated.

To this end, the simultaneous interpretation apparatus 1 stores the speaker's voice model 540 in advance before generating the speaker's voice data. The simultaneous interpretation apparatus 1 generates a speaker's voice model 540 or obtains and stores the speaker's voice model 540 from the outside.

4 is a schematic diagram of a speaker voice model, according to an embodiment of the present application.

Referring to FIG. 4 , a speaker voice model 540 may include a basic voice model 501 and a first tuning model 545 . Here, including the model means including at least a part of the model or having the same parameter value and/or structure as at least a part of the model.

When the content of the second language of the translation unit 51 is input, the speaker voice model 540 inputs the content of the second language into the basic voice model 501 to generate basic voice data of the content of the second language. do.

The basic voice model 501 is a model learned to utter a text in a second language as a basic voice according to a pronunciation rule of the second language. Since the basic voice model outputs voice data of citizens who speak the second language, it may be treated as a second station voice model.

This basic voice may be a representative voice, such as a statistical or average analysis of voices of people who use the second language, or a machine voice. For example, the basic voice may be an average voice of voices of a plurality of training samples below.

The basic speech model 501 uses a plurality of training samples and when text (eg, text data in the second language) recorded in the second language is input, the result of pronunciation in the second language (eg, speech in the second language) is input. data).

Each of the plurality of training samples for the basic speech model 501 includes a pair of text data of the second language and speech data of the second language matching the text data of the second language. Here, the voice data that matches the text data means that the text of the second language is uttered as the voice of the second language. The plurality of training samples may be generated from voice data in which each of a plurality of people utters a single text in the second language with their own voice. That is, the plurality of training samples may have one or more training samples for a single text.

The first tuning model 545 generates a synthesized speaker voice of the content of the second language by reflecting the characteristics of the speaker's voice to the basic voice of the content of the second language.

In one embodiment, the first tuning model 545 may include a conditional auto-regressive model.

The first tuning model 545 is trained to reflect the speaker's voice characteristics to the basic voice. The basic voice and the speaker's voice are implemented by parameters of voice characteristics. The first tuning model 545 is trained to adjust the basic voice characteristic values to the input speaker's voice characteristic values.

In one embodiment, the first tuning model 545 may be trained to reflect speaker voice characteristics to the basic voice of the second language through a cross lingual learning method. The cross language learning method may include, for example, a Selvy deep method, but is not limited thereto.

In this way, when the control module 40 supplies the 2-1 control command according to the input of the speaker's voice request to the synthesis unit 55, the synthesis unit 55 uses the speaker's voice model 540 to generate the second control command. It is also possible to generate speaker synthesized speech data of content in a language.

5 is a schematic diagram of a process of generating content in a second language in which a specific person's voice request is reflected, according to an embodiment of the present application.

Referring to FIG. 5 , when the simultaneous interpretation device 1 receives a specific person's voice request, the control module 40 controls the synthesis unit 55 to convert the specific person's voice into a second language translated by the translation unit 51. A 2-2 control command to be synthesized with the contents of is generated. The 2-2 control command is supplied to the synthesis unit 55.

In one embodiment, the 2-2 control command may include previously stored voice characteristic information of a specific person. The specific person's voice characteristic information may be extracted from the specific person's voice by the feature extraction unit 30 and stored in advance.

Then, the synthesis unit 55 generates a synthesized voice that reflects the specific voice information of the 2-2 control command. The synthesized voice of a specific person is obtained by synthesizing the content of the second language with the voice of the specific person, and is audio data in which voice characteristics specific to the voice of the content of the second language are reflected.

In one embodiment, the synthesis unit 55 may generate synthesized speech specific to the content of the second language using the specific speech model 560 .

The simultaneous interpretation device 1 inputs the content of the second language generated by the translation unit 51 and the specific voice characteristic information stored in advance to the specific voice model 560 to generate synthesized voice specific to the content of the second language. can also create

To this end, the simultaneous interpretation apparatus 1 stores a specific voice model 560 in advance before generating a specific synthesized voice. The simultaneous interpretation device 1 generates a specific voice model 560 or obtains and stores it from the outside.

6 is a schematic diagram of a process of generating a specific voice model according to an embodiment of the present application.

Referring to FIG. 6 , a specific voice model 560 may include a basic voice model 501 and a second tuning model 565 .

When the content of the second language of the translation unit 51 is input, the specific person voice model 560 inputs the content of the second language into the basic voice model 501 to generate basic voice data of the content of the second language. do.

The basic voice model 501 is a model learned to output a result of uttering text in the second language as voice according to the pronunciation rules of the second language, and has been described above with reference to FIG. 4, so detailed descriptions will be omitted. .

The second tuning model 565 reflects voice characteristics specific to the basic voice of the content in the second language to generate synthesized voice specific to the content in the second language.

In one embodiment, the second tuning model 565 may include a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), or a Generative Adversarial Network (GAN). In addition to this, the second tuning model 565 may include a neural network having a different structure.

The second tuning model 565 is learned to reflect voice characteristics specific to the base voice. The basic voice and the specific voice are implemented by parameters of voice characteristics. The second tuning model 565 is trained to adjust basic voice characteristic values to input specific voice characteristic values.

The second tuning model 565 is a learning algorithm that can be used for neural network structures such as convolution neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory models (LSTMs), and generative adversarial networks (GANs). It is learned to reflect speech characteristics specific to the basic speech data of the content of the second language using .

In this way, when the 2-2 control command according to the specific voice request input is supplied to the synthesis unit 55 by the control module 40, the synthesis unit 55 uses the specific voice model to generate the content of the second language. It is also possible to generate synthetic speech data for a specific person.

In one embodiment, the simultaneous interpretation device 1 receives a user input for selecting one of a plurality of specific persons, and transmits the second language content generated by the translation unit 51 to the selected specific person's voice. and synthesized speech specific to the second language content may be generated.

The simultaneous interpretation apparatus 1 may store one or more specific voice characteristic information. The simultaneous interpretation device 1 may induce a user input by providing an interface screen M2 for selecting one of one or more pieces of stored specific voice characteristic information.

The simultaneous interpretation device 1 generates a synthesized voice of the selected person of the content in the second language based on the content in the second language and the voice characteristics of the selected person.

In some embodiments, the simultaneous interpretation device 1 may include a plurality of second tuning models 565 for each of a plurality of specific persons. For example, for n specific persons, the simultaneous interpretation apparatus 1 has a 2-1 tuning model 565-1, . . . ., 2-n tuning models 565-n. Each second tuning model 565 is learned to adjust the voice characteristic value of the input voice (eg, basic voice) to the corresponding specific voice characteristic value.

7 is a schematic diagram of a process of generating content in a second language in which a request for a reference voice is reflected, according to an embodiment of the present application.

Referring to FIG. 7 , when the simultaneous interpretation device 1 receives a request for a reference voice, the control module 40 converts the synthesizer 55 into a second language translated by the translator 51 into the sample machine voice. A 2-3 control command to be synthesized with the contents of is generated. The 2-3 control command is supplied to the synthesis unit 55.

In one embodiment, the 2-3 control commands may include pre-stored sample machine voice characteristic information. As described above, the sample machine voice characteristic information may be characteristic information resulting from average or statistical analysis of voices of a plurality of people.

In some embodiments, the simultaneous interpretation device 1 may store the sample machine voice characteristic information as a reference template in advance. The reference template is generated by using a plurality of people who speak the second language as samples, and includes corresponding sample machine voice characteristic information.

In one embodiment, the synthesizer 55 may generate a reference synthesized voice of the content of the second language using the reference voice model 580 . This reference synthesized voice is voice data obtained by synthesizing the content of the second language with the sample machine voice associated with the reference of the reference voice request.

The simultaneous interpretation device 1 inputs the contents of the second language generated by the translation unit 51 and the pre-stored sample machine voice characteristic information (eg, a reference template) into the reference speech model 580 to input the second language content to the reference speech model 580. It is also possible to generate a reference synthesized voice of the content of

To this end, the simultaneous interpretation apparatus 1 stores the reference voice model 580 in advance before generating the reference synthesized voice. The simultaneous interpretation apparatus 1 generates a reference speech model 580 or obtains and stores it from the outside.

8 is a schematic diagram of a process of generating a reference speech model according to an embodiment of the present application.

Referring to FIG. 8 , a reference voice model 580 may include a basic voice model 501 and a third tuning model 585 .

When the content of the second language generated by the translation unit 51 is input, the reference speech model 580 inputs the content of the second language to the basic speech model 501 to provide a basic voice of the content of the second language. generate data

The basic voice model 501 is a model learned to output a result of uttering text in the second language as voice according to the pronunciation rules of the second language, and has been described above with reference to FIG. 4, so detailed descriptions will be omitted. .

The third tuning model 585 generates a reference synthesized voice of the content of the second language by reflecting sample machine voice characteristics to the basic voice of the content of the second language.

In one embodiment, the third tuning model 585 may include Gaussian mixture model (GMM), restricted Boltzmann machines (RBM), deep neural network (DNN), CycleGAN, or StarGAN.

In another embodiment, the third tuning model 585, like the second tuning model 565, is a Convolution Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), or GAN. (Generative Adversarial Network) structure.

The third tuning model 585 is learned to reflect voice characteristics specific to the base voice. Basic voices and sample machine voices are implemented by parameters of voice characteristics. The third tuning model 585 is trained to adjust basic voice characteristic values to input sample machine voice characteristic values.

The third tuning model 585 is a learning algorithm that can be used for neural network structures such as convolution neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory models (LSTMs), and generative adversarial networks (GANs). It is learned to reflect the sample machine speech characteristics to the basic speech data of the content of the second language using

In this way, when the second-third control command according to the reference speech request is supplied to the synthesis unit 55 by the control module 40, the synthesis unit 55 uses the reference speech model to sample the contents of the second language. It is also possible to generate machine synthesized voice data.

In one embodiment, the simultaneous interpretation device 1 receives a user input for selecting one of a plurality of sample machine voices, and converts the content of the second language generated by the translation unit 51 to the selected sample. It is also possible to generate a reference synthesized voice of content in the second language by synthesizing it with the machine voice.

The simultaneous interpretation apparatus 1 may store one or more sample machine voice characteristic information. The simultaneous interpretation device 1 may store one or more reference templates associated with each of one or more sample machine voices. The simultaneous interpretation apparatus 1 may induce a user input by providing an interface screen M2 for selecting one of one or more stored pieces of sample machine-voice characteristic information.

The simultaneous interpretation device 1 generates a reference synthesized voice of the content of the second language based on the content of the second language and a reference template associated with the selected sample machine voice.

In some embodiments, the simultaneous interpretation device 1 may include a plurality of third tuning models 283 for each of a plurality of sample machine voices. For example, for n samples of machine speech, the simultaneous interpretation apparatus 1 uses a 3-1 tuning model 585-1, . . . ., 2-n tuning models 585-n. Each of the second tuning models 585 is learned to adjust voice characteristic values of an input voice (eg, basic voice) to corresponding sample machine voice characteristic values.

In addition, the simultaneous interpretation apparatus 1 may additionally modulate the generated synthesized voice.

In certain embodiments, the simultaneous interpretation apparatus 1 may modulate the synthesized voice by adjusting voice characteristic values of the generated synthesized voice. When voice characteristic values (ie, modulation values) of the modulated synthetic voice are input to the simultaneous interpretation apparatus 1, the voice characteristic values are adjusted to the modulation values.

A parameter value of a voice characteristic of the synthesized voice, that is, already generated voice data is an initial value, and a parameter value of a voice characteristic of the final voice data is a modulation value.

In one embodiment, the simultaneous interpretation apparatus 1 may provide the user with an interface screen M3 for fine control, which displays voice characteristic values of the generated synthesized voice. The interface screen M3 may be configured to obtain a modulation value based on a relative position or directly obtain a modulation value as a number.

For example, when the synthesized speaker voice is generated, the simultaneous interpretation apparatus 1 may provide the interface screen M3 of FIG. 3 for displaying at least some voice characteristic values of the synthesized speaker voice.

When a synthesized voice of a specific person is generated, the simultaneous interpretation apparatus 1 may provide the interface screen M3 of FIG. 5 for displaying at least some voice characteristic values of the generated synthesized voice of a specific person.

When the reference synthesized voice is generated, the simultaneous interpretation apparatus 1 may provide the interface screen M3 of FIG. 7 for displaying at least some voice characteristic values of the generated reference synthesized voice.

Through this interface screen M3, the simultaneous interpretation apparatus 1 interacts with the user to acquire modulation values of voice characteristics.

The control module 40 transmits an adjustment command including modulation value information to the synthesis unit 55. For example, when the interface screen M3 displaying the voice characteristic values of the synthesized voice of FIG. 3 is displayed, the control module 400 generates an adjustment command including input modulation value information. The modulation value information includes a voice characteristic to be adjusted and parameter values of the voice characteristic to be adjusted.

The synthesis unit 55 adjusts voice characteristic values of the generated synthesized voice based on the control command. Then, the synthesis unit 55 modulates the synthesized voice into a voice corresponding to the modulation value.

In one embodiment, synthesizer 55 may modulate the synthesized voice using a modulation algorithm. A modulated voice is implemented by parameters of voice characteristics having adjusted modulation values.

The modulation algorithm may include a first modulation algorithm using parallel data or a second modulation algorithm using non-parallel data. Here, parallel data means that voice data before and after modulation originate from the voice of the same person.

In one embodiment, the first modulation algorithm may adjust a parameter value of a speech characteristic of synthesized speech to a corresponding modulation value when a modulation value is input after performing an alignment operation in units of frames.

For example, the first modulation algorithm may be a dynamic time warping (DTW) or an iterative combination of a nearest neighbor search step and a conversion step alignment (INCA) algorithm.

In another embodiment, the first modulation algorithm may adjust an initial value of a parameter of a voice characteristic to a modulation value using a learning model.

This learning model may be similar to third tuning model 585 . For example, the learning model of the first modulation algorithm may include Gaussian mixture model (GMM), restricted Boltzmann machines (RBM), and deep neural network (DNN). A learning model of the first modulation algorithm is learned using parallel data.

In one embodiment, the second algorithm may adjust an initial value of a parameter of a voice characteristic to a modulation value using a learning model.

This learning model may be similar to third tuning model 585 . For example, the learning model of the second modulation algorithm may include CycleGAN or StarGAN. The learning model of the second modulation algorithm is learned using non-parallel data.

The learning model of the second modulation algorithm, including the Cycle GAN, is trained to modulate the voices of two people in both directions, thereby learning to adjust initial values of parameters of voice characteristics to modulation values.

The learning model of the second modulation algorithm including the Star GAN is trained to simultaneously output a plurality of voice data using a mask vector, thereby learning to adjust initial values of parameters of voice characteristics to various modulation values. The Star GAN is learned using training sample sets for multiple domains. The training sample set is subsetted by domain. Each training sample included in the subset of training samples for each domain includes voice data, a domain label, and a mask vector, respectively. The domain label is used by the learning model to learn to change voice data under the target domain. The mask vector specifies the domain to focus on during learning. Labels that are not specified are ignored during training, and labels that are specified are focused during training.

In one embodiment, a domain may be negative in a training sample set for multiple domains. That is, the learning model of the second modulation algorithm is learned using voice data subdivided for each person as an individual data set. A mask vector is added to the training samples, so that the learning model of the second modulation algorithm can perform combined learning between domains of different datasets.

In addition to this, the synthesis unit 55 may generate a modulated synthesized voice by adjusting an initial value of a voice characteristic parameter of the synthesized voice to a modulation value using another voice modulation technique.

The output unit 60 outputs audio data for content in the second language in which the user's request is reflected. The output unit 60 may include a device such as a speaker that converts an electrical signal into a voice signal and outputs the converted audio signal.

The output unit 60 may output audio of content in a second language in which a situation request is reflected.

Alternatively, the output unit 60 may output synthesized voice of content in the second language in which a speaker's voice request, a specific person's voice request, or a reference voice request is reflected.

Also, the output unit 60 may output the modulated synthesized voice.

It will be clear to those skilled in the art that the simultaneous interpretation device 1 may include other components not described herein. For example, it may further include a data input device, an output device such as a display and/or print, a storage device such as a memory, a network, a network interface, and/or a protocol.

A simultaneous interpretation method according to another aspect of the present invention is performed by a computing device including a processor. The computing device may be the simultaneous interpretation device 1 or other data processing device such as a computer. Hereinafter, for clarity of explanation, the simultaneous interpretation method will be described in more detail based on the embodiments performed by the simultaneous interpretation apparatus 1 of FIG. 1 .

9 is a flow diagram of a method for simultaneous interpretation, according to one aspect of the present application.

Referring to FIG. 9 , the simultaneous interpretation method includes: receiving content in a first language (S10); generating a corresponding control command by receiving a user request to be reflected in content in the second language (S40); and reflecting a user request corresponding to the control command to generate audio of content in a second language obtained by simultaneous interpretation of content in the first language (S50). Also, the simultaneous interpretation method includes outputting audio of content in the second language (S60).

In this simultaneous interpretation method, audio of content in a second language obtained by simultaneous interpretation of content in a first language is generated according to an input method for content in a first language and a user request.

10 is a flowchart of a simultaneous interpretation method when content in a first language is acquired as voice, according to an embodiment of the present application.

Referring to FIG. 10 , the simultaneous interpretation method includes: receiving (eg, through the first input unit 11) a speaker's voice of content in a first language spoken by the speaker (S1010); converting the received speaker's voice of the content in the first language into text in the first language (S1020); receiving a user request to be reflected in the simultaneous interpretation result and generating a corresponding control command (S1040); and reflecting a user request corresponding to the control command to generate audio of content in a second language obtained by simultaneous interpretation of content in the first language (S1050). The audio of the content in the second language may be a voice in which a user request is reflected.

The step (S1050) includes: translating the content in the first language into the content in the second language (S1051) and generating audio data of the content in the second language (S1055).

In certain embodiments, the simultaneous interpretation method of FIG. 10 may further include extracting the speaker's voice characteristics from the speaker's voice (eg, by the feature extraction unit 30) (S1030).

In steps S1010 and S1020, the speaker's voice and content information of the first language are obtained.

In step S1030, signal information related to the input voice signal on the time domain or frequency domain is analyzed to extract the voice characteristics of the input voice signal. Here, the signal information related to the input voice signal may include a waveform, frequency, period, and/or intensity, but is not limited thereto.

In one embodiment, the voice characteristics may include timbre, vocal range, tone and/or gender.

In one embodiment, the step (S1040) includes: receiving the situation request as a user request before step (S1051) and generating and transmitting a first control command to the translation unit 51 (S1041); and/or receiving a speaker's voice request, a specific person's voice request, or a reference voice request as a user request before step S1055 and generating and transmitting a second control command to the synthesis unit 55 (S1045). .

When a contextual request is input in step S1041, the content in the first language is translated into the content in the second language based on the content context of the contextual request.

In one embodiment, in the step of translating into text in the second language (S1051), the text of the content in the first language is converted into the text in the content in the second language through a rule-based method and/or an artificial intelligence-based method. It may also include a translation step.

The text of the content in the second language is converted into audio of the content in the second language (S1055).

In step S1041, the situation request may be received by a user input for selecting one of a plurality of pre-stored content situations.

In one embodiment, the step (S1041) includes: providing an interface screen configured to select any one of a plurality of pre-stored content situations; and generating a first control command including the selected content status information and transmitting the first control command to the translation unit 51 when an input for selecting one content status is received. The first control command is a command for translating content in a first language into content in a second language under a selected content situation. The selected content context represents the content context of the first language.

Then, the translation unit 51 translates the text of the content in the first language into the text of the content in the second language in step S1020 under the context of the content (S1051). This translation result is translated specifically for the specified content situation.

In step S1045, the speaker's voice request, the specific person's voice request, or the reference voice request may be received by a user input for selecting one of the speaker's voice request, the specific person's voice request, and the reference voice request as the user request.

In one embodiment, the step S1041 includes: providing an interface screen M1 configured to select any one of the speaker's voice request, the specific person's voice request, and the reference voice request as a user request; if an input for selecting a speaker's voice request is received (eg, by the control module 40), generating a 2-1 control command including the speaker's voice characteristic information; generating a 2-2 control command including pre-stored voice characteristic information of the selected specific person when an input for selecting a specific person's voice request is received; generating a 2-3 control command including pre-stored sample machine voice information when an input for selecting a reference voice request is received; A step of transmitting the generated 2-1 control command, 2-2 control command or 2-3 control command to the synthesis unit 55 may be included.

The 2-1 control command is a command for synthesizing the translated second language content with the speaker's voice in order to provide a simultaneous interpretation result in which the speaker's voice characteristics are reflected. The speaker's voice characteristic information is extracted from the voice signal of the content of the first language spoken by the speaker, and is obtained from the feature extractor 30 .

The 2-2 control command is a command for synthesizing the translated content in the second language with the voice of the selected specific person in order to provide a simultaneous interpretation result in which the voice characteristics of the specific person are reflected. The voice characteristic information of a specific person is pre-stored in the simultaneous interpretation apparatus 1, and is voice characteristic information extracted from one or more voice signals of a specific person.

The simultaneous interpretation method may be used to reflect the voice of a specific person selected from among a plurality of specific persons.

In one embodiment, the step (S1041) includes providing an interface screen M1 configured to select any one of the speaker's voice request, the specific person's voice request, and the reference voice request as a user request, and then selecting the specific person's voice request. When a user input is obtained, the method may further include providing an interface screen M2 for inducing selection of one of a plurality of pre-stored selectable specific persons. For example, when a specific person's voice request is selected on the interface screen M1 of FIG. 5 , an interface screen M2 displaying a specific person having a selectable voice may be provided.

The 2-3 control command is a command for synthesizing the translated second language content with the selected sample machine voice to provide a simultaneous interpretation result in which the sample machine voice characteristics are reflected. The sample machine voice characteristic information is previously stored in the simultaneous interpretation apparatus 1, and is voice characteristic information extracted from one or more specific person's voice signals.

The simultaneous interpretation method may be used to reflect a selected sample machine voice from among a plurality of sample machine voices.

In one embodiment, the step (S1041) includes providing an interface screen M1 configured to select any one of the speaker's voice request, the specific person's voice request, and the reference voice request as a user request, and then selecting the specific person's voice request. When the user input is obtained, the method may further include providing an interface screen M2 for inducing selection of one of a plurality of pre-stored selectable sample machine voices. For example, if a specific person's voice request is selected on the interface screen M1 of FIG. 7 , an interface screen M2 displaying a specific person having a selectable voice may be provided.

When there is no user request in terms of voice (for example, when there is only a first control command), a voice for the content in the second language is generated from the text of the content in the second language using the basic voice model 501 (S1055). ).

When there is a user request in terms of voice (eg, when there is a second control command), the voice for the second language content (i.e., , synthetic voice) is generated (S1055). The second control command is a command for the synthesis unit to synthesize the second language content with a speaker's voice, a characteristic voice, or a sample machine voice associated with a reference.

For example, the synthesis unit 55 obtains speaker voice-related information including speaker voice characteristic information through the 2-1 control command. The synthesis unit 55 generates a synthesized speaker of the content of the second language based on the content of the second language and the speaker's voice characteristic information in step S1051 (S1055).

In one embodiment, the synthesizer 55 may generate a synthesized speaker voice of the content of the second language using the speaker voice model 540 .

The 2-1 control command and the speaker voice model 540 for generating a synthesized speaker voice based on the control command have been described above with reference to FIGS. 3 and 4, and detailed description thereof will be omitted.

Alternatively, the synthesis unit 55 obtains information related to the specific person's voice including the specific person's voice characteristic information through the 2-2 control command. The synthesis unit 55 generates synthesized speech specific to the content in the second language based on the content in the second language and the specific voice characteristic information in step S1051 (S1055).

In one embodiment, the synthesis unit 55 may generate synthesized speech specific to the content of the second language using the specific speech model 560 .

The process of generating the 2-2 control command and the specific person's voice model 560 for generating the specific person's voice data based on the 2-2 control command have been described above with reference to FIGS. 5 and 6, so detailed descriptions are omitted. do.

Alternatively, the synthesis unit 55 obtains information related to a specific person's voice including sample machine voice characteristic information through the 2-3 control command. The synthesis unit 55 generates a reference synthesized voice of the content of the second language based on the content of the second language and the sample machine voice characteristic information in step S1051 (S1055). As described above, the reference synthetic voice of the content in the second language is audio data obtained by synthesizing the content in the second language and the sample machine voice having the voice characteristics of the 2-3 control commands.

In one embodiment, the synthesizer 55 may generate a reference synthesized voice of the content of the second language using the reference voice model 580 .

The 2-3 control command and the reference speech model 580 generating the reference synthesized speech based thereon have been described above with reference to FIGS. 7 and 8 , and detailed description thereof will be omitted.

The simultaneous interpretation method includes: outputting audio of the second language content generated in step S1050 (S1060).

The simultaneous interpretation method includes: outputting the voice data of step S1050 as a voice signal (eg, by the output unit 60) (S1060).

Also, the simultaneous interpretation method may further include storing the voice data of step S1050 (S970).

In addition, the step of outputting the audio signal (S1060) may further include: listening to audio data of content in the second language in advance. Here, pre-listening is to output a part of the audio data as a voice signal. The partial section may be an initial section of the entire section of the synthesized voice, but is not limited thereto.

In addition, the step (S1050) includes a step (S1057) of modulating the audio (eg, synthesized voice) of the second language content generated in the step (S1055) before outputting the audio signal in the step (S1060). may include more.

When the adjustment value of the audio characteristics of the content in the second language is input, the audio of the content in the second language having the input adjustment value is generated (S1057). For example, if the adjustment value is input after generating the synthesized voice of the content in the second language according to the second control command, the modulated synthesized voice of the content in the second language is generated (S1057).

In one embodiment, the step (S1050) may further include: providing an interface screen for inducing an input of an adjustment value (S1056). For example, the interface screen M3 of FIGS. 3, 5, and 7 may be provided for inputting an adjustment value (S1056).

Since the process of generating the modulated voice (eg, modulated synthesized voice) has been described above with reference to FIGS. 3 to 8 , a detailed description thereof will be omitted.

11 is a flowchart of a simultaneous interpretation method when content in a second language is acquired as non-voice, according to an embodiment of the present application.

Since the simultaneous interpretation method of FIG. 11 is similar to the simultaneous interpretation method of FIG. 10, the differences will be mainly described.

Referring to FIG. 10 , the simultaneous interpretation method includes: receiving text of content in a first language (eg, through the second input unit 12) (S1110); generating a corresponding control command by receiving a user request to be reflected in content in the second language (S1140); and reflecting a user request corresponding to the control command to generate audio of the content in the second language obtained by simultaneous interpretation of the content in the first language (S1150). Also, in certain embodiments, the simultaneous interpretation method may include: outputting audio of the content in the second language generated in step S1150 (S1160).

In step S1110, content information of the first language is obtained. For example, in step S1110, text data of content in the first language is obtained.

The step (S1140) includes: receiving the situation request as a user request before step (S1150), generating a first control command, and transmitting it to the translation unit 51 (S1141); and/or receiving a specific voice request or reference voice request as a user request before step S1155 and generating and transmitting a second control command to the synthesis unit 55 (S1145).

The step (S1150) includes: translating the first language content into the second language content (S1151) and generating audio data of the second language content (S1155).

When a contextual request is input (S1141), the first language content is translated into the second language content under the content context of the contextual request (S1151).

In step S1155, audio of content in the second language is generated.

When a specific person's voice request is input, a synthesized voice may be generated by synthesizing the specific person's voice and the content of the second language (S1155).

When a reference voice request is input, a synthesized voice obtained by synthesizing the corresponding sample machine voice and the contents of the second language may be generated (S1155).

When the contextual request and the audio aspect request are input, synthesized voice of the second language content translated under the context of the content may be generated (S1155).

In this way, when only the text of the first language content is input, a simultaneous interpretation result reflecting a specific person's voice or sample machine voice may be provided.

In addition, when only the contextual request is input, the voice of the content in the second language translated under the context of the content may be generated (S1155). Here, the text of the content in the second language is converted into a voice set for a case in which a voice aspect request is not input. The set voice may be a basic voice output using the basic voice model 501 . Then, the content of the second language translated under the content situation is output as a basic voice (S1160).

12 is a flowchart of a simultaneous interpretation method when content in a second language is acquired as non-voice, according to another embodiment of the present application.

Since the simultaneous interpretation method of FIG. 12 is similar to a combination of at least a part of the simultaneous interpretation method of FIG. 10 and at least a part of the simultaneous interpretation method of FIG. 11, the differences will be mainly described.

The simultaneous interpretation method may include: receiving text of content in a first language (eg, through the second input unit 12) (S1210); inputting a speaker's voice (eg, by the first input unit 11) (S1211); extracting the speaker's voice characteristics from the speaker's voice (eg, by the feature extraction unit 30) (S1230); generating a corresponding control command by receiving a user request to be reflected in content in the second language (S1240); and generating voice data obtained by simultaneously interpreting the contents of the first language into the contents of the second language by reflecting the user request corresponding to the control command (S1250). Also, in specific embodiments, the simultaneous interpretation method may include: outputting audio of the content in the second language generated in step S1250 (S1260).

In step S1210, content information of the first language is obtained, and in step S1211, the speaker's voice is obtained. That is, the simultaneous interpretation target and the speaker's voice are input through different input means.

The speaker's voice in step S1211 is a voice different from the one in which the speaker uttered the entire contents of the first language. This speaker's voice may be a voice that utters a part of the content in the first language or another expression not included in the content in the first language.

The step (S1240) includes: receiving the situation request as a user request before step (S1250), generating a first control command, and transmitting the generated first control command to the translation unit 51 (S1241); and/or receiving a speaker's voice request, a specific person's voice request, or a reference voice request as a user request before step S1255 and generating and transmitting a second control command to the synthesis unit 55 (S1245).

Since the step of generating the audio of the second language content of FIG. 12 ( S1250 ) is similar to the step of generating the audio of the second language content of FIG. 10 ( S950 ), a detailed description thereof will be omitted.

In this way, when the text of the first language content and the speaker's voice are input, a simultaneous interpretation result reflecting the speaker's voice may be provided.

The operations of the simultaneous interpretation apparatus 1 and method according to the above-described embodiments may be at least partially implemented as a computer program and recorded on a computer-readable recording medium. For example, implemented together with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor to perform any or all steps, operations, or processes described.

The computer may be any device that may be integrated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device that has one or more alternative and special purpose processors, memory, storage, and networking components (whether wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, Apple's OS X or iOS, a Linux distribution, or an operating system such as Google's Android OS.

The computer-readable recording medium includes all types of recording and identification devices in which data readable by a computer is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage and identification devices, and the like. In addition, computer-readable recording media may be distributed in computer systems connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

The present invention reviewed above has been described with reference to the embodiments shown in the drawings, but this is only exemplary, and those skilled in the art will understand that various modifications and variations of the embodiments are possible therefrom. However, such modifications should be considered within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

Claims (16)

A simultaneous interpretation method for generating a simultaneous interpretation result reflecting a user's request, performed by a simultaneous interpretation apparatus including a translation unit and a synthesis unit, comprising:
receiving a voice of a speaker who uttered content in a first language;
converting the received speaker's voice of the content in the first language into text in the first language;
extracting a speaker's voice characteristics from the speaker's voice;
generating a corresponding control command by receiving a user request to be reflected in a result of simultaneous interpretation; and
generating audio of content in a second language obtained by simultaneously interpreting content in a first language by reflecting a user request corresponding to a control command;
Generating the audio of the simultaneously interpreted second language content includes:
translating the content in the first language into content in a second language; and
A method of simultaneous interpretation comprising generating audio of content in a second language.
The method according to claim 1, wherein receiving the user request and generating a corresponding control command comprises:
receiving the context request as a user request and generating and sending the first control command to the translation unit; and
Receiving a speaker's voice request, a specific person's voice request, or a reference voice request as a user request, and generating and transmitting a second control command to a synthesis unit.
includes at least one of
the first control command is a command for the translation unit to translate content in a first language into content in a second language under a content situation;
The second control command is a command for causing the synthesis unit to synthesize the contents of the second language with a speaker's voice, a characteristic voice, or a sample machine voice associated with a reference.
The method according to claim 2, wherein the generating of the audio of the second language content comprises:
When receiving a speaker voice request, the synthesis unit generates a 2-1 control command for synthesizing the speaker's voice with content of a second language translated by the translation unit, and transmits the 2-1 control command to the synthesis unit. step - the 2-1 control command includes speaker voice characteristic information; and
and generating a synthesized speaker voice by synthesizing the speaker's voice with the content of the second language translated by the translation unit based on the speaker's voice characteristic of the 2-1 control command and the content of the second language. method.
The method according to claim 2, wherein the generating of the audio of the second language content comprises:
When receiving a request for a specific voice, the synthesis unit generates a 2-2 control command for synthesizing the specific voice with content in a second language translated by the translation unit, and transmits the 2-2 control command to the synthesis unit. step - the 2-2 control command includes pre-stored specific voice characteristic information; and
and generating a synthesized specific voice synthesized by synthesizing a pre-stored specific voice with the second language content translated by a translation unit based on the specific voice characteristic of the 2-2 control command and the content of the second language. Interpretation method.
The method of claim 4,
The synthesis unit generates synthesized speech specific to the contents of the second language using specific speech models including a basic speech model and a second tuning model;
The basic voice model is a model learned to utter text in a second language as a basic voice according to pronunciation rules of the second language,
The second tuning model is a model learned to generate a specific synthetic voice by reflecting pre-stored specific voice characteristics to the basic voice.
The method of claim 5,
Generating the audio of the content in the second language,
Providing an interface screen for selecting any one of one or more specific voice characteristic information stored; further comprising,
The synthesis unit is configured to use one or more second tuning models for each of one or more specific persons, and synthesizes the specific person by reflecting the voice characteristics of the specific person to the basic voice of the content in the second language using the second tuning model for the selected specific person. A method of simultaneous interpretation characterized by generating voice.
The method according to claim 2, wherein the generating of the audio of the second language content comprises:
When a reference voice request is input, a 2-3 control command for synthesizing the sample machine voice associated with the reference of the reference voice request with the content of the second language translated by the translation unit is generated, and the synthesis unit generates the 2-3 control command. 3 sending a control command, wherein the 2-3 control command includes pre-stored sample machine voice characteristic information; and
and generating a reference synthesized voice synthesized by synthesizing a pre-stored sample machine voice with the second language content translated by a translation unit based on the second language content and the sample machine voice specification of the 2-3 control command. simultaneous interpretation method.
The method of claim 7,
The synthesis unit generates a reference synthesized speech of content of the second language using a reference speech model including a basic speech model and a third tuning model;
The basic voice model is a model learned to utter text in a second language as a basic voice according to pronunciation rules of the second language,
The third tuning model is a model learned to generate the reference synthesized voice by reflecting pre-stored sample machine voice characteristics to the basic voice.
The method of claim 5,
Generating the audio of the content in the second language,
Further comprising providing an interface screen for selecting any one of the one or more references associated with the stored one or more sample machine speech characteristic information;
The synthesis unit is configured to use one or more third tuning models for each of the sample machine voices associated with the one or more references, wherein the corresponding sample machine speech characteristics are determined by using the third tuning model for the sample machine speeches associated with the selected reference. A simultaneous interpretation method characterized by generating a reference synthesized voice reflected in a basic voice of two languages.
The method of claim 1,
The voice characteristics are extracted by analyzing an input voice signal in a time domain or a frequency domain, and include at least one of timbre, vocal range, tone, and gender.
The method of claim 2,
The simultaneous interpretation method according to claim 1 , wherein the content context includes at least one of a business context, an academic context, and a daily life context.
The method of claim 1,
wherein the sample machine voice characteristic information is stored as a reference template for each associated reference.
The method according to claim 1, wherein the generating of the audio of the second language content comprises:
Modulating the voice of the generated second language content;
The method of simultaneous interpretation characterized in that the modulated voice of the content in the second language is generated by adjusting the voice characteristic value of the generated voice of the content in the second language to the input modulation value.
A simultaneous interpretation method for generating a simultaneous interpretation result reflecting a user's request, performed by a simultaneous interpretation apparatus including a translation unit and a synthesis unit, comprising:
receiving text of the content in the first language;
receiving a speaker's voice of a different expression from the contents of the first language;
extracting speaker voice characteristics from the speaker voice;
generating a corresponding control command by receiving a user request to be reflected in a result of simultaneous interpretation; and
generating audio of content in a second language obtained by simultaneously interpreting content in a first language by reflecting a user request corresponding to a control command;
Generating the audio of the simultaneously interpreted second language content includes:
translating the content in the first language into content in a second language; and
A method of simultaneous interpretation comprising generating audio of content in a second language.
A computer-readable recording medium recording a program for performing the simultaneous interpretation method according to any one of claims 1 to 14.
a first input unit for obtaining information in the form of voice;
a second input unit for acquiring information in non-speech form;
a text generation unit that converts the audio signal into text when the audio signal is input to the second input unit;
a feature extractor extracting voice features from the voice signal;
When a user request is input, a control module generating a control command for generating a simultaneous interpretation result reflecting the user request; and
a simultaneous interpretation module generating audio of content in a second language obtained by simultaneous interpretation of content in a first language by reflecting a user request corresponding to a control command;
The simultaneous interpretation module,
a translation unit which translates the contents of the first language obtained by the first input unit or the second input unit into contents of a second language; and
and a synthesis unit for generating voice of the content in the second language.

Images