KR20220099083A - System, user device and method for providing automatic interpretation service based on speaker separation - Google Patents

Abstract

Provided is an automatic interpretation method based on a speaker separation in a user terminal. The method comprises: a step of receiving a first speech signal comprising at least one among a user speech and user surrounding speech from an automatic interpretation service providing terminal; a step of separating the first speech signal into a speech signal for each speaker; a step of interpreting the speech signal for each speaker into a language selected by the user based on an interpretation mode; and a step of providing a second speech signal generated as a result of the interpretation to at least one among a counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode. Therefore, the present invention is capable of having an advantage of being able to obtain the information conveniently.

Description

SYSTEM, USER DEVICE AND METHOD FOR PROVIDING AUTOMATIC INTERPRETATION SERVICE BASED ON SPEAKER SEPARATION

The present invention relates to a system, a user terminal, and a method for providing an automatic interpretation service based on speaker separation.

In general, automatic interpretation (or automatic translation) has a structure in which a user's first language is recognized, interpreted in a second language, and then transmitted to a counterpart terminal, or an interpretation result received from the counterpart terminal is synthesized and presented to the user.

This approach basically aims for a face-to-face conversation in which the user and the other party exist. However, in most situations that require automatic interpretation, when a user goes to a foreign country, it is necessary to have a conversation with a foreigner (tourism, restaurant, shopping, hotel, business, etc.) This is to understand the voices of foreigners around you.

That is, it is necessary to provide the user with an interpretation of foreign voices heard in the vicinity as well as face-to-face conversation, but such a technology is absent.

Patent Publication No. 10-2019-0015081 (2019.02.13.)

A system for providing an automatic interpretation service based on speaker separation, in which the voices of a plurality of speakers input to an individual device to be solved by the present invention are divided for each speaker and all of the voices of the user's desired language are converted to provide interpretation results, a user terminal and to provide a method.

However, the problems to be solved by the present invention are not limited to the problems described above, and other problems may exist.

In a speaker separation-based automatic interpretation method in a user terminal according to a first aspect of the present invention for solving the above problems, a first voice signal including at least one of a user's voice and a user's surrounding voice is received from an automatic interpretation service providing terminal to do; separating the first voice signal into voice signals for each speaker; interpreting the speech signal for each speaker into a language selected by a user based on the interpretation mode; and providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode.

In some embodiments of the present invention, the step of receiving the first voice signal including at least one of the user's voice and the user's surrounding voice from the automatic interpretation service providing terminal may include automatically interpreting the first voice signal based on the interpretation mode. Received from a service providing terminal, wherein the automatic interpretation service providing terminal may receive a user's voice through a microphone when the interpretation mode is a conversation mode, and may receive a user's surrounding voice through a microphone in the listening mode.

In some embodiments of the present invention, the step of interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode may include selecting a user voice from among the separated voice signals for each speaker when the interpretation mode is a conversation mode. performing the interpretation, and providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode may include: It may be provided to the counterpart terminal, and an interpretation result of the counterpart's voice received from the counterpart terminal may be provided to the automatic interpretation service providing terminal.

In some embodiments of the present invention, the step of interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode may include: when the interpretation mode is the listening mode, the voice surrounding the user among the separated voice signals for each speaker The step of selectively performing an interpretation and providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode may include: It can be divided and provided to the automatic interpretation service providing terminal.

In some embodiments of the present invention, the step of interpreting the speech signal for each speaker into a language selected by the user based on the interpretation mode includes at least one of speaker information, context information, and noise information from the user's surrounding voice. extracting information; storing the extracted context information in a buffer; and interpreting a speaker-specific voice signal included in the user's surrounding voice based on the extracted context information.

In some embodiments of the present invention, the step of providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode includes: It is possible to classify and provide the interpretation result to the automatic interpretation service providing terminal by reflecting the strength information and the echo information of the voice signal divided for each speaker.

In some embodiments of the present invention, the step of providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode may include: An interpretation result corresponding to the at least one speaker selected by , may be provided to the automatic interpretation service providing terminal.

In addition, the user terminal providing the automatic interpretation service based on speaker separation according to the second aspect of the present invention includes a communication module for transmitting and receiving data with an automatic interpretation service providing terminal and a counterpart terminal, and a user's voice and user's surrounding voice from the automatic interpretation service providing terminal. and a memory in which a program for generating a second voice signal, which is an interpretation result, based on the reception of a first voice signal including at least one of is executed, the first voice signal is separated into voice signals for each speaker, and the voice signal for each speaker is interpreted into the language selected by the user based on the interpretation mode. 2 The voice signal is provided to at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the interpretation mode.

In some embodiments of the present invention, the automatic interpretation service providing terminal may receive a user's voice through a microphone when the interpretation mode is a conversation mode, and may receive a user's surrounding voice through a microphone in the listening mode.

In some embodiments of the present invention, when the interpretation mode is a conversation mode, the processor selects a user's voice from among the separated voice signals for each speaker and performs an interpretation, and the processor performs an interpretation of the user's voice through the communication module. 2 The voice signal may be provided to the counterpart terminal, and the result of interpretation of the counterpart's voice received from the counterpart terminal may be provided to the automatic interpretation service providing terminal.

In some embodiments of the present invention, when the interpretation mode is the listening mode, the processor selects a user's surrounding voice from among the separated speaker-specific voice signals to perform interpretation, and divides the interpretation result of the user's surrounding voice for each speaker Thus, it can be provided to the automatic interpretation service providing terminal through the communication module.

In some embodiments of the present invention, the processor extracts context information including at least one of speaker information, context information, and noise information from the surrounding voice of the user and stores the extracted context information in the buffer of the memory, and based on the extracted context information Thus, it is possible to interpret the voice signal for each speaker included in the voice around the user.

In some embodiments of the present invention, the processor includes: a source separation model for each speaker for separating the first voice signal into a voice signal for each speaker based on the first and second voice signals and the extracted context information; The performance of the speech recognition model for recognizing the first speech signal, the machine translation model for translating the first speech signal, and the combined automatic interpretation model may be automatically updated.

In some embodiments of the present invention, the processor classifies a voice signal for each speaker from the extracted context information, and reflects the strength information and echo information of the voice signal divided for each speaker to automatically interpret the interpretation result through the communication module It can be provided as a service providing terminal.

In some embodiments of the present invention, the processor may provide an interpretation result corresponding to at least one speaker selected by the user among the interpretation results for the voice around the user to the automatic interpretation service providing terminal through the communication module. .

In addition, the speaker separation-based automatic interpretation service providing system according to the third aspect of the present invention receives a first voice signal including at least one of a user voice and a user's surrounding voice through a multi-channel microphone, and An automatic interpretation service providing terminal that provides an interpretation result and an interpretation result for the voice of the other party through a speaker, receives the first voice signal and separates it into a voice signal for each speaker, and then provides a voice signal for each speaker to the user based on the interpretation mode After interpreting in a language selected by and a counterpart terminal that interprets the counterpart's voice according to the interpretation mode and provides it to the user terminal.

A computer program according to another aspect of the present invention for solving the above problems executes a system, a user terminal and a method for providing a speaker separation-based automatic interpretation service in combination with a computer, which is hardware, and is stored in a computer-readable recording medium do.

Other specific details of the invention are included in the detailed description and drawings.

According to the above-described embodiment of the present invention, compared to the conventional face-to-face automatic interpretation method, the user can conveniently acquire information by converting the voices of a plurality of surrounding speakers into their native language.

In addition, an embodiment of the present invention can provide an automatic interpretation method in which both a conversation mode and a listening mode are provided by mixing with the conventional face-to-face method.

In addition, an embodiment of the present invention can be utilized as an auxiliary tool for language learning, and the user can also apply to foreign language conversation learning by interpreting and listening to surrounding native language voices in a desired language.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram schematically illustrating a speaker separation-based automatic interpretation service providing system according to an embodiment of the present invention.
2 is a block diagram illustrating a user terminal according to an embodiment of the present invention.
3 is a flowchart of an automatic interpretation method based on speaker separation according to an embodiment of the present invention.
4 is a diagram for explaining a user's surrounding voice.
5A and 5B are diagrams for explaining the content of dividing a first voice signal into a voice signal for each speaker.
6 is a diagram for explaining a conversation mode provided by an embodiment of the present invention.
7 is a diagram for explaining a listening mode provided in an embodiment of the present invention.
8 and 9 are diagrams for explaining the content of extracting and reflecting context information in an embodiment of the present invention.
FIG. 10 is a view showing the contents of providing a second sound line signal based on the collected situation information.
11 is a diagram for explaining each process performed in an automatic interpretation service providing terminal.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully inform those skilled in the art of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural, unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly specifically defined.

The present invention relates to a system (1), a user terminal (100) and a method for providing an automatic interpretation service based on speaker separation.

Unlike the conventional method for the purpose of face-to-face conversation in automatic interpretation, the present invention divides the voices of a plurality of mixed speakers input to individual devices and converts them all into the language voice desired by the user. and providing the voices of a plurality of speakers to the user terminal 100 in consideration of the characteristics of each speaker. Here, the language desired by the user may generally be the native language, but is not necessarily limited thereto.

Through this, one embodiment of the present invention helps the user to easily understand the surrounding situation while being exposed to a heterogeneous language environment (eg, overseas travel, etc.) such as being in a foreign country, and automatically interprets information around the user face-to-face It can be used in the system to improve automatic interpretation performance and user convenience.

In addition, an embodiment of the present invention can be applied in a native language environment, for example, by receiving a user's surrounding voice in the native language environment and receiving the translated result into English, there is an advantage that it can be used for learning a foreign language. In particular, the present invention is characterized in that, when an interpretation result for the user's surrounding voice is provided, context information is additionally extracted and reflected in the synthesized voice signal. Interpretation results for each of the voices of a plurality of speakers included in the user's surrounding voice In addition, location information is more reflected and provided in their interpretation results, thereby enabling more lively foreign language learning.

Meanwhile, in an embodiment of the present invention, simultaneous interpretation in an environment in which both the speaker and the listener use different languages, but is not necessarily limited thereto.

Hereinafter, the speaker separation-based automatic interpretation service providing system 1 and the user terminal 100 according to an embodiment of the present invention will be described with reference to FIGS. 1 and 2 .

1 is a diagram schematically illustrating a speaker separation-based automatic interpretation service providing system 1 according to an embodiment of the present invention. 2 is a block diagram illustrating the user terminal 100 according to an embodiment of the present invention.

The system 1 for providing automatic interpretation service based on speaker separation according to an embodiment of the present invention includes a user terminal 100 , an automatic interpretation service providing terminal 200 , and a counterpart terminal 300 .

First, the automatic interpretation service providing terminal 200 receives a first voice signal including at least one of a user's voice and a user's surrounding voice through the multi-channel microphone 210 . In addition, the automatic interpretation service providing terminal 200 receives the interpretation result for the first voice signal from the user terminal 100 or receives the interpretation result for the counterpart's voice from the counterpart terminal 300 and provides it through the speaker 230 . do.

In this case, the automatic interpretation service providing terminal 200 may be provided with an interpretation mode selection button 220, and provides voice reception and interpretation results according to the interpretation mode selected by the user.

In an embodiment, the automatic interpretation service providing terminal 200 may be provided in the form of a hands-free terminal including a speaker 230 and a microphone 210, but is not necessarily limited thereto, and the speaker 230 and the microphone 210 All terminals provided with may be the automatic interpretation service providing terminal 200 . Accordingly, it goes without saying that the automatic interpretation service providing terminal 200 may be provided integrally with the user terminal 100 having the microphone 210 and the speaker 230 or the counterpart terminal 300 .

Referring to FIG. 2 , the user terminal 100 includes a communication module 110 , a memory 120 , and a processor 130 .

The communication module 110 transmits and receives data to and from the automatic interpretation service providing terminal 200 and the counterpart terminal 300 . That is, the communication module 110 may provide an interpretation result of the user's voice to the counterpart terminal 300 in real time, and receives the interpretation result of the counterpart's voice received from the counterpart terminal 300 to provide an automatic interpretation service providing terminal 200 . ) can be provided.

In this case, the communication module 110 is preferably configured as a wireless communication module, but the concept does not necessarily exclude the wired communication module. The wired communication module may be implemented as a power line communication device, a telephone line communication device, a cable home (MoCA), Ethernet, IEEE1294, an integrated wired home network, and an RS-485 control device. In addition, the wireless communication module may be implemented by wireless LAN (WLAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60GHz WPAN, Binary-CDMA, wireless USB technology, wireless HDMI technology, and the like.

The memory 120 stores a program for generating a second voice signal, which is an interpretation result, based on the reception of a first voice signal including at least one of a user voice and a voice around the user from the automatic interpretation service providing terminal 200 . do.

In this case, the memory 120 collectively refers to a non-volatile storage device and a volatile storage device that continuously maintain stored information even when power is not supplied. For example, memory may include compact flash (CF) cards, secure digital (SD) cards, memory sticks, solid-state drives (SSDs) and micro SD cards, etc. It may include a magnetic computer storage device such as a NAND flash memory, a hard disk drive (HDD), and the like, and an optical disc drive such as a CD-ROM or DVD-ROM.

As the program stored in the memory 120 is executed, the processor 130 separates the first voice signal received from the automatic interpretation service providing terminal 200 into a voice signal for each speaker, and a voice for each speaker based on the interpretation mode. After the signal is interpreted into the language selected by the user, the second voice signal generated as a result of the interpretation through the communication module is provided to the counterpart terminal 300 or the automatic interpretation service providing terminal 200 according to the interpretation mode.

The counterpart terminal 300 receives the voice signal in the same manner as the user terminal 100 and provides the interpretation result to the user terminal 100 or the automatic interpretation service providing terminal 200 paired with the counterpart terminal 300 . That is, in an embodiment of the present invention, the user terminal 100 and the counterpart terminal 300 are named according to the subjects they use, and their configurations and functions are the same.

On the other hand, the user terminal 100 and the counterpart terminal 300 in the present invention are a server system, a laptop equipped with a web browser (WEB Browser), a desktop, a laptop, etc. It may be an intelligent device that adds computer support functions such as information search, and also a mobile phone, smart phone, pad, or smart watch that can install and run a number of applications (that is, applications) that the user wants. (Smart watch), a wearable (wearable) device, may be other mobile communication devices.

Hereinafter, a speaker separation-based automatic interpretation method performed in the user terminal 100 according to an embodiment of the present invention will be described in more detail with reference to FIGS. 3 to 11 .

3 is a flowchart of an automatic interpretation method based on speaker separation according to an embodiment of the present invention. 4 is a diagram for explaining a user's surrounding voice. 5A and 5B are diagrams for explaining the content of dividing a first voice signal into a voice signal for each speaker.

First, according to the request of the automatic interpretation service providing terminal 200 or the user terminal 100, the user terminal 100 performs pairing with the automatic interpretation service providing terminal 200 (S110). At this time, the user terminal 100 initializes the interpretation environment as it is paired with the automatic interpretation service providing terminal 200 . As an example of initialization of the interpretation environment, context information (speaker information, context information, and noise information) in the interpretation result previously performed may be initialized. However, since there may be cases in which surrounding voices are received in the same environment or conversations are continuously conducted with the same counterpart, it goes without saying that the interpretation environment initialization process may be selectively performed according to a user's input.

Next, a first voice signal including at least one of a user's voice and a user's surrounding voice is received from the automatic interpretation service providing terminal 200 (S120).

In this case, the user's voice corresponds to the utterance of the user who is the subject using the user terminal 100 , and the user's surrounding voice corresponds to the remaining voices except for the user's voice received through the multi-channel microphone 210 . That is, according to an embodiment of the present invention, interpretation is performed on a utterance that is a user's voice and provided to the counterpart terminal 300 , and an interpretation result for the counterpart's voice is received from the counterpart terminal 300 to receive the user terminal 100 and The face-to-face interpretation service provided by the automatic interpretation service providing terminal 200 is basically provided.

According to an embodiment of the present invention, the multi-channel microphone 210 receives ambient voices from various speakers rather than face-to-face conversations, and performs interpretation for them, thereby providing the user terminal 100 and automatic interpretation service. It may be provided to the terminal 200 . Here, various speakers do not specifically limit their subjects, such as direct speech by a person, as well as a guide voice output through the speaker 230 .

For example, referring to FIG. 4 , the user's surrounding voice P1 may include 'ambient noise', 'neighbor 1 voice', and 'neighborhood person 2' voices, in which case people 1 and 2 are the user. It corresponds to an unspecified person who does not communicate directly with

In such a situation, in an embodiment of the present invention, when a user's surrounding voice is received through the multi-channel microphone 210 of the automatic interpretation service providing terminal 200, the automatic interpretation service providing terminal 200 transmits the user's surrounding voice to the user. provided to the terminal 100, and the user terminal 100 generates an interpretation result (Korean) corresponding to each voice language (English, Korean) of the person around 1 and the person around 2, as will be described later, to provide an automatic interpretation service. It may be provided to be output through the speaker 230 of 200 .

In this case, the automatic interpretation service providing terminal 200 may be provided with an interpretation mode selection button 230, and provides voice reception and interpretation results according to the interpretation mode selected by the user.

Next, the user terminal 100 separates the first voice signal into a voice signal for each speaker ( S130 ).

According to an embodiment of the present invention, a user's voice and a voice around the user are first separated into voice signals for each speaker.

As shown in FIGS. 5A and 5B , the user's voice and the user's surrounding voice can be easily distinguished according to the interpretation mode. That is, in the face-to-face conversation mode, the voice received through the microphone 210 may be recognized as the user's voice, and in the listening mode instead of the conversation mode, it may be recognized as the user's surrounding voice rather than the user's voice. This may also be provided in advance by the automatic interpretation service providing terminal 200 .

In the process of separating the voice signal for each speaker, the user's voice is divided into the user's voice and the noise signal because there is only one speaker by the source separation model P2 for each speaker. In addition, in the case of the user's surrounding voice, a plurality of speaker-spoken voices and noise signals may be separated by the speaker-specific source separation model P2.

Referring back to FIG. 3 , next, the user terminal 100 interprets the voice signal for each speaker into the language selected by the user based on the interpretation mode ( S140 ). Then, the second voice signal generated as a result of the interpretation is provided to at least one of the counterpart terminal 300 and the automatic interpretation service providing terminal 200 according to the interpretation mode (S150).

6 is a diagram for explaining a conversation mode provided by an embodiment of the present invention, and FIG. 7 is a diagram for explaining a listening mode provided by an embodiment of the present invention. 8 and 9 are diagrams for explaining the content of extracting and reflecting context information in an embodiment of the present invention. FIG. 10 is a view showing the contents of providing a second sound line signal based on the collected situation information.

In one embodiment, the present invention provides a conversation mode and a listening mode as the interpretation mode.

First, referring to FIG. 6 , the conversation mode is a case in which the user and the other party have a face-to-face conversation. In the conversation mode, the user terminal 100 selects a user voice from among the separated voice signals for each speaker (S141), and the user voice After interpreting is performed for the user (S142), the second voice signal, which is the result of the interpretation of the user's voice, is provided to the counterpart terminal 300 (S143). In addition, when the other party also has a conversation such as an answer to this, the result of the interpretation of the voice of the other party received from the other party terminal 300 is received (S144) and provided to the automatic interpretation service providing terminal 200 (S145).

Next, referring to FIG. 7 , the listening mode is for receiving and interpreting the voice of the user's surroundings rather than face-to-face conversation. is selected (S146), and interpretation is performed on the user's surrounding voice (S147). In addition, the user terminal 100 may classify the interpretation result for the user's surrounding voice for each speaker (S148) and provide the result to the automatic interpretation service providing terminal 200 (S149).

In this case, the user terminal 100 may further perform a process of extracting context information from the user's surrounding voice in the listening mode. Referring to FIG. 8 , the user terminal 100 inputs a speaker-specific voice signal separated from the user's surrounding voice to the voice encoder A1, and receives speaker information, context information, and noise from the user's peripheral voice separated for each speaker. Context information including at least one of the information is extracted, and the extracted context information is stored in a buffer.

Then, after inputting the extracted contextual information to the contextual information encoder (A2), it is summed with the output information of the voice encoder and inputted to the translation decoder (A3), so that the interpretation result reflecting the contextual information when interpreting the voice around the user is displayed. can provide

At this time, the user terminal 100 classifies the voice signal for each speaker from the context information extracted as shown in FIG. 9 , and further reflects the intensity information and the echo information of the voice signal divided for each speaker through the context information to provide an automatic interpretation service. It may be provided to the providing terminal 200 . That is, the user terminal 100 estimates the speaker's location information based on the intensity information and the echo information (B1), checks whether the corresponding speaker information exists in the buffer (B2), and outputs the automatic interpretation result. Location information for each speaker may be further reflected and provided.

Accordingly, the speaker-specific voice signals output from the automatic interpretation service providing terminal 200 may be output to have characteristics similar to those of actual speakers.

In the existing general automatic interpretation system, the automatic interpretation result is provided to the user as a preset single-speaker voice without considering the characteristics of the conversational speaker. On the other hand, in an embodiment of the present invention, through the process of separating voice signals for each speaker and the process of extracting and applying context information, the results of automatic interpretation of multiple speakers can be synthesized to have characteristics similar to those of the actual speaker. Therefore, it is possible to provide a more natural automatic interpretation result to the user.

On the other hand, speaker information can include information such as whether it is male or female or computer voice, age, nationality, emotional state, etc., and context information can be checked through the meaning of words or sentences before and after the real-time interpretation result, and noise information is a concept that includes information such as location, time, and space.

FIG. 10 shows the contents of providing an interpretation result as a second voice signal by reflecting the location information in FIG. 9, each speaker's voice signal extracted from the user's surrounding voice, speaker information and location information extracted from context information; are input to the text encoder (C1) and the context encoder (C2), respectively. Then, as the output information of the text encoder (C1) and the output information of the context encoder (C2) are summed and input to the speech synthesis decoder (C3), the interpretation result for the voice around the user is divided into a plurality of speakers ( C4) The second voice signal in which the location information of a plurality of speakers is reflected may be synthesized (C5) and provided to the automatic interpretation service providing terminal 200 through the user terminal 100 .

In addition, an embodiment of the present invention provides a source separation model for each speaker, a voice recognition model for recognizing the first voice signal, and a first voice based on context information extracted through the first and second voice signals and the user's surrounding voice. It is possible to automatically update the performance of a machine translation model for translating a signal and an automatic interpretation model combined with them. That is, each model is based on an artificial intelligence model learning method such as a neural network model, a deep learning model, by setting the first and second voice signals and context information as input data, and setting the existing output information in each model as output data. performance can be automatically updated. Through this, an embodiment of the present invention has an advantage in that it is possible to provide a more improved automatic interpretation result to the user as the automatic interpretation service is continuously used.

11 is a diagram for explaining each process performed by the automatic interpretation service providing terminal 200. Referring to FIG.

When the automatic interpretation service providing terminal 200 receives the user's automatic interpretation request (S210), it requests the user terminal 100 to initialize interpretation as it is paired with the user terminal 100 (S220).

Next, by dividing the interpretation mode (S230), the user's voice is received through the multi-channel microphone 210 in the conversation mode (S231), and the user's surrounding voice is received through the multi-channel microphone 210 in the listening mode (S231) do (S232).

Next, the received user voice or the user's surrounding voice is transmitted to the user terminal 100 (S240), and the second voice signal according to the interpretation result from the user terminal 100 or the counterpart terminal 300 through the user terminal 100 ), when the second voice signal according to the interpretation result is received (S250), the second voice signal is output to the user through the speaker 230 (S260).

Meanwhile, in the above description, steps S110 to S260 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be changed. In addition, even if other contents are omitted, the contents of FIGS. 1 to 2 may also be applied to the speaker separation-based automatic interpretation method of FIGS. 3 to 11 .

The embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-mentioned program, in order for the computer to read the program and execute the methods implemented as a program, C, C++, JAVA, Ruby, which the processor (CPU) of the computer can read through the device interface of the computer; It may include code coded in a computer language such as machine language. Such code may include functional code related to a function defining functions necessary for executing the methods, etc. can do. In addition, the code may further include additional information necessary for the processor of the computer to execute the functions or code related to memory reference for which location (address address) in the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code uses the communication module of the computer to determine how to communicate with any other computer or server remotely. It may further include a communication-related code for whether to communicate and what information or media to transmit and receive during communication.

The storage medium is not a medium that stores data for a short moment, such as a register, a cache, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program may be stored in various recording media on various servers accessible by the computer or in various recording media on the computer of the user. In addition, the medium may be distributed in a computer system connected by a network, and computer-readable codes may be stored in a distributed manner.

The foregoing description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

1: Automatic interpretation service provision system
100: user terminal
110: communication module
120: memory
130: processor
200: automatic interpretation service providing terminal
210: multi-channel microphone
220: Interpretation mode selection button
230: speaker
300: counterpart terminal

Claims (16)

In the automatic interpretation method based on speaker separation in a user terminal,
Receiving a first voice signal including at least one of a user's voice and a user's surrounding voice from an automatic interpretation service providing terminal;
separating the first voice signal into voice signals for each speaker;
interpreting the speech signal for each speaker into a language selected by a user based on the interpretation mode; and
providing a second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode;
Automatic interpretation method based on speaker separation.
According to claim 1,
Receiving a first voice signal including at least one of a user's voice and a user's surrounding voice from the automatic interpretation service providing terminal,
receiving the first voice signal from an automatic interpretation service providing terminal based on the interpretation mode,
The automatic interpretation service providing terminal receives a user's voice through a microphone when the interpretation mode is a conversation mode, and receives a user's surrounding voice through a microphone in the listening mode,
Automatic interpretation method based on speaker separation.
According to claim 1,
The step of interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode comprises:
When the interpretation mode is a conversation mode, an interpretation is performed by selecting a user's voice from among the separated voice signals for each speaker;
providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode;
providing a second voice signal, which is an interpretation result of the user's voice, to a counterpart terminal, and providing an interpretation result of the counterpart's voice received from the counterpart terminal to an automatic interpretation service providing terminal;
Automatic interpretation method based on speaker separation.
According to claim 1,
The step of interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode comprises:
When the interpretation mode is the listening mode, interpretation is performed by selecting a user's surrounding voice from among the separated speaker-specific voice signals;
providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode;
The interpretation result for the user's surrounding voice is divided by speaker and provided to the automatic interpretation service providing terminal.
Automatic interpretation method based on speaker separation.
5. The method of claim 4,
The step of interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode comprises:
extracting context information including at least one of speaker information, context information, and noise information from the voice around the user;
storing the extracted context information in a buffer; and
Interpreting the speaker-specific voice signal included in the user's surrounding voice based on the extracted context information;
Automatic interpretation method based on speaker separation.
6. The method of claim 5,
providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode;
Separating a voice signal for each speaker from the extracted context information, and providing an interpretation result to the automatic interpretation service providing terminal by reflecting the strength information and echo information of the voice signal divided for each speaker,
Automatic interpretation method based on speaker separation.
5. The method of claim 4,
providing the second voice signal generated as a result of the interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to the interpretation mode;
providing an interpretation result corresponding to at least one speaker selected by a user among the interpretation results for the voices around the user to the automatic interpretation service providing terminal;
Automatic interpretation method based on speaker separation.
A user terminal providing an automatic interpretation service based on speaker separation, the user terminal comprising:
A communication module that transmits and receives data to and from an automatic interpretation service providing terminal and a counterpart terminal;
When a first voice signal including at least one of a user's voice and a user's surrounding voice is received from the automatic interpretation service providing terminal, a program for generating a second voice signal that is an interpretation result based on the received memory;
A processor for executing the program stored in the memory;
As the program is executed, the processor separates the first voice signal into a voice signal for each speaker, interprets the voice signal for each speaker into a language selected by the user based on the interpretation mode, and then through the communication module providing the second voice signal generated as a result of interpretation to at least one of a counterpart terminal and an automatic interpretation service providing terminal according to an interpretation mode;
A user terminal that provides automatic interpretation service based on speaker separation.
9. The method of claim 8,
The automatic interpretation service providing terminal receives a user's voice through a microphone when the interpretation mode is a conversation mode, and receives a user's surrounding voice through a microphone in the listening mode,
A user terminal that provides automatic interpretation service based on speaker separation.
9. The method of claim 8,
When the interpretation mode is a conversation mode, the processor selects a user's voice from among the separated voice signals for each speaker for interpretation, and provides a second voice signal, which is a result of the interpretation of the user's voice, to the counterpart terminal through the communication module and providing an interpretation result of the counterpart's voice received from the counterpart terminal to an automatic interpretation service providing terminal,
A user terminal that provides automatic interpretation service based on speaker separation.
9. The method of claim 8,
When the interpretation mode is the listening mode, the processor selects a user's surrounding voice from among the separated voice signals for each speaker to perform interpretation, divides the interpretation result for the user's surrounding voice for each speaker, and uses the communication module to automatically which is provided by an interpreter service providing terminal,
A user terminal that provides automatic interpretation service based on speaker separation.
12. The method of claim 11,
The processor extracts context information including at least one of speaker information, context information, and noise information from the user's surrounding voice and stores it in a buffer of the memory, and based on the extracted contextual information, the user's surrounding voice includes Interpreting voice signals for each speaker,
A user terminal that provides automatic interpretation service based on speaker separation.
13. The method of claim 12,
The processor is configured to: based on the first and second voice signals and the extracted context information, a source separation model for each speaker for separating the first voice signal into a voice signal for each speaker, and a method for recognizing the first voice signal Automatically updating the performance of the speech recognition model, the machine translation model for translating the first speech signal, and the automatic interpretation model combined with them,
A user terminal that provides automatic interpretation service based on speaker separation.
13. The method of claim 12,
The processor classifies the voice signal for each speaker from the extracted context information, reflects the strength information and the echo information of the voice signal divided for each speaker, and provides the interpretation result to the automatic interpretation service providing terminal through the communication module. ,
A user terminal that provides automatic interpretation service based on speaker separation.
12. The method of claim 11,
wherein the processor provides an interpretation result corresponding to at least one speaker selected by the user among the interpretation results for the voices around the user to the automatic interpretation service providing terminal through the communication module,
A user terminal that provides automatic interpretation service based on speaker separation.
In the speaker separation-based automatic interpretation service providing system,
Automatic interpretation service that receives a first voice signal including at least one of a user's voice and a user's surrounding voice through a multi-channel microphone, and provides an interpretation result for the first voice signal and an interpretation result for the other party's voice through a speaker terminal,
After receiving the first voice signal, separating the voice signal for each speaker, interpreting the voice signal for each speaker into the language selected by the user based on the interpretation mode, and then converting the second voice signal generated as a result of the interpretation into the interpretation mode a user terminal provided to at least one of the counterpart terminal and the automatic interpretation service providing terminal according to the
and a counterpart terminal that receives and provides the second voice signal from a user terminal and interprets the other party's voice according to the interpretation mode and provides it to the user terminal;
A system for providing automatic interpretation service based on speaker separation.

Images