RU2585974C2 - Method of providing communication between people speaking in different languages - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to electronics, particularly to means of receiving and transmitting speech of subscribers, speaking in different languages. Mobile terminal uses language models configured to given topic (e.g., tourism, business, everyday life, etc.). This enables to achieve high quality on devices with low computational power (for example, smart phones). For communication between people speaking languages A and B, an information processing conveyor is set up, comprising steps of: inputting speech in a language A, conversion of speech to text in language A, translation of text to B language, speech synthesis in language B, outputting speech in language B. System also makes it possible to train acoustic and language models, targeted at a specific user and/or conversation topics, and save them in a cloud service. Flexibility is provided by possibility of localisation of different pipeline stages of information processing terminals of participants of a conversation, as well as ability to download models used in cloud service.

EFFECT: providing protection of transmitted and received speech from unauthorised access, high accuracy of transmitting speech, high reliability of receiving and transmitting speech.

12 cl, 12 dwg

Description

The invention relates to the field of electronics, in particular to a technique for realizing communication of people speaking different languages, in particular, but not exclusively, to systems that convert an input speech audio signal in an input language into an output speech audio signal or text in an output language.

Hereinafter, it is customary to call the input speech audio signal in the input language A of the conversation participant A, and the output speech audio signal in the output language B of the conversation participant B, terminal devices used by the conversation participants: the first (transmitting) terminal A and the second (receiving) terminal B, which use languages A and B, respectively.

A known method of communication of multilingual interlocutors, including a translation center and a tool in the form of a mobile communication terminal for interlocutors to transmit information to each other through a translation center, the mobile radio terminal contains a loud-speaking element (RU No. 31288).

The disadvantage of this device is the complexity of the communication system, providing for the appeal to an intermediate translation center, which is due to the limited scope and low reliability.

A known method of communication of multilingual interlocutors and the simultaneous translation of speech information from one language to another, in which a memory device switching is connected to an analog-to-digital converter / ADC / 1 terminal of the electronic system (functionally transmitting in this consideration) is input analog audio signal in speech form in the input language. Then, through the ADC, analog-to-digital processing and conversion of the speech form of an analog audio signal into a text text form are performed in the 1st, input, language. In the process of exchanging information - code signals (in text form) between the 1st communication terminal and the second (receiving) terminal, provide electronic translation of the code text form in the 1st, input, language of the transmitting 1st terminal into the code text form in the 3rd, output, language of the 2nd (receiving) terminal. At the final stage of the transceiver, the code form of the text in the 3rd, output, language of the 2nd terminal is converted into an analog audio signal in speech form in the 3rd, output, language of this 2nd terminal by means of a speech synthesizer and the speech form of the analog audio signal is output to 3rd, output, language through the speaker of the 2nd communication terminal, functionally receiving in this case (RU No. 2419142).

The disadvantage of the prototype is the complexity of the hardware circuit and the complexity of the conversion of the audio signal, in particular taking into account the need to translate the code text form of the input audio signal into an intermediate language, the duration of the translation, which does not provide real synchronous translation in practice.

There is a method of communication between people speaking different languages, an electronic transceiver system that implements simultaneous translation of oral speech from one language to another, including at least one transceiver communication terminal of an incoming analog audio signal in a speech form, which contains: non-volatile storage device mainly flash memory; speech input means of an analog audio signal in the input language to the communication terminal, which is functionally transmitting; means for analog-to-digital conversion and processing of the speech form of an analog audio signal into a code text form in the input language; means for translating the code form of the text in the input language into the code form of the text in the output language; means for digital-to-analog conversion and processing of the code form of the text in the output language into an analog audio signal in the same language in speech form, as well as means for outputting the speech form of an analog audio signal in the output language through a communication terminal that is functionally receiving (RU No. 2070734, prototype).

The disadvantages of this prior art solution include relatively low functional and technical characteristics due to significant errors in the conversion of the speech form of the audio signal input into the transmitting terminal into a code text form.

In general, there are currently systems of speech recognition and machine translation of a fairly high quality. Such systems require significant computing resources. For this, distributed processing is used on servers on the Internet or in cloud structures. This approach has both pros (the ability to provide high quality) and cons (works only when accessing the Internet and full control by the service provider). Google and Apple (Siri) systems provide high-quality speech recognition using a large vocabulary and networked cloud resources. Local versions of these systems provide good quality only for relatively small tasks, such as voice input of phone book entries.

Local speech recognition and machine translation systems are limited in their capabilities by the computing resources of the terminals (computers or smartphones). The main reason for this is the large size of dictionaries (up to several hundred thousand words), language models and the complexity of such systems. However, when narrowing the subject of conversation, the complexity of the task of speech recognition and translation is significantly reduced. For a good coverage of the text on a given topic (for example, business, travel, sports, etc.), a dictionary of 10-20 thousand words is enough. Modern speech recognition systems and machine translation are able to solve such a problem several times faster than real time (the time the phrase is pronounced).

The technical task of the utility model is to create an effective way to ensure communication between people speaking different languages, and to expand the arsenal of ways to ensure communication between people speaking different languages.

The technical result of the invention is to provide high translation accuracy and flexible use of mobile terminals without attracting network resources while expanding the capabilities realized by the claimed invention, in particular: the possibility of high-quality speech recognition and machine translation on the terminal without access to network resources through the use of thematic language models; the ability to use for the organization of communication of people speaking different languages, various, including low-power, terminals due to the flexible distribution of stages of the conveyor of information processing between terminals; the ability to control the set of used acoustic and language models, including downloading models from the cloud service, adapting models to the participant’s speech and the subject of the conversation, saving models in the cloud service.

To achieve a technical result, the present invention proposes to use a set of thematic language models of a relatively small size for speech recognition and machine translation systems operating on the terminals of conversation participants. The average volume of such models is from 5 to 20 megabytes, which allows you to store a set of models in the permanent memory of the terminal. Participants in the conversation can choose the model used depending on the topic of the conversation.

The present invention also offers the possibility of using additional services, in particular, the service of distribution and storage of models in the cloud and the service of training acoustic and language models. Users of the proposed system can access the acoustic and language models stored in the cloud service and install them on their terminal. Models for several languages and / or themes can be installed on the terminal. The speech signal of a conversation participant can be used to train an acoustic model adapted to the voice of that conversation participant. The speech signal of the participant in the conversation in the form of text can be used to teach thematic language models. Trained models are stored in a local storage device or in a cloud service for storing models.

The essence of the invention lies in the fact that a method for ensuring communication of people speaking different languages is presented, according to which the input analog speech audio signal in the input language A is introduced into the first terminal A; the speech signal is converted to digital form, the digital speech signal in language A using speech recognition software using only the local resources of terminal A, including the thematic language model located in the storage device of terminal A, is converted into text in language A, the text in language A using a machine tool for speech translation using only the local resources of terminal A, including a thematic language model located in the storage device of terminal A, etc. is generated into the text in the output language B, and the text in the language B is transmitted via the communication channel to the second terminal B, the text of the response message in the language B is received by the terminal A through the communication channel and transmitted to the speech synthesis software in the language B, which generates a digital audio signal in the language B, converted by the means of outputting speech into an analog audio signal, which is reproduced by terminal A. Moreover, terminal B functions in a similar way, up to replacing language A with language B and vice versa.

In this case, the speech recognition means, and / or the machine translation means, and / or the speech synthesis means have a hardware or hardware-software implementation.

In particular cases of implementation, the speech synthesis tool in language B operates on terminal A, the speech synthesis tool in language A operates on terminal B, and a speech signal or an encoded speech signal is transmitted via a communication channel. At the same time, terminal B functions similarly up to the replacement of language A with language B and vice versa.

In this case, speech recognition means, and / or machine translation, and / or speech synthesis means have a hardware or hardware-software implementation.

In addition, a machine translation tool from language B to language A and a machine translation tool from language A to language B operate on terminal A.

In this case, the speech recognition means and / or the machine translation means and / or the speech synthesis means have a hardware or hardware-software implementation.

In special cases of implementation, speech synthesis tool in language B, machine translation tool from language B to language A and speech synthesis tool in language B operate on terminal A, and a device with low processing power (landline or low-power) is used as terminal B.

At the same time, speech recognition and / or machine translation means and / or speech synthesis means have hardware or hardware-software implementation.

In special cases of implementation, mobile electronic devices connected via Bluetooth 2.0 / 4.0 and not using cellular communication to organize a conversation are used as communication terminals.

At the same time, speech recognition and / or machine translation means and / or speech synthesis means have hardware or hardware-software implementation.

In special cases of implementation, mobile electronic devices connected via the NFC protocol and not using cellular communication to organize a conversation are used as communication terminals.

At the same time, speech recognition and / or machine translation means and / or speech synthesis means have hardware or hardware-software implementation.

In particular cases of implementation, the same communication terminal is functionally used as the first and second terminal.

Preferably, the method provides for training acoustic and language models focused on a specific user and / or topic of conversation, and the storage of trained models in a cloud service and downloading them from a cloud service.

Preferably, the method provides for downloading from a cloud service acoustic and language models oriented to different languages and topics of conversation.

Figure 1 shows the general diagram of the hardware of the terminal, figure 2 is a general diagram of the software of the terminal, figure 3 is a flowchart of the communication process, when participant A speaks and participant B listens, in figure 4 - block a process diagram in which a translation into an intermediate more common language B is used, in FIGS. 5-6 are block diagrams of terminals A and B with a symmetrical distribution of stages of the information processing pipeline and transmission of text information between the terminals, FIGS. 7-8 are a block diagrams of terminals A and B with a symmetric distribution of diy conveyor processing information and transmitting an encoded speech signal (for example, according to the GSM standard) between the terminals, Figs. 9-10 are block diagrams of terminals A and B, provided that terminal A has more processing power than terminal B, and transmission text information between the terminals, in Fig. 11 is a block diagram of a terminal A, provided that the terminal B is a stationary telephone or low-power cell phones and transmitting an encoded speech signal (for example, according to the GSM standard) between the terminals, in Fig. 12 is a block terminal circuit a A, provided that terminal B is a pager or other device that allows input / output of text information.

Structurally, the terminal (cell mobile phone, smartphone, personal computer) is a case 9 (shown conditionally) in which the processor 1 is located. A processor 2, a monitor 3, a microphone 4, a speaker 5, a keyboard 6, a power supply 7 are connected to the processor 1 , transceiver 8 (GSM / GPRS / Wi-Fi transceiver, network card, hybrid Bluetooth 2.0 / 4.0, NFC, etc.). A general diagram of the hardware of the terminal is shown in FIG.

The terminal software includes the following components:

control system 10, speech input means 11, speech output means 12, speech encoding means 13, speech recognition means 14, speech synthesis means 15, machine translation means 16, acoustic model learning means 17, language model learning means 18, means for interacting with 19 cloud service 20. Used acoustic and language models for speech processing systems are stored in storage device 2. In figure 2. marked:

21 - the first (transmitting) terminal A;

22 - processor (at terminal A);

23 - storage device (at terminal A);

24 - language models stored in a storage device (at terminal A);

25 - microphone (at terminal A);

26 - means of inputting voice information (at terminal A);

27 - speech recognition means for input language A (at terminal A);

28 - means of translating a text in language A into text in language B (at terminal A);

29 - speech synthesis tool for language A (at terminal A);

30 - means for outputting voice information (at terminal A);

31 - speaker (at terminal A);

32 - channel for the transfer of information between terminals A and B;

33 - the second (receiving) terminal B;

34 - processor (at terminal B);

35 - storage device (at terminal B);

36 — language models stored in a storage device (at terminal B);

37 - microphone (at terminal B);

38 - means of inputting voice information (at terminal B);

39 - speech recognition means for language B (at terminal B);

40 - means of translating text in language B into text in language A (on the terminal

B)

41 - a means of speech synthesis for language B (at terminal B);

42 - means for outputting voice information (at terminal B);

43 - speaker (at terminal B);

44 - means of translating text in language B into text in language A (at terminal A);

45 - speech synthesis tool for language A (at terminal A);

46 - speech synthesis tool for language B (at terminal A);

47 — speech encoding means (at terminal A);

48 - speech encoding means (at terminal B);

49 - speech recognition means for language B (at terminal A);

5-12 some essential blocks and means of communication (such as a screen, keyboard, power supply, etc.) are not shown, since they are well-known, and their functional affiliation with the considered terminals is undeniable, and they are not principal objects claimed inventions.

The control system 10 is responsible for coordinating the operation of all components of the terminal software and controlling the operating modes of the terminal. The speech input means 11 is responsible for inputting a speech signal from a microphone. The speech output means 12 is responsible for outputting a speech signal to a speaker or other sound reproducing device. Speech encoding means 13 converts a speech signal into a set of code vectors in speech recognition and transmission to another terminal. Speech recognition means 14 converts the speech acoustic signal into text form. Speech synthesis tool 15 performs the conversion of the text into a speech acoustic signal. Means 16 machine translation converts text in one language into text in another language. The acoustic model training means 17 accumulates information about the speech signal of the terminal user and adapts the acoustic models to the specific speech features of the user. The tool 18 for teaching language models accumulates information about phrases pronounced by a terminal user and adapts language models to the specific features of a given user's speech on a given topic. The tool 19 interacting with the cloud service 20 is responsible for downloading acoustic and language models from the cloud and storing them in the cloud.

The process of communication between people speaking different languages naturally breaks down into several stages, Fig. 3 illustrates the process when participant A speaks and participant B listens:

a) speech input in language A;

b) the translation of speech in language A into text in language A (speech recognition);

c) translation of a text in language A into a text in language B (machine translation);

d) translation of a text in language B into speech in language B (speech synthesis);

e) speech output in language B.

Stage a-e form an information processing pipeline for the communication of people speaking different languages. Stage a and e can, in principle, be performed on a terminal that does not have processing power, such as a stationary telephone or a low-power cell phone. Performing stages b-d requires significant processing power. The terminal on which these tasks are performed can be, for example, a smartphone or laptop. Since stage a is performed at terminal A, and stage e is performed at terminal B, in the interval between these stages, information must be transmitted from terminal A to terminal B.

The present invention proposes to give conversation participants the opportunity to distribute the stages of the conveyor between terminals A and B, for example, if the terminals A and B are of the same type, stages a-c can be performed on terminal A, and stages de on terminal B. If terminal B is stationary By telephone, the ad stages will be performed on terminal A, and the e-stage will be performed on terminal B. The distribution of work between terminals A and B can be set automatically, semi-automatically or manually, depending on the processing power of the term catch A and B, the presence of linguistic models of terminals and / or the choice of participants in the call.

The type of information transmitted between terminals A and B is determined by the distribution of pipeline stages between the terminals. For example, if steps a-c are performed on terminal A, text in language B will be transmitted. If steps a-d are performed on terminal A, a speech signal (encoded speech signal) will be transmitted. Text transmission is preferable due to its smaller size and ease of protection of transmitted data, but not mandatory.

The above information processing pipeline is not the only one possible. For example, in the absence of means of translation from language A to language B, a translation into the intermediate more common language B can be used (see Fig. 4). The implementation of the method of the illustrated drawings:

5-6 are block diagrams of terminals A and B with a symmetric distribution of stages of the information processing pipeline and transmission of text information between the terminals.

7-8 are block diagrams of terminals A and B with a symmetrical distribution of stages of the information processing pipeline and transmission of the encoded speech signal (for example, according to the GSM standard) between the terminals.

9-10 are block diagrams of terminals A and B, provided that terminal A has more processing power than terminal B and transmission of text information between the terminals.

11 is a block diagram of a terminal A, provided that the terminal B is a stationary telephone or low-power cell phones, and the transmission of an encoded speech signal (for example, according to the GSM standard) between the terminals.

12 is a block diagram of a terminal A, provided that the terminal B is a pager or other device capable of inputting / outputting text information.

According to the claimed method, the input analog speech signal in language A is input into terminal A 21 (Fig. 5-6) through microphone 25. Then, through the ADC included in the speech input means 26, the analog speech signal is converted into a digital speech signal in language A. The speech signal in digital form is input to the speech recognition means 27 using one of the language models 24 stored in the storage device 23. The speech recognition means 27 converts the speech signal in language A into a text form (text n language A) and transfers it to the machine 28 means of translating into the language B. The means 28 of the machine translation uses one of the language models 24 stored in the storage device 23, and converts the text in the language A into the text in the language B, and transmits it over the communication channel 32 to terminal B 33. The response from terminal B 33 in the form of text in language A comes through a data channel 32 and is input to speech synthesis means in language A 29. Language synthesis tool 29 converts text in language A into a digital speech signal in language A and transmits it to the means 30 output speech. Means 30 for outputting speech by means of its DAC converts the digital speech signal into an analog speech signal and outputs it through the speaker 31. Means 27 for speech recognition, means 28 for machine translation and means for synthesizing speech in language A, operating on terminal A 21, use processor 22 and may have a software, hardware-software or hardware implementation.

The above diagram illustrates an embodiment of the invention, shown in Fig.5-6, and is not exhaustive or solely possible.

The features of the claimed method is as follows.

When organizing simultaneous interpretation of oral speech, only local computing resources of the terminals are used and network resources are not used (on the Internet, in the cloud, etc.).

To convert the input speech audio signal into text in language A, a local speech recognition system with a language model focused on a specific area (for example, travel, business, everyday life, etc.) is used, which allows to increase the recognition accuracy and reduce the computing power requirements of the terminal .

To translate text in language A into text in language B or C, a local machine translation system is used with a language model that focuses on a specific area (for example, travel, business, everyday life, etc.), which allows to increase recognition accuracy and reduce requirements for computing power of the terminal.

When organizing simultaneous interpretation of oral speech, a conveyor of information transfer from one participant to the conversation is formed.

The conveyor of the transfer of information from the participant in conversation A to the participant in conversation B includes the following stages: inputting speech in language A, converting speech into text in language A, translating text from language A to language B, synthesizing speech in language B, outputting speech in language B.

In the absence of a machine translation system from language A to language B, the pipeline for transferring information from the participant in conversation A to the participant in conversation B includes the following stages: entering speech in language A, converting speech into text in language A, translating text from language A to language B, translation text from language B to language B, speech synthesis in language B, speech output in language B.

The conveyor of information transfer from the participant in conversation B to the participant in conversation A is formed in the same way (language A is replaced by language B and vice versa).

Various methods of distributing stages of the information processing pipeline between terminals A and B are allowed. After the last stage of the information processing pipeline at terminal A (B) is completed, information (audio signal, encoded audio signal or text) is transmitted to terminal B (A), where the remaining stages are performed information processing pipeline.

When establishing a connection, terminals A and B distribute the stages of the information processing pipeline depending on the computing capabilities of the terminals and presets set by the participants in conversation A and B.

As terminals A and B, devices connected via a mobile communication system (for example, GSM or CDMA standard) or the Internet can be used. Also, devices connected via Bluetooth or NFC protocols and not using cellular communication can be used as terminals A and B (including to ensure confidentiality).

The proposed system allows you to train the used acoustic and language models for a specific user and / or topic of conversation. Trained models can be stored in the cloud service and subsequently used on this or another terminal of this user. In addition, the cloud service is used to distribute acoustic and language models for other languages or topics of conversation.

One of the particular private applications of the technical implementation of the claimed method is an electronic transceiver system with the function of simultaneous translation of oral speech from language A to language B between two terminals A and B with a symmetrical distribution of stages of the information processing pipeline and transmission of text information between the terminals (Fig. 5 -6).

Another particular private application for the technical implementation of the claimed method is an electronic transceiver system with the function of simultaneous translation of oral speech from language A to language B between two terminals A and B with a symmetrical distribution of stages of the information processing pipeline transmitting an encoded speech signal (for example, according to the standard GSM) between the terminals (Figs. 7-8).

Another particular private application of the technical implementation of the claimed method is an electronic transceiver system with the function of simultaneous translation of oral speech from language A to language B between two terminals A and B, provided that terminal A has more processing power than terminal B, and the transmission of text information between the terminals (Fig.9-10).

Another particular private application for the technical implementation of the claimed method is an electronic transceiver system with the function of simultaneous translation of oral speech from language A to language B between two terminals A and B, provided that terminal B is a stationary telephone or a low-power cell phone (Fig. eleven).

Another particular private application of the technical implementation of the claimed method is an electronic transceiver system with the function of simultaneous translation of oral speech from language A to language B between two terminals A and B, provided that terminal B is a pager or other device that allows input / output textual information (Fig.12).

As a result of the implementation of the present invention are provided:

- the possibility of high-quality speech recognition and machine translation on the terminal without access to network resources through the use of thematic language models;

- the ability to use for the organization of communication of people speaking different languages, various, including low-power, terminals due to the flexible distribution of stages of the conveyor of information processing between terminals;

- the ability to control the set of used acoustic and language models, including downloading models from the cloud service, adapting models to the speech of the participant in the conversation and the subject of the conversation, saving models in the cloud service.

Claims (12)

1. A method for transmitting and receiving speech of subscribers speaking different languages, according to which a communication channel is established between two communication terminals, and using local computing resources of terminals in which acoustic and language thematic models are provided, an audio signal processing pipeline isolated from network resources is formed, which one of the terminals receives the input analog speech audio signal in the input language, using the local computing resources of the terminals are recognized speech and its translation into coded text in the input language, and then using acoustic and language thematic models — machine translation of this text into coded text in the output language, followed by conversion of the translated coded text to synthesize an analog speech audio signal in the output language, which is output by another terminal. 2. The method according to p. 1, characterized in that one of the terminals receives an input analog speech audio signal in the input language and the same terminal using its local computing resources, which provide acoustic and language thematic models, speech recognition and translation into encoded text in the input language, machine translation of this text into encoded text in the output language, followed by conversion of the translated text to synthesize an analog speech audio signal in the output language, distributed for output via a communication channel to another terminal. 3. The method according to p. 1, characterized in that one of the terminals receives an input analog speech audio signal in the input language, transmitted via a communication channel to another terminal, which, using its local computing resources, which provide acoustic and language thematic models, recognizes speech and its translation into encoded text in the input language, machine translation of this text into encoded text in the output language, followed by conversion of the translated text for the synthesis of analog speech of the audio signal in the target language, which displays the same terminal. 4. The method according to p. 1, characterized in that one of the terminals receives the input analog speech audio signal in the input language and using its local computing resources, which provide acoustic and language thematic models, speech recognition and translation into encoded text are carried out on the input language transmitted over the communication channel to another terminal, which, using its local computing resources, which provide acoustic and language thematic models, performs machine translation d of this text into a coded text in the target language, followed by converting the translated text to speech synthesis of the analog audio signal in the target language which is output by another terminal. 5. The method according to any one of paragraphs. 1-4, characterized in that the analogue speech audio signal of the response message in the input or output language is received by the corresponding terminal, followed by the formation of an information processing pipeline for synthesizing the analogue speech audio signal in the output or input language, respectively, which is output by another terminal. 6. The method according to p. 5, characterized in that the reverse input and output speech signals are transmitted using two terminals connected by a communication channel, configured to form an information processing conveyor for converting the input and output speech signals, both in the direction from one terminal to another and in the opposite direction. 7. The method according to any one of paragraphs. 2, 3, characterized in that, as one of the terminals, a device from the group is used: a landline telephone, a mobile phone using a wire or cellular connection to organize a conversation. 8. The method according to any one of paragraphs. 1-4, 6, characterized in that at least one of the means of speech recognition, machine translation, speech synthesis has a hardware or hardware-software implementation. 9. The method according to any one of paragraphs. 1-4, 6, characterized in that as the communication terminals use mobile electronic devices connected by a wireless communication channel according to a protocol from the group: Bluetooth, NFC. 10. The method according to any one of paragraphs. 1-4, 6, characterized in that the communication channel for communication in different languages is established with the formation of two communication terminals by means of the same reversible terminal device. 11. The method according to any one of paragraphs. 1-4, 6, characterized in that thematic acoustic and language models are formed, which are stored in the permanent memory of at least one terminal or in a cloud service with the possibility of downloading them to communication terminals if necessary. 12. The method according to p. 11, characterized in that at least one terminal uses thematic acoustic and language models in various languages.

Images