RU2808582C2 - Method and system for recognizing user's speech fragment - Google Patents

Abstract

FIELD: natural speech processing.

SUBSTANCE: method includes: receiving an audio signal that is an audio representation of a user's speech fragment, which is a response to a computer-generated speech fragment previously issued by an electronic device to the user; obtaining a computer-generated text string that is a text representation of the computer-generated speech fragment; and generating, using a speech-to-text (STT) model, another text string being a textual representation of the user's speech fragment based on the audio signal and a computer-generated text string taken into account as the context of the user's speech fragment.

EFFECT: improves the quality of speech recognition and overall satisfaction of users using virtual assistant applications.

20 cl, 8 dwg

Description

Field of technology to which the invention relates

[01] This technology generally relates to natural speech processing and, in particular, to methods and systems for recognizing spoken human speech.

State of the art

[02] Speech-to-text conversion systems (STT, Speech-To-Text) allow you to process spoken human speech in order to identify individual words in it and thus convert fragments of human speech into text. For example, such systems can be used in combination with so-called virtual assistant applications and provide improved interaction between the user and certain online services (for example, online shopping platforms, online booking systems, etc.) and/or his electronic device. This is especially important for novice users and/or for users with disabilities who at some point in time are unable to use the user-machine interfaces of such online services and/or electronic device to effectively interact with them. For example, a driver or visually impaired user may not be able to use the touch screen keyboard associated with their electronic device to navigate a doctor's website to make an appointment or an online shopping platform website. in order to form an order. However, the customer service staff of these online services may not be able to quickly assist such users, for example, due to a high volume of requests from other users.

[03] Thus, a virtual assistant application with support for the virtual interlocutor function (also known as a chatbot) can be used to navigate the user to the relevant online service and generate requests. In particular, a virtual assistant application can initiate a dialogue with the user in text or audio format, receive a request in the form of a speech fragment from the user, and use the received speech fragment recognized by the STT system to generate additional clarifying questions. We can say that in this way the virtual assistant application imitates a person helping the user by maintaining a dialogue with him.

[04] For example, when a virtual assistant application is used to schedule patient appointments with a doctor, it can be activated by a user contacting the doctor, such as by telephone, or starting a conversation on his website, and can greet the user and determine his needs with phrases “Hello, you have reached the office of Doctor House. How can I be of service to you today?” (“Hello, you have reached out to Doctor House’s office. How can I help you today?”) . In response, the user can say the following phrase: “I'd like to make an appointment with the doctor.” In addition, the virtual assistant application can receive an audio representation of this user request, generate a text representation using the STT system for further processing and provide the next follow-up question, such as requesting specific user profile information, user availability, meeting urgency, etc. the corresponding response to which the virtual assistant application is capable of processing in the same way using the STT system. Finally, based on the user's speech responses thus recognized and, for example, the doctor's availability, the virtual assistant application can fulfill the user's request by scheduling an appointment for him and storing data indicating him in the doctor's schedule, which can be confirmed by the virtual assistant application generating another a response indicating the following phrase: “Your appointment with Doctor House has been scheduled for December 20, 2021 at 11 AM.”

[05] However, one disadvantage of such applications may be the inability of the STT system used with them to determine the context for words from the resulting audio representations of the user's speech fragments. In particular, without identifying the context of a phrase, the STT system may be unable to properly identify semantic relationships between words and may therefore form incorrect textual representations of them. In the appointment scheduling example described above, the STT system may be pre-configured to expect the full form of "the Doctor House" and may therefore be unable to understand the word "the doctor" in the user's response, which may lead to additional unnecessary questions for the user. In another example, upon receiving an audio representation of the user's first and last name "Neville Longbottom", the STT system may produce a text representation of "Neville long bottom" that does not correspond to the user's profile data. This may result in a lower-quality virtual assistant application maintaining conversations with the user during the user's request, thereby negatively impacting the user's experience with the virtual assistant application.

[06] To solve the technical problem described above, some known approaches have been proposed.

[07] Patent application US20190005138A1 “Obtaining responsive information from multiple corpora” (Google LLC, published January 3, 2019) describes methods for automated assistants to search information in various alternative corpora.

[08] Patent application US20210166678A1 "Electronic device and controlling the electronic device" (Samsung Electronics Co Ltd, published June 3, 2021) describes an electronic device containing a processor capable of determining whether speech needs to be transmitted to a server storing a first dialogue system. user input via the microphone, wherein, based on determining that the user's speech is transmitted to the server, the communicator is controlled to transmit the user's speech and at least a portion of the stored conversation history information to the server.

[09] Patent application US2020382448A1 “Contextual feedback to a natural understanding system in a chat bot” (Microsoft Technology Licensing LLC, published July 21, 2020) describes a chatbot computing system containing a robot program controller and a natural speech processor.

Disclosure of the Invention

[010] The purpose of the present invention is to eliminate at least some of the disadvantages of known solutions.

[011] The developers of the present technology have determined that the quality of speech recognition provided by an STT system can be improved if the STT system is able to take into account the last phrase generated for the user by the virtual assistant application as context for the response. In particular, the developers implemented the STT system based on machine learning techniques (such as neural networks), which involve learning to recognize the user's speech fragments based not only on the corresponding text representations, but also on the phrases in response to which the user provides these speech fragments.

[012] An STT system trained in this manner can convert each word from a speech fragment into text within its specific associated context, which is expected to improve speech recognition performance and overall satisfaction for users using virtual assistant applications.

[013] According to the first aspect of the present technology, a method for generating textual representations of a user's speech fragment is implemented. The user's speech fragment is received by an electronic device associated with the user. The user's speech fragment is provided in response to computer-generated speech fragments output by the electronic device. The electronic device is capable of communicating with the server. The server is designed to generate computer-generated speech fragments. The method is performed by the server. The method includes: receiving by the server from an electronic device an audio signal that is an audio representation of a user's speech fragment, which is a response to a computer-generated speech fragment previously issued by the electronic device to the user; receiving by the server a computer-generated text string that is a text representation of the computer-generated speech fragment; and generating by the server, using the STT model, another text string, which is a text representation of the user's speech fragment, based on the audio signal and a computer-generated text string, taken into account as the context of the user's speech fragment.

[014] In some embodiments of the method, it further includes: the server, using the STT model, generating a first text string being a text representation of the user's first speech fragment based on the first audio signal being an audio representation of the user's first speech fragment; generating by the server, using a different model, a computer-generated text string based on the first text string; generation by the server, using a text-to-speech (TTS, Text-To-Speech) model, of a computer-generated audio signal, which is an audio representation of a computer-generated speech fragment, based on a computer-generated text string.

[015] In some embodiments of the method, the STT model is a neural network (NN, Neural Network) with an encoder-decoder architecture containing a stack of encoder layers and a stack of decoder layers, and generating another text string includes: generating by the server an audio vector representing audio signal using an audio vectorization algorithm; input by the server of the audio vector into the NN network encoder stack; generating by the server, using a text vectorization algorithm, a text vector representing a computer-generated text string; and the server inputting the text vector into the NN network decoder stack.

[016] In some embodiments of the method, the NN is a transformer-based NN.

[017] In some embodiments of the method, it further includes training an STT model in a training phase based on a training data set containing a plurality of training objects, wherein the training object comprises: (a) an indication of a training audio signal generated from the training speech fragment user spoken by the training user; (b) a first training text string, which is a text representation of the user's training speech fragment; and (c) a second training text string used to provide context for the first text string and being a textual representation of the corresponding computer-generated speech fragment in response to which the training user uttered the user's training speech fragment.

[018] In some embodiments of the method, it further includes the server receiving a first audio signal from the electronic device.

[019] In some embodiments of the method, the method further includes the server sending a computer-generated audio signal to the electronic device.

[020] In some embodiments of the method, the other model comprises a Natural Language Processing (NLP) model.

[021] In some embodiments of the method, one or more models are hosted on another server.

[022] In some embodiments of the method, the NLP model is a transformer-based NN model.

[023] In some embodiments of the method, the TTS model is a transformer-based NN model.

[024] In some embodiments of the method, it further includes: the server, using the STT model, generating a second text string based on a second audio signal that is an audio representation of the user's second speech fragment corresponding to the request following the computer-generated speech fragment; generating by the server, based on the second text string using a different model, another computer-generated text string that is a text representation of another computer-generated speech fragment to be provided to the user; generating by the server, from the other computer-generated text string using the TTS model, another computer-generated audio signal that is an audio representation of another computer-generated speech fragment to be provided to the user in response to the second speech fragment of the user; receiving by the server from the electronic device a third audio signal, which is an audio representation of a third speech fragment of the user, corresponding to another request following another computer-generated speech fragment; and generating by the server, using the STT model, based on the third audio signal, a computer-generated text string and the other computer-generated text string, a third text string that is a textual representation of the third speech portion of the user, taking into account the computer-generated text string and the other computer-generated text string as the context of the other next request.

[025] In some embodiments of the method, it further includes the server sending another computer-generated audio signal to the electronic device.

[026] According to the second aspect of the present technology, a server is implemented for generating text representations of a user's speech fragment. The user's speech fragment is received by an electronic device associated with the user and capable of communicating with the server. The user's speech fragment is provided in response to computer-generated speech fragments output by the electronic device. The server contains a processor and a machine-readable physical storage medium that stores instructions. When executing commands, the processor is capable of: receiving from an electronic device an audio signal, which is an audio representation of a user's speech fragment, which is a response to a computer-generated speech fragment previously issued by the electronic device to the user; receive a computer-generated text string that is a text representation of a computer-generated speech fragment; and generating, using the STT model, another text string, which is a text representation of the user's speech fragment, based on the audio signal and a computer-generated text string, taken into account as the context of the user's speech fragment.

[027] In some server embodiments, the processor is further capable of: generating, using the STT model, a first text string that is a textual representation of the user's first speech segment based on the first audio signal that is an audio representation of the user's first speech segment; generating, using another model, a computer-generated text string based on the first text string; and generating, using the TTS model, a computer-generated audio signal that is an audio representation of the computer-generated speech fragment based on the computer-generated text string.

[028] In some embodiments of the server, the STT model is an NN network with an encoder-decoder architecture containing a stack of encoder layers and a stack of decoder layers, and the processor is capable of generating another text string by: generating an audio vector representing the audio signal using an audio vectorization algorithm ; inputting the audio vector into the NN network encoder stack; generating, using a text vectorization algorithm, a text vector representing a computer-generated text string; and inputting the text vector into the NN network decoder stack.

[029] In some server embodiments, the NN is a transformer-based NN.

[030] In some embodiments of the server, the processor is further capable of training an STT model in a training phase based on a training data set containing a plurality of training objects, wherein the training object comprises: (a) an indication of a training audio signal generated from a training speech fragment of the user, spoken by the training user; (b) a first training text string, which is a text representation of the user's training speech fragment; and (c) a second training text string used to provide context for the first text string and being a textual representation of the corresponding computer-generated speech fragment in response to which the training user uttered the user's training speech fragment.

[031] In some server embodiments, the other model comprises an NLP model.

[032] In some server embodiments, one or more models are hosted on another server associated with the server.

[033] In the context of the present description, a transformer model is a model with an encoder-decoder architecture that uses attention mechanisms. Attention mechanisms can be used in encoder processing, decoder processing, and encoder-decoder interactions. A variety of attention mechanisms can be used in the transformer model.

[034] One component of the transformer model may be a self-attention mechanism. The difference between an attention mechanism and a self-attention mechanism is that a self-attention mechanism operates on similar representations, such as all encoder states in one layer. The self-attention mechanism is part of the transformer model, in which tokens interact with each other. Each token, in a sense, “observes” the other tokens in the sentence through an attention mechanism, gathers context, and updates its previous representation. Each input token in a self-attention mechanism receives three representations: (a) query, (b) key, and (c) value. A query is used when a token observes other tokens: it seeks information to better “understand” itself. The key reacts to the appearance of a request - it is used to calculate attention weights. The value is used to calculate the attention result: it provides information about the tokens that report that they need it (i.e., those tokens are given higher weights).

[035] Another component of the transformer model may be a masked self-attention mechanism. The decoder typically contains this special self-attention mechanism, which is different from the self-attention mechanism in the encoder. The encoder receives all the tokens simultaneously, while the tokens can observe all the tokens in the input sentence, and in the decoder the tokens are formed one by one, i.e. During the formation of the model, it is unknown which tokens will be formed in the future. To prevent the decoder from “looking ahead,” the transformer model uses a mechanism of masked self-attention, i.e. future tokens are masked.

[036] Another component of the transformer model may be a multi-headed attention mechanism. It should be noted that understanding the role of a word in a sentence requires understanding how it relates to the different parts of the sentence. This is important not only when processing the initial sentence, but also when forming goals. As a result, thanks to this type of attention mechanism, the transformer model can “concentrate” on various things. The multi-headed attention mechanism, instead of one attention mechanism, contains several independently working heads. It can be implemented in the form of several attention mechanisms, the results of which are combined.

[037] The transformer model encoder may comprise a self-attention encoder mechanism and a feed-forward network block. The encoder self-attention mechanism may be a multi-headed attention mechanism used to keep tokens observing each other. Queries, keys, and values are calculated based on encoder states. The forward distribution network block receives information from tokens and processes this information.

[038] A transformer model decoder may comprise a (masked) decoder self-attention mechanism, a decoder-encoder attention mechanism, and a feed-forward network. The decoder's masked self-attention mechanism may be a masked multi-headed attention mechanism used to observe previous tokens on the part of the tokens. Queries, keys, and values are calculated based on the decoder states. The decoder-encoder attention mechanism can be a multi-headed attention mechanism used to view source information by target tokens. Queries are calculated based on decoder states, and keys and values are calculated based on encoder states. The forward distribution network block receives information from tokens and processes this information.

[039] The tokens in the encoder can be said to communicate with each other and update their representations. It can also be said that in a decoder, the target token first looks at the previously formed target tokens, then the source, and finally updates its representations. This can be repeated in several layers. In one non-limiting embodiment, this may be repeated six times.

[040] As described above, in addition to the attention mechanism, the layer contains a feedforward network block. For example, a feedforward network block can be represented by two linear layers with a nonlinear connection of the form “truncated linear transformation” (ReLU, Rectifier Linear Unit) between them. After viewing other tokens using the attention mechanism, the model uses a feedforward network block to process this new information. The transformer model may further contain residual connections to add the block's inputs to its outputs. Residual links can be used to combine layers. In the transformer model, residual connections can be used after the corresponding attention mechanism and feedforward network block. For example, the Add & Norm layer may be provided with (a) attention engine inputs through a residual connection and (b) attention engine outputs. The result of the Add & Norm layer can then be provided to a feedforward network block or other attention mechanism. In another example, the Add & Norm layer may be provided with (a) feedforward network block inputs via a residual link and (b) feedforward network block outputs. As described above, the transformer model may contain "Add & Norm" layers. In general, such a layer can independently normalize the vector representation of each example in the batch. This is done to control the "flow" into the next layer. Layer normalization allows you to increase convergence stability and, in some cases, even quality.

[041] As used herein, the term “server” means a computer program executed by associated hardware and capable of receiving requests (eg, from client devices) over a network and executing or initiating execution of those requests. The hardware may be one physical computer or one computer system, which is not essential for this technology. In the present context, the expression "server" does not mean that every task (for example, a received command or request) or any particular task is accepted, executed or launched by the same server (i.e. the same software and/or hardware ). This expression means that any number of software or hardware can receive, send, execute, or initiate the execution of any task or request, or the results of any tasks or requests. All of these software and hardware may be a single server or multiple servers, both of which are implied by the expression “at least one server.”

[042] As used herein, the term “client device” means any computer hardware capable of executing programs suitable for solving a given task. Thus, some (non-limiting) examples of client devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches and gateways. It should be noted that in this context, a device functioning as a client device may also function as a server for other client devices. The use of the expression “client device” does not exclude the use of multiple client devices to receive, send, perform, or initiate execution of any task or request, or the results of any tasks or requests, or steps of any method described herein.

[043] As used herein, the term “database” means any structured collection of data, regardless of its specific structure, database management software, or computer hardware for storing, using, or otherwise making use of that data. The database may reside in the same hardware as the process that stores or uses the information stored in the database, or the database may reside in separate hardware, such as a dedicated server or multiple servers.

[044] As used herein, the expression “information” includes information of any kind or kind capable of being stored in a database. Thus, information includes audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc. .etc.), documents, spreadsheets, word lists, etc., but is not limited to them.

[045] As used herein, the expression "component" includes a designation of software (suitable for certain hardware) necessary and sufficient to perform a particular function or multiple functions.

[046] As used herein, the expression "computer-readable storage medium" means media of any kind or form, including random access memory (RAM), read-only memory (ROM), disks (CD-ROM, DVD, floppy disks, hard drives, etc.), USB flash drives, solid state drives, tape drives, etc.

[047] As used herein, the numerals "first", "second", "third", etc. are used only to indicate differences between the nouns they refer to, but not to describe any specific relationships between those nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy or classification, in this case, of servers, nor that their use (in itself) does not imply the presence of a “second server” in any situation. Moreover, as occurs herein in other contexts, reference to a “first” element and a “second” element does not preclude the fact that the two elements may in fact be the same element. Thus, for example, in some cases the “first” server and the “second” server may be the same software and/or hardware, and in other cases different software and/or hardware.

[048] Each embodiment of the present technology relates to at least one of the above-mentioned objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology associated with attempting to achieve the above purpose may not be consistent with that purpose and/or may be consistent with other purposes not expressly mentioned herein.

[049] Additional and/or alternative features, aspects and advantages of embodiments of the present technology are set forth in the following description, the accompanying drawings, and the claims.

Brief description of drawings

[050] The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the accompanying drawings.

[051] In FIG. 1 is a diagram of an example computer system for implementing certain embodiments of systems and/or methods in accordance with the present technology.

[052] In FIG. 2 illustrates a networked computing environment suitable for some embodiments of the present technology.

[053] In FIG. 3 shows a diagram of the process of creating a virtual assistant application hosted on a server from the network computing environment shown in FIG. 2, a corresponding speech response to a speech fragment according to non-limiting embodiments of the present technology.

[054] In FIG. 4 is a diagram of a machine learning model architecture suitable for use in certain non-limiting embodiments of the present technology.

[055] In FIG. 5 shows a diagram of the TTS model for generating an appropriate speech response to a user's speech fragment, implemented based on the architecture of the machine learning model presented in FIG. 4, and hosted on a server from the network computing environment shown in FIG. 2, according to non-limiting embodiments of the present technology.

[056] In FIG. 6 is a diagram of the STT model implemented based on the machine learning model architecture shown in FIG. 4, during its training by a server from the network computing environment shown in FIG. 2, generating a textual representation of the user's speech fragment for the TTS model to generate an appropriate speech response to it in accordance with non-limiting embodiments of the present technology.

[057] In FIG. 7 is a diagram of the STT model shown in FIG. 6 during its application in the use phase according to certain non-limiting embodiments of the present technology.

[058] In FIG. 8 is a flow diagram of a method for generating a server from the network computing environment shown in FIG. 2, a textual representation of a user's speech fragment according to non-limiting embodiments of the present technology.

Carrying out the invention

[059] The examples and conventional language presented here are intended to provide a better understanding of the principles of this technology and not to limit its scope to such specifically provided examples and conditions. It is obvious that those skilled in the art are capable of developing various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its spirit and scope.

[060] In addition, to facilitate a better understanding, the following description may contain simplified embodiments of the present technology. Those skilled in the art will appreciate that other embodiments of the present technology may be significantly more complex.

[061] In some cases, useful examples of modifications to the present technology are provided. They promote understanding, but also do not define the scope or boundaries of the actual technology. The presented list of modifications is not exhaustive and other modifications may be developed by one skilled in the art within the scope of this technology. In addition, if in some cases modifications are not described, this does not mean that they are impossible and/or that the description contains the only possible implementation of a particular element of this technology.

[062] Moreover, the description of the principles, aspects and embodiments of the present technology, as well as specific examples thereof, are intended to cover their structural and functional equivalents, whether currently known or developed in the future. For example, those skilled in the art will appreciate that any block diagrams described herein correspond to the conceptual representations of illustrative circuit diagrams implementing the fundamentals of the present technology. It should also be clear that any flowcharts, process diagrams, state transition diagrams, pseudo codes, etc. correspond to various processes that may be represented on a computer-readable physical storage medium and may be performed by a computer or processor, whether such computer or processor is explicitly shown or not.

[063] The functions of the various elements shown in the drawings, including any functional unit designated as a “processor” or a “graphics processor,” may be implemented using dedicated hardware, as well as using hardware capable of executing the associated software. If a processor is used, these functions may be performed by one dedicated processor, one shared processor, and/or a plurality of separate processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor, such as a central processing unit (CPU), or a special purpose processor, such as a graphics processing unit (GPU). In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to hardware capable of executing software and may include, but is not limited to, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), software ROM, RAM, and/or non-volatile storage device. Other hardware, general purpose and/or custom, may also be included.

[064] Software modules, or simply modules intended to be implemented in software, may be represented herein as any combination of flowchart elements or other elements indicating the execution of process steps and/or containing a textual description. Such modules may be executed by hardware, shown explicitly or implicitly.

[065] Given the foregoing principles, some non-limiting examples are provided below to illustrate various embodiments of aspects of the present technology.

Computer system

[066] In FIG. 1 illustrates a computer system 100 suitable for use in some embodiments of the present technology. The computer system 100 includes various hardware elements, including one or more single-core or multi-core processors, generally represented by a processor 110, a graphics processing unit (GPU) 111, a solid-state drive 120, a RAM 130, a display interface 140, and an input/output interface 150.

[067] Communication between various elements of the computer system 100 may occur through one or more internal and/or external buses 160 (such as a PCI bus, a USB bus, an IEEE 1394 FireWire bus, a SCSI bus, a Serial-ATA bus, etc. ) to which various hardware elements are connected electronically.

[068] I/O interface 150 may couple to touch screen 190 and/or one or more internal and/or external buses 160. Touch screen 190 may be included in a display. In some embodiments, the touch screen 190 is a display. The touch screen 190 may also be referred to as a screen 190. In FIGS. 1 embodiments, touch screen 190 includes touch hardware 194 (e.g., pressure-sensitive cells embedded in the display layer to capture physical interaction between the user and the display) and a touch I/O controller 192 that provides communication with interface 140 display and/or with one or more internal and/or external buses 160. In some embodiments, the I/O interface 150 may be coupled to a keyboard (not shown), mouse (not shown), or touch pad (not shown) that provides user interaction with the computer system 100 in addition to or instead of the touch screen 190. In some embodiments, computer system 100 may include one or more microphones (not shown). Microphones can record audio, such as a user's speech. The user's speech fragments may be converted into commands to control the computer system 100.

[069] It should be noted that in some non-limiting embodiments of the invention, certain elements of the computer system 100 may be missing. For example, there may be no touch screen 190, particularly if the computer system is implemented as a smart speaker device (but not limited to).

[070] According to embodiments of the present technology, solid state drive 120 stores software instructions suitable for loading into RAM 130 and execution by processor 110 and/or GPU processor 111. Program instructions may, for example, be part of a library or application.

Network computing environment

[071] In FIG. 2 is a diagram of a networked computing environment 200 suitable for use with certain embodiments of systems and/or methods of the present technology. Networked computing environment 200 includes a server 250 coupled through a communications network 240 to an electronic device 210. In non-limiting embodiments of the present technology, the electronic device 210 may be coupled to a user 230.

[072] In some non-limiting embodiments of the present technology, server 250 is implemented as a traditional computer server and may include some or all of the elements of the computer system 100 shown in FIG. 1. In one non-limiting example, server 250 is implemented as a Dell™ PowerEdge™ server running the Microsoft™ Windows Server™ operating system, but it may also be implemented using any other suitable hardware, application software and/or or embedded software or a combination thereof. In the illustrated non-limiting embodiments of the present technology, server 250 is a single server. In other non-limiting embodiments of the present technology (not shown), the functions of the server 250 may be distributed among multiple servers.

[073] In addition, electronic device 210 may be any computer hardware capable of executing programs suitable for solving a given task. Thus, some non-limiting examples of electronic device 210 include personal computers (desktops, laptops, netbooks, etc.), smartphones, and tablets. In some non-limiting embodiments of the present technology, electronic device 210 may also include some or all of the elements of the computer system 100 shown in FIG. 1. It should be noted that the electronic device 210 may include additional elements, such as a microphone (not separately shown) for converting received sounds recorded in the vicinity of the electronic device 210, such as speech fragments of the user 230, into a machine-readable format, such as a digital audio format, including, for example, MP3, Ogg, etc., and a speaker (also not separately shown) for reproducing incoming audio signals in the vicinity of the electronic device 210, as described below.

[074] Additionally, in some non-limiting embodiments of the present technology, the networked computing environment 200 may include a second server 260 coupled via a communications network 240 to the electronic device 210 and to the server 250. It is contemplated that the second server 260 may be implemented similarly server 250.

[075] According to some non-limiting embodiments of the present technology, networked computing environment 200 may provide and/or maintain automatic communication with user 230, as described below.

[076] For example, in some non-limiting embodiments of the present technology, the second server 260 may be associated with a service provider providing services to users, such as user 230, which the second server 260 may include a virtual assistant application 265 to assist in obtaining. Accordingly, the second server 260 may provide access to the virtual assistant application 265 to an electronic device associated with the communications network 240, such as the electronic device 210.

[077] In general, a virtual assistant application 265, also referred to herein as a chatbot application, may initiate and then maintain automatic communication with a user 230 and assist in obtaining services from a service provider associated with a second server 260. For example, a virtual assistant application 265 The assistant may be enabled by activating the appropriate control of the user 230 accessing the service provider using the electronic device 210, such as by dialing the appropriate telephone number or launching the virtual assistant application 265 on a website associated with the service provider. In addition, while maintaining automatic communication with user 230, virtual assistant application 265 may receive user speech 235 containing a user request from user 230 and provide the user with a computer-generated speech 270 that may contain additional questions that clarify the details of user 230 accessing the user. service provider.

[078] The implementation of the virtual assistant application 265 is not subject to any restrictions and may depend on the services provided by the service provider associated with the second server 260. For example, if the service provider is a doctor or medical clinic, then the virtual assistant application 265 may Manage patient appointments (e.g., schedule, change, or cancel appointments). In another example, the service provider may be an online shopping platform that sells various products, such as goods and services, and the virtual assistant application 265 may receive and send online orders from user 230. In yet another example, the service provider may be a brick-and-mortar (not online) grocery store or brick-and-mortar restaurant, and the virtual assistant application 265 can accept and send grocery orders for the user 230. Other options for service providers and respective users of the virtual assistant application 265 are also possible without going beyond the boundaries of the present technology.

[079] In a specific, non-limiting example, the virtual assistant application 265 may be implemented as the ALISA™ virtual assistant application provided by Yandex LLC (16 Lev Tolstogo St., Moscow, 119021, Russia). However, it should be noted that the virtual assistant application 265 may be implemented as any other commercially available or proprietary virtual assistant application.

[080] Specifically, according to some non-limiting embodiments of the present technology, to generate a corresponding computer-generated speech fragment 270 for a user speech fragment 235, the virtual assistant application 265 may: (1) cause the microphone of the electronic device 210 to receive the speech fragment 235 the user to generate from it an audio representation 245 representing the user's speech fragment 235; (2) initiate transmission of an audio representation 245 of the speech fragment to the server 250 to generate a text representation 255 of the speech fragment; (3) initiate transmission of the textual speech fragment representation 255 to the second server 260 to generate a computer-generated speech fragment 270 based on the textual speech fragment representation 255; and (4) initiate transmission of data indicating the corresponding computer-generated speech fragment 270 to the electronic device 210 for reproduction to the user 230.

[081] According to some non-limiting embodiments of the present technology, the audio representation 245 of a speech fragment may correspond to the amplitude-time representation of the user's speech fragment 235, such as a digitized representation of the corresponding audio signal in one of the above-mentioned formats. In other non-limiting embodiments of the present technology, the audio representation of the speech fragment may correspond to the time-frequency representation of the user's speech fragment 235. In some non-limiting embodiments of the present technology, the time-frequency representation may be a corresponding spectrogram associated with the user's speech fragment 235. Formation of spectrograms is not the task of this technology. However, in brief, server 250 can generate a corresponding spectrogram of user speech 235 by applying a Fourier transform to portions of the time-amplitude representation of user speech 235 sequentially determined based on a predetermined time window. In some non-limiting embodiments of the present technology, the corresponding spectrogram can be converted using a chalk scale. For example, a corresponding spectrogram may be generated as described in the same applicant's patent application US17408958, "Method and system for classifying a user of an electronic device" (filed August 23, 2021), the contents of which are incorporated herein by reference in their entirety.

[082] Additionally, according to some non-limiting embodiments of the present technology, the textual speech fragment representation 255 corresponds to a text string representing the user's speech fragment 235, which may be stored and sent by the server 250 in one of the text formats, such as plain text.

[083] Below with reference to FIG. 3-7 describe the generation by the server 250 and the second server 260 of a corresponding computer-generated speech fragment 270 for the user's speech fragment 235. It should be noted that the server 250 and the second server 260 may be implemented by the same element or by different elements without going beyond the scope of the present technology.

[084] In FIG. 3 is a diagram of a process for a networked computing environment 200 to generate a corresponding computer-generated speech fragment 270 in response to receiving a user speech fragment 235 in accordance with certain non-limiting embodiments of the present technology.

[085] First, as described above, the virtual assistant application 265 may (a) cause the user's speech fragment 235 to be received by the electronic device 210, thereby generating a corresponding audio representation 245 of the speech fragment, and (b) cause the electronic device 210 to transmit the audio representation. 245 of the speech fragment to the server 250 to recognize speech in the user's speech fragment 235 and generate a text representation 255 of the speech fragment.

[086] To generate a text representation 255 of a speech fragment from an audio representation 245 of a speech fragment, the server 250 may contain (or otherwise access) an STT model 302 . The architecture and configuration of STT model 302 for speech recognition are described in more detail below with reference to FIG. 6 and 7. However, in general, the STT model 302 is capable of processing spoken natural speech, such as a speech fragment 235 of a user 230, to recognize individual words therein and thereby generate a textual representation 255 of the speech fragment, such as a text string, such as “Hello, it's John Smith, I want to make an appointment with my family doctor” or “Hi, I'd like to order a pizza" (“Hello, I would like to order pizza”), etc.

[087] In some non-limiting embodiments of the present technology, having generated a textual speech fragment representation 255, the server 250 may send it to a second server 260 for further processing of natural speech into a textual speech fragment representation 255.

[088] To this end, according to some non-limiting embodiments of the present technology, the second server 260 may contain (or otherwise access) an NLP model 304 that can (1) understand the language in the textual speech fragment representation 255, those. determine the contextual and grammatical relationships between words therein, and (2) generate, based on the textual representation 255 of a speech fragment, a computer-generated text string 370, which is a textual representation of the corresponding computer-generated speech fragment 270, to be presented to the user 230. Architecture and configuration of the model 304 NLPs are described below.

[089] In some non-limiting embodiments of the present technology, in order to generate a corresponding computer-generated speech fragment 270 for the user 230, the virtual assistant application 265 may initiate transmission of a computer-generated text string 370 to a TTS model 306, which may convert the computer-generated text string 370 to natural language speech element and then transmit its data to electronic device 210 for playback to user 230. The architecture and configuration of the TTS model 306 is described below.

[090] It should be clear that, although in FIGS. In 3 embodiments of the invention, the NLP model 304 and the TTS model 306 are hosted on the second server 260; in other non-limiting embodiments of the present technology, each of these models may contain and therefore train and use the server 250. In other words, the server 250 and the second server 260 may be implemented as a single server. In other non-limiting embodiments of the present technology, the NLP model 304 and the TTS model 306 may be hosted on separate third-party servers (not shown) connected to the communications network 240.

Communication network

[091] In some non-limiting embodiments of the present technology, communications network 240 is the Internet. In alternative, non-limiting embodiments of the present technology, communications network 240 may be implemented as any suitable Local Area Network, WAN, private communications network, or the like. Obviously, the communication network embodiments are provided for illustrative purposes only. The implementation of the respective communication lines (not specifically designated) between the server 250, the second server 260 and the electronic device 210 on the one hand and the communication network 240 on the other hand depends, among other things, on the implementation of the server 250, the second server 260 and the electronic device 210. Only in As a non-limiting example, in those embodiments of the present technology where the electronic device 210 is implemented as a wireless communication device, such as a smartphone, the communication link may be implemented as a wireless communication link. Examples of wireless communication links include a 3G network channel, a 4G network channel, etc. The communications network 240 may also utilize a wireless connection to the server 250 and to the second server 260.

Machine learning model architecture

[092] In FIG. 4 illustrates a machine learning model architecture 400 suitable for use in certain non-limiting embodiments of the present technology. The machine learning model architecture 400 is based on a transformer-based neural network model architecture, as described, for example, in Vaswani et al. “Attention Is All You Need,” Proceedings of the ^31st Conference on Neural Information Processing Systems (NIPS, 2017), the contents of which are incorporated herein by reference in their entirety.

[093] Thus, the machine learning model architecture 400 may comprise an encoder layer stack 402 and a decoder layer stack 403 that are capable of processing input data 412 and target data 417 of the machine learning model architecture 400, respectively.

[094] In addition, the encoder block 404 of the encoder layer stack 402 includes a multi-head attention (MHA) encoder layer 406 and an encoder feedforward NN network layer 408. MHA encoder layer 406 contains dependencies between parts of the input data 412 provided to it. For example, if input data 412 contains text data, such as a text sentence, MHA encoder layer 406 may contain dependencies between words of the sentence. In another example, where the input data 412 of the encoder layer stack 402 contains an audio signal, for example, representing a fragment of human speech (such as the user speech fragment 235 described above), the MHA encoder layer 406 may contain dependencies between specific sounds and/or acoustic features of the human speech fragment. speech. Such dependencies may be used by the MHA encoder layer 406 to determine the context information of a portion of the input data 412 of the encoder layer stack 402 (e.g., representing a word from a sentence or an acoustic feature of a user's speech fragment 235) associated with another portion of the input data 412.

[095] In addition, the feedforward NN layer 408 is capable of transforming its input data from the MHA encoder layer 406 into a format accepted by one or more subsequent layers of the machine learning model architecture 400, such as the MHA encoder-decoder layer 409 (described below). The encoder forward propagation NN layer 408 typically does not contain the dependency of the encoder MHA layer 406, so the input data of the encoder forward propagation NN layer 408 can be processed in parallel.

[096] In addition, the input data 412 of the encoder layer stack 402 may be represented by a plurality of 414 input vectors generated by the input data vectorization algorithm 410 . In general, the input data vectorization algorithm 410 is capable of generating fixed-dimensional vector representations of the input data 412 in a corresponding vector representation space. In other words, if the input data 412 contains data indicative of a user's speech fragment 235, such as an audio representation 245 of the speech fragment, then the input data vectorization algorithm 410 may generate a set of input vectors 414 in which the coordinates of vectors representing similar acoustic features of the audio representation 245 speech fragments (eg, representing similar-sounding parts from the user's speech fragment 235) are located closer to each other in the corresponding vector representation space.

[097] In general, the implementation of the input data vectoring algorithm 410 depends on the format of the input data 412 provided to it. If the input data 412 contains fragments of human speech, as in the example presented above, then the input data vectoring algorithm 410 may be implemented as an audio vectorizing algorithm , including but not limited to Seq2Seq (Sequence-to-Sequence Autoencoder) audio vectorization algorithm, Convolutional Vector Regression audio vectorization algorithm, Letter-based n-gram audio vectorization algorithm, Model-based audio vectorization algorithm LSTM (Long Short-Term Memory), etc.

[098] A vector of a set of vectors 414 may contain numeric values, such as 468 floating point values, representing a corresponding portion of the input data 412, such as a word, a portion of a user's speech 235, or the like.

[099] Generating the plurality of input vectors 414 may also include the use of a positional vectorization algorithm (not shown) capable of capturing positional data in portions of the input data 412. For example, if the input data 412 contains a text sentence, then the positional vectorization algorithm may generate a vector indicating the positional data of words in this text sentence. In another example, where input data 412 represents a speech fragment 235 of a user, a positional vectorization algorithm may capture positional data of acoustic features associated with that fragment. In other words, the position vectorization algorithm may generate a vector containing context information from the input data 412, which may be added to the plurality of input vectors 414. There are no restrictions imposed on the implementation of the positional vectorization algorithm. For example, it may (among other things) comprise a positional sine vectorization algorithm, a positional frame overlay vectorization algorithm, and a positional convolutional vectorization algorithm.

[0100] It should be noted that the encoder layer stack 402 may contain multiple encoder blocks, such as 6 or 12, implemented similar to encoder block 404.

[0101] The decoder block 405 of the decoder layer stack 403 of the machine learning model architecture 400 includes (a) an MHA decoder layer 407 and (b) a decoder forward propagation NN network layer 411, which can typically be implemented like an MHA encoder layer 406 and an NN network layer 408 direct propagation encoder, respectively. However, the architecture of the decoder block 405 differs from the architecture of the encoder block 404 in that the decoder block 405 further includes an encoder-decoder MHA layer 409. The encoder-decoder MHA layer 409 is capable of (a) receiving input vectors from the encoder layer stack 402 and from the decoder MHA layer 407 and therefore (b) learning, as described below, the dependencies between the input data 412 and the target data 417 ( such as text data) of the machine learning model architecture 400 input into the decoder layer stack 403. That is, the outputs of the encoder-decoder MHA layer 409 are attention vectors containing data indicative of relationships between corresponding portions of the input data 412 and the target data 417.

[0102] As with input data 412, for the purpose of supplying target data 417 to decoder block 405, a target data vectorization algorithm 415 may be applied to target data 417 to generate a plurality of target vectors 419 containing numeric representations of corresponding portions of target data 417. Similarly input data vectorization algorithm, the target data vectorization algorithm 415 may form mappings of the target data 417 into a corresponding vector representation space. For example, if the target data 417 contains text data, such as a text sentence, the target data vectorization algorithm 415 may generate a set of target vectors 419 in which the coordinates of vectors representing words of a text sentence with similar meaning are located closer to each other in the corresponding vector representation space . Thus, the target data vectorization algorithm 415 may be implemented as a text vectorization algorithm, including, but not limited to, a Word2Vec (Word to Vector) text vectorization algorithm, a GloVe (Global Vectors for Word Representation) text vectorization algorithm, and the like.

[0103] It is contemplated that the target data vectorization algorithm 415 may be implemented similar to the input data vectorization algorithm 410 . Additionally, a positional algorithm may be applied to the plurality of target vectors 419 to capture positional data from portions of the target data 417, as described above with respect to the plurality of input vectors 414.

[0104] As described below, machine learning model architecture 400 may obtain input data 412 and target data 417 from a digital object, such as, for example, STT training digital object 640, as described with reference to FIG. 6.

[0105] It should be noted that the decoder layer stack 403 may contain multiple decoder blocks, such as 6 or 12, implemented similar to the decoder block 405. Once the machine learning model architecture 400 is trained, all blocks of the encoder layer stack 402 and decoder layer stack 403 are assumed to have different weights used in generating the output data 425. A backpropagation algorithm may be used to adjust the weights during training against the machine learning model architecture 400. and differences between input data 415 and output data 425 can be determined and minimized. Such differences can be expressed by a loss function, such as a cross-entropy loss function.

[0106] It should be understood that in non-limiting embodiments of the present technology, other embodiments of the loss function are also possible, such as the root mean square error loss function, the Huber loss function, the piecewise linear loss function, etc.

[0107] The output 425 of the machine learning model architecture 400 may include an output vector corresponding to a vector from a plurality of 414 input vectors. For example, as described below, in those embodiments of the invention where the input data 412 of the machine learning model architecture 400 includes an audio speech fragment representation 245 for a user's speech fragment 235, the output vector may contain probabilities indicating the corresponding portion of the text speech fragment representation 255.

[0108] It should be understood that the machine learning model architecture 400 described with reference to FIG. 4 is simplified for better understanding and that the actual implementation of the machine learning model architecture 400 may contain additional layers and/or blocks, for example, as described in the aforementioned work of Vaswani et al. In particular, in some embodiments of the machine learning model architecture 400, each encoder block 404 and each decoder block 405 may also include a layer of normalization operations. Additionally, generating output data 425 may include applying a softmax normalization function to the output of decoder layer stack 403, etc. Those skilled in the art will appreciate that these operations are widely used in neural networks and deep learning models, such as machine learning model architecture 400.

NLP model

[0109] As used herein, an NLP model, such as the NLP model 304 described below, is a machine learning model trained to read, understand, and generate snippets of natural speech. In other words, it can be said that the NLP model 304 can perform two different processes: (a) a natural language understanding (NLU) process, for example, to understand the text representation 255 of a speech fragment generated by the STT model 302, transforming the text representation 255 speech fragment into structured data, and (b) a natural language generation (NLG) process for generating a computer-generated text string 370 from the structured data.

[0110] Thus, the computer-generated text string 370 may, for example, point to the next line of dialogue of the virtual assistant application corresponding to the user's speech fragment 235. In particular, in the case of speech fragment 235 of the user “Hello, it's John Smith, I want to make an appointment with my family doctor” (“Hello, this is John Smith, I want to make an appointment with my family doctor”) a computer-generated text string 370 may indicate the following phrase: “Sure! Can I have your medical insurance card number?” (“Of course! May I have your health insurance card number?”).

[0111] In some non-limiting embodiments of the present technology, the NLP model 304 may be implemented based on an NN network, such as an LSTM NN or a recurrent NN. However, according to some non-limiting embodiments of the present technology, the NLP model 304 may be implemented as a transformer-based NN model. However, the NLP model 304 may contain some or all of the elements of the machine learning model architecture 400 described above.

[0112] In particular, in some non-limiting embodiments of the present technology, NLP model 304 may also include an encoder layer stack 402 and a decoder layer stack 403, including multiple encoder and decoder blocks. However, the respective numbers of such blocks in the encoder layer stack 402 and in the decoder layer stack 403 may differ from their number in the machine learning model architecture 400.

[0113] For example, in some non-limiting embodiments of the present technology, NLP model 304 may include one encoder block and thirteen decoder blocks, implemented like encoder block 404 and decoder block 405, respectively, as described above. In other non-limiting embodiments of the present technology, the NLP model 304 may contain no encoder blocks and contain several decoder blocks, for example, 6, 12, or 96 (in which case the model 304 may be referred to as a Generative Pre-trained Transformer (GPT) Transformer)). However, in other non-limiting embodiments of the present technology, the NLP model 304 may contain only encoder blocks, such as 12, 24, or 36, and no decoder blocks (in which case, the NLP model 304 may be referred to as "Bidirectional Encoder Views from Transformers" ( BERT, Bidirectional Encoder Representations from Transformers)).

[0114] Other configurations of encoder layer stack 402 and decoder layer stack 403 are also possible to implement NLP model 304 without going beyond the boundaries of the present technology.

[0115] In addition, to train the NLP model 304 to generate computer-generated text strings in response to text representations of speech fragments generated by STT model 302, for example, computer-generated text string 370 for text speech fragment representation 255 for user speech fragment 235, a second server 260 may feed the NLP model 304 an NLP training dataset containing a plurality of NLP training digital objects.

[0116] According to some non-limiting embodiments of the present technology, an NLP training digital object of a plurality of NLP training digital objects may comprise (a) a first user-generated text string and (b) a second user-generated text string corresponding to the first user-generated text string. For example, for this purpose, an NLP training digital object can be obtained on the Internet from public conversations on social media, such as comments or messages on a social networking website or discussions on a forum website. In another example, the first user-generated text string and the second user-generated text string may be generated based on conversations between various technical support services, for example, between service provider services (described above) and their clients.

[0117] Additionally, according to some non-limiting embodiments of the present technology, the second server 260 may provide (a) a first user-generated text string to the encoder layer stack 402 and (b) a second user-generated text string to the decoder layer stack 403, as described above. Thus, the second server 260 may feed a plurality of NLP training digital objects, containing, for example, thousands or hundreds of thousands of similar NLP training digital objects, to the NLP model 304 and thereby train the NLP model 304 to generate output 425 containing probability distribution vectors for the words. a computer-generated text string of the use stage, which is a text representation of the speech fragment of the user of the use stage.

[0118] In addition, a loss function representing the differences between input data 412 and output data 425 may be determined, and by minimizing such a loss function, the second server 260 may determine weights for the nodes of the NLP model 304 using a backpropagation algorithm. Additional information on training an NLP model 304 can be found, for example, in the article “Towards a Human-like Open-Domain Chatbot” (Adiwardana et al., Google Research), the contents of which are incorporated herein by reference in their entirety.

[0119] In those embodiments of the invention where the NLP model 304 contains only an encoder layer stack 402 or only a decoder layer stack 403, the NLP model 304 may first be pre-trained based on a more general training dataset and then fine-tuned based on the training set the NLP data described above, as described in detail, for example, in the article “Language Models are Few-Shot Learners” (Brown et al., OpenAI), the contents of which are incorporated herein by reference in their entirety.

[0120] Thus, the second server 260 may train the NLP model 304 to generate a computer-generated text string 370 in response to the textual speech fragment representation 255 for the user's speech fragment 235.

Model TTS

[0121] As described above with reference to FIG. 3, upon receiving a computer-generated text string 370 from the NLP model 304, the virtual assistant application 265 may send the computer-generated text string 370 to the TTS model 306 to generate a computer-generated audio signal indicating a corresponding computer-generated speech fragment 270 for the user's speech fragment 235.

[0122] In general, as used herein, TTS model 306 is a machine learning model capable of converting elements of natural speech as text, such as spoken language. There are no restrictions on the implementation of the 306 TTS. It may include, but is not limited to, a TTS WaveNet model, a TTS Deep Voice model, a TTS Tacotron model, etc.

[0123] According to some non-limiting embodiments of the present technology, the TTS model 306 may be implemented as a transformer-based NN model and thus may comprise some or all of the elements of the machine learning model architecture 400 described above with reference to FIG. . 4.

[0124] In FIG. 5 is a diagram of a TTS model 306 based on the machine learning model architecture 400 shown in FIG. 4, according to some non-limiting embodiments of the present technology. Thus, as described below, input data 412 of TTS model 306 may contain text strings, and target data 417 may contain audio representations of user speech fragments corresponding to text strings of input data 412.

[0125] In some non-limiting embodiments of the present technology, TTS model 306 may include additional elements for processing input data 412 and target data 417. In particular, for input data 412, TTS model 306 may include a text-to-phoneme conversion algorithm 502, capable of forming a phonetic representation of an input string such as words or sentences. For example, the phonetic representation of a word may contain a corresponding phonetic transcription according to the International Phonetic Alphabet (IPA).

[0126] According to some non-limiting embodiments of the present technology, to generate a plurality of input vectors 414 corresponding to vector representations of the phonetic representation of the input text string in the corresponding vector representation space, as described above, TTS model 306 includes an additional NN network - pre-network 504 coder. In some non-limiting embodiments of the present technology, the pre-encoder network 504 may be implemented as a Convolutional NN (CNN) containing a predetermined number of layers, such as three.

[0127] In some non-limiting embodiments of the present technology, the audio representation of a user's speech fragment input into the TTS model 306 as part of the target data 417 may be generated similar to the audio speech representation 245, as described above with reference to FIG. 2. To this end, in some non-limiting embodiments of the present technology, the audio representation for TTS model 306 may, for example, contain a corresponding chalk spectrogram associated with the user's speech fragment.

[0128] In some non-limiting embodiments of the present technology, to obtain an audio representation by the decoder layer stack 403, the TTS model 306 may further comprise yet another additional NN network capable of generating a plurality of target vectors 419 based on the audio representations of a user's speech segment. In particular, this NN network can contain a predetermined number of fully connected layers (for example, two), each of which contains, in particular, 256 hidden nodes with an activation function of the form ReLU.

[0129] Additionally, according to some non-limiting embodiments of the present technology, TTS model 306 may be trained, for example, by second server 260, to determine correspondences between phonemes of words or phrases from input data 412 and audio representations of user speech fragments from target data 417. Thus, the second server 260 may train the TTS model 306 to generate computer-generated audio signals indicative of computer-generated speech fragments to be provided in response to receiving a corresponding text string, such as computer-generated text string 370.

[0130] To this end, the second server 260 may train a TTS model 306 based on a TTS training dataset containing a plurality of training TTS digital objects. In particular, according to some non-limiting embodiments of the present technology, a TTS training digital object of a plurality of TTS training digital objects may comprise (1) a training text string and (2) a corresponding training audio representation, such as a chalk spectrogram (described above) , a user speech fragment generated by a training user pronouncing a training text string.

[0131] In some non-limiting embodiments of the present technology, a training text string may be prepared in advance to be spoken by a training user to form a corresponding audio representation associated with it. For example, the training text string “Hello! You have reached the office of Doctor House. How can we help you today?” (“Hello, you have reached Dr. House's office. How can we help you today?”) and the training user can speak this training text string and record his speech portion using an electronic device similar to the electronic device 210 described above. Based on the recording of the user's speech fragment, the second server 260 can generate a corresponding training audio representation associated with the training text string.

[0132] However, in other non-limiting embodiments of the present technology, the second server 260 may receive (1) a TTS training audio signal, e.g., representing pre-recorded speech of relatively long duration, e.g., 25 hours, and (2) the corresponding TTS training text representation of that pre-recorded speech represented by the TTS training audio signal. In addition, the second server 260 may (1) divide the TTS training audio signal into multiple portions, for example, of equal length such as 10 seconds, (2) generate a corresponding training audio representation for a portion of the TTS training signal, (3) determine corresponding portions in the training a TTS text representation of the prerecorded speech corresponding to a plurality of portions of the TTS training audio signal, and (4) generating a TTS training digital object containing a training text string corresponding to a portion of the TTS training text representation of the prerecorded speech and a corresponding training audio representation.

[0133] Thus, having generated (or otherwise obtained) a TTS training data set as described above, the second server 260 can train the TTS model 306 to based on the computer-generated text string 370 to generate a computer-generated audio signal indicative of a computer-generated speech fragment 270. In particular, as described above, first, the second server 260 may supply a training text string using the text-to-phonemes algorithm 502 and the encoder pre-network 504 as described above to the encoder layer stack 402. Second, the second server 260 may provide the corresponding training audio representation associated with the text string via the decoder pre-network 505, as described above, to the decoder layer stack 403. Thus, the second server 260 may input each object of the plurality of TTS training digital objects.

[0134] Then, as described above, using a backpropagation algorithm and minimizing a loss function indicating differences between input data 412 and output data 425, the second server 260 can train a TTS model 306 to generate in response to a use stage text string, such as computer-generated text string 370, output data 425 containing probability distribution vectors for portions of the computer-generated audio signal corresponding to the computer-generated text string 370. Then, based on the corresponding audio representation of the use step, the second server 260 can generate a computer-generated audio signal representing the corresponding computer-generated speech fragment 270 virtual assistant application 265 for the user's speech fragment 235.

[0135] Additional information regarding the implementation of the TTS model 306 according to some non-limiting embodiments of the present technology is provided in the article "Close to Human Quality TTS with Transformer" (Li et al., University of Electronic Science and Technology of China), the contents of which is incorporated herein by reference in its entirety. In addition, additional information about the Model 306 TTS, which can be used to implement some non-limiting embodiments of the present technology, is provided in the article “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (Yi Ren et al., Zhejiang University, Microsoft Research Asia and Microsoft Azure Speech), the contents of which are incorporated herein by reference in their entirety. Additional information about the Model 306 TTS, which can be used to implement some non-limiting embodiments of this technology, is also provided in the article "Fastpitch: parallel text-to-speech with pitch prediction" (Adrian Lancuck, NVIDIA Corporation), the contents of which is incorporated herein by reference in its entirety.

[0136] Thus, the network computing environment 200 can generate a corresponding computer-generated speech fragment 270 for transmission to the electronic device 210 in response to receiving the user speech fragment 235 (see FIG. 3).

[0137] In doing so, the developers of the present technology have determined that the quality and relevance of the corresponding computer-generated speech fragment 270 of the virtual assistant application 265 can be improved by increasing the accuracy of the STT model 302's generation of the textual representation 255 of the speech fragment. It should be noted that the second server 260 uses the output of the STT model 302 in the processing performed by the NLP model 304 and the TTS model 306. By improving the quality of the output generated by the STT model 302, it is also possible to improve the efficiency of other downstream elements of the network computing environment 200 used by the virtual assistant application 265 to generate computer-generated speech snippets for the user 230.

[0138] In this regard, the developers have determined that the STT model 302 can more accurately generate a textual speech fragment representation 255 if it is able to take into account the previous computer-generated text string 360 of the NLP model 304 as the context of the audio speech fragment representation 245 for the user's speech fragment 235. In some non-limiting embodiments of the present technology, a previous computer-generated text string 360 may be generated by an NLP model 304 for a previous corresponding computer-generated speech fragment generated by a TTS model 306, as described above, in response to which the user 230 provided a user speech fragment 235 .

[0139] Thus, developers have implemented the methods and systems described herein to allow the TTS model 302 to consider a previous computer-generated text string 360 as context for a user's speech 235 to generate a corresponding text representation 255 of the speech that is then used to generate a computer-generated speech. fragment 270. As a result, with the present methods and systems, it is possible to improve the accuracy of the STT model 302, for example, by eliminating the problem of lexical ambiguity when processing the audio speech fragment representation 245 for the user's speech fragment 235.

[0140] For example, if in user speech 235 the user 230 states his last name, such as "It's Wordsworth", then without contextual information the STT model 302 can generate a text representation 255 of the speech 255 "It's words worth" ( "It's worth a word") because she lacks the information that it is a surname, which should be represented as one word. By feeding STT model 302 as feedback data a previous computer-generated text string 360 that was used to generate a previous corresponding computer-generated speech fragment for user 230, such as “Can I have your last name, please?” (“May I know your last name?”), it is possible to provide the proper context for the user's speech fragment 235 to allow the 302 STT to recognize the famous surname “Wordsworth.” In another example, the STT model 302, which has not been provided with the proper context, may, based on a user's speech fragment 235 containing an indication of the user's address 230, such as "4245, Bellechasse" ("4245, Bellechasse"), generate a text representation 255 of the speech fragment "4245 belle chasse" ("4245, beautiful hunt"), without recognizing the famous French toponym "Bellechasse" ("Bellechasse"), which in this example corresponds to the name of the street.

[0141] Thus, through the methods and systems described herein, it is possible to improve the speech recognition accuracy of the STT model 302, which in turn allows the user 230 to generate more relevant speech responses to his speech fragments provided to the virtual assistant application 265, resulting in a better user experience 230 with the virtual assistant application 265 .

[0142] Below with reference to FIG. 6 and 7 describe the architecture and configuration of the Model 302 STT in accordance with certain non-limiting embodiments of the present technology.

Model STT

[0143] As described above, according to some non-limiting embodiments of the present technology, STT model 302 may be implemented as a machine learning model. In some non-limiting embodiments of the present technology, the machine learning model comprises an NN network-based model. Non-limiting examples of NN network-based models that can be used to implement this technology include a recurrent NN (RNN) model or an LSTM-based NN network model.

[0144] In some non-limiting embodiments of the present technology, the STT model 302 may be implemented as a transformer-based NN model. To this end, STT model 302 may comprise some or all of the components of the machine learning model architecture 400 described above with reference to FIG. 4.

[0145] However, according to some non-limiting embodiments of the present technology, STT model 302 may include additional components. In fig. 6 is a diagram of a STT model 302 implemented in accordance with certain non-limiting embodiments of the present technology.

[0146] According to some non-limiting embodiments of the present technology, STT model 302 may include a stack of STT decoder layers 603, in which the architecture of STT decoder block 605 is different from the architecture of decoder block 405 in machine learning model architecture 400. Specifically, in addition to the components of the decoder block 405 described above, the STT decoder block 605 may include an additional MHA layer 607 to provide additional input data 621. In some non-limiting embodiments of the present technology, additional input data 621 may be used to provide additional context for input data 412 and target data 417, as described in more detail below. Thus, in some non-limiting embodiments of the present technology, additional MHA layer 607 may define dependencies between corresponding portions of input data 412, target data 417, and additional input data 621.

[0147] It is contemplated that to input additional input data 621 to STT decoder block 605, server 250 may employ a second input data vectoring algorithm 610, implemented similar to input data vectoring algorithm 410 or target data vectoring algorithm 415. In addition, as described above with respect to input data 412 and target data 417, server 250 may apply to additional input data 621 a positional vectorization algorithm (not shown) capable of generating a vector pointing to positional data in portions of the provided data.

[0148] As in the above examples regarding training the NLP model 304 and the TTS model 306, the server 250 may receive input data 412, target data 417, and additional input data 621 for input to the STT model 302 from a corresponding digital object, such as training digital object 640 STT, as described below.

[0149] According to some non-limiting embodiments of the present technology, to use the STT model 302 to generate textual speech representations for input human speech audio representations, such as speech text representation 255, the server 250 may perform two corresponding processes in regarding the 302 STT. The first process of the two processes is a training process, where the server 250 is able, based on the training data set, to train the STT model 302 to generate a text representation 255 of a speech fragment, as described below. The second process is a use phase process where the server 250 is capable of applying the thus trained STT model 302 to the input audio representations according to certain non-limiting embodiments of the present technology, as described below.

Learning process

[0150] According to some non-limiting embodiments of the present technology, server 250 may train an STT model 302 based on an STT training dataset containing a plurality of training STT digital objects generated by server 250 using training speech fragments of training users.

[0151] In particular, in some non-limiting embodiments of the present technology, an STT training digital object 640 of a plurality of STT training digital objects may comprise (a) a training audio speech representation 642 generated from a training user's training speech speech similar to as described above with respect to the generation of an audio representation 245 of a speech fragment, and (b) a training text representation 644, for example, generated manually (eg, entered from a keyboard) by a training user during the generation of the STT training digital object 640 (see FIG. 6). In additional non-limiting embodiments of the present technology, the training speech text representation 644 may be pre-generated for further provision to the training user, who then generates the training speech fragment based on the training speech text representation 644.

[0152] In other non-limiting embodiments of the present technology, server 250 may generate STT training digital object 640 from pre-recorded speech, as described above with respect to training TTS model 306.

[0153] In addition, according to some non-limiting embodiments of the present technology, the STT training digital object 640 may also comprise a training computer-generated text string 646 in response to which a training audio speech representation 642 and/or were generated. or a training text representation 644 of a speech fragment. If the training audio speech fragment 642 points to the training speech fragment of the user "4245, Bellechasse" (see one of the examples of lexical ambiguity presented above), which is further reflected in the training text speech fragment representation 644, then the training computer-generated test string 646 may indicate the question “Can I have your address?” (“May I know your address?”). Thus, the training computer-generated text string 646 may be used to provide context for the user's training speech fragment, represented by the training audio speech fragment 642 and the training text speech fragment 644. In some non-limiting embodiments of the present technology, the training computer-generated text string 646 may be generated, for example, by the NLP model 304 (or other similar model) in response to receiving a previous training speech fragment of the user, as described above.

[0154] Additionally, as described above, to supply the training digital STT object 640 to the STT model 302, the server 250 may convert the training audio speech fragment representation 642, the training speech text representation 644, and the training computer-generated text string 646 into their respective vector representations. . In particular, server 250 may apply input vectorization algorithm 410 to training audio speech representation 642 to generate a plurality of input vectors 414 as described above. Likewise, the server 250 may apply the target data vectorization algorithm 415 to the training text representation 644 of the speech fragment to generate a plurality of target vectors. Finally, server 250 may apply a second input vectorization algorithm 610 to the training computer-generated text string 646 to generate a plurality of additional input vectors 627.

[0155] Additionally, according to some non-limiting embodiments of the present technology, server 250 may provide a plurality of input vectors 414 representing a training audio representation 642 of a speech fragment to a layer stack 402 of an STT model 302 encoder, such as a block 404 encoders, (b) a plurality of 419 target vectors representing the training text representation 644 of the speech fragment, into the STT decoder layer stack 603 of the STT model 302, for example, into the STT decoder block 605, and (c) a plurality 627 of additional input vectors representing the training generated the computer text string 646, also into the STT decoder block 605. As described above, in some non-limiting embodiments of the present technology, server 250 may provide a plurality of target vectors 419 and a plurality of additional input vectors 627 to various MHA layers of the STT decoder block 605, i.e. to the decoder MHA layer 407 and to the additional MHA layer 607, respectively.

[0156] Thus, server 250 may provide STT model 302 with a plurality of STT training digital objects, for example, containing thousands or hundreds of thousands of STT training digital objects similar to STT training digital object 640. Additionally, as described above, server 250 may adjust the initial node weights of each layer from the encoder layer stack 402 and from the STT decoder layer stack 603 by using a backpropagation algorithm and minimizing the loss function.

[0157] Thus, the server 250 can train the STT model 302 to generate textual speech representations, such as textual speech representation 255, for human speech fragments based on their corresponding audio speech fragment representations.

Use phase process

[0158] According to some non-limiting embodiments of the present technology, during the use phase, the server 250 may use the STT model 302 thus trained to recognize portions of human speech. In fig. 7 is a schematic diagram of the Model 302 STT shown in FIG. 6, which is executed by server 250 during the use phase, according to some non-limiting embodiments of the present technology.

[0159] In particular, the server 250 may obtain an audio speech fragment representation 245 for a user speech fragment 235 received from the user 230. In addition, the server 250 may apply an input data vectorization algorithm 410 to the audio speech fragment representation 245 to generate a representation thereof. a plurality of 414 input vectors to feed into the Model 302 STT encoder layer stack 402. The server 250 may also receive from the NLP model 304 the previous computer-generated text string 360 that was used to generate the TTS model 306's previous corresponding computer-generated speech fragment in response to which the user 230 provided a user speech fragment 235. Thus, according to some non-limiting embodiments of the present technology, server 250 may apply a second input data vectorization algorithm 610 to the previous computer-generated text string 360 to generate a plurality of additional input vectors 627. The server 250 may then provide the thus generated plurality 627 of additional input vectors representing the previous computer-generated text string 360 to an additional MHA layer 607 of the STT decoder block 605 of the STT model 302.

[0160] Thus, in some non-limiting embodiments of the present technology, in response to receiving a plurality of input vectors 414 and a plurality of additional input vectors 627, the STT model 302 may generate output data 425 comprising probability distribution vectors indicative of a textual speech representation 255 fragment for the user's speech fragment 235.

[0161] As a result, by executing the STT model 302 so trained and providing it with the previous computer-generated text string used to generate the TTS model 306's corresponding previous speech responses from the virtual assistant application 265, the server 250 can provide context for each user speech fragment to the user 230 , which makes it possible to improve the quality of the STT model 302's generation of corresponding textual representations of speech fragments, for example, by eliminating the problems with lexical ambiguity described above.

[0162] For example, for each subsequent instance of user speech 235, server 250 may provide STT model 302 with a corresponding previous computer-generated text string previously output by NLP model 304 of TTS model 306 to generate the corresponding previous speech response for virtual assistant application 265 as context. for the user's next speech fragment. In this way, the server 250 can improve the speech recognition accuracy of the STT model 302, whereby the virtual assistant application 265 is able to provide more expected and relevant speech responses to the user 230, thereby improving the user's experience interacting with the virtual assistant application 265.

Way

[0163] The architecture and examples described above enable a method of generating a textual representation of a user's speech fragment, for example, a textual speech fragment representation 255 for a user's speech fragment 235. In fig. 8 is a flow diagram of a method 800 according to non-limiting embodiments of the present technology. Method 800 may be performed by server 250.

Step 802: The server receives from the electronic device an audio signal that is an audio representation of the user's speech fragment, which is a response to the computer-generated speech fragment previously provided by the electronic device to the user.

[0164] According to some non-limiting embodiments of the present technology, method 800 begins at step 802, in which the server 250 may receive an audio representation of a speech fragment 235 of a user 230, such as an audio representation 245 of a speech fragment as described above with reference to FIG. . 3.

[0165] As described above, the user 230 may provide a user speech fragment 235 during his interaction with the virtual assistant application 265 executed by the second server 260 in response to a previous corresponding computer-generated speech fragment generated for him by the TTS model 306.

[0166] Method 800 continues at step 804.

Step 804: The server receives a computer-generated text string that is a text representation of the computer-generated speech fragment.

[0167] At step 804, server 250 may receive, via communication network 240 from NLP model 304 hosted on second server 260, the previous computer-generated text string 360 used to generate the previous corresponding computer-generated speech fragment in response to which user 230 provided speech user fragment 235, as described with reference to FIG. 7.

[0168] According to some non-limiting embodiments of the present technology, the server 250 may use a previous computer-generated text string 360 as context for the user's speech fragment 235 when generating, using the STT model 302, a text speech fragment representation 255 for the user's speech fragment 235. as described above with reference to FIG. 7.

[0169] Method 800 then proceeds to step 806.

Step 806: The server generates, using the STT model, another text string that is a text representation of the user's speech fragment based on the audio signal and a computer-generated text string taken into account as the context of the user's speech fragment.

[0170] At step 806, the server 250 may generate, using the STT model 302, a speech text representation 255 for the user's speech fragment 235, taking into account as context therefor the previous computer-generated text string 360. As described above, the previous computer-generated text string 360 is used. to generate a previous corresponding computer-generated speech fragment, in response to which the user 230 provided a user speech fragment 235.

[0171] According to some non-limiting embodiments of the present technology, STT model 302 may be implemented as a machine learning model. For example, as described above with reference to FIG. 6, the STT model 302 may be implemented as a transformer-based NN model with an encoder-decoder architecture such as the machine learning model architecture 400 described above with reference to FIG. 4. According to some non-limiting embodiments of the present technology, the STT model 302 may contain some additional components beyond those provided in the machine learning model architecture 400.

[0172] In particular, instead of a decoder layer stack 403, the STT model 302 may include an STT decoder layer stack 603, in which the architecture of the STT decoder block 605 differs from the architecture of the decoder block 405 in that an additional MHA layer 607 is provided after the encoder-decoder MHA layer 409 .

[0173] The server 250 may, based on the STT training data set, train the STT model 302 to generate a text representation 255 of a speech fragment, taking as its context a previous computer-generated text string 360, as described above with reference to FIG. 6. In particular, according to some non-limiting embodiments of the present technology, the STT training data set may comprise a plurality of STT training digital objects, wherein the STT training digital object 640 may include (a) a training audio representation 642 of a speech fragment, generated from the training user's speech fragment in a manner similar to that described above with respect to the generation of the audio speech fragment 245, (b) the training text representation 644 of the speech fragment, and (c) the training computer-generated text string 646, in response to which the speech fragment was generated a training audio presentation 642 of a speech fragment and/or a training text representation 644 of a speech fragment.

[0174] Thus, to train the STT model 302 to generate a text representation 255 of a speech fragment, the server 250 may feed it a plurality of STT training digital objects, wherein the feeding of the STT training digital object 640 includes (a) input using an input vectorization algorithm 410 data of the training audio representation 642 of the speech fragment into the encoder layer stack 402, for example, into the encoder block 404, (b) input using the vectorization algorithm 415 of the target data of the training text representation 644 of the speech fragment into the STT decoder layer stack 603 and, in particular, into the layer 409 MHA encoder-decoder block 605 STT decoder, and (c) input using a second algorithm 610 input vectorization of training computer-generated text string 646 into an additional layer 607 MHA layer 607 of STT decoder block 605.

[0175] In addition, the server 250 may use the STT model 302 trained in this manner to generate a text representation 255 of the speech fragment. In particular, as described above with reference to FIG. 7, to generate a text representation 255 of a speech fragment, the server 250 may (a) input, using an input data vectorization algorithm 410, the audio representation 245 of a speech fragment into the STT model 302 encoder layer stack 402 and (b) input, using a second input data vectorization algorithm 610, the previous one. computer-generated text string 360 into the STT decoder layer stack 603 and, in particular, into the additional MHA layer 607 of the STT decoder block 605, as described above with reference to FIG. 7.

[0176] The server 250 may then send, via communications network 240, a text representation 255 of the speech fragment to a second server 260 comprising an NLP model 304 and a TTS model 306. Thus, first, using the NLP model 304, the second server 260 can generate a computer-generated text string 370 based on the text representation 255 of the speech fragment. Second, using the TTS model 306, based on the computer-generated text string 370, the second server 260 can generate a corresponding computer-generated speech fragment 270 for sending to the electronic device 210 in response to the user speech fragment 235, as described above with reference to FIG. 4 and 5.

[0177] According to some non-limiting embodiments of the present technology, the NLP model 304 and the TTS model 306 may be implemented as a transformer-based NN model and trained as described above with reference to FIG. 4 and 5.

[0178] Thus, the server 250, using the STT model 302, can generate textual representations of speech fragments for the following speech fragments of the user 230 for the virtual assistant application 265. For example, for the purpose of generating an appropriate textual representation of a speech segment for the user's next speech fragment provided by user 230 in response to computer-generated speech fragment 270, server 250 may use computer-generated text string 370 from the NLP model 304 as the context for the user's next speech fragment. as described above with respect to user speech 235.

[0179] Thus, some embodiments of method 800 improve the speech recognition accuracy of STT model 302 in user speech fragments, which in turn improves the performance of NLP model 304 and TTS model 306 in providing computer-generated speech fragments in response to speech user fragments. As a result, the user's experience of interacting with virtual assistant applications can be improved.

[0180] This completes the method 800.

[0181] Obviously, not all technical effects mentioned in this description need to be present in every embodiment of the present technology.

[0182] Possible changes and improvements to the above-described embodiments of the present technology may be apparent to one skilled in the art. The foregoing description is provided for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the attached claims.

Claims (47)

1. A method of generating textual representations of a user's speech fragment received by a user-associated electronic device and provided in response to computer-generated speech fragments output by an electronic device capable of communicating with a server for generating computer-generated speech fragments, wherein the method is performed by the server and includes into yourself: - the server receives from an electronic device an audio signal, which is an audio representation of a user’s speech fragment, which is a response to a computer-generated speech fragment previously issued by the electronic device to the user; - receipt by the server of a computer-generated text string, which is a text representation of a computer-generated speech fragment; And - generation by the server, using the speech-to-text (STT) model, of another text string, which is a text representation of the user’s speech fragment, based on the audio signal and a computer-generated text string, taken into account as the context of the user’s speech fragment, wherein the speech-to-text (STT) model is a neural network (NN). 2. The method according to claim 1, characterized in that it additionally includes: - generation by the server, using the STT model, of a first text line, which is a text representation of the user’s first speech fragment, based on the first audio signal, which is an audio representation of the user’s first speech fragment; - generation by the server using a different model of a computer-generated text string based on the first text string; And - generation by the server, using a text-to-speech (TTS) model, of a computer-generated audio signal, which is an audio representation of a computer-generated speech fragment, based on a computer-generated text string. 3. The method according to claim 1, where the neural network (NN) has an encoder-decoder architecture containing a stack of encoder layers and a stack of decoder layers, and generating another text string includes: - formation by the server of an audio vector representing an audio signal using an audio vectorization algorithm; - input by the server of the sound vector into the NN network encoder stack; - generation by the server, using a text vectorization algorithm, of a text vector representing a computer-generated text string; And - input by the server of a text vector into the stack of the NN network decoder. 4. The method according to claim 3, characterized in that the NN network is a transformer-based NN network. 5. The method according to claim 1, characterized in that it further includes training an STT model at the training stage based on a training data set containing a plurality of training objects, wherein the training object contains (a) an indication of a training audio signal generated based on of a training speech fragment of the user spoken by the training user, (b) a first training text string being a textual representation of the user's training speech fragment, and (c) a second training text string used to provide context for the first text string and being a textual representation of the corresponding computer-generated speech fragment, in response to which the training user uttered the user's training speech fragment. 6. The method according to claim 2, characterized in that it further includes the server receiving a first audio signal from the electronic device. 7. The method of claim 2, further comprising the server sending a computer-generated audio signal to the electronic device. 8. The method according to claim 2, characterized in that the other model contains a natural speech processing (NLP) model. 9. Method according to claim 1 or 2, characterized in that one or more models are hosted on another server. 10. The method according to claim 8, characterized in that the NLP model is a transformer-based NN model. 11. The method according to claim 2, characterized in that the TTS model is a transformer-based NN model. 12. The method according to claim 2, characterized in that it additionally includes: - generation by the server, using the STT model, of a second text string based on a second audio signal, which is an audio representation of the second speech fragment of the user, corresponding to the request following the computer-generated speech fragment; - generation by the server, based on the second text string using a different model, of another computer-generated text string, which is a text representation of another computer-generated speech fragment to be provided to the user; - generating by the server, using the TTS model, based on another computer-generated text string, another computer-generated audio signal, which is an audio representation of another computer-generated speech fragment to be provided to the user in response to the second speech fragment of the user; - receipt by the server from the electronic device of a third audio signal, which is an audio representation of the third speech fragment of the user, corresponding to another request following another computer-generated speech fragment; And - generating by the server, using the STT model, based on the third audio signal, a computer-generated text string and another computer-generated text string, a third text string that is a textual representation of the user's third speech fragment, taking into account the computer-generated text string and another computer-generated text string as the context of the other next request. 13. The method of claim 12, further comprising the server sending another computer-generated audio signal to the electronic device. 14. A server for generating text representations of a user's speech fragment received by an electronic device associated with the user and capable of communicating with the server, and provided in response to computer-generated speech fragments output by the electronic device, wherein the server includes a processor and a machine-readable physical storage medium storing commands, and the processor is configured to perform the following actions when executing commands: - receiving from an electronic device an audio signal, which is an audio representation of a user’s speech fragment, which is a response to a computer-generated speech fragment previously issued by the electronic device to the user; - obtaining a computer-generated text string, which is a text representation of a computer-generated speech fragment; And - generation, using the STT model, of another text string, which is a text representation of the user’s speech fragment, based on the audio signal and a computer-generated text string, taken into account as the context of the user’s speech fragment, wherein the speech-to-text (STT) model is a neural network (NN). 15. Server according to claim 14, characterized in that the processor is additionally configured to: - generating, using the STT model, a first text line, which is a text representation of the user's first speech fragment, based on the first audio signal, which is an audio representation of the user's first speech fragment; - generating, using another model, a computer-generated text string based on the first text string; - generating, using the TTS model, a computer-generated audio signal, which is an audio representation of a computer-generated speech fragment, based on a computer-generated text string, 16. The server according to claim 14, characterized in that the NN neural network has an encoder-decoder architecture containing a stack of encoder layers and a stack of decoder layers, and the processor is configured to generate another text string by: - generating a sound vector representing an audio signal using a sound vectorization algorithm; - inputting the sound vector into the NN network encoder stack; - generating, using a text vectorization algorithm, a text vector representing a computer-generated text string; And - entering a text vector into the NN network decoder stack. 17. The server according to claim 16, characterized in that the NN network is a transformer-based NN network. 18. The server according to claim 14, characterized in that the processor is additionally configured to train the STT model at the training stage based on a training data set containing a plurality of training objects, wherein the training object contains (a) an indication of a training audio signal generated based on of a training speech fragment of the user spoken by the training user, (b) a first training text string being a textual representation of the user's training speech fragment, and (c) a second training text string used to provide context for the first text string and being a textual representation of the corresponding computer-generated speech fragment, in response to which the training user uttered the user's training speech fragment. 19. The server according to claim 15, characterized in that the other model contains an NLP model. 20. The server according to claim 14 or 15, characterized in that one or more models are hosted on another server connected to the server.