RU2798362C2 - Method and server for teaching a neural network to form a text output sequence - Google Patents

Abstract

FIELD: neural networks.

SUBSTANCE: invention relates to a method and a server for training an attention neural network (ANN) to generate a text output sequence. The technical result consists in the possibility of generating a relevant text output sequence, which is a summary of the content of a plurality of content resources. In the method, the server inputs a training request and a training text input sequence into the encoder subnet, while the training text input sequence (a) is formed as a sequence of training content snippets and (b) is divided into a sequence of input groups, and the input group is associated with a training content snippet and contains words from this learning content snippet; the encoded representation of the training text input sequence is generated by the server using the encoder subnet, including the generation of the attention output for the corresponding words from the training text input sequence by applying an attention restriction mask to the training text input sequence, and the attention output is used to form an encoded representation of the training text input sequence, when generating output data of the "attention" type for a word from the input group, the attention restriction mask allows you to "take into account" only words from this input group so that the output data of the "attention" type is formed based on the context this input group, and not the contexts of other input groups in the training text input sequence; generating, by the server using the decoder subnet, a decoded representation for the training text input sequence corresponding to the predicted text output sequence; generating, by the server, a penalty estimate for the training iteration by comparing the predicted text output sequence with a predetermined text output sequence representing a predetermined response to the training request; and adjusting the ANN network server based on the penalty estimate.

EFFECT: possibility of generating a relevant text output sequence, which is a summary of the content of a plurality of content resources.

32 cl, 7 dwg

Description

The field of technology to which the invention belongs

[01] The present technology relates to intelligent personal assistant (IPA, Intelligent Personal Assistant) systems and, in particular, to a method and electronic device for teaching a neural network to form a text output sequence.

State of the art

[02] Electronic devices such as smartphones and tablets provide access to an ever-growing variety of applications and services for processing and/or accessing information of various kinds. However, novice users and/or users with disabilities and/or users driving a vehicle may not be able to interact effectively with such devices, mainly due to the variety of functions provided by such devices or the inability to use user interfaces. - a machine provided by such devices (such as a keyboard or touch screen). For example, a user driving a vehicle or a visually impaired user may not be able to use the touch screen keyboard implemented in some of these devices.

[03] IPA systems have been developed to perform user-requested functions. For example, IPA systems can be used for information retrieval and for navigation. A traditional IPA system, such as the IPA ^Siri® system, for example, can receive simple human input in natural language from a device and perform a variety of tasks for the user. For example, the user may communicate with the IPA ^Siri® system by providing speech cues (via the ^Siri® voice interface), such as the current weather, the location of the nearest shopping mall, and the like. The user may also request the execution of various applications installed on the electronic device.

[04] Conventional IPA systems are capable of allowing users to enter speech search queries. For example, traditional IPA systems can generate a textual representation of a spoken query and trigger a search for relevant information.

Disclosure of invention

[05] IPA systems are used to support voice interaction between users and electronic devices. This provides a convenient user interaction with such devices, for example, when the user has a visual impairment, and also when the device does not provide a traditional visual display (for example, in a smart speaker) and / or when the user's hands or eyes are busy (for example, when driving).

[06] The developers of the present technology have discovered certain technical shortcomings associated with existing IPA systems. At least one disadvantage of conventional IPA systems is their limited ability to handle speech requests and, in particular, limitations in providing audio responses. This is mainly because users who expect an audio response generally prefer short responses, and designing an IPA system that can succinctly provide information in response to the many queries that users could potentially enter is a technical challenge. For example, some conventional systems are not suitable for generating a summary of relevant content that is concise enough to be presented to the user in audio format. In another example, other conventional systems are not suitable for using the context to which the content relates to form a better summary of the content.

[07] The aim of the present invention is to eliminate at least some of the shortcomings of the known solutions. The developers of the present technology have developed a system in which some content determined to be relevant to a query is processed by a neural network (NN, Neural Network). In at least some non-limiting embodiments of the present technology, the NN may be an Attention-type NN (ANN) containing an encoder subnet and a decoder subnet capable of processing data sequences, while in the encoder subnet implemented a specific attention mechanism.

[08] In at least some non-restrictive embodiments of the present technology, the encoder's subnet attention mechanism can be said to be capable of applying an attention restriction mask to an input data sequence. It is contemplated that the input data sequence may comprise input groups corresponding to content snippets from query-relevant content received, for example, from a search engine server capable of performing web searches in response to user search queries. As a result of applying the attention constraint mask to an element of an input sequence in which such input groups are labeled, the encoder subnet can "take into account" the context of that element's input group, rather than the context of other input groups. Thus, we can say that the context obtained for an element using the attention mechanism is based on the context of the input group of this element, and not on other contexts of other input groups.

[09] According to a first aspect of the present technology, a method for teaching an ANN to generate a text output sequence based on a text input sequence and a query is implemented. The text input sequence is formed as a sequence of content snippets obtained from the corresponding content resources relevant to the query. The text output sequence is a summary of the content of the sequence of content snippets to be used as a response to a query. The ANN contains an encoder subnet and a decoder subnet. The method is performed by the server on which the ANN network is implemented. The method at the training iteration of the ANN network includes the input by the server of the training query and the training text input sequence into the encoder subnet. The training text input sequence (a) is formed as a sequence of training content snippets and (b) is divided into a sequence of input groups. The input group is associated with the training content snippet and contains the words from the training content snippet. The method at the training iteration of the ANN network includes the formation by the server using the encoder subnet of an encoded representation of the training text input sequence, including the generation of output data of the form "attention" for the corresponding words from the training text input sequence by applying an attention restriction mask in relation to the training text input sequence . The output data of the "attention" type is used to form an encoded representation of the training text input sequence. When generating attention-type output for a word from an input group, the attention constraint mask allows you to "take into account" only words from this input group so that the attention-type output is generated based on the context of this input group, and not the contexts of other input groups. groups in the training text input sequence. The method at the training iteration of the ANN network includes generating, by the server using the decoder subnet, a decoded representation of the training text input sequence. Such a decoded representation corresponds to the predicted text output sequence. The method at the training iteration of the ANN network includes generating a penalty estimate for the training iteration by the server by comparing a predicted text output sequence with a predetermined text output sequence. The predefined text output sequence is a predefined response to the training query. The method in the ANN training iteration includes updating the ANN network by the server based on the penalty estimate.

[10] In some embodiments of the method, the generated attention output is further based on the context from the training query.

[11] In some embodiments of the method, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The method at another training iteration further includes training the ANN based on the other training dataset. The other training dataset contains (a) a training query, (b) a modified training text input sequence, and (c) a predefined text output sequence. The input groups in the modified training text input sequence are in an order different from the order of the input groups in the training text input sequence.

[12] In some embodiments, the method further includes generating a training content snippet from a sequence of training content snippets. Such formation includes input by the server of the training request and content from the training content resource into a machine learning algorithm (MLA, Machine Learning Algorithm) capable of producing a predictable content snippet. The MLA algorithm is trained to generate a predictable content snippet such that the predicted content snippet is similar to a predefined text output sequence. A predictable content snippet is a training content snippet.

[13] In some embodiments of the method, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The training iteration is performed at the first stage of training the ANN, which includes the first set of training iterations. The method further includes performing a second step of training the ANN. The second stage of ANN training contains a second set of training iterations. In another training iteration of the second set of training iterations, the ANN is trained on a different training dataset. The other training dataset is a set of a plurality of predefined training datasets to be used in performing the second plurality of training iterations. Another training dataset is selected by the human estimator.

[14] In some embodiments of the method, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The predetermined textual output sequence is determined by the human evaluator. The human evaluator determines a predetermined textual output sequence from the content of at least one of the learning content resources associated with the learning content snippets.

[15] In some embodiments, the method further includes (a) generating, by the server, a text representation of the query based on an audio representation of the user's speech indicative of a query by the user associated with the electronic device, (b) determining, by the server, search results corresponding to the text representation of the query. , wherein the search result contains a content snippet and points to a content resource relevant to the query, (c) the server generates a text input sequence in the form of a sequence of content snippets based on the content snippets of the search results, (d) the server enters a text input sequence and a textual representation of the query to an ANN capable of providing a text output sequence representing a summary of the content of a sequence of content snippets to be used as a response to the request, (e) generating by the server an audio representation of the text output sequence representing a computer-generated speech fragment of the response, and (f) providing by the server to the electronic device associated with the user, the audio representation of the text output sequence.

[16] In some embodiments of the method, updating the ANN includes applying backpropagation to the ANN by the server.

[17] In some embodiments of the method, it further includes obtaining content snippets from corresponding content resources relevant to the query.

[18] In some embodiments of the method, such obtaining includes performing a search using a search engine and a query as input.

[19] In some embodiments of the method, such obtaining further includes obtaining the N most relevant snippets associated with the N best search results.

[20] In some embodiments of the method, it further includes obtaining a predetermined text output sequence.

[21] In some embodiments of the method, such acquisition is from a human evaluator.

[22] In some embodiments of the method, it further includes obtaining an indication of the content resource used by the human evaluator to generate a predetermined text output sequence.

[23] In some embodiments of the method, the content resource is used to validate a predetermined textual output sequence.

[24] In some embodiments of the method, the content resource is a network resource corresponding to the training query and obtained by performing a search using a search engine.

[25] According to a second aspect of the present technology, a server is implemented for training an ANN to generate a text output sequence based on a text input sequence and a query. The text input sequence is formed as a sequence of content snippets obtained from the corresponding content resources relevant to the query. The text output sequence is a summary of the content of the sequence of content snippets to be used as a response to a query. The ANN is implemented on the server and contains an encoder subnet and a decoder subnet. The server in the ANN training iteration is able to inject the training query and the training text input sequence into the encoder subnet. The training text input sequence (a) is formed as a sequence of training content snippets and (b) is divided into a sequence of input groups. The input group is associated with the training content snippet and contains the words from the training content snippet. The server at the training iteration of the ANN network is able to generate an encoded representation of the training text input sequence using the encoder subnet. The generation capability includes the capability for the server to generate an attention-like output for corresponding words from the training text input sequence by applying an attention restriction mask to the training text input sequence. The output data of the "attention" type is used to form an encoded representation of the training text input sequence. When generating attention-type output for a word from an input group, the attention constraint mask allows you to "take into account" only words from this input group so that the attention-type output is generated based on the context of this input group, and not the contexts of other input groups. groups in the training text input sequence. The server at the training iteration of the ANN network is able to generate a decoded representation of the training text input sequence using the decoder subnet. The decoded representation corresponds to the predicted text output sequence. The server at the training iteration of the ANN is able to generate a penalty estimate for the training iteration by comparing the predicted text output sequence with a predetermined text output sequence. The predefined text output sequence is a predefined response to the training query. The server at the ANN training iteration is able to adjust the ANN based on the penalty estimate.

[26] In some server embodiments, the generated attention output is further based on the context from the training request.

[27] In some server embodiments, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The server at another training iteration is additionally able to train the ANN based on a different training dataset. The other training dataset contains (a) a training query, (b) a modified training text input sequence, and (c) a predefined text output sequence. The input groups in the modified training text input sequence are in an order different from the order of the input groups in the training text input sequence.

[28] In some server embodiments, it is further capable of generating a training content snippet from a sequence of training content snippets. Such generation includes input by the server of the training request and content from the training content resource into an MLA algorithm capable of producing a predictable content snippet. The MLA algorithm is trained to generate a predictable content snippet such that the predicted content snippet is similar to a predefined text output sequence. A predictable content snippet is a training content snippet.

[29] In some server embodiments, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The training iteration is performed at the first stage of training the ANN, which includes the first set of training iterations. The server is further capable of performing the second stage of training the ANN. The second stage of training the ANN network includes a second set of training iterations. In another training iteration of the second set of training iterations, the ANN is trained on a different training dataset. The other training dataset is a set of a plurality of predefined training datasets to be used in performing the second plurality of training iterations. Another training dataset is selected by the human estimator.

[30] In some server embodiments, in a training iteration, the ANN is trained based on a training dataset containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence. The predetermined textual output sequence is determined by the human evaluator. The human evaluator determines a predetermined textual output sequence from the content of at least one of the learning content resources associated with the learning content snippets.

[31] In some server embodiments, it is further capable of (a) generating a textual representation of the query based on the audio representation of the user's speech fragment indicating the user's query associated with the electronic device, (b) determining search results corresponding to the textual representation of the query, wherein the search result contains a content snippet and points to a content resource relevant to the query, (c) generate a text input sequence as a sequence of content snippets based on the content snippets of the search results, (d) input a text input sequence and a text representation of the query into an ANN capable of produce a text output sequence representing a summary of the content of a sequence of content snippets to be used as a response to a request, (e) generate an audio representation of the text output sequence representing a computer-generated speech fragment of the response, and (f) provide to an electronic device associated with the user , the audio representation of the text output sequence.

[32] In some server embodiments, updating the ANN includes applying backpropagation to the ANN by the server.

[33] In some embodiments, the server is further capable of obtaining content snippets from corresponding content resources relevant to the request.

[34] In some embodiments of the server, it is capable of performing a search using a search engine and a query as input to do so.

[35] In some embodiments of the server, for the purpose of such obtaining, it is additionally able to receive the N most relevant snippets associated with the N best search results.

[36] In some server embodiments, it is further capable of receiving a predefined text output sequence.

[37] In some embodiments, the server is capable of receiving a predefined textual output sequence from a human evaluator.

[38] In some server embodiments, it is further capable of obtaining an indication of the content resource used by the human evaluator to generate a predefined text output sequence.

[39] In some server embodiments, the content resource is used to validate a predefined text output sequence.

[40] In some embodiments of the server, the content resource is a network resource corresponding to a training query and obtained by performing a search using a search engine.

[41] In the context of the present description, the term "server" means a computer program executed by appropriate hardware and capable of receiving requests (for example, from devices) via a network and fulfill these requests or initiate their execution. The relevant hardware may be one physical computer or one computer system, which is not essential to the present technology. In the present context, the expression "server" does not mean that every task (e.g. received command or request) or some particular task is received, executed or started by the same server (i.e. the same software and/or hardware ). This expression means that any number of software or hardware can receive, send, execute or initiate the execution of any task or request or the results of any tasks or requests. All of these software and hardware may be a single server or multiple servers, both of which are meant by the expression "at least one server".

[42] In the context of the present description, the term "device" means any computer hardware capable of executing programs suitable for solving the task. Thus, some (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, and the like), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that in this context, a device functioning as a device may also function as a server for other devices. The use of the term "device" does not preclude the use of multiple devices to receive, send, perform, or initiate any task or request, or sequences of any tasks, requests, or steps in any manner described herein.

[43] In the context of the present description, the term "database" means any structured set of data, regardless of its specific structure, database management software or computer hardware for storing this data, using them or providing them with use in another way. The database may reside on the same hardware as the process for storing or using the information stored in the database, or the database may reside on separate hardware such as a specialized server or multiple servers.

[44] In the context of the present description, the expression "information" includes information of any kind or type that can be stored in a database. Thus, information includes audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.). .d.), but not limited to documents, spreadsheets, word lists, etc.

[45] In the context of the present description, the expression "component" includes the designation of software (suitable for certain hardware), necessary and sufficient to perform a particular function or several functions.

[46] In the context of the present description, the term "computer usable storage medium" means media of any kind and form, including random access memory (RAM), read only memory (ROM), disks (CD-ROM, DVD, floppy disks, hard drives, etc.), USB sticks, solid state drives, tape drives, etc.

[47] In the context of the present description, the numerals "first", "second", "third", etc. are used only to indicate the difference between the nouns they refer to, but not to describe any specific relationship between these nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy, or classification of, in this case, servers, and that their use (by itself) does not imply a "second server" in every situation. In addition, as occurs herein in another context, reference to a "first" element and a "second" element does not exclude that the two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, and in other cases they may be different software and/or hardware.

[48] Each embodiment of the present technology relates to at least one of the above objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology, associated with an attempt to achieve the above goal, may not meet this goal and/or may meet other goals not explicitly mentioned here.

[49] Additional and/or alternative features, aspects, and advantages of embodiments of the present technology are contained in the following description, in the accompanying drawings, and in the claims.

Brief description of the drawings

[50] The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the attached drawings.

[51] FIG. 1 is a diagram of a system suitable for implementing non-limiting embodiments of the present technology.

[52] FIG. 2 shows a search engine result page (SERP) generated by the server of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[53] FIG. 3 shows the structure of the data stored in the database of the system shown in FIG. 1 according to non-limiting embodiments of the present technology.

[54] FIG. 4 is a diagram of an iteration of learning the MLA algorithm performed by the system shown in FIG. 1 and capable of generating predictive content snippets for respective content resources, in accordance with non-limiting embodiments of the present technology.

[55] FIG. 5 is a diagram of an iteration of ANN training performed by the system shown in FIG. 1 and capable of generating predictive text output sequences that are summaries of the content of the respective content resources, according to non-limiting embodiments of the present technology.

[56] FIG. 6 is a block diagram of the processing of a user's speech fragment in the system shown in FIG. 1 to provide a computer-generated speech to a user, in accordance with non-limiting embodiments of the present technology.

[57] FIG. 7 is a flowchart of the ANN training method of FIG. 5 in accordance with other non-limiting embodiments of the present technology.

Implementation of the invention

[58] FIG. 1 is a diagram of a system 100 suitable for non-limiting embodiments of the present technology. It should be understood that the system 100 is provided only to demonstrate an implementation of the present technology. Thus, the following description of the system is a description of examples illustrating the present technology. This description is not intended to define the scope or boundaries of the present technology. In some cases, useful examples of modifications to system 100 are provided. These provide insight but also do not define the scope or boundaries of the present technology.

[59] These modifications do not constitute an exhaustive list. As one skilled in the art will appreciate, other modifications are likely to be possible. In addition, if modifications are not described in some cases (i.e., there are no examples of modifications), this does not mean that they are impossible and / or that this description contains the only possible implementation of one or another element of the present technology. One skilled in the art will appreciate that this may not be the case. In addition, it should be understood that system 100 may, in some instances, be a simplified implementation of the present technology, and that such variations are presented to facilitate a better understanding thereof. Those skilled in the art will appreciate that other embodiments of the present technology may be significantly more complex.

[60] In general, system 100 is capable of providing responses to user requests that can be said to result in a "dialogue" between the user and the electronic device. For example, an audio signal 152 (such as a speech fragment) from a user 102 can be detected by an electronic device 104 (or simply device 104), which is capable of providing audio signals 154 (such as speech fragments or computer-generated speech fragments) in response. Thus, it can be said that as a result of this, a dialogue 150 is provided between the user 102 and the device 104, consisting of (a) a sound signal 152 and (b) a sound signal 154.

[61] The following describes the various elements of system 100 and how these elements ensure that an audio signal 154 is provided by the electronic device 104 in response to an audio signal 152 of the user 102.

User device

[62] As mentioned above, system 100 includes device 104. There are no particular restrictions on the implementation of device 104. For example, device 104 may be implemented as a personal computer (desktop, laptop, netbook, etc.), a wireless communications device (smartphone, cell phone, tablet, smart speaker, etc.), or network equipment (router, switch). , gateway). Device 104 may sometimes be referred to as an electronic device, a terminal device, a client electronic device, or simply a device. It should be noted that the association of the electronic device 104 with the user 102 is not meant to suggest or imply any mode of operation, such as logging in, registering, or the like.

[63] It is contemplated that device 104 includes hardware and/or application software and/or firmware (or a combination thereof) for (a) detecting or capturing audio signal 152 and (b) providing or playing audio signal 154. For example, device 104 may include one or more microphones for detecting or capturing audio signals 152 and one or more speakers for providing or reproducing audio signals 154.

[64] Device 104 also includes hardware and/or application software and/or firmware (or a combination thereof) for executing IPA application 105. In general, the IPA application 105, also known as a chatbot, is designed to allow the user 102 to send requests in the form of speech fragments (for example, in the form of an audio signal 152) and in response to provide the user 102 with answers in the form of speech fragments ( for example, in the form of a sound signal 154).

[65] Sending requests and providing responses can be performed by the IPA application 105 using a natural language user interface. In general, the natural language user interface of the IPA application 105 may be any kind of human-machine interface in which linguistic units such as verbs, phrases, sentences, and the like act as user interface controls for retrieving, selecting, modifying or otherwise generating data in the 105 IPA application.

[66] For example, when device 104 detects (i.e., captures) user 102's speech patterns (e.g., audio signal 152) by device 104, IPA application 105 can use its natural language user interface to parse user 102's speech patterns and extract data from them, indicating user requests. In addition, data indicative of the responses received by the device 104 are parsed by the natural language user interface of the IPA application 105 to provide or play utterances (eg, beep 154) indicative of those responses.

[67] It is contemplated that the IPA application 105 may alternatively or additionally use a text-based human-machine interface to allow the user 102 to send instructions for requests in text form, and in response to provide the user 102 with instructions for responses in text form, without leaving the the limits of the present technology.

Communication network

[68] In an illustrative non-limiting example of system 100, device 104 is associated with communication network 110 to access and transmit data packets to or from server 106 and/or other web resources (not shown). In some non-limiting embodiments of the present technology, the communication network 110 may be the Internet. In other non-limiting embodiments of this technology, communications network 110 may be implemented in other ways, such as any wide area network, local area network, private network, or the like. The implementation of a communication link (not labeled) between device 104 and communication network 110 depends, among other things, on the implementation of device 104.

[69] By way of a non-limiting example only, in those embodiments of the present technology in which the device 104 is implemented as a wireless communication device (such as a smartphone), the communication link may be implemented as a wireless communication link (such as a 3G network link). , 4G network channel, Wireless Fidelity or WiFi ^® , Bluetooth ^® for short, etc.). In instances where device 104 is implemented as a laptop, the link may be wireless (such as Wireless Fidelity or ^WiFi® , ^Bluetooth® for short, etc.) or wired (such as an Ethernet-based connection).

Many resource servers

[70] As described above, a plurality of resource servers 108 can be accessed via communication network 110. A plurality of resource servers 108 may be implemented as conventional computer servers. In a non-limiting embodiment of the present technology, a server of multiple resource servers 108 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. The server of the plurality of resource servers 108 may also be implemented using any other suitable hardware and/or application software and/or firmware, or a combination thereof.

[71] The plurality of resource servers 108 is capable of containing resources (or web resources) that can be accessed by the device 104 and/or server 106. There are no restrictions on the kind of resources contained in the plurality of resource servers 108. However, in some embodiments of the present technology, these resources may contain digital documents or simply documents that are web pages.

[72] For example, a plurality of resource servers 108 may contain web pages, ie. a plurality of resource servers 108 may store documents that are web pages and are available to device 104 and/or server 106. A document may be written in a markup language and may, among other things, contain (a) the content of the corresponding web page, and (b) machine-readable commands to display the corresponding web page (the content it contains). Thus, in at least some embodiments of the present technology, a plurality of resource servers 108 may be said to contain a plurality of content resources, each of which is associated with a respective content.

[73] The device 104 may contact a server of a plurality of resource servers 108 to obtain a document (eg, content resource) stored on that server. For example, user 102 may enter a web address associated with a web page into a browser application (not shown) of device 104. In response, device 104 may contact a resource server containing the web page to obtain a document representing this web page to display the content of that web page using a browser application.

[74] It should be noted that the server 106 may also access a server of the plurality of resource servers 108 in order to obtain a document (eg, content resource) stored on that resource server. The purpose of a server 106 that accesses and retrieves documents from a plurality of resource servers 108 is described in more detail below.

Server

[75] As mentioned above, system 100 also includes a server 106, which may be implemented as a conventional computer server. In an exemplary embodiment of the present technology, server 106 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. It should be understood that the server 106 may be implemented using any other suitable hardware, application software and/or firmware, or combinations thereof. In the present non-limiting embodiments of the present technology, the server 106 is a single server. In other non-limiting embodiments of the present technology, the functions of the server 106 may be distributed among multiple servers.

[76] In general, server 106 is capable of (a) receiving data associated with audio signal 152 from device 104, (b) parsing that data, and in response (c) determining data to generate audio signal 154. For example, server 106 can receive data indicative of a user request and in response may generate data indicative of a response to that user request. To this end, the server 106 is capable of performing, among other things, one or more computer procedures, referred to in this document as the search engine 130.

[77] In some embodiments, the server 106 may be managed and/or administered by a search engine service provider (not shown), such as the Yandex™ search engine operator. The server 106 may include a search engine 130 to perform one or more searches in response to queries submitted by users of the search engine 130.

[78] In some embodiments, the server 106 may receive data from the device 104 indicative of a request from the user 102. For example, the IPA application 105 of the device 104 may generate a digital audio representation of the audio signal 152 (e.g., the user's speech fragment) and send it a digital audio representation via communications network 110 to server 106. In this example, server 106 may generate a textual representation of the request sent by user 102 based on the digital audio representation of the user's speech and perform searches based on the textual representation of the request thus generated. In at least some embodiments of the present technology, the user 102 may use the electronic device 104 and manually enter a textual representation of the request, and the response provided to the user 102 may be provided as a digital audio representation without departing from the present technology.

[79] It should be noted that text-to-speech for the digital audio representation of the user's speech can be performed locally on device 104 (eg, by IPA application 105). Therefore, the device 104 may additionally or alternatively send a textual representation of the request to the server 106 in addition to or instead of the digital audio representation.

[80] As described above, the server 106 may use the textual representation of the query to perform a search (corresponding to the query) and thus generate search results relevant to the query. For example, server 106 using search engine 130 may generate data indicative of one or more search results relevant to user query 102.

[81] To better illustrate this, the following description is given with reference to FIG. 2, which contains the view 200 of the SERP page. When a user 102 submits their query to a search engine 130, such as through a browser application, the search engine 130 is typically used to generate a so-called SERP page containing one or more search results relevant to the query. As shown in FIG. 2, the search engine 130 is able to use the textual representation 202 of the query to generate a ranked list of search results, which in this case contains the first search result 210, the second search result 220, and the third search result 230.

[82] It should be noted that in FIG. 2 in a non-limiting example, (a) the first search result 210 is associated with the first content resource 212 and with the first content snippet 214, (b) the second search result 220 is associated with the second content resource 222 and contains the second content snippet 224, (c ) the third search result 230 is associated with the third content resource 232 and with the third content snippet 234 .

[83] The search engine 130 is able to determine content resources that are relevant to the user's query and rank those content resources based on, among other things, the relevance of their content to the query, as is known in the art.

[84] In at least some non-limiting embodiments of the present technology, the developers of the present technology have developed methods and systems for generating user-oriented content snippets for respective content resources. The developers of the present technology have found that providing the user with more concise content snippets (ie, concise and meaningful) can increase user satisfaction with search engine services. Such user-friendly content snippets can reduce the amount of time required to parse search results provided by a search engine.

[85] To this end, in at least some non-limiting embodiments of the present technology, the server 106 is capable of executing one or more computer procedures, which are referred to below as the snippet engine 135. In general, the snippet engine 135 is capable of generating a corresponding content snippet for a content resource that is a concise summary of that resource's content.

[86] The developers of the present technology have also found that the snippet engine 135, capable of generating concise snippets of content for various content resources, can be used in the context of processing speech requests. Obviously, in some embodiments of the present technology, where the device 104 is, for example, a smart speaker, it is desirable to provide the user 102 with voice search engine services. In these embodiments, the generation of concise content snippets can be used to create computer-generated speech fragments that more concisely and meaningfully convey content relevant to a query. The snippet engine 135 generates short content snippets for the respective content resources in more detail below.

[87] It should also be noted that the developers of the present technology have found that in the context of speech queries, additional processing of such concise content snippets may be required to further increase user satisfaction from the brevity and richness of computer-generated speech fragments. It is clear that the ranked list of content snippets provided in audio format is less convenient than the SERP page visually displayed to the user.

[88] Based on this, the developers of the present technology have developed methods and systems that allow the server 106, in a sense, summarize the content of a plurality of content resources relevant to a user request, and provide this summary of content to the user 102 in audio format (i.e., provide computer-generated speech fragment). In at least some non-limiting embodiments of the present technology, the developers of the present technology have developed an ANN network 140 capable of generating a text output sequence representing such a content summary.

[89] It is also contemplated that at least some of the methods and systems described herein are capable of using concise content snippets (generated by snippet engine 135) to train the ANN network 140 to generate a concise and meaningful summary of content for a plurality of content resources relevant to a query. The ANN 140 generates this text output sequence, which is a summary of the content of a plurality of content resources, in more detail below.

[90] It should be noted that the server 106 is capable of providing the device 104 with data indicative of a summary of the content in text format to be processed by the IPA application 105 to generate an appropriate digital audio representation (eg, beep 154) and played back to the user 102. However however, in other embodiments of the invention, the server 106 may generate a digital audio representation of the content summary based on the corresponding text output sequence and thus provide data indicative of the content summary in digital audio format.

Database system

[91] The server 106 is connected to the database system 120. In general, database system 120 may store information received and/or generated by server 106 during processing. For example, database system 120 is capable of receiving data received and/or generated by server 106 during processing from server 106 for temporary and/or permanent storage and providing the stored data to server 106 for later use. To better illustrate this, the following description refers to FIG. 3, which is a representation 300 of the data stored in the database system 120, in accordance with at least some non-limiting embodiments of the present technology.

[92] The database system 120 is capable of storing search engine data 310. In general, search engine data 310 stored in database system 120 contains data that allows server 106 to provide search engine services to users of search engine 130. In some embodiments, search engine data 310 may contain data about a large number of content resources (for example, , documents) received by the crawler application (not shown). Thus, the database system 120 can provide the server 106 with access to this large number of documents during document retrieval.

[93] It should be noted that the server 106 may run a crawler application as part of the search engine 130. In general, the crawler application may be used by the server 106 to "visit" resources available through the communication network 110 and to obtain or download documents for the purpose of further use. For example, a crawler application may be used by server 106 to access multiple resource servers 108 and to retrieve or download documents that are web pages contained on multiple resource servers 108 . It is contemplated that the crawler application may be periodically executed by the server 106 to retrieve or download documents that have been updated and/or made available via the communication network 110 since the previous execution of the crawler application.

[94] It should be noted that data on a large number of content resources may be stored in the database system 120 in an indexed format, i. database system 120 may store an indexing structure known in the art, such as an inverted index of content resources. For example, the database system 120 may store a plurality of lists of occurrences associated with corresponding terms (eg, words) identified in the content of the content resources. Essentially, a list of occurrences points to a set of content resources whose content contains the corresponding term.

[95] In some embodiments, search engine data 310 stored in database system 120 may contain information about searches previously performed by search engine 130. For example, search engine data 310 may contain query data associated with corresponding queries. The request data associated with a request can be of various kinds and is not subject to any restrictions. For example, the request data may include (but is not limited to):

- query popularity;

- the frequency of sending the request;

- the number of "clicks" associated with the request;

- indications of other sent requests related to the request;

- indications of documents related to the request;

- other statistics related to the request;

- text representation of the request;

- the number of characters in the request;

- audio representation (or representations) of the request;

- other characteristics inherent in the request.

[96] It is also contemplated that search engine data 310 stored in database system 120 may contain information about content resources provided by search engine 130 as search results. The resource data associated with a content resource can be of various kinds and is not subject to any restrictions. For example, resource data may include (but is not limited to):

- the popularity of the content resource;

- coefficient of "clicks" for the content resource;

- time per "click" associated with the content resource;

- references to other documents related to the content resource;

- indications of requests related to the content resource;

- other statistical data related to the content resource;

- text data associated with the content resource;

- other characteristics of the content resource inherent in the document.

[97] The database system 120 may store the data 330 of the human evaluator. In general, human evaluator data 330 contains information at least partially generated by human evaluators and used by server 106 to train the MLA algorithm 400 (see FIG. 4) implemented as part of the snippet engine 135 and the ANN network 140. The server 106's use of the human estimator data 330 to train the MLA algorithm 400 of the snippet engine 135 and the server 106's ANN network 140 is described in more detail below.

[98] It should be noted that human evaluator data 330 may comprise a plurality of human-rated datasets. The human-rated dataset may contain (a) an indication of a query (Q), (b) an indication of a content resource relevant to that query, and (c) an indication of a human-rated content snippet for the corresponding content resource. A human-assessed data set may be generated by providing a query to the human assessor and then collecting the data provided by the human assessor. It should be noted that the human estimator data 330 may contain a large number of human-rated training datasets, as described below.

[99] For example, in response to a query, the human evaluator may use the search engine 130 to perform a search corresponding to the query and retrieve a plurality of search results related to the corresponding content resources that the search engine 130 determined to be relevant to the query. The human evaluator is then tasked with parsing the content of the content resources provided as search results and generating a human-rated content snippet for that query based on the content of at least one of the content resources provided as search results. In some non-limiting embodiments of the present technology, the human evaluator is also tasked with providing an indication of at least one of the content resources provided as search results used to generate the summary (e.g., validation, quality control, etc.). ).

[100] To better illustrate this, it can be assumed that the human evaluator uses the content of the second most relevant content resource for the query to generate a human-rated content snippet. In this case, the human-rated dataset may contain (a) an indication of a query, (b) an indication of the second most relevant content resource, and (c) an indication of a human-rated content snippet generated from the content of the second most relevant content resource.

[101] In some non-limiting embodiments of the invention, the human-rated content snippet may be the original text sequence found by the human evaluator in the content of the corresponding content resource. For example, if the query is the expression "John Malkovich", then the human evaluator might determine as the human-rated content snippet the following text sequence from the second most relevant content resource: "John Gavin Malkovich was born on December 9, 1953 in the United States of America. He is a voice actor, director, producer, and fashion designer. He received Academy award nominations for his performances in Places in the Hear (1984) and In the Line of Fire (1993)."

[102] In other non-limiting embodiments of the invention, a human-rated content snippet may be a text sequence based on the content of the corresponding content resource, partially modified by the human evaluator for brevity. For example, if the query is the expression "John Malkovich", then the human evaluator can determine the following text sequence from the second most relevant content resource: "John Gavin Malkovich was born on December 9, 1953 in the United States of America. He is a voice actor, director, producer, and fashion designer. He received Academy award nominations for his performances in Places in the Hear (1984) and In the Line of Fire (1993)" and can then partially modify this text sequence to form a more concise, human-rated content snippet based on it. For example, in this case, a human evaluator might generate the following human-rated content snippet: "John Gavin Malkovich is an Academy award nominated American actor born on December 9, 1953, and who is also a director, producer, and fashion designer." This means that, in some cases, the human evaluator may partially modify the original text sequence found in the content of the corresponding content resource so that it is a content summary that is more concise and meaningful (i.e., more concise) than the original. a text sequence from the corresponding content resource.

[103] The snippet engine 135, comprising the MLA algorithm 400, generates short content snippets for the respective content resources in more detail below.

MLA algorithm of the snippet mechanism

[104] As described above and shown in FIG. 4, the snippet engine 135 contains the MLA algorithm 400. In general, MLA algorithms are able to learn from training samples and make predictions based on new (unknown) data. MLA algorithms are typically used to first build a model based on training inputs, and then make predictions or decisions based on the data, expressed as outputs, rather than executing static machine-readable instructions. MLA algorithms can be used as scoring models, ranking models, classification models, and the like.

[105] It should be understood that in some non-limiting embodiments of the present technology, different kinds of MLA algorithms with different architectures and/or topologies can be used to implement the MLA algorithm 400. However, in the context of the present technology, the MLA algorithm 400 can be said to be an MLA algorithm that, based on a request-resource pair, learns to generate a predictable content snippet (eg, in text format).

[106] In general, the implementation of the MLA algorithm 400 by the server 106 can be divided into two main phases: a learning phase and a usage phase. First, the MLA algorithm 400 is trained in the training phase. Then, when the MLA algorithm 400 knows what input data is expected and what output data should be given, the MLA algorithm 400 is executed with real data at the stage of use. As described below, it is contemplated that the server 106 may also use the MLA algorithm 400 to generate at least some training data for training the ANN network 140.

[107] FIG. 4 is a representation 480 of one training iteration of the MLA algorithm 400. However, it should be noted that server 106 may perform a large number of training iterations based on the respective training datasets, similar to server 106 being able to perform a single training iteration shown in FIG. 4.

[108] During the training iteration shown, the server 106 is able to train the MLA algorithm 400 based on the training dataset 410. In at least some non-limiting embodiments of the present technology, training dataset 410 may correspond to a corresponding human-rated dataset from a plurality of human-rated datasets stored in database system 120. In other words, training dataset 410 may contain data indicative of (a) a request 416, (b) a corresponding content resource 414, and (c) a corresponding target content snippet 412 (e.g., a human-rated content snippet for a corresponding request- resource).

[109] The server 106 is capable of using (a) the request 416 and the corresponding content resource 414 as input to the MLA algorithm 400 and (b) the corresponding target content snippet 412 as the training target. In some embodiments of the invention, it is contemplated that data indicative of the request 416 and the corresponding content resource 414 can be entered into the MLA algorithm 400 in vector format. For example, the content of the request 416 and the corresponding content resource 414 may be entered as vector representations based on the respective bag-of-words representations.

[110] In response to input representing the request 416 and the corresponding content resource 414, the MLA algorithm 400 is able to produce a predictable content snippet 420. In other words, the predicted content snippet 420 is a content snippet prediction generated by the MLA algorithm 400 for the corresponding content resource 414, for example, when the content resource 414 is provided as a search result relevant to query 416. It is contemplated that the predicted content snippet 420 can be returned by the algorithm 400 MLA in vector format without going beyond the real technology.

[111] The server 106 is then able to generate a penalty estimate 430 for the training iteration based on the predicted content snippet 420 and the target content snippet 412 (ie, based on a comparison of the prediction against the target). The type of penalty metric used to generate the penalty score 430 based on the predicted content snippet 420 and the target content snippet 412 is not particularly limited. However, it should be noted that the penalty score 430 indicates a difference between the target content snippet 412 and the predicted content snippet 420. It can also be said that the penalty score 430 may indicate a degree of similarity or difference between the target content snippet 412 and the predicted content snippet 420.

[112] At the training iteration, the server 106 is able to train the MLA algorithm 400 based on the penalty estimate 430 . For example, server 106 may use backpropagation to adjust MLA algorithm 400 based on penalty estimate 430 . Thus, it can be said that as a result of the execution of the training iteration by the server 106, the MLA algorithm 400 is trained to generate corresponding predicted content snippets similar to the corresponding target content snippets (i.e., generating predictions closer to the corresponding targets). Given that the target content snippets may be the corresponding human-rated content snippets from the database system 120, the MLA algorithm 400 can thus be trained to generate predictive content snippets similar to the human-rated content snippets (e.g., short content snippets).

[113] As described above, after performing a large number of training iterations, the server 106 may apply the trained MLA algorithm 400 at its use stage. In a non-limiting example, the server 106 may use the MLA algorithm 400 when generating content snippets for search results corresponding to a query submitted to search engine 130. To better illustrate this, FIG. 2 shows that the server 106 can use the MLA algorithm 400 thus trained to generate:

- the first content snippet 214 for the first search result 210 based on the query 202 and the first content resource 212;

- a second content snippet 224 for a second search result 220 based on a query 202 and a second content resource 222;

- a third content snippet 234 for third search results 230 based on query 202 and third content resource 232.

[114] In addition, in at least some non-limiting embodiments of the present technology, in addition to using the MLA algorithm 400 when generating the SERP page by the search engine 130 of the server 106, the MLA algorithm 400 can also be used by the server 106 to train the ANN network 140. The following describes in more detail the training and subsequent use of the ANN network 140 by the server 106 to generate a summary of the content of the SERP page.

ANN networks

[115] As described above, the server 106 may contain a network 140 ANN. In general, NN networks are a special class of MLA algorithms and consist of interconnected groups of artificial “neurons” that process information using a connection approach to computing. NNs are used to model complex relationships between inputs and outputs (without actual knowledge of those relationships) or to look for patterns in data. The NNs are first adapted in the training phase when they are provided with a known set of inputs and information to adapt the NNs to generate appropriate outputs (in the situation for which the simulation is being attempted). In this learning phase, the NN adapts to the situation being learned and changes its structure so that the NN is able to provide an adequate output prediction for the input in the new situation (based on what has been learned). Thus, instead of trying to determine complex statistical distributions or mathematical algorithms for a given situation, the NN network tries to provide an "intuitive" answer based on the "perception" of the situation. Thus, the NN network is a kind of trained black box that can be used in situations where the contents of the box may be less important than that this box provides adequate responses to the input data. For example, NNs are widely used to optimize the distribution of web traffic between servers and data processing, including filtering, clustering, signal separation, compression, vectorization, and so on.

[116] It should be noted that encoder-decoder models are a class of models based on NN networks. Encoder-decoder models typically consist of (a) an encoder subnet used to encode input data into their "hidden representations" and (b) a decoder subnet used to decode these hidden representations into output data. They are often used to solve what are known in the art as "sequence-to-sequence" (seq2seq) prediction problems. Such tasks include machine translation, abstracting of texts, implementation of chatbots, etc.

[117] In practice, some NNs are also called seq2seq models and are capable of transforming an input data sequence into an output data sequence. Seq2seq models can use recurrent neural networks (RNN, Recurrent NN), for example, in order to, in a certain sense, "extract" the context in which the input data is used in the appropriate sequence. For simplicity, we can say that the encoder subnet of the seq2seq model transforms each input into the corresponding hidden vector (containing the input and the context), and the decoder subnet reverses the process and transforms the vector into the output (using the previous output as the input context).

[118] More modern seq2seq models are called transformers. Like models based on RNNs, transformers are designed to process serial data in tasks such as translation and summarization of texts. However, unlike RNN-based models, transformers do not require that these serial data be processed in a particular order. For example, if the input is a natural language sentence, then the transformer does not need to process its beginning before processing its end. Due to this feature, transformers allow much more parallelization than models based on RNN networks and, therefore, can reduce training time. Since the development of transformers, they have been used to solve various problems related to natural language processing (NLP, Natural Language Processing). An example of such an implementation is the transformer-based model used in the Bidirectional Encoder Representations from Transformers (BERT) system, which is trained on large language datasets and can be fine-tuned for specific language tasks.

[119] In the context of the present technology, ANN 140 may be implemented as an encoder-decoder model containing an encoder subnet and a decoder subnet.

[120] It is assumed that the ANN encoder subnet 140 is capable of implementing a particular attention mechanism. For example, the encoder subnetwork may contain an attention layer capable of applying an attention restriction mask to input data. As described below, when determining the context for a word from the input sequence, the attention constraint mask can "take into account" the words from the input group to which the word from the input sequence belongs, and therefore allows only the context of the corresponding input group to be taken into account, and not other contexts of other inputs. groups of the input sequence. Below with reference to FIG. 5 describes training the ANN network 140, determining the input groups from the input sequence, and setting the encoder subnet attention constraint mask in more detail.

ANN network training

[121] FIG. 5 shows one iteration of training the network 140 ANN. However, it should be noted that the server 106 may perform a large number of iterations of training the ANN network 140 . This large number of iterations of training the network 140 ANN can be performed at the main stage of training the network 140 ANN. It is contemplated that in at least some non-limiting embodiments of the present technology, the ANN network 140 may be trained in multiple training phases or in a multi-stage training phase.

[122] It is assumed that the ANN network 140 can perform an appropriate set of training iterations during each training step. In some non-limiting embodiments of the present technology, the ANN network 140 may have (a) a preliminary training phase, during which a preliminary set of training datasets is used to tune the model, (b) a main training phase, during which, to train the model to form satisfactory predictions uses the main set of training datasets, and (c) a "hallucination avoidance" training phase, during which predefined training datasets are used to adjust the model so that the model avoids making "out of the ordinary" predictions. It should be noted that the training iteration during any training step of the ANN network 140 can be performed in a manner similar to the training iteration shown in FIG. 5. However, as described below, training iterations of each respective training step of the ANN network 140 may be performed by the server 106 on the respective training datasets for the respective training goals.

[123] The ANN network 140 contains an encoder subnet 602 and a decoder subnet 604. In the training iteration, the server 106 may input to the encoder subnetwork 602 a training query 642 and a training text input sequence 610.

[124] It should be noted that the training text input sequence 610 is formed as a sequence of training content snippets, including the i th training content snippet 612, the first training content snippet 614, the second training content snippet 616, the third training content snippet 618, and the j th training content snippet 620 . In one non-limiting embodiment of the present technology, the training text input sequence 610 may be generated as a sequence of fifteen training content snippets. However, the training text input sequence from the ANN 140 training dataset may contain more or less than fifteen training content snippets without departing from the present technology.

[125] It should be noted that the training content snippets from the training text input sequence 610 may be generated by the server 106 in various ways. In one non-limiting example, server 106 may use search engine 130 to generate training content snippets for the search result set corresponding to training query 642. In some embodiments of the present technology, server 106 may generate training content snippets for the search result set in accordance with known methods.

[126] In other embodiments of the present technology, appropriate training content snippets from the training text input sequence 610 may be generated by the MLA algorithm 400 of the snippet engine 135 . For example, server 106 may determine fifteen content resources (or any other suitable number of content resources) that are relevant to training query 642 and use MLA algorithm 400 to generate an appropriate content snippet for each content resource from among the fifteen content resources, as described higher. The server 106 may then generate the training text input sequence 610 by generating a sequence of content snippets of the respective content resources (by concatenation). Thus, as described above, it can be said that the server 106 can use the MLA algorithm 400 to generate at least some of the training data for training the ANN network 140 without going beyond the present technology.

[127] It should also be noted that the training text input sequence 610 is divided into a sequence of input groups. In the context of the present technology, an input group is associated with a training content snippet and contains words from the training content snippet. In one non-limiting example, server 106 may determine words from training text input sequence 610 to be combined into an input group. To this end, the server 106 may use the appropriate training content snippets from the training text input sequence 610 to match the words from the training text input sequence 610 to the corresponding input groups. In the presented example, the sequence of input groups contains:

- the i -th input group 632 containing the words from the i -th training content snippet 612;

- the first input group 634 containing the words from the first training content snippet 614;

- a second input group 636 containing the words from the second training content snippet 616;

- a third input group 638 containing words from the third training content snippet 618;

- the j th input group 640 containing the words from the j th training content snippet 620.

[128] In some embodiments of the present technology, each word from the training text input sequence 610 may be associated with a corresponding position in the training text input sequence 610. Additionally or alternatively, each input group from the sequence of input groups may be associated with a corresponding position in the training text input sequence 610. text input sequence 610. In other embodiments, it is contemplated that data indicative of training query 642 and training text input sequence 610 can be input to MLA algorithm 400 in vector format. For example, server 106 may generate vector representations for training query 642 and training text input sequence 610 and use these vector representations as input to encoder subnet 602.

[129] The encoder subnetwork 602 is capable of generating an encoded representation 650 of the training text input sequence 610. In some embodiments of the present technology, it is contemplated that the encoded representation 650 generated by the encoder subnetwork 602 may consist of one or more vectors generated by the encoder subnetwork 602 at a training iteration. and to be further processed by the subnet 604 of the decoder.

[130] In general, the decoder subnetwork 604 is capable of generating a decoded representation for the training text input sequence 610. In the example shown, this decoded representation corresponds to the predicted text output sequence 652 of the ANN network 140. It should be noted that the predicted text output sequence 652 corresponds to the prediction generated by the ANN network 140 and is the predicted summary of the content of the sequence of training content snippets from the training text input sequence 610.

[131] The server 106 may compare the predicted text output sequence 652 with a predetermined text output sequence 644 (the prediction is compared to the target). It is contemplated that the predefined text output sequence 644 may be a human-rated content snippet. For example, as described above, server 106 may obtain a request and a corresponding human-rated content snippet from human rater data 330 stored in database system 120 and then use them as training query 642 and predefined text output sequence 644, respectively.

[132] The server 106 is also capable of generating a penalty estimate 655 based on a comparison of the predicted text output sequence 652 with the predefined text output sequence 644. text output sequence 644 is not subject to any particular restrictions. However, it should be noted that the penalty score 655 indicates the difference between the predicted text output sequence 652 and the predetermined text output sequence 644. It can also be said that the penalty score 655 may indicate the degree of similarity or difference between the predicted text output sequence 652 and the predetermined text output sequence 652. given text output sequence 644.

[133] In the training iteration, the server 106 is able to train the ANN network 140 based on the penalty estimate 655 . For example, server 106 may use backpropagation to update ANN network 140 based on penalty estimate 655 . However, other learning methods may also be used in conjunction with the penalty estimate 655 to adjust the ANN network 140 to improve predictions. Regardless of the particular training method used, it can be said that as a result of the training iteration performed by the server 106, the ANN network 140 is trained to generate predictive text output sequences for queries that are similar to the predefined text output sequences (i.e., generate predictions closer to the corresponding goals ). Given that the predetermined text output sequences may be human-rated content snippets from the database system 120, the ANN network 140 can thus be trained to generate predictive text output sequences similar to human-rated content snippets for queries (e.g., with brief content snippets).

[134] As described above, encoder subnetwork 602 may apply a particular attention mechanism to training text input sequence 610 to generate encoded representation 650. In general, encoder subnetwork 602 may apply an attention constraint mask 606 to training text input sequence 610 when generating attention output data to be used in generating the encoded representation 650.

[135] In some embodiments of the present technology, the encoder subnet 602 attention output may be said to indicate contexts in which the words of the training text input sequence 610 are used. "attention" for a word from the input group of the training text input sequence 610, the attention constraint mask 606 allows "taking into account" only the words from this input group so that the output of the type "attention" is formed based on the context obtained from this input group, and not contexts derived from other input groups in the training text input sequence 610. As a result, the encoder subnetwork 602 can obtain content for a word from the training text input sequence 610 depending only on the words from the training content snippet to which the word refers, without considering the words from other training content snippets from the training text input sequence 610.

[136] For example, it can be assumed that the encoder subnetwork 602 processes the word associated with the second input group 636 from the second training snippet 616 of content containing three words, as shown in the view 690. As shown, when the encoder subnetwork 602 generates for some word attention output 680, the attention restriction mask 606 is applied to the training text output sequence 610 such that the encoder subnetwork 602 can only "take into account" two other words from the same input group (from the second input group 636) and " does not take into account" words from other input groups of the training text input sequence 610. As a result, it can be said that the encoder subnetwork 602, which has an attention restriction mask 606, uses the context of the corresponding input when generating the attention output data for a word from the training text input sequence 610 groups (in this case, the context obtained from the second input group 636), and not the contexts of other input groups.

[137] It should also be noted that in addition to using the context of the corresponding input group, the encoder subnetwork 602 may use the context of the training request 642. input sequence 610 uses the context of the training query 642 and the context of the corresponding input group (in this case, the context obtained from the second input group 636), and not the contexts of other input groups. In at least some embodiments of the invention, it is contemplated that the context of training query 642 may be one or more of the terms contained in training query 642 without departing from the present technology.

[138] Briefly, the ANN network 140 can be trained during the main stage of its training based on a large number of iterations performed similar to the training iteration shown in FIG. 5. As described above, the server 106 may perform this large number of iterations based on the respective training datasets.

[139] It should be noted that in at least some embodiments of the present technology, it is assumed that during the main training phase, the server 106 can generate and use modified training datasets to train the ANN network 140 .

[140] For a better illustration of this, the following is shown in FIG. 5, a training dataset comprising a training text input sequence 610, a training query 642, and a predefined text output sequence 644. Given that the training text input sequence 610 is generated as a sequence of training content snippets, in some non-limiting embodiments of the present technology, the server 106 can generate a modified training dataset by (a) reordering the training content snippets in the sequence of training content snippets of the training text input sequence 610 and (b) using the training text input sequence thus rearranged (instead of sequence 610), training query 642, and predefined text output sequence 644 as the modified training dataset. Thus, it is contemplated that the server 106 can generate a modified training dataset based on the training dataset by generating a modified training text input sequence in which the order of the input groups is different from the order of the input groups in the training text input sequence of the training dataset.

[141] The developers of the present technology have found that by training the ANN network 140 based on the training datasets and based on the modified datasets, the ANN network 140 can generate predictions that are less dependent on the order of the input groups in the text input sequence, and much more dependent on the contexts of the corresponding input groups. In at least some embodiments of the present technology, if the ANN network 140 is used to generate a content summary for the SERP page based on the query and content snippets of the search results on the SERP page, then training the ANN network 140 based on the training datasets and based on the modified datasets allows the ANN network 140 to provide summaries of content less dependent on the order of the search results on the SERP page. As a result, the content summaries generated by the ANN 140 for a search result set are likely to be similar to each other (more consistent content summaries) regardless of the order in which the search result set is ranked on the SERP page.

[142] As described above, in some non-limiting embodiments of the present technology, the server 106, after the main training step of the ANN network 140, may additionally perform its training step to prevent "hallucinations". The developers of the present technology have found that in some non-limiting embodiments of the invention, the ANN network 140 trained during the main stage of its training can form "unusual" predictions for some text input sequences of the use stage. In other words, the ANN 140 may "hallucinate" in the sense that the prediction generated for some text input sequences may be clearly wrong.

[143] To correct these mispredictions, in some non-limiting embodiments of the present technology, the server 106 may perform a "hallucination prevention" training step during which a set of "best" training datasets is selected to adjust the prediction quality of the ANN network 140. This set of best training datasets can be pre-specified by human estimators. Server 106 may also preselect these training datasets as a subset of training datasets in which the predetermined text output sequence is present in one or more snippets of the corresponding training text input sequence.

[144] Having trained the ANN network 140, the server 106 can apply the ANN network 140 at the stage of its use. Below with reference to FIG. 6 describes the possible use by server 106 of ANN 140 at the stage of its use in at least some non-limiting embodiments of the present technology.

ANN use phase

[145] FIG. 6 is a representation 700 of how the server 106 can use the ANN network 140 at the stage of using it. As shown, the audio signal 152 may be captured and an audio representation 702 of the user's speech 102 may be generated by the electronic device 104. The audio representation 702 may be processed to generate a textual representation 704 of the user's request. It should be noted that the STT (Speech-To-Text) procedure 720 may be performed by the electronic device 104 and/or server 106 in various ways without departing from the present technology.

[146] The server 106 may search in response to a textual representation 704 of the user's request. In other words, the server 106 may perform a search procedure 730 to obtain one or more content resources relevant to the user's query. In some embodiments, the server 106 may use the search engine 130 to retrieve a variety of content resources relevant to the user's query. It should be noted that the server 106 may also use the snippet engine 135 to generate use-phase content snippets for appropriate content resources from a plurality of content resources. Thus, the server 106 may generate a SERP page 706 containing a plurality of search results, each associated with a respective content resource relevant to the user's query and having a respective usage stage content snippet generated based on the content from the respective content resource.

[147] The server 106 may use the information generated by the search engine 130 and the snippet engine 135 to perform a usage step data set generation procedure 740 . In general, when executing use-phase data set generation procedure 740, server 106 may use the information generated by search engine 130 and snippet engine 135 to generate use-phase text input sequence 708. For example, server 106 may generate use-phase text input sequence 708 as a sequence of use-phase content snippets generated by snippet engine 135 for appropriate content resources relevant to a user request.

[148] The server 106 may input a text representation 704 of the user request and a text input sequence 708 of the use stage of the use stage into the trained ANN network 140. Thus, it can be said that the server 106 can perform a summary procedure 750 during which iteration of the usage step of the ANN network 140 is performed on the entered user query textual representation 704 and the usage step textual input sequence 708. As a result, the trained ANN network 140 provides server 106 with a use-stage text output sequence 710 representing a (predicted) summary of the content of a sequence of use-stage content snippets (e.g., generated by a search engine according to some known methods and/or generated by snippet engine 135 as described above). ) and to be used as a response to a user request.

[149] The server 106 may also generate an audio representation 712 of the text output sequence 710. The audio representation 712 is to be used as a computer-generated speech fragment of the response. It is contemplated that Text-To-Speech (TTS) procedure 750 may be performed by server 106 and/or electronic device 104. Having received and/or generated audio representation 712, electronic device 104 may use this representation to generate an audio signal. 154 to user 102, for example, using one or more speakers.

[150] In some non-limiting embodiments of the present technology, the server 106 may perform the method 800 shown in FIG. 7. The various steps of method 800 are described in more detail below.

Step 802: Entering a training query and a training text input sequence.

[151] Method 800 begins at step 802, where the server 106 may inject the training query 642 and training text input sequence 610 into the encoder subnetwork 602 (see FIG. 5) in the training iteration of the ANN network 140 (see FIG. 5). The training text input sequence 610 is (a) formed as a sequence of training content snippets (e.g., a sequence containing content snippets 612, 614, 616, 618, 620) and (b) split into a sequence of input groups (e.g., a sequence containing input groups 632, 634, 636, 638, 640). Note that the input group is associated with the training content snippet and contains words from the training content snippet.

[152] In some embodiments, training content snippets from a training text input sequence may be generated by server 106 using search engine 130. For example, server 106 may search for and use content snippets associated with at least some relevant search results, in as relevant training content snippets. In some embodiments of the invention, training content snippets may be generated by methods known in the art.

[153] In some embodiments, server 106 may obtain training content snippets from appropriate content resources relevant to training query 642. It is contemplated that server 106 may obtain training content snippets by performing a search using search engine 130 and training query 642 as input data. It is also contemplated that server 106 can obtain training content snippets by obtaining the N most relevant snippets associated with the top N search results.

[154] In other embodiments, server 106 may generate a training content snippet from a sequence of training content snippets. In some embodiments, the server 106 may input the training request 642 and content from the training content resource into an MLA algorithm 400 capable of producing a predictable content snippet. As described above, the MLA algorithm 400 may be trained by the server 106 to generate a predictive content snippet that is similar to a predetermined text output sequence and is a training content snippet to be used as part of the training text input sequence 610.

Step 804: generating an encoded representation of the training text input sequence.

[155] Method 800 continues at step 804, where server 106 may, using encoder subnet 602, generate an encoded representation 650 of training text input sequence 610. To generate encoded representation 650, server 106 may generate attention-type output (such as 680 attention) for the corresponding words from the training text input sequence 610 by applying an attention restriction mask 606 to the training text input sequence 610. The attention output is used by the server 106 to generate an encoded representation 650 of the training text input sequence 610.

[156] It should be noted that when generating attention output 680 for a word from input group 636, the attention constraint mask 606 allows only words from input group 636 to be "taken into account" so that attention output 680 is generated on based on the input group context 636 rather than other input group contexts from the training text input sequence 610.

[157] It should also be noted that in addition to using the context of the corresponding input group, the encoder subnetwork may use the context of the training request 642. It is assumed that it can be said that the encoder subnetwork 602 when generating the attention output for a word from the training text input sequence 610 uses the context of the training query 642 and the context of the corresponding input group (in this case, the context obtained from the second input group 636), and not the contexts of other input groups. In at least some embodiments of the invention, it is contemplated that the context of training query 642 may be based on one or more of the terms contained in training query 642 without departing from the present technology.

Step 806: generating a decoded representation of the training text input sequence.

[158] The method 800 continues at step 806, where the server 106, using the decoder subnet 604, can generate a decoded representation 652 for the training text input sequence 610 corresponding to the predicted text output sequence of the ANN network 140. It is contemplated that decoded representation 652 can be provided by ANN 140 in vector format without departing from the present technology.

[159] In some embodiments of the invention, it is contemplated that the decoder subnetwork 604 may use its own attention mechanism. For example, the attention mechanism of the decoder subnetwork 604 can be implemented similar to the attention mechanism of the decoder from the transformer network. In some cases, it can be said that the attention mechanism of the decoder subnetwork 604 can be implemented in accordance with art-known variations for transformer networks.

Step 808: generating a penalty estimate for a given training iteration.

[160] The method 800 continues at step 808, where the server 106 may generate a penalty estimate 655 for the training iteration by comparing the predicted text output sequence 652 with the predefined text output sequence 644. The predefined text output sequence is a predefined response to the training query. 642.

[161] It should be noted that in some embodiments, the server 106 may additionally receive a predefined text output sequence 644. In some embodiments, the predefined text output sequence 644 may be obtained from a human evaluator (e.g., generated by a human evaluator and stored in the database system 120 as described above). In some embodiments, the server 106 may also obtain an indication of the content resource used by the human evaluator to generate the predefined text output sequence 644. In some embodiments, the server 106 may obtain from the database system 120 an indication of the content resource that may be associated with a predetermined textual output sequence 644 from a human-assessed data set. In one non-limiting example, a content resource may be used to test a predefined text output sequence 644. It is also contemplated that the content resource may be a network resource corresponding to training query 642 and obtained by performing a search using search engine 130, without going beyond the current technology.

Step 810: Adjusting the ANN based on the penalty estimate.

[162] Method 800 continues at step 810, where server 106 may update ANN network 140 based on penalty estimate 655. In one non-limiting example, the server 106 to correct the ANN network 140 may backpropagate the ANN network 140 based on the penalty estimate 655 .

[163] In some embodiments of the present technology, the server 106 may perform a large number of training iterations, similar to the training iteration shown in FIG. 5. For example, in a training iteration, the ANN network 140 is trained based on a training dataset containing (a) a training query 642, (b) a training text input sequence 610, and (c) a predefined text output sequence 644. In some embodiments of the present technology, the server 106 may also, in another training iteration, train the ANN network 140 based on a different training dataset (modified training dataset). For example, another training dataset may contain (a) a training query 642, (b) a modified training text input sequence, and (c) a predefined text output sequence 644, where the input groups in the modified training text input sequence are in an order other than the order of the input groups in the training text input sequence 610.

[164] In some embodiments of the present technology, the server 106 may perform the configuration shown in FIG. 5 iteration of training during the first (and/or main) stage of training the network 140 ANN. However, the server 106 may also perform a second training step for the ANN 140 (eg, a training step to prevent "hallucinations") including a second set of training iterations. On a different training iteration of the second set of training iterations, the server 106 may train the ANN network 140 based on a different training dataset from the set of predefined training datasets to be used in performing the second set of training iterations. For example, a different training dataset may be selected by the human evaluator.

[165] In some embodiments of the present technology, the predefined text output sequence 644 may be determined by a human evaluator. For example, a predetermined textual output sequence 644 may be obtained from human appraiser data 330 stored in the database system 120. Thus, the human evaluator may determine a predefined text output sequence 644 from the content of at least one of the training content resources associated with the training content snippets from the training text input sequence 610.

[166] In other embodiments of the present technology, the server 106 may also perform the step of using the ANN network 140. For example, the server 106 may generate a textual representation 704 (see FIG. 6) of the query based on the audio representation 702 of the user's speech indicating the user's request 102 associated with the electronic device 104. The server 106 may also determine search results corresponding to the textual representation 704 query, where the search result contains a content snippet and points to a content resource relevant to the query. To do this, the server 106 may perform a search in response to a textual representation 704 of the user's query. In other words, the server 106 may perform a search procedure 730 to obtain one or more content resources relevant to the user's query. In some embodiments, the server 106 may use the search engine 130 to retrieve a variety of content resources relevant to the user's query. It should be noted that the server 106 may also use the snippet engine 135 to generate use-phase content snippets for appropriate content resources from a plurality of content resources. Thus, the server 106 may generate a SERP page 706 containing a plurality of search results, each associated with a content resource relevant to the user's query and having a usage stage content snippet generated based on the content from the corresponding content resource.

[167] The server 106 may also generate the text input sequence 708 as a sequence of content snippets based on the content snippets of the search results. The server 106 may also input a text input sequence 708 and a text representation 704 of the request into the ANN network 140 capable of providing a text output sequence 710. The text output sequence 710 may be a summary of the content of the content snippet sequence that may be used as a response to the request. The server 106 may also generate an audio representation 712 of the text output sequence 710, which is a computer generated speech fragment of the response. The server 106 may also provide the electronic device 104 with an audio representation 712 of the textual output sequence.

[168] For a person skilled in the art, possible changes and improvements in the above-described embodiments of the present technology may be obvious. The foregoing description is for illustrative purposes only and is not intended to limit the scope of the invention. The scope of protection of this technology is determined solely by the scope of the appended claims.

Claims (54)

1. A method for training an attentional neural network (ANN) containing an encoder subnetwork and a decoder subnetwork to generate a text output sequence based on a query and a text input sequence generated as a sequence of content snippets obtained from the corresponding content resources relevant to the query, while the text output sequence represents a summary of the content of a sequence of content snippets and is to be used as a response to a request, and the method is performed by a server containing the ANN, and in the training iteration of the ANN, includes: - the server inputs a training request and a training text input sequence into the encoder subnet, while the training text input sequence (a) is formed as a sequence of training content snippets and (b) is divided into a sequence of input groups, and the input group is associated with the training content snippet and contains words from this learning content snippet; - the formation by the server using the encoder subnet of an encoded representation of the training text input sequence, including the generation of output data of the "attention" type for the corresponding words from the training text input sequence by applying an attention restriction mask in relation to the training text input sequence, and the output data of the "attention" type are used to generate an encoded representation of the training text input sequence, when generating output data of the “attention” type for a word from the input group, the attention restriction mask allows you to “take into account” only words from this input group so that the output data of the “attention” type is formed based on the context of this input group, and not the contexts of other input groups in the training text input sequence; - the formation by the server using the decoder subnet of a decoded representation for the training text input sequence corresponding to the predicted text output sequence; - the formation by the server of a penalty estimate for the training iteration by comparing the predicted text output sequence with a predefined text output sequence representing a predefined response to the training request; And - adjustment by the ANN network server based on the penalty estimate. 2. The method of claim 1, wherein the generated attention type output is additionally based on the context from the training query. 3. The method according to claim 1, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence, while the method at another training iteration further includes training the ANN based on another training dataset containing (a) a training query, (b) a modified training text input sequence, and (c) a predefined text output sequence, and the input groups in the modified training of the text input sequence are arranged in an order different from the order of the input groups in the training text input sequence. 4. The method of claim. 1, characterized in that it further includes generating a training content snippet from a sequence of training content snippets, including input by the server of the training query and content from the training content resource into a machine learning algorithm (MLA) capable of producing a predictive a content snippet, wherein the MLA algorithm is trained to generate a predictable content snippet such that the predicted content snippet is similar to a predetermined text output sequence and is a training content snippet. 5. The method according to claim 1, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence, while the training iteration is performed at the first training step of the ANN, including the first set of training iterations, the method further includes performing the second training step of the ANN, including the second set of training iterations, and at the other training iteration of the second set of training iterations, the ANN is trained on the basis of another training dataset, which is a set of a plurality of predefined training datasets to be used in performing the second plurality of training iterations, and is selected by the human evaluator. 6. The method according to claim 1, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence determined by the estimator a human, wherein the human evaluator determines a predetermined text output sequence from the content of at least one of the learning content resources associated with the learning content snippets. 7. The method according to p. 1, characterized in that it additionally includes: - the formation by the server of a textual representation of the request based on the audio representation of the user's speech fragment indicating the request of the user associated with the electronic device; - determination by the server of search results corresponding to the text representation of the request, while the search result contains a content snippet and points to a content resource relevant to the request; - formation by the server of a textual input sequence in the form of a sequence of content snippets based on content snippets of the search results; - input by the server of a textual input sequence and a textual representation of the request to the ANN capable of producing a textual output sequence representing a summary of the content of the sequence of content snippets and to be used as a response to the request; - formation by the server of an audio representation of a text output sequence, which is a speech fragment of the response generated by the computer; And - providing by the server to the electronic device associated with the user, an audio representation of the text output sequence. 8. The method of claim. 1, characterized in that the adjustment includes the server application of the method of backpropagation in relation to the network ANN. 9. The method of claim 1, further comprising obtaining content snippets from corresponding content resources relevant to the query. 10. The method of claim 9, wherein the acquisition includes performing a search using a search engine and a query as input. 11. The method of claim 10, wherein the acquisition includes obtaining the N most relevant snippets associated with the N best search results. 12. The method of claim. 1, characterized in that it further includes obtaining a predetermined text output sequence. 13. The method according to claim 12, characterized in that the receipt is carried out from a human appraiser. 14. The method of claim 13, further comprising obtaining an indication of the content resource used by the human evaluator to generate the predetermined text output sequence. 15. The method of claim 14, wherein the content resource is used to validate a predetermined textual output sequence. 16. The method according to claim 15, characterized in that the content resource is a network resource corresponding to the training query and obtained by performing a search using a search engine. 17. A server for training an attentional neural network (ANN) comprising an encoder subnetwork and a decoder subnetwork to generate a text output sequence based on a request and a text input sequence formed as a sequence of content snippets obtained from the corresponding content resources relevant to the request, with a text output the sequence is a summary of the content of a sequence of content snippets and is to be used as a response to a request, and the server contains the ANN network and is configured, at the training iteration of the ANN network: - input of a training query and a training text input sequence into the encoder subnet, while the training text input sequence (a) is formed as a sequence of training content snippets and (b) is divided into a sequence of input groups, and the input group is associated with the training content snippet and contains words from this tutorial content snippet; - generating, using the subnet of the encoder, an encoded representation of the training text input sequence, including generating output data of the "attention" type for the corresponding words from the training text input sequence by applying an attention restriction mask in relation to the training text input sequence, while the output data of the "attention" type is used to form an encoded representation of the training text input sequence, and when generating output data of the “attention” type for a word from the input group, the attention restriction mask allows you to “take into account” only words from this input group so that the output data of the “attention” type is formed based on the context of this input group, and not the contexts of other input groups in the training text input sequence; - generating, using the decoder subnet, a decoded representation for the training text input sequence corresponding to the predicted text output sequence; - generating a penalty estimate for the training iteration by comparing the predicted text output sequence with a predetermined text output sequence representing a predetermined response to the training request; And - adjustments to the ANN network based on the penalty estimate. 18. The server according to claim 17, characterized in that the generated output data of the "attention" type is additionally based on the context from the training request. 19. The server according to claim 17, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence, and the server is additionally capable, at another training iteration, of training the ANN network based on a different training dataset containing (a) a training query, (b) a modified training text input sequence, and (c) a predefined text output sequence, wherein the input groups in the modified training text input the sequences are in an order different from the order of the input groups in the training text input sequence. 20. The server according to claim 17, characterized in that it is additionally configured to generate a training content snippet from a sequence of training content snippets, including inputting a training query and content from a training content resource into an MLA algorithm capable of issuing a predictable content snippet, while the MLA algorithm is trained to generate a predictable content snippet such that the predicted content snippet is similar to a predefined text output sequence and is a training content snippet. 21. The server according to claim 17, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence, while the training iteration is performed at the first training step of the ANN, including the first set of training iterations, the server is further configured to perform the second training step of the ANN, including the second set of training iterations, and at the other training iteration of the second set of training iterations, the ANN is trained on the basis of another training dataset, which is a set of a plurality of predefined training datasets to be used in performing the second plurality of training iterations, and is selected by the human evaluator. 22. The server according to claim 17, characterized in that at the training iteration, the ANN is trained on the basis of a training data set containing (a) a training query, (b) a training text input sequence, and (c) a predefined text output sequence determined by the estimator a human, wherein the human evaluator determines a predetermined text output sequence from the content of at least one of the learning content resources associated with the learning content snippets. 23. The server according to claim 17, characterized in that it is additionally configured to: - generating a text representation of the request based on the audio representation of the user's speech fragment indicating the request of the user associated with the electronic device; - determining search results corresponding to the textual representation of the query, wherein the search result contains a content snippet and points to a content resource relevant to the query; - formation of a text input sequence in the form of a sequence of content snippets based on content snippets of search results; inputting a textual input sequence and a textual representation of the request into an ANN capable of producing a textual output sequence representing a summary of the content of the sequence of content snippets to be used as a response to the request; - formation of an audio representation of the text output sequence, which is a speech fragment of the response generated by the computer; And - providing the electronic device associated with the user with an audible representation of the text output sequence. 24. The server as claimed in claim 17, wherein the adjustment includes applying backpropagation by the server to the ANN. 25. The server according to claim 17, characterized in that it is additionally configured to receive content snippets from the corresponding content resources relevant to the request. 26. The server of claim 25, wherein the acquisition includes performing a search using a search engine and a query as input. 27. The server of claim 26, wherein the acquisition includes obtaining the N most relevant snippets associated with the N best search results. 28. The server according to claim 17, characterized in that it is additionally configured to receive a predetermined text output sequence. 29. The server according to claim 28, characterized in that it is configured to receive a predefined text output sequence from a human evaluator. 30. The server of claim 29, further configured to obtain an indication of the content resource used by the human evaluator to generate the predetermined text output sequence. 31. The server according to claim 30, characterized in that the content resource is used to check a predetermined text output sequence. 32. The server according to claim 31, characterized in that the content resource is a network resource corresponding to the training request and obtained by performing a search using a search engine.

Images