RU2778385C1 - Method and system for formation of training set for machine learning algorithm - Google Patents

Abstract

FIELD: machine learning algorithms.

SUBSTANCE: computer method for the formation of a training set for training a machine learning algorithm (hereinafter – MLA) implemented in an information search system and performed by a server is disclosed. In this case, the method is implemented by the server and includes: extraction from a search log database on the server of a set of search engine results page (hereinafter – SERP), while a page of the set of SERP pages is formed in response to a search request of a set of search requests, and it contains a set of documents responding to this search request; formation by the server of a graph, in which the set of documents on the set of SERP pages and the set of requests are combined in such a way that: a graph node contains a search request of the set of search requests, and it is connected to the first subset of nodes, while each node of the first subset of nodes contains a document of the set of documents, related to this search request, and the node of the first subset of nodes is connected to the second subset of nodes, while each node of the second subset of nodes contains search requests related to the document of this node; reception of an indication to a search request for the formation of a negative training example for training MLA algorithm; identification of the node in the graph, related to this search request; and formation of a training set for use as the negative training example for the search request in training MLA algorithm containing the search request in a pair with the document located in another node being at a distance of the predetermined number of nodes from this node.

EFFECT: provision of the formation of training sets for training machine learning algorithms implemented in an information search system.

20 cl, 8 dwg

Description

The field of technology to which the invention belongs

[001] The present technology relates generally to machine learning algorithms, and more specifically, to a method and system for generating a training set for a machine learning algorithm.

State of the art

[002] Machine learning algorithms (MLA, Machine Learning Algorithm) are used to meet many needs in computer technology. Typically, MLA algorithms are used to generate predictions related to user interaction with a computing device. An example of an area where such a forecast is required is the user's actions in relation to content available on the Internet.

[003] The amount of information available on various Internet resources has grown exponentially over the past two years. Several solutions have been developed to help the typical user find the information they need. One example of such solutions is a search engine. Examples include search engines GOOGLE™, YANDEX™, YAHOO!™, etc. The user may access the interface of such a search engine and submit a search query related to information to be found on the Internet. In response, the search engine returns a ranked list of search results that are potentially relevant to the search query. The ranked list of search results is generated based on various ranking algorithms applied by the particular search engine used by the user to search. The general goal of such ranking algorithms is to present the most relevant search results at the top of the ranked list, while the less relevant search results are placed in the less prominent positions of the ranked list of search results (the positions with the least relevant search results are located at the bottom of the ranked list). search results list).

[004] Search engines are usually an effective search tool in the case of a search query, when the user knows in advance what he wants to find. In other words, if the user is interested in information about the most popular places in Italy (i.e. when the search topic is known), he can send the search query "The most popular places in Italy?" After that, the search engine presents a ranked list of Internet resources that are potentially relevant to this search query. Then the user can view the ranked list of search results and get information about the sights of Italy of interest to him. If the user is not satisfied for any reason with the presented search results, he can re-run the search, for example, with a refined search query, such as "The most popular places in Italy in the summer?", "The most popular attractions in southern Italy?", "The most popular destinations for a romantic getaway in Italy?

[005] When a user submits a search query, the search engine generates a list of relevant web resources (based on the analysis of web resources viewed by the crawler, indications of which are stored in the crawler database as posting lists, etc.). The search engine then uses the MLA algorithm to rank the resulting list of search results. The MLA algorithm ranks a list of search results based on their relevance to the search query. Such an MLA algorithm is trained to predict the degree of relevance of a search result to a search query based on a large number of features associated with that search result and indications of user behavior on search results when submitting similar search queries in the past.

[006] Before using the MLA algorithm to predict the relevance of search results to the query submitted by the user, this MLA algorithm must be trained. Depending on the implementation of the MLA algorithm, learning can be done in various ways such as supervised learning, unsupervised learning, etc. The exemplary MLA training approach provides several training examples (labeled training sets) used to train the MLA algorithm to make inferences within a given task (such as predicting the relevance of documents to a search query). Typically, the MLA algorithm is provided with positive examples (in this case, relevant documents) and negative examples (in this case, irrelevant documents).

[007] Patent application US2017220575A1 "Identification of Synthetic Examples for Improving Search Ranking", Shutterstock Inc., 07/03/2017 discloses methods, systems, and computer-readable media for identifying synthetic media examples for training with the teacher of the MLA algorithm, designed to rank media files by relevance to a search query. In one aspect, the method includes identifying a search engine query for a collection of media files from previously received search queries and selecting at least one training media file from that collection as a synthetic negative example for that search query. The method also includes providing a training set for supervised learning of the ranking algorithm. The training set contains a search query identifier, a copy of the training media, and a first indication that the training media is a synthetic negative example for that search. The method further includes providing a search query and a collection to the algorithm, and receiving ranking data from the algorithm about the collection.

[008] Xuanhui Wang et al. “Improved Retrieval Accuracy for Difficult Queries using Negative Feedback” (Conference on Information and Knowledge Management (CIKM), 2007), discusses the situation when the search results are so bad that none from documents with a high rank is not relevant to the user's query. To improve the accuracy of the search for such complex queries, it is proposed to use negative feedback.

Disclosure of invention

[009] The developers of the present technology have discovered at least one technical problem associated with known approaches.

[0010] The present technology relates to machine learning algorithms (in general) and to methods and systems for generating a training set for a machine learning algorithm (in particular). As described above, when training various supervised MLA algorithms such as a decision tree based MLA algorithm or a neural network (NN, Neural Network), training sets are provided to the MLA algorithm. Training sets are labeled examples illustrating both positive examples and negative examples. Positive examples when training the MLA algorithm correspond to the presence of the target output, negative examples when training the MLA algorithm correspond to the absence of the target output.

[0011] In general, positive examples are easy to form. For example, examples for search query and document pairs can be obtained using the search query logs of the search engine server, with user "clicks" being a good indicator of the relevance of a past document to a past user query. Forming negative examples can be more difficult.

[0012] The exemplary MLA algorithm learning system uses random negative examples on the assumption that most random examples will be drastically different from positive examples, and therefore they will be suitable negative examples. However, in practice, the effectiveness of negative examples is not 100% (closer to 80%).

[0013] In accordance with the first broad aspect of the present technology, a computer method for generating a training set for training a server of an MLA algorithm in an information retrieval system is implemented. The method is performed by the server and includes: extracting from the search log database on the server a plurality of search engine result pages (SERP, Search Engine Result Page), wherein a page of a plurality of SERP pages is formed in response to a search query from a plurality of search queries and contains a plurality documents that answer this search query; the formation by the server of a graph that maps a set of documents on a set of SERP pages and a set of queries in such a way that the graph node contains a search query from the set of search queries and is connected to the first subset of nodes, wherein each node from the first subset of nodes contains a document from the set of documents associated with this search query, and a node from the first subset of nodes is connected to the second subset of nodes, each node from the second subset of nodes contains search queries associated with the document of this node; obtaining an indication of a search query for generating a negative training example for training the MLA algorithm; identifying a node in the graph associated with this search query; and forming a training set to be used as a negative training example for the search query when training the MLA algorithm, wherein the training set contains the search query paired with a document located at another node that is a predetermined number of nodes away from that node.

[0014] In some non-limiting embodiments of the method, the predetermined number of nodes is a random number between five and nine.

[0015] In some non-limiting embodiments of the method, the predetermined number of nodes is an odd number of nodes.

[0016] In some non-limiting embodiments of the method, an association between a document and a search query is established when the document is included in the corresponding SERP page or when it is selected in response to displaying the corresponding SERP page.

[0017] In some non-limiting embodiments of the method, said other node is included in a third subset of nodes that are a predetermined number of nodes away from that other node, the method further comprising selecting that other node from the third subset of nodes.

[0018] In some non-limiting embodiments of the method, selecting another node from a third subset of nodes includes selecting that other node if that other node has the highest user interaction parameter indicative of one or more activities of one or more users in the third subset of nodes .

[0019] In some non-limiting embodiments of the method, a user interaction parameter for a document indicates the duration of hovering over that document and/or a "click" event on that document.

[0020] In some non-limiting embodiments of the method, the graph is a data tree.

[0021] In some non-limiting embodiments of the method, identifying a node in a graph associated with a search query includes identifying a root node.

[0022] In some non-limiting embodiments of the method, the graph is a mesh graph.

[0023] In accordance with another broad aspect of the present technology, a computerized method for generating a training set for training a server of an MLA algorithm in an information retrieval system is implemented. The method is performed by the server and includes: extracting from the search log database the first SERP page associated with the first request and the second SERP page associated with the second request, wherein the first SERP page contains the search result in the first position, and the second SERP page contains this search result in the second position; identification on the first page of the SERP of the search result with the highest engagement; and forming a training set to be used as negative training examples when training the MLA algorithm if the product of the first position and the second position exceeds a predetermined threshold, wherein the training set contains the second query paired with the most interacting search result.

[0024] In some non-limiting embodiments of the method, it further includes generating a second training set for use as a positive example, wherein the second training set contains the first query and the most interacting search result.

[0025] In some non-limiting embodiments of the method, the most engaged search result contains the search result on the first page of the SERP selected by the user.

[0026] In some non-limiting embodiments of the method, the search result is one of the plurality of interaction search results, and the longest interaction search result is one of the plurality of interaction search results having the longest interaction.

[0027] In some non-limiting embodiments of the method, the predetermined threshold value is fifty.

[0028] In accordance with another broad aspect of the present technology, a system is implemented for generating a training set for a server to train an MLA algorithm in an information retrieval system. The server comprises a processor capable of: retrieving a plurality of SERP pages from a search log database on the server, wherein a page of the plurality of SERP pages is formed in response to a search query from the plurality of search queries and contains a plurality of documents that respond to that search query; form a graph mapping a set of documents on a set of SERP pages and a set of queries such that a graph node contains a search query from the set of search queries and is connected to the first subset of nodes, each node from the first subset of nodes contains a document from the set of documents associated with it a search query, a node from the first subset of nodes is connected to a second subset of nodes, with each node from the second subset of nodes containing search queries associated with the document of this node; receive an indication of a search query for generating a negative training example for training the MLA algorithm; identify the node in the graph associated with this search query; and generate a training set for use as a negative training example for the search query when training the MLA algorithm, wherein the training set contains this search query paired with a document located at another node located at a distance of a predetermined number of nodes from this node.

[0029] In some non-limiting embodiments of the system, the predetermined number of nodes is an odd number of nodes.

[0030] In accordance with another broad aspect of the present technology, a system is implemented for generating a training set for a server to train an MLA algorithm in an information retrieval system. The server comprises a processor capable of: extracting from the search log database the first SERP page associated with the first query and the second SERP page associated with the second query, wherein the first SERP page contains the search result in the first position and the second SERP page contains the result search in the second position; identify the most engaged search result on the first page of the SERP; and generate a training set to be used as negative training examples when training the MLA algorithm if the product of the first position and the second position exceeds a predetermined threshold value, wherein the training set contains the second query paired with the most interacting search result.

[0031] In some non-limiting embodiments of the system, the processor is further configured to generate a second training set for use as a positive example, wherein the second training set contains the first query and the most interacting search result.

[0032] In the context of the present description, the term "server" means a computer program executed by appropriate hardware and capable of receiving requests via a network (for example, from electronic devices) and fulfill these requests or initiate their execution. The hardware may be one physical computer or one computer system, which is not essential to the present technology. In the present context, the expression "server" does not mean that every task (e.g. received command or request) or some specific task is received, executed or started by the same server (i.e. the same software and/or hardware ). This means that any number of software or hardware can receive, send, execute or initiate the execution of any task or request, or the results of any tasks or requests. All of these software and hardware may be a single server or multiple servers, both of which are meant by the expression "at least one server".

[0033] In the context of the present description, the term "electronic device" means any computer hardware capable of executing programs suitable for solving the task. Thus, some (non-limiting) examples of electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that in this context, a device functioning as an electronic device may also function as a server in relation to other electronic devices. The use of the term "electronic device" does not preclude the use of multiple electronic devices to receive, send, perform or initiate any task or request, or the results of any tasks or requests or steps of any method described herein.

[0034] In the context of the present description, the term "database" means any structured set of data, regardless of its specific structure, database management software or computer hardware for storing this data, using them, or providing them with use in another way. The database may reside on the same hardware as the process that stores or uses the information stored in the database, or the database may reside on separate hardware such as a dedicated server or multiple servers.

[0035] In the context of the present description, the expression "information" means information of any kind or type that can be stored in a database. Thus, the information includes, without limitation, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages etc.), documents, spreadsheets, etc.

[0036] In the context of the present description, the term "computer usable storage medium" means media of any kind and form, including RAM, ROM, disks (CD-ROMs, DVDs, floppy disks, hard disks, etc.), USB -drives, solid state drives, tape drives, etc.

[0037] In the context of the present description, unless explicitly stated otherwise, an information element can be referred to as the information element itself, as well as a pointer, link, hyperlink, or other indirect means by which the recipient of the data can find a place in the network, memory , database, or other machine-readable storage medium from which the information element can be retrieved. For example, a reference to a document may include the document itself (i.e., its contents), or the reference may be a unique document descriptor pointing to a file on a specific file system, or some other means to indicate a network location to the recipient of the data. , memory addresses, a table in a database, or other places where the file can be accessed. It should be apparent to one skilled in the art that the degree of precision required for such an indication depends on the amount of prior clarification regarding the interpretation of the information exchanged between the sender and recipient of the data. For example, if it is known before the communication between the sender and the recipient that the reference to the information element is the database key for the element in a certain table of the predefined database containing this information element, then in order to efficiently transmit this information element to the recipient, it is enough to send the database key data, even if the information element itself is not transmitted between the sender and receiver of the data.

[0038] In the context of the present description, the numerals "first", "second", "third", etc. are used only to indicate the difference between the nouns they refer to, but not to describe any specific relationship between these nouns. For example, it should be clear that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy, or classification of, in this case, servers, and that their use (by itself) does not imply a "second server" in every situation. Also, as occurs here in another context, referring to a "first" element and a "second" element does not preclude that the two elements may in fact be the same element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, and in other cases different software and/or hardware.

[0039] Each embodiment of the present technology relates to at least one of the above objectives and/or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology, associated with an attempt to achieve the above goal, may not meet this goal and/or may meet other goals not explicitly mentioned here.

[0040] Additional and/or alternative features, aspects, and advantages of implementations of the present technology are set forth in the following description, in the accompanying drawings, and in the appended claims.

Brief description of the drawings

[0041] The following description is provided for a better understanding of the present technology, as well as other aspects and their features, and should be used in conjunction with the attached drawings.

[0042] FIG. 1 is a diagram of a system implemented in accordance with non-limiting embodiments of the present technology.

[0043] FIG. 2 is a schematic representation of the query log.

[0044] FIG. 3 shows an example of a learning process for a machine learning algorithm used by the system shown in FIG. one.

[0045] FIG. 4 is a data tree generated according to a non-limiting embodiment of the present technology.

[0046] FIG. 5 shows two pages of search results implemented in accordance with a non-limiting embodiment of the present technology.

[0047] FIG. 6 shows a graph implemented in accordance with a non-limiting embodiment of the present technology.

[0048] FIG. 7 is a flowchart of a first method for generating a negative training example according to a non-limiting embodiment of the present technology.

[0049] FIG. 8 is a flowchart of a second method for generating a negative training example according to another non-limiting embodiment of the present technology.

Implementation of the invention

[0050] The examples and conventions provided herein are intended to provide a better understanding of the principles of the present technology, and not to limit its scope to such specifically given examples and terms. It should be obvious that specialists in the art are able to develop various methods and devices that are not explicitly described or shown, but implement the principles of the present technology within its essence and scope.

[0051] In addition, to facilitate a better understanding, the following description may contain simplified implementations of the present technology. Those skilled in the art will appreciate that various embodiments of the present technology can be significantly more complex.

[0052] In some cases, useful examples of modifications to the present technology are provided. They contribute to understanding, but also do not define the scope or boundaries of the present technology. The presented list of modifications is not exhaustive and a person skilled in the art can develop other modifications within the scope of this technology. In addition, if modifications are not described in some cases, this does not mean that they are impossible and / or that the description contains the only possible implementation of one or another element of the present technology.

[0053] Moreover, the description of the principles, aspects, and embodiments of the present technology, as well as their specific examples, is intended to cover their structural and functional equivalents, whether they are currently known or will be developed in the future. For example, it should be apparent to those skilled in the art that any block diagrams described herein correspond to conceptual representations of illustrative circuit diagrams that implement the principles of the present technology. It should also be obvious that any flowcharts, process diagrams, state transition diagrams, pseudocodes, etc. correspond to various processes that may be represented on a computer-readable physical storage medium and may be performed by a computer or processor, whether such a computer or processor is explicitly shown or not.

[0054] The functions of the various elements shown in the drawings, including any functional unit labeled "processor" or "graphics processing unit", may be implemented using specialized hardware, as well as using hardware capable of executing the corresponding software. If a processor is used, these functions may be performed by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor such as a central processing unit (CPU) or a specialized processor such as a graphics processing unit (GPU). In addition, explicit use of the term "processor" or "controller" should not be construed as referring solely to the hardware capable of executing the software and may refer to, but is not limited to, digital signal processor (DSP) hardware, network processor, ASIC (ASIC), Field Programmable Gate Array (FPGA), Software ROM, RAM, and NVRAM. Other general purpose and/or specialized hardware may also be contemplated.

[0055] Software modules, or simply modules that are intended to be implemented in software, may be represented here as any combination of flowchart elements or other elements indicating the steps of a process and/or containing a textual description. Such modules may be implemented in hardware, as shown or implied.

[0056] In the following, in light of the foregoing principles, some non-limiting examples are provided to illustrate various embodiments of aspects of the present technology.

[0057] As shown in FIG. 1, system 100 is an information retrieval system and includes an electronic device 102. Electronic device 102 is typically associated with a user (not shown) and may sometimes be referred to as a "client device".

[0058] In the context of the present description, unless expressly indicated otherwise, the term "electronic device" means any computer hardware capable of executing programs suitable for solving a corresponding task. Thus, some (non-limiting) examples of electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that in this context, a device functioning as an electronic device may also function as a server in relation to other electronic devices. The use of the term "electronic device" does not preclude the use of multiple client devices to receive, send, perform or initiate any task or request or the results of any tasks or requests or steps of any method described herein.

[0059] The electronic device 102 includes a non-volatile memory (memory) 104. The non-volatile memory 104 may include one or more storage media and generally provides storage space for computer instructions executed by the processor 106. For example, the non-volatile memory 104 may be implemented as computer-readable media, including ROMs, hard disk drives (HDDs), solid state drives (SSDs), and flash memory cards.

[0060] The electronic device 102 includes hardware and/or software and/or firmware (or a combination thereof), as is known in the art, to execute the search engine application 108. In general, through the search application 108, a user (not shown) can search, for example, on the Internet using a server-based (described below) search engine. To do this, the search engine application 108 includes a search query interface 110 and a search results interface 112.

[0061] There are no particular restrictions on the implementation of the search engine application 108. One example of a search engine application 108 may be implemented on a user's device (not shown) accessing a search engine associated website to access the search engine application 108. For example, the search engine application 108 can be accessed by entering a Uniform Resource Locator (URL) associated with the Yandex™ search engine at www.yandex.ru. It should be clearly understood that the search engine application 108 can be accessed using any other commercially available or proprietary search engine.

[0062] In alternative non-limiting embodiments of the present technology, the search engine application 108 may be implemented as a browser application on a portable device (such as a wireless communication device). For example, if the electronic device 102 is implemented as a portable device, such as a Samsung™ Galaxy™ S10, the Yandex™ browser application may be running on the electronic device 102. It should be clearly understood that any other commercially available or proprietary browser application may be used to implement non-limiting embodiments of the present technology.

[0063] In general, the electronic device 102 includes an input user interface (not shown), such as a keyboard, for receiving user input into, for example, a search query interface 110. The implementation of the input user interface is not particularly limited, but depends on the implementation of the electronic device 102. By way of non-limiting example only, in those embodiments of the present technology where the electronic device 102 is implemented as a wireless communication device (such like an iPhone™ smartphone), the input user interface can be implemented as a software keyboard (also called an on-screen or virtual keyboard). If the electronic device 102 is implemented as a personal computer, the input user interface may be implemented as a hardware keyboard.

[0064] While only one electronic device 102 is shown here, it should be understood that this is for ease of illustration and that system 100 may include more than one electronic device.

[0065] The electronic device 102 is connected to the communication network 114 using a communication line 116 . In some non-limiting embodiments of the present technology, communication network 114 may be the Internet. In other embodiments of the present technology, communication network 114 may be implemented differently, such as an arbitrary wide area network, local area network, private network, and so on.

[0066] The implementation of the link 116 is not subject to any particular restrictions, it depends on the implementation of the electronic device 102. By way of non-limiting example only, in those embodiments of the present technology where the electronic device 102 is implemented as a wireless device communication link (such as a smartphone), a communication link (not shown) may be implemented as a wireless link (such as a 3G network link, a 4G network link, Wireless Fidelity or WiFi®, Bluetooth® for short, etc.). In those examples where the electronic device 102 is implemented as a laptop, the link may be wireless (such as Wireless Fidelity or WiFi®, Bluetooth®, etc. for short) or wired (such as an Ethernet-based connection).

[0067] It should be obvious that the embodiments of the electronic device 102, communication line 116, and communication network 114 are for illustration purposes only. A person skilled in the art should be clear and other specific details of the implementation of the electronic device 102, communication line 116 and communication network 114. As such, the above examples are not intended to limit the scope of the present technology.

[0068] System 100 also includes a server 118 connected to a communication network 114. Server 118 may be implemented as a conventional computer server. In an exemplary embodiment of the present technology, server 118 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. Obviously, the server 118 may be implemented in any other suitable hardware and/or software and/or firmware, or a combination thereof. In the present non-limiting embodiment of the present technology, the server 118 is implemented as a single server. In other non-limiting embodiments of the present technology, the functions of the server 118 may be distributed among multiple servers.

[0069] The practical implementation of the server 118 is well known. In general, server 118 includes a communication interface (not shown) whose structure and functions allow communication with various entities (such as electronic device 102 and other devices that may be connected to communication network 114) via communication network 114.

[0070] Server 118 includes server memory 120, including one or more storage media, and generally provides storage space for computer program instructions executed by server processor 122. For example, server memory 120 may be implemented as a physical computer-readable storage medium, including ROM and/or RAM. Server memory 120 may also include one or more persistent storage devices such as hard disk drives (HDDs), solid state drives (SSDs), and flash memory cards.

[0071] In some non-limiting embodiments of the present technology, the server 118 may be operated by the same organization that provides the search application 108 described above. For example, server 118 may be operated by Yandex LLC (119021, Russia, Moscow, Lev Tolstoy St., 16). In other embodiments, server 118 may be operated by an organization other than the organization providing search application 108.

[0072] In some embodiments of the present technology, the server 118 provides access to the search engine 124 (such as Yandex.Search™) accessible via the communication network 114 using the search application 108. The method of practical implementation of the search engine 124 is known in the art and is not described in detail here. Suffice it to say that the search engine 124 is capable of performing one or more search queries on the Internet after a user (not shown) enters a search string into the search query interface 110 . The search engine 124 may then send a set of search results to the electronic device 102 for display to the user of the electronic device 102 via the search results interface 112 as a SERP page.

[0073] Server 118 is capable of generating a ranked list of search results, including general and vertical Internet search results. There are several algorithms for ranking search results that can be implemented by the server 118.

[0074] By way of non-limiting example, some well-known methods for ranking search results in terms of relevance to a user's search query are based on some or all of the following criteria: (a) the popularity of the search query or corresponding response in performing searches; (b) the number of results; (c) the presence of defining terms in the query (such as "images", "movies", "weather", etc.); (d) frequency of use by other users of this search query with defining terms; (e) the frequency with which other users performing similar searches selected a particular resource or particular vertical search results when the results were submitted using the SERP page. The search engine server 118 may calculate and assign a relevance score (based on the various criteria presented above) for each search result obtained for the user's search query, and also generate a SERP page where the search results are ranked according to their relevance scores. In this embodiment, the search engine server 118 may execute a plurality of machine learning algorithms for ranking documents and/or generate features for ranking documents.

[0075] In some non-limiting embodiments of the present technology, the server 118 is connected to the log database 126 via a dedicated channel (not indicated). Typically, the log base 126 can support a query log 128 .

[0076] The query log 128 is for recording searches performed using the server 118. In particular, the query log 128 stores search query terms (ie, search words) and associated search results. It should be noted that the query log 128 may be maintained in an anonymous form when search queries cannot be correlated with the users who submitted those search queries.

[0077] In particular, the request log 128 may contain a list of requests with relevant terms and information about documents, the list of which was transmitted by the server 118 in response to the corresponding request, a timestamp, and may also contain a list of users with their anonymous identifiers and corresponding documents, chosen by them after sending the request. In some embodiments of the present technology, the query log 128 may be updated each time a new search is performed on the server 118. In other embodiments, the request log 128 may be updated at predetermined times. In some embodiments of the invention, there may be multiple copies of the query log 128, each corresponding to the query log 128 at different points in time.

[0078] FIG. 2 schematically shows part 200 of query logs 128.

[0079] In the illustrated example, portion 200 contains five previously submitted search queries (namely, first search query 202, second search query 204, third search query 206, fourth search query 208, and fifth search query 210). Each of these five queries is associated with a corresponding SERP page (namely, the first SERP page 212, the second SERP page 214, the third SERP page 216, the fourth SERP page 218, and the fifth SERP page 220).

[0080] In some non-limiting embodiments of the present technology, these five search queries are received from electronic device 102. Alternatively, these five search queries may be received from one or more electronic devices.

[0081] Each of these five SERP pages contains multiple search results. Although in the embodiment presented here, each search result is shown as a combination of a letter and a number (eg, "A1"), this is done for ease of illustration, and it should be understood that each search result corresponds to some page on the Internet.

[0082] It should also be understood that when performing different search queries, the generated SERP pages may contain some common search result. For example, the search result A1 for the first search query 202 may be the same as the search result B4 for the second search query 204, despite the difference between the first search query 202 and the second search query 204.

[0083] In some non-limiting embodiments of the present technology, the query log 128 further includes user interaction parameters indicative of actions such as "clicks" on one or more documents on the SERP page.

[0084] Non-limiting examples of user interaction parameters stored in request log 128 include (but are not limited to):

- lose/win - was the search result in the response to the search query selected or not (click event);

- residence time - the time spent by the user on the search result before returning to the SERP page or until the search is terminated (for example, due to the fact that the user found the desired information);

- hover duration - the time for which the user hover over the search result;

- long/short "click": whether the user's action on the search result was long or short compared to the user's actions on other documents on this SERP page.

[0085] Of course, the above list is not exhaustive and may include other types of user actions without going beyond the boundaries of the present technology.

[0086] FIG. 1 also shows that the learning server 130 is connected to the server 118 via a dedicated channel. The learning server 130 may be implemented as a traditional computer server. In an exemplary embodiment of the present technology, the learning server 130 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. Obviously, the learning server 130 may be implemented using any other suitable hardware and/or application software and/or firmware, or a combination thereof. In the present non-limiting embodiment of the present technology, the learning server 130 is a single server. In other non-limiting embodiments of the present technology, the functions of the learning server 130 may be distributed among multiple servers. In the context of the present technology, the methods and systems described herein may be implemented on the learning server 130 in part. In some embodiments of the present technology, the learning server 130 is controlled and/or administered by a search engine operator. Alternatively, the learning server 130 may be managed and/or administered by another service provider. Although the learning server 130 is shown in the drawings as a server other than the server 118, it is contemplated that the functions of the learning server 130 described below can be performed by the server 118.

[0087] In general, the purpose of the training server 130 is to train one or more MLA algorithms used by the search service 124 by forming a training set.

[0088] The learning server 130 may, for example, train one or more MLA algorithms associated with the server 118 to improve general web search, vertical web search, recommendation, result prediction, and other applications. Training and improvement of the MLA algorithm can be performed at predetermined periods of time or when the search engine provider deems it necessary.

MLA Algorithm Training

[0089] FIG. 3 is a schematic illustration of the MLA algorithm 300 learning process. The MLA algorithm 300 is used by the search service 124.

[0090] For a better understanding of the basic concepts of the present technology, it should be understood that, in general, the training of the MLA algorithm 300 can be divided into the first and second stages. In the first step, training input 304 (described below) is generated. In the second step, the MLA algorithm 300 is trained using the training input 304. In addition, although the training steps of the MLA algorithm 300 are presented as being performed by the training server 130, it is contemplated that the MLA algorithm 300 may be trained by another entity, such as the server 118.

[0091] The generation of training input 304 is explained next with reference to two non-limiting embodiments.

First option

[0092] According to a non-limiting embodiment of the present technology, the generation of training input 304 begins with the training server 130 receiving the data packet 302 from the log database 126 . The data packet 302 contains a query log 128 . In some non-limiting embodiments of the present technology, the learning server 130 may retrieve the query log 128 periodically, such as once a day, once a week, or the like. In another non-limiting embodiment of the present technology, the learning server 130 may retrieve one or more query logs that match predetermined criteria, such as query logs with search queries completed in the last hour, last day, and the like.

[0093] Upon receipt of the query logs 128, the learning server 130 is able to parse a plurality of query logs 128 to identify one or more search results submitted in response to two or more search queries (described in more detail below).

[0094] FIG. 6 shows a mesh graph 600 generated by a learning server 130.

[0095] In the illustrated embodiment of graph 600, each of the circles corresponds to a unique search query, and each of the rectangles corresponds to a unique search result. For simplicity, all circles and rectangles are referred to here as nodes.

[0096] For example, the first node 602 corresponds to the first search query 202 associated with four nodes (in particular, the second node 604), each of which corresponds to the search result presented on the first page 212 SERP. Although the first node 602 is shown as being connected to four nodes, this is for illustration purposes only. It is contemplated that the first node 602 may be connected to more or less than four nodes. For example, the first node 602 may be associated with all or only a subset (eg, 10, 20, etc.) of the search results on the first SERP page 212. In another example, the first node 602 may be associated with a subset of the search results on the first SERP page 212 having the highest user experience parameter.

[0097] Let the second node 604 (corresponding to the search result A1) be associated with the first node 602. Then the second node 604 is connected to two other nodes, each of which corresponds to a search query different from the first search query 202 and associated with the SERP page containing the result A1 search. As an example, search result B4 can be assumed to match search result A1, then second node 604 is connected to third node 606 corresponding to second search query 204. In another example, search result C3 can be assumed to match search result A1 (i.e., second node 604), then the fourth node 608 matches the third search query 206.

[0098] In other words, the node in column 600 corresponds to a search query or search result obtained from a plurality of query logs 128. If the node is a search term, then it is connected to one or more nodes corresponding to the search result displayed on the SERP page generated in response to this search term.

[0099] If a node is a search result, then it is connected to one or more nodes that match the search query associated with the SERP page containing that search result.

[00100] Although the graph 600 is described as being generated based on only a portion 200 of the query logs 128, it should be understood that this is done for ease of explanation. It should be understood that graph 600 is formed from a plurality of query logs 128 with search results and search queries sent by a plurality of electronic devices.

[00101] After graph 600 is generated, training server 130 may generate training input 304 (see FIG. 3). In some non-limiting embodiments of the present technology, learning server 130 is capable of pairing a node that matches a search query with a node that is a predetermined number of nodes away from that node as a negative training example. For example, the learning server 130 may pair the first node 602 and one of the nodes that are an odd number of nodes away from the first node 602.

[00102] In some non-limiting embodiments of the present technology, the odd number of nodes is a random or predetermined number between five and nine. For example, let the odd number of nodes be seven. The learning server 130 can then find a node that is seven nodes away from the first node 602, such as a fifth node 610 corresponding to a particular search result (dashed lines show the trajectory from the first node 602). Obviously, the learning server 130 may pair the first node 602 with each node that is seven nodes away from the first node 602, or alternatively pair the first node 602 with the node that is seven nodes away and has the largest user interaction parameter.

[00103] In some non-limiting embodiments of the present technology, the learning server 130 can further generate a positive training example by pairing a node (such as the first node 602) with one of its directly connected nodes (such as the second node 604). The training input 304 then corresponds to a negative training example and a positive training example (if any).

[00104] Although graph 600 is formed by the training server as a mesh graph in the above example, this is not a limitation. In FIG. 4 shows a data tree 400 (or graph 400) generated by the learning server 130 after analysis.

[00105] The data tree 400 comprises a root node 412 at a first level 402 connected to two child nodes at a second level 404. Each of the child nodes is connected to two child nodes at a third level 406, and so on. In the example shown, data tree 400 is a five-level data tree (first level 402, second level 404, third level 406, fourth level 408, and fifth level 410), it should be understood that this is for illustration purposes only and that data tree 400 is an N-level tree. data. Moreover, although each node is shown as being connected to two corresponding child nodes, this is not a limitation and it is contemplated that each node can be connected to more or less child nodes, respectively.

[00106] In some non-limiting embodiments of the present technology, each of the nodes at odd levels of the data tree 400 corresponds to a search query retrieved from a plurality of query logs 128, and each of the nodes at even levels of the data tree 400 corresponds to a search result retrieved from a plurality of query logs 128 .

[00107] The following describes the relationship between nodes. As briefly discussed above, the learning server 130 is capable of parsing a plurality of query logs 128 to identify one or more search results common to one or more search queries. In other words, the data tree 400 illustrates the association of one or more search results with two or more search queries.

[00108] For example, let the root node 412 correspond to the first search query 202. The root node 412 is connected to two child nodes, namely the first node 414 and the second node 416, each of which is the result of a search on the first SERP page 212 ( namely, A1 and A3).

[00109] Each of the first node 414 and the second node 416 is connected to two respective child nodes. For example, the first node 414 is connected to the third node 418 and to the fourth node 420. Each of the third node 418 and the fourth node 420 corresponds to a search query different from the first search query 202 and associated with the SERP page containing the search result available in its parent node. (for the first node 414, the search result A1). Assuming, for example, that search result B4 corresponds to search result A1 (i.e., first node 414), then third node 418 corresponds to second search query 204. Assuming, in another example, that search result C3 corresponds to search result A1 (i.e., i.e. the first node 414), then the fourth node 420 matches the third search query 206.

[00110] In other words, a node in the data tree 400 corresponds to a search query or search result obtained from a plurality of query logs 128. If the node is a search term, then it is a child node for a parent node and a parent node for two child nodes, where the parent and two child nodes correspond to the search result displayed on the SERP page generated in response to this search term.

[00111] If a node is a search result, then it is a child node for a parent node and a parent node for two child nodes, where the parent and two child nodes match the search query associated with the SERP page containing that search result.

[00112] Although the formation of the data tree 400 is shown based on only a portion 200 of the query logs 128, it should be understood that this is done for ease of explanation. It should be understood that the data tree 400 is formed from a plurality of query logs 128 with search results and search queries sent by a plurality of electronic devices.

[00113] In some non-limiting embodiments of the present technology, each of the nodes at even levels (ie, nodes with search results) corresponds to the search results with the largest user interaction parameter. For example, taking the root node 412 (corresponding to the first search query 202) as an example, then the first node 414 and the second node 416 (two child nodes of the root node 412) correspond to the search result whose interaction parameter is maximum in the entire set of query logs 128. In other words, the learning server 130 is able to identify in the query logs 128 the two search results with the highest interaction parameter presented in response to the first search query 202.

[00114] It is also assumed here that each node can be connected to more than two child nodes. For example, the root node 412 may be connected to a number of child nodes, each of which corresponds to, for example, the top ten search results on the first SERP page 212 or all search results on the first SERP page 212.

[00115] After the data tree 400 is generated, the training server 130 generates training input 304 (see FIG. 3). In general, the training input 304 is a negative training case for training the MLA algorithm 300 in the second phase and contains one of the nodes corresponding to the search query paired with one of the nodes corresponding to the search result.

[00116] In some non-limiting embodiments of the present technology, the learning server 130 may combine the first search query 202 (included in the root node 412) with a node that is a predetermined number of nodes from the root node 412. In particular, the learning server 130 may combine the root node 412 is paired with one of the nodes that is an odd number of nodes away from the root node 412, or, in other words, located at an even level of the data tree. 400.

[00117] In some non-limiting embodiments of the present technology, the odd number of nodes is a random or predetermined number between five and nine. For example, let the odd number of nodes be five. The learning server 130 then has access to the sixth layer (not shown) and is able to randomly select a node corresponding to the search result. The choice of a node at the sixth level is not limited, for example, it can be chosen randomly. In some non-limiting embodiments of the present technology, the learning server 130 selects the node within the sixth layer associated with the largest user experience parameter.

[00118] Although in the above example, only the root node 412 is paired with the search result, it should be understood that other nodes can be paired to form additional negative training examples, for example, the third node 418 with the search query, included the number of nodes spaced by an odd number of nodes from the third node 418.

[00119] In addition, although only one data tree 400 is generated in the above example, it should be understood that this is done for ease of illustration and that the learning server 130 may generate multiple data trees with different search queries as parent nodes.

[00120] Finally, although only one negative training example is generated in the above example, this is not a limitation. It should be understood that the learning server 130 may generate more than one negative training example by pairing the root node 412 with one or more other sixth level search results. In another embodiment, the learning server 130 may also generate one or more negative training examples by selecting one or more search results from another level (e.g., from the eighth level (not shown)) to be paired with the root node 412.

[00121] In some non-limiting embodiments of the present technology, the learning server 130 further allows a positive training example to be generated by pairing a given node (such as the root node 412) and one of its directly connected child nodes (such as the first node 414). In this case, training input 304 corresponds to a negative training example and a positive training example (if any).

[00122]

Second option

[00123] Next, with reference to FIG. 3 describes a second non-limiting embodiment for generating training input 304.

[00124] Similar to the first embodiment, here the learning server 130 may receive a data packet 302 containing a plurality of query logs 128.

[00125] The learning server 130 then selects from the plurality of query logs 128 two SERP pages containing at least one common search result.

[00126] FIG. 5 shows the first SERP page 212 and the second SERP page 214.

[00127] For example, suppose the search result A4 corresponds to the search result B13. In the example shown, the search result A4 is located in the first position on the first page 212 of the SERP, and the search result B13 is located in the thirteenth position on the second page 214 of the SERP.

[00128] Let also the second search result (search result A2) on the first SERP page 212 have the highest user interaction parameter on the SERP page 212. In some non-limiting embodiments of the present technology, the search result with the most interaction corresponds to the search result selected by the user on the first page 212 of the SERP, or the search result with the longest user interaction time.

[00129] In some non-limiting embodiment of the present technology, the learning server 130 is able to determine the product of the position of the search result A4 and the position of the search result B13. In other words, the learning server 130 allows the positions of the overall search result on the first SERP page 212 and the second SERP page 214 to be multiplied. Considering that the search result A4 is the fourth search result on the first page 212 SERP, and the search result B13 is in the thirteenth position on the second page 214 SERP, this product is 52.

[00130] In some non-limiting embodiments of the present technology, if this product exceeds a predetermined threshold value (eg, 50), the training server 130 generates a negative training example by pairing the second search query 204 with the search result with the most interaction on the SERP page 212 ( which corresponds to the search result A2).

[00131] For example, if the predetermined threshold is 50, the training server 130 may pair the second search query 204 with the search result A2 as a negative training example. On the other hand, if the predetermined threshold is 60 (ie, greater than 52), the training server 130 does not pair the second search query 204 with the search result A2 (ie, does not generate a negative training example).

[00132] The assignment of the predetermined threshold is not limited, for example, it may be determined by the administrator of the learning server 130. From the above, it should be understood that when the product of two positions gives a higher value, this indicates that the document was ranked lower in the search results, therefore, even if this document is relevant to both queries, it is not the most relevant document for the search target for at least one of the search queries. In other words, non-limiting embodiments of the present technologies allow two search queries (and their associated SERP pages) to be found that are related (by a common document) but not very close (the common document is in low ranking positions).

[00133] In some non-limiting embodiments of the present technology, the learning server 130 further generates a positive training example by pairing the first search query 202 with the most engaged search result on the first SERP page 212 (i.e., search result A2).

[00134] In this case, training input 304 corresponds to a negative training example and a positive training example (if any).

[00135] Although the above description of generating a negative training example (and a positive training example) refers to two SERP pages (the first SERP page 212 and the second SERP page 214), it should be understood that this is done for ease of explanation and that training input 304 may contain more negative and positive training examples generated from other search results.

MLA training

[00136] Next, with reference to FIG. 3 describes training the MLA algorithm 300 using training inputs 304 generated in accordance with the first and second embodiments.

[00137] Training input 304 is entered into the MLA algorithm 300. The MLA algorithm 300 contains learning logic for determining a set of features associated with each negative learning example. There are no restrictions on the implementation of the MLA algorithm 300. In some non-limiting embodiments of the present technology, the MLA algorithm 300 is implemented as a neural network.

[00138] In particular, given that each of the training examples is a negative training example, the MLA algorithm 300 can determine which set of features indicates a low relevance of the search result to the search query.

[00139] Accordingly, the MLA algorithm 300 may generate an intended function when it is executed by the server 118 as part of a search service 124 capable of assigning a ranking parameter to a search result in response to a received usage stage search query based on a set of features associated with the search query and with the result. search.

[00140] Although only one example of training the MLA algorithm 300 is presented here, this is done for ease of illustration. It should be clearly understood that the training of the MLA algorithm 300 is performed iteratively using a plurality of both negative and positive training examples.

[00141] Shown in FIG. 4, the architecture and the examples described above make it possible to carry out the above method implemented on a computer. In FIG. 7 is a flow diagram of a method 700 for generating a negative training example. Method 700 is performed in accordance with non-limiting embodiments of the present technology. Method 700 may be performed by learning server 130.

[00142] Step 702: Retrieving a plurality of SERP pages from the search log database on the server, wherein a page of the plurality of SERP pages is generated in response to a search query from the plurality of search queries and contains a plurality of documents responding to that search query.

[00143] The method 700 begins at step 702 where the learning server 130 retrieves the data packet 302 from the log base 126. The data packet 302 contains a plurality of query logs 128 .

[00144] Query logs 128 are designed to record search queries made using server 118. In particular, query logs 128 store search query terms (ie, search words) and associated search results. For example, each query log included in the plurality of query logs 128 is associated with a respective electronic device (such as electronic device 102). It should be noted that the query logs 128 may be maintained in an anonymized form, and search queries cannot be associated with the users who submitted those search queries.

[00145] Step 704: Server generates a graph in which a set of documents in a set of SERP pages is mapped to a set of queries, wherein the graph node contains a search query from the set of search queries and is connected to a first subset of nodes, each node from the first subset of nodes contains a document from of the set of documents associated with this search query, a node from the first subset of nodes is connected to the second subset of nodes, and each node from the second subset of nodes contains search queries associated with the document of this node.

[00146] At step 704, after receiving the data packet 302, the training server generates a data tree 400 or graph 600.

[00147] For example, in column 600, each of the circles corresponds to a unique search query, and each of the boxes corresponds to a unique search result. For simplicity, all circles and rectangles are referred to here as nodes.

[00148] For example, the first node 602 corresponds to the first search query 202 associated with four nodes (in particular, the second node 604), each of which corresponds to the search result presented on the first page 212 SERP. Although the first node 602 is shown as being connected to four nodes, this is for illustration purposes only. It is contemplated that the first node 602 may be connected to more or less than four nodes. For example, the first node 602 may be associated with all or only a subset (eg, 10, 20, etc.) of the search results on the first SERP page 212. In another example, the first node 602 may be associated with a subset of the search results on the first SERP page 212 having the highest user experience parameter.

[00149] Let the second node 604 (corresponding to the search result A1) be connected to the first node 602. Then the second node 604 is connected to two other nodes, each of which corresponds to a search query different from the first search query 202 and associated with the SERP page containing the result A1 search. As an example, search result B4 can be assumed to match search result A1, then second node 604 is connected to third node 606 corresponding to second search query 204. In another example, search result C3 can be assumed to match search result A1 (i.e., second node 604), then the fourth node 608 matches the third search query 206.

[00150] Step 706: Obtaining an indication of a search query for generating a negative training example for training the MLA algorithm.

[00151] In step 706, the learning server 130 selects a node in column 600 that matches a particular search query. For example, the learning server 130 may select the root node 412.

[00152] Step 708: identifying the node in the graph associated with the search query.

[00153] In step 708, the learning server 130 selects a node in graph 600 that is a given number of nodes away from the root node 412.

[00154] Step 710: Generate a training set to use as a negative training example for a search query when training the MLA algorithm, wherein the training set contains the search query paired with a document located at another node that is a predetermined number of nodes away from this node.

[00155] At step 710, the learning server 130 generates a negative training example corresponding to the search query of the root node 412 with a node that is a given number of nodes away from it.

[00156] Method 700 then ends or returns to step 702.

[00157] Shown in FIG. 5, the architecture and the examples described above make it possible to carry out the above method implemented on a computer. In FIG. 8 is a flow diagram of a method 800 for generating a negative training example. Method 800 is performed in accordance with non-limiting embodiments of the present technology. Method 800 may be performed by learning server 130.

[00158] Step 802: Retrieve from the search log database the first SERP page associated with the first query and the second SERP page associated with the second query, where the first SERP page contains the search result in the first position and the second SERP page contains the result search in the second position.

[00159] The method 800 begins at step 802, where the learning server 130 retrieves a plurality of logs 128 from the log base 126.

[00160] The learning server 130 then selects from the plurality of query logs 128 two SERP pages containing at least one common search result, such as the first SERP page 212 and the second SERP page 214.

[00161] For example, suppose the search result A4 corresponds to the search result B13. In the example shown, the search result A4 is located in the fourth position on the first page 212 SERP, and the search result B13 is located in the thirteenth position on the second page 214 SERP.

[00162] Step 804: Identification on the first page of the SERP of the search result with the most engagement.

[00163] In step 804, the learning server 130 determines on the first SERP page 212 the search result with the highest user interaction parameter.

[00164] Step 806: Generating a training set to use as negative training examples when training the MLA algorithm if the product of the first position and the second position exceeds a predetermined threshold, this training set containing the second query paired with the most interacting search result .

[00165] In step 806, the learning server 130 determines the product of the position of the search result A4 and the position of the search result B13. In other words, the learning server 130 is able to multiply the positions of the overall search result on the first SERP page 212 and on the second SERP page 214. Considering that the search result A4 is the fourth search result on the first page 212 SERP, and the search result B13 is in the thirteenth position on the second page 214 SERP, this product is 52.

[00166] In some non-limiting embodiments of the present technology, if this product exceeds a predetermined threshold, the learning server 130 generates a negative training example by pairing the second search query 204 and the most engaged search result on the first SERP page 212 (corresponding to result A2 search).

[00167] Method 800 then ends or returns to step 802.

[00168] Those skilled in the art should appreciate that at least some embodiments of the present technology are directed to expanding the arsenal of technical means to solve a specific technical problem specific to the traditional method of generating negative training examples.

[00169] It is obvious that not all of the technical effects mentioned in this description should be present in every embodiment of the present technology. For example, there may be embodiments of the present technology where the user does not receive some of these technical effects, or other implementations where the user receives other technical effects, or when there is no technical effect.

[00170] Modifications and improvements to the embodiments of the present technology described above may be apparent to those skilled in the art. The foregoing description is given by way of example only, and is not intended to limit the scope of the invention. Therefore, the scope of the present technology is limited solely by the scope of the appended claims.

[00171] Although the embodiments described above are given with reference to specific steps performed in a certain order, it should be understood that these steps can be combined, separated, or that their order can be changed without deviating from the present technology. Accordingly, the order and grouping of steps is not limiting to the present technology.

Claims (40)

1. A computer method for forming a training set for training a machine learning algorithm (MLA) implemented in an information retrieval system and performed by a server, wherein the method is performed by the server and includes: - extracting from the search log database on the server a plurality of search engine result pages (SERPs), wherein a page of the plurality of SERP pages is formed in response to a search query from a plurality of search queries and contains a plurality of documents that respond to this search query; - the formation by the server of a graph in which a set of documents on a set of SERP pages and a set of requests are compared in such a way that: the graph node contains a search query from a plurality of search queries and is connected to a first subset of nodes, wherein each node from the first subset of nodes contains a document from a plurality of documents associated with this search query; and - a node from the first subset of nodes is connected to the second subset of nodes, with each node from the second subset of nodes containing search queries associated with the document of this node; - receiving an indication of a search query to form a negative training example for training the MLA algorithm; - identification of the node in the graph associated with this search query; and - formation of a training set for use as a negative training example for a search query when training an MLA algorithm containing a search query paired with a document located at another node located at a distance of a predetermined number of nodes from this node. 2. The method according to claim. 1, characterized in that the predetermined number of nodes is an odd number of nodes. 3. The method according to claim 2, characterized in that the odd number of nodes is a random number from five to nine. 4. The method according to claim 2, characterized in that the odd number of nodes is a predetermined number from five to nine. 5. The method according to claim 1, characterized in that the document is associated with the search query based on the inclusion of this document in the corresponding SERP page or the selection of this document in response to the display of the corresponding SERP page. 6. The method according to claim. 1, characterized in that the other node is included in the third subset of nodes located at a distance of a predetermined number of nodes from this other node, and the method includes selecting this other node from the third subset of nodes. 7. The method of claim 6, wherein selecting another node from the third subset of nodes includes selecting that other node if it has the highest user interaction parameter indicative of one or more types of actions performed by one or more users in third subset of nodes. 8. The method according to claim 7, characterized in that the user interaction parameter for the document indicates the duration of hovering over this document and/or a "click" event on this document. 9. The method of claim. 1, characterized in that the method further includes generating a second training set for use as a positive example, while the second training set contains a search query paired with a document located at another node directly connected to this node. 10. The method according to claim 9, characterized in that the graph is a data tree. 11. The method of claim. 10, characterized in that the identification of the node in the graph associated with the search query includes the identification of the root node of the data tree. 12. The method according to claim 9, characterized in that the graph is a grid graph. 13. A computer method for forming a training set for training a machine learning algorithm (MLA) implemented in an information retrieval system and performed by a server, performed by a server and including: - obtaining from the search log database the first search engine results page (SERP) associated with the first query and the second SERP page associated with the second query, wherein the first SERP page contains the search result in the first position, and the second SERP page contains this result search in the second position; - identification on the first page of the SERP search result with the greatest engagement; - formation of a training set to be used as negative training examples when training the MLA algorithm, if the product of the first position and the second position exceeds a predetermined threshold value, while the training set contains the second query paired with the search result with the greatest interaction. 14. The method of claim 13, characterized in that it includes generating a second training set for use as a positive example, wherein the second training set contains the first query and the search result with the highest interaction. 15. The method of claim 13, wherein the most engaged search result contains the selected search result on the first page of the SERP. 16. The method of claim 14, wherein the search result is one of the plurality of interaction search results, and the longest interaction search result is one of the plurality of interaction search results having the longest interaction. 17. The method according to claim 13, characterized in that the predetermined threshold is 50. 18. A system for generating a training set for training a machine learning algorithm (MLA) implemented in an information retrieval system and executed by a server, wherein the server contains a processor configured to: - extracting from the search log database on the server a plurality of search engine result pages (SERPs), wherein a page of the plurality of SERP pages is formed in response to a search query from a plurality of search queries and contains a plurality of documents that respond to this search query; - formation of a graph in which a set of documents on a set of SERP pages and a set of queries are compared in such a way that: the graph node contains a search query from a plurality of search queries and is connected to a first subset of nodes, wherein each node from the first subset of nodes contains a document from a plurality of documents associated with this search query; and - a node from the first subset of nodes is connected to the second subset of nodes, with each node from the second subset of nodes containing search queries associated with the document of this node; - obtaining an indication of the search query to form a negative training example for training the MLA algorithm; - identifying the node in the graph associated with this search query; and - forming a training set for use as a negative training example for a search query when training an MLA algorithm containing a search query paired with a document located at another node located at a distance of a predetermined number of nodes from this node. 19. The system according to claim 18, characterized in that the predetermined number of nodes is an odd number of nodes. 20. A system for generating a training set for training a machine learning algorithm (MLA) implemented in an information retrieval system and executed by a server, wherein the server contains a processor configured to: - obtaining from the search log database the first search engine results page (SERP) associated with the first query and the second SERP page associated with the second query, whereby the first SERP page contains the search result in the first position, and the second SERP page contains this result search in the second position; - identification on the first page of the SERP search result with the greatest engagement; - forming a training set for use as negative training examples when training the MLA algorithm, if the product of the first position and the second position exceeds a predetermined threshold value, while the training set contains the second query paired with the search result with the greatest interaction.

Images