RU2775815C2 - Methods and servers for ranking digital documents in response to a query - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: group of inventions relates to search engine technologies and can be used for ranking digital documents for a query. A search engine associated with an inverted index is stored on a server. The method includes accessing the inverted index to retrieve query-independent data for a first document-term pair and a second document-term pair, wherein the query-independent data indicates (i) a term-dependent occurrence of the first term in the document and (ii) a term-dependent occurrence of the second term in the document. The method includes generating, using the query-independent data, a query-dependent feature indicating a group occurrence of the first term with the second term in the document. The method includes generating a ranking feature for the document based on at least the first term, the second term and the query-dependent feature, and ranking the document based on at least the ranking feature.

EFFECT: increase in the performance speed.

20 cl, 7 dwg

Description

Technical field

[01] The present technology relates to search engine technologies. In particular, the present technology is directed to methods and servers for ranking digital documents in response to a query.

State of the art

[02] The Internet provides access to a wide range of resources, such as video files, image files, audio files, or web pages. Search engines are used to find these resources. For example, digital images that satisfy a user's information needs may be identified by a search engine in response to receiving a user query submitted by the user. User requests can consist of one or more terms (term, word) of the request. The search engine selects and ranks search results based on their relevance to the user's query and their importance or quality compared to other search results, and provides the user with the best search results.

[03] Search engine systems contain a component called an inverted index, which includes a large number of document lists (posting lists) associated with the corresponding search terms, and stores indications of documents that contain the corresponding search terms. Such data structures reduce the time, memory, and processing resources for performing searches.

[04] During the search, the system extracts information related to terms (from the query used) from the inverted index and is configured to use this information to rank documents in response to the query used. The data stored in an inverted index is query independent and term dependent. Processing real-time extracted data for document ranking is challenging due to the trade-off between ranking quality and the required processing time to do so.

[05] U.S. Patent No. 10,417,687, entitled "Generating modified query to identify similar items in a data store" and published September 17, 2019, discloses techniques for identifying similar items in a data store by generating a modified query from a user query and from information about a specific item stored in the data store. Predefined descriptive terms may be designated as useful for identifying elements, and these terms may be located in the keyword data for a particular element. Element correlations can also be identified with respect to the element's category designation. The modified query can be generated based on the predefined descriptive terms in the keyword data for the element and element correlations.

The essence of the invention

[06] Embodiments of the present technology were developed based on developers' identification of at least one technical problem related to prior art approaches to object classification.

[07] In at least one broad aspect of the present technology, a server is provided that runs a plurality of computer-implemented algorithms, referred to herein as a "search engine," which is broadly configured to provide search results in response to a query submitted by a user of an electronic device.

[08] Broadly speaking, a web search engine is a software system for performing web searches. Search results are usually presented in a ranked format, that is, search results are presented in a list ranked based on relevance to the query. The information is usually displayed to the user through a search engine results page (SERP). Search results may be digital documents such as, but not limited to, web pages, images, videos, infographics, articles, research papers, and other types of digital files.

[09] In at least one aspect of the present technology, a server is communicatively coupled to a storage device storing a structured set of data, referred to herein as an "inverted index". In a broad sense, an inverted index is a data structure that is used as a component of search engine indexing algorithms. For example, a direct index can first be generated that stores word lists for each document - then the direct index can be "inverted" in some sense so that it stores lists of documents for each word. A direct index query may require successive iterations over each document and each word to check for a matching document. The time, memory, and processing resources to complete such a request are not always technically realistic. On the contrary, one of the advantages of an inverted index is that it takes comparatively less time, memory, and processing resources to query such a data structure.

[10] In at least one broad aspect of the present technology, an inverted index stores data for respective term-document pairs. The data stored in association with the corresponding term-document (DT) pairs is assumed to contain "query-independent" data. In at least some embodiments of the present technology, query-independent data for a given pair of TDs may indicate a term-dependent occurrence of the corresponding term in the content associated with the corresponding document. We can say that an inverted index stores "term-dependent" data for the corresponding terms, as opposed to data that depends on more than one term from the query.

[11] In at least some aspects of the present technology, the developers of the present technology have developed methods and servers for using query-independent data stored in an inverted index to generate in real time during a given search performed in response to a submitted query, one or more "query-dependent features" based on the extracted query-independent data for one or more terms from the submitted query. In at least some embodiments of the present technology, it can be said that data previously stored in an inverted index can be retrieved during a real-time search and processed by a variety of computer-implemented algorithms, which are referred to here as a "dynamic feature generator", which in broadly configured to generate one or more dynamic features, where each dynamic feature depends on more than one term from the query.

[12] In at least one embodiment, the server-executed dynamic feature generator is configured to (i) use query-independent data associated with corresponding terms to (ii) generate a query-specific feature indicative of a "group occurrence" of at least a pair of terms from the query into the content associated with the corresponding document. It is assumed that the server is configured to use a query-specific feature to rank a corresponding document among a set of potentially relevant documents in response to a query.

[13] In a first broad aspect of the present technology, a method for ranking digital documents in response to a query is provided. The digital documents are potentially relevant to the query containing the first and second terms. The request was sent by the user of an electronic device that is connected with the ability to communicate with the server hosting the search engine. The search engine is associated with an inverted index that stores information associated with document-term (DT) pairs. The method is executed on the server. The method comprises, for a given document from a plurality of potentially relevant documents, accessing the inverted index by the server to retrieve query-independent data for the first pair of DTs and the second pair of DTs. The first DT pair has the given document and the first term. The second DT pair has the given document and the second term. The query-independent data indicates (i) a term-dependent occurrence of the first term in the content associated with the given document, and (ii) a term-dependent occurrence of the second term in the content associated with the given document. The method comprises, for a given document from a plurality of potentially relevant documents, generating a query-dependent feature by the server using query-independent data retrieved for the first DT pair and the second DT pair. The query-dependent attribute indicates the group occurrence of the first term with the second term in the content associated with this document. The method comprises, for a given document from a set of potentially relevant documents, generating by the server a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature. The method comprises ranking by the server a given document from a plurality of potentially relevant documents based on at least a ranking feature.

[14] In some embodiments of the method, generating a ranking feature for a given document is performed by a neural network (NN).

[15] In some embodiments of the method, the method further comprises training by the NN server to generate a ranking feature. The training of the NN comprises the server generating a training set for the training document-query (DQ) pair to be used during a given iteration of training the NN. A training pair DQ has a training query and a training document. The generation comprises generating by the server a plurality of training term embeddings based on the corresponding terms from the training query. The generation comprises accessing by the server an inverted index associated with the search engine to retrieve a plurality of query-independent datasets associated with corresponding pairs from the plurality of training pairs DT. The given one of the set of training pairs DT includes the training document and the corresponding one of the set of terms from the training query. The generation comprises generating by the server a plurality of training feature vectors for a plurality of training pairs DT using a plurality of query-independent datasets. Training the NN comprises, during a given iteration of training the NN, inputting by the server into the NN said set of training term embeddings and said set of training feature vectors to generate a predicted ranking feature for the training pair DQ. NN training comprises, during a given iteration of NN training, setting the NN by the server based on a comparison between the label and the predicted ranking feature, such that the NN generates, for a given used DQ pair, an appropriate predicted ranking feature that indicates the relevance of the corresponding used document to the corresponding used query.

[16] In some embodiments of the method, query-independent data has been stored in the inverted index prior to receiving a query from the electronic device, and the query-dependent feature is generated after receiving a query from the electronic device.

[17] In some embodiments of the method, a query-dependent feature is generated using real-time query-independent data during a search engine document ranking procedure.

[18] In some embodiments of the method, the ranking is performed using a machine learning algorithm (MLA) based on a decision tree configured to rank a set of potentially relevant documents based on their relevance to a query.

[19] In some embodiments of the method, the method further comprises determining a similar term for a given one of the plurality of terms. When accessing the inverted index to retrieve query-independent data, the retrieved query-independent data contains the query-independent data for a third pair of DTs, the third pair of DTs having the given document and said similar term.

[20] In some embodiments of the method, the inverted index access is for further retrieval of query-independent data based on the content associated with the first DT pair and the second DT pair. Content-based query-independent data points to the textual context of the corresponding term in the content associated with the given document.

[21] In some embodiments of the method, the term-dependent occurrence of the first term contains at least one of: one or more positions of the first term in the header associated with this document; one or more positions of the first term in the URL associated with this document; and one or more positions of the first term in the body of this document.

[22] In some embodiments of the method, the grouping of the first term with the second term in the content associated with a given document contains at least one of: the number of times the second term from the query is included in addition to the first term in the header associated with the given document; the number of times the second term from the query is included in addition to the first term in the URL associated with the document; and the number of times the second term from the query is included in addition to the first term in the body of the given document.

[23] In a second broad aspect of the present technology, a server is provided for ranking digital documents in response to a request. The digital documents are potentially relevant to the query containing the first and second terms. The request was sent by the user of an electronic device that is connected with the ability to communicate with the server hosting the search engine. The search engine is associated with an inverted index that stores information associated with document-term (DT) pairs. The server is configured to, for a given document from a plurality of potentially relevant documents, access the inverted index to retrieve query-independent data for the first DT pair and the second DT pair. The first DT pair has the given document and the first term, and the second DT pair has the given document and the second term. The query-independent data indicates (i) a term-dependent occurrence of the first term in the content associated with the given document, and (ii) a term-dependent occurrence of the second term in the content associated with the given document. The server is configured to, for a given document from a plurality of potentially relevant documents, generate a query-dependent feature using the query-independent data retrieved for the first DT pair and the second DT pair. The query-dependent attribute indicates the group occurrence of the first term with the second term in the content associated with this document. The server is configured to, for a given document from a plurality of potentially relevant documents, generate a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature. The server is configured to rank a given document from a plurality of potentially relevant documents based on at least a ranking feature.

[24] In some server implementations, the server uses a neural network to generate a ranking feature for a given document.

[25] In some server embodiments, the server is further configured to teach the NN to generate a ranking feature. The server is configured to generate a training set for a training document-query (DQ) pair to be used during a given training iteration of the NN. A training pair DQ has a training query and a training document. To generate the training set, the server is configured to generate a plurality of training term embeddings based on the corresponding terms from the training query. To generate the training set, the server is configured to access an inverted index associated with the search engine to retrieve a plurality of query-independent datasets associated with matching pairs from the plurality of training pairs DT. The given one of the set of training pairs DT includes the training document and the corresponding one of the set of terms from the training query. To generate a training set, the server is configured to generate a plurality of training feature vectors for a plurality of DT training pairs using a plurality of query-independent datasets. The server is configured to, during a given training iteration of the NN, input by the server into the NN said set of training term embeddings and said set of learning feature vectors to generate a predicted ranking feature for the training pair DQ. The server is configured to, during a given iteration of training the NN, tune the NN based on a comparison between the label and the predicted ranking feature such that the NN generates for a given used DQ pair an appropriate predicted ranking feature that indicates the relevance of the corresponding used document to the corresponding used query.

[26] In some server embodiments, query-independent data has been stored in an inverted index prior to receiving a query from the electronic device, and the query-dependent feature is generated after receiving a query from the electronic device.

[27] In some server embodiments, a query-dependent feature is generated using real-time query-independent data during a search engine document ranking procedure.

[28] In some server embodiments, the server is configured to rank using a machine learning algorithm (MLA) based on a decision tree configured to rank a set of potentially relevant documents based on their relevance to a query.

[29] In some server embodiments, the server is further configured to determine a similar term for a given one of the plurality of terms. When accessing the inverted index to retrieve query-independent data, the retrieved query-independent data contains the query-independent data for a third pair of DTs, the third pair of DTs having the given document and said similar term.

[30] In some server embodiments, the server is configured to access the inverted index to further retrieve query-independent data based on the content associated with the first DT pair and the second DT pair. Content-based query-independent data points to the textual context of the corresponding term in the content associated with the given document.

[31] In some server implementations, a term-dependent occurrence of a first term contains at least one of: one or more first term positions in a header associated with a given document; one or more positions of the first term in the URL associated with this document; and one or more positions of the first term in the body of this document.

[32] In some server implementations, the grouping of the first term with the second term in the content associated with a given document includes at least one of: the number of times the second term from the request is included in addition to the first term in the header associated with the with this document the number of times the second term from the query is included in addition to the first term in the URL associated with the document; and the number of times the second term from the query is included in addition to the first term in the body of the given document.

[33] As used herein, unless expressly stated otherwise, "electronic device", "electronic device", "server", "remote server", and "computer system" are any hardware and/or software suitable for the corresponding task. Thus, some non-limiting examples of hardware and/or software include computers (servers, desktops, laptops, netbooks, etc.), smartphones, tablets, network equipment (routers, switches, gateways, etc.). ) and/or combinations thereof.

[34] As used herein, unless specifically noted otherwise, the terms "computer-readable media" and "memory" are intended to include media of any nature and kind, non-limiting examples of which include RAM, ROM, disks (CD-ROM , DVDs, floppy disks, hard drives, etc.), USB keys, flash memory cards, solid state drives and tape drives.

[35] As used herein, unless expressly provided otherwise, an "indication" of an information element may be the information element itself, or a pointer, link, hyperlink, or other indirect mechanism that allows the recipient of such indication to locate a location in a network, memory, database, or other a computer-readable medium from which the information element can be retrieved. For example, a document designation may include the document itself (i.e., its contents), or it may be a unique document descriptor identifying a file with respect to some particular file system, or some other means of directing the recipient of such designation to a network location, a base table data or other location where the file can be accessed. One skilled in the art will appreciate that the degree of precision required in an indication depends on the degree of any prior understanding of what interpretation will be provided of the information exchanged in the interaction between the sender and recipient of such indication. For example, if it is understood that, prior to communication between the sender and recipient, the indication of the information element will be in the form of a database key to be recorded in some particular table of the predefined database containing the information element, then sending the database key is all that is required to efficiently convey the information element. element to the recipient, even if the information element itself was not transmitted in the interaction between the sender and recipient of such an indication.

[36] In the context of the present description, unless specifically provided otherwise, the words "first", "second", "third", etc. They were used as adjectives only for the purpose of providing a distinction between the nouns they modify from one to another, and not for the purpose of describing any specific relationship between these nouns. Thus, for example, it should be understood that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy or ranking (for example) of such / between such servers, nor does their use ( by itself) does not mean that some "second server" must necessarily exist in any given situation. Also, as discussed elsewhere in this document, reference to a "first" element and a "second" element does not exclude that the two elements are in fact the same real world element. Thus, for example, in some cases the "first" server and the "second" server may be the same software and/or hardware, in other cases they may be different software and/or hardware.

[37] Each of the implementations of the present technology has at least one of the above aspects and/or goals, but not necessarily all of them. It should be understood that some aspects of the present technology that have arisen in an attempt to achieve the aforementioned goal may not satisfy this goal and/or satisfy other goals that are not explicitly described in this document. Additional and/or alternative features, aspects and advantages of implementations of the present technology will become apparent from the following description, the accompanying drawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[38] For a better understanding of the present technology, as well as other aspects and additional features thereof, reference is made to the following description, which is to be used in conjunction with the accompanying drawings, in which:

[39] FIG. 1 is a diagram of a system implemented in accordance with non-limiting embodiments of the present technology.

[40] FIG. 2 depicts a view of the data stored in the database subsystem of the system shown in FIG. 1, in accordance with non-limiting embodiments of the present technology.

[41] FIG. 3 depicts a representation of a learning iteration of the ranking feature generator of the system shown in FIG. 1, in accordance with non-limiting embodiments of the present technology.

[42] FIG. 4 is a representation of how the dynamic feature generator of the system shown in FIG. 1 is configured to generate query-dependent data based on query-independent data obtained from the inverted system index in FIG. 1, in accordance with non-limiting embodiments of the present technology.

[43] FIG. 5 is a representation of the usage phase of the ranking feature generator of the system shown in FIG. 1, in accordance with non-limiting embodiments of the present technology.

[44] FIG. 6 is a representation of how the ranking model of the system shown in FIG. 1 is configured to generate a ranked list of documents in accordance with non-limiting embodiments of the present technology.

[45] FIG. 7 is a schematic representation of a method performed by a server of the system shown in FIG. 1, in accordance with non-limiting embodiments of the present technology.

Detailed description

[46] The examples and conventions used herein are intended primarily to help the reader understand the principles of this technology, and not to limit its scope to such specific examples and terms. It should be understood that those skilled in the art will be able to develop various mechanisms that, while not explicitly described herein, nevertheless embody the principles of the present technology and are included within its spirit and scope.

[47] In addition, the following description may describe implementations of the present technology in a relatively simplified manner for purposes of ease of understanding. Those skilled in the art will appreciate that various implementations of the present technology may be more complex.

[48] In some cases, examples of modifications to the present technology that are considered useful may also be set forth. This is only to promote understanding and, again, not to rigorously define the scope or delineate the boundaries of the present technology. These modifications are not an exhaustive list, and other modifications may be made by one skilled in the art while still remaining within the scope of the present technology. In addition, cases where examples of modifications are not given should not be interpreted to mean that no modifications can be made and/or that what is described is the only way to implement such an element of the present technology.

[49] In addition, all statements contained herein that identify principles, aspects, and implementations of the present technology, as well as specific examples thereof, are intended to cover both structural and functional equivalents, whether or not they are currently known. time or will be developed in the future. Thus, for example, those skilled in the art should understand that any block diagrams herein represent conceptual views of an illustrative circuit embodying the principles of the present technology. Likewise, it should be understood that any and all flowcharts, sequence diagrams, state transition diagrams, pseudo-codes, and the like represent various processes that may per se be represented on computer-readable media and executed by a computer or processor, whether or not whether such a computer or processor is explicitly shown or not.

[50] The functions of the various elements shown in the figures, including any functional block labeled "processor" or "graphics processing unit", can be provided using specialized hardware, as well as hardware capable of executing software and associated with proper software. When provided by a processor, the functions may be provided by one dedicated processor, one shared processor, or multiple individual processors, some of which may be shared. In some embodiments of the present technology, the processor may be a general purpose processor such as a central processing unit (CPU) or a processor dedicated to a specific purpose such as a graphics processing unit (GPU). In addition, explicit use of "processor" or "controller" should not be construed as referring solely to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), read-only memory (ROM) for software storage, random access memory (RAM), and non-volatile storage. Other hardware, traditional and/or specialized, may also be included.

[51] Software modules, or simply modules, which may be referred to as software, may be represented herein as any combination of flowchart elements or other elements indicating the execution of process steps and/or a textual description. Such modules may be executed by hardware that is explicitly or implicitly shown.

[52] Given these fundamentals, consider some non-limiting examples to illustrate various implementations of aspects of the present technology.

[53] With reference to FIG. 1 illustrates a schematic representation of system 100, system 100 being suitable for implementing non-limiting embodiments of the present technology. It should be clearly understood that the depicted system 100 is only an illustrative implementation of the present technology. Thus, the following description is only intended to be used as a description of illustrative examples of the present technology.

[54] In the illustrated example, system 100 may be used to provide one or more online services to a given user. To this end, system 100 includes, among other things, an electronic device 104 associated with a user 101, a server 112, a plurality of resource servers 108, and a database subsystem 150.

[55] In the context of the present technology, system 100 is used to provide search engine services. For example, user 101 may send a given request via electronic device 104 to server 112, which, in response, is configured to provide search results to user 101. Server 112 generates these search results based on information that has been retrieved from, for example, a plurality of resource servers 108 and stored in the database subsystem 150. These search results provided by system 100 may be relevant to the request submitted.

[56] It can be said that the server 112 is configured to execute a variety of computer-implemented algorithms, which are hereinafter referred to as the "search engine" 160. As will become apparent from the description below, the search engine 160 is generally configured to identify potentially relevant digital documents for the query and ranking them based on their relevance to the query.

ELECTRONIC DEVICE

[57] As mentioned above, system 100 includes an electronic device 104 associated with a user 101. Thus, electronic device 104 or simply "device" 102 may sometimes be referred to as a "client device", "end user device", or "client electronic device" . It should be noted that the fact that the electronic device 104 is associated with the user 101 does not necessarily imply or imply any mode of operation, such as needing to log in, needing to register, or the like.

[58] In the context of the present description, unless expressly stated otherwise, "electronic device" or "device" is any computer hardware that is capable of running software suitable for the corresponding task being solved. Thus, some non-limiting examples of device 104 include personal computers (desktops, laptops, netbooks, etc.), smartphones, tablets, and the like. Device 104 includes hardware and/or software and/or firmware (or a combination thereof), as is known in the art, to run a given browser application (not shown).

[59] In a broad sense, the purpose of a particular browser application is to allow the user 101 to access one or more web resources. The implementation of a specific browser application is not particularly limited. One example of a given browser application that runs on device 104 could be implemented as a browser Yandex™. For example, the user 101 may use a given browser application to (i) navigate to a given search engine website and (ii) send a request in response to which he/she should be provided with relevant search results.

[60] The device 104 is configured to generate a request 180 to communicate with the server 112. The request 180 may take the form of one or more data packets containing information indicating, in one example, the request submitted by the user 101. The device 104 is also configured to receive a response 190 from the server 112. The response 190 may take the form of one or more data packets containing information indicating, in one example, search results that are relevant to the request submitted, and computer-readable instructions for a given browser application to display those search results to the user 101.

Communication network

[61] System 100 includes a communication network 110. In one non-limiting example, communication network 110 may be implemented as the Internet. In other non-limiting examples, communication network 110 may be implemented in other ways, such as any wide area network, local area network, private network, and the like. In fact, how the communications network 110 is implemented is not a limitation and will depend, among other things, on how the other components of the system 100 are implemented.

[62] The purpose of the communication network 110 is to communicate with at least some components of the system 100, such as the device 104, a plurality of resource servers 108, and a server 112. For example, this means that the plurality of resource servers 108 may accessed via communication network 110 by device 104. In another example, this means that a plurality of resource servers 108 can be accessed via communication network 110 by server 112. In another example, this means that server 112 can be accessed via communication network 110 by device 104 .

[63] Communication network 110 may be used to transfer data packets between device 104, a plurality of resource servers 108, and server 112. For example, communication network 110 may be used to transmit request 180 from device 104 to server 112. In another example, communication network 110 may be used to send a response 190 from the server 112 to the device 104.

Many resource servers

[64] As mentioned above, a plurality of resource servers 108 can be accessed via the communications network 110 . A plurality of resource servers 108 may be implemented as conventional computer servers. In a non-limiting example of an embodiment of the present technology, this one of the plurality of resource servers 108 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. This one of the many resource servers 108 may also be implemented in any other suitable hardware and/or software and/or firmware, or combinations thereof.

[65] The plurality of resource servers 108 are configured to host (web) resources that can be accessed by the device 104 and/or the server 106. What type of resources are hosted by the plurality of resource servers 108 is not limited. However, in some embodiments of the present technology, the resources may contain digital documents or simply "documents" that represent web pages.

[66] For example, a plurality of resource servers 108 may contain web pages, which means that a plurality of resource servers 108 may store documents representing web pages and available to device 104 and/or server 112. This document may be written in a markup language and may contain, among other things, (i) the content (content) of the relevant web page and (ii) computer-readable instructions for displaying the relevant web page (its content).

[67] The device 104 may access a given one of the plurality of resource servers 108 to retrieve a given document stored on that one of the plurality of resource servers 108. For example, the user 101 may enter a web address associated with a given web page into a given browser application of the device 104, and in response, the device 104 may access a given resource server hosting the given web page to obtain a document representing this web page to display the contents of the web page through this browser application.

[68] The server 112 may access a given one of the plurality of resource servers 108 to retrieve a given document stored on that one of the plurality of resource servers 108. The purpose by which server 112 accesses and retrieves documents from a plurality of resource servers 108 will be described in more detail hereinafter.

Database Subsystem

[69] The server 112 is connected with the possibility of communication with the subsystem 150 of the database. Broadly, the database engine 150 is configured to receive data from the server 112, store the data, and/or provide the data to the server 106 for later use.

[70] In some embodiments, the database engine 150 can be configured to store information associated with the server 112, herein referred to as "search engine data" 175. For example, the database engine 150 can store information about previously performed searches by the search engine. 160, information about previously sent requests to the server 112 and about the documents that were provided by the search engine 160 of the server 112 as search results.

[71] It is contemplated that, as part of the search engine data 175, the database engine 150 may store query data associated with corresponding queries sent to the search engine 160. The query data associated with a given query can be of various types and is not limiting. For example, the database engine 150 may store query data for appropriate queries, such as, but not limited to:

the popularity of this request;

frequency of sending this request;

the number of clicks (clicks) associated with this request;

indications of other sent requests related to this request;

indication of documents related to this request;

other statistics related to this request;

search terms associated with a given query;

the number of characters in this request; and

other query-specific characteristics of that query.

[72] It is contemplated that, as part of the search engine data 175, the database subsystem 150 may also store document data associated with corresponding documents. The document data associated with a given document can be of various types and is not limiting. For example, the database engine 150 may store document data for relevant documents, such as, but not limited to:

the popularity of this document;

the ratio of the number of clicks to the number of impressions for this document;

time per click associated with this document;

indication of queries related to this document;

other statistics related to this document;

the text associated with this document;

file size of this document; and

other document-specific characteristics of that document.

[73] As will be discussed in more detail with reference to FIG. 2 below, database engine 150 may be configured to store content associated with corresponding digital documents on a document-by-document basis. For example, the database engine 150 may be configured to store content associated with a given digital document.

[74] It is contemplated that, as part of the search engine data 175, the database subsystem 150 may also store user data associated with respective users. The user data associated with a given user can be of various types and is non-limiting. For example, the database engine 150 may store user data for respective users, such as, but not limited to:

past web session data associated with a given user;

past requests sent by this user;

click history data associated with a given user;

other data about the interaction with this user and documents; and

user preferences.

[75] As illustrated in FIG. 1, the database subsystem 150 is also configured to store a structured set of data, hereinafter referred to as an "inverted index" 170. In a broad sense, an inverted index 170 is a data structure that can be called a component of the search engine 160. For example, a direct index may be generated first , which stores the lists of words (or terms) for each document - then the direct index can be inverted so that it stores the lists of documents for each term. A direct index query may require sequential iteration over each document and each term to check for a matching document. The time, memory, and processing resources to complete such a request are not always technically realistic. On the contrary, one of the advantages of the inverted index 170 is that such a real-time data structure requires comparatively less time, memory, and processing resources to query such a data structure.

[76] As will be described in more detail below with reference to FIG. 2, the inverted index 170 is configured to store a plurality of lists of documents, each of which is associated with a corresponding term, and wherein this list of documents contains a plurality of documents containing the corresponding term. In addition, the inverted index 170 is configured to store data for the respective term-document pairs.

Server

[77] System 100 includes a server 112, which may be implemented as a conventional computer server. In an exemplary embodiment of the present technology, server 112 may be implemented as a Dell™ PowerEdge™ server running a Microsoft™ Windows Server™ operating system. It goes without saying that the server 106 may be implemented in any other suitable hardware and/or software and/or firmware, or combinations thereof. In the depicted non-limiting embodiment of the present technology, server 112 is a single server. In alternative non-limiting embodiments of the present technology, the functionality of the server 106 may be distributed and may be implemented by multiple servers.

[78] As shown in FIG. 1, server 112 is configured to host a search engine 160 to provide search engine services. In some embodiments, server 112 may be under the control and/or administration of a search engine provider (not shown), such as, for example, a search engine operator. Yandex™ machines. As such, server 112 may be configured to host search engine 160 to perform one or more searches in response to queries submitted by users of search engine 160.

[79] For example, server 112 may receive a request 180 from device 104 indicating a request submitted by user 101. Server 112 may search in response to the submitted request to generate search results relevant to the submitted request. As a result, the server 112 may be configured to generate a response 190 indicating the search results and may transmit the response 190 to the device 104 to display the search results to the user 101, such as through a given browser application.

[80] The search results generated for a submitted request can take many forms. However, in one non-limiting example of the present technology, the search results generated by the server 112 may point to documents that are relevant to the submitted request. How the server 112 is configured to determine and retrieve documents that are relevant to the submitted request will become clear from the description given here.

[81] The server 106 may also be configured to execute the crawler application 120. In a broad sense, the crawler application 120 may be used by the server 112 to "visit" resources available via the communications network 110 and retrieve/download them for further use. For example, the crawler application 120 may be used by the server 106 to access a plurality of resource servers 108 and to retrieve/download documents representing web pages hosted on the plurality of resource servers 108.

[82] It is contemplated that the crawler application 120 may be periodically executed by the server 112 to retrieve/download documents that have been updated and/or made available over the communications network 110 since the previous execution of the crawler application 120.

[83] The server 112 may also be configured to execute a "ranking model" 130 broadly configured to use information about a given query and a set of potentially relevant documents to rank those documents in response to the query. In at least one embodiment of the present technology, the ranking model 130 may be implemented as one or more machine learning (MLA) algorithms. It is contemplated that a plurality of potentially relevant documents can be identified by the server 112 using the information stored in the inverted index 170.

[84] In a broad sense, a given MLA is first "built" (or trained) using training data and training targets. During this training iteration, the MLA is given training input and generates the appropriate prediction. The server 112 is then configured to "tune" the MLA in some sense based on a comparison of the prediction result with the corresponding training target for the training input. For example, tuning may be performed by server 112 using one or more machine learning techniques such as, but not limited to, backpropagation. Thus, after a large number of training iterations, the MLA "tunes" in such a way as to make predictions based on the input data so that these predictions are close to the corresponding training goals.

[85] In at least some embodiments of the present technology, the ranking model 130 may be implemented as an MLA based on a given decision tree. Broadly speaking, a given decision tree based MLA is a machine learning model having one or more "decision trees" that are used to (i) move from observations of an object (represented in branches) to inferences about the target value of an object (represented in leaves). In one non-limiting implementation of the present technology, a decision tree-based MLA may be implemented in accordance with the CatBoost framework.

[86] How a decision tree based MLA can be trained in accordance with at least some embodiments of the present technology is disclosed in US Patent Publication No. 2019/0164084 entitled "METHOD OF AND SYSTEM FOR GENERATING PREDICTION QUALITY PARAMETER FOR A PREDICATION MODEL EXECUTED IN A MAHCINE LEARNING ALGORITHM" published May 30, 2019, the contents of which are hereby incorporated by reference in their entirety. Additional information about the CatBoost library, its implementation, and gradient boosting algorithms is available at https://catboost.ai.

[87] What data is used by ranking model 130 to generate a ranked list of documents in response to a query will be discussed in more detail hereinafter with reference to FIG. 6. However, it should be noted that the ranking model 130 can be configured to use a "ranking feature" for a given "document-query pair" to rank the corresponding document in response to a query.

[88] The server 106 is configured to execute a "ranking feature generator" 140, which is broadly configured to use query-dependent data, to generate one or more ranking features to be used by the ranking model 130. In at least one embodiment of the present technology, the ranking feature generator 140 may be implemented as a neural network (NN).

[89] Broadly speaking, NNs are a special class of MLAs consisting of interconnected groups of artificial "neurons" that process information using a connectionist approach to computation. NNs are used to model complex relationships between inputs and outputs (without actually knowing those relationships) or to look for patterns in data. NNs are first trained during a training phase during which they are given some known set of "inputs" and information to adapt the NN to generate the proper output (for some particular situation that is being modeled). During this learning phase, this NN adapts to the situation being learned and changes its structure so that the given NN can provide reasonable predictive output for certain inputs during some new situation (based on what has been learned). Thus, instead of trying to define complex statistical schemes or mathematical algorithms for some specific situation, this NN tries to give an "intuitive" answer based on the "perception" of the situation. Thus, this NN is a kind of trained "black box" that can be used in a situation where what is in the "box" may be less important; and when it's more important to have a "box" that gives reasonable answers to the given inputs. For example, NNs are commonly used to optimize the distribution of web traffic between servers and data processing, including filtering, clustering, signal separation, compression, vector generation, and the like.

[90] How a ranking feature generator (eg, NN) 140 can be trained to generate a given ranking feature for a given query document pair will be described in more detail hereinafter with reference to FIG. 3. How the ranking feature generator 140 can then be used in real time during its use phase by the search engine 160 will be described in more detail hereinafter with reference to FIG. 5. However, it should be mentioned that the ranking feature generator 140 may use query-specific data that is dynamically generated for a given query-document pair.

[91] Server 112 may also be configured to execute one or more computer-implemented algorithms, hereinafter referred to as "dynamic feature generator" 150, which is broadly configured to use query-independent data retrieved from inverted index 170 for one or more terms in the following order: to generate real-time (dynamically) query-dependent data. How the query-independent data may be used by the dynamic feature generator 155 to generate query-dependent data, and how the dynamic feature generator may be implemented by the server 112, will be discussed in more detail here below with reference to FIG. four.

[92] With reference to FIG. 2 shows a representation 200 of at least some of the data stored in the database engine 150. For example, a representation 202 of at least some of the document data stored in the database engine 150 as part of the search engine data 175 is shown, as explained above.

[93] As shown, the database engine 150 may be configured to store a plurality of documents 204 associated with respective content data. For example, the database engine 150 may store the document 210 along with the content data 212 . In this example, the document 210 may be a given web page that has been crawled by the crawler application 120 and downloaded from one of the resource servers 108 .

[94] Content data 212 may contain multiple content types 214 . For example, one of the set of content types 214 may include a title of the document 210. In the same example, another of the set of content types 214 may include a Uniform Resource Locator (URL) associated with the document 210. In the same example, another of the set content types 214 may include the main content of document 210. Additionally or alternatively, other content types other than those non-exhaustively listed above may be stored as part of the content data for a given document without departing from the scope of the present technology.

[95] Also shown is a representation 222 of at least some of the data stored as part of the inverted index 170. As explained above, the inverted index 170 may store a plurality of lists 224 of documents associated with respective terms. In a broad sense, a given list of documents for a given search term will contain links in the form of document numbers, for example, to those documents in which this search term occurs. The links in a given list of documents may themselves be in numerical order, while there will be spaces between document numbers, as the search term does not occur in documents whose numbers are omitted and form spaces. For example, the inverted index 170 is configured to store a list 250 of documents associated with term 220 (T1) and a plurality of documents 230. As illustrated, the plurality of documents contains document 210 (D1) and other documents that contain term 220 (T1) in their content.

[96] It should be noted that this list of documents may contain data pointing to one or more location indicators regarding the position of the corresponding term in the corresponding documents.

[97] In some embodiments, it is contemplated that the set of document lists 224 may contain lists of documents for "like terms". For example, the first term may be similar to the second term if the second term is, but is not limited to, a synonym for the first term, a normalized version of the first term, and the like. In this example, the first term could be "occupation" and the second term could be "work". In this example, the first term could be "work" and the second term could be "work". As will become apparent from the description below, the server 112 can be configured to identify one or more terms similar to the terms from the query and use them in addition to the terms from the query to retrieve data from the inverted index 170.

[98] In the context of the present technology, the inverted index 170 is configured to store query-independent data for corresponding document-term (DT) pairs. For example, server 112 may be configured to store query-independent data 260 for the D1-T1 pair, i.e., query-independent data 260 is associated with a pair containing term 220 and document 210.

[99] It can be said that query-independent data 260 includes a data set dependent on a DT pair stored for a corresponding D1-T1 pair. Similarly, the inverted index 170 may be configured to store a set of query-independent data 240 (a set of data sets dependent on a DT pair) for the respective DT pairs.

[100] It should be noted that, in the context of the present technology, "query-independent" data may refer to data that is determined and stored without prior knowledge of what the currently used query is. The query-independent data 260 is assumed to point to (i) a term-dependent occurrence of the term 220 (T1) in the content associated with the document 210 (D1).

[101] It is contemplated that a term-dependent occurrence of term 220(T1) may include at least one of: (i) one or more positions of term 220(T1) in the header associated with document 210(D1), (ii ) one or more term positions 220 (T1) in the URL associated with document 210 (D1), and (iii) one or more term positions 220 (T1) in the body of document 210 (D1).

[102] In some embodiments of the present technology, query-independent data 260 for the D1-T1 pair may further comprise content-based query-independent data associated with the D1-T1 pair. In these embodiments, the content-based query-independent data may point to the textual context of the corresponding term in the content associated with the given document. In the example of a D1-T1 pair, query-independent data based on content in query-independent data 260 may point to one or more neighboring terms with term 220 (T1) in the content of document 210 (D1).

[103] As will become apparent from the description below, dynamic feature generator 155 can use query-independent data stored in inverted index 170 to generate real-time query-specific data for a given document-query (DQ) pair. Ranking feature generator 140 can then use query-specific data and data associated with various terms from the query to generate a given ranking feature for a given DQ pair.

[104] How server 112 is configured to use real-time, in response to receiving a current request, dynamic feature generator 155 to generate a real-time ranking feature using ranking feature generator 140 will be discussed here further below with reference to FIG. 5. However, how the server 112 is configured to use the dynamic feature generator 155 to generate training data for training the ranking feature generator 140 will first be discussed with reference to FIG. 3 and 4.

[105] In FIG. 3 shows a representation 300 of training data 310 and a representation 350 of one training iteration of a ranking feature (NN) generator 140. Server 112 may be configured to use training data 310, which may be used to generate a training set that will be used to perform one training iteration.

[106] The training data 310 comprises a DQ training pair 302 having a training query 304 (Qa) and a training document 306 (Da). The training query 304 has a set of terms 330 containing a first term 332 (Ta), a second term 334 (Tb), and a third term 336 (Tc). Training data 310 is associated with label 320. Label 320 indicates that training document 306 (Da) matches training query 304 (Qa).

[107] For example, the label 320 may be determined by a human evaluator who has been instructed to "score" the relevance of the training document 306 (Da) to the training query 304 (Qa). Similarly, one or more evaluators may be tasked with estimating a plurality of DQ training pairs, and a plurality of DQ training pairs may be stored in the database system 150 along with the corresponding estimated labels.

[108] Server 112 may be configured to use training data 310 and label 320 to generate training set 380 to perform one iteration of training the ranking feature generator 140 shown in FIG. 3. The training set 380 contains a set of training term embeddings 382, a set of training feature vectors 384, and a label 320.

[109] The server 112 can be configured to generate a set of training term embeddings 382 based on the corresponding terms from the training query 304. In a broad sense, an embedding is a relatively low-dimensional space into which high-dimensional vectors can be translated. Embedding simplifies machine learning for large inputs, such as sparse vectors representing words. Ideally, nesting captures some of the semantics of the input by placing semantically similar inputs close together in the nesting space. It is also assumed that embeddings can be learned and reused across different models. Server 112 may be configured to execute one or more computer-implemented nesting algorithms configured to receive a given training term as input and output a corresponding training term nesting. For example, server 112 may be configured to generate a first training term embedding 342 based on first term 332 (Ta), a second training term embedding 344 based on second term 334 (Tb), and a third training term embedding 346 based on third term 336 (Tc) .

[110] As mentioned above, training set 380 contains a plurality of feature vectors 384. It should be noted that one or more features from the set of feature vectors 384 may be generated by the dynamic feature generator 155. As will now be discussed in more detail with reference to FIG. 4, one or more features from the set of feature vectors 384 may contain query-dependent data generated by the dynamic feature generator 155 based on query-independent data extracted for the set of training terms 330 from the inverted index 170.

[111] In FIG. 4 depicts a representation 400 of how dynamic feature generator 155 generates a set of feature vectors 384 for training set 380. Server 112 is configured to access an inverted index 170 to retrieve a set of query-independent datasets associated with the respective DT training pairs, i.e., the server 112 may retrieve a first query-independent dataset 402 for the Da-Ta training pair, a second query-independent dataset 404 for the Da-Tb training pair, and a third query-independent dataset 406 for the Da-Tc training pair. It can be said that the first query-independent dataset 402, the second query-independent dataset 404, and the third query-independent dataset 406 represent query-independent data extracted from the inverted index 170 for the set of training pairs DT.

[112] The server 112 is configured to input the query-independent data thus retrieved into the dynamic feature generator 155 . In a broad sense, the dynamic feature generator 155 is configured to generate, for each training term from the training query 304, a corresponding feature vector. In the illustrated example, dynamic feature generator 155 uses the thus extracted query-independent data to generate a first feature vector 352 for training term 332 Ta, a second feature vector 354 for training term 334 Tb, and a third feature vector 356 for training term 336 Tc.

[113] It should be noted that the dynamic feature generator 155 may comprise a plurality of computer-implemented algorithms that are configured to process query-independent data to generate a plurality of features (for each training term) whose types are predetermined. For example, the types of features to be generated by the dynamic feature generator 155 can be determined by the operator of the server 112. In this example, the operator can define the types of features that he considers useful for a particular implementation of the present technology.

[114] In some embodiments, feature types may include query-independent features and query-dependent features. The developers of the present technology have recognized that the query-independent data stored in the inverted index 170 can be retrieved in real time and processed to generate query-dependent data. In particular, the developers of the present technology have found that the query-independent datasets extracted for the respective DT pairs, taken individually, do not include information that depends on more than one term from the query (hence query-independent). However, when query-independent data sets are retrieved for each pair of DTs and parsed in combination with each other, server 112 may in some sense "extract" query-dependent data from it in the form of one or more query-dependent features.

[115] To better illustrate this, suppose that request 304 is "table border css" ("table border css") and therefore term 332 (Ta) is "table" ("table"), term 334 (Tb ) is "border" and term 336 (Tc) is "css". Also suppose that document 306 (Da) is linked to the following URL: "https://www.w3schools.com/css/css_table.asp", has the title "CSS styling tables", and the body contains the following sentences: "Table Borders", "To specify table borders in CSS, use the border property", and "The example below specifies a black border for <table>, <th>, and <td> elements" ("The example below specifies a black border for <table>, <th>, and <td> elements").

[116] In this example, the first query-independent dataset 402 specifies one or more term-dependent occurrences of the "table" term 332 (Ta) in document-related content. For example, the first query-independent dataset 402 may indicate the presence of a "table" term 332 (Ta) in the document header 306. Alternatively, the first query-independent dataset 402 may indicate the presence of a "table" term 332 (Ta) in the document body. document 306. In an additional case, the first query-independent data set 402 may indicate the presence of a "table" term 332 (Ta) in the URL of the document 306. In yet another case, the first query-independent data set 402 may indicate the position (or offset) of the term 332 (Ta) "table" in the content of the document 306. Optionally, the first query-independent data set 402 may indicate how many times the term 332 (Ta) "table" occurs in the content of the document 306.

[117] Similarly, the second query-independent dataset 404 specifies one or more term-dependent occurrences of the (Tb) "table" term 334 in the content associated with the document 306. Similarly, the third query-independent dataset 406 specifies one or more term-dependent from the term occurrences of the term 336 (Tc) "table" in the content associated with the document 306. It should be noted that character.

[118] It should be noted that query-independent data can be used to generate a query-specific feature that indicates a group occurrence of a given term with one or more other terms from query 304.

[119] For example, the operator may predetermine that one type of query-specific feature generator 155 should generate based on query-independent data is the number of times a given term from request 304 is included in addition to the second term in the URL associated with the document 306. Thus, when generating the first feature vector 352 for the Da-Ta pair, the server 112 may determine that the "table" term 332 (Ta) does not occur in addition to the "border" term 334 (Tb) in the URL associated with document 306 ("https://www.w3schools.com/css/css_table.asp"). In this case, the server 112 may determine the value of this feature in the first feature vector 352 as "0".

[120] In another example, the operator may predetermine that another type of query-specific feature that generator 155 should generate based on query-independent data is the number of times a given term from request 304 is included in addition to the third term in the URL. associated with the document 306. Thus, when generating the first feature vector 352 for the Da-Ta pair, the server 112 can determine that the "table" term 332 (Ta) occurs once in addition to the presence of the "css" term 336 (Tc) in The URL associated with document 306 ("https://www.w3schools.com/css/css_table.asp"). In this case, the server 112 may determine the value of this feature in the first feature vector 352 as "1".

[121] In an additional example, the operator may predetermine that another type of query-specific feature that generator 155 should generate based on query-independent data is the number of times a given term from request 304 is included in addition to the first term in the URL. associated with the document 306. Thus, when generating the third feature vector 356 for the Da-Tc pair, the server 112 may determine that the "css" term 336 (Tc) occurs twice in addition to the presence of the "table" term 332 (Ta) in the URL. associated with document 306 ("https://www.w3schools.com/css/css_table.asp"). In this case, the server 112 may determine the value of this feature in the third feature vector 356 as "2".

[122] In yet another example, the operator may predetermine that another type of query-dependent feature that the generator 155 should generate based on the query-independent data is the percentage of terms from request 304 that appear with the given term in the URL associated with the query. with the document 306. Thus, when generating the first feature vector 352 for the Da-Ta pair, the server 112 may determine that two of the three terms from the request 304 occur with the "table" term 332 (Ta) in the URL associated with the document 306 (" https://www.w3schools.com/css/css_table.asp"). In this case, the server 112 may determine the value of this feature in the first feature vector 352 as "2/3". However, when generating the second feature vector 354 for the Da-Tb pair, the server 112 may determine that none of the terms from the request 304 occur with the "border" term 332 (Tb) in the URL associated with the document 306 ("https://www. .w3schools.com/css/css_table.asp") because term 332 (Tb) "border" does not occur in this URL. In this case, the server 112 may determine the value of this feature in the second feature vector 354 as "0".

[123] It should be noted that when generating feature vectors for respective DT pairs, generator 155 can be configured to generate feature vectors of a predetermined size. For example, the size may be determined by the operator based on, for example, the number of query-dependent features that he (she) considers useful for a particular implementation. However, it is contemplated that a given feature vector may contain, in addition to one or more query-dependent features of various types, query-independent features for the corresponding term. In one implementation of the present technology, the size of the feature vectors generated by the generator 155 is "5".

[124] Returning to the description of FIG. 3, server 112 generates for training set 380 a first training feature vector 352, a second training feature vector 354, and a third training feature vector 356 as described above. The server 112 inserts term embeddings 342, 344 and 346 and training feature vectors 352, 354 and 356 into the trainable NN (ranking feature generator 140). In some embodiments, the implementation of the NN may contain multiple parallel input layers and multiple fully connected layers.

[125] It is contemplated that server 106 can be configured to concatenate nested terms with corresponding training feature vectors. For example, server 106 may be configured to concatenate term embedding 342 with training feature vector 352 to generate concatenated training inputs. In the same example, server 106 may be configured to concatenate term embedding 344 with training feature vector 354 to generate concatenated training inputs. In the same example, server 106 may be configured to concatenate term embedding 346 with training feature vector 356 to generate concatenated training inputs. Server 106 can be configured to feed concatenated inputs into the NN so that the NN, in a sense, understands which term embeddings are associated with which training feature vectors.

[126] In response to input, the NN outputs predicted ranking feature 360. Server 112 is configured to compare predicted ranking feature 360 with label 320. For example, server 112 may apply a given penalty function while comparing predicted ranking feature 360 with label 320. The result of the comparison then used by server 112 to, in a sense, "tune" the NN so that the NN generates predicted ranking features that are close to the corresponding labels. For example, tuning may be performed using one or more NN learning methods (eg, backpropagation). After a large number of iterations performed in a similar manner, NN is trained to generate predicted ranking features, which are predictions of the relevance of the relevant documents to the relevant queries.

[127] How the server 112 is configured to use the ranking feature generator 140 and the real-time dynamic feature generator 155 for the current query sent by the user 102 will now be described with reference to FIG. 5. Shown here is a representation 500 of how server 112 is configured to generate predicted ranking feature 540 for a given DQ pair.

[128] Suppose that the server 112 receives the current (used) request 502 sent by the user 102 from the electronic device 104 through the request 180. It can be said that once the request 502 is received by the server 112, the server 112 is configured to use the search engine 160 in real time to generate a given SERP. Request 502 (Qw) has term 506 (Tx), term 507 (Ty), and term 508 (Tz).

[129] The server 112 is configured to determine the set of potentially relevant documents 504 for the query 502 in use. In some embodiments, the server 112 may use one or more computer-implemented procedures to determine the set of documents that are potentially relevant to the query 502 in use. It is assumed that the server 112 may use methods known in the art to identify documents that should be included in the set of potentially relevant documents 504.

[130] Server 112 is configured to generate predicted ranking features for the respective ones from a plurality of potentially relevant documents 504. In one example, server 112 can be configured to generate predicted ranking features 540 for a pair including query 502 (Qw) and document 505 (Dw) from a set of potentially relevant documents 504.

[131] The server 112 is configured to access the inverted index 170 to retrieve query-independent data 510. The query-independent data 510 comprises a first query-independent data set 512 for the Dw-Tx pair, a second query-independent data set 514 for the Dw pair. -Ty and a third query-independent dataset 516 for the Dw-Tz pair.

[132] In some embodiments, query-independent data may be said to be "term dependent" data. It should be noted that the first query-independent data set 512 for the Dw-Tx pair indicates a term-dependent occurrence of the term 506 (Tx) in the content of the document 505 (Dw), the second query-independent data set 514 for the Dw-Ty pair indicates a dependent term 507 (Tx) occurrence in document content 505 (Dw), and the third query-independent data set 516 for the Dw-Tz pair indicates a term-dependent occurrence of term 508 (Tz) in document content 505 (Dw).

[133] In some embodiments, query-independent data 510 may further comprise content-based query-independent (term-dependent) data indicating the textual context of the corresponding term in the content associated with the given document. For example, the text context may include the previous term and the next term for the corresponding term in the content associated with the given document.

[134] Server 112 is configured to input query-independent data 510 into dynamic feature generator 155 to generate a plurality of feature vectors comprising first feature vector 532, second feature vector 534, and third feature vector 536. As explained above, the first feature vector 532, the second feature vector 534, and the third feature vector 536 contain at least one query-dependent feature indicating the group occurrence of the corresponding term with one or more other terms from the query 502.

[135] In at least some embodiments of the present technology, it can be said that the query-independent data was stored in the inverted index 170 prior to the receipt of the query 502 from the electronic device 104, and the query-dependent feature is generated after the query 502 is received from the electronic device 104. It can also be said that the query-specific feature is generated using real-time query-independent data during the document ranking procedure of the search engine 160. It should be noted that data indicative of the query-specific feature could not be stored in the inverted index 170 for given pair of DTs before request 502 is received because it depends on one or more other terms from request 502.

[136] Server 112 is also configured to generate term attachments 518, 520, and 522 for terms 506, 507, and 508, respectively. It can be said that the server 112 can be configured to generate a used set 550 for input to the ranking feature generator 140 (trained NN), and where the used set 550 contains term embeddings 518, 520 and 522 and feature vectors 532, 534 and 536.

[137] The server 112 is configured to input the used set 550 into the ranking feature generator 140, which, in response, generates a predicted ranking feature 540 for the pair of document 505 and request 502. It is contemplated that the server 106 can be configured to generate concatenated inputs by concatenating term embeddings 518 with feature vector 532, term embeddings 520 with feature vector 534, and term embeddings 522 with feature vector 536, and inputting the concatenated input data into ranking feature generator 140.

[138] As mentioned above, server 112 may be configured to use predicted ranking feature 540 to rank document 505 among a set of potentially relevant documents 504 in response to query 502. With reference to FIG. 6 depicts a representation 600 of how server 112 is configured to use a ranking model 130 (e.g., a decision tree based MLA) to generate a ranked list 680 of documents for user 102 in response to request 502. As can be seen, server 112 can input ranking model 130 request data 602. For example, server 112 may access database engine 150 and retrieve information associated with request 502, such as past interaction data associated with request 502.

[139] Server 112 may also input document data 604 and predicted ranking feature 540 for document 505. For example, server 112 may be configured to access database engine 150 and retrieve information associated with document 505. For example, document data 604 may contain content-based information and/or past interaction data associated with document 505. Similarly, server 112 may also input document data 606 and predicted ranking feature 620 for document 506 (another of a plurality of potentially relevant documents 504). Server 112 may be configured to generate a predicted ranking feature 620 for a document 509-query pair 502 in a manner similar to how the server 112 is configured to generate a predicted ranking feature 640 for a document 505-query pair 502. The ranking model 130 is configured to generate as output a ranked list of documents 68 0, ranked based on their relevance to query 502. Server 112 can be configured to use a ranked list 680 of documents to generate SERPs and send data indicating this via response 190 to electronic device 104 for display to user 102.

[140] In FIG. 7 schematically shows a method 700 for ranking digital documents in response to a query. For example, server 112 may be configured to perform method 700, the steps of which will now be discussed in more detail.

STEP 702: for a given document, accessing the inverted index to retrieve query-independent data for the first document-term (DT) pair and the second DT pair

[141] Method 700 begins at step 702, where server 112 is configured to access the inverted index 170 to retrieve query-independent data for the first DT pair and the second DT pair. For example, suppose a query contains a first term and a second term. The first DT pair has the given document and the first term, and the second DT pair has the given document and the second term.

[142] As explained above, the query-independent data received by the server 112 indicates (i) the term-dependent occurrence of the first term in the content associated with the given document, and (ii) the term-dependent occurrence of the second term in the content associated with the document. by this document.

[143] In some embodiments of the present technology, server 112 may be configured to retrieve query-independent data for a term that is similar to one of the first or second terms. For example, server 112 may be configured to, for a given one of a plurality of terms from a query in use, determine a similar term, and when accessing inverted index 170 to retrieve query-independent data, server 112 may be configured to retrieve query-independent data. for the third DT pair, where the third DT pair has the given document and the similar term referred to.

[144] In additional embodiments, it is contemplated that the server 112 can access the inverted index 170 to retrieve content-based query-independent data associated with the first DT pair and the second DT pair. Content-based query-independent data can point to the textual context of the corresponding term in the content associated with the given document. Additionally or alternatively, the database engine 150 may store a forward index in addition to the inverted index 170, which may be used by the server 112 to thus obtain query-independent data based on the content (the text context of the terms in the document).

[145] It is contemplated that a term-dependent occurrence of a given term may contain at least one of: (i) one or more first term positions in a title associated with a given document, (ii) one or more first term positions in a URL associated with a with this document, and (iii) one or more positions of the first term in the body of this document.

STEP 704: for a given document, generating a query-dependent feature using the query-independent data extracted for the first DT pair and the second DT pair

[146] The method 700 proceeds to step 704, where the server 112 is configured to generate a query-specific feature using the query-independent data retrieved for the first DT pair and the second DT pair, where the query-specific feature indicates a group occurrence of the first term with a second term into the content associated with this document.

[147] In one non-limiting example shown in FIG. 5, server 112 may be configured to access inverted index 170 to retrieve query-independent data 510 (eg, block 702). The query-independent data 510 comprises a first query-independent data set 512 for the Dw-Tx pair, a second query-independent data set 514 for the Dw-Ty pair, and a third query-independent data set 516 for the Dw-Tz pair.

[148] In some embodiments, query-independent data may be said to be "term dependent" data. It should be noted that the first query-independent data set 512 for the Dw-Tx pair indicates a term-dependent occurrence of the term 506 (Tx) in the content of the document 505 (Dw), the second query-independent data set 514 for the Dw-Ty pair indicates a dependent term 507 (Tx) occurrence in document content 505 (Dw), and the third query-independent data set 516 for the Dw-Tz pair indicates a term-dependent occurrence of term 508 (Tz) in document content 505 (Dw).

[149] As part of step 704, server 112 may be configured to input query-independent data 510 into dynamic feature generator 155 to generate a plurality of feature vectors comprising first feature vector 532, second feature vector 534, and third feature vector 536. As explained above, the first feature vector 532, the second feature vector 534, and the third feature vector 536 contain at least one query-dependent feature indicating the group occurrence of the corresponding term with one or more other terms from the query 502.

[150] In at least some embodiments of the present technology, it can be said that the query-independent data was stored in the inverted index 170 prior to the receipt of the query 502 from the electronic device 104, and the query-specific feature is generated after the query 502 is received from the electronic device 104. It can also be said that the query-specific feature is generated using real-time query-independent data during the document ranking procedure of the search engine 160. It should be noted that data indicative of the query-specific feature could not be stored in the inverted index 170 for given pair of DTs before request 502 is received because it depends on one or more other terms from request 502.

[151] In at least some embodiments of the present technology, the grouping of a first term with a second term in the content associated with a given document comprises at least one of: (i) the number of times the second term from the query is included in addition to the first term in the title associated with the given document, (ii) the number of times the second term from the query is included in addition to the first term in the URL associated with the given document, (iii) the number of times the second term from the query is included in addition to the first term in the body of the given document, and (iv) a positional offset between the first term and the second term in the body of the given document.

STEP 706: for a given document, generating a ranking feature for a given document based on at least the first term

[152] The method 700 proceeds to step 706, where the server 112 is configured to generate a ranking feature for a given document based on at least the first term, the second term, and the query-dependent feature. For example, server 112 may be configured to use ranking feature generator 140 to generate a ranking feature for a given document.

[153] In some embodiments of the present technology, the ranking feature generator 140 may be implemented as a given NN.

STEP 708: ranking a given document from a set of potentially relevant documents based on at least a ranking feature

[154] The method 700 proceeds to block 708, where the server 112 is configured to rank a given document from a set of potentially relevant documents based on at least the ranking feature determined in block 706. For example, the server 112 may be configured to execute the model 130 ranking.

[155] In at least some embodiments of the present technology, the ranking model 130 may be implemented as an MLA based on a given decision tree. Broadly speaking, a given decision tree based MLA is a machine learning model having one or more "decision trees" that are used to (i) move from observations of an object (represented in branches) to inferences about the target value of an object (represented in leaves). In one non-limiting implementation of the present technology, a decision tree-based MLA may be implemented in accordance with the CatBoost framework.

[156] It is contemplated that server 112 can be configured to rank a given document using an MLA based on a decision tree configured to rank a set of potentially relevant documents based on their relevance to a query.

[157] It should be clearly understood that not all of the technical effects mentioned herein will necessarily be achieved in each and every embodiment of the present technology. For example, embodiments of the present technology may be implemented without achieving some of these technical effects, while other embodiments may be implemented with or without other technical effects.

[158] Some of the above steps, as well as sending/receiving signals, are well known in the art and, as such, have been omitted from parts of this description for simplicity. Signals can be sent/received using optical means (for example, a fiber optic connection), electronic means (for example, using a wired or wireless connection), as well as mechanical means (for example, pressure-based, temperature-based, or any other suitable physical parameter).

Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The preceding description is intended to be exemplary and not limiting. Therefore, the scope of the present technology is intended to be limited only by the scope of the appended claims.

Claims (66)

1. A method for ranking digital documents in response to a request, wherein the digital documents are potentially relevant to a request having a first term and a second term, the request being sent by a user of an electronic device communicatively connected to a server hosting a search engine, the search engine the machine is associated with an inverted index storing information associated with document-term (DT) pairs, the method being executed by a server, the method comprising: for a given document from a set of potentially relevant documents: the server accesses the inverted index to retrieve query-independent data for the first DT pair and the second DT pair, where the first DT pair has the given document and the first term, the second DT pair has the given document and the second term, wherein the query-independent data indicates (i) a term-dependent occurrence of the first term in the content associated with the given document, and (ii) a term-dependent occurrence of the second term in the content associated with the given document; a query-dependent feature is generated by the server using the query-independent data retrieved for the first DT pair and the second DT pair, at the same time, the query-dependent attribute indicates the group occurrence of the first term with the second term in the content associated with this document; generating by the server a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature; and the server ranks this document from a set of potentially relevant documents based on at least a ranking feature. 2. The method of claim 1, wherein the ranking feature for a given document is generated by a neural network (NN). 3. The method of claim 2, wherein the method further comprises training the NN server to generate a ranking feature, wherein training the NN comprises: a training set is generated by the server for a training document-request (DQ) pair to be used during a given training iteration NN, wherein the training pair DQ has a training query and a training document, the training document is associated with a label, the label indicating the relevance of the training document to the training request, and the generation contains the steps in which: generating by the server a plurality of training term embeddings based on the corresponding terms from the training query; access by the server to the inverted index associated with the search engine to extract a set of query-independent data sets associated with the corresponding pairs from the set of training pairs DT, wherein the given one of the plurality of training pairs DT includes the training document and the corresponding one of the plurality of terms from the training query; generating by the server a plurality of training feature vectors for the plurality of training pairs DT using the plurality of query-independent data sets; during a given iteration of NN training: introducing said set of training term embeddings and said set of learning feature vectors into the NN by the server to generate a predicted ranking feature for the training pair DQ; and tuned by the server NN based on the comparison between the label and the predicted ranking feature, so that the NN generates for a given used DQ pair an appropriate predicted ranking feature that indicates the relevance of the corresponding used document to the corresponding used query. 4. The method of claim 1, wherein the query-independent data was stored in the inverted index prior to receiving a query from the electronic device, and wherein the query-dependent feature is generated after receiving a query from the electronic device. 5. The method of claim 1, wherein the query-dependent feature is generated using real-time query-independent data during a search engine document ranking procedure. 6. The method of claim 1, wherein the ranking is performed by a decision tree-based machine learning algorithm (MLA) configured to rank a set of potentially relevant documents based on their relevance to a query. 7. The method of claim 1, wherein the method further comprises, for a given one of the plurality of terms, defining a similar term, and wherein: when accessing the inverted index to retrieve query-independent data, the retrieved query-independent data contains query-independent data for a third pair of DTs, the third pair of DTs having the given document and said similar term. 8. The method of claim 1, wherein the inverted index access is for further retrieval of query-independent data based on the content associated with the first DT pair and the second DT pair, query-independent content-based data points to the textual context of the corresponding term in the content associated with the given document. 9. The method of claim 1, wherein the term-dependent occurrence of the first term comprises at least one of: one or more positions of the first term in the heading associated with this document; one or more positions of the first term in the URL associated with this document; and one or more positions of the first term in the body of this document. 10. The method of claim. 1, in which the group occurrence of the first term with the second term in the content associated with this document contains at least one of: the number of times the second term from the query is included in addition to the first term in the heading associated with the document; the number of times the second term from the query is included in addition to the first term in the URL associated with the document; and the number of times the second term from the query is included in addition to the first term in the body of the given document. 11. A server for ranking digital documents in response to a request, wherein the digital documents are potentially relevant to a request having a first term and a second term, the request being sent by a user of an electronic device communicatively connected to a server hosting the search engine, wherein the search engine is associated with an inverted index storing information associated with document-term (DT) pairs, and the server is configured to: for a given document from a set of potentially relevant documents: access the inverted index to retrieve query-independent data for the first DT pair and the second DT pair, where the first DT pair has the given document and the first term, the second DT pair has the given document and the second term, wherein the query-independent data indicates (i) a term-dependent occurrence of the first term in the content associated with the given document, and (ii) a term-dependent occurrence of the second term in the content associated with the given document; generate a query-dependent feature using the query-independent data retrieved for the first DT pair and the second DT pair, at the same time, the query-dependent attribute indicates the group occurrence of the first term with the second term in the content associated with this document; generate a ranking feature for the given document based on at least the first term, the second term, and the query-dependent feature; and to rank the given document from a set of potentially relevant documents based on at least a ranking feature. 12. The server of claim 11, wherein the server uses a neural network to generate a ranking feature for a given document. 13. The server of claim 12, wherein the server is further configured to teach the NN to generate a ranking feature, the server is configured to: generate a training set for a training document-query pair (DQ) to be used during a given training iteration NN, where the training pair DQ has a training query and a training document, where the training document is associated with a label, where the label indicates the relevance of the training document to the training query , and to generate the training set, the server is configured to: generate a plurality of training term embeddings based on the corresponding terms from the training query; access the inverted index associated with the search engine to retrieve a set of query-independent datasets associated with the corresponding pairs from the set of training pairs DT, wherein the given one of the plurality of training pairs DT includes the training document and the corresponding one of the plurality of terms from the training query; generate a plurality of training feature vectors for the plurality of training pairs DT using the plurality of query-independent datasets; during a given iteration of NN training: input by the server into the NN said set of training term embeddings and said set of learning feature vectors to generate a predicted ranking feature for the training pair DQ; and adjust the NN based on the comparison between the label and the predicted ranking feature, so that the NN generates, for a given used DQ pair, an appropriate predicted ranking feature that indicates the relevance of the corresponding used document to the corresponding used query. 14. The server of claim 11, wherein the query-independent data was stored in the inverted index prior to receiving a query from the electronic device, and wherein the query-dependent feature is generated after receiving a query from the electronic device. 15. The server of claim 11, wherein the query-dependent feature is generated using real-time query-independent data during a search engine document ranking procedure. 16. The server of claim 11, wherein the server is configured to rank using a machine learning algorithm (MLA) based on a decision tree configured to rank a plurality of potentially relevant documents based on their relevance to a query. 17. The server of claim 11, wherein the server is further configured to determine a similar term for a given one of the plurality of terms, and wherein: when accessing the inverted index to retrieve query-independent data, the retrieved query-independent data contains query-independent data for a third pair of DTs, the third pair of DTs having the given document and said similar term. 18. The server of claim 11, wherein the server is configured to access the inverted index to further retrieve query-independent data based on the content associated with the first DT pair and the second DT pair, wherein the query-independent content-based data points to the textual context of the corresponding term in the content associated with the given document. 19. The server of claim 11, wherein the term-dependent occurrence of the first term includes at least one of: one or more positions of the first term in the heading associated with this document; one or more positions of the first term in the URL associated with this document; and one or more positions of the first term in the body of this document. 20. The server of claim 11, wherein the group occurrence of the first term with the second term in the content associated with the document contains at least one of: the number of times the second term from the query is included in addition to the first term in the heading associated with the document; the number of times the second term from the query is included in addition to the first term in the URL associated with the document; and the number of times the second term from the query is included in addition to the first term in the body of the given document.

Images