EP4569391A1 - Contextualized and optimal content load and placement in a web search environment using generative models - Google Patents

Abstract

Example embodiments of the present disclosure provide for an example method for content item load and placement. The method includes obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result. The method includes determining for each search result: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result. The method includes responsive to determining a number of content item slots to generate adjacent to the first search result, generating content slots adjacent to the first search result. The method includes populating the content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving content items.

Description

CONTEXTUALIZED AND OPTIMAL CONTENT LOAD AND PLACEMENT IN A WEB SEARCH ENVIRONMENT USING GENERATIVE MODELS

PRIORITY

[0001] The present application claims the benefit of priority of U.S. Provisional Patent Application No. 63/591,915, filed on October 20, 2023, which is incorporated by reference herein.

FIELD

[0002] The present disclosure relates generally to machine learning. More particularly, the present disclosure relates to implementing machine-learned models to facilitate content load and placement in search interfaces.

BACKGROUND

[0003] A computer can execute instructions to generate outputs provided some input(s) according to a parameterized model. The computer can use an evaluation metric to evaluate its performance in generating the output with the model. The computer can update the parameters of the model based on the evaluation metric to improve its performance. In this manner, the computer can iteratively “learn” to generate the desired outputs. The resulting model is often referred to as a machine-learned model.

SUMMARY

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] In one example aspect, the present disclosure provides for an example computer- implemented method. The example computer-implemented method includes obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results. The example computer-implemented method includes determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result. The example computer-implemented method includes, responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data. The example computer-implemented method includes populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

[0006] In an example aspect, the present disclosure provides for an example system for content load and placement in search interfaces, including one or more processors and one or more memory devices storing instructions that are executable to cause the one or more processors to perform operations. In some implementations, the one or more memory devices can include one or more transitory or non-transitory computer-readable media storing instructions that are executable to cause the one or more processors to perform operations. In the example system, the operations can include obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results. In the example system, the operations can include determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result. In the example system, the operations can include responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data. In the example system, the operations can include populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

[0007] In an example aspect, the present disclosure provides for an example transitory or non-transitory computer readable medium embodied in a computer-readable storage device and storing instructions that, when executed by a processor, cause the processor to perform operations. In the example transitory or non-transitory computer-readable medium, the operations include obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results. In the example transitory or non-transitory computer-readable medium, the operations include determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result. In the example transitory or non-transitory computer-readable medium, the operations include responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data. In the example transitory or non-transitory computer-readable medium, the operations include populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0009] Figure 1 depicts a block diagram of example data flow to perform content item load and placement according to example embodiments of the present disclosure;

[0010] Figure 2 depicts a block diagram of an example system to perform content item load and placement according to example embodiments of the present disclosure;

[0011] Figure 3 depicts an example method for performing content item loading and placement according to example embodiments of the present disclosure; and

[0012] Figure 4 A depicts a block diagram of an example computing system that performs content item load and placement according to example embodiments of the present disclosure.

[0013] Figure 4B depicts a block diagram of an example computing device that performs content item load and placement according to example embodiments of the present disclosure.

[0014] Figure 4C depicts a block diagram of an example computing device that performs content item load and placement according to example embodiments of the present disclosure.

DETAILED DESCRIPTION

[0015] This disclosure relates to content item loading and placement within a search interface. For instance, as search results are selected for display, the system can use a machine-learned model to determine if content items should be displayed alongside the search result and, if so, how many content items should be displayed. The machine-learned model can be a large language model or other generative model. The machine-learned model can make this determination for each search result of a number of search results. Using this method, each search interface can be customized based on user data, query context data, and search results data to create a bespoke search interface experience for each search session. [0016] Existing methods provide for static content slots within search results where content items can be provided at the top or beginning of a search result page or the bottom or end of the search result page. A single call to a content provider service engine is made following receipt of the entire set of search results generated responsive to a search query. The present disclosure allows for bespoke search result interfaces that allow for improved integration between search result items and content items within a search result page. This is accomplished using language models to ingest input data including user context data, search query context data, and search results to generate an output including whether a content item can be displayed alongside the search result (e.g., a determination of authorization to display a content item adjacent to the search result), and if a content item can be displayed (e.g., is authorized to be displayed adjacent to the search result), how many content item slots and the placement of the content item slots. This allows for adjusting the display of search results and associated selected content items based on the intent of the user’s search query. For instance, if a search query is purely information seeking, less or no content item slots can be generated or populated, however if instead, the search query is associated with a purchase intent, one or more content item slots can be generated for the respective search results depending on the relationship between the search results and the query.

[0017] The machine-learned model can take into account user context data and query context data to make the content load and placement determinations. User context data can include historical search history, prior content item interaction data, or other relevant user data. Query context data can include query subject matter, query intent, or query sub-intent.

[0018] The technology of the present disclosure provides for a number of technical effects and benefits. For instance, aspects of the described technology can allow for improved utilization of limited display as well as conservation of computing resources by determining a number of calls to make to a content provider service engine based on user context and query context. For instance, based on data indicative of a user device being located within an area that generally has constrained bandwidth resources, the computing system can determine that calls to fill content item slots should occur a more limited number of times than for a user device located within an area that generally has larger bandwidth resources available. Further, the present disclosure provides for improved computing resource utilization by determining when to make calls to a content provider service engine based on the type of search query, including intent or sub-intents, and the individual search results that are received. As such, calls for content items can be made in such a way that redundant content is not provided, relevant search results are not obstructed by irrelevant content items, or system resources are not wasted on calls when there are limited resources or an indication that a user does not interact with content items. [0019] The improvements associated with the systems and methods discussed herein can be further understood with reference to the figures.

[0020] Reference now is made to the figures, which provide example arrangements of computing systems, model structures, and data flows for illustration purposes only. FIG. 1 illustrates an example data flow 100 to perform content load and placement according to example embodiments of the present disclosure.

[0021] Data flow 100 can include input signals 105, content load and placement model 125, output data 130, and content selection engine 145. For instance, input signals 105 can include user context data 110, query context data 115, and search results data 120. Content load and placement model 125 can obtain input signals 105 and generate output data 130. Output data 130 can include authorized to place content 135 and/or number of content slots 140. For instance, for each generated search result, a call can be made to content load and placement model 125. Content load and placement model 125 can include a machine learned model. In some instances, content load and placement model 125 can include a language model, such as a large language model.

[0022] Content load and placement model 125 can generate output for each search result indicating if a content item can be placed alongside the search result (e.g., authorized to place content 135). If the indication is yes, that content can be placed alongside the search result, then the system can determine a number of content slots 140 that are available next to the content item. In some instances, the authorization of placement and number of content items can be determined based on user context data 110 and query context data 115 in addition to search results data 120.

[0023] Upon receipt of output data 130 comprising an indication of authorization to place a content item, the system can make a call to content selection engine 145. Content selection engine 145 can utilize output data 130, user context data 110, query context data 115, and search results data 120 to select one or more content items based on the number of content slots that are permitted to be presented alongside the search result. This can be performed for each search result that is generated as output from a search engine. This is described in more detail with regard to FIG. 2.

[0024] FIG. 2 depicts an example system to perform content load and placement according to example embodiments of the present disclosure. FIG. 2 includes client device 202 which can include a client application. A client application can be, for example, a browser. Client device 202 can obtain user input via client application. For instance, user input can include user query data 205. User query data 205 can be provided as input into search engine 210. Additionally, user query data 205 can be stored in a query context data 215. Additionally, user query data 205 can be stored in user context data 220.

[0025] User context data 220 can maintain a trace of recent actions taken in client application. Client application can provide a first interface for submitting a search query. Client application can process a search query via search engine 210. Search engine 210 can generate search results based on user query data 205, query context data 215, and/or search index data 225.

[0026] Search results data 230 can be generated as output by search engine 210. In some implementations each search result (e.g., search results 1 235A, result 2 235B, result 3 235C, and result n 235D) can be output consecutively (e.g., one at a time). As each respective search result is generated, the search result can be provided as input into content load and placement module 240. Content load and placement module 240 can include one or more machine learned model(s) 245. Machine learned model(s) 245 can include language models (e.g., large language models). Machine learned model(s) 245 can generate output 250 for each respective search result.

[0027] For instance, result 1 235 A and associated data can be provided as input into machine learned model(s) 245 to generate output 1 205A. Output 250A can include an indication to show content or not show content. In this example embodiment, the system includes an indication of “yes” for show content. Due to the show content indication being “yes”, the machine-learned model(s) 245 also generates a number of content items to be provided for display alongside result 1 235 A. If a show content indication is “yes”, a call can be made to content selection engine 255.

[0028] Content selection engine 255 can output selected content items 260. For result 1 235 A, the selected content items 260 can include content items 260 A which include a first content item and a second content item. Content items 260A can be provided to be displayed alongside result 1 235 A via an interface of client device 202. The interface can be associated, for example, with a client application (e.g., browser).

[0029] As depicted in FIG. 2, the content load and placement module 240 can be called for each respective search result of search result data 230. For instance, machine learned model(s) 245 can obtain result 2 235B and associated data to determine output 250B. Output 250B can include an indication not to show content alongside result 235B. As such, the system can transmit data to cause result 2 235B to be provided for display and repeat the process for result 3 235C. [0030] Continuing with the example, machine learned model(s) 245 can obtain result 3 235C and associated data to determine output 250C. Output 250C can include an indication not to show content alongside result 235C. As such, the system can transmit data to cause result 3 235C to be provided for display and repeat the process for as many search results as are generated by search engine 210.

[0031] Machine learned model(s) 245 can obtain result n 235D and associated data as input and generate output 4 250D can include an indication of “yes” for show content. Due to the show content being “yes”, the machine-learned model(s) 245 also generates a number of content items to be provided for display alongside result 4 235D. As a result of the content indication being “yes”, a call can be made for content selection engine 255. Content selection engine 255 can generate content item 260D as output. The client device 202 can obtain result 2 235B, result 3 235C, result 4 235D, and/or the selected content item 260D to be provided for display via client device 202 (e.g., via a client application).

[0032] The content load and placement module can, in addition to search result data, obtain user context data 220 or query context data 215 as input to determine an indication of show content or number of content items for each respective search result.

[0001] FIG. 3 depicts a flow diagram of an example method 300 to perform content load and placement in accordance with some embodiments of the present disclosure. The method 300 can be performed by processing logic that can include hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, method 300 is performed by a server computing system (e.g., server computing system 604) or client computing system (e.g., client computing system 602). Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processors can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0033] At operation 302, processing logic can obtain, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results.

[0034] The input data can further include bandwidth access data. A number of permitted transmitted requests to the content item selection engine is determined based on the bandwidth access data. For instance, a device with less bandwidth access can be permitted to transmit less requests than a device with more bandwidth access.

[0035] User context data can include at least one of: (i) historical data or (ii) user preference data. As described herein, historical data can include past search history, past websites visited, past content interacted with, or other relevant prior activity. User preference data can include types of content preferred, format of content, likes, dislikes, or other relevant preference data.

[0036] The query context data can include at least one of: (i) the query, (ii) an intent of the query, or (iii) one or more sub-intents of the query. The intent of the query can include at least one of: (i) information seeking or (ii) purchase seeking. In some instances, a query can be processed by the computing system to determine an intent of the query. For instance, a query containing the words “car for sale” can be associated with a purchase seeking intent whereas a query containing the words “brief history of Main street” can be associated with an information seeking intent. A machine-learned model can be utilized to determine the intent of the query and/or one or more sub-intents of the query.

[0037] The data associated with a first search result of a plurality of search results can include at least one of (i) content of the search result, (ii) associated webpage, or (iii) an association between the query and the search result.

[0038] At operation 304, processing logic can determine, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result. The machine-learned model can include a large language model. The determination of authorization to display a content item adjacent to the search result can be based on the (i) user context data, (ii) query context data, (iii) and data associated with the first search result of the plurality of search results. For instance, based on preferences associated with the user (e.g., to see less sponsored content items), the system can determine to not show a content item next to a search result. Additionally, a purchase intent can be associated with a higher weight toward generating a content item slot next to the search result whereas an information seeking intent can be associated with a lower weight toward generating a content item slot next to the search result. In some instances, a content slot can be prevented from being generated if the system predicts that the populated content items would be duplicative of the search results. For instance, if the query is “car for sale”, and the first three results are all dealerships in the user’s geographic area, a content item for one of those results can be prevented from being displayed alongside the natural search results.

[0039] At operation 306, processing logic can, responsive to determining a number of content item slots to generate adjacent to the first search result, generate one or more content slots adjacent to the first search result based on the input data and determined output data. For instance, a first search result can be associated with displaying a first content slot next to the search result. As such, a first content slot can be generated adjacent to the search result. In some instances, multiple content slots can be generated next to a first search result.

[0040] At operation 308, processing logic can populate the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content item. The content items can be selected using the content item selection engine based on the query context data. Query context data can include search query data.

[0041] In some instances, processing logic can obtain interaction data indicative of a user interaction with one or more content items populated in one or more content item slots. Processing logic can generate a label for each set of interaction data. Processing logic can train the machine-learned model based on the labeled interaction data. For instance, if a content item is interacted with, this can indicate at least that a content item can be shown next to that search result, that a content item can be shown for that query or similar queries, and/or that the particular content item selected can be shown next to the search result. If a user continues to scroll without interacting with any content item, the system can utilize this information to prevent further content slots from being rendered alongside further search results. This can help conserve compute resources by preventing rendering of content slots as well as preventing unnecessary transmission of data requests for content items to populate the respective content item slots.

[0042] FIG. 4A depicts a block diagram of an example computing system 600 that performs prompt generation and recommendations for input into generative models to improve the output of the generative models according to example embodiments of the present disclosure. The computing system 600 includes a client computing system 602, a server computing system 604, a training computing system 606, a content provider computing system 608, and a search engine computing system 610 that are communicatively coupled over a network 630. [0043] The client computing system 602 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0044] The client computing system 602 includes one or more processors 612 and a memory 614. The one or more processors 612 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 614 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 614 can store data 616 and instructions 618 which are executed by the processor 612 to cause the client computing system 602 to perform operations.

[0045] In some implementations, the client computing system can include an application. The application can include an application that is downloaded on a user device. Additionally, or alternatively, the application can include a web-based application. The application can communicate with an application programming interface to interface with a search system. For instance, the API can facilitate interaction between client computing system 602, server computing system 604, and search engine computing system 610.

[0046] In some implementations, the client computing system can include a user interface. The user interface can include a graphical user interface, audio user interface, touch user interface, or any other user interface. The client computing system can include a user input component. The user input component can be associated with user interface and can be capable of obtaining user input. For instance, user input can include touch, audio, or other user input. In some instances, user input component can be capable of obtaining user input and translating the user input into a computer readable form. The client computing system 602 can also include one or more user input components that receives user input. For example, the user input component can be a touch-sensitive component (e.g., a touch- sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0047] As described above, the client computing system 602 can store or otherwise include one or more models 620 (e.g., generative model 622). For example, the models 620 (e.g., generative models 622) can be or can otherwise include various statistical or machine-learned models. Example machine-learned models include neural networks or other multi-layer nonlinear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Models 620 can include generative models 622. Generative models 622 can be configured to generate one or more output prompts responsive to obtaining input prompt data. The output prompts can include, for example, responses to user search queries, follow-up questions, or context tailored landing page data. The confidence model 642 and generative model 622 are discussed with reference to Figure 1 and Figure 2.

[0048] In some implementations, the one or more models 620 can be received from the server computing system 604 over network 630, stored in the client computing system memory 614, and then used or otherwise implemented by the one or more processors 612. In some implementations, the client computing system 602 can implement multiple parallel instances of a single model 620.

[0049] More particularly, the model can determine whether to generate a content slot next to a content item. In some instances, the model can determine that no more content slots should be generated. For instance, based on a detection of limited network bandwidth resources, the model can determine that no further content slots should be generated to prevent utilization of bandwidth resources to populate the content slots.

[0050] Additionally or alternatively, one or more models 640 can be included in or otherwise stored and implemented by the server computing system 604 that communicates with the client computing system 602 according to a client-server relationship. For example, the models 640 can be implemented by the server computing system 604 as a portion of a web service (e.g., a search service). Thus, one or more models 620 can be stored and implemented at the client computing system 602 and/or one or more models 640 can be stored and implemented at the server computing system 604.

[0051] The server computing system 604 includes one or more processors 632 and a memory 634. The one or more processors 632 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 634 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 634 can store data 636 and instructions 638 which are executed by the processor 632 to cause the server computing system 604 to perform operations. [0052] In some implementations, the server computing system 604 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 604 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0053] Server computing system 604 can be configured to obtain data from client computing system 602 (e.g., via an application). For instance, server computing system 604 can utilize the obtained user input data to update or train one or more models 620 or 640 (e.g., confidence model 642, generative model 644, generative model 622).

[0054] As described above, the server computing system 604 can store or otherwise include one or more models 640 (e.g., confidence model 642, generative model 644, generative model 622). For example, the models 640 (e.g., confidence model 642, generative model 644, generative model 622) can be or can otherwise include various statistical or machine-learned models. Example machine-learned models include neural networks or other multi-layer nonlinear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Models 640 can include confidence model 642 and generative model 644. Confidence model 642 can determine a confidence level in a generative model’s accuracy. Generative model 644 can be configured to generate one or more output prompts responsive to obtain input prompt data. The output prompts can include, for example, responses to user search queries, follow-up questions, or context tailored landing page data. The confidence model 642 and generative model 644 are discussed with reference to Figure 1 and Figure 2.

[0055] The client computing system 602 or the server computing system 604 can train the models 620, 640 via interaction with the training computing system 606 that is communicatively coupled over the network 630. The training computing system 606 can be separate from the server computing system 604 or can be a portion of the client computing system 602.

[0056] The content provider computing system 608 includes one or more processors 672 and a memory 674. The one or more processors 672 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 674 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 674 can store data 676 and instructions 678 which are executed by the processor 672 to cause the content provider computing system 608 to perform operations.

[0057] In some implementations, the content provider computing system 608 includes or is otherwise implemented by one or more server computing devices. In instances in which the content provider computing system 608 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0058] Content provider computing system 608 can include database 680. Database 680 can store content element data 681. Content element data 681 can include content elements, asset groups, or other content related data.

[0059] Content provider computing system 608 can be communicatively connected over network 630 to server computing system 604. In some instances, content provider computing system 608 can be a first party computing system associated with the server computing system 604. In some instances, content provider computing system 608 can be associated with a third-party content provider (e.g., advertiser). There can be more than one content provider computing system 608.

[0060] The search engine computing system 610 includes one or more processors 682 and a memory 684. The one or more processors 682 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 684 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 684 can store data 686 and instructions 688 which are executed by the processor 682 to cause the search engine computing system 610 to perform operations.

[0061] In some implementations, the search engine computing system 610 includes or is otherwise implemented by one or more server computing devices. In instances in which the search engine computing system 610 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof. [0062] Search engine computing system 610 can include search index 690 comprising an index that can be parsed responsive to receipt of a user input query. The search engine computing system 610 can generate search results from search index 690 using additional data obtained from client computing system 602, server computing system 604 and/or content provider computing system 608.

[0063] The training computing system 606 includes one or more processors 652 and a memory 654. The one or more processors 652 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 654 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 654 can store data 656 and instructions 658 which are executed by the processor 652 to cause the training computing system 606 to perform operations. In some implementations, the training computing system 606 includes or is otherwise implemented by one or more server computing devices.

[0064] The training computing system 606 can include a model trainer 660 that trains the machine-learned models 620, 640 stored at the client computing system 602, the server computing system 604, the content provider computing system 608, or the search engine computing system 610 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0065] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 660 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0066] In particular, the model trainer 660 can train the models 620, 640 based on a set of training data. The training data can include, for example, historic signal data, publisher- rendered native content item data, user input data, conversion data, user device location data, click data, or any other relevant data (e.g., data stored in database 680, and the like). [0067] In some implementations, if the user has provided consent, the training examples can be provided by the client computing system 602. Thus, in such implementations, the models 620, 640 provided to the client computing system 602 can be trained by the training computing system 606 on user-specific data received from the client computing system 602. In some instances, this process can be referred to as personalizing the model.

[0068] The model trainer 660 includes computer logic utilized to provide desired functionality. The model trainer 660 can be implemented in hardware, firmware, or software controlling a general purpose processor. For example, in some implementations, the model trainer 660 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 660 includes one or more sets of computer-executable instructions that are stored in a tangible computer- readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0069] The network 630 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 630 can be carried via any type of wired or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL).

[0070] The machine-learned models described in this specification may be used in a variety of tasks, applications, or use cases.

[0071] In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

[0072] In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

[0073] In some implementations, the input to the machine-learned model(s) of the present disclosure can be speech data. The machine-learned model(s) can process the speech data to generate an output. As an example, the machine-learned model(s) can process the speech data to generate a speech recognition output. As another example, the machine-learned model(s) can process the speech data to generate a speech translation output. As another example, the machine-learned model(s) can process the speech data to generate a latent embedding output. As another example, the machine-learned model(s) can process the speech data to generate an encoded speech output (e.g., an encoded or compressed representation of the speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, the machine-learned model(s) can process the speech data to generate a prediction output.

[0074] In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.). The machine-learned model(s) can process the latent encoding data to generate an output. As an example, the machine-learned model(s) can process the latent encoding data to generate a recognition output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent encoding data to generate a search output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent encoding data to generate a prediction output.

[0075] In some implementations, the input to the machine-learned model(s) of the present disclosure can be statistical data. Statistical data can be, represent, or otherwise include data computed or calculated from some other data source. The machine-learned model(s) can process the statistical data to generate an output. As an example, the machine-learned model(s) can process the statistical data to generate a recognition output. As another example, the machine-learned model(s) can process the statistical data to generate a prediction output. As another example, the machine-learned model(s) can process the statistical data to generate a classification output. As another example, the machine-learned model(s) can process the statistical data to generate a segmentation output. As another example, the machine-learned model(s) can process the statistical data to generate a visualization output. As another example, the machine-learned model(s) can process the statistical data to generate a diagnostic output.

[0076] In some cases, the machine-learned model(s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g. input audio or visual data).

[0077] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[0078] In some cases, the input includes audio data representing a spoken utterance and the task is a speech recognition task. The output may comprise a text output which is mapped to the spoken utterance. In some cases, the task comprises encrypting or decrypting input data. In some cases, the task comprises a microprocessor performance task, such as branch prediction or memory address translation.

[0079] Figure 4A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the client computing system 602 can include the model trainer 660 and the training data. In such implementations, the models 620, 640 can be both trained and used locally at the client computing system 602. In some of such implementations, the client computing system 602 can implement the model trainer 660 to personalize the models 620, 622 based on user-specific data.

[0080] Figure 4B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0081] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0082] As illustrated in Figure 4B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0083] Figure 4C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0084] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0085] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0086] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0087] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken, and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0088] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, and equivalents.

[0089] The depicted or described steps are merely illustrative and can be omitted, combined, or performed in an order other than that depicted or described; the numbering of depicted steps is merely for ease of reference and does not imply any particular ordering is necessary or preferred.

[0090] The functions or steps described herein can be embodied in computer-usable data or computer-executable instructions, executed by one or more computers or other devices to perform one or more functions described herein. Generally, such data or instructions include routines, programs, objects, components, data structures, or the like that perform particular tasks or implement particular data types when executed by one or more processors in a computer or other data-processing device. The computer-executable instructions can be stored on a computer-readable medium such as a hard disk, optical disk, removable storage media, solid-state memory, read-only memory (ROM), random-access memory (RAM), or the like. As will be appreciated, the functionality of such instructions can be combined or distributed as desired. In addition, the functionality can be embodied in whole or in part in firmware or hardware equivalents, such as integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or the like. Particular data structures can be used to implement one or more aspects of the disclosure more effectively, and such data structures are contemplated to be within the scope of computer-executable instructions or computer-usable data described herein.

[0091] Although not required, one of ordinary skill in the art will appreciate that various aspects described herein can be embodied as a method, system, apparatus, or one or more computer-readable media storing computer-executable instructions. Accordingly, aspects can take the form of an entirely hardware embodiment, an entirely software embodiment, an entirely firmware embodiment, or an embodiment combining software, hardware, or firmware aspects in any combination.

[0092] As described herein, the various methods and acts can be operative across one or more computing devices or networks. The functionality can be distributed in any manner or can be located in a single computing device (e.g., server, client computer, user device, or the like).

[0093] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, or variations within the scope and spirit of the appended claims can occur to persons of ordinary skill in the art from a review of this disclosure. For example, one or ordinary skill in the art can appreciate that the steps depicted or described can be performed in other than the recited order or that one or more illustrated steps can be optional or combined. Any and all features in the following claims can be combined or rearranged in any way possible.

[0094] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Numerous other embodiments, modifications, or variations within the scope and spirit of the appended claims can occur to persons of ordinary skill in the art from a review of this disclosure. Any and all features in the following claims can be combined or rearranged in any way possible. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Lists joined by a particular conjunction such as “or,” for example, can refer to “at least one of’ or “any combination of’ example elements listed therein, with “or” being understood as “and/or” unless otherwise indicated. Also, terms such as “based on” should be understood as “based at least in part on.”

[0095] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, or equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations, or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure covers such alterations, variations, or equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results; determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result; responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data; and populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

2. The computer-implemented method of claim 1, wherein the input further comprises bandwidth access data, and wherein a number of permitted transmitted requests to the content item selection engine is determined based on the bandwidth access data.

3. The computer-implemented method of claim 1, wherein the content items are selected using the content item selection engine based on the query context data.

4. The computer-implemented method of claim 1, wherein the machine-learned model comprises a large language model.

5. The computer-implemented method of claim 1, wherein the user context data comprises at least one of: (i) historical data or (ii) user preference data.

6. The computer-implemented method of claim 1, wherein query context data comprises at least one of: (i) the query, (ii) an intent of the query, or (iii) one or more subintents of the query.

7. The computer-implemented method of claim 6, wherein the intent of the query comprises at least one of: (i) information seeking or (ii) purchase seeking.

8. The computer-implemented method of claim 1, wherein data associated with a first search result of a plurality of search results comprises at least one of (i) content of the search result, (ii) associated webpage, or (iii) an association between the query and the search result.

9. The computer-implemented method of claim 1, comprising: obtaining interaction data indicative of a user interaction with one or more content items populated in one or more content item slots; generating a label for each set of interaction data; and training the machine-learned model based on the labeled interaction data.

10. A computing system comprising: one or more processors; and one or more memory devices storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results; determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result; responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data; and populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

11. The computing system of claim 10, wherein the input further comprises bandwidth access data, and wherein a number of permitted transmitted requests to the content item selection engine is determined based on the bandwidth access data.

12. The computing system of any of claim 10 or claim 11, wherein the content items are selected using the content item selection engine based on the query context data.

13. The computing system of any of claim 10 to claim 12, wherein the machine- learned model comprises a large language model.

14. The computing system of any of claim 10 to claim 13, wherein the user context data comprises at least one of: (i) historical data or (ii) user preference data.

15. The computing system of any of claim 10 to claim 14, wherein query context data comprises at least one of: (i) the query, (ii) an intent of the query, or (iii) one or more sub-intents of the query.

16. The computing system of any of claim 10 to claim 15, wherein the intent of the query comprises at least one of: (i) information seeking or (ii) purchase seeking.

17. The computing system of any of claim 10 to 16, wherein data associated with a first search result of a plurality of search results comprises at least one of (i) content of the search result, (ii) associated webpage, or (iii) an association between the query and the search result.

18. One or more transitory or non-transitory computer-readable media storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: obtaining, by a machine-learned model, input data comprising: (i) user context data, (ii) query context data, and (iii) data associated with a first search result of a plurality of search results; determining, by the machine-learned model, for each search result of the plurality of search results: (i) a determination of authorization to display a content item adjacent to the search result or (ii) a number of content item slots to generate adjacent to the search result; responsive to determining a number of content item slots to generate adjacent to the first search result, generating one or more content slots adjacent to the first search result based on the input data and determined output data; and populating the one or more content item slots adjacent to the first search result by transmitting a request for content items to a content item selection engine and receiving one or more content items.

19. The one or more transitory or non-transitory computer-readable media of claim 18, wherein the input further comprises bandwidth access data, and wherein a number of permitted transmitted requests to the content item selection engine is determined based on the bandwidth access data.

20. The one or more transitory or non-transitory computer-readable media of claim 18 or 19, wherein the content items are selected using the content item selection engine based on the query context data.