EP4695701A1 - Augmenting large language models understanding of web information with factual and subjective location based data - Google Patents

Abstract

Provided are systems and methods for improving response to user requests by providing structured location data associated with the request as context to a large language model. A request response system can respond to a user's request with information relevant to their request. To efficiently and accurately respond to these user requests, the request response system can use a large language model to produce synthetic responses that are more accurate and useful than conventional query response techniques. In some examples, the requests are associated with particular locations. In order to enable the machine learning language model to accurately provide responses to location-based requests, the user response system can, prior to providing input to the machine learning model, access the database of structured location data, retrieve relevant location data, and generate a model input that includes the user request and the retrieved structured location data.

Description

AUGMENTING LARGE LANGUAGE MODELS UNDERSTANDING OF WEB

INFORMATION WITH FACTUAL AND SUBJECTIVE LOCATION BASED DATA

PRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application 63/501,126 having a filing date of May 9, 2023, which is incorporated by reference herein.

FIELD

[0002] The present disclosure relates generally to large language models. More particularly, the present disclosure relates to improving the accuracy and usefulness of location-based search results by using a multi-model system to access structured location data for use as context by a machine-learned large language model.

BACKGROUND

[0003] Services that provide information in response to user requests can use a variety of information to provide the most accurate and useful information possible. One technology that has improved the ability of such sendees is the use of machine-learned large language models. Existing large language models can struggle to provide satisfactory results for requests that are associated with a particular location.

SUMMARY

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] One example aspect of the present disclosure is directed to a computer- implemented method. The method includes receiving, by a computing system with one or more processors, a user request, wherein the user request has an associated location. The method includes accessing, by the computing system, location-based data responsive to the user request from a location-based database. The method includes generating, by the computing system, model input, the model input including the user request and the locationbased data responsive to the user request. The method includes providing, by the computing system, model input as input to a machine-learned language model. The method includes receiving, by the computing system, a request response as an output of the machine-learned language model processing the model input. The method includes providing, by the computing system, the request response to the user for display.

[0006] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices. [0007] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which: [0009] FIG. 1 depicts a block diagram of an example computing system that uses machine-learned models to respond to user requests with respect to text extracted from an image according to example embodiments of the present disclosure;

[0010] FIG. 2A illustrates an example method for providing responses to user requests according to example embodiments of the present disclosure;

[0011] FIG. 2B illustrates an example method for providing responses to user requests according to example embodiments of the present disclosure;

[0012] FIG. 3 illustrates a system for responding to user requests according to example embodiments of the present disclosure;

[0013] FIG. 4A is an example user interface according to example embodiments of the present disclosure;

[0014] FIG. 4B shows an example interface for navigation application according to example embodiments of the present disclosure;

[0015] FIG. 5 A illustrates an example user interface for displaying the comparison between two locations as output by large language model according to example embodiments of the present disclosure;

[0016] FIG. 5B illustrates an example interface for comparing tw o separate locations along one or more axes according to example embodiments of the present disclosure;

[0017] FIGS. 6A-6D illustrate examples of a user interface for conducting a multiquestion session according to example embodiments of the present disclosure; [0018] FIG. 7 is an example user interface for a navigation application for displaying responses generated by a large language model according to example embodiments of the present disclosure;

[0019] FIG. 8 in an example user interface for a navigation application for displaying responses generated by a large language model according to example embodiments of the present disclosure;

[0020] FIGS 9A-9D can illustrate an example flow within the user interface for displaying results associated with a request to a location application according to example embodiments of the present disclosure;

[0021] FIG. 10 is an example user interface for presenting responses to a user request in a navigation application according to example embodiments of the present disclosure;

[0022] FIG. 11 depicts a block diagram of an example computing device that performs according to example embodiments of the present disclosure;

[0023] FIG. 12 depicts a block diagram of an example computing device that performs according to example embodiments of the present disclosure; and

[0024] FIG. 13 depicts an example flow diagram for a method of using structured location data to improve responses to user requests according to example embodiments of the present disclosure.

[0025] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.

DETAILED DESCRIPTION

[0026] Generally, the present disclosure is directed towards systems and methods for improving response to user requests by providing structured location data associated with the request as context to a large language model. In particular, a computing system can provide a service for responding to user requests (such as queries, information requests for particular locations, and so on). A request response system can respond to a user’s request with information relevant to their request. To efficiently and accurately respond to these user requests, the request response system can use a large language model to produce synthetic responses that are more accurate and useful than conventional query response techniques. In some examples, the requests are associated with particular locations. In order to enable the machine learning language model to accurately provide responses to location-based requests, the user response system can, prior to providing input to the machine learning model, access the database of structured location data, retrieve relevant location data, and generate a model input that includes the user request and the retrieved structured location data.

[0027] The machine-learned model (e.g., a large language model) can use the query and the context of the associated location structured data to generate a response (or a list of candidate responses) to a user request. The generated response(s) can include relevant material from the structured location data as well as general data on which the machine- learned language model was trained. Using the structured location data as context, the output of the machine-learned language model can incorporate location-specific information to provide better responses to the user request. The output of the machine learning model can be presented to the user in response to that user's request.

[0028] For example, a user can submit a request that includes the question "‘Where should I go on a sunny day in Greenville?” The user computing device can transmit the request to the request response system. The request response system can be located at a remote server system and the request can be transmitted via a computer network. The request response system can analyze the content of the request as well as any location data provided by the user computing device to identify a location associated with the query. In this example, the request response system can determine that the request is associated with Greenville South Carolina based on GPS data provided by the user computing system (with the user’s permission). The request response system can access a database of structured location data to retrieve information about the location (in this case Greenville. South Carolina) as well as any information about the query (e.g., locations that may be appropriate when the sun is shining in that location). In some cases the request can include multiple modalities of data, such as a combination of image, text, and/or audio.

[0029] The request response system can generate an input to the model. The input can be a prompt. The prompt can include the user request, the structured location data associated with the request, as well as additional instructions that may be appropriate to generate the requested output. For example, the additional instructions may direct the machine-learned model to include citations to the source of any information presented in a candidate response. In some examples, the instructions can indicate a format for the output. For example, if the user requests a comparison between two locations, the instructions may indicate that the results should be displayed in a table. In some examples, the machine-learned model can output the results in markup language (or other structured format) and the content can be displayed using the format indicated in the markup language. In some examples, the instructions can include the specific factors or axis on which a comparison can be made. In other examples, the output of the machine-learned model can include a determination on which factors of axes should be used for a comparison.

[0030] The machine-learned model can analyze the input and generate an appropriate response. For example, the output can be a ranked list of potential locations that may satisfy the user’s request. The ranked list can include objective and subjective data included in the structured location data. The ranked list can then be returned to the requesting user and presented for their review.

[0031] In some examples, the query can be directed towards information that is difficult for large language models to properly produce. For example, if the query⁷ is directed towards a topic which requires specific information about a location (e.g., address, opening time, prices, user ratings, and so on), machine-learned models may have trouble accurately producing this information without errors (e.g., hallucinations). These errors can sometimes occur when the output of the machine-learned model has the correct form but is unable to provide the specific information needed for a specific response due to training data that is out of date or which includes many examples of the type of information that should be returned by few examples of the correct information for a particular location or time.

[0032] One way to reduce the likelihood of factual error in the model output is to use an additional machine-learned model to provide accurate contextual information for use by the main machine-learned model. This additional machine-learned model can be referred to as a research model. The research model can be trained to output a series of database queries based on the prompt received based on a user query.

[0033] The research model can determine which database should be queried and the specific database query to use for each specified database. For example, potential databases can be a structured database of geographic data. This structured database can be used primarily for a navigation service and thus includes a large amount of factual data for many locations. In addition to the structured database of geographic data, the research model can also access a database of subjective user data. This database can include user ratings, reviews, information about previous searches or queries, user movement data, and so on.

[0034] The query response model can include a post processing model that can combine the data retrieved from each database (based on the generated database queries from the research model). In some examples, the post-processing model can associate structured geographic data with the associated subject user data before it is provided to the response model. [0035] In some examples, employing a two-tiered system (e.g., with a research model and a response model) to respond to user queries can consume more resources (e.g., power and processor time) than a typical response model. As a result, the query response system may not wish to employ the two-tiered system for all queries. Instead, the query response system can determine (e.g., using a classifier or other machine-learned model) to determine, for each query, whether the need for accuracy in the response is sufficient enough to employ the more resource intensive multi-tiered system.

[0036] If so, the query can be provided to the multi-tiered system to ensure accuracy when required. If not, the query can be provided directly to the response model to provide a response without the extra expenditure or resources.

[0037] More specifically, a user computing device can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console, a gaming controller, a wearable computing device, an embedded computing device, or any other ty pe of computing device. In some examples, the user computing device can include a network communication device for communicating via a computer network.

[0038] In some examples, a server system can be any computing system available to receive requests over a computing network. For example, a server system can provide navigation services and/or web search services upon request. User computing systems can transmit a user request via one or more computer networks and receive responses from the server computing system via those same computer networks. In some examples, the format of the request is defined by an API of the server system. In other examples, a web interface allows users to submit natural language requests without having to format the requests in accordance with the specific API.

[0039] The server computing system can provide a user request response system. User requests can include search queries, navigational queries, itinerary requests, location queries, and so on. The server computing system can respond to a user request with a natural language answer to a query’, a list of candidate results, navigational instructions, an itinerary, or any other appropriate result.

[0040] In some examples, to provide this service, the server computing system can include a large language model that is machine-trained to receive natural language queries from users and output synthetic natural language responses. The large language model can be trained on a large corpus of data such that the model is able to provide useful and accurate information with a conversational or natural language tone response. Indeed, the large language model can also use previous queries from the user and responses from the model as input. Using past entries in the conversation as context enables the large language model to correctly respond to a series of questions by the user (even questions that reference previous questions from the user or responses by the model).

[0041] To improve the output of the machine learning model, the request response system can provide relevant data to the model as context and/or background. For example, if a request is received from a user, the request response system can access one or more databases that contain relevant data. This data can be retrieved and provided as context and background to the machine-learned large language model. For example, if the user request is associated with a particular location or set of locations, the request response system can access a database of structured location data.

[0042] The structured location data can be first person data collected, stored, and maintained by the request response system (or an associated organization). Alternatively or additionally, the structured location data can include user-generated content such as user- submitted textual reviews, user-submitted photographs, or the like. In another example, the structured location data can include structured data received or obtained from various third- party sources of information, such as general web content. The structured data can be stored in a knowledge graph, a relational database, or any other appropriate method for storing location data. In some cases the structured location data can include multiple modalities of data, such as a combination of image, text, and/or audio.

[0043] The request response system can access one or more databases of structured location content to help ensure that the machine-learned model can provide vetted factual knowledge about the particular locations the request is associated with. For example, when a request is received from a user, the request response system can analyze that request to determine whether a location is associated with the request. For example, the text of the request may include a reference to a location or the request itself may be accompanied with location data from the user's computing device.

[0044] In response to determining that a location is associated with the request, the request response system can access a database of structured geographic content data and access relevant location data based on the request. For example, if the request is for a seafood restaurant recommendation in a particular city, the request response system can access the database of structured location data and retrieve information about seafood restaurants that are associated with the particular city. This structured location data can be used along with the text of the query itself to generate a prompt for the large language response model. [0045] The request response system can include a second machine-learned model that can be used to identify the location associated with the particular query and generate one or more research queries to request context information from one or more databases (e.g., a database of structured location data). Such a machine-learned model can be referred to as a large language research model and can retrieve additional information from a plurality of sources and other services to use as context when generating model input for the response model. For example, the research model can access structured location data from a location database, weather data from a weather database, web search data from a general search platform, and other search corpuses. All identified background data can be used to generate a prompt for the machine-learned large language response model.

[0046] In some examples, the research queries can involve a multiple step process in which one query needs to be answered before a second query can be answered. For example, if the user submits a request that asks, “What is the population of the capital of North Dakota?”, the research model can first use a query to determine the capital of North Dakota and use that information to determine the correct population amount.

[0047] In some examples, the research model can receive conversational context that represents previous user requests received from the same user as context for generating one or more search queries to search the various databases. Similarly, the research model can be trained using a large volume of previous user requests to generate the proper search queries. [0048] Once the research model has generated, as output, one or more queries for use in searching one or more of the various sub databases, the request response system can execute the queries with the appropriate database or search service. The result of these searches can be the receipt of a variety' of diverse ty pes of data such as structured location data, web result data, weather data, and other data that is structured in a variety of ways.

[0049] The request response system can generate a prompt for the machine-learned large language response model that includes, among other things, the user request itself, past conversation context for the request, the results from the various sub searches, including but not limited to structured location data, and information describing the format and requirements for the response. For example, the request response system can. to ensure that the responses generated by the large language response model are accurate, include a clause in all model prompts that require the model to generate specific references to the source location of any data it provides.

[0050] The large language response model can receive the prompt generated by the request response system as model input. The large language response model can be trained to receive model input of this ty pe and respond with response data that is helpful to the user. It should be noted that the response data can include a combination of factual data stored in databases, subjective data accessed through databases, web searches, and other sources, as well as synthetic text generated by the large language response model based on previous training. In some cases the response data can include multiple modalities of data, such as a combination of image, text, and/or audio.

[0051] In some examples, the request response system can use the large language model system to provide a variety of services to users. One example of the service can include receiving natural language queries and providing natural language responses as output by the large language response model for display to the user. In some examples, this can result in a conversational query and response style in which the user can ask clarifying queries about previous responses and the request response system can provide accurate responses based on the previous statements in the conversation by both the user and the system. As noted above, this can be accomplished in part by providing the context of past conversations to the large language response model as part of a prompt for the current query. [0052] Another example service provided by the request response system can be a request for a category list at a particular location. For example, the user can ask for the best place to throw axes in Wichita. The request response system can access structured location data for Wichita and provide that data as input to the large language response model. The large language response model can then generate a response that combines geographic facts from the structured location data, subjective user experience can be based on user reviews and feedback, and natural language synthetic response responses. This information can be combined into a single response and formatted for display to the user. Thus, the displayed response can contain a mixture of subjective content and objective content. Likewise, the displayed response can contain a mixture of synthetic, model-generated content and factual content. The displayed response can synthesize these different types of sources of data into a single coherent display that represents a bespoke, useful response to the user's request.

[0053] In some examples, the user response system can also provide a place comparison service. In response to a request to compare two places, the large language response model can access the appropriate structured location data and generate a response that compares the two places along a plurality of axes. The axes can be chosen by the large language request model response model as part of the output of the model. In some examples, the output of the large language response model can be a structured format (e.g., in a table) that can present information in a manner that is most useful to the user. In addition, any data included in the response (e.g., factual data, user reviews, ratings, and so on) can also include a citation to the source so users can verify the accuracy of the data provided by the large language response model.

[0054] Another service potentially provided by the request response system can be a sendee that can provide answers to user’s questions about a particular place. The answers can include information derivable from the structured location data such as the hours, phone number, address, services, and products available, and so on. The large language response model can use the structured location data as context to ensure that the output can provide a satisfactory’ answer to virtually any question the user may have about a particular location. [0055] Y et another example of a service provided by the request response system can be itinerary planning. An itinerary planner can access structured location data as well as information searchable on the web (e.g., user feedback provided by users in comment sections or reviews) to identify appropriate information about a proposed plan for the users. [0056] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the system and methods can provide an improved request response system for location-associated requests. In particular, the systems and methods disclosed herein can use existing structured location data to improve the reliability and usefulness of large language models. The addition of structured location data can improve the accuracy of systems that employ these models to provide responses to user requests. Improving the efficiency and accuracy of a request response system based on a large language model can improve user experience, reduce costs associated with these requests, and save time.

[0057] The systems and methods of the present disclosure also enable efficient use of resources. For example, for some user queries, the likelihood of factual errors can be unacceptably high. The research model can enable the response system to reduce the likelihood of errors. However, doing so can incur additional resource usage. For example, the system can use additional power and processor time when using the research model to generate search queries and retrieve data from databases and services not generally included in the training data for the response model. As such, the system can first evaluate each query and determine whether to use the research model to respond. If the system determines that the research model is not needed to provide increased factual accuracy (e.g., the response model itself can reach the desired accuracy without needing additional information), the system can provide the query directly to the response system. Thus, the query’ response system can provide the technical benefit of increasing accuracy when needed and preserving resources when the existing accuracy is already sufficient. Doing so can increase the overall accuracy of the system without too great an increase in the costs of providing the system.

[0058] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.

[0059] FIG. 1 depicts a block diagram of an example computing system 100 that uses machine-learned models to respond to user requests with respect to text extracted from an image according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, and a server computing system 130 that are communicatively coupled over a network 180.

[0060] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0061] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA. a controller, a microcontroller, etc.) and can be one processor or a plurality’ of processors that are operatively connected. The memoty 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 1 12 to cause the user computing device 102 to perform operations. [0062] In some implementations, the user computing device 102 can store or include one or more models 120 for responding to user requests by accessing a machine-learned model and structured location content. In some implementations, the user computing device 102 can store or include one or more models 120. In some implementations, the user computing device 102 can store or include one or more models 120. For example, the models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks), large language models (LLMs) or other types of machine- learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models (e.g., transformer models). [0063] In some implementations, the one or more models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single model 120 (e.g., to perform parallel optimization for large language models across multiple instances of the models 120).

[0064] More particularly, model(s) 120 can. in some implementations, include a machine-learned large language model. The machine-learned large language model can be, or otherwise include a model that has been trained on a large corpus of language training data in a manner that provides the machine-learned large language model with the capability to perform multiple language tasks. For example, the machine-learned large language model can be trained to perform query response tasks, comparison tasks, planning tasks, conversational question and answer tasks, and list generation tasks, etc. In particular, the machine-learned large language model can be trained to process a variety of inputs to generate an appropriate response to a user request. For example, the machine-learned large language model can process a model input that can include, but is not limited to. a user response request, associated structured location data, contextual data, and one or more instructions.

[0065] More particularly, in some environments, machine learning models 120 can include two distinct models. The first model is a research model. The large language research model can be trained to receive a user request and accompanying location data as import. Based on the input, the large language research model can generate one or more queries for associated databases. For example, the databases can include a database of location data such as map facts, a weather database, a web search service or database, and other search databases. The search queries generated by the large language research model can then be executed by the user response system to receive responses to the queries from the associated services. For example, other queries can result in structured location data, web result data, weather data, and so on.

[0066] The second model can be a response model. The response model can receive model input that includes a prompt generated by the request response system. The prompt can be created such that it includes the user request, the results of any searches generated by the research model, including but not limited to structured location data, previous contacts associated with the current request, as well as instructions used to ensure the output is acceptable to the request response system. [0067] The structured location data can be structured as a knowledge graph, a relational database, or any other data structure useful for storing and retrieving location information. In some examples, the structured location data can include facts such as a location of a place, the hours of operation, the services and products available at that location, a category of services available at that location, and so on. The structured location data can also include subjective information. Subjective information can include user reviews, rankings, feedback, and other information submitted by users either directly to the structured location database or received via a web query.

[0068] The response model can generate an output based on the model input. In some examples, the model output can have a combination of objective data as well as subjective data. In some examples, the output can be structured to display results in our format. In some examples, the output can be a list of relevant locations to the query, a comparison between two locations, answer to a uery about a location, or our planned itinerary based on a request. Other services and output types can be generated as needed.

[0069] In some examples, the output of the research model can be formatted such that it can be displayed in a web page or in an application. This information can be transmitted to the user computing system for display.

[0070] Additionally, or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 140 as a portion of a web sendee (e.g., a service providing responses to user requests). Thus, one or more models 120 can be stored and implemented at the user computing device 102 and/or one or more models 140 can be stored and implemented at the server computing system 130.

[0071] The user computing device 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g.. a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0072] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory' 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0073] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the serv er computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0074] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models).

[0075] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

[0076] The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

[0077] In some implementations, the input to the machine-learned model(s) of the present disclosure can be one or more of: the text of a user request, retrieve structured location data associated with the request, previous requests or responses from the user or the system that are contextually important for the request, additional instructions from the request response system that provide ensure the output of the model is acceptable. [0078] In some examples the output of the machine learning model can be text responsive to a particular query, one or more candidate locations that maybe appropriate for the user request, comparison between one or more locations, specific information responsive to a query the user had about a particular location, and one or more proposed itineraries based on the user request.

[0079] FIG. 1 illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the models 120 can be both trained and used locally at the user computing device 102.

[0080] FIG. 2A illustrates an example method for providing responses to user requests according to example embodiments of the present disclosure. In this example, the request response system can receive a request at 202 (e.g., a query or other request). The large language model can accept the query as input. The large language model can then process the query to understand the query and identify one or more characteristics of the query. For example, a research model 204 can enable the system to understand the context of the query 202. The output of the research model (e.g.. a specifically trained large language model) can include one or more topics 206 associated with the query. In some examples, the output of the research model can also include inputs to other services or databases such as services or databases that provide access to web snippets, place snippets, structured location data, user reviews, geographic data lists, and so on. The request response system can use those topics and the inputs to request data from one or more other services. For example, the request for websites snippets can be sent to a search engine or other method that provides results associated with the web. The structured geographic data inputs can be sent to a database of structured location data. The structured location database can provide data associated with the generated input.

[0081] In some examples, once the generated list of topics 206 and the supplemental or context data has been received, a post processing model can determine the top inputs 210 based on ranking of the topics to determine which data to provide as context with the query’. [0082] The selected information can be included as input to a large language model. The large language model can generate a response that summarizes 212 the received input and its own general knowledge into a model output. The model output can include information describing a particular location or itinerary of multiple locations. [0083] The query response system can then process 214 the output and generate a user interface for displaying the information. This information can then be transmitted to the user or system that submitted the query or request.

[0084] FIG. 2B illustrates an example method for providing responses to user requests according to example embodiments of the present disclosure. In this example, a query⁷ 252 is provided to a query⁷ response system. The query response system can include a query evaluation system 250.

[0085] The query' evaluation system 250 can determine, based on the query, characteristics of the user, and any instructions received from the user along with the query', whether the response model is likely to make a factual error responding to the query. In accordance with a determination that the likelihood that the response model will make a factual error when responding to the query⁷ exceeds a threshold value (e.g., more than 5%), the query⁷ evaluation system 250 can cause the query to be multi-model response model 262. [0086] The multi-model response system 262 can include a research model that collects additional supplemental data associated with the query. This data can be collected from a plurality of databases and the research model can generate queries to retrieve the appropriate data from particular databases. The supplemental data can be provided to a response model as part of a prompt. The prompt can also include the query⁷ and any other information necessary to respond to the query. For example, the prompt can include information describing previous queries from the user and so on.

[0087] If the query evaluation system 250 determines that the likelihood of the response model making a factual error does not exceed a threshold (e.g., less than 5% or another value), the query evaluation system 250 can provide the query⁷ as input to the single model response system 260. In this context the single model response system 260 is a response system that does not include a research model. However, other machine-learned models may be used to provide other functionality. Thus, the single model response system 260 does not employ a research model to generate queries to gather appropriate factual data from other databases or services.

[0088] FIG. 3 illustrates a system for responding to user requests according to example embodiments of the present disclosure. A user can submit a request 320. In this example, the request 302 is “plan me a date night in Chelsea NYC this weekend.” The user request response system 310 can use that request as input to a multi-tier large language model. The multi-tier large language model can include a research model 312. The research model 312 can process the request 302 to generate a series of further search queries (314-1 to 314-5). The queries (314-1 to 314-5) can be used to retrieve data from a variety of other services or databases. In this example, the additional queries can access a local search service 316-1, a weather search service 316-2, a web search 316-3, and one or more other search corpuses 314-4. In some examples, a local search system can access a database that includes structured location content. The structured location content can be information about locations organized in the hierarchy or in other data structures. For example, structured location data can be stored in a knowledge graph or other data structure for storing location data.

[0089] Each query can be used to receive additional information from the databases or sendees. The additional information (e.g., 318-1 to 318-5) can provide context and factual grounding for responding to the user request 302. A post processing model 324 can process the output of each service. The post processing model 324 can determine the specific results that are useful for the output large language response model to respond to the request. [0090] The post processing model 324 can provide the received data to the large language response model 320. The large language response model 320 can also access conversation context 322 from the past history’ that the user has with the request response system. In some examples, the request response system can format a prompt as model input to the large language response model. The prompt can include any structured location data or other search result that was generated by the queries of the large language research model, the user request itself, and any other instructions that may be necessary to perform an accurate response to the user's request.

[0091] In some examples, the large language response model 320 can take the model input as input. The model input can be processed to generate an appropriate response to these requests. In this example, the request is for date night in a particular location on a particular date. The output 326 can be a suggestion for a particular date and activity at a particular location and time. In some examples, the output of the large language response model 320 can include a list of potential candidate itineraries and the user can select the candidate itinerary’ they prefer based on that information provided by the response model. The output can include natural language, synthetic language generated by the model, factual data received from the structure location data, and subjective data generated by users. In some examples, the response can include a combination of these types of data. In some examples, the model training and data pipeline 311 can provide information for training the research model 312 and the LLM response model 320.

[0092] FIG. 4A is an example user interface 400 according to example embodiments of the present disclosure. In this example, a user interface 100 is displayed. The user interface can display information from a map application for use on the user computer device. In this example, the user is able to input a request 402 in the user input field. In some examples, the user can input the request via an input device (e.g., a keyboard, a touch screen that accepts input, a microphone that can receive spoken commands and so on).

[0093] Once the user has input the request 402 into the input field, the user computing device can transmit the request to the request response system including a machine-learned large language model. As discussed above, the machine-learned large language model can access structured location information from a database or other sources and provide that information to the machine-learned large language model.

[0094] Using the user's request and the structured location data as input, the large language model can generate an output 404. In this example, the user request is a recommendation for a place to stay in Myrtle Beach. The response generated by the machine- learned large language model can include a summary of the location 404 and one or more candidate results (406-1, 406-2, and 406-3). Each candidate result can include the name of the location, subjective information such as star reviews, a list of one or more accommodations, and a written description of the location. In some examples, each result can include a combination of factual information, subjective information, and synthetic information generated by the large language model.

[0095] The output of the large language model can be formatted for display in a particular user computing device and transmitted to the user. The user computing device can then display the information to the user and give the user the option of selecting a result or responding to the output with another request for information. In some examples, the output can also include a visual indication of the location of one or more results in a map format 408.

[0096] FIG. 4B shows an example interface 450 for navigation application according to example embodiments of the present disclosure. In this example, the user submits a request 452 for a recommendation for a place to eat while visiting New York. The response 454 generated by the machine-learned model includes a description of the different categories of restaurants available and a recommended restaurant from each category (e.g., 456-1. 456-2, and 456-3).

[0097] For example, the description generated by the large language model indicates that there are acclaimed eateries, classic restaurants, and buzzy hotspots. One of each type of restaurant is included in the list of proposed eating locations. In some examples, the user interface can include a map 458 that displays the locations of one or more recommended locations.

[0098] FIG. 5A illustrates an example user interface 500 for displaying the comparison between two locations as output by large language model according to example embodiments of the present disclosure. In this example, the user interface has a field that a user can use to input a request 502. In this example, the user asks the system to compare the two locations to determine which is better for families with young kids.

[0099] The system can analyze the request 504 and, based on this analysis, retrieve structured location data from a location database. The structured location data and the request can be provided to a machine-learned large language model. In this example, the model generates a response 504 (as output). The interface can include two options to be compared (e.g., 506-1 and 506-2).

[0100] The output can include a brief recommendation of which resort may be better as well as a comparison between the two locations along several axes. The information can include information retrieved from a structured database and is thus highly likely to be accurate. The information can also include subjective user data (e.g., ratings) integrated into the response.

[0101] In this example, the axes can include the price, the ratings, the water attractions, other amenities, and dining opportunities. In some examples, the specific axis along which the locations are compared are selected as output from the machine-learned model. The output of the machine-learned model may also include formatting information to allow the response to be shown in as a table in the user interface.

[0102] FIG. 5B illustrates an example interface 550 for comparing two separate locations along one or more axes according to example embodiments of the present disclosure. In this example, the user requests 552 that the system determine what the prices are like at the first location (556-1) and the second location (556-2). The large language model can use the request as input as well as any previous messages and any relevant location data.

[0103] The large language response model can, based on the model input, generate an output 554. In this example, the output compares two locations (556-1 and 556-2) along several axes with respect to price. The data can be displayed in a structured table format 558. The responses include factual content such as the cost for various courses as well as subjective content such as star ratings based on user reviews. In some examples, the responses also include images relevant to the locations being compared. The responses also include information generated by the large language model itself. This generated information can be referred to as synthetic content.

[0104] FIGS. 6A-6D illustrate examples of a user interface for conducting a multiquestion session according to example embodiments of the present disclosure. In the user interface 600 displayed in FIG. 6, the initial request 602 from a user in 6A is to recommend a neighborhood in Las Vegas for the user to stay at. The large language response model generates response 604. The response can include a list of neighborhoods in Las Vegas at which to stay. The generated interface also includes images and information 608 for those locations as well as a map 606 with pictographic images showing where the specific hotels are located.

[0105] In FIG. 6B. the interface 610 includes a user request in which the user has selected a particular neighborhood, (the strip), and now asks ‘'what is the best hotel on the strip?” 612 The previous answer can be used as context for the current request 612. In this way, the large language response model is able to identify that the strip in question is the strip in Las Vegas. In response to the query, the large language response model can generate a response as a model output 614. The response can include a list of available hotels (616-1 to 616-3) organized by type. Each hotel can be listed with objective information, the cost, and subjective information, the user ratings as well as descriptions that can be drawn at least in part, from user reviews.

[0106] In FIG. 6C. the interface 620 can receive a user request 622 that prompts the system to compare two of the candidate hotels from FIG. 6B to determine which is more suitable for a family with young children. Using the context from the previous question as shown in FIG. 6B, the large language model can generate a response 624. The response includes information describing why either the first option 626 or the second option 628 may be preferable. This information can be presented to the user in a summary of information found for each potential response.

[0107] In FIG. 6D, the interface 630 includes a user request in which the user submits a specific question about the Excalibur. Specifically, the user submits a question 632 to determine whether that hotel includes a pool. Using context from previous questions as well as the appropriate location data retrieved from a database and used as context, the large language response model can provide a response 634. The response can include information that indicates whether the pool at the hotel is appropriate for children. In some examples, the large language model can select specific user reviews 638 that mention the pool on whether it is kid friendly. This information (including snippets of the reviews) can be presented to the user along with images 636 and synthetic text 634 generated by the large language module. [0108] FIG. 7 is an example user interface 700 for a navigation application for displaying responses generated by a large language model according to example embodiments of the present disclosure. In this example, the navigation application includes a field in which a request 702 can be submitted. In this example, a request 702 can be submitted via text or via vocal input. The request can be analyzed to identify one or more associated locations. The navigation application can access location data for the associated locations. The location data and the request can then be used as input to a machine learned model. The machine learning model can output an appropriate response to the request. In this example, the user request is a request for an itinerary for a Mother's Day in Chicago that is vegetarian and kid friendly.

[0109] The output generated by the machine learning model includes a brief description 704 of the response generated by the machine learning model, a map that indicates the location of each potential activity, and a list of activities 706 along with associated times that can be displayed are recommended to the user. The navigation application can format the information to display in the user interface. In addition, additional elements of the interface, such as a like or dislike button, an option to edit the itinerary 708, and an option to book the recommended activities can be presented in the interface.

[0110] FIG. 8 in an example user interface 800 for a navigation application for displaying responses generated by a large language model according to example embodiments of the present disclosure. In this example, the user submits a request 802 that the navigation application provides recommendations for a location to take a child in Chicago that likes comic books. In this case, the location is determined to be Chicago and the activity is determined to be comic book associated locations or activities.

[0111] The navigation application can retrieve location information associated with Chicago and/ or comic book associated activities. This location information along with the user request 802 can be used as input to the machine learning model. In this example, the output from the machine-learned model can indicate a plurality of locations that may be suitable for children who like comic books in Chicago. Each location can be represented by an image (e.g., 806 and 810) along with a brief description generated by the machine learning model as well as one or more user comments or reviews (808 and 812). User comments and reviews can be identified as appropriate for determining whether a location is suitable for a child who likes comics in Chicago. In some examples, a map 804 can also be displayed showing the location of each recommended location.

[0112] FIGS 9A-9D can illustrate an example flow within the user interface for displaying results associated with a request to a location application according to example embodiments of the present disclosure.

[0113] In this example, the navigation application 900 as shown in FIG. 9 A includes a user input field 904 that allows a user to input a request. The user interface can also include a map 904. The user can select the search field to initiate the process of submitting a user request.

[0114] The user interface in FIG. 9B 810 is updated as the user inputs the text of their request 912. In FIG. 9C the user interface 920 can display one or more candidate locations as a response 922 for review by the user. The user can select one of the candidate locations and the user interface 930 can be updated as seen in FIG. 9D to display more information about the location 932, and images associated with the patient, generated text associated with the location, as well as one or more user comments or reviews 934 appropriate to the risk request and the location. This information can allow the user to more clearly understand the suitability of each location for their request.

[0115] FIG. 10 is an example user interface 1000 for presenting responses to a user request in a navigation application. The user can submit a request 1002 requesting information on how to spend a rainy day in Austin. In response the machine-learned model can generate one or more candidate locations (1004). Each candidate location can be represented in the interface using images, subjective data like user ratings 1006, objective data about its location at the time to travel there, as well as user feedback previously received. [0116] FIG. 11 depicts a block diagram of an example computing device 1110 that performs according to example embodiments of the present disclosure. The computing device 1110 can be a user computing device or a sen' er computing device.

[0117] The computing device 1110 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine- learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0118] As illustrated in Figure 11, each application can communicate with a number of other components of the computing device, such as. for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0119] FIG. 12 depicts a block diagram of an example computing device that performs according to example embodiments of the present disclosure. The computing device can be a user computing device or a server computing device.

[0120] The computing device includes a number of applications (e.g., applications I through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0121] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 12, a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device.

[0122] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository' of data for the computing device. As illustrated in Figure 12, the central device data layer can communicate with a number of other components of the computing device, such as. for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0123] FIG. 13 depicts an example flow diagram for a method of using structured location data to improve responses to user requests according to example embodiments of the present disclosure. One or more portion(s) of the method can be implemented by one or more computing devices such as, for example, the computing devices described herein. Moreover, one or more portion(s) of the method can be implemented as an algorithm on the hardware components of the device(s) described herein. FIG. 13 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, and/or modified in various ways without deviating from the scope of the present disclosure. The method can be implemented by one or more computing devices, such as one or more of the computing devices depicted in FIGS. 1 and 2.

[0124] A computing device (e.g., user computing device 102 in FIG. 1 or server computing system 130) can include one or more processors, memory, and one or more sensors. The user computing device can include other components that, together, enable the computing device to receive user requests and provide responses to those requests.

[0125] To do so, the computing device can receive, at 1302, a user request, wherein the user request has an associated location. The computing device can access, at 1304, locationbased data responsive to the user request from a location-based database. In some examples, the location-based database includes data describing factual information associated with a plurality of locations. The location-based database includes subjective data associated with a plurality of locations.

[0126] The computing device can provide the user request and the associated location to a machine-learned research model as input. The computing device can receive one or more suggested search queries, wherein a respective suggested search uery is associated with a target database. The computing device can retrieve location data from one or more location databases using the one or more suggested search queries.

[0127] The computing device can analyze the user request to identify one or more topics of interest. The computing device can retrieve web results associated with the topic of interest. In some examples, the web results can be included in the model input. The computing device can generate, at 1306. model input, the model input including the user request and the location-based data responsive to the user request.

[0128] Generating a model input can include generating a prompt for the machine- learned language model, the prompt including the user request, the location data, and additional instructions to the machine-learned model. In some examples, the additional instructions include a directive to cite specific sources for information included in a request response.

[0129] The location-based data is structured location data arranged hierarchically. The structured location data includes objective data and subjective data. In some examples, the subjective data includes user reviews and rating data for particular locations. The computing device can provide, at 1308, model input as input to a machine-learned language model. The computing device can receive, at 1310, a request response as an output of the machine- learned language model processing the model input; and

[0130] In some examples, the output of the machine-learned model includes objective data, subjective data, and data generated by the machine-learned model. The user request includes a query⁷, and the request response includes a plurality of query responses and wherein the query results are ordered based on likelihood of usefulness to a user. The query response can include a comparison of more than one candidate response to the user request. The comparison compares the candidate responses along more than one axis and the axes are determined by the machine-learned language model.

[0131] In some examples, the output of the machine-learned model is structured to display a comparison of candidate responses in a table. The computing device can provide, at 1312, the request response to the user for display. For example, the results can be displayed in the user interface of a smartphone, tablet computer, personal computing device, and so on. [0132] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0133] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, the method comprising: receiving, by a computing system with one or more processors, a user request, wherein the user request has an associated location; accessing, by the computing system, location-based data responsive to the user request from a location-based database; generating, by the computing system, model input, the model input including the user request and the location-based data responsive to the user request; providing, by the computing system, model input as input to a machine-learned language model; receiving, by the computing system, a request response as an output of the machine- learned language model processing the model input; and providing, by the computing system, the request response to the user for display.

2. The computer-implemented method of claim 1. wherein accessing location-based data responsive to the user request from a location-based database further comprises; providing, by the computing system, the user request to a machine-learned research model as input; and receiving, by the computing system, one or more suggested search queries, wherein a respective suggested search query is associated with a target database.

3. The computer-implemented method of claim 2, wherein accessing location-based data responsive to the user request from a location-based database further comprises: retrieving, by the computing system, location data from one or more location databases using the one or more suggested search queries.

4. The computer-implemented method of claim 2, wherein the method further comprises, prior to providing the user request to the machine-learned research model: determining, by the computing system and based on the user request, a likelihood value that the machine-learned language model will produce a factually incorrect response without employing the machine-learned research model to retrieve location data; in accordance with a determination that the likelihood value exceeds a predetermined threshold, providing the user request to the machine-learned research model; and in accordance with that the likelihood value does not exceed the predetermined threshold, providing the user request to the machine-learned language model without first providing the user request to the machine-learned research model.

5. The computer-implemented method of claim 1, wherein the user request includes a query and the request response includes a plurality of query responses and wherein the queryresults are ordered based on likelihood of usefulness to a user.

6. The computer-implemented method of claim 1 , wherein the request response includes a comparison of more than one candidate response to the user request.

7. The computer-implemented method of claim 6, wherein the output of the machine- learned model is structured to display a comparison of candidate responses in a table.

8. The computer-implemented method of claim 1. wherein the location-based database includes data describing factual information associated with a plurality of locations.

9. The computer-implemented method of claim 1, wherein the location-based database includes subjective data associated with a plurality of locations.

10. The computer-implemented method of claim 1, further comprising: analyzing, by the computing system, the user request to identify one or more topics on interest; and retrieving, by the computing system, web results associated with the topic of interest.

11. The computer-implemented method of claim 10, wherein the web results are included in the model input.

12. The computer-implemented method of claim 1, wherein generating a model input includes generating a prompt for the machine-learned language model, the prompt including the user request, the location data, and additional instructions to the machine-learned model.

13. The computer-implemented method of claim 12, wherein the additional instructions include a directive to cite specific sources for information included in a request response.

14. The computer implemented method of claim 1, wherein the location-based data is structured location data arranged hierarchically.

15. The computer implemented method of claim 14, wherein the structured location data includes objective data and subjective data.

16. The computer implemented method of claim 15, wherein the subjective data includes user reviews and rating for particular locations.

17. The computer implemented method of claim 16, wherein the output of the machine- learned language model includes objective data, subjective data, and data generated by the machine-learned language model.

18. The computer implemented method of claim 1, wherein one or both of the user request or the request response comprises multiple data modalities.

19. A computer system configured to perform the method of any preceding claim.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any preceding claim.