KR20230003388A - Digital supplement association and retrieval for visual search - Google Patents

Abstract

Systems and methods for identifying and retrieving content for visual search are provided. An exemplary method includes receiving data specifying a digital supplement. The data may identify a digital supplement and a supplement anchor for associating the digital supplement with visual content. The method also includes creating a data structure instance that specifies the digital supplement and the supplement anchor, and after creating the data structure instance, storing the data structure instance in a database that includes a plurality of other data structure instances. and enabling triggering of the digital supplement by an image based at least on Each of the other data structure instances may specify a digital supplement and one or more supplement anchors.

Description

Digital Supplement Association and Search for Visual Search {DIGITAL SUPPLEMENT ASSOCIATION AND RETRIEVAL FOR VISUAL SEARCH}

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority to U.S. Application Serial No. 16/014,520, “Digital Supplement Association and Retrieval for Visual Search,” filed on June 21, 2018, which is incorporated herein by reference in its entirety.

Mobile computing devices, such as smartphones, often include cameras. Such cameras may be used to capture images of entities in the environment around the computing device. Various types of content or experiences associated with these entities may be presented to users via mobile computing devices.

This disclosure describes systems and methods for digital supplement association and retrieval for visual retrieval. For example, the systems and techniques described herein may be used to provide digital supplements, such as augmented reality (AR) content or experiences that respond to visual searches. A visual search may be based, for example, on an image or an entity identified within an image. For example, a digital supplement may include providing information or functionality associated with an image.

One aspect is a computer-implemented method comprising receiving data specifying a digital supplement, the data identifying a digital supplement and a supplement anchor for associating the digital supplement with visual content. The method also includes creating a data structure instance specifying the digital supplement and the supplement anchor. The method further comprises, after creating the data structure instance, enabling triggering of the digital supplement by an image based at least on storing the data structure instance in a database containing a plurality of other data structure instances. do. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Another aspect is a computing device that includes at least one processor and a memory storing instructions. The instructions, when executed by the at least one processor, cause the computing device to receive data specifying a digital supplement, the data including a digital supplement and a supplement anchor for associating the digital supplement with visual content; Identifies contextual information. The instructions also cause the computing device to create a data structure instance specifying the digital supplement, the supplement anchor, and the context information. The instructions enable a computer device to trigger the digital supplement by an image based at least on creating the data structure instance and then storing the data structure instance in a database containing a plurality of other data structure instances. . Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Another aspect is a computer-implemented method comprising receiving a visual content query from a computing device and identifying a supplement anchor based on the visual content query. The method also includes generating an ordered list of digital supplements based on the identified supplement anchors and transmitting the ordered list to the client computing device.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other configurations will become apparent from the following description, drawings and claims.

1 is a block diagram illustrating a system according to an example implementation.
FIG. 2 is a third-person view of an exemplary physical space from which an embodiment of the client computing device of FIG. 1 accesses a digital supplement.
3 is a diagram of an exemplary method for enabling triggering of a digital supplement in accordance with implementations described herein.
4 is a diagram of an exemplary method for enabling triggering of digital supplements in accordance with implementations described herein.
5 is a diagram of an exemplary method for retrieving and presenting digital supplements in accordance with implementations described herein.
6 is a diagram of an example method for identifying and presenting a digital supplement based on an image, in accordance with an implementation described herein.
7A-7C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content searches and displaying digital supplements.
8A-8C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content searches and displaying digital supplements.
9A-9C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content searches and displaying digital supplements.
10A-10C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content searches and displaying digital supplements.
11A-11C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device of FIG. 1 for performing a visual content search within a repository.
12A-12C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 during visual content retrieval.
13 is a schematic diagram of an example of a computer device and a mobile computer device that can be used to implement the techniques described herein.
Detailed reference will be made to non-limiting embodiments of the present invention, examples of which are shown in the accompanying drawings. Examples are described below with reference to the drawings, where like reference numbers refer to like elements. When the same reference number is shown, the corresponding description(s) are not repeated and the interested reader is referred to the above-discussed drawing(s) for description of similar element(s).

This disclosure describes technological improvements that simplify the identification and presentation of digital supplements based on visual content. Some implementations of the techniques described herein create an index of digital supplements associated with particular types of visual content, and provide these digital supplements in response to visual content queries received from client computing devices. This index allows users to access relevant digital supplements provided by network-accessible resources (e.g. webpages) deployed globally. This may provide functional data structures that enable more efficient information retrieval.

For example, a client computing device, such as a smartphone, may capture an image of a supplemental anchor, such as an entity. The client computing device may send an image-based visual content query to the server computing device to retrieve the digital supplement associated with the identified supplement anchor. In some implementations, the supplemental anchor is based on the physical environment surrounding the client computing device, and the digital supplement is virtual content that can supplement the user's experience in the physical environment.

The visual content query may include an image or data determined from the image (eg, an indicator of an identified supplemental anchor). An example of data determined from an image is text extracted from an image, for example using optical character recognition. Other examples of data extracted from an image are a value read from a barcode, QR code, etc. in the image, an identifier or description of the entity, product or entity type identified in the image.

For example, a neural network system such as a convolutional neural network system can be used to identify entities, products, or entity types in images. An identifier or description of an entity, product or entity type may include metadata or a reference to a record in a database associated with the entity, product or entity type. Non-limiting examples of entities include buildings, works of art, products, books, posters, photographs, catalogs, signs, documents (eg business cards, receipts, coupons, catalogs), people, and body parts.

Various types of digital supplements associated with supplemental anchors are available. A digital supplement may be provided by a network accessible resource such as a web page available on the Internet. We need a way to find and deliver these digital supplements in response to visual content queries. Some implementations create and maintain an index of digital supplements associated with entities for use in responding to visual content queries. For example, the index may be populated by crawling network accessible resources to determine whether the network accessible resources contain or provide digital supplements, and to determine supplement anchors associated with such digital supplements.

For example, a network accessible resource may include metadata identifying a supplemental anchor (eg, text, code, entity, or entity type) with which the digital supplement is associated. Metadata can be included in network-accessible resources in response to Hypertext Transfer Protocol (HTTP) requests. Metadata can be provided in a variety of formats, such as Extensible Markup Language (XML), JavaScript Object Notation (JSON), or other formats.

Metadata for a digital supplement may include one or more of the following: type indicator, anchor indicator, name, description, snippet of content (i.e., an excerpt or preview of a piece of content), associated images, digital Links such as URLs to supplements and identifiers of applications associated with digital supplements. Metadata may also include information about the publisher of the digital supplement. For example, the metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some implementations, the metadata includes contextual information related to providing the digital supplement. For example, metadata may also include conditions associated with providing or accessing the digital supplement (eg, geographic conditions, required applications).

The identified digital supplement may be added to an index stored in memory. In at least some implementations, an associated supplement anchor for a digital supplement is used as a key to an index. Digital supplements can also be associated with various scores. For example, the digital supplement may provide information about how many other links were found that referenced the digital supplement or network-accessible resources associated with the digital supplement (eg, while crawling network-accessible resources) and provided those links. may be associated with a reputation score based on the prestige of a network accessible resource. As another example, a digital supplement may be associated with one or more relevance scores corresponding to the relevance of the digital supplement (or associated network accessible resource) to a particular anchor. Relevance scores can be associated with keywords or topics. The relevance score may be determined based on one or more of the content of the digital supplement, the content of the network-accessible resource, and the content (eg, text) of a link to the network-accessible resource.

1 is a block diagram illustrating a system 100 according to an example implementation. System 100 may associate a digital supplement with an entity or entity type, and may retrieve a digital supplement in response to a visual search. Visual search is a search based on visual content. For example, a visual search may be performed based on a visual content query. A visual content query is a query based on images or other visual content. For example, a visual content query may include images. In some implementations, the visual content query may include text or data based images. For example, text or data may be created by recognizing one or more entities in an image. Some visual content queries do not include images (e.g., visual content queries only include text or data generated from images). In some implementations, system 100 includes client computing device 102 , search server 152 and digital supplement server 172 . Also shown is a network 190 in which the client computing device 102 , search server 152 and digital supplement server 172 may communicate.

The client computing device 102 may include a processor assembly 104 , a communication module 106 , a sensor system 110 and a memory 120 . The sensor system 110 may include various sensors such as a camera assembly 112 , an inertial motion unit (IMU) 114 , and a Global Positioning System (GPS) receiver 116 . Implementations of sensor system 110 may include, for example, a light sensor, an audio sensor, an image sensor, a distance and/or proximity sensor, a contact sensor such as a capacitive sensor, a timer and/or other sensors and/or sensors. may include different combinations of In some implementations, the client computing device 102 is a mobile device (eg, a smartphone).

Camera assembly 112 captures images or video of the physical space around client computing device 102 . Camera assembly 112 may include one or more cameras. Camera assembly 112 may also include an infrared camera. Images captured with the camera assembly 112 may be used to identify supplemental anchors and form visual content queries.

In some implementations, images captured with camera assembly 112 are also sent to client computing device 102 within a physical space, such as an interior space, based on a representation of that physical space received from memory 120 or an external computing device. It can be used to determine the position and orientation of In some implementations, the representation of the physical space can include visual features of the physical space (eg, features extracted from images of the physical space). The representation may also include location determination data associated with the corresponding features that may be used by a visual positioning system to determine a position and/or position within a physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structures in physical space. In some implementations, the representation does not include a three-dimensional model of physical space.

IMU 114 may detect motion, movement, and/or acceleration of the client computing device. IMU 114 may include a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the client computing device 102 may be detected and tracked based on data provided by the IMU 114 or GPS receiver 116 .

The GPS receiver 116 may receive signals emitted from GPS satellites. The signal includes the time and position of the satellite. Based on receiving signals from several satellites (eg, at least four), the GPS receiver 116 can determine the global position of the client computing device 102 .

Memory 120 may include applications 122 , other applications 140 , and device positioning system 142 . Other applications 140 include any other applications installed or otherwise available for execution on client computing device 102 . In some implementations, application 122 can cause one of the other applications 140 to be launched to provide the digital supplement. In some implementations, some digital supplements may be available only when other applications 140 are associated with the digital supplement or include specific applications needed to provide the digital supplement.

Device positioning system 142 determines the position of client computing device 102 . Device positioning system 142 can use sensor system 110 to determine the position and orientation of client computing device 102 globally or within physical space. In some implementations, device positioning system 142 determines the location of client computing device 102 based, for example, on cellular triangulation.

In some implementations, the client computing device 102 uses an image captured by the camera assembly 112 to determine a 6 DOF pose (eg, position and orientation) of the client computing device 102 within physical space ( or features extracted from such images) to known configurations of features in a representation of physical space.

The application 122 may include a supplement anchor identification engine 124 , a digital supplement search engine 126 , a digital supplement presentation engine 128 and a user interface engine 130 . Some implementations of application 122 may include fewer, additional or different components.

Supplemental anchor identification engine 124 identifies supplemental anchors based, for example, on images captured with camera assembly 112 . In some implementations, supplement anchor identification engine 124 analyzes the image to identify text. The text can then be used to identify anchors. For example, text can be mapped to nodes in a knowledge graph. For example, text may be recognized as the name of an entity such as a person, place, product, building, artwork, movie, or other tangible entity. In some implementations, text can be recognized as a phrase generally associated with a particular entity or a phrase describing a particular entity. For example, text may be recognized as an anchor associated with a particular entity.

In some implementations, supplement anchor identification engine 124 identifies one or more codes within the image, such as barcodes, QR codes, or other types of codes. The code can then be mapped to the replenishment anchors.

Supplemental anchor identification engine 124 may include a machine learning module capable of recognizing at least some types of entities within an image. For example, a machine learning module may include a neural network system. A neural network is a computational model used in machine learning and consists of nodes organized into layers with weighted connections. Neural network training uses training examples, each through a series of iterative rounds, to determine weight values for connections between layers that increase the probability of the neural network giving the desired output for a given input input and desired output. to be. During each training round, the weights are adjusted to correct for erroneous output values. Once trained, neural networks can be used to predict outputs based on inputs provided.

In some implementations, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer that computes the values of a layer based on applying a kernel function to a subset of the previous layer's values. Neural network training may include adjusting the weights of kernel functions based on training examples. In general, the same kernel function is used to compute each value of the convolutional layer. Thus, in neural networks, convolutional layers require far fewer weights to learn while training than fully connected layers (i.e., layers where each value in a layer is computed as an independently adjusted weight combination of each value in the previous layer). Because convolutional layers generally have fewer weights, training and using convolutional layers may require less memory, processor cycles, and time than equivalent fully-connected layers.

After the supplement anchor identification engine 124 recognizes an entity or entity type in an image, a textual description of the entity or entity type may be created. Entities or entity types can also be mapped to supplemental anchors. In some implementations, a supplemental anchor is associated with one or more digital supplements.

In some implementations, the supplemental anchor identification engine 124 determines confidence scores for recognized anchors. A higher confidence score may indicate that content from the image (eg, image, extracted text, barcode, QR code) is more likely to be associated with the determined anchor than if a lower confidence score was determined.

1 shows supplement anchor identification engine 124 as a component of application 122 on client computing device 102 , some implementations include supplement anchor identification engine on search server 152 . . For example, client computing device 102 can transmit an image captured by camera assembly 112 to search server 152, which can then identify supplemental anchors within the image. .

In some implementations, supplemental anchor identification engine 124 identifies potential supplemental anchors. For example, supplement anchor identification engine 124 may identify (recognize) various entities within an image. Identifiers of recognized entities may be sent to search server 152, which may determine which of the entities are associated with any supplement anchor. In some implementations, search server 152 may use the identified entity as contextual information even if the identified entity is not a supplement anchor.

The digital supplement search engine 126 searches for digital supplements. For example, digital supplement search engine 126 may search for digital supplements associated with supplement anchors identified by supplement anchor identification engine 124 . In some implementations, digital supplement search engine 126 retrieves digital supplements from search server 152 or digital supplement server 172 .

For example, after a supplement anchor is identified, digital supplement search engine 126 may retrieve one or more digital supplements associated with the identified supplement anchor. Digital supplemental search engine 126 may generate a visual content query that includes the image (or identifier of a supplemental anchor or entity within the image) and transmit the visual content query to search server 152 . The visual content query may also include contextual information, such as the location of the client computing device 102 . In some implementations, data associated with the digital supplement, such as a name, image, or description, is retrieved and presented to the user (eg, by user interface engine 130). If multiple digital supplements are presented, the user may select one of the digital supplements through the user interface generated by user interface engine 130 .

The digital supplement presentation engine 128 presents or causes the digital supplement to be presented by the client computing device 102 . In some implementations, the digital supplement presentation engine 128 causes the client computing device to launch one of the other applications 140 . In some implementations, digital supplement presentation engine 128 causes information or content to be displayed. For example, digital supplement presentation engine 128 can cause user interface engine 130 to generate a user interface that includes information or content from the digital supplement to be displayed by client computing device 102 . In some implementations, digital supplement presentation engine 128 is triggered by digital supplement search engine 126 to search for digital supplements. The digital supplement presentation engine 128 can trigger the display device 108 to display content associated with the digital supplement. In some implementations, digital supplement presentation engine 128 causes digital supplements to be displayed at a different time than when digital supplement search engine 126 searches for digital supplements. For example, a digital supplement can be retrieved in response to a visual content query a first time and the digital supplement can be presented a second time. For example, to retrieve a digital supplement in response to a visual content query based on home furnishings or images of furniture in a catalog or store at a first time (eg, while the user is viewing a catalog or visiting a store). can A digital supplement comprising the home furniture or AR content of the furniture may be presented at a second time (eg, while the user is in the home furniture or a room in which the furniture may be placed).

User interface engine 130 creates a user interface. User interface engine 130 can also cause client computing device 102 to display the generated user interface. The generated user interface may display, for example, information or content of a digital supplement. In some implementations, user interface engine 130 generates a user interface that includes a number of user-operable controls each associated with a digital supplement. For example, a user may actuate one of the user-operable controls (e.g., by touching a control on a touch screen, clicking on a control using a mouse or other input device, or otherwise activating a control). ).

Search server 152 is a computing device. Search server 152 may respond to search requests, such as visual content queries. The response may include one or more digital supplements potentially related to the visual content query. In some implementations, search server 152 includes memory 160 , processor assembly 154 and communication module 156 . Memory 160 may include content crawler 162 , digital supplement search engine 164 and digital supplement data store 166 .

Content crawler 162 may crawl network accessible resources to identify digital supplements. For example, content crawler 162 may access webpages accessible over the Internet, such as webpages served by digital supplement server 172 . Crawling network accessible resources may include requesting the resource from a web server and parsing at least a portion of the resource. A digital supplement may be identified based on metadata provided by a network accessible resource, such as XML or JSON data providing information about the digital supplement. In some implementations, the crawler identifies network accessible resources based on extracting links from previously crawled network accessible resources. Content crawler 162 may also identify network accessible resources to crawl based on receiving input submitted by a user. For example, a user may submit a URL (or other information) to a network accessible resource containing a digital supplement via a web form or application programming interface (API). In some implementations, content crawler 162 creates an index of identified digital supplements. Content crawler 162 may also generate a score associated with the digital supplement, such as a relevance score or a popularity (reputation) score.

The digital supplement search engine 164 receives the search query and generates a response that may include one or more potentially related digital supplements. For example, digital supplement search engine 164 may receive a visual content query from client computing device 102 . Visual content queries can include images. The digital supplement search engine 164 may identify supplement anchors in the image and, based on the identified supplement anchors, may identify related or potentially related digital supplements. The digital supplement search engine 164 can send a response to the client computing device 102 that includes the digital supplement or information that can be used to access the digital supplement. In some implementations, digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a digital supplement list may be included in the response to a query. The list may be ordered based on relevance to the replenishment anchor, popularity, or other attribute of the digital supplement.

The visual content query may include, for example, an image captured by camera assembly 112 or text or other data associated with an image captured by camera assembly 112 . The visual content query may also include other information, such as the location of the client computing device 102 or the identifier of the user of the client computing device 102 . In some implementations, search server 152 can determine the approximate location of client computing device 102 from a user identifier (e.g., a client computing device to which a user associates information about the user's location with an account of the user). (102) if you have enabled location services).

The digital supplement data store 166 stores information about digital supplements. In some implementations, digital supplement data store 166 includes an index of digital supplements. For example, the index may be created by the content crawler 162. Digital supplement search engine 164 may use the index to respond to search queries.

Digital supplement server 172 is a computing device. Digital supplement server 172 provides digital supplements. In some implementations, digital supplement server 172 includes memory 180 , processor assembly 174 and communication module 176 . Memory 180 may include digital supplement 182 and metadata 184 . In some implementations, memory 180 may include other network accessible resources such as webpages that are not necessarily digital supplements. For example, memory 180 may store webpages that include metadata to provide details about one or more digital supplements and how to access those digital supplements. Additionally, memory 180 may include a resource serving engine, such as a web server, that responds to requests, such as HTTP requests, to network accessible resources such as web pages and digital supplements.

Digital supplement 182 is any type of content that can be provided as a supplement to something in the physical environment around the user. Digital supplement 182 may also include any type of content that may supplement stored images (eg, the previous physical environment around the user). For example, a digital supplement may be associated with a supplement anchor such as an image, object, or product identified in an image or location. Digital supplement 182 may include one or more images, audio content, text data, videos, games, data files, applications, or structured text documents. Examples of structured text documents include Hypertext Markup Language (HTML) documents, XML documents, and other types of structured text documents.

Digital supplement 182 may cause an application to be launched and may define parameters for the application. Digital supplement 182 may also cause a request to be sent to a server (eg, an HTTP request) and may define parameters for that request. In some implementations, digital supplement 182 is initiated as a workflow for completing an activity, such as a workflow for completing a purchase. For example, digital supplement 182 may send an HTTP request to a server to add a particular product to the user's shopping cart, add a coupon code, and retrieve a purchase confirmation page.

Metadata 184 is data describing the digital supplement. Metadata 184 may describe one or more digital supplements provided by digital supplement server 172 or provided elsewhere. Metadata 184 for a digital supplement may include one or more of the following: type indicator, anchor indicator, name, description, preview snippet or excerpt, associated image, URL to the digital supplement, and the like. The identifier of the application associated with the link and digital supplement. Metadata may also include information about the publisher of the digital supplement, such as a publisher name, a publisher description, and an image or icon associated with the publisher. In some implementations, the metadata also includes context information about the digital supplement or context information that must be satisfied to provide the digital supplement. For example, metadata may include conditions that must be met to access the digital supplement (eg, geographic conditions, client computing device requirements, required applications). Exemplary contextual information includes a location, an entity identified within an image, or multiple entities identified within an image (eg, some digital supplements may require a combination of entities to be recognized within an image). A recognized entity may be a supplemental anchor. In some implementations, a recognized entity provides context information rather than a supplemental anchor. Metadata 184 may also include a supplemental anchor (eg, text, code, entity, or entity type) associated with the digital supplement.

Metadata 184 may be stored in a variety of formats. In some implementations, metadata 184 is stored in a database. Metadata 184 may also be stored as XML files, JSON files, or other format files. In some implementations, digital supplement server 172 retrieves metadata 184 from a database, and formats metadata 184 in XML, JSON, or otherwise in a request from a client or search server 152. provides a response to For example, search server 152 may access metadata 184 to generate data stored in digital supplement data store 166 and used to respond to search requests from client computing device 102 . there is.

Communications module 106 includes one or more devices for communicating with other computing devices, such as search server 152 or digital supplement server 172 . Communications module 106 can communicate over a wireless or wired network, such as network 190 . The communication module 156 of the search server 152 and the communication module 176 of the digital supplement server 172 may be similar to the communication module 106 .

The display device 108 may include, for example, a liquid crystal display (LCD) screen, a light emitting diode (LED) screen, an organic light emitting diode (OLED) screen, a touch screen, or other screen or display for displaying images or information to a user. can In some implementations, display device 108 includes a light projector arranged to project light onto a portion of the user's eye.

Memory 120 may include one or more non-transitory computer readable storage media. Memory 120 may generate visual content queries based on captured images, send visual content queries, receive responses to visual content, and present identified digital supplements in response to visual content queries. It may store instructions and data usable by the client computing device 102 to implement the techniques described herein. The memory 160 of search server 152 and the memory 180 of digital supplement server 172 may be similar to memory 120, the technology of search server 152 and digital supplement server 172, respectively. can store data instructions that can be used to implement

Processor assembly 104 includes one or more devices capable of executing instructions, such as instructions stored by memory 120, to perform various tasks associated with digital supplement association and retrieval for visual retrieval. For example, the processor assembly 104 may include a central processing unit (CPU) and/or a graphics processing unit (GPU). For example, if there is a GPU, some image/video rendering tasks such as user interface creation and display or some display of digital supplements may be offloaded from the CPU to the GPU. In some implementations, some image recognition tasks can be offloaded from the CPU to the GPU.

Although FIG. 1 does not show it, some implementations include a head-mounted display (HMD) device. The HMD may be a separate device from the client computing device 102 or the client computing device 102 may include the HMD. In some implementations, computing device 102 communicates with HMD 104 via a cable. For example, the client computing device 102 can send video signals and/or audio signals to the HMD for display for a user, and the HMD can send motion, position and/or orientation information to the client computing device 102. can

The client computing device 102 may also include various user input components (not shown) such as a controller that communicates with the client computing device 102 using a wireless communication protocol. In some implementations, the client computing device 102 is connected via a wired connection (eg, Universal Serial Bus (USB) cable) or via a wireless communication protocol (eg, any WiFi protocol, any BlueTooth protocol, ZigBee, etc.) ) to communicate with the HMD (not shown). In some implementations, the client computing device 102 is a component of an HMD and can be contained within a housing of the HMD.

Network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or other networks. Client computing device 102 may receive audio/video signals over a network, which may be provided as part of a digital supplement in an example implementation.

2 is a third-person view of an exemplary physical space 200 from which an embodiment of a client computing device 102 accesses a digital supplement. In this example, physical space 200 includes object 222 . Here, the object 222 is a work hung on the wall of the physical space 200 . Object 222 is included within field of view 204 of camera assembly 112 of client computing device 102 .

An example user interface screen 206 is also shown. User interface screen 206 may be generated, for example, by user interface engine 130 of client computing device 102 . The user interface screen 206 includes an image display panel 208 and a digital supplement selection panel 210 . The image display panel 208 shows images. For example, the image display panel 208 can show an image corresponding to a real-time feed from the camera assembly 112 of the client computing device 102 . In some implementations, image display panel 208 shows previously captured images or images retrieved from memory 120 of client computing device 102 .

In some implementations, user interface screen 206 is displayed to a user on a display device of client computing device 102 . In some implementations, the user interface screen 206 can be overlaid on an image of the physical space (or a video feed captured by a camera of the computing device). In addition, the user interface screen 206 may be displayed as AR content in the user's field of view using the HMD worn by the user.

The image display panel 208 may also include annotations or user interface elements that may be associated with images. For example, image display panel 208 may include an indicator indicating that an object in the image (eg, object 222 ) has been recognized as a supplemental anchor. The indicator may include user-operable controls for accessing or viewing information regarding the digital supplement associated with the identified supplement anchor. In some circumstances, an image displayed on image display panel 208 may include multiple objects that are recognized as supplementary anchors, and image display panel 208 may overlay multiple images to identify such supplemental anchors. may contain comments.

A supplement anchor may be recognized by a supplement anchor identification engine of the client computing device 102 . In some implementations, supplemental anchors are identified by sending images to search server 152 . Search server 152 may then analyze the image and identify supplemental anchors in the image. In some implementations, search server 152 may send client computing device 102 one or more of the location (eg, image coordinates) or dimensions of any identified object associated with the supplement anchor. The client computing device 102 can then update the user interface screen to show annotations identifying supplemental anchors (or associated objects) in the image. In some implementations, client computing device 102 may send camera assembly 112 (eg, supplement anchor identification engine 124 may track supplement anchors identified by search server 152 ). track the position of a supplemental anchor (or associated object) in a video stream captured by (e.g., images captured sequentially in sequence).

The digital supplement selection panel 210 allows the user to select a digital supplement to present. For example, the digital supplement selection panel 210 may include a menu containing user-operable controls each associated with a digital supplement. In this example, digital supplement selection panel 210 includes user-operable controls 212 and user-operable controls 214 that each contain information about an associated digital supplement. For example, the user-operable control may display one or more of a name (or title), brief description, and image associated with the digital supplement that may be received from search server 152 . Upon actuation of user-operable control 212 or user-operable control 214, the content of the associated digital supplement may be presented to the user. Presenting the digital supplement to the user may include causing the client computing device 102 to display a user interface screen that includes images, video, text, other content, or combinations thereof from the digital supplement. In some implementations, the digital supplement content is displayed as an overlay on the image display panel 208 over the image or camera feed. The digital supplement content may be three-dimensional augmented reality content.

In some implementations, presenting the digital supplement includes operating an application (eg, one of the other applications 140 ) installed on the client computing device 102 . Presenting the digital supplement may also include sending a request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as the identifier of the product or object identified within the image. In some implementations, the image (or other content) of the visual content query is passed as a parameter along with the request. Images may be provided through APIs associated with digital supplement server 172 . In some implementations, client computing device 102 transmits the image to digital supplement server 172 . In some implementations, search server 152 may transmit images to digital supplement server 172 . For example, in response to a user selecting a digital supplement, client computing device 102 can send an indicator of the selection to search server 152, which retrieves an image of the corresponding digital supplement. can be sent to the server. Client computing device 102 can also send a URL to a location on search server 152 that digital supplement server 172 can use to access the image. Advantageously, such an implementation can reduce the amount of data that a client computing device must transmit.

A digital supplement associated with user-operable control 212 may cause information about object 222 to be displayed, such as information from a museum. A digital supplement associated with user-operable control 214 may cause information related to the museum tour to be displayed. For example, presenting a digital supplement may mark a stop on a museum tour as complete and cause information about the next stop to be displayed.

3 is a diagram of an example method 300 that enables triggering of digital supplements in accordance with implementations described herein. Method 300 may be performed, for example, by content crawler 162 of search server 152 to allow users to access digital supplements based on visual content queries.

At operation 302, data specifying a digital supplement is received. The data may identify the digital supplement and the circumstances in which the digital supplement should be provided. Data specifying the digital supplement may be received in a variety of ways. For example, data specifying a digital supplement may be received from a network accessible resource such as a webpage containing metadata about the digital supplement. Data specifying the digital supplement may also be received via, for example, an API or form provided by search server 152 . Data specifying the digital supplement may be received from a memory location or data store.

Data for the digital supplement may include access data usable by the client computing device to access the digital supplement. For example, the access data may include the URL of the digital supplement and parameters to be passed to that URL. Access data may include an application identifier and parameters for the application. The data for the digital supplement may include descriptive data for the digital supplement. The descriptive data can be used by the client computing device to present information about the digital supplement to the user (eg, in a menu from which the user can select the digital supplement). Description data may include, for example, a name (or title, description, publisher name, and image). Data relating to the digital supplement may also include an identifier of the supplement anchor.

At operation 304, a data structure instance based on the received data is created. For example, a data structure can be a record in a database. The database may be a relational database, and a data structure instance may be linked (eg, via a foreign key) with one or more records associated with a supplemental anchor.

At operation 306, after creating the data structure instance, retrieval of the digital supplement by visual content query is enabled. For example, a database field associated with a data structure instance may be set active so that digital supplement search engine 164 can access and return the associated digital supplement. In some implementations, triggering a digital supplement can include saving or committing a database record. In some implementations, enabling retrieval of the digital supplement includes enabling triggering of the digital supplement by the client computing device. For example, after being instantiated, the digital supplement may be returned to the client computing device in response to a search and activated or presented by the client computing device.

4 is a diagram of an exemplary method 400 that enables triggering of digital supplements in accordance with implementations described herein. Method 400 may be performed, for example, by content crawler 162 of search server 152 to allow users to access digital supplements based on visual content queries.

At operation 402, network accessible resources are analyzed. In some implementations, the network accessible resource is a webpage served by digital supplement server 172, for example. In some implementations, a set of network accessible resources are analyzed. A set of network accessible resources can be created based on submission via a form or API. In some implementations, the set of network accessible resources can be created by crawling other network accessible resources to identify URLs. This crawling process can be performed recursively.

At operation 404, metadata associated with the digital supplement in the network accessible resource is identified. In some implementations, the network accessible resource may include an indicator of metadata associated with the digital supplement. For example, a network accessible resource may include a tag identifying a portion of the network accessible resource that includes metadata. Tags can be XML tags with specific types or attributes. Tags can be HTML tags, such as script tags that contain a JSON data structure containing metadata.

At operation 406, a digital supplement data structure instance based on the metadata is created. Operation 406 may be similar to operation 304.

At operation 408, a visual content query is received. The visual content query may be sent by a client computing device, such as client computing device 102 , for example. In some implementations, the visual content query includes images. A visual content query may include textual data describing an image. For example, the text data may include an identifier of a supplemental anchor in an image captured by a camera assembly of the client computing device. In some implementations, the visual content query also includes other information, such as the location of the client computing device or an identifier of a user account associated with the client computing device.

At operation 410, multiple digital supplement data structure instances are identified based on the visual content query. In some implementations, supplemental anchors are identified within images provided in the visual content query. Supplement anchors can be used to query an index or database for related digital supplements. In some implementations, other data provided with the query may also be used to identify the digital supplement, such as the location of the client computing device or information associated with the user account. In some embodiments, multiple supplement anchors are used to identify related supplement anchors.

At operation 412, an alignment of a number of digital supplement data structure instances is determined. The sorting may be based on the relevance of the digital supplement to various scores or visual content queries associated with the digital supplement. In some implementations, a relevance score corresponding to the relevance of the digital supplement to the visual content query is used to order multiple digital supplement data structure instances.

A relevance score may include one or more of the content of a digital supplement, content of a network-accessible resource that links to a digital supplement (or a network-accessible resource associated with a digital supplement), or proximity to a link to a digital supplement of another network-accessible resource. It can be determined from a number of factors, such as link text or content.

The score may be based on a popularity metric. A reputation metric is an example of a popularity metric. The reputation metric may be based on a combination of how many other network resources are linked to the digital supplement and the reputation scores of other network accessible resources. In some implementations, the popularity score can be based on how often the digital resource is or has been selected. In some implementations, the popularity score may correspond to how often a digital resource is selected for visual content query.

The score can be determined or retrieved from a data store or API. In some implementations, an API is accessed to retrieve scores for digital supplements. For example, the score may be retrieved from a search engine that has determined relevance and/or popularity to a digital resource in relation to a search term based on a supplemental anchor.

The number of digital supplement data structures may also be ordered based on frequency of use by a particular user (eg, user of the client computing device) or recent use by a particular user. In some implementations, multiple digital supplemental data structures are randomly aligned.

At operation 414, the visual content query is answered based on the number of digital supplement data structure instances. For example, information associated with multiple digital supplement data structure instances may be transmitted to the client computing device in the order determined in operation 412 . In some implementations, the information includes descriptive data that can be displayed in a menu or other type of user interface configured to receive user selection of the digital supplement. Information may also include access data that may be used by a client computing device to access or present a digital supplement.

5 is a diagram of an exemplary method 500 for retrieving and presenting digital supplements in accordance with implementations described herein. Method 500 may be performed, for example, by application 122 of client computing device 102 to identify and access a digital supplement based on a visual content query.

At operation 502, a visual content query based on the image is sent to a server computing device (eg, search server 152). For example, an image may be captured with the camera assembly 112 of the client computing device 102 . The image may also be a stored image, such as an image previously captured by the camera assembly 112 . In some implementations, the visual content query includes only images. In some implementations, the visual content query includes additional information. For example, the visual content query may include information such as the location of the client computing device 102 or the identifier of an account associated with the user of the client computing device 102 . Application 122 may also identify anchors in the image (eg, using supplemental anchor identification engine 124 ). The visual content query may include the identified anchor's identifier (eg, text, number, or other type of identifier). In at least some implementations, the visual content query does not include images.

In some implementations, sending the visual content query to the server includes an API call. In some implementations, sending the visual content query to the server includes calling an API provided by the server. In some implementations, sending the visual content query to the server includes submitting a form (eg, submitting a GET or POST request) using the HTTP protocol.

At operation 504, a response to the visual content query identifying the digital supplement is received. A response may be received from search server 152 over network 190 . The response may include one or more digital supplements that were identified by search server 152 based on the visual content query. For example, the response may include an array of data associated with the digital supplement. In some implementations, the data associated with the digital supplement can include descriptive data that can be used to present digital supplement options from which a user can select. For example, descriptive data may include name, short description, publisher name, and image. The data may include access data such as a URL and parameters to include with the request via the URL or application name and associated parameters. The data may also include the location, coordinates, or dimensions of the supplement anchor in the image sent along with the visual content query (eg, if the supplement anchor is identified by search server 152).

At operation 506, a user interface screen including information associated with the digital supplement is displayed. In some implementations, the user interface screen includes annotations overlaying the identified supplemental anchors (eg, based on provided coordinates). The annotation may provide information about the object of the image associated with the identified supplemental anchor. The annotation may include a user-operable control that can be actuated to present or activate the digital supplement. The user interface screen may also include a digital supplement selection panel that may be used to select from a number of digital supplements identified in the response received at operation 504 . In some implementations, the user interface screen may be generated by a web browser opening a URL specified by the digital supplement. User interface screens may be created by other applications that are launched to provide digital supplements.

6 is a diagram of an exemplary method 600 for identifying and presenting digital supplements based on images, in accordance with implementations described herein. Method 600 may be performed, for example, by application 122 on client computing device 102 to identify and access a digital supplement based on a visual content query.

At operation 602, an image is captured. For example, an image may be captured by the camera assembly 112 of the client computing device 102 . In some implementations, a sequence of images (ie, video) may be captured by camera assembly 112 .

At operation 604 , a visual content query based on the image is sent to a server computing device, such as search server 152 . Operation 604 may be similar to operation 502 . In implementations where image sequences are captured, the visual content query may include multiple images or image sequences. In some implementations, the image sequence can be streamed to a server computing device.

At operation 606, a response to the visual content query identifying a number of digital supplements is received. Operation 606 may be similar to operation 504 previously described.

At operation 608, a user interface screen is displayed that includes user-operable controls for selecting a digital supplement from a plurality of digital supplements. For example, a digital supplement selection panel may be displayed. The digital supplement selection panel may include multiple user-operable controls each associated with one of the multiple digital supplements identified in the response. The digital supplement selection may arrange a user-operable control based on an order or ranking of digital supplements provided by the server computing device. The digital supplemental selection panel may arrange the user-operable controls vertically, horizontally or otherwise. The user-operable control may be associated with or include information about an associated digital supplement that the user may consider when deciding whether to select a digital supplement. For example, the displayed information may include one or more of a name, description, image, and publisher name for the digital supplement.

At step 610, user input for selecting a digital supplement is received. The user input may be a click using a mouse or other device. The user input may be a stylus or finger touch input. Another example of user input is near touch input (eg, holding a finger or pointing device close to the screen). In some implementations, user input can also include hand gestures, head motions, eye actions, or voice input.

At operation 612, information is provided to a resource associated with the selected digital supplement. For example, information about the user of the client computing device may be transmitted to a server that provides the digital supplement (provided that informational rights are provided). Information may be provided to applications that provide digital supplements. Various types of information may be provided. For example, the information may include user information such as user name, user preferences or location.

Information may also include information related to visual content queries, such as images or image sequences. The information may also include an identifier and/or position of one or more supplemental anchors in the image. This information can be used to provide digital supplements to users. For example, AR content in a digital supplement may be sized and positioned based on images.

Information may be transmitted directly by the client computing device 102 to a resource associated with the digital supplement (eg, digital supplement server 172 ). In some implementations, information is provided by search server 152 to resources associated with digital supplements (eg, client computing devices do not need to transmit much data). In at least some of these implementations, the client computing device 102 can send selection information identifying the selected digital supplement to the search server 152 . Upon receiving the selection and confirming that the user has authorized sharing of the information, search server 152 may transmit the information to the resource providing the digital supplement. The client computing device 102 can also prompt the user to allow information sharing. In some implementations, search server 152 may determine information to send to a resource based on the digital supplement data structure instance (which may be based on metadata associated with the digital supplement).

At operation 614, the user interface is updated based on the selected digital supplement. Operation 614 may be similar to operation 506 .

7A-7C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. 7A shows a user interface screen 700a. The user interface screen 700a includes an image display panel 708 and an information panel 730 . In this example, image display panel 708 is displaying an image of a shelf full of wine bottles (eg, as you might find in a store). Image display panel 708 also includes indicator 740 and indicator 742 . Each of these indicators indicates that the wine bottle shown in the image below the indicator has been recognized as a supplement anchor (ie, in this case a recognized product). Indicator 740 and indicator 742 are examples of user-operable controls. Information panel 730 is provided with the instruction "Tap the item of interest."

In FIG. 7B , user interface screen 700b is shown after the user activates indicator 740 . After operation, annotations 744 from the digital supplement are displayed. Annotation 744 includes information regarding the grade of the wine, which may assist the user in selecting a wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after the user activates the indicator 740. User interface screen 700c may be displayed in place of or in addition to user interface screen 700b (e.g., after activating annotation 744 or when the user swipes up from information panel 730 in FIG. 7B). if you do). In FIG. 7C, an expanded information panel 732 is shown. The expanded information panel 732 occupies more of the user interface screen 700c than does the information panel 730 in FIGS. 7A and 7B.

The extended information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734 . The digital supplement selection panel 710 includes a user-operable control 712, a user-operable control 714, and a user-operable control 716 (only partially visible). In some implementations, additional user-operable controls may be displayed when the user swipes the digital supplement selection panel 710 . The user-operable controls of the digital supplement selection panel 710 may be arranged in a ranked order. A user-operable control 712 is associated with a digital supplement for meal pairing. Upon actuation of the user-operable control 712, a digital supplement displaying food and meal pairing information for the selected wine may be displayed. A user-operable control 714 is associated with a digital supplement that stores photos. Upon operation, the application that stores the picture is activated so that the image can be presented. Additional information may be stored along with the photo, such as identified replenishment anchors.

The digital supplement content display panel 734 can display content from the digital supplement. The digital supplement content display panel 734 may display the default digital supplement or the highest ranked digital supplement associated with the identified supplement anchor. In this example, digital supplement content display panel 734 includes product information pertaining to a product associated with the selected supplement anchor. In this case, the wine name, grade, country of origin, image and comments are provided.

8A-8C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. In this example, the visual content search is based on the receipt image.

A user interface screen 800a is shown in FIG. 8A. The user interface screen 800a includes an image display panel 808 and an information panel 830 . In this example, the image display panel 808 is displaying an image of the restaurant's receipt. Image display panel 808 also includes indicator 840 , indicator 842 , annotation 844 and highlight overlay 846 . In this case, indicator 840 is associated with the receipt as a document, and indicator 842 is associated with the particular restaurant named on the receipt. The identified receipt document and the identified restaurant name are both examples of supplemental anchors.

Annotation 844 is associated with a digital supplement that provides a tip calculator. In this example, an example tip calculation is included in annotation 844 and overlaid at the appropriate position on image display panel 808 . In some implementations, a digital supplement may be selected by default and displayed upon identifying an appropriate supplement anchor. A highlight overlay 846 is overlaid over a portion of the receipt document that contains information used by the tip calculator digital supplement.

In this example, the item displayed in information panel 830 is a document associated with a receipt as if indicator 840 had been activated. In some implementations, an identified supplemental anchor is, for example, a user's past action, another user's action on a similar image, a confidence score for a supplemental anchor, or a position or size of a portion of an image associated with a supplemental anchor. is ranked based on the user's relevance or interest. Information panel 830 may then display, in at least some implementations, an item associated with the highest ranking supplement anchor. Alternatively, if indicator 842 has been activated, information panel 830 may include an item for a particular restaurant.

Here, the information panel 830 includes a digital supplement selection panel 810 . The digital supplement panel includes a user-operable control 812 , a user-operable control 814 and a user-operable control 816 . In this example, user-operable control 812 is associated with tip calculator digital supplement, user-operable control 814 is associated with bill split digital supplement, and user-operable control 816 is associated with cost. Associated with the report digital supplement. For example, upon actuation of the user-operable control 812, a user interface control for adjusting a parameter of the tip calculator may be displayed (eg, to adjust the percentage).

In FIG. 8B , the user interface screen 800b is shown after the user has actuated the user-activatable control 814 . After activation, an expanded information panel 832 is shown that includes items to assist the user in calculating bill splits. For example, you can enter the number of people splitting a bill to determine how much each person has to pay.

In FIG. 8C , user interface screen 800c is shown after the user actuates user-activatable control 816 . After activation, an expanded information panel 834 is displayed that includes items to help the user save the receipt to the expense report. For example, a user can select an expense report that should be associated with a receipt (e.g. "Sydney trip 2018"). When an expense report is selected, an image of the receipt can be uploaded to the expense report submission or management system. In some implementations, the entire image displayed on image display panel 808 is uploaded. In some implementations, a portion of the image is uploaded (eg, the image is cropped to contain only the receipt).

9A-9C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. Visual content search in this example is based on face images.

A user interface screen 900a is shown in FIG. 9A. The user interface screen 900a includes an image display panel 908 and an information panel 930 . In this example, the image display panel 908 is displaying an image of a face. Here, the face is an example of a supplemental anchor. Information panel 930 includes user-operable controls 912 for digital supplements that have been identified for supplemental anchors in an image (ie, face). A user-operable control 912 is associated with a digital supplement for strapping eyeglasses.

In FIG. 9B , user interface screen 900b is shown after the user actuates user activatable control 912 . After actuation, an expanded information panel 932 is displayed containing items that assist the user in visually putting the glasses on the face in the image. Here, several eyeglass styles are displayed, and the user can select a pair to try on. When a pair of glasses is selected, AR content 960 is overlaid on the image display panel 908 . Here, the AR content 960 corresponds to the selected glasses and is scaled to match the face in the image. In some implementations, when a digital supplement for wearing glasses is selected, the image displayed on the image display panel 908 is transmitted to a server providing the digital supplement, which analyzes the image to determine the position and size of the AR content 960. They can decide what to do or recommend glasses to try on.

10A-10C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. In this example, visual content retrieval is based on furniture images in a catalog.

In FIG. 10A, a user interface screen 1000a is shown. The user interface screen 1000a includes an image display panel 1008 . In this example, the image display panel 1008 is displaying an image of a portion of a page of a furniture catalog. The image display panel also includes indicator 1040 , indicator 1042 and indicator 1044 . In this example, indicator 1040 is associated with a bed, indicator 1042 is associated with a decorative item, and indicator 1044 is associated with a rug. Beds, decorative items, and rugs in catalogs are examples of supplementary anchors.

In FIG. 10B , user interface screen 1000b is shown after the user selects indicator 1040 (eg, by touching the screen at or near where indicator 1040 is displayed). The user interface screen 1000b includes a digital supplement selection panel 1010 and an information panel 1030 . Information panel 1030 includes information about the supplement anchor associated with the selected indicator (eg, product name, description, image).

The digital supplement selection panel 1010 includes a user-operable control 1012 and a user-operable control 1014 . A user-operable control 1012 is associated with a digital supplement that provides an in-home view. User-operable controls 1014 are associated with other digital supplements (eg, digital supplements for posting on social media sites).

In FIG. 10C , user interface screen 1000c is shown after actuation of user-operable control 1012 . The user interface screen 1000c includes an image display panel 1008 , a digital supplement selection panel 1010 and a reduced information panel 1032 . The reduced information panel 1032 may include user-operable controls that, when actuated, may cause the information panel to pop up and be displayed.

Here, the image display panel 1008 now displays an image of the room and includes AR content 1060 . AR content 1060 includes a 3D model of the bed associated with indicator 1040 overlaid on an image panel. The user may adjust the position of the AR content 1060 within the room to see how the bed fits into the room. In some implementations, when a digital supplement for an in-home view is selected, the image displayed on the image display panel 1008 is sent to a server providing the digital supplement to analyze the image to determine the position and position of the AR content 1060. Decide where and how to size. In some implementations, AR content 1060 may be provided after a visual content query.

11A-11C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing a visual content search within a repository. In this example, visual content retrieval is based on product images captured within the store.

In FIG. 11A , a user interface screen 1100a is shown. The user interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 is displaying images captured within the store. The image display panel 1108 also includes an indicator 1140 associated with the vase. A vase displayed on the image display panel 1108 is an example of a replenishment anchor. The information panel 1130 displays product information about the vase and digital supplements including a vase purchasing function. For example, a digital supplement may include a workflow that initiates the purchase of a vase. The digital supplement in this example is such that if the client computing device is within the store, the digital supplement published by (or associated with) the store from which the image was captured is identified and presented as a high ranking result to a visual content query and client Identified based on the location of the computing device. In some implementations, another digital supplement is provided for the same image if the location of the client computing device has changed.

In FIG. 11B, a user interface screen 1100b is shown. The user interface screen 1100b includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 is displaying other images captured within the store. The image display panel 1108 also includes an indicator 1142 associated with the lug. A lug displayed on the image display panel 1108 is an example of a supplemental anchor. The information panel 1130 displays product information about the rug and a digital supplement including the ability to select a size and purchase the rug. As in FIG. 11A , the digital supplement is identified based on the image content and the location of the client computing device.

In FIG. 11C , a user interface screen 1100c is shown. The user interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 is displaying other images captured within the store. The image display panel 1108 also includes an indicator 1144 associated with the vase. A vase displayed on the image display panel 1108 is an example of a replenishment anchor. Information panel 1130 displays a digital supplement containing product information about the vase. The information panel 1130 also includes a coupon indicator 1132 and functionality to use coupons. Redeeming the coupon may include purchasing an item at a discounted price on a website associated with the store. In some implementations, a coupon code is provided that can be used to secure a discount during checkout. As in FIGS. 11A and 11B , the digital supplement is identified based on the image content and the location of the client computing device.

12A-12C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 during a visual content search. In this example, the visual content search is based on an image of a movie poster (eg, may be captured in a movie theater).

* In FIG. 12A, a user interface screen 1200a is shown. The user interface screen 1200a includes an image display panel 1208 . In this example, the image display panel 1208 is displaying an image of a movie poster. The image display panel 1208 also includes an indicator 1240 associated with the movie poster identified in the image. A movie poster is an example of a supplementary anchor. Indicator 1240 may include a user-operable control that, when actuated, will display the digital supplement or a menu for selecting a digital supplement.

In FIG. 12B, a user interface screen 1200b is shown. The image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified in the image. For example, preview digital supplement 1242 can be displayed after actuation of indicator 1240 (FIG. 12A) indicator 1240. The preview digital supplement 1242 can overlay an image or video from a movie associated with the identified movie poster onto the image of the movie poster.

In FIG. 12C, a user interface screen 1200c is shown. Image display panel 1208 also includes rating indicator 1244 and rating indicator 1246 . Rating indicator 1244 and rating indicator 1246 may be generated by one or more digital supplements in response to a visual content query including a movie poster. A digital supplement may, for example, overlay rating information for a movie associated with a movie poster on an image. Rating indicator 1244 and rating indicator 1246 may include user-operable controls that, when activated, cause the rating and additional information about the associated movie to be displayed.

13 illustrates a computer device 1300 that may be used with the techniques described herein (eg, to implement client computing device 102, search server 152, and digital supplement server 172) and An example mobile computer device 1350 is shown. Computing device 1300 includes processor 1302, memory 1304, storage device 1306, high-speed interface 1308 coupled to memory 1304 and high-speed expansion port 1310 and low-speed bus 1314 and storage device and a low-speed interface 1312 coupled to 1306. Each of the components 1302, 1304, 1306, 1308, 1310 and 1312 are interconnected using various buses and may be mounted on a common motherboard or in other ways suitable. Processor 1302 processes instructions for execution within computing device 1300, including instructions stored in memory 1304 or storage device 1306, such as display 1316 coupled to high-speed interface 1308. Graphical information for the GUI can be displayed on an external input/output device. In another implementation, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Additionally, multiple computing devices 1300 may be coupled with each device providing portions of the necessary operation (eg, a server bank, a group of blade servers, or a multi-processor system).

Memory 1304 stores information within computing device 1300 . In one implementation, memory 1304 is a volatile memory unit or units. In another implementation, memory 1304 is a non-volatile memory unit or units. Memory 1304 may also be another form of computer readable medium, such as a magnetic or optical disk.

Storage device 1306 can provide mass storage for computing device 1300 . In one implementation, the storage device 1306 is a device comprising a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid state memory device, or a storage area network or other configuration of devices. It can be or include a computer readable medium, such as an array. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. An information carrier is a computer or machine readable medium, such as memory 1304 , storage device 1306 , or memory on processor 1302 .

High speed controller 1308 manages bandwidth intensive operations for computing device 1300 , while low speed controller 1312 manages low bandwidth intensive operations. The assignment of these functions is exemplary only. In one implementation, high-speed controller 1308 includes memory 1304, display 1316 (eg, via a graphics processor or accelerator), and a high-speed expansion port (not shown) that can accommodate various expansion cards (not shown). 1310) is connected to. In an implementation, low speed controller 1312 is coupled to storage device 1306 and low speed expansion port 1314 . A low-speed expansion port, which may include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), is one or more input/output devices such as keyboards, pointing devices, scanners, or via a network adapter, for example, a switch or It may be coupled to a networking device such as a router.

Computing device 1300 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 1320 or multiple in a group of such servers. It can also be implemented as part of the rack server system 1324. It may also be implemented in a personal computer, such as laptop computer 1322. Alternatively, components from computing device 1300 may be combined with other components in a mobile device (not shown), such as device 1350 . Each of the devices may include one or more of computing devices 1300 and 1350, and the entire system may be composed of multiple computing devices 1300 and 1350 communicating with each other.

Computing device 1350 includes processor 1352 , memory 1364 , input/output devices such as display 1354 , communication interface 1366 and transceiver 1368 , among other components. Device 1350 may also be provided with a storage device such as a microdrive or other device to provide additional storage. Each of components 1350, 1352, 1364, 1354, 1366 and 1368 are interconnected using various buses, and some components may be mounted on a common motherboard or in other ways suitable.

Processor 1352 may execute instructions within computing device 1350 , including instructions stored in memory 1364 . The processor may be implemented as a chipset of chips that includes discrete and multiple analog and digital processors. The processor may provide coordination of other components of device 1350, such as, for example, user interfaces, applications executed by device 1350, and wireless communications by device 1350.

Processor 1352 can communicate with a user through control interface 1358 and display interface 1356 coupled to display 1354 . Display 1354 may include, for example, TFT LCD (Thin Film Transistor Liquid Crystal Display) and LCD (Light Emitting Diode) or OLED (Organic Light Emitting Diode) displays or other suitable display technologies. Display interface 1356 may include suitable circuitry for driving display 1354 to present graphics and other information to a user. Control interface 1358 may receive commands from a user and transform them for submission to processor 1352 . Additionally, external interface 1362 can be provided for communication with processor 1352 to enable short-range communication of device 1350 with other devices. External interface 1362 can be provided, for example, for wired communication in some implementations or for wireless communication in other implementations, and multiple interfaces can also be used.

Memory 1364 stores information within computing device 1350 . Memory 1364 may be implemented as one or more of a computer readable medium or media, a volatile memory unit or units, and a non-volatile memory unit or units. Expansion memory 1374 may also be provided and connected to device 1350 via expansion interface 1372, which may include, for example, a single in line memory module (SIMM) card interface. The expansion memory 1374 may provide additional storage space for device 1350 or may store applications or other information about device 1350 . In particular, expansion memory 1374 may include instructions that perform or supplement the processes described above, and may also include secure information. Thus, for example, expansion memory 1374 can be provided as a secure module for device 1350 and can be programmed with instructions that allow secure use of device 1350 . Additionally, secure applications may be provided with additional information via the SIMM cards, such as placing identification information on the SIMM card in an unhackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product also includes instructions that, when executed, perform one or more methods as described above. An information carrier is a computer or machine readable medium, such as memory 1364, expansion memory 1374, or memory on processor 1352, for example, that can be received via transceiver 1368 or external interface 1362.

Device 1350 may communicate wirelessly via communication interface 1366, which may include digital signal processing circuitry as needed. Communications interface 1366 may provide for communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur via radio frequency transceiver 1368, for example. Additionally, short-range communications may occur, such as using Bluetooth, Wi-Fi or other transceivers (not shown). Additionally, Global Positioning System (GPS) receiver module 1370 may provide additional navigation and location related radio data to device 1350 that may be used as appropriate by applications running on device 1350 .

Device 1350 can also communicate audibly using audio codec 1360 that can receive information uttered from a user and convert it into usable digital information. Audio codec 1360 may likewise produce audible sound for a user, such as through a speaker in a handset of device 1350, for example. Such sound may include sound from voice telephone calls, may include recorded sound (eg, voice messages, music files, etc.), and may also include sound generated by applications running on device 1350. can include

Computing device 1350 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 1380. It may also be implemented as part of a smartphone 1382, personal digital assistant (PDA), or other similar mobile device.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. . These various implementations may include implementation in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, and may include a storage system , can be coupled to receive data and instructions from and transmit data and instructions to the at least one input device and the at least one output device.

These computer programs (also known as programs, software, software applications or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural language and/or object oriented programming language and/or assembly/machine language. . As used herein, the terms "machine readable medium", "computer readable medium" include machine readable medium that receives machine instructions as a machine readable signal, such that machine instructions and/or data are programmable. Refers to any computer program product, apparatus and/or device used to provide to a processor, eg magnetic disks, optical disks, memory, programmable logic devices (PLDs). The term “machine readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may use a display device (a light-emitting diode (LED), or organic LED (OLED) or liquid crystal display (LCD) to display information to a user). ) monitor/screen) and a keyboard and pointing device through which the user can provide input to the computer, such as a mouse or trackball. Other types of devices may also be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form including acoustic, audio, or tactile input. can

The systems and techniques described herein may be implemented by a user computer or user having a graphical user interface, eg, a backend component such as a data server, a middleware component such as an application server, or an implementation of the systems and techniques described herein. It may be implemented in a computing system that includes a front-end component, such as an interactive web browser, or any combination of one or more of the foregoing back-end, middleware, or front-end components. Components of the system may be interconnected by any form or medium of digital data communication, for example a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

A computing system can include users and servers. Users and servers are generally remote from each other and typically interact through a communication network. The relationship between user and server arises by means of computer programs running on the respective computers and having a user-server relationship with each other.

In some implementations, the computing device shown in FIG. 13 can include sensors that interface with the AR headset/HMD device 1390 to create an augmented environment for viewing content embedded within a physical space. For example, one or more sensors included in computing device 1350 or other computing devices shown in FIG. 13 may provide input to AR headset 1390 or to the AR space in general. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device 1350 can use sensors to determine a detected rotation and/or an absolute position of the computing device within AR space, which can be used as an input into AR space. For example, the computing device 1350 may be integrated into the AR space as a virtual object such as a controller, laser pointer, keyboard, weapon, and the like. The positioning of the computing device/virtual object by the user when integrated into the AR space causes the user to position the computing device to view the virtual object in a particular way in the AR space. For example, if the virtual object represents a laser pointer, the user may operate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in a circle, etc., and can use the device in a similar way to using a laser pointer. In some implementations, a user can aim at a target location using a virtual laser pointer.

In some implementations, one or more input devices included in or coupled to computing device 1350 may be used as input to the AR space. Input devices include a touch screen, keyboard, one or more buttons, trackpad, touchpad, pointing device, mouse, trackball, joystick, camera, microphone, earphone or bud with input, game controller, or other connectable input device. You can, and now you are not limited. When the computing device is incorporated into the AR space, a user interacting with an input device included in computing device 1350 can cause certain actions to occur in the AR space.

In some implementations, the touch screen of computing device 1350 can be rendered as a touch pad in AR space. A user may interact with the touch screen of computing device 1350 . Interactions are rendered according to movement on the touchpad rendered in AR space, for example in the AR headset 1390. The rendered movement can control virtual objects in AR space.

In some implementations, one or more output devices included in computing device 1350 can provide output and/or feedback to a user of AR headset 1390 in AR space. Output and feedback can be visual, tactile or audible. Outputs and/or feedbacks may include, but are not limited to, vibrating, turning one or more lights or strobes on or off or blinking and/or flashing, sounding an alarm, striking a chime, playing a song, playing an audio file, and the like. Output devices may include, but are not limited to, vibration motors, vibrating coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes and speakers.

In some implementations, computing device 1350 can be represented as another object in a computer-generated 3D environment. Interactions by a user with computing device 1350 (eg, rotation, shaking, touching a touchscreen, swiping a finger across a touchscreen) may be interpreted as interactions with an object in AR space. In the example of a laser pointer in AR space, computing device 1350 is viewed as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates the computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from interactions with the AR environment on computing device 1350 or with computing device 1350 on AR headset 1390 . A user's interaction with the computing device may be converted into an interaction with a user interface generated in the AR environment for the controllable device.

In some implementations, computing device 1350 can include a touchscreen. For example, a user may interact with a touchscreen to interact with a user interface for a controllable device. For example, a touchscreen can include user interface elements such as sliders that can control properties of a controllable device.

Computing device 1300 is intended to represent various types of digital computers, including but not limited to laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1350 is intended to represent various types of mobile devices such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components depicted herein, their connections and relationships, and their function are meant to be illustrative only and are not meant to limit implementations of the invention described and/or claimed herein.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.

Additionally, the logic flows depicted in the figures do not necessarily require a specific depicted order, or chronological order, to achieve desired results. Additionally, other steps may be provided, or steps may be omitted from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

Although specific configurations of the described implementations have been described as described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the implementation. It should be understood that these embodiments are illustrative only and not restrictive, and that various changes in form and detail may be made. Any part of the devices and/or methods described herein may be combined in any combination, except for mutually exclusive combinations. Implementations described herein may include various combinations and/or subcombinations of functions, components and/or configurations of the other implementations described.

Claims (25)

As a computer-implemented method,
sending, by the client computing device, a visual content query to a server computing device;
receiving, by the client computing device, a response to the visual content query that identifies a digital supplement and a digital supplement server;
causing, by the client computing device, the visual content query to be transmitted to the digital supplement server; and
causing, by the client computing device, a user interface including information associated with the digital supplement to be displayed. The method of claim 1 , wherein sending the visual content query comprises sending an image to the server computing device. The method of claim 1 , wherein sending the visual content query comprises sending an identifier of an entity in an image to the server computing device. The method of claim 1 , further comprising sending context information to the server computing device. The method of claim 4,
Wherein the context information comprises a location. The method of claim 1 , wherein receiving a response to the visual content query comprises receiving a URL associated with the digital supplement. The method of claim 1 , wherein receiving a response to the visual content query comprises receiving an identifier of an application associated with the digital supplement. The computer implemented method of claim 1 , wherein receiving a response to the visual content query comprises receiving a name, description and image associated with the digital supplement. The computer-implemented method of claim 1 , wherein receiving a response to the visual content query comprises receiving a list of digital supplements comprising the digital supplements. 10. The method of claim 9, wherein the user interface includes information about a number of digital supplements from the list of digital supplements. The method of claim 10,
determining an ordering for the list of digital supplements based on whether an application associated with the digital supplement is installed. The method of claim 1,
further comprising the step of capturing an image;
wherein the visual content query is based on the image. In the computing device,
at least one processor; and
a memory storing instructions, which instructions, when executed by the at least one processor, cause the computing device to:
to capture an image;
send a visual content query based on the image to a server computing device;
receive a response to the visual content query that identifies a digital supplement, the digital supplement comprising augmented reality content that is sized and positioned based on the image; And
A computing device that causes a user interface to be displayed that includes information associated with the digital supplement. The method of claim 13,
Wherein receiving a response to the visual content query comprises receiving an ordered list of digital supplements including the digital supplement. The method of claim 14,
wherein the user interface includes a digital supplement selection panel including user-operable controls associated with a plurality of digital supplements in the ordered list. The method of claim 15
wherein the user-operable controls are ordered on the digital supplement selection panel based on an order provided by the sorted list. The method of claim 13,
The digital supplement is associated with a digital supplement server, and the instructions cause the computing device to:
transmit information to the digital supplement server; And
and receive digital supplement content from the digital supplement server, the digital supplement content based on the transmitted information. 18. The computing device of claim 17, wherein the transmitted information includes context information. As a computer-implemented method,
capturing, by the client computing device, an image;
sending, by the client computing device, a visual content query based on the image to a search server;
receiving, by the client computing device, a response to the visual content query that identifies a digital supplement server;
causing, by the client computing device, the image to be transmitted to the digital supplement server;
receiving, by the client computing device, digital supplement content from the digital supplement server; and
causing the digital supplement content to be displayed by the client computing device. The method of claim 19
wherein the visual content query includes the image, and causing the image to be transmitted to the digital supplement server comprises sending instructions to the search server to provide the image to the digital supplement server. How it is implemented with . The method of claim 19
wherein causing the image to be transmitted to the digital supplement server comprises transmitting the image to the digital supplement server. The method of claim 19
wherein the digital supplement comprises augmented reality content that is sized and positioned based on the image. As a computing device,
at least one processor; and
A computing device comprising a memory that stores instructions, the instructions, when executed by the at least one processor, causing the computing device to perform the method of any one of claims 1-12 or 19-22. A computer program stored on a computer-readable storage medium, the computer program comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 12 or 19 to 22. A computer program that makes it happen. A computer-readable storage medium containing instructions, which when executed by one or more processors cause the one or more processors to perform the method of any one of claims 1-12 or 19-22. media.

Images