KR20200136030A - Digital replenishment association and search for visual search - Google Patents

Abstract

Systems and methods for identifying and searching content for visual search are provided. An exemplary method includes receiving data specifying a digital supplement. The data may identify a digital supplement and a supplement anchor for associating the digital supplement with visual content. The method also includes the steps of creating a data structure instance specifying the digital supplement and the supplement anchor, and after creating the data structure instance, storing the data structure instance in a database including a plurality of other data structure instances. And enabling triggering of the digital supplement by an image based at least on what it does. Each of the other data structure instances may specify a digital supplement and one or more supplement anchors.

Description

Digital replenishment association and search for visual search

Cross-reference to related applications

This application claims priority to U.S. Application No. 16/014,520, "Digital Supplement Association and Search for Visual Search," filed June 21, 2018, which is incorporated herein by reference in its entirety.

Mobile computing devices such as smartphones often include cameras. Such cameras can be used to capture images of entities in the environment around the computing device. Various types of content or experiences associated with these entities may be provided to users through mobile computing devices.

The present disclosure describes a system and method for digital supplement association and retrieval for visual retrieval. For example, the systems and techniques described herein can be used to provide digital supplements such as augmented reality (AR) content or experiences responsive to visual search. The visual search may be based on an image or an entity identified within the image, for example. For example, digital supplements may include providing information or functionality associated with an image.

One aspect is a computer-implemented method comprising receiving data specifying a digital supplement, the data identifying a digital supplement and a supplement anchor for associating the digital supplement with visual content. The method also includes creating a data structure instance specifying the digital replenishment and the replenishment anchor. The method further comprises, after creating the data structure instance, enabling triggering of the digital supplement by an image based at least on storing the data structure instance in a database comprising a plurality of other data structure instances. do. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Yet another aspect is a computing device that includes at least one processor and a memory that stores instructions. The instructions, when executed by at least one processor, cause the computing device to receive data specifying a digital supplement, the data being a digital supplement and a supplement anchor for associating the digital supplement with visual content and Identify context information. The instructions also cause the computing device to create a data structure instance specifying the digital supplement, the supplement anchor, and the context information. The instructions enable a computer device to create the data structure instance and then trigger the digital replenishment by an image based at least on storing the data structure instance in a database containing a plurality of other data structure instances. . Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Another aspect is a computer-implemented method comprising receiving a visual content query from a computing device and identifying a replenishment anchor based on the visual content query. The method also includes generating an ordered list of digital supplements based on the identified supplement anchors and transmitting the ordered list to the client computing device.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other configurations will become apparent from the description, drawings and claims below.

1 is a block diagram illustrating a system according to an exemplary implementation.
2 is a third person view of an exemplary physical space in which the embodiment of the client computing device of FIG. 1 accesses a digital supplement.
3 is a diagram of an exemplary method for enabling triggering of a digital replenishment according to an implementation described herein.
4 is a diagram of an exemplary method for enabling triggering of digital replenishment in accordance with an implementation described herein.
5 is a diagram of an exemplary method of retrieving and presenting a digital supplement according to an implementation described herein.
6 is a diagram of an exemplary method of identifying and presenting a digital supplement based on an image, in accordance with an implementation described herein.
7A-7C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content retrieval and displaying digital supplements.
8A-8C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content retrieval and displaying digital supplements.
9A-9C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content retrieval and displaying digital supplements.
10A-10C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content searches and displaying digital supplements.
11A-11C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 for performing visual content retrieval within a repository.
12A-12C are schematic diagrams of user interface screens displayed by the embodiment of the client computing device of FIG. 1 during visual content retrieval.
13 is a schematic diagram of an example of a computer device and a mobile computer device that can be used to implement the techniques described herein.
Detailed reference will be made to non-limiting embodiments of the invention, examples of which are shown in the accompanying drawings. Examples are described below with reference to the drawings, and like reference numerals refer to like elements. When the same reference numerals are shown, the corresponding description(s) are not repeated and the interested reader refers to the drawing(s) discussed above for description of similar element(s).

This disclosure describes improvements in technology that simplify identification and presentation of digital supplements based on visual content. Some implementations of the techniques described herein generate an index of digital supplements associated with certain types of visual content and provide such digital supplements in response to a visual content query received from a client computing device. This index allows users to access related digital supplements provided by network-accessible resources (e.g. web pages) deployed around the world. This can provide a functional data structure that enables more efficient information retrieval.

For example, a client computing device, such as a smartphone, can capture an image of a replenishment anchor, such as an entity. The client computing device can send a visual content query based on the image to the server computing device to retrieve the digital refill associated with the identified replenishment anchor. In some implementations, the supplement anchor is based on the physical environment around the client computing device, and the digital supplement is virtual content that can supplement the user's experience in the physical environment.

The visual content query may include an image or data determined from the image (eg, an indicator of an identified supplement anchor). An example of data determined from an image is text extracted from an image using optical character recognition, for example. Other examples of data extracted from an image are a value read from a barcode, a QR code, etc. of an image, an entity identified in the image, an identifier or description of a product or entity type.

For example, a neural network system, such as a convolutional neural network system, can be used to identify entities, products, or entity types in an image. The identifier or description of an entity, product, or entity type may include metadata or references to records in a database associated with the entity, product, or entity type. Non-limiting examples of entities include buildings, works of art, products, books, posters, photos, catalogs, signs, documents (eg, business cards, receipts, coupons, catalogs), people and body parts.

Various types of digital refills associated with refill anchors are available. Digital supplements may be provided by network-accessible resources such as web pages available on the Internet. There is a need for a way to find and deliver these digital supplements in response to visual content queries. Some implementations create and maintain an index of digital supplements associated with the entity for use in responding to visual content queries. For example, the index may be populated by determining whether the network accessible resource contains or provides a digital supplement, and crawling the network accessible resource to determine the supplement anchor associated with the digital supplement.

For example, a network accessible resource may include metadata that identifies the replenishment anchor (eg, text, code, entity, or entity type) to which the digital replenishment is associated. Metadata may be included in a network accessible resource in response to a Hypertext Transfer Protocol (HTTP) request. Metadata can be provided in various formats such as Extensible Markup Language (XML), JavaScript Object Notation (JSON), or other formats.

Metadata for digital supplements may include one or more of the following: type indicator, anchor indicator, name, description, snippet of content (i.e., an excerpt or preview of a portion of the content), associated image, digital A link, such as a URL to the supplement, and the identifier of the application associated with the digital supplement. The metadata may also include information about the publisher of the digital supplement. For example, the metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some implementations, the metadata includes contextual information related to providing the digital supplement. For example, the metadata may also include conditions (eg, geographic conditions, required applications) associated with providing or accessing digital supplements.

The identified digital replenishment can be added to an index stored in memory. In at least some implementations, the associated replenishment anchor for the digital replenishment is used as the key for the index. Digital supplements can also be associated with various scores. For example, a digital replenishment refers to the digital replenishment or a network accessible resource associated with the digital replenishment (e.g., while crawling a network-accessible resource) and how many other links are found and provide these links. May be associated with a prestige score based on the prestige of a network accessible resource. As another example, the digital replenishment may be associated with one or more relevance scores corresponding to the relevance of the digital replenishment (or associated network accessible resource) to a particular anchor. The relevance score can be associated with a keyword or topic. The relevance score may be determined based on one or more of the content of the digital supplement, the content of a resource accessible to the network, and the content of a link to the network accessible resource (eg, text).

1 is a block diagram illustrating a system 100 according to an exemplary implementation. System 100 may associate a digital supplement with an entity or entity type, and may search for a digital supplement in response to a visual search. Visual search is a search based on visual content. For example, a visual search may be performed based on a visual content query. Visual content queries are queries based on images or other visual content. For example, a visual content query can include an image. In some implementations, the visual content query can include text or data based on images. For example, text or data may be generated by recognizing one or more entities in an image. Some visual content queries do not include images (for example, visual content queries contain only text or data generated from images). In some implementations, system 100 includes a client computing device 102, a search server 152 and a digital supplement server 172. Also shown is a network 190 through which the client computing device 102, the search server 152, and the digital supplement server 172 can communicate.

The client computing device 102 may include a processor assembly 104, a communication module 106, a sensor system 110 and a memory 120. The sensor system 110 may include various sensors such as a camera assembly 112, an inertial motion unit (IMU) 114, and a Global Positioning System (GPS) receiver 116. Implementations of the sensor system 110 include, for example, a light sensor, an audio sensor, an image sensor, a distance and/or proximity sensor, for example a contact sensor such as a capacitive sensor, a timer and/or other sensors and/or sensors. Different combinations of them. In some implementations, the client computing device 102 is a mobile device (eg, a smartphone).

The camera assembly 112 captures an image or video of a physical space around the client computing device 102. The camera assembly 112 may include one or more cameras. Camera assembly 112 may also include an infrared camera. Images captured with camera assembly 112 can be used to identify replenishment anchors and form visual content queries.

In some implementations, the image captured by camera assembly 112 may also be based on a representation of that physical space received from memory 120 or an external computing device, within a physical space, such as an internal space, of the client computing device 102 Can be used to determine the location and direction of In some implementations, the representation of the physical space may include a visual feature of the physical space (eg, a feature extracted from an image of the physical space). The representation may also include positioning data associated with corresponding features that can be used by the visual positioning system to determine a location and/or position within the physical space based on one or more images of the physical space. The representation may also include a three-dimensional model of at least some structure within the physical space. In some embodiments, the representation does not include a three-dimensional model of physical space.

The IMU 114 may detect motion, movement, and/or acceleration of the client computing device. IMU 114 may include a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the client computing device 102 may be detected and tracked based on data provided by the IMU 114 or GPS receiver 116.

The GPS receiver 116 may receive a signal emitted from a GPS satellite. The signal contains the time and position of the satellite. Based on receiving signals from multiple satellites (eg, at least four), the GPS receiver 116 can determine the global position of the client computing device 102.

The memory 120 may include an application 122, another application 140 and a device positioning system 142. Other applications 140 include any other applications installed or otherwise available for execution on the client computing device 102. In some implementations, application 122 may cause one of the other applications 140 to be launched to provide a digital supplement. In some implementations, some digital supplements may be available only if other applications 140 are associated with the digital supplement or include specific applications required to provide the digital supplement.

The device positioning system 142 determines the position of the client computing device 102. The device positioning system 142 may use the sensor system 110 to determine the location and orientation of the client computing device 102 globally or within a physical space. In some implementations, the device positioning system 142 determines the location of the client computing device 102 based on, for example, cellular triangulation.

In some implementations, the client computing device 102 has an image captured by the camera assembly 112 to determine a 6 DOF pose (e.g., position and orientation) of the client computing device 102 within the physical space. Alternatively, it may include a visual positioning system that compares features extracted from such images with known configurations of features in the representation of the physical space.

Applications 122 may include replenishment anchor identification engine 124, digital replenishment search engine 126, digital replenishment presentation engine 128 and user interface engine 130. Some implementations of application 122 may include fewer, additional, or other components.

Refill anchor identification engine 124 identifies replenishment anchors based, for example, on images captured with camera assembly 112. In some implementations, the supplement anchor identification engine 124 analyzes the image to identify the text. You can then use the text to identify the anchor. For example, text can be mapped to nodes in a knowledge graph. For example, text may be recognized as the name of an entity such as a person, place, product, building, work of art, movie, or other type of entity. In some implementations, text may be recognized as phrases generally associated with a particular entity or phrases describing a particular entity. For example, text may be recognized as an anchor associated with a specific entity.

In some implementations, the supplement anchor identification engine 124 identifies one or more codes within the image, such as barcodes, QR codes, or other types of codes. Then you can map the code to the replenishment anchor.

The replenishment anchor identification engine 124 may include a machine learning module capable of recognizing at least some types of entities within an image. For example, the machine learning module may include a neural network system. A neural network is a computational model used in machine learning and consists of nodes organized in layers with weighted connections. Neural network training uses training examples, each of which is an input and a desired output to determine the weighted values for the connections between the layers, increasing the likelihood of the neural network giving the desired output for a given input over a series of iterative rounds. to be. During each training round, the weights are adjusted to correct the wrong output values. Once trained, the neural network can be used to predict the output based on the input provided.

In some embodiments, the neural network system comprises a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. The convolutional layer is a layer that calculates a layer value based on applying a kernel function to a subset of previous layer values. Neural network training may include adjusting the weights of the kernel function based on the training example. In general, the same kernel function is used to calculate each value of the convolutional layer. Thus, in a neural network, there are far fewer weights that must be learned while training a convolutional layer than a fully connected layer (e.g., a layer in which each value of the layer is calculated as an independently adjusted weight combination of each value of the previous layer). In general, the convolutional layer has a low weight, so training and using the convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.

After the replenishment anchor identification engine 124 recognizes the entity or entity type in the image, a textual description of the entity or entity type may be generated. Also, an entity or entity type can be mapped to a supplement anchor. In some implementations, the replenishment anchor is associated with one or more digital replenishments.

In some implementations, the replenishment anchor identification engine 124 determines a confidence score for the recognized anchor. A higher confidence score may indicate that content from the image (eg, image, extracted text, barcode, QR code) is more likely to be associated with the determined anchor than if a lower confidence score was determined.

The example of FIG. 1 shows the replenishment anchor identification engine 124 as a component of the application 122 on the client computing device 102, but some implementations include the replenishment anchor identification engine on the search server 152. . For example, the client computing device 102 can send the image captured by the camera assembly 112 to the search server 152, which can then identify the replenishment anchor in the image. .

In some implementations, the refill anchor identification engine 124 identifies potential refill anchors. For example, the replenishment anchor identification engine 124 can identify (recognize) various entities within the image. The identifier of the recognized entity may be sent to the search server 152, which may determine which of the entities are associated with any supplement anchor. In some implementations, the search server 152 may use the identified entity as context information even if the identified entity is not a supplement anchor.

The digital supplement search engine 126 searches for digital supplements. For example, the digital replenishment search engine 126 may search for a digital replenishment associated with the replenishment anchor identified by the replenishment anchor identification engine 124. In some implementations, digital supplement search engine 126 searches for digital supplements from search server 152 or digital supplement server 172.

For example, after a replenishment anchor has been identified, the digital replenishment search engine 126 may search for one or more digital replenishments associated with the identified replenishment anchor. The digital replenishment search engine 126 may generate a visual content query including an image (or an identifier of a replenishment anchor or entity in the image) and transmit the visual content query to the search server 152. The visual content query may also include contextual information such as the location of the client computing device 102. In some implementations, data associated with a digital supplement, such as a name, image, or description, is retrieved and presented to a user (eg, by user interface engine 130). When multiple digital supplements are presented, the user can select one of the digital supplements through a user interface generated by the user interface engine 130.

The digital supplement presentation engine 128 presents the digital supplement or causes it to be presented by the client computing device 102. In some implementations, the digital supplement presentation engine 128 causes the client computing device to launch one of the other applications 140. In some implementations, the digital supplement presentation engine 128 causes information or content to be displayed. For example, the digital supplement presentation engine 128 may cause the user interface engine 130 to create a user interface that includes information or content from the digital supplement to be displayed by the client computing device 102. In some implementations, the digital supplement presentation engine 128 is triggered by the digital supplement search engine 126 that searches for digital supplements. The digital supplement presentation engine 128 may trigger the display device 108 to display content associated with the digital supplement. In some implementations, the digital supplement presentation engine 128 causes the digital supplement to be displayed at a different time than when the digital supplement search engine 126 searches for the digital supplement. For example, a digital supplement may be retrieved in response to a visual content query at a first time, and the digital supplement may be presented at a second time. For example, at the first time (for example, while a user is viewing a catalog or visiting a store), a catalog or store might search for a digital supplement in response to a query for visual content based on the image of the home or furniture. I can. A digital supplement containing the home furniture or AR content of the furniture may be presented at a second time (eg, while the user is in the home furniture or room where the furniture can be placed).

The user interface engine 130 creates a user interface. The user interface engine 130 may also cause the client computing device 102 to display the generated user interface. The generated user interface may, for example, display information or content of a digital supplement. In some implementations, the user interface engine 130 creates a user interface that includes a number of user operable controls each associated with a digital supplement. For example, the user can activate one of the controls that the user can operate (e.g., by touching the control on the touch screen, clicking the control using a mouse or other input device, or otherwise activating the control). ).

Search server 152 is a computing device. The search server 152 may respond to a search request, such as a visual content query. The response may include one or more digital supplements potentially related to the visual content query. In some implementations, search server 152 includes memory 160, processor assembly 154 and communication module 156. The memory 160 may include a content crawler 162, a digital supplement search engine 164 and a digital supplement data store 166.

Content crawler 162 may crawl network accessible resources to identify digital supplements. For example, the content crawler 162 may access webpages accessible through the Internet, such as webpages provided by the digital supplement server 172. Crawling the network accessible resource may include requesting the resource from a web server and parsing at least a portion of the resource. Digital supplements can be identified based on metadata provided by network-accessible resources, such as XML or JSON data that provides information about the digital supplement. In some implementations, the crawler identifies network accessible resources based on extracting links from previously crawled network accessible resources. The content crawler 162 may also identify network accessible resources to crawl based on receiving input submitted by the user. For example, a user may submit a URL (or other information) to a network accessible resource, including a digital supplement, through a web form or an application programming interface (API). In some implementations, the content crawler 162 creates an index of the identified digital supplement. The content crawler 162 may also generate a score associated with a digital supplement, such as a relevance score or popularity (fame) score.

The digital supplement search engine 164 receives the search query and generates a response that may include one or more potentially related digital supplements. For example, the digital supplement search engine 164 can receive a visual content query from the client computing device 102. Visual content queries can include images. The digital replenishment search engine 164 can identify the replenishment anchor in the image and, based on the identified replenishment anchor, can identify a related or potentially associated digital replenishment. The digital replenishment search engine 164 can send a response to the client computing device 102 that includes the digital replenishment or information that can be used to access the digital replenishment. In some implementations, digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a list of digital supplements could be included in the response to a query. The list may be sorted based on relevance to the replenishment anchor, popularity, or other attributes of the digital replenishment.

The visual content query may include, for example, an image captured by the camera assembly 112 or text or other data associated with an image captured by the camera assembly 112. The visual content query may also include other information such as the location of the client computing device 102 or an identifier of a user of the client computing device 102. In some implementations, the search server 152 may determine the approximate location of the client computing device 102 from the user identifier (e.g., a client computing device where the user associates information about the user's location with the user's account). (If you have enabled location services at 102).

The digital replenishment data store 166 stores information about digital replenishment. In some implementations, the digital supplement data store 166 includes an index of digital supplements. For example, the index may be created by the content crawler 162. The digital supplement search engine 164 may use the index to respond to the search query.

Digital replenishment server 172 is a computing device. Digital replenishment server 172 provides digital replenishment. In some implementations, digital supplement server 172 includes memory 180, processor assembly 174 and communication module 176. The memory 180 may include a digital supplement 182 and metadata 184. In some implementations, memory 180 may include other network accessible resources such as webpages that are not necessarily digital supplements. For example, memory 180 may store webpages including metadata to provide details about one or more digital supplements and how to access such digital supplements. Additionally, the memory 180 may include a resource service engine such as a web server that responds to requests such as HTTP requests with network accessible resources such as web pages and digital supplements, for example.

Digital supplement 182 is any type of content that can be provided as a supplement to something in the physical environment around the user. The digital supplement 182 may also include any type of content that may supplement the stored image (eg, the previous physical environment around the user). For example, a digital supplement may be associated with a supplement anchor, such as an image, object, or product identified in an image or location. The digital supplement 182 may include one or more images, audio content, text data, video, games, data files, applications, or structured text documents. Examples of structured text documents include Hypertext Markup Language (HTML) documents, XML documents, and other types of structured text documents.

Digital replenishment 182 can cause an application to be launched and define parameters for that application. The digital supplement 182 may also cause the request to be sent to the server (eg, HTTP request) and may define parameters for that request. In some implementations, digital replenishment 182 is initiated as a workflow for completing an activity, such as a workflow for completing a purchase. For example, the digital supplement 182 may send an HTTP request to the server to add a specific product to the user's shopping cart, add a coupon code, and retrieve a purchase confirmation page.

Metadata 184 is data describing the digital supplement. Metadata 184 may describe one or more digital supplements provided by digital supplement server 172 or provided elsewhere. Metadata 184 for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a preview snippet or excerpt, an associated image, a URL to the digital supplement, such as The identifier of the application associated with the link and digital supplement. The metadata may also include information about the publisher of the digital supplement, such as the publisher name, publisher description, and an image or icon associated with the publisher. In some implementations, the metadata also includes context information about the digital supplement or context information that must be satisfied to provide the digital supplement. For example, the metadata may include conditions that must be met in order to access the digital supplement (eg, geographic conditions, client computing device requirements, required applications). Exemplary context information includes a location, an entity identified within the image, or multiple entities identified within the image (eg, some digital supplements may require a combination of entities to be recognized within the image). The recognized entity may be a supplement anchor. In some implementations, the recognized entity provides context information rather than a supplement anchor. The metadata 184 may also include a supplement anchor (eg, text, code, entity or entity type) associated with the digital supplement.

The metadata 184 may be stored in various formats. In some implementations, metadata 184 is stored in a database. Metadata 184 may also be stored as an XML file, a JSON file, or other format file. In some implementations, the digital supplement server 172 retrieves the metadata 184 from the database, and formats the metadata 184 in XML, JSON, or other way to request a request from the client or search server 152. Provides a response to. For example, search server 152 may access metadata 184 to generate data that is stored in digital replenishment data store 166 and used to respond to search requests from client computing device 102. have.

Communication module 106 includes one or more devices for communicating with another computing device, such as search server 152 or digital supplement server 172. The communication module 106 may communicate through a wireless or wired network such as the network 190. The communication module 156 of the search server 152 and the communication module 176 of the digital supplement server 172 may be similar to the communication module 106.

The display device 108 may include, for example, a liquid crystal display (LCD) screen, a light emitting diode (LED) screen, an organic light emitting diode (OLED) screen, a touch screen or other screen or display for displaying an image or information to a user. I can. In some implementations, the display device 108 includes a light projector arranged to project light onto a portion of the user's eye.

Memory 120 may include one or more non-transitory computer-readable storage media. Memory 120, such as generating a visual content query based on the captured image, sending a visual content query, receiving a response to the visual content, and presenting the identified digital supplement in response to the visual content query. It may store instructions and data usable by the client computing device 102 to implement the techniques described herein. The memory 160 of the search server 152 and the memory 180 of the digital supplement server 172 may be similar to the memory 120, and the description of the search server 152 and the digital supplement server 172, respectively. You can store data commands that can be used to implement.

Processor assembly 104 includes one or more devices capable of executing instructions, such as instructions stored by memory 120, to perform various tasks associated with digital replenishment association and retrieval for visual retrieval. For example, the processor assembly 104 may include a central processing device (CPU) and/or a graphics processing device (GPU). For example, if you have a GPU, some image/video rendering tasks, such as creating a user interface and displaying or displaying some of the digital supplements, can be offloaded from the CPU to the GPU. In some implementations, some image recognition tasks may be offloaded from the CPU to the GPU.

Although Figure 1 does not show it, some implementations include a head-mounted display (HMD) device. The HMD may be a separate device from the client computing device 102 or the client computing device 102 may comprise an HMD. In some implementations, computing device 102 communicates with HMD 104 via a cable. For example, the client computing device 102 may transmit a video signal and/or an audio signal to the HMD for display for a user, and the HMD may transmit motion, position and/or orientation information to the client computing device 102. I can.

The client computing device 102 may also include various user input components (not shown) such as a controller that communicates with the client computing device 102 using a wireless communication protocol. In some implementations, the client computing device 102 is via a wired connection (e.g., a Universal Serial Bus (USB) cable) or a wireless communication protocol (e.g., any WiFi protocol, any BlueTooth protocol, ZigBee, etc. ) Through the HMD (not shown). In some implementations, the client computing device 102 is a component of the HMD and can be contained within the housing of the HMD.

The network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or another network. The client computing device 102 may receive audio/video signals over a network that may be provided as part of a digital supplement in an example implementation.

2 is a third person view of an exemplary physical space 200 in which an embodiment of a client computing device 102 accesses a digital supplement. In this example, physical space 200 includes object 222. Here, the object 222 is a work hung on the wall of the physical space 200. The object 222 is contained within the field of view 204 of the camera assembly 112 of the client computing device 102.

An exemplary user interface screen 206 is also shown. The user interface screen 206 may be generated, for example, by the user interface engine 130 of the client computing device 102. The user interface screen 206 includes an image display panel 208 and a digital supplement selection panel 210. The image display panel 208 shows an image. For example, the image display panel 208 can show an image corresponding to a real-time feed from the camera assembly 112 of the client computing device 102. In some implementations, the image display panel 208 shows a previously captured image or an image retrieved from the memory 120 of the client computing device 102.

In some implementations, the user interface screen 206 is displayed to the user on a display device of the client computing device 102. In some implementations, the user interface screen 206 may be overlaid on an image of the physical space (or a video feed captured by the camera of the computing device). In addition, the user interface screen 206 may be displayed as AR content in the user's field of view using the HMD worn by the user.

The image display panel 208 may also include annotations or user interface elements that may be associated with the image. For example, the image display panel 208 may include an indicator indicating that an object (eg, object 222) in the image has been recognized as a supplement anchor. The indicator may include user-operable controls to access or view information about the digital refill associated with the identified refill anchor. In some situations, the image displayed on the image display panel 208 may include multiple objects that are recognized as replenishment anchors, and the image display panel 208 overlays the image to identify such replenishment anchors. May include comments of.

The replenishment anchor may be recognized by the replenishment anchor identification engine of the client computing device 102. In some implementations, the replenishment anchor is identified by sending the image to the search server 152. The search server 152 can then analyze the image and identify replenishment anchors in the image. In some implementations, search server 152 may send one or more of a location (eg, image coordinates) or dimension of any identified object associated with the replenishment anchor to the client computing device 102. The client computing device 102 may then update the user interface screen to show the annotation identifying the replenishment anchor (or associated object) in the image. In some implementations, the client computing device 102 is to the camera assembly 112 (e.g., the refill anchor identification engine 124 can track the refill anchor identified by the search server 152). The position of the replenishment anchor (or associated object) in the video stream captured by (eg, sequentially captured images in sequence) can be tracked.

The digital supplement selection panel 210 allows a user to select a digital supplement to be presented. For example, the digital replenishment selection panel 210 may include a menu including user-operable controls each associated with the digital replenishment. In this example, the digital replenishment selection panel 210 includes a user operable control 212 and a user operable control 214 each containing information about the associated digital replenishment. For example, the user operable control may display one or more of a name (or title), a brief description, and an image associated with the digital supplement that may be received from the search server 152. Upon activation of user operable control 212 or user operable control 214, the content of the associated digital supplement may be presented to the user. Presenting the digital supplement to the user may include causing the client computing device 102 to display a user interface screen comprising images, videos, text, other content, or combinations thereof from the digital supplement. In some implementations, the digital supplement content is displayed as an overlay on the image display panel 208 over the image or camera feed. The digital supplement content may be 3D augmented reality content.

In some implementations, presenting the digital supplement includes running an application installed on the client computing device 102 (eg, one of the other applications 140). Presenting the digital supplement may also include sending the request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as the identifier of the product or object identified within the image. In some implementations, the image (or other content) of the visual content query is passed as a parameter with the request. Images may be provided through an API associated with digital supplement server 172. In some implementations, the client computing device 102 transmits the image to the digital supplement server 172. In some implementations, the search server 152 can send the image to the digital supplement server 172. For example, in response to the user selecting a digital supplement, the client computing device 102 may send an indicator of the selection to the search server 152, and the search server 152 may send the image to the corresponding digital supplement. Can be sent to the server. The client computing device 102 can also send the URL to a location on the search server 152 that the digital supplement server 172 can use to access the image. Advantageously, such implementations can reduce the amount of data the client computing device has to transmit.

Digital supplements associated with user-operable controls 212 can cause information about the object 222, such as information from a museum, to be displayed. Digital replenishment associated with user-operable control 214 may cause information related to museum tours to be displayed. For example, presenting a digital supplement may indicate that a stop on a museum tour has been completed and information about the next stop may be displayed.

3 is a diagram of an exemplary method 300 that enables triggering of digital replenishment in accordance with an implementation described herein. This method 300 is performed, for example, by the content crawler 162 of the search server 152 to allow a user to access the digital supplement based on a visual content query.

In operation 302, data specifying the digital supplement is received. The data can identify digital supplements and situations in which digital supplements should be provided. Data specifying the digital supplement can be received in a variety of ways. For example, data specifying the digital supplement may be received from a network accessible resource, such as a webpage containing metadata about the digital supplement. Data specifying the digital supplement may also be received via an API or form provided by, for example, search server 152. Data specifying the digital supplement can be received from a memory location or data store.

The data for the digital supplement may include access data usable by the client computing device to access the digital supplement. For example, the access data may include a URL of the digital supplement and a parameter to be passed to the URL. The access data may include an application identifier and parameters for the application. The data for the digital supplement may include descriptive data for the digital supplement. The descriptive data can be used by the client computing device to present information about the digital supplement to the user (eg, in a menu where the user can select the digital supplement). The description data may include, for example, a name (or title, description, publisher name and image). Data regarding the digital replenishment may also include the identifier of the replenishment anchor.

In operation 304, a data structure instance is created based on the received data. For example, the data structure can be a record of a database. The database may be a relational database, and the data structure instance may be associated with one or more records (eg, via foreign keys) associated with the supplement anchor.

In operation 306, after creating the data structure instance, the digital replenishment by visual content query is enabled. For example, a database field associated with a data structure instance can be set to active so that the digital supplement search engine 164 can access and return the associated digital supplement. In some implementations, triggering the digital replenishment may include saving or committing a database record. In some implementations, enabling retrieval of the digital supplement includes enabling triggering of the digital supplement by a client computing device. For example, after the instance is created, the digital replenishment can be returned to the client computing device in response to the search, and activated or presented by the client computing device.

4 is a diagram of an exemplary method 400 for enabling triggering of digital replenishment in accordance with an implementation described herein. This method 400 is performed, for example, by the content crawler 162 of the search server 152 to enable a user to access the digital supplement based on a visual content query.

In operation 402, the network accessible resource is analyzed. In some implementations, the network accessible resource is, for example, a webpage served by the digital supplement server 172. In some implementations, a set of network accessible resources is analyzed. The set of network-accessible resources can be created based on a form or submission via API. In some implementations, the set of network accessible resources can be created by crawling other network accessible resources to identify URLs. This crawling process can be performed recursively.

In operation 404, metadata associated with the digital supplement in the network accessible resource is identified. In some implementations, the network accessible resource may include an indicator of metadata associated with the digital supplement. For example, the network accessible resource may include a tag that identifies a portion of the network accessible resource including metadata. Tags can be XML tags with specific types or attributes. The tag may be an HTML tag such as a script tag including a JSON data structure including metadata.

In operation 406, a digital supplement data structure instance based on the metadata is created. Operation 406 may be similar to operation 304.

In operation 408, a visual content query is received. The visual content query may be sent by a client computing device such as client computing device 102, for example. In some implementations, the visual content query includes images. The visual content query can include textual data describing the image. For example, the textual data may include an identifier of a replenishment anchor in an image captured by the camera assembly of the client computing device. In some implementations, the visual content query also includes other information, such as the location of the client computing device or an identifier of a user account associated with the client computing device.

In operation 410, multiple instances of the digital supplement data structure are identified based on the visual content query. In some implementations, replenishment anchors are identified within the image provided in the visual content query. Replenishment anchors can be used to query an index or database for a related digital replenishment. In some implementations, other data provided with the query may also be used to identify the digital supplement, such as information associated with the location of the client computing device or user account. In some embodiments, multiple replenishment anchors are used to identify related replenishment anchors.

In operation 412, an alignment of multiple instances of the digital supplement data structure is determined. The alignment may be based on various scores associated with the digital supplement or the relevance of the digital supplement to a visual content query. In some implementations, a relevance score corresponding to the relevance of the digital supplement to a visual content query is used to sort multiple instances of the digital supplement data structure.

The relevance score is based on one or more of the content of the digital supplement, the content of a network accessible resource that is linked to the digital supplement (or a network accessible resource associated with the digital supplement), and a link to the digital supplement of another network accessible resource. It can be determined from a number of factors such as link text or content.

The score can be based on a popularity metric. The reputation metric is an example of a popularity metric. The reputation metric may be based on a combination of how many other network resources are linked to the digital supplement and the reputation score of other network accessible resources. In some implementations, the popularity score may be based on how often or how often the digital resource was selected. In some implementations, the popularity score may correspond to how often digital resources are selected for visual content query.

The score can be determined or retrieved from a data store or API. In some implementations, the API is accessed to retrieve scores for digital supplements. For example, the score may be retrieved from a search engine that has determined relevance and/or popularity to a digital resource in relation to a search term based on a supplement anchor.

The multiple digital supplement data structures may also be ordered based on frequency of use by a particular user (eg, a user of a client computing device) or recent usage by a particular user. In some implementations, multiple digital supplement data structures are randomly ordered.

In operation 414, the visual content query is answered based on the number of instances of the digital supplement data structure. For example, information associated with multiple instances of the digital supplement data structure may be transmitted to the client computing device in the order determined in operation 412. In some implementations, the information includes descriptive data that may be displayed in a menu or other type of user interface configured to receive user selections of the digital supplement. The information may also include access data that can be used by the client computing device to access or present the digital supplement.

5 is a diagram of an exemplary method 500 of retrieving and presenting digital supplements in accordance with an implementation described herein. The method 500 may be performed, for example, by the application 122 of the client computing device 102 to identify and access the digital supplement based on a visual content query.

In operation 502, a visual content query based on the image is transmitted to a server computing device (eg, search server 152). For example, an image may be captured with the camera assembly 112 of the client computing device 102. The image may also be a stored image, such as an image that was previously captured by the camera assembly 112. In some implementations, the visual content query includes only images. In some implementations, the visual content query includes additional information. For example, the visual content query may include information such as the location of the client computing device 102 or an identifier of an account associated with a user of the client computing device 102. Application 122 can also identify anchors in the image (eg, using replenishment anchor identification engine 124). The visual content query may include the identifier of the identified anchor (eg, text, number, or other type of identifier). In at least some implementations, the visual content query does not include an image.

In some implementations, sending the visual content query to the server includes an API call. In some implementations, sending the visual content query to the server includes calling an API provided by the server. In some implementations, sending the visual content query to the server includes submitting the form (eg, submitting a GET or POST request) using the HTTP protocol.

In operation 504, a response to a visual content query identifying a digital supplement is received. The response may be received from the search server 152 over the network 190. The response may include one or more digital supplements that have been identified based on the visual content query by the search server 152. For example, the response may include an arrangement of data associated with the digital supplement. In some implementations, the data associated with the digital supplement can include descriptive data that can be used to present a user selectable digital supplement option. For example, the description data may include a name, a brief description, a publisher name and an image. The data may include access data such as URLs and parameters to include with requests via URL or application name and associated parameters. The data may also include the location, coordinates, or dimensions of the replenishment anchor in the image sent with the visual content query (eg, if the replenishment anchor is identified by search server 152).

In operation 506, a user interface screen including information associated with the digital supplement is displayed. In some implementations, the user interface screen includes annotations overlaying the identified replenishment anchor (eg, based on the coordinates provided). The annotation may provide information about the object in the image associated with the identified replenishment anchor. Annotations can include user-operable controls that can be activated to present or activate digital supplements. The user interface screen may also include a digital supplement selection panel that can be used to select from a number of digital supplements identified in the response received in operation 504. In some implementations, the user interface screen may be generated by a web browser that opens a URL specified by the digital supplement. User interface screens can be created by other applications launched to provide digital supplements.

6 is a diagram of an exemplary method 600 of identifying and presenting a digital supplement based on an image, in accordance with an implementation described herein. The method 600 may be performed, for example, by the application 122 of the client computing device 102 to identify and access the digital supplement based on a visual content query.

In operation 602, an image is captured. For example, an image may be captured by the camera assembly 112 of the client computing device 102. In some implementations, a sequence of images (ie, video) may be captured by the camera assembly 112.

In operation 604, a visual content query based on the image is transmitted to a server computing device, such as search server 152. Operation 604 may be similar to operation 502. In embodiments in which image sequences are captured, the visual content query may include multiple images or image sequences. In some implementations, the sequence of images can be streamed to a server computing device.

In operation 606, a response to a visual content query identifying a number of digital supplements is received. Operation 606 may be similar to operation 504 previously described.

In operation 608, a user interface screen is displayed including a user operable control for selecting a digital supplement from a plurality of digital supplements. For example, a digital supplement selection panel may be displayed. The digital replenishment selection panel may include a number of user-operable controls each associated with one of the plurality of digital replenishments identified in the response. The digital replenishment selection may arrange user-operable controls based on the order or ranking of the digital replenishments provided by the server computing device. The digital refill selection panel can arrange user-operable controls vertically, horizontally or otherwise. The user operable control may be associated with or include information about the associated digital supplement that the user may consider when deciding whether to select a digital supplement. For example, the displayed information may include one or more of a name, description, image, and publisher name for the digital supplement.

In step 610, a user input for selecting a digital supplement is received. The user input can be a click using a mouse or other device. The user input may be a stylus or a finger touch input. Another example of user input is near touch input (eg, holding a finger or pointing device close to the screen). In some implementations, user input may also include hand gestures, head motion, action using eyes, or voice input.

In operation 612, the information is provided to the resource associated with the selected digital supplement. For example, information about a user of a client computing device may be transmitted to a server that provides a digital supplement (if the right to provide information is provided). The information can be provided to applications that provide digital supplements. Various types of information can be provided. For example, the information may include user information such as a user name, user preference or location.

The information may also include information related to a visual content query, such as an image or sequence of images. The information may also include an identifier and/or position of one or more supplement anchors in the image. This information can be used to provide digital supplements to users. For example, the AR content of a digital supplement can be sized and positioned based on the image.

The information may be transmitted directly by the client computing device 102 to a resource associated with the digital supplement (eg, digital supplement server 172 ). In some implementations, the information is provided by search server 152 to a resource associated with the digital supplement (eg, the client computing device does not need to transmit a lot of data). In at least some of these implementations, the client computing device 102 can send selection information to the search server 152 that identifies the selected digital supplement. Upon receiving the selection and confirming that the user has authorized the sharing of the information, the search server 152 may transmit the information to a resource that provides a digital supplement. Client computing device 102 may also prompt the user to allow information sharing. In some implementations, search server 152 may determine the information to send to the resource based on the digital supplement data structure instance (which may be based on metadata associated with the digital supplement).

In operation 614, the user interface is updated based on the selected digital supplement. Operation 614 may be similar to operation 506.

7A-7C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. A user interface screen 700a is shown in FIG. 7A. The user interface screen 700a includes an image display panel 708 and an information panel 730. In this example, image display panel 708 is displaying an image of a shelf full of wine bottles (eg, as found in a store). The image display panel 708 also includes an indicator 740 and an indicator 742. Each of these indicators indicates that the wine bottle shown in the image below the indicator has been recognized as a replenishment anchor (eg, recognized product in this case). Indicator 740 and indicator 742 are examples of user operable controls. The information panel 730 is provided with instructions to "Tap the item of interest."

In FIG. 7B, a user interface screen 700b is shown after the user operates the indicator 740. After operation, the annotation 744 from the digital replenishment is displayed. The annotation 744 contains information about the grade of the wine, which can help the user select the wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after the user operates the indicator 740. The user interface screen 700c may be displayed instead of or in addition to the user interface screen 700b (for example, after the operation of the annotation 744 or the user swipes upward on the information panel 730 of FIG. 7B ). If so). In Fig. 7C, an expanded information panel 732 is shown. The expanded information panel 732 occupies more of the user interface screen 700c than that of the information panel 730 in FIGS. 7A to 7B.

The extended information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734. The digital replenishment selection panel 710 includes a user operable control 712, a user operable control 714 and a user operable control 716 (shown in part only). In some implementations, additional user operable controls may be displayed when the user swipes the digital supplement selection panel 710. The user operable controls of the digital supplement selection panel 710 may be arranged in a ranked order. The user operable control 712 is associated with a digital supplement for meal pairing. Upon activation of the user operable control 712, a digital supplement may be displayed displaying food and meal pairing information for the selected wine. User operable control 714 is associated with a digital replenishment that stores photos. In operation, an application that stores a photo is activated and an image can be provided. Additional information can be stored along with a picture, such as an identified replenishment anchor.

The digital supplement content display panel 734 may display content from the digital supplement. The digital supplement content display panel 734 may display the default digital supplement or the highest ranked digital supplement associated with the identified supplement anchor. In this example, digital replenishment content display panel 734 includes product information regarding a product associated with the selected refill anchor. In this case, the wine name, grade, country of origin, image and comment are provided.

8A-8C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content retrieval and displaying digital supplements. The visual content search in this example is based on the receipt image.

A user interface screen 800a is shown in FIG. 8A. The user interface screen 800a includes an image display panel 808 and an information panel 830. In this example, image display panel 808 is displaying an image of the restaurant's receipt. The image display panel 808 also includes an indicator 840, an indicator 842, an annotation 844 and a highlight overlay 846. In this case, the indicator 840 is associated with the receipt as a document, and the indicator 842 is associated with the specific restaurant named in the receipt. Both the identified receipt document and the identified restaurant name are examples of replenishment anchors.

Annotation 844 is associated with a digital supplement that provides a tip calculator. In this example, an exemplary tip calculation is included in the annotation 844 and is overlaid at the appropriate position on the image display panel 808. In some implementations, digital replenishment can be selected by default and displayed when identifying an appropriate replenishment anchor. A highlight overlay 846 is overlaid over a portion of the receipt document containing information used by the tip calculator digital supplement.

In this example, the item displayed on the information panel 830 is a document associated with the receipt as if the indicator 840 had been activated. In some embodiments, the identified replenishment anchor is, for example, a user's past action, another user's action on a similar image, a confidence score for a replenishment anchor, or the position or size of a portion of the image associated with the replenishment anchor. Based on, it is ranked based on the user's relevance or interest. The information panel 830 may then, in at least some implementations, display the item associated with the highest ranked replenishment anchor. Instead, if indicator 842 has been activated, information panel 830 can include items for a particular restaurant.

Here, the information panel 830 includes a digital supplement selection panel 810. The digital replenishment panel includes a user operable control 812, a user operable control 814, and a user operable control 816. In this example, user-operable control 812 is associated with a tip calculator digital replenishment, user-operable control 814 is associated with a bill division digital replenishment, and user-operable control 816 is associated with cost. Associated with the report digital supplement. For example, upon actuation of the user operable control 812, a user interface control for adjusting the parameters of the tip calculator may be displayed (eg, to adjust the percentage).

In FIG. 8B, a user interface screen 800b is shown after the user activates the user-operable control 814. After operation, an expanded information panel 832 is shown containing items that help the user calculate the bill split. For example, you can enter the number of people dividing the bill to determine how much each person should pay.

In FIG. 8C, a user interface screen 800c is shown after the user activates the user-operable control 816. After operation, an expanded information panel 834 is displayed containing items that help the user to store receipts in an expense report. For example, a user can select an expense report that should be associated with a receipt (eg "Sydney trip 2018"). Once an expense report is selected, an image of the receipt can be uploaded to the expense report submission or management system. In some implementations, the entire image displayed on the image display panel 808 is uploaded. In some implementations, a portion of the image is uploaded (eg, the image is truncated to contain only the receipt).

9A-9C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. In this example, the visual content search is based on a face image.

A user interface screen 900a is shown in FIG. 9A. The user interface screen 900a includes an image display panel 908 and an information panel 930. In this example, the image display panel 908 is displaying an image of a face. Here, the face is an example of a replenishment anchor. The information panel 930 includes a user operable control 912 for the digital supplement that was identified for the supplement anchor in the image (ie, face). The user operable control 912 is associated with a digital supplement to tie the glasses.

In FIG. 9B, a user interface screen 900b is shown after the user activates the user operable control 912. After operation, an expanded information panel 932 containing items that help the user visually wear glasses on the face in the image is displayed. Here, several glasses styles are displayed, and the user can select a pair to try. When a pair of glasses is selected, AR content 960 is overlaid on the image display panel 908. Here, the AR content 960 corresponds to the selected glasses and is resized to match the face in the image. In some implementations, when a digital supplement for wearing glasses is selected, the image displayed on the image display panel 908 is sent to the server providing the digital supplement to analyze the image where and where the AR content 960 position and size are determined. You can decide what to do or recommend glasses to wear.

10A-10C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing visual content searches and displaying digital supplements. In this example, the visual content search is based on the furniture image in the catalog.

In FIG. 10A, a user interface screen 1000a is shown. The user interface screen 1000a includes an image display panel 1008. In this example, image display panel 1008 is displaying images of a portion of a page of a furniture catalog. The image display panel also includes an indicator 1040, an indicator 1042 and an indicator 1044. In this example, indicator 1040 is associated with a bed, indicator 1042 is associated with a decorative item, and indicator 1044 is associated with a rug. Beds, decorative items, and rugs in the catalog are examples of replenishment anchors.

In FIG. 10B, the user interface screen 1000b is shown after the user selects the indicator 1040 (eg, by touching the screen at or near where the indicator 1040 is displayed). The user interface screen 1000b includes a digital supplement selection panel 1010 and an information panel 1030. The information panel 1030 includes information (eg, product name, description, image) on a supplement anchor associated with the selected indicator.

The digital replenishment selection panel 1010 includes a user operable control 1012 and a user operable control 1014. User operable control 1012 is associated with a digital supplement that provides a view into the home. User-operable controls 1014 are associated with other digital supplements (eg, digital supplements for posting on social media sites).

In FIG. 10C, a user interface screen 1000c is shown after operation of the user operable control 1012. The user interface screen 1000c includes an image display panel 1008, a digital supplement selection panel 1010, and a reduced information panel 1032. The reduced information panel 1032 may include user operable controls that may cause the information panel to pop up and display when activated.

Here, the image display panel 1008 now displays an image of the room and includes AR content 1060. AR content 1060 includes a 3D model of a bed associated with an indicator 1040 overlaid on an image panel. The user can adjust the position of the AR content 1060 within the room to see how the bed enters the room. In some implementations, when a digital supplement for in-home view is selected, the image displayed on the image display panel 1008 is sent to a server providing the digital supplement to analyze the image to determine the position and position of the AR content 1060. Decide where and how to size. In some implementations, AR content 1060 may be provided later than visual content queries.

11A-11C are schematic diagrams of user interface screens displayed by an embodiment of a client computing device 102 for performing a visual content search within a repository. In this example, the visual content search is based on product images captured within the store.

In Fig. 11A, a user interface screen 1100a is shown. The user interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying images captured within a store. The image display panel 1108 also includes an indicator 1140 associated with the vase. The vase displayed on the image display panel 1108 is an example of a refill anchor. The information panel 1130 displays product information on a vase and a digital supplement including a vase purchase function. For example, a digital refill may include a workflow to initiate a vase purchase. In this example, the digital replenishment is the image content and client so that if the client computing device is in the store, the digital replenishment published (or associated with the store) from the store where the image was captured is identified and presented as a high ranking result in the visual content query. It is identified based on the location of the computing device. In some implementations, if the location of the client computing device has changed, a different digital supplement is provided for the same image.

In FIG. 11B, a user interface screen 1100b is shown. The user interface screen 1100b includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying another image captured within the store. The image display panel 1108 also includes an indicator 1142 associated with the lug. The lugs displayed on the image display panel 1108 are an example of a replenishment anchor. The information panel 1130 displays a digital supplement that includes product information for the rug and a function of selecting a size and purchasing a rug. As in Fig. 11A, the digital supplement is identified based on the image content and the location of the client computing device.

In FIG. 11C, a user interface screen 1100c is shown. The user interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying another image captured within the store. The image display panel 1108 also includes an indicator 1144 associated with the vase. The vase displayed on the image display panel 1108 is an example of a refill anchor. The information panel 1130 is displaying a digital supplement including product information for the vase. The information panel 1130 also includes a coupon indicator 1132 and the ability to use the coupon. Using a coupon may include purchasing an item at a discounted price on a website associated with the store. In some implementations, coupon codes are provided that can be used to secure discounts during checkout. As in FIGS. 11A and 11B, the digital supplement is identified based on the image content and the location of the client computing device.

12A-12C are schematic diagrams of user interface screens displayed by an embodiment of the client computing device 102 during visual content retrieval. In this example, the visual content search is based on an image of a movie poster (for example, it may be captured in a movie theater).

In FIG. 12A, a user interface screen 1200a is shown. The user interface screen 1200a includes an image display panel 1208. In this example, image display panel 1208 is displaying an image of a movie poster. The image display panel 1208 also includes an indicator 1240 associated with the movie poster identified in the image. Movie posters are examples of replenishment anchors. Indicator 1240 may include a user operable control to display a digital supplement or a menu for selecting a digital supplement when activated.

In FIG. 12B, a user interface screen 1200b is shown. The image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified in the image. For example, preview digital supplement 1242 may be displayed after activation of indicator 1240 (of FIG. 12A) indicator 1240. Preview digital supplement 1242 may overlay an image or video from the movie associated with the identified movie poster onto the image of the movie poster.

In FIG. 12C, a user interface screen 1200c is shown. The image display panel 1208 also includes a rating indicator 1244 and a rating indicator 1246. Rating indicator 1244 and rating indicator 1246 may be generated by one or more digital supplements in response to a visual content query comprising a movie poster. The digital supplement may, for example, overlay rating information for a movie associated with a movie poster in the image. Rating indicator 1244 and rating indicator 1246 may include user-operable controls that, when activated, allow additional information about the rating and associated movie to be displayed.

13 illustrates a computer device 1300 that may be used in conjunction with the techniques described herein (e.g., to implement a client computing device 102, a search server 152 and a digital supplement server 172) and An example of a mobile computer device 1350 is shown. The computing device 1300 includes a processor 1302, a memory 1304, a storage device 1306, a high speed interface 1308 and a low speed bus 1314 connected to the memory 1304 and a high speed expansion port 1310, and a storage device. It includes a low speed interface 1312 connected to 1306. Each of the components 1302, 1304, 1306, 1308, 1310 and 1312 may be interconnected using various buses and mounted on a common motherboard or in any other suitable manner. Processor 1302 processes instructions for execution within computing device 1300, including instructions stored in memory 1304 or storage device 1306, such as display 1316 connected to high speed interface 1308. Graphical information for a GUI can be displayed on an external input/output device. In other implementations, multiple processors and/or multiple buses may be used as appropriate, with multiple memories and multiple types of memory. Further, multiple computing devices 1300 may be connected with each device to provide portions of the required operation (eg, a server bank, a group of blade servers, or a multi-processor system).

Memory 1304 stores information within computing device 1300. In one implementation, memory 1304 is a volatile memory unit or units. In another implementation, memory 1304 is a non-volatile memory unit or units. Further, the memory 1304 may be another form of computer-readable medium such as a magnetic or optical disk.

Storage device 1306 can provide large storage for computing device 1300. In one implementation, the storage device 1306 is a device including a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid state memory device, or a storage area network or other configuration device. It may be a computer-readable medium, such as an array, or may include a computer-readable medium. The computer program product can be tangibly carried on an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer or machine readable medium such as memory 1304, storage device 1306, or memory on processor 1302.

The high speed controller 1308 manages the bandwidth intensive operations for the computing device 1300, while the low speed controller 1312 manages the low bandwidth intensive operations. The assignment of these functions is merely exemplary. In one implementation, the high speed controller 1308 includes a high speed expansion port (not shown) capable of receiving a memory 1304, a display 1316 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). 1310). In an implementation, low speed controller 1312 is coupled to storage device 1306 and low speed expansion port 1314. Low-speed expansion ports, which may include a variety of communication ports (e.g. USB, Bluetooth, Ethernet, wireless Ethernet), can be used with one or more input/output devices such as keyboards, pointing devices, scanners, or switches or via network adapters, for example. It can be coupled to a networking device such as a router.

The computing device 1300 can be implemented in a number of different forms as shown in the figure. For example, it may be implemented as a standard server 1320 or in multiples in a group of such servers. It may also be implemented as part of the rack server system 1324. It may also be implemented in a personal computer such as laptop computer 1322. Alternatively, components from computing device 1300 may be combined with other components within a mobile device (not shown) such as device 1350. Each of the devices may include one or more of the computing devices 1300 and 1350, and the entire system may include a plurality of computing devices 1300 and 1350 communicating with each other.

Computing device 1350 includes a processor 1352, memory 1364, input/output devices such as display 1354, communication interface 1366, and transceiver 1368, among other various components. Device 1350 may also be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of the components 1350, 1352, 1364, 1354, 1366 and 1368 are interconnected using various buses, and some components may be mounted on a common motherboard or in any other suitable manner.

The processor 1352 may execute instructions in the computing device 1350 including instructions stored in the memory 1364. The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. The processor may provide coordination of other components of the device 1350 such as, for example, user interfaces, applications executed by the device 1350 and wireless communication by the device 1350.

The processor 1352 may communicate with a user through a control interface 1358 and a display interface 1356 connected to the display 1354. The display 1354 may include, for example, a TFT LCD (thin film transistor liquid crystal display) and an LCD (light emitting diode) or OLED (organic light emitting diode) display or other suitable display technology. Display interface 1356 may include suitable circuitry for driving display 1354 to provide graphics and other information to a user. Control interface 1358 can receive commands from a user and transform them for submission to processor 1352. Additionally, an external interface 1362 may be provided for communication with the processor 1352 to enable short-range communication of the device 1350 with other devices. External interface 1362 may be provided, for example, for wired communication in some implementations or for wireless communication in other implementations, and multiple interfaces may be used.

Memory 1364 stores information within computing device 1350. The memory 1364 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, and a nonvolatile memory unit or units. Further, the expansion memory 1374 may be provided and connected to the device 1350 through an expansion interface 1372, which may include, for example, a single in line memory module (SIM) card interface. The extended memory 1374 may provide additional storage space for the device 1350 or may store applications or other information about the device 1350. In particular, the extended memory 1374 may include instructions for performing or supplementing the above-described processes, and may also include security information. Thus, for example, extended memory 1374 may be provided as a secure module for device 1350 and may be programmed with instructions that allow secure use of device 1350. In addition, security applications can be provided with additional information via SIMM cards, such as placing identification information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one implementation, the computer program product is tangibly carried on an information carrier. The computer program product also includes instructions that, when executed, perform one or more methods as described above. The information carrier is, for example, a computer or machine readable medium such as memory 1362, extended memory 1374, or memory on processor 1352, which may be received via transceiver 1368 or external interface 1362.

Device 1350 may communicate wirelessly through communication interface 1366, which may include digital signal processing circuitry as needed. The communication interface 1366 may be provided for communication under various modes or protocols such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may take place, for example, through a radio frequency transceiver 1368. In addition, short-range communications such as using Bluetooth, Wi-Fi or other transceivers (not shown) may occur. Additionally, the Global Positioning System (GPS) receiver module 1370 may provide additional navigation and location-related wireless data to the device 1350 that can be appropriately used by applications running on the device 1350.

The device 1350 can also communicate audibly using an audio codec 1360 that can receive spoken information from a user and convert it into usable digital information. The audio codec 1360 may likewise generate an audible sound for the user, such as, for example, through a speaker in the handset of the device 1350. These sounds may include sounds from voice telephone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1350. It may include.

The computing device 1350 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular phone 1380. It may also be implemented as part of a smartphone 1382, personal digital assistant (PDA), or other similar mobile device.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. . These various implementations may include implementations in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general purpose, and storage system , Receive data and instructions from at least one input device and at least one output device and may be connected to transmit data and instructions thereto.

These computer programs (also known as programs, software, software applications or code) contain machine instructions for a programmable processor, and can be implemented in high-level procedural and/or object-oriented programming languages and/or assembly/machine language. . As used herein, the terms "machine-readable medium" and "computer-readable medium" include machine-readable media that receive machine instructions as machine-readable signals, including programmable machine instructions and/or data. Refers to any computer program product, apparatus, and/or device used to provide a processor, such as magnetic disk, optical disk, memory, programmable logic device (PLD). The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

In order to provide interaction with a user, the systems and techniques described herein include a display device (a light-emitting diode) or OLED (organic LED) or liquid crystal display (LCD) for displaying information to a user. ) Monitor/screen) and a keyboard and pointing device through which the user can provide input to the computer, for example a mouse or a trackball. Other types of devices can also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be received in any form including acoustic, voice, or tactile input. I can.

The systems and techniques described herein include, for example, a backend component such as a data server, a middleware component such as an application server, or a user computer having a graphical user interface or an implementation of the systems and techniques described herein. It may be implemented in a computing system that includes a frontend component such as an interactive web browser, or any combination of one or more of the backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, for example a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include users and servers. Users and servers are generally far apart from each other and usually interact through a communication network. The relationship between the user and the server arises by computer programs running on each computer and having a user-server relationship with each other.

In some implementations, the computing device shown in FIG. 13 may include sensors that interface with the AR headset/HMD device 1390 to create an augmented environment for viewing embedded content within a physical space. For example, one or more sensors included in the computing device 1350 or other computing device shown in FIG. 13 may provide input to the AR headset 1390 or, in general, to the AR space. The sensor may include, but is not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. Computing device 1350 may use sensors to determine the absolute position and/or detected rotation of the computing device within AR space that can be used as input into AR space. For example, the computing device 1350 may be integrated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, and the like. When integrated into the AR space, the location of the computing device/virtual object by the user allows the user to locate the computing device and view the virtual object in a specific manner in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were a real laser pointer. Users can move the computing device left and right, up, down, in a circle, etc., and use the device in a manner similar to using a laser pointer. In some implementations, a user can aim at a target location using a virtual laser pointer.

In some implementations, one or more input devices included or connected to the computing device 1350 may be used as input to the AR space. Input devices include touch screens, keyboards, one or more buttons, track pads, touch pads, pointing devices, mice, trackballs, joysticks, cameras, microphones, earphones or buds with input capabilities, game controllers or other connectable input devices. Can, and is not limited anymore. When the computing device is integrated into the AR space, a user interacting with the input device included in the computing device 1350 may cause a specific action to occur in the AR space.

In some implementations, the touch screen of computing device 1350 may be rendered as a touch pad in AR space. The user may interact with the touch screen of the computing device 1350. The interactions are rendered according to the motion on the touch pad rendered in AR space in the AR headset 1390, for example. The rendered motion can control virtual objects in AR space.

In some implementations, one or more output devices included in computing device 1350 may provide output and/or feedback to a user of AR headset 1390 in AR space. Output and feedback can be visual, tactile or audible. Output and/or feedback may include, but are not limited to, vibration, turning on/off or blinking one or more lights or strobes and/or flashing, ringing an alarm, hitting a chime, playing a song, and playing an audio file. Output devices can include, but are not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe and a speaker.

In some implementations, computing device 1350 may be displayed as another object in a computer generated 3D environment. Interactions with the computing device 1350 by the user (eg, rotation, shaking, touch screen touch, and finger swiping across the touch screen) may be interpreted as interactions with objects in AR space. In the example of a laser pointer in AR space, computing device 1350 is viewed as a virtual laser pointer in a computer generated 3D environment. As the user manipulates the computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from an AR environment on computing device 1350 or interactions with computing device 1350 on AR headset 1390. The interaction between the computing device and the user may be converted into an interaction with a user interface generated in an AR environment for a controllable device.

In some implementations, computing device 1350 may include a touch screen. For example, a user may interact with a touchscreen to interact with a user interface for a controllable device. For example, the touchscreen may include a user interface element such as a slider to control properties of the controllable device.

Computing device 1300 is intended to represent various types of digital computers including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1350 is intended to represent various types of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functionality are meant to be exemplary only and are not meant to limit implementation of the invention described and/or claimed herein.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.

Additionally, the logic flows shown in the figures do not necessarily require a particular depicted order, or a time series order, to achieve the desired results. Additionally, other steps may be provided, or steps may be omitted from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are also within the scope of the following claims.

While certain configurations of the described embodiments have been described as described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of implementation. It is to be understood that these embodiments are illustrative and not restrictive, and various changes in form and detail may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except combinations that are mutually exclusive. Implementations described herein may include various combinations and/or sub-combinations of functions, components, and/or configurations of the other implementations described.

Claims (51)

As a computer-implemented method,
Receiving data specifying a digital supplement, the data identifying a digital supplement and a supplement anchor for associating the digital supplement with visual content;
Creating a data structure instance specifying the digital replenishment and the replenishment anchor; And
After creating the data structure instance, enabling triggering of the digital supplement by an image based at least on storing the data structure instance in a database containing a plurality of other data structure instances, the plurality of Each of the different data structure instances of the computer-implemented method specifies a digital supplement and one or more supplement anchors. The method of claim 1, wherein receiving data specifying the digital supplement comprises:
Analyzing the web page; And
And identifying metadata associated with the digital supplement represented within the webpage. The method of claim 2, wherein receiving data specifying the digital supplement comprises:
Before analyzing the web page, further comprising the step of crawling a plurality of web pages including the web page, computer-implemented method. The method of claim 2 or 3, wherein receiving data specifying the digital supplement comprises:
Prior to analyzing the webpage, receiving an identifier of the webpage. 5. The computer-implemented method of any of claims 2-4, further comprising associating the data structure instance with a popularity score for the webpage. 6. The computer-implemented method of any of claims 2-5, further comprising associating the data structure instance with a relevance score for the replenishment anchor based on the webpage. The computer-implemented method of any of the preceding claims, wherein the digital supplement comprises a name, description, image and uniform resource locator (URL). The computer-implemented method of any of the preceding claims, wherein the digital supplement comprises an identifier of an application. The method according to any of the preceding claims, after the step of enabling triggering of the triggering of the digital supplement:
Receiving a visual content query including an image;
Determining whether the image matches the replenishment anchor specified by the created data structure instance; And
In response to determining that the image matches the supplement anchor, providing the digital supplement from the data structure instance in response to the visual content query. 11. The computer-implemented method of claim 9, wherein determining that the image matches the supplement anchor comprises identifying an entity within the image. 11. The method of claim 9 or 10, wherein the data structure instance specifies context information, and the step of providing the digital supplement from the data structure instance provides the data from the data structure instance based on a determination that the context has been satisfied. A computer-implemented method comprising the step of: The method of any preceding claim, wherein providing the digital supplement from the data structure instance in response to the visual content query comprises providing a list of digital supplements, the list comprising the data structure A digital supplement from and a digital supplement from one of the other data structure instances, and are ordered based on one or more of a popularity score and a relevance score. As a computing device,
At least one processor; And
A memory for storing instructions, wherein the instructions, when executed by the at least one processor, cause the computing device to:
Receive data specifying the digital supplement, the data identifying the digital supplement and supplement anchor and context information for associating the digital supplement with visual content;
Create a data structure instance specifying the digital replenishment, the replenishment anchor, and the context information; And
After creating the data structure instance, enabling the triggering of the digital supplement by an image based at least on storing the data structure instance in a database containing a plurality of different data structure instances, and the plurality of different data structures Each of the instances specifies a digital replenishment and one or more replenishment anchors. The computing device of claim 13, wherein the context information includes a location. 15. The computing device of claim 13 or 14, wherein the context information includes an identified entity within the image. 16. The computing device of any of claims 13-15, wherein the context information includes a number of entities identified within the image. As a computer-implemented method,
Receiving a visual content query from a computing device;
Identifying a replenishment anchor based on the visual content query;
Generating an ordered list of digital supplements based on the identified supplement anchors; And
Transmitting the ordered list to a computing device. 18. The computer-implemented method of claim 17, wherein the visual content query includes an image and the digital supplement includes a URL to a webpage presenting information about the entity identified in the image. 19. The computer-implemented method of claim 17 or 18, wherein the digital supplement comprises a URL to a webpage presenting a coupon code for the entity identified in the image. 20. The computer-implemented method of any of claims 17-19, wherein the digital supplement comprises video content identified based on the image. 21. The computer-implemented method of any of claims 17-20, wherein the digital supplement comprises a request to a server and parameters for that request. 22. The computer-implemented method of claim 21, wherein the parameter initiates a workflow. 23. The computer-implemented method of any of claims 17-22, wherein the digital supplement of the list is ordered based on one or more of a popularity score and a relevance score. As a computing device,
At least one processor; And
A computing device comprising a memory storing instructions that, when executed by the at least one processor, cause the computing device to perform the method of any one of claims 1-12 or 17-23. A computer program product comprising an instruction, wherein the instruction, when executed by one or more processors, causes the one or more processors to perform the method of any one of claims 1 to 12 or 17 to 23. A computer-readable storage medium containing instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1-12 or 17-23. media. As a computer-implemented method,
Sending a visual content query to the server computing device;
Receiving a response to the visual content query identifying a digital supplement; And
And causing a user interface to be displayed that includes information associated with the digital supplement. 28. The computer-implemented method of claim 27, wherein sending the visual content query comprises sending an image to the server computing device. 29. The computer-implemented method of claim 27 or 28, wherein transmitting the visual content query comprises transmitting an identifier of an entity in an image to the server computing device. 30. The computer-implemented method of any of claims 27-29, further comprising transmitting context information to the server computing device. The method of claim 30,
Wherein the context information includes a location. 32. The computer-implemented method of any of claims 27-31, wherein receiving a response to the visual content query comprises receiving a URL associated with the digital supplement. 33. The computer-implemented method of any of claims 27-32, wherein receiving a response to the visual content query comprises receiving an identifier of an application associated with the digital supplement. 34. The computer-implemented method of any of claims 27-33, wherein receiving a response to the visual content query comprises receiving a name, description, and image associated with the digital supplement. 35. The computer-implemented method of any of claims 27-34, wherein receiving a response to the visual content query comprises receiving a list of digital supplements, including the digital supplement. 36. The computer-implemented method of claim 35, wherein the user interface includes information regarding a plurality of digital supplements from the list of digital supplements. The method of claim 36,
Further comprising determining an alignment for the list of digital supplements based on whether an application associated with the digital supplement has been installed. The method according to any one of claims 27 to 37,
Further comprising the step of capturing the image,
The visual content query is based on the image. In the computing device,
At least one processor; And
A memory for storing instructions, wherein the instructions, when executed by the at least one processor, cause the computing device to:
Let the image be captured;
Send a visual content query based on the image to a server computing device;
Receive a response to the visual content query identifying a digital supplement; And
And cause a user interface to be displayed including information associated with the digital supplement. The method of claim 39,
The computing device, wherein receiving a response to the visual content query includes receiving an ordered list of digital supplements that include the digital supplement. The method of claim 40,
Wherein the user interface includes a digital replenishment selection panel including user operable controls associated with a plurality of digital replenishments in the sorted list. The method of claim 41,
The user operable control is ordered on the digital replenishment selection panel based on an order provided by the sorted list. The method according to any one of claims 39 to 42,
The digital replenishment is associated with a digital replenishment server, and the instructions cause the computing device to:
Send information to the digital replenishment server; And
Receive digital replenishment content from the digital replenishment server, the digital replenishment content based on the transmitted information. 44. The computing device of claim 43, wherein the transmitted information includes context information. As a computer-implemented method,
Capturing an image;
Transmitting a visual content query to a search server based on the image;
Receiving a response to the visual content query identifying a digital supplement server;
Causing the image to be transmitted to the digital supplement server;
Receiving digital supplement content from the digital supplement server; And
Causing the digital supplement content to be displayed. The method of claim 45,
Wherein the visual content query includes the image, and causing the image to be transmitted to the digital replenishment server comprises sending instructions to the search server to provide the image to the digital replenishment server. How to be implemented as. The method of any one of claims 45 to 46,
Wherein causing the image to be transmitted to the digital replenishment server comprises transmitting the image to the digital replenishment server. The method of any one of claims 45 to 47,
The digital supplement comprises augmented reality content sized and positioned based on the image. As a computing device,
At least one processor; And
A computing device comprising a memory for storing instructions, wherein the instructions, when executed by the at least one processor, cause the computing device to perform the method of any one of claims 27-38 or 45-48. A computer program product comprising an instruction, wherein the instruction, when executed by one or more processors, causes the one or more processors to perform the method of any one of claims 27 to 38 or 45 to 48. A computer-readable storage medium comprising instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 27 to 38 or 45 to 48. media.

Images