JP2021522614A - Digital supplemental association and search for visual search - Google Patents

Abstract

Systems and methods for identifying and searching content for visual search are provided. An exemplary method involves receiving data that specifies a digital supplement. The data can distinguish between digital supplements and supplemental anchors for associating the digital supplements with image content. The method also involves creating a data structure instance that specifies digital supplements and supplementary anchors, and after creating the data structure instance, storing the data structure instance in a database that contains multiple other data structure instances. It may include, at least based on, enabling the triggering of digital supplementation by images. Each other data structure instance can specify a digital supplement and one or more supplement anchors.

Description

Reference to Related Application This application is a continuation of US Non-Provisional Patent Application No. 16/014,520 entitled "DIGITAL SUPPLEMENT ASSOCIATION AND RETRIEVAL FOR VISUAL SEARCH" filed on June 21, 2018, and its priority It is a claim and its disclosure is incorporated herein by reference in its entirety.

Background Mobile computing devices such as smartphones often include cameras. These cameras can be used to capture images of entities in the environment surrounding the computing device. Various types of content or experience associated with these entities may be available to the user through the mobile computing device.

Summary This disclosure describes a system and method for digital supplemental association and search for visual search. For example, the systems and techniques described herein can be used to provide digital supplements such as augmented reality (AR) content or experience in response to visual searches. The visual search may be based on, for example, an image or an entity identified within the image. Digital supplements may include, for example, providing information or functionality associated with an image.

One aspect is a method performed on a computer that involves receiving data that specifies a digital supplement, which identifies the digital supplement from a supplement anchor for associating the digital supplement with image content. The method also involves creating data structure instances that specify digital supplements and supplement anchors. The method further includes enabling the triggering of digital supplementation with images, based on creating the data structure instance and then storing at least the data structure instance in a database containing a plurality of other data structure instances. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Another aspect is a computing device that includes at least one processor and memory that stores instructions. When executed by the at least one processor, the instruction causes the computing device to receive data specifying a digital supplement, which is a digital supplement and a supplement for associating the digital supplement with image content. Receive the anchor and the context information. The instruction also causes the computing device to generate a data structure instance that specifies digital supplements, supplemental anchors, and contextual information. The instruction also triggers a digital supplement with an image, at least based on creating a data structure instance on the computing device and then storing the data structure instance in a database containing multiple other data structure instances. Activate. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Yet another aspect is a method performed on a computer that includes receiving an image content query from a computing device and identifying a supplemental anchor based on the image content query. The method also includes generating an ordered list of digital supplements based on the identified supplement anchors and sending the ordered list to client computing devices.

Details of one or more embodiments will be described in the accompanying drawings and the following description. Other features will become apparent from the description and drawings as well as the claims.

FIG. 6 is a block diagram showing a system according to an exemplary embodiment. The embodiment of the client computing device of FIG. 1 is a third person perspective of an exemplary physical space accessing a digital supplement. FIG. 5 is a diagram of an exemplary method of enabling a digital supplement trigger according to an embodiment described herein. FIG. 5 is a diagram of an exemplary method of enabling a digital supplement trigger according to an embodiment described herein. FIG. 5 is a diagram of an exemplary method of retrieving and presenting digital supplements according to embodiments described herein. FIG. 5 is a diagram of an exemplary method of identifying and presenting digital supplements based on images, according to the embodiments described herein. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing an image content search and displaying a digital supplement. It is the schematic of the user interface screen displayed by the embodiment of the client computing device of FIG. 1 in order to perform various image content search in a store. It is the schematic of the user interface screen displayed by the embodiment of the client computing device of FIG. 1 in order to perform various image content search in a store. It is the schematic of the user interface screen displayed by the embodiment of the client computing device of FIG. 1 in order to perform various image content search in a store. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 during various image content searches. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 during various image content searches. FIG. 5 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 during various image content searches. It is a schematic diagram of an example of a computer device and a mobile computer device that can be used to carry out the techniques described herein.

Detailed Description This disclosure describes technical improvements that simplify the identification and presentation of digital supplements based on image content. Certain embodiments of the techniques described herein generate indexes for digital supplements associated with a particular type of image content and digitally respond to image content queries received from client computing devices. Provide supplements. This index gives users access to relevant digital supplements provided by network-accessible resources (eg, web pages) located around the world. This may provide a functional data structure that allows for more efficient retrieval of information.

For example, a client computing device such as a smartphone may acquire an image of a supplementary anchor, such as an entity. The client computing device may then send an image-based image content query to the server computing device to search for the digital supplement associated with the identified supplement anchor. In certain embodiments, the supplemental anchor is based on the physical environment surrounding the client computing device, and the digital supplement is virtual content that can supplement the user's experience in the physical environment.

The image content query may include an image or data determined from the image (eg, such as an indicator of an identified supplemental anchor). An example of data determined from an image is, for example, text extracted from an image using optical character recognition. Other examples of data extracted from an image include a barcode, QR® code, etc. in the image, the identifier or description of the entity, the product, or the type of entity identified in the image.

An entity, product, or entity type can be identified in an image using a neural network system, such as a convolutional neural network system. An identifier or description of an entity, product, or entity type can include metadata or references to records in the database associated with the entity, product, or entity type. Non-limiting examples of entities include buildings, works of technology, products, books, posters, photographs, catalogs, signs, documents (eg, business cards, receipts, coupons, catalogs), people, and body parts.

Supplements Various types of digital supplements associated with anchors may be available. Digital supplements may be provided by network-accessible resources, such as web pages available on the Internet. There is a need for a way to locate and provide these digital supplements in response to image content queries. One embodiment generates and maintains an index of digital supplements associated with the entity used in responding to image content queries. The index, for example, by crawling network-accessible resources to determine whether the network-accessible resource contains or provides digital supplements and determines the supplemental anchors associated with those digital supplements. , Can be added.

For example, a network-accessible resource may contain metadata that identifies a supplemental anchor (eg, text, code, entity, or type of entity) with which a digital supplement is associated. Metadata can be included by network-accessible resources in response to Hypertext Transfer Protocol (HTTP) requests. Metadata can be provided in a variety of formats, such as XML (extensible markup language), JavaScript Object Notation (JSON), or other formats.

Metadata for digital supplements can be one or more of the following: type indicators, anchor indicators, names, descriptions, content snippets (ie, excerpts or previews of parts of content), related images, digital supplements. It may include a link such as a URL to, and an application identifier associated with the digital supplement. Metadata may also include information about the publisher of digital supplements. For example, the metadata may include an issuer name, an issuer description, and one or more of the images or icons associated with that issuer. In certain embodiments, the metadata includes contextual information associated with providing digital supplements. For example, metadata may also include conditions associated with providing digital supplements or accessing digital captures (eg, geographic conditions, required applications).

The identified digital supplement may be added to an index stored in memory. At least in certain embodiments, the associated supplement anchor for digital supplement is used as a key to the index. Digital supplements can also be associated with various scores. For example, a digital supplement is how many other links (eg, while crawling a network accessible resource) are found that refer to the digital supplement or the network accessible resource associated with the digital supplement. It can be associated with a fame score based on the fame of the network accessible resources that provide those links. As another example, a digital supplement can be associated with one or more relevance scores that correspond to the relevance of the digital supplement (or associated network accessible resource) to a particular anchor. Relevance scores can also be associated with keywords or themes. The relevance score is one of the content of the digital supplement, the content of the network-accessible resource, the content of the site that links to the network-accessible resource, and the content of the link to the network-accessible resource (for example, text). Or it can be determined on the basis of more than one.

FIG. 1 is a block diagram showing a system 100 according to an exemplary embodiment. System 100 can associate a digital supplement with an entity or entity type and retrieve the digital supplement in response to a visual search. The visual search is a search based on image content. For example, a visual search can be performed based on an image content query. An image content query is a query based on an image or other image content. For example, an image content query may include an image. In certain embodiments, the image content query may include image-based text or data. For example, text or data can be generated by recognizing one or more entities in an image. Some image content queries do not include images (for example, image content queries may only contain data or text generated from images). In certain embodiments, the system 100 includes a client computing device 102, a search server 152, and a digital supplement server 172. Also shown is the network 190 through which the client computing device 102, the search server 152, and the digital supplementary server 172 can communicate.

The client computing device 102 may include a processor assembly 104, a communication module 106, a sensor system 110, and a memory 120. The sensor system 110 may include various sensors such as a camera assembly 112, an inertial motion unit (IMU) 114, and a Global Positioning System (GPS) receiver 116. Embodiments of the sensor system 110 also include, for example, contact sensors such as optical sensors, voice sensors, image sensors, distance and / or proximity sensors, capacitance sensors, timers, and / or other sensors, and / or sensors. It may include other sensors containing different combinations. In certain embodiments, the client computing device 102 is a mobile device (eg, a smartphone).

The camera assembly 112 acquires an image or video of the physical space around the client computing device 102. The camera assembly 112 may include one or more cameras. The camera assembly 112 may also include an infrared camera. The image acquired in the camera assembly 112 can be used to supplement the anchor and to identify it to form an image content query.

In certain embodiments, the image acquired by the camera assembly 112 is also a client computing device within a physical space, such as an internal space, based on a representation of the physical space received from the memory 120 or an external computing device. It can be used to determine the position and orientation of 102. In certain embodiments, the representation of physical space may include visual features of the physical space (eg, features extracted from an image of real space). The representation is also associated with those features and can be used by a visual positioning system to determine a position and / or location in physical space based on one or more images of physical space. May include decision data. The representation may also include a three-dimensional model of at least some structure in physical space. In certain embodiments, the representation does not include a three-dimensional model of physical space.

The IMU 114 may detect the movement, movement, and / or acceleration of the client computing device. The IMU 114 may include various different types of sensors such as accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of the client computing device 102 can be detected and tracked based on the data provided by the IMU 114 or GPS receiver 116.

The GPS receiver 116 may receive signals transmitted by GPS satellites. The signal includes the time and position of the satellite. Based on receiving signals from several (eg, at least 4) satellites, the GPS receiver 116 may determine the global position of the client computing device 102.

The memory 120 may include an application 122, another application 140, and a device positioning system 142. The other application 140 includes any other application that is installed or otherwise available to run on the client computing device 102. In one embodiment, application 122 may allow one of the other applications 140 to provide digital supplements. In certain embodiments, some digital supplements may only be available if the other application 140 includes a particular application that is associated with or is required to provide the digital supplement. ..

The device positioning system 142 determines the position of the client computing device 102. The device positioning system 142 may use the sensor system 110 to determine the position and orientation of the client computing device 102 globally or in physical space. In one embodiment, the device positioning system 142 locates the client computing device 102, for example, based on cellular triangulation.

In certain embodiments, the client computing device 102 compares the images acquired by the camera assembly 112 (or features extracted from those images) with a known arrangement of features within a representation of physical space and is physically physical. It may include a visual positioning system that determines the orientation (eg, position and orientation) of the client computing device 102 in a physical space with six degrees of freedom.

Application 122 may include a supplementary anchor identification engine 124, a digital supplementary search engine 126, a digital supplementary presentation engine 128, and a user interface engine 130. Certain embodiments of application 122 may include fewer, additional, or other components.

The supplementary anchor identification engine 124 identifies the supplementary anchor based on, for example, the image acquired by the camera assembly 112. In certain embodiments, the supplemental anchor identification engine 124 analyzes the image to identify the text. The text can then be used to identify the anchor. For example, the text can be mapped to a node in the knowledge graph. For example, text can be recognized as the name of an entity, such as a person, place, product, building, artwork, movie, or other type of entity. In certain embodiments, the text may be recognized as a phrase commonly associated with a particular entity or as a phrase describing a particular entity. For example, the text can then be recognized as an anchor associated with that particular entity.

In certain embodiments, the supplemental anchor identification engine 124 identifies one or more codes in the image, such as a barcode, QR code®, or another type of code. The code can then be mapped to a supplemental anchor.

Supplement The anchor identification engine 124 may include a machine learning module capable of recognizing at least some type of entity in the image. For example, a machine learning module may include a neural network system. A neural network is a computational model used for machine learning and consists of nodes organized in layers with weighted connections. Training a neural network uses multiple training examples, each of which is an input and a desired output, allowing the neural network to provide the desired output for a given input over a series of iterative rounds. Determine the weight value of the connection between layers that increases the sex. During each training round, the weights are adjusted to deal with incorrect output values. Once trained, neural networks can be used to predict outputs based on the inputs provided.

In certain embodiments, the neural network system comprises a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolution layer is a layer in which the values of a layer are calculated based on applying kernel functions to a subset of the values of the previous layer. Training a neural network can include adjusting the weights of kernel functions based on training examples. Typically, the same kernel function can be used to calculate each value in the convolution layer. Therefore, the weights that must be learned while training the convolutional layer are such that each value in a fully connected layer in the neural network (eg, each value in one layer is independently adjusted for each value in the previous layer). Much less than layers calculated as weighted combinations). Because the convolution layer weighs less than usual, training and using the convolution layer can require less memory, processor cycles, and time than an equivalent fully connected layer.

After the anchor identification engine 124 recognizes an entity or entity type in the image, a text description of the entity or entity type can be generated. In addition, entities or entity types can be mapped to supplemental anchors. In certain embodiments, the supplemental anchor is associated with one or more digital supplements.

In certain embodiments, the supplemental anchor identification engine 124 determines the confidence score of the recognized anchor. A higher confidence score associates content from the image (eg, image, extracted text, barcode, QR code®) with the determined anchor than if a lower confidence score was determined. It can be shown that it is likely to be.

The example of FIG. 1 shows the supplementary anchor identification engine 124 as a component of application 122 on the client computing device 102, but one embodiment includes a supplementary anchor identification engine on the search server 152. For example, the client computing device 102 can send the image acquired by the camera assembly 112 to the search server 152, which can identify the supplemental anchors in the image.

In certain embodiments, the supplemental anchor identification engine 124 identifies potential supplementary anchors. For example, the supplemental anchor identification engine 124 may identify (recognize) various entities in the image. The identifier of the recognized entity can then be sent to the search server 152, which may determine if any of the entities are associated with any supplemental anchor. In certain embodiments, the search server 152 may use the identified entity as contextual information even if the identified entity is not a supplemental anchor.

The digital supplement search engine 126 searches for digital supplements. For example, the digital supplement search engine 126 may search for the digital supplement associated with the supplement anchor identified by the supplement anchor identification engine 124. In certain embodiments, the digital supplement search engine 126 searches for digital supplements from search server 152 or digital supplement server 172.

For example, after the supplement anchor has been identified, the digital supplement search engine 126 may search for one or more digital supplements associated with the identified supplement anchor. The digital supplemental search engine 126 may generate an image content query containing an image (or an identifier of a supplementary anchor or entity within the image) and may send the image content query to the search server 152. The image content query may also include contextual information such as the location of the client computing device 102. In certain embodiments, data related to digital supplements, such as names, images, or descriptions, is retrieved and presented to the user (eg, by the user interface engine 130). When multiple digital supplements are presented, the user may select one of the digital supplements via the user interface generated by the user interface engine 130.

The digital supplement presentation engine 128 presents or causes the digital supplement to be presented or presented on the client computing device 102. In one embodiment, the digital supplementary presentation engine 128 causes the client computing device to initiate one of the other applications 140. In certain embodiments, the digital supplementary presentation engine 128 displays information or content. For example, the digital supplement presentation engine 128 may cause the user interface engine 130 to generate a user interface containing information or content from the digital supplement displayed by the client computing device 102. In certain embodiments, the digital supplement presentation engine 128 is triggered by the digital supplement search engine 126 searching for digital supplements. The digital supplement presentation engine 128 may then trigger the display device 108 to display the content associated with the digital supplement. In one embodiment, the digital supplement presentation engine 128 displays the digital supplement at a different time than when the digital supplement search engine 126 searches for the digital supplement. For example, digital supplements may be searched in response to an image content query at a first point in time, and digital supplements may be presented at a second point in time. For example, a digital supplement responds to an image content query based on an image of a household item or furniture from a catalog or store during the first time (eg, while the user is viewing the catalog or in the store). Can be searched. Digital supplements, including AR content for household items or furniture, may be presented at a second time (eg, while the user is in a room where household items or furniture can be installed).

The user interface engine 130 creates a user interface. The user interface engine 130 can also cause the client computing device 102 to display the generated user interface. The generated user interface may display information or content from, for example, digital supplements. In certain embodiments, the user interface engine 130 produces a user interface, each containing a plurality of user-operable controls associated with a digital supplement. For example, the user can activate the user (for example, by touching the control on the touch screen, clicking the control using a mouse or another input device, or otherwise activating the control). One of the controls can be activated.

The search server 152 is a computing device. The search server 152 can respond to a search request such as an image content query. The response may include one or more digital supplements that are potentially related to the image content query. In certain embodiments, the search server 152 includes memory 160, processor assembly 154, and communication module 156. The memory 160 may include a content crawler 162, a digital supplemental search engine 164, and a digital supplementary data storage unit 166.

The content crawler 162 may crawl network accessible resources to identify digital supplements. For example, the content crawler 162 may access a web page accessible via the Internet, such as a web page provided by the digital supplement server 172. Crawling a network-accessible resource can include requesting a resource from a web server and analyzing at least a portion of that resource. Digital supplements can be identified based on metadata provided by network-accessible resources, such as XML or JSON data that provides information about digital supplements. In one embodiment, the crawler identifies network-accessible resources based on extracting links from previously crawled network-accessible resources. The content crawler 162 may also identify network accessible resources to crawl based on the receipt of input submitted by the user. For example, a user may submit a URL (or other information) to a network accessible resource, including digital supplements, via a web form or application programming interface (API). In certain embodiments, the content crawler 162 produces an index of identified digital supplements. Content crawlers 162 may also generate scores associated with digital supplements, such as relevance scores or popularity (fame) scores.

The digital supplement search engine 164 receives a search query and generates a response that may include one or more potentially related digital supplements. For example, the digital supplemental search engine 164 may receive an image content query from the client computing device 102. The image content query can include images. The digital supplement search engine 164 can identify the supplement anchors in the image and can identify associated or potentially related digital supplements based on the identified supplement anchors. The digital supplement search engine 164 may send a response to the client computing device 102 that includes the digital supplement or information that may be used to access the digital supplement. In certain embodiments, the digital supplement search engine 164 may return information associated with a plurality of digital supplements. For example, a list of digital supplements can be included in the response to a query. The list may be ordered based on relevance, popularity, or other characteristics of digital supplements to supplemental anchors.

The image content query may include, for example, the image captured by the camera assembly 112, or text or other data associated with the image captured by the camera assembly 112. The image content query may also include other information such as the location of the client computing device 102 or the identifier of the user of the client computing device 102. In certain embodiments, the search server 152 (eg, if the user enables a location service on the client computing device 102 that associates information about the user's location with the user's account) from the user identifier to the client. It can determine the expected position of the computing device 102.

The digital supplement data storage unit 166 stores information related to the digital supplement. In certain embodiments, the digital supplemental data storage unit 166 includes a digital supplementary index. For example, the index can be generated by the content crawler 162. Digital supplemental search engine 164 may use the index to respond to search queries.

The digital supplementary server 172 is a computing device. Digital supplement server 172 provides digital supplement. In certain embodiments, the digital supplement server 172 includes a memory 180, a processor assembly 174, and a communication module 176. Memory 180 may include digital supplement 182 and metadata 184. In certain embodiments, the memory 180 may also include other network accessible resources such as web pages that are not necessarily digital supplements. For example, memory 180 may store a web page containing details about one or more digital supplements and metadata for providing a way to access those digital supplements. Further, the memory 180 may include a resource service delivery engine such as a web server that responds to requests such as HTTP requests using network accessible resources such as web pages and digital supplements.

Digital supplement 182 is any kind of content that can be provided as a supplement to something in the physical environment around the user. The digital supplement 182 may also include any type of content that can supplement the stored image (eg, in the previous physical environment around the user). For example, a digital supplement can be associated with a supplement anchor such as an image, an object or product identified within the image, or a location. The digital supplement 182 may include one or more images, audio content, text data, video, games, data files, applications, or structured text documents. Examples of structured text documents include hypertext markup language (HTML) documents, XML documents, and other types of structured text documents.

Digital Supplement 182 can launch an application and define parameters for that application. The digital supplement 182 can also send a request to a server (eg, an HTTP request) and define parameters for that request. In certain embodiments, the digital supplement 182 begins as a workflow for completing an activity, such as a workflow for completing a purchase. For example, Digital Supplement 182 may add a particular item to the user's shopping cart, add a coupon code, and send an HTTP request to a server that searches the purchase confirmation page.

Metadata 184 is data that describes a digital supplement. Metadata 184 may describe one or more digital supplements provided by the digital supplement server 172 or provided elsewhere. Metadata 184 for digital supplements may include one or more of the following: type indicators, anchor indicators, names, descriptions, preview snippets or excerpts, related images, links such as URLs to digital supplements, and The application identifier associated with the digital preview. Metadata may also include information about the publisher of the digital supplement, such as the publisher name, the publisher description, and the image or icon associated with the publisher. In certain embodiments, the metadata also includes contextual information about the digital supplement, or contextual information that must be satisfied in order to provide the digital supplement. For example, the metadata may include conditions that must be met in order to access digital supplements (eg, geographic conditions, client computing device requirements, required applications). Illustrative contextual information requires a combination of entities identified in the image, or entities identified in the image, or multiple entities identified in the image (eg, some digital supplements require a combination of entities recognized in the image. In some cases). The recognized entity can be a supplementary anchor. In some embodiments, the recognized entity does not supplement the anchor and instead provides contextual information. Metadata 184 may also include supplemental anchors (eg, text, code, entities, or types of entities) associated with digital supplements.

Metadata 184 can be stored in various formats. In certain embodiments, the metadata 184 is stored in a database. Metadata 184 can be stored as an XML file, JSON file, or other format file. In some embodiments, the digital supplement server 172 retrieves the metadata 184 from the database and formats the metadata 184 as XML, JSON, or otherwise provides a response to a request from the client or search server 152. For example, the search server 152 may access the metadata 184 to generate data stored in the digital supplementary data storage unit 166 and used to respond to search requests from the client computing device 102. ..

Communication module 106 includes one or more devices for communicating with other computing devices, such as search server 152 or digital supplement server 172. The communication module 106 may communicate via a wireless or wired network such as network 190. The communication module 156 of the search server 152 and the communication module 176 of the digital supplement server 172 can be similar to the communication module 106.

The display device 108 may be, for example, an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying an image or information to the user. May include. In certain embodiments, the display device 108 includes an optical projector configured to project light onto a portion of the user's eyes.

Memory 120 may include one or more non-temporary computer-readable storage media. The memory 120 generates an image content query based on the acquired image, sends an image content query, receives a response to the image content query, and presents a digital supplement identified in the response to the image content query. , Such as, the instructions and data that can be used by the client computing device 102 to carry out the techniques described herein. The memory 160 of the search server 152 and the memory 180 of the digital supplement server 172 may be similar to the memory 120 and may store data instructions that can be used to implement the techniques of the search server 152 and the digital supplement server 172, respectively. ..

Processor assembly 104 provides one or more devices capable of executing instructions, such as instructions stored by memory 120, to perform digital supplemental associations for visual search and various tasks associated with the search. include. For example, the processor assembly 104 may include a central processing unit (CPU) and / or a graphics processor unit (GPU). For example, in the presence of a GPU, some image / video rendering tasks, such as generating and displaying a user interface, or displaying parts of a digital supplement, can be offloaded from the CPU to the GPU. In certain embodiments, some image recognition tasks can also be offloaded from the CPU to the GPU.

Although not shown in FIG. 1, certain embodiments include a head mounted display device (HMD). The HMD may be a device separate from the client computing device 102, or the client computing device 102 may include an HMD. In one embodiment, the client computing device 102 communicates with the HMD over a cable. For example, the client computing device 102 can transmit video and / or audio signals to the HMD for display to the user, which can provide motion, position, and / or orientation information to the client computing device 102. Can be sent to.

The client computing device 102 may also include various user input components (not shown), such as a controller that communicates with the client computing device 102 using a wireless communication protocol. In certain embodiments, the client computing device 102 uses a wired connection (eg, a universal serial bus (USB) cable) or a wireless communication protocol (eg, any WiFi protocol, any BlueTooth protocol, Zigbee®). ) Etc.) can communicate with the HMD (not shown). In certain embodiments, the client computing device 102 is a component of the HMD and may be housed within the housing of the HMD.

The network 190 can be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network. The client computing device 102 can receive, for example, audio / video signals, which, in an exemplary embodiment, may be provided over the network as part of a digital supplement.

FIG. 2 is a third person view of an exemplary physical space 200 in which an embodiment of the client computing device 102 accesses a digital supplement. In this example, physical space 200 includes object 222. Here, the object 222 is an artwork on the wall of the physical space 200. Object 222 is included in the field of view 204 of the camera assembly 112 of the client computing device 102.

An exemplary user interface screen 206 is also shown. The user interface screen 206 may be generated, for example, by the user interface engine 130 of the client computing device 102. The user interface screen 206 includes an image display panel 208 and a digital supplementary selection panel 210. The image display panel 208 displays an image. For example, the image display panel 208 may show an image corresponding to a real-time feed from the camera assembly 112 of the client computing device 102. In certain embodiments, the image display panel 208 shows a previously acquired image or an image retrieved from memory 120 of the client computing device 102.

In certain embodiments, the user interface screen 206 is presented to the user on the display device of the client computing device 102. In certain embodiments, the user interface screen 206 may be so overlaid on an image of physical space (or a video feed captured by a camera of a computing device). In addition, the user interface screen 206 may be displayed as AR content over the user's field of view using the HMD worn by the user.

The image display panel 208 may also include annotations or user interface elements associated with the image. For example, the image display panel 208 may include an indicator that an object in the image (eg, object 222) has been recognized as a supplementary anchor. The indicator may include a user-enabled control for accessing or viewing information about the digital supplement associated with the identified supplement anchor. In some situations, the image displayed on the image display panel 208 may contain multiple objects that are recognized as supplemental anchors, and the image display panel 208 may include multiple annotations that overlay the images to identify those supplementary anchors. Can include.

The supplementary anchor may be recognized by the supplementary anchor identification engine of the client computing device 102. In certain embodiments, the supplemental anchor is identified by sending the image to the search server 152. The search server 152 can then analyze the image and identify supplemental anchors within the image. In certain embodiments, the search server 152 may send one or more of the positions associated with the supplementary anchors (eg, image coordinates) or the dimensions of any identified object to the client computing device 102. The client computing device 102 may then update the user interface screen to show annotations that identify supplemental anchors (or associated objects) in the image. In certain embodiments, the client computing device 102 may track the location of a supplementary anchor (or associated object) within a video stream (eg, a sequence of images acquired sequentially) acquired by the camera assembly 112 (for example, a sequence of images acquired sequentially). For example, the supplemental anchor identification engine 124 may track the supplementary anchors identified by the search server 152).

The digital supplement selection panel 210 allows the user to select a digital supplement for presentation. For example, the digital supplement selection panel 210 may include a menu, each containing a user-enabled control associated with the digital supplement. In this example, the digital supplement selection panel 210 includes a user-enabled control 212 and a user-enabled control 214, each containing information about the associated digital supplement. For example, the user-enabled control may display one or more of a name (or title), a brief description, and an image associated with the digital supplement that may be received from the search server 152. When the user-enabled control 212 or the user-enabled control 214 is activated, the associated digital supplemental content may be presented to the user. Presenting a digital supplement to a user may include causing the client computing device 102 to display a user interface screen that includes images, videos, text, other content, or a combination thereof from the digital supplement. In certain embodiments, the digital supplementary content is displayed as an overlay on the image display panel 208 on the image or camera feed. Digital supplementary content can be three-dimensional augmented reality content.

In certain embodiments, presenting a digital supplement comprises launching an application installed on the client computing device 102 (eg, one of the other applications 140). Presenting a digital supplement may also include sending a request to the URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as the identifier of the product or object identified in the image. In certain embodiments, the image (or other content) from the image content query is passed a parameter with the request. The image may also be provided via the API associated with the digital supplement server 172. In one embodiment, the client computing device 102 transmits an image to the digital supplement server 172. In certain embodiments, the search server 152 may transmit the image to the digital supplement server 172. For example, in response to a user selecting a digital supplement, the client computing device 102 can send a selection indicator to the search server 152, which in turn sends an image to the corresponding digital supplement server. obtain. The client computing device 102 may also send the URL to a location on the search server 152 that the digital supplement server 172 can use to access the image. Beneficially, these practices can reduce the amount of data that the client computing device needs to send.

A digital supplement associated with the user-enabled control 212 may display information about the object 222, such as information from a museum. A digital supplement associated with the user-enabled control 214 may display information related to the museum. For example, the presentation of a digital supplement may mark a stop on a museum tour as complete and display information about the next stop.

FIG. 3 is a diagram of an exemplary method 300 for enabling a digital supplement trigger according to an embodiment described herein. The method 300 may be performed, for example, by the content crawler 162 of the search server 152 to allow the user to access the digital supplement based on the image content query.

In operation 302, data that specifies digital supplementation is received. The data may identify the digital supplement and the circumstances in which the digital supplement should be provided. Data specifying digital supplements can be received in a variety of ways. For example, data specifying a digital supplement may be received from a network accessible resource such as a web page that contains metadata about the digital supplement. The data specifying the digital supplement may also be received, for example, via the API or form provided by the search server 152. Data that specifies digital supplements can also be received from memory locations or data stores.

The data about the digital supplement may include access data that can be used by the client computing device to access the digital supplement. For example, the access data may include a digital supplement URL and parameters passed to that URL. Access data may also include the identifier of the application and the parameters for that application. Data on digital supplements may also include descriptive data on digital supplements. The descriptive data can be used by a client computing device to present information about digital supplements to the user (eg, on a menu where the user can select digital supplements). The descriptive data may include, for example, a name (or title), a description, a publisher's name, and an image. Data on digital supplements may also include the identifier of the supplement anchor.

In operation 304, a data structure instance based on the received data is generated. The data structure can be, for example, a record in the database. The database may be a relational database, and data structure instances can be linked (eg, via an outside key) to one or more records associated with a supplemental anchor.

In operation 306, after the data structure instance is generated, the digital supplement can be searched by the image content query. For example, the database fields associated with a data structure instance may be actively configured so that the digital supplement search engine 164 can access and return the associated digital supplement. In certain embodiments, the digital supplement trigger may include saving or committing a record in the database. In certain embodiments, enabling the search for digital supplements comprises enabling the triggering of digital supplements by the client computing device. For example, after the instance is created, the digital supplement may be returned to the client computing device in response to the search and activated or presented by the client computing device.

FIG. 4 is a diagram of an exemplary method 400 for enabling a digital supplement trigger, according to an embodiment described herein. The method 400 may be performed, for example, by the content crawler 162 of the search server 152 to allow the user to access the digital supplement based on the image content query.

In operation 402, network accessible resources are analyzed. In some embodiments, the network accessible resource is, for example, a web page presented by the digital supplement server 172. In one embodiment, a set of network accessible resources is analyzed. The set of network accessible resources may be generated based on submissions via forms or APIs. In certain embodiments, the set of network accessible resources can be generated by crawling other network accessible resources to identify the URL. This crawling process can be performed recursively.

In operation 404, the metadata associated with the digital supplement in the network accessible resource is identified. In certain embodiments, network accessible resources may include metadata indicators associated with digital supplements. For example, a network accessible resource may include a tag that identifies a portion of the network accessible resource that contains metadata. The tag can be an XML tag with a particular type or attribute. The tag can be an HTML tag such as a script tag containing a JSON data structure containing metadata.

In operation 406, a digital supplemental data structure instance based on the metadata is generated. The operation 406 can be similar to the operation 304.

In operation 408, an image content query is received. The image content query can be sent by a client computing device such as, for example, the client computing device 102. In certain embodiments, the image content query includes an image. The image content query can also include textual data that describes the image. For example, the text data may include the identifier of the supplemental anchor in the image obtained by the camera assembly of the client computing device. In certain embodiments, the image content query also includes other information, such as the location of the client computing device or the identifier of the user account associated with the client computing device.

In operation 410, the plurality of digital supplemental data structure instances are identified based on the image content query. In certain embodiments, the supplemental anchor is identified within the image provided by the image content query. The supplement anchor can then be used to query the index or database for the associated digital supplement. In certain embodiments, other data provided with the query may also be used to identify digital supplements, such as information associated with the location of the client computing device or user account. In certain embodiments, multiple supplemental anchors can be used to identify the supplementary anchors associated with them.

In operation 412, the ordering of the plurality of digital supplemental data structure instances is determined. The ordering may be based on the various scores associated with the digital supplement, or the relevance of the digital supplement to the image content query. In certain embodiments, the relevance score corresponding to the relevance of a digital supplement to an image content query can be used to order multiple digital supplement data structure instances. The relevance score is the content of the digital supplement, the digital supplement. Content, link text, or content near a link to a digital supplement on a network-accessible resource (or network-accessible resource associated with a digital supplement) that links to. It can be determined from multiple factors, such as one or more of them.

Scores can also be obtained based on popularity metrics. The fame metric is an example of a popular metric. The fame metric may be based on a combination of how many other network resources link to the digital supplement and the fame scores of those other network accessible resources. In certain embodiments, the popularity score may be based on how often or how often digital resources are selected. In certain embodiments, the popularity score may correspond to how often digital resources are selected for image content queries.

Scores may be determined or may be retrieved from a data store or API. In certain embodiments, the API is accessed to retrieve the digital supplemental score. For example, scores can be searched by search engines that determine the relevance and / or popularity of digital resources with respect to search terms based on supplemental anchors.

Multiple digital supplemental data structures may also be ordered based on frequency of use by a particular user (eg, a user of a client computing device) or recent recency by that particular user. In certain embodiments, the plurality of digital supplemental data structures are randomly ordered.

In operation 414, the image content query is replied based on a plurality of digital supplemental data structure instances. For example, the information associated with a plurality of digital supplemental data structure instances may be transmitted to the client computing device in the order determined in operation 412. In certain embodiments, the information includes descriptive data that may be presented within a menu or another type of user interface configured to receive digital supplemental user selection. The information may also include access data that may be used by client computing devices to access or present digital supplements.

FIG. 5 is a diagram of an exemplary method 500 for retrieving and presenting digital supplements according to embodiments described herein. The method 500 may be performed, for example, by application 122 of the client computing device 102 to identify and access the digital supplement based on the image content query.

In operation 502, the image-based image content query is sent to a server computing device (eg, search server 152). For example, the image may be acquired by the camera assembly 112 of the client computing device 102. The image can also be a stored image, such as an image previously acquired by camera assembly 112. In certain embodiments, the image content query comprises only images. In certain embodiments, the image content query includes additional information. For example, the image content query may include information such as the location of the client computing device 102 or the identifier of the account associated with the user of the client computing device 102. Application 122 may also identify anchors in the image (eg, using the supplemental anchor identification engine 124). The image content query may include an identifier of the identified anchor (eg, a text, a number, or some other type of identifier). At least in certain embodiments, the image content query does not include an image.

In certain embodiments, sending an image content query to a server involves calling an API. In certain embodiments, sending an image content query to a server involves invoking an API provided by the server. In certain embodiments, sending an image content query to a server involves submitting a form using the HTTP protocol (eg, submitting a GET or POST request).

In operation 504, a response to an image content query that identifies the digital supplement is received. The response may be received from the search server 152 via network 190. The response may include one or more digital supplements identified by the search server 152 based on the image content query. For example, the response may include an array of data associated with the digital supplement. In certain embodiments, the data associated with the digital supplement may include descriptive data that can be used to present digital supplement options for the user to select. For example, the descriptive data may include a name, a short description, an issuer name, and an image. The data may also include access data such as the URL and parameters to be included in the request via the URL, or application names and related parameters. The data may also include the position, coordinates, or dimensions of the supplementary anchor in the image transmitted in the image content query (eg, if the supplementary anchor is identified by the search server 152).

In operation 506, a user interface screen containing information associated with the digital supplement is displayed. In certain embodiments, the user interface screen includes annotations that overlay the identified supplemental anchors (eg, based on the coordinates provided). Annotations may provide information about the objects in the image associated with the identified supplemental anchor. The annotation may include a user-operable control that may be activated to present or activate a digital supplement. The user interface screen may also include a digital supplement selection panel that can be used to select from a plurality of digital supplements identified in the response received in operation 504. In certain embodiments, the user interface screen can be generated by a web browser that opens the URL specified by the digital supplement. The user interface screen can also be generated by another application launched to provide digital supplements.

FIG. 6 is a diagram of an exemplary method 600 for identifying and presenting digital supplements based on images, according to the embodiments described herein. The method 600 can be performed, for example, by application 122 on the client computing device 102 to identify and access the digital supplement based on the image content query.

In operation 602, an image is acquired. For example, the image may be acquired by the camera assembly 112 of the client computing device 102. In certain embodiments, a series of images (ie, video) may be acquired by the camera assembly 112.

In operation 604, the image-based image content query is sent to a server computing device such as the search server 152. The operation 604 can be similar to the operation 502. In an embodiment in which a sequence of images is obtained, the image content query may include multiple images or sequences of images. In certain embodiments, the sequence of images may be streamed to the server computing device.

In operation 606, a response to an image content query that identifies a plurality of digital supplements is received. The operation 606 can be the same as the above-mentioned operation 504.

At operation 608, a user interface screen is displayed that includes a user-operable control for selecting one digital supplement from a plurality of digital supplements. For example, a digital supplemental selection panel may be displayed. The digital supplement selection panel may include multiple user-operable controls, each associated with one of the plurality of digital supplements identified in the response. Digital supplement selection may provide user-enabled controls based on the ordering or ranking of digital supplements provided by the server computing device. The digital supplemental selection panel allows the user to arrange the actuable controls vertically, horizontally, or otherwise. The user-enabled control may be associated with or include information about the associated digital supplement that the user can consider when deciding whether to select the digital supplement. For example, the information displayed may include one or more of the digital supplement's name, description, image, and publisher's name.

In operation 610, a user input for selecting a digital supplement is received. User input can be a click with a mouse or other device. The user input can also be a stylus or touch input from a finger. Another example of user input is approximately touch input (eg, holding a finger or pointing device close to the screen). In certain embodiments, the user input may also include hand gestures, head movements, eye movements, or speech inputs.

In operation 612, the information is provided to the resource associated with the selected digital supplement. For example, information about a user of a client computing device may be sent to a server that provides digital supplements (if permission is provided to provide the information). Information may also be provided to applications that provide digital supplements. Various types of information can be provided. For example, the information may include user information such as username, user preference, or location.

The information may also include information associated with an image content query, such as an image or sequence of images. The information may also include the identifier and / or position of one or more supplemental anchors in the image. This information can be used to provide digital supplements to users. For example, digital supplement AR content can be formed and positioned based on images.

The information may be transmitted directly by the client computing device 102 to a resource associated with the digital supplement (eg, digital supplement server 172). In certain embodiments, the information is provided by the search server 152 to the resources associated with the digital supplement (eg, the client computing device does not need to send too much data). In at least some of these practices, the client computing device 102 may send selection information that identifies the selected digital supplement to the search server 152. Upon receiving the selection and confirming that the user has allowed the information to be shared, the search server 152 may send the information to a resource that provides digital supplements. The client computing device 102 may also prompt the user to share information. In certain embodiments, the search server 152 may determine the information to send to a resource based on a digital supplement data structure instance, which can be based on the metadata associated with the digital supplement.

In operation 614, the user interface is updated based on the selected digital supplement. The operation 614 can be similar to the operation 506.

7A-7C are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 for performing an image content search and displaying a digital supplement. FIG. 7A shows the user interface screen 700a. The user interface screen 700a includes an image display panel 708 and an information panel 730. In this example, the image display panel 708 displays an image of a shelf full of wine bottles (eg, which you might find in the store). The image display panel 708 also includes an indicator 740 and an indicator 742. Each of these indicators indicates that the wine bottle shown in the image below the indicator is recognized as a supplemental anchor (eg, in this case, as a recognized product). Indicator 740 and indicator 742 are examples of user-operable controls. Within the information panel 730, the instruction "Tap what you are interested in" is provided.

In FIG. 7B, the user interface screen 700b is shown after the user activates the indicator 740. After activation, note 744 from the digital supplement is displayed. Note 744 contains information on wine ratings that may help the user select a bottle of wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after the user activates the indicator 740. The user interface screen 700c is shown instead of or in addition to displaying the user interface screen 700b (eg, after the activation of Note 744 or when the user swipes up on the information panel 730 of FIG. 7B). Is done. FIG. 7C shows an expanded information panel 732. The expanded information panel 732 occupies more user interface screens 700c than the information panel 730 featured in FIGS. 7A and 7B.

The extended information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734. The digital supplementary selection panel 710 includes a user-operable control 712, a user-operable control 714, and a (partially visible) user-operable control 716. In certain embodiments, a user swipes over the digital supplementary selection panel 710 to display additional user-enabled controls. The user-operable controls on the digital supplement selection panel 710 may be arranged in a ranked order. The user actuable control 712 is associated with a digital supplement for meal pairing. When the user activable control 712 is activated, a digital supplement may be displayed that displays food and meal pairing information for the selected wine. The user-enabled control 714 is associated with a digital supplement that stores a photo. When activated, an application for saving photos may be activated to provide an image. Additional information may be stored with photographs such as identified supplemental anchors.

The digital supplement content display panel 734 may display content from the digital supplement. The Digital Supplement Content Display Panel 734 may display the default digital supplement or the highest ranked digital supplement associated with the identified supplement anchor. In this example, the digital supplemental content display panel 734 contains product information about the product associated with the selected supplementary anchor. In this case, the wine name, rating, place of origin, image, and comments are provided.

8A-8C are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 for performing an image content search and displaying a digital supplement. In this example, the image content search is based on the image on the receipt.

FIG. 8A shows the user interface screen 800a. The user interface screen 800a includes an image display panel 808 and an information panel 830. In this example, the image display panel 808 displays an image of the receipt from the restaurant. The image display panel 808 also includes an indicator 840, an indicator 842, an annotation 844, and a highlight overlay 846. In this case, the indicator 840 is associated with the receipt as a document and the indicator 842 is associated with the particular restaurant named for the receipt. The identified receipt document and the identified restaurant name are both examples of supplemental anchors.

Note 844 is associated with a digital supplement that provides an advanced calculator. In this example, an exemplary advanced calculator is included in annotation 844 and is superposed at the appropriate location on the image display panel 808. In certain embodiments, the digital supplement may be selected by default and may be displayed once the appropriate supplement anchor is identified. The highlight overlay 846 is overlaid on a portion of the receipt document containing the information used by the digital supplement of the advanced calculator.

In this example, the items displayed on the information panel 830 relate to the receipt as a document as if the indicator 840 was activated. In certain embodiments, the identified supplemental anchor is, for example, the user's past behavior, another user's behavior on a similar image, the confidence score of the supplementary anchor, or the position or size of a portion of the image to which the supplementary anchor is associated. Based on the user's potential relevance or interests, the ranking is based on. The information panel 830 may then display items related to the highest ranked supplemental anchor, at least in certain embodiments. If instead the indicator 842 is activated, the information panel 830 may include items relating to a particular restaurant.

Here, the information panel 830 includes a digital supplementary selection panel 810. The digital supplement panel includes a user-operable control 812, a user-operable control 814, and a user-operable control 816. In this example, the user-operable control 812 is associated with the advanced computer digital supplement, the user-operable control 814 is associated with the check-split digital supplement, and the user-operable control 816 is associated with the expense reporting digital supplement. Has been done. For example, when the user activable control 812 is activated, a user interface control for adjusting the parameters of the advanced calculator may be displayed (eg, to adjust the percentage).

In FIG. 8B, the user interface screen 800b is shown after the user activates the user actuable control 814. After activation, an extended information panel 832 is shown that contains items that help the user calculate how to split the invoice. For example, the number of people splitting an invoice can be entered to determine the amount each should pay.

In FIG. 8C, the user interface screen 800c is shown after the user activates the user actuable control 816. After activation, an extended information panel 834 is presented containing items that help the user remember the receipt in the expense report. For example, the user can select an expense report to which the receipt should be associated (eg, "Sydney Travel 2018"). Once the expense report is selected, the image of the receipt may be uploaded to the expense report submission or management system. In one embodiment, all images displayed on the image display panel 808 are uploaded. In some embodiments, a portion of the image is uploaded (eg, the image is cropped to include only the receipt).

9A and 9B are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 for performing an image content search and displaying digital supplements. In this example, the image content search is based on facial images.

FIG. 9A shows the user interface screen 900a. The user interface screen 900a includes an image display panel 908 and an information panel 930. In this example, the image display panel 908 displays a face image. Here, the face is an example of a supplementary anchor. The information panel 930 includes a user-enabled control 912 for digital supplement identified for the supplement anchor in the image (ie, face). The user actuable control 912 is associated with a digital supplement for tying to eyeglasses.

In FIG. 9B, the user interface screen 900b is shown after the user activates the user actuable control 912. After activation, an extended information panel 932 is shown that includes items to help the user visually try the eyeglasses on the face in the image. Here, multiple eyeglass styles are displayed, allowing the user to select a pair to try. When a pair of glasses is selected, the AR content 960 is overlaid on the image display panel 908. Here, the AR content 960 corresponds to the selected eyeglasses and has a size that matches the face in the image. In one embodiment, when a digital supplement for trying on eyeglasses is selected, the image shown on the image display panel 908 is sent to a server that provides the digital supplement, which causes the image to display AR content 960. It can be analyzed to determine where and how it should be placed, what size it should be, or recommended to try on eyeglasses.

10A-10C are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 for performing an image content search and displaying a digital supplement. In this example, the image content search is based on the image of furniture in the catalog.

FIG. 10A shows the user interface screen 1000a. The user interface screen 1000a includes an image display panel 1008. In this example, the image display panel 1008 displays an image of a part of the furniture catalog page. The image display panel also includes an indicator 1040, an indicator 1042, and an indicator 1044. In this example, indicator 1040 is associated with the bed, indicator 1042 is associated with the ornament, and indicator 1044 is associated with the rug. Images of beds, ornaments, and rugs in the catalog are examples of supplemental anchors.

In FIG. 10B, the user interface screen 1000b is shown after the user has selected indicator 1040 (eg, by touching the screen at or near where the indicator 1040 is displayed). The user interface screen 1000b includes a digital supplementary selection panel 1010 and an information panel 1030. The information panel 1030 contains information (eg, product name, description, image) about the supplemental anchor associated with the selected indicator.

The digital supplementary selection panel 1010 includes a user-operable control 1012 and a user-operable control 1014. The user actuable control 1012 is associated with a digital supplement that provides a home view. User actuable control 1014 is associated with another digital supplement (eg, a digital supplement for posting on social media sites).

In FIG. 10C, the user interface screen 1000c is shown after activation of the user operable control 1012. The user interface screen 1000c includes an image display panel 1008, a digital supplementary selection panel 1010, and a reduced information panel 1032. The reduced information panel 1032 may include a user-operable control that, when activated, can pop up and display the information panel.

Here, the image display panel 1008 displays an image of the room and includes AR content 1060. The AR content 1060 includes a 3D model of the bed associated with the indicator 1040 overlaid on the image panel. The user may adjust the position of the AR content 1060 in the room to see how the bed fits into the room. In one embodiment, when a digital supplement for home view is selected, the image shown on the image display panel 1008 is sent to a server that provides the digital supplement, whereby the image displays AR content 1060. It can be analyzed to determine where and how it should be positioned and what size it should be. In certain embodiments, the AR content 1060 may be provided after the image content query.

11A-11C are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 for performing various image content searches in the store. In this example, the image content search is based on the image of the product acquired in the store.

FIG. 11A shows the user interface screen 1100a. The user interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 displays an image taken in the store. The image display panel 1108 also includes an indicator 1140 associated with the vase. The vase displayed on the image display panel 1108 is an example of a supplementary anchor. The information panel 1130 displays product information about the vase and a digital supplement that includes functionality for purchasing the vase. Digital supplements may include, for example, a workflow for initiating a vase purchase. In this example, the digital supplement is identified based on the image content and the location of the client computing device, thereby identifying the digital supplement published (or associated with) the store from which the image was obtained. It can also be provided as a result of high ranking for image content queries when the client computing device is in the store. In one embodiment, if the client computing device is repositioned, different digital supplements will be provided for the same image.

FIG. 11B shows the user interface screen 1100b. The user interface screen 1100b includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 displays another image acquired in the store. The image display panel 1108 also includes an indicator 1142 associated with a lag. The lug displayed on the image display panel 1108 is an example of a supplementary anchor. The information panel 1130 displays product information about the rug and a digital supplement that includes functionality for selecting the size and purchasing the rug. As shown in FIG. 11A, the digital supplement is identified based on the image content and the location of the client computing device.

FIG. 11C shows the user interface screen 1100c. The user interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, the image display panel 1108 displays another image acquired in the store. The image display panel 1108 also includes an indicator 1144 associated with the vase. The vase displayed on the image display panel 1108 is an example of a supplementary anchor. The information panel 1130 displays a digital supplement containing product information about the vase. The information panel 1130 also includes a coupon indicator 1132 and functionality for redeeming coupons. Redeeming a coupon may include purchasing merchandise at a discounted price from the website associated with the retailer. In certain embodiments, a coupon code that can be used to secure a discount during clearing is presented. Similar to FIGS. 11A and 11B, digital supplements are identified based on image content and the location of the client computing device.

12A-12C are schematic views of a user interface screen displayed by an embodiment of the client computing device 102 during various image content searches. In this example, the image content search is based on the image of a movie poster (which may be obtained, for example, in a movie theater).

FIG. 12A shows the user interface screen 1200a. The user interface screen 1200a includes an image display panel 1208. In this example, the image display panel 1208 displays an image of a movie poster. The image display panel 1208 also includes an indicator 1240 associated with the movie poster identified in the image. A movie poster is an example of a supplementary anchor. Indicator 1240 may include a user-operable control that, when activated, displays a digital supplement or a menu for selecting a digital supplement.

FIG. 12B shows the user interface screen 1200b. The image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified in the image. For example, preview digital supplement 1242 may be shown after activation of indicator 1240 (FIG. 12A). The preview digital supplement 1242 may superimpose an image or video from the movie associated with the identified movie poster on top of the movie poster image.

FIG. 12C shows the user interface screen 1200c. The image display panel 1208 also includes a rating indicator 1244 and a rating indicator 1246. The rating indicator 1244 and rating indicator 1246 may be generated by one or more digital supplements in response to an image content query that includes a movie poster. Digital supplements can superimpose rating information within an image, for example for a movie associated with a movie poster. The rating indicator 1244 and rating indicator 1246 may include a user actuable control that, when activated, provides additional information about the rating and associated movie.

FIG. 13 is a computer device 1300 and a mobile computer device 1350 that can be used with the techniques described herein (eg, for implementing client computing device 102, search server 152, and digital supplementary server 172). An example is shown. The computing device 1300 includes a processor 1302, a memory 1304, a storage device 1306, a high-speed interface 1308 for connecting to the memory 1304 and the high-speed expansion port 1310, and a low-speed interface for connecting to the low-speed bus 1314 and the storage device 1306. Includes 1312 and. Each of the components 1302, 1304, 1306, 1308, 1310 and 1312 is interconnected using various buses and can be installed on a common motherboard or otherwise as needed. Processor 1302 includes instructions stored in memory 1304 or storage device 1306 to display graphical information for the GUI on an external input / output device such as display 1316 coupled to high speed interface 1308. It can process instructions for execution within the computing device 1300. In other embodiments, multiple processors and / or multiple buses may be used with multiple memory and types of memory, if desired. Also, a plurality of computing devices 1300 may be connected, and each device provides a part of the required operation (eg, as a server bank, a group of blade servers, or a multiprocessor system).

The memory 1304 stores information in the computing device 1300. In one embodiment, memory 1304 is one or more volatile memory units. In another embodiment, memory 1304 is one or more non-volatile memory units. Memory 1304 can also be another form of computer-readable storage medium, such as magnetic or optical discs.

The storage device 1306 may provide large capacity storage for the computing device 1300. In one embodiment, the storage device 1306 is in a floppy (registered trademark) disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or storage area network or other configuration. Can be, or can include, a computer-readable storage medium, such as an array of devices containing the device. Computer program products can be tangibly embodied in information carriers. Computer program products may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1304, storage device 1306, or memory on processor 1302.

The high-speed controller 1308 manages bandwidth-intensive operation for the computing device 1300, while the low-speed controller 1312 manages lower bandwidth-intensive operation. The assignment of such functions is merely an example. In one embodiment, the high speed controller 1308 is coupled to memory 1304, to display 1316 (eg, via a graphics processor or accelerator), and to fast expansion port 1310 capable of accepting various expansion cards (not shown). Has been done. In this embodiment, the slow controller 1312 is coupled to the storage device 1306 and the slow expansion port 1314. Slow expansion ports that may include various communication ports (eg, USB, Bluetooth®, Ethernet®, Wireless Ethernet®) include keyboards, pointing devices, scanners, for example, via network adapters. , Or it can be coupled to one or more input / output devices, such as networking devices such as switches or routers.

The computing device 1300 can be implemented in several different forms, as shown in the figure. For example, it may be performed multiple times as a standard server 1320 or in a group of such servers. It can also be implemented as part of the rack server system 1324. In addition, it can be implemented as a personal computer such as a laptop computer 1322. Alternatively, the components from the computing device 1300 can be combined with other components within a mobile device (not shown) such as device 1350. Each such device may include one or more of the computing devices 1300, 1350, and the entire system may consist of a plurality of computing devices 1300, 1350 communicating with each other.

The computing device 1350 includes, among other components, a processor 1352, a memory 1364, an input / output device such as a display 1354, a communication interface 1366, and a transceiver 1368. Device 1350 may also include storage devices such as microdrives or other devices to provide additional storage. Each of the components 1350, 1352, 1364, 1354, 1366 and 1368 are interconnected using various buses, some of which are on a common motherboard or, as needed, others. Can be installed in a way.

Processor 1352 can execute instructions within the computing device 1350, including instructions stored in memory 1364. The processor may be implemented as a chipset of chips containing multiple separate analog and digital processors. The processor may provide coordination of other components of device 1350, such as control of the user interface, applications executed by device 1350, and wireless communication by device 1350.

Processor 1352 may communicate with the user via control interface 1358 and display interface 1356 coupled to display 1354. The display 1354 can be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal display), an LED (Light Emitting Diode), an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 1356 may include suitable circuits for driving the display 1354 to present graphics and other information to the user. The control interface 1358 may receive commands from the user, translate them and submit them to the processor 1352. In addition, the external interface 1362 may be provided in communication with the processor 1352 to allow short-range communication between the device 1350 and other devices. The external interface 1362 may be prepared, for example, for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 1364 stores information in the computing device 1350. Memory 1364 may be implemented as one or more computer-readable storage media, one or more volatile memory units, or one or more of one or more non-volatile memory units. Expansion memory 1374 may also be provided and connected to device 1350 via expansion interface 1372, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such extended memory 1374 can provide additional storage space for device 1350 or can store applications or other information for device 1350. Specifically, the extended memory 1374 can include instructions for executing or supplementing the above-mentioned processes, and may also include secure information. Thus, for example, extended memory 1374 may be provided as a security module for device 1350 and may be programmed with instructions that allow safe use of device 1350. In addition, secure applications can be provided via the SIMM card along with additional information, such as placing the identity information on the SIMM card in an unhackable manner.

The memory may include, for example, flash memory and / or NVRAM memory as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. Computer program products, when executed, include instructions that perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1364, extended memory 1374, or memory on processor 1352, which can be received, for example, via transceiver 1368 or external interface 1362.

The device 1350 may communicate wirelessly via a communication interface 1366, which may optionally include a digital signal processing circuit. The communication interface 1366 is under various modes or protocols such as GSM® voice call, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA®, CDMA2000, or GPRS. Can provide communication. Such communication may take place, for example, via the radio frequency transceiver 1368. In addition, short-range communications can occur, such as with Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, the GPS (Global Positioning System) receiver module 1370 may provide device 1350 with additional navigation and location-related radio data that may be used as needed by applications running on device 1350. ..

The device 1350 may also communicate audibly using the voice codec 1360, which can receive utterance information from the user and convert it into usable digital information. The voice codec 1360 can also generate audible sound for the user, for example via a speaker in the handset of device 1350. Such sounds may include sounds from voice phones, recorded sounds (eg, voice messages, music files, etc.), and may also include sounds produced by applications running on the device 1350. ..

The computing device 1350 can be implemented in several different forms, as shown in the figure. For example, it can be implemented as a mobile phone 1380. It can also be implemented as part of a smartphone 1382, personal digital assistant, or other similar mobile device.

Various embodiments of the systems and techniques described herein include digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or them. It can be realized by the combination of. These various implementations are dedicated or general purpose, combined to receive data and instructions from the storage system, at least one input device, and at least one output device and send the data and instructions to them. Obtaining may include implementation in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor.

These computer programs (also known as programs, software, software applications or code) include machine instructions for programmable processors, in high-level procedures and / or object-oriented programming languages, and / or assemblies. / Can be implemented in machine language. As used herein, the terms "machine-readable medium" and "computer-readable storage medium" include machine-readable media that receive machine instructions as machine-readable signals. Refers to any computer program product, device, and / or device (eg, magnetic disk, disk, memory, programmable logic device (PLD)) used to provide instructions and / or data to a programmable processor. The term "machine readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

To provide user interaction, the systems and techniques described herein are display devices (LEDs (Light Emitting Diodes) or OLEDs (Organic LEDs) or LCDs (Liquid Crystal Display)) for displaying information to the user. It can be performed on a computer having a monitor / screen) and a keyboard and pointing device (eg, mouse or trackball) from which the user can give input to the computer. Other types of devices can also be used to provide interaction with the user, eg, the feedback provided to the user is any form of sensory feedback (eg, visual feedback, auditory feedback, etc.). Or tactile feedback), and input from the user can be received in any form, including acoustic input, audio input, or tactile input.

The systems and techniques described herein include a back-end component (eg, as a data server), or a middleware component (eg, as an application server), or a front-end component (eg, a user). Includes (a client computer with a graphical user interface or web browser capable of interacting with embodiments of the systems and techniques described herein), or any combination of such backend, middleware, or frontend components. Can be implemented in computing systems, including. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), and the Internet.

Computing systems can include clients and servers. Clients and servers are generally remote from each other and typically interact over a communication network. The client-server relationship arises from computer programs that run on their respective computers and have a client-server relationship with each other.

In one embodiment, the computing device shown in FIG. 13 interfaces with an AR headset / HMD device 1390 to provide a sensor for creating an augmented environment for viewing inserted content in physical space. Can include. For example, one or more sensors included in the computing device 1350 or other computing device shown in FIG. 13 may provide input to the AR headset 1390, or may generally provide input to AR space. .. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 1350 can use sensors to determine the absolute position and / or detected rotation of the computing device in AR space, and that absolute position and / or detected rotation to AR space. Can be used as input. For example, the computing device 1350 can be embedded in AR space as virtual objects such as controllers, laser pointers, keyboards, and weapons. When incorporated into AR space, the user's positioning of the computing device / virtual object may allow the user to position the computing device to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user can operate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in a circle, and can use the device in the same way as using a laser pointer. In certain embodiments, the user can use a virtual laser pointer to aim at a target location.

In certain embodiments, one or more input devices included in or coupled to the computing device 1350 can be used as inputs into AR space. Input devices include touch screens, keyboards, one or more buttons, trackpads, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, earphones with input capabilities or earbuds that plug into the ear canal. (Buds), game controllers, or other connectable input devices may be included, but not limited to. When the computing device is embedded in the AR space, the user interacting with the input device contained in the computing device 1350 can cause a particular action in the AR space.

In certain embodiments, the touch screen of the computing device 1350 can be rendered as a touchpad in AR space. The user can interact with the touch screen of the computing device 1350. Interactions are rendered as movements on the touchpad rendered in AR space, for example in an AR headset 1390. The rendered motion can control the virtual object in AR space.

In certain embodiments, one or more output devices included in the computing device 1350 can provide output and / or feedback to the user of the AR headset 1390 in AR space. The output and / or feedback can be visual, tactile, or audio. Outputs and / or feedback are vibrations, turning on and off or flashing or flashing and / or flashing one or more lights or strobes, sounding alarms, chimes, playing music, and audio files. It may include, but is not limited to, performances. Output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In one embodiment, the computing device 1350 may appear as another object in a computer-generated 3D environment. User interaction with the computing device 1350 (eg, rotating, shaking, touching the touch screen, swiping a finger across the touch screen) is interpreted as interacting with an object in AR space. Can be done. In the example of a laser pointer in AR space, the computing device 1350 appears as a virtual laser pointer in a computer-generated 3D environment. When the user operates the computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 1350 in the AR environment on the computing device 1350 or AR headset 1390. User interactions with computing devices can be translated into interactions with user interfaces generated in an AR environment for controllable devices.

In certain embodiments, the computing device 1350 may include a touch screen. For example, the user can interact with the touch screen to interact with the user interface for controllable devices. For example, a touch screen may include user interface elements such as sliders that can control the characteristics of controllable devices.

Computing device 1300 represents various forms of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Is intended. The computing device 1350 is intended to represent various forms of mobile devices such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and limit embodiments of the invention described and / or claimed herein. Is not intended.

Several embodiments have been described. However, it will be appreciated that various modifications can be made without departing from the spirit and scope of this specification.

In addition, the logical flow shown in the figure does not require the specific order or sequential order shown to achieve the desired result. In addition, other steps may be provided, or the steps may be excluded from the described flow, other components may be added to the described system, or the systems, etc. Can be removed. Therefore, other embodiments are within the scope of the following claims.

Although some features of the embodiments described have been described herein, many modifications, substitutions, modifications, and equivalents will be recalled to those of skill in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and modifications contained within the scope of the embodiment. It should be understood that they are presented as examples, not limitations, and that various changes in form and detail can be made. Any part of the apparatus and / or method described herein can be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

Claims (51)

It ’s a method that runs on a computer.
The data comprises receiving data specifying a digital supplement, which identifies the digital supplement and a supplement anchor for associating the digital supplement with the image content, the method further comprising.
Creating a data structure instance that specifies the digital supplement and the supplement anchor,
After generating the data structure instance, the plurality of said data structure instances include enabling the triggering of digital capture by an image based on at least storing the data structure instance in a database containing a plurality of other data structure instances. A method in which each of the other data structure instances specifies a digital supplement and one or more supplement anchors. Receiving data that specifies the digital supplement is
Analyzing web pages and
The method performed on a computer according to claim 1, comprising identifying the metadata associated with the digital supplement represented within the web page. Receiving data that specifies the digital supplement is
The method performed on a computer according to claim 2, further comprising crawling a plurality of web pages including the web page before analyzing the web page. Receiving data that specifies the digital supplement is
The method performed on a computer according to claim 2 or 3, further comprising receiving an identifier of the web page before analyzing the web page. The method performed on a computer according to any one of claims 2-4, further comprising associating the data structure instance with the popularity score of the web page. The method performed on a computer according to any one of claims 2-5, further comprising associating the data structure instance with a relevance score of a supplementary anchor based on the web page. The method performed on the computer according to any of the preceding claims, wherein the digital supplement comprises a name, a description, an image, and a unified resource location specifier. The method performed on the computer according to any of the preceding claims, wherein the digital supplement comprises an identifier of the application. After enabling the digital supplement trigger
Receiving image content queries that include images and
Determining that the image matches the supplemental anchor specified by the generated data structure instance.
The preceding claim, further comprising providing digital supplements from the data structure instance in response to the image content query in response to determining that the image matches the supplement anchor. How it runs on your computer. The computer-implemented method of claim 9, wherein determining that the image matches the supplemental anchor comprises identifying an entity within the image. The data structure instance also specifies context information and providing the digital supplement from the data structure instance provides the data from the data structure instance based on determining that the context is satisfied. The method performed on the computer according to claim 9 or 10. Providing the digital supplement from the data structure instance in response to the image content query comprises providing a list of digital supplements, the list being the digital supplement from the data structure instance and the other. Performed on the computer according to any of the preceding claims, including digital supplements from one of the data structure instances and ordered based on one or more of the popularity and relevance scores. How to be done. It ’s a computing device,
With at least one processor
It comprises a memory for storing an instruction, and when the instruction is executed by at least one processor, the computing device receives the instruction.
The data specifying the digital supplement is received, and the data identifies the digital supplement, the supplement anchor for associating the digital supplement with the image content, and the context information.
Generate a data structure instance that specifies the digital supplement, the supplement anchor, and the context information.
After generating the data structure instance, enable the digital supplemental trigger with an image based on storing at least the data structure instance in a database containing a plurality of other data structure instances to enable the plurality of other data. Each of the structure instances is a computing device that specifies a digital supplement and one or more supplement anchors. 13. The computing device of claim 13, wherein the contextual information includes location. 13. The computing device of claim 13, wherein the context information includes an entity identified in the image. The computing device according to any one of claims 13 to 15, wherein the context information includes a plurality of entities identified in the image. It ’s a method that runs on a computer.
Receiving image content queries from computing devices and
Identifying supplemental anchors based on the image content query
To generate an ordered list of digital supplements based on the identified supplement anchors,
A method performed on a computer, including sending an ordered list to a computing device. The computer-executed method of claim 17, wherein the image content query comprises an image, and the digital supplement comprises a URL to a web page that presents information about an entity identified within the image. The computer-executed method of claim 17 or 18, wherein the digital supplement comprises a URL to a web page presenting a coupon code for the entity identified in the image. The method performed on a computer according to any one of claims 17 to 19, wherein the digital supplement comprises video content identified based on the image. The method performed on a computer according to any one of claims 17 to 20, wherein the digital supplement comprises a request to a server and parameters to the request. 21. The computer-executed method of claim 21, wherein the parameter initiates a workflow. The computer-executed method of any of claims 17-22, wherein the digital supplements in the list are ordered based on one or more of a popularity score and a relevance score. It ’s a computing device,
With at least one processor
With memory for storing instructions, the instructions, when executed by at least one processor, cause the computing device to perform the method according to any one of claims 1-12 or 17-23. , Computing device. A computer program product comprising an instruction, wherein when the instruction is executed by one or more processors, the method according to any one of claims 1-12 or 17-23 to the one or more processors. A computer program product that runs. The method of any of claims 1-12 or 17-23, which is a computer-readable storage medium containing instructions and, when executed by one or more processors, to the one or more processors. A computer-readable storage medium to run. Sending image content queries to server computing devices and
Receiving a response to an image content query that identifies a digital supplement and
A method performed on a computer, including displaying a user interface that includes information associated with the digital supplement. 28. The method performed on a computer according to claim 27, wherein sending an image content query comprises sending an image to the server computing device. 28. The method performed on a computer according to claim 27 or 28, wherein sending an image content query comprises sending an identifier of an entity in the image to said server computing device. The method performed on a computer according to any of claims 27-29, further comprising transmitting contextual information to the server computing device. The computer-executed method of claim 30, wherein the context information includes a location. The method performed on a computer according to any of claims 27-31, wherein receiving the response to the image content query comprises receiving a URL associated with the digital supplement. The method performed on a computer according to any of claims 27-32, wherein receiving the response to the image content query comprises receiving an identifier of the application associated with the digital supplement. The method performed on a computer according to any of claims 27-33, wherein receiving the response to the image content query comprises receiving a name, description, and image associated with the digital supplement. The method performed on a computer according to any of claims 27-34, wherein receiving a response to the image content query comprises receiving a list of digital supplements including said digital supplements. 35. The method performed on a computer according to claim 35, wherein the user interface comprises information about a plurality of digital supplements from the list of digital supplements. 36. The method performed on a computer according to claim 36, further comprising determining the order of the digital supplements with respect to the list based on whether or not an application associated with the plurality of digital supplements is installed. The method of the computer according to any of claims 27-37, further comprising acquiring an image, wherein the image content query is based on the image. It ’s a computing device,
With at least one processor
It comprises a memory that stores an instruction, and when the instruction is executed by the at least one processor, the computing device receives the instruction.
Get the image
An image content query based on the image is sent to the server computing device.
Receive a response to the image content query that identifies the digital supplement,
A computing device that displays a user interface that contains information associated with the digital supplement. 39. The computing of claim 39, wherein the instruction to cause the computing device to receive a response to the image content query comprises an instruction to cause the computing device to receive an ordered list of digital supplements including said digital supplement. device. 40. The computing device of claim 40, wherein the user interface comprises a digital supplement selection panel that includes a user-operable control associated with a plurality of digital supplements from the ordered list. 41. The computing device of claim 41, wherein the user-operable controls are ordered on the digital supplementary selection panel based on the order provided by the ordered list. The digital supplement is associated with a digital supplement server, and when the instruction is executed by at least one processor, the computing device is further injured.
Send the information to the digital supplement server
The computing device according to any one of claims 39 to 42, wherein the digital supplementary content is received from the digital supplementary server, and the digital supplementary content is based on the transmitted information. The computing device of claim 43, wherein the transmitted information includes contextual information. To get an image and
Sending an image content query based on the image to the search server and
Receiving a response to an image content query that identifies the digital supplement server,
Sending the image to the digital supplement server and
Receiving digital supplementary content from the digital supplementary server and
A method performed on a computer, including displaying the digital supplementary content. 45. The image content query includes the image, and causing the image to be transmitted to the digital supplement server comprises transmitting an instruction to provide the image to the digital supplement server to the search server. The method described on the computer described. The method performed on a computer according to claim 45 or 46, wherein transmitting the image to the digital supplement server comprises transmitting the image to the digital supplement server. The method performed on a computer according to any of claims 45-47, wherein the digital supplement comprises augmented reality content that is sized and positioned based on the image. It ’s a computing device,
With at least one processor
It comprises a memory storing instructions, and when the instructions are executed by the at least one processor, the computing device performs the method according to any of claims 27-38 or 45-48. Let the computing device. A computer program product comprising an instruction, wherein when the instruction is executed by one or more processors, the one or more processors are subjected to any of claims 27-38 or 45-48. A computer program product that lets you perform a method. A computer-readable storage medium containing an instruction that, when executed by one or more processors, causes the one or more processors to receive any of claims 27-38 or 45-48. A computer-readable storage medium that performs the method described in.

Images