US20230342388A1 - Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking - Google Patents
Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking Download PDFInfo
- Publication number
- US20230342388A1 US20230342388A1 US18/344,512 US202318344512A US2023342388A1 US 20230342388 A1 US20230342388 A1 US 20230342388A1 US 202318344512 A US202318344512 A US 202318344512A US 2023342388 A1 US2023342388 A1 US 2023342388A1
- Authority
- US
- United States
- Prior art keywords
- input
- user
- scene
- processor
- gesture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/532—Query formulation, e.g. graphical querying
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
- G06F16/9554—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL] by using bar codes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0179—Display position adjusting means not related to the information to be displayed
- G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
Definitions
- AR augmented reality
- HMD head mounted displays
- AR is a technology in which a user's view of the real world is enhanced with additional information generated from a computer model.
- the enhancements may include labels, 3D rendered models, or shading and illumination changes. AR allows a user to work with and examine the physical real world, while receiving additional information about the objects in it.
- AR devices typically include an optical see-through HMD and one or more user input mechanisms that allow users to simultaneously see and interact with their surroundings while interacting with applications, such as e-mail and media players.
- User input mechanisms may include one or more of gesture recognition technology, eye tracking technology, and other similar mechanisms.
- AR devices also referred to herein as HMD systems
- virtual objects displayed in the HMD screen augment real world scenes viewed by a user through the HMD.
- a person in the field of view of a user may be augmented with her name
- an artwork may be augmented with descriptive information
- a book may be augmented with its price and so on.
- a user can perform a visual search to obtain more information about a specific object visible in a real-world scene, i.e., within the field of view of the user.
- the HMD system generally takes a snapshot of the scene in the field of view, conducts a search using an online visual search engine (e.g. Google Goggles), retrieves the results, and displays the results.
- the results are displayed on the HMD screen to provide an augmented reality scene to the user that includes the real-world scene, with the object just searched visible through the HMD, and the augmented reality search result.
- a problem with this visual search approach is that other objects may be present in the real-world scene resulting in inaccurate search results and redundant or extraneous data transfer to the online visual search engine.
- a user may have to move closer to the object of interest so that the object of interest approximately covers her entire field of view.
- this may work for visual searching with a Smartphone, wherein a user zooms in on the object to be searched and takes a picture, this approach is not desirable for optical see-through HMD systems in that its requires the user to move and thus may not provide a good user experience.
- a method, an apparatus, and a computer program product for conducting online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD) are disclosed.
- An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD.
- the portion includes searchable content, such as a barcode.
- the user interaction may be an eye gaze or a gesture.
- a user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region.
- the apparatus captures an image of the portion.
- the identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view.
- the apparatus transmits the image to a visual search engine.
- FIG. 1 is a diagram illustrating an online visual search system for conducting online searches of objects visible through an AR device having an optical see-through HMD.
- FIG. 2 is an illustration of an AR device in the form of a pair of eyeglasses.
- FIG. 3 is an illustration of a real-world scene through an optical see-through HMD with augmented reality.
- FIG. 4 is a diagram illustrating elements of an AR device.
- FIG. 5 is an illustration of a real-world scene visible to a user through an optical see-through HMD, including an object of interest to be searched on line.
- FIG. 6 is an illustration of an augmented reality scene visible to a user that includes the real-world scene of FIG. 5 and the augmented reality search result.
- FIG. 7 is a flow chart of a method of conducting online searches of objects visible through an AR device having an optical see-through HMD.
- FIG. 8 is a diagram illustrating elements of an AR device that conducts online searches of objects visible through an optical see-through HMD.
- FIG. 9 is a diagram illustrating an example of a hardware implementation for an AR device employing a processing system.
- processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure.
- DSPs digital signal processors
- FPGAs field programmable gate arrays
- PLDs programmable logic devices
- state machines gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure.
- One or more processors in the processing system may execute software.
- Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
- the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium.
- Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer.
- such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
- Disk and disc includes CD, laser disc, optical disc, digital versatile disc (DVD), and floppy disk where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
- FIG. 1 is a diagram illustrating a visual online search system 100 for conducting online searches of objects visible through an optical see-through HMD with augmented reality, herein referred to as an AR device or an HMD system.
- the online search system 100 includes an AR device 102 , a communications network 104 , and a visual search engine 106 .
- the AR device 102 captures an image of an object seen through a HMD and transmits the image to the visual search engine 106 over the communications network 104 .
- the visual search engine 106 conducts an online search of the object captured in the image using known technology and sends the results to the AR device 102 over the communications network 104 .
- the AR device 102 displays the search results as an augmented reality in the HMD.
- FIG. 2 is an illustration of an example AR device 200 in the form of a pair of eyeglasses.
- the AR device 200 is configured such that the user of the device is able to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs, including both two-dimensional (2D) and three-dimensional (3D) AR content.
- the AR device 200 may also be configured to allow the user to interact with the content and possibly with remote devices, systems or networks through wireless communication.
- the AR device may also provide feedback to the user as a result of such interactions, including for example, audio, video or tactile feedback.
- the example AR device 200 includes a pair of optical see-through HMDs 202 , 204 , an on-board processing system 206 , one or more sensors, such as a scene camera 208 , one or more eye tracking components (not visible) for each of the right eye and left eye, one or more user-interaction feedback devices 210 and a transceiver 212 .
- the processing system 206 and the eye tracking components provide eye tracking capability.
- eye tracking components may include one or both of eye cameras and infra-red emitters, e.g. diodes.
- the processing system 206 and the scene camera 208 provide gesture tracking capability.
- the feedback devices 210 provide perception feedback to the user in response to certain interactions with the AR device.
- Feedback devices 210 may include a speaker or a vibration device. Perception feedback may also be provided by visual indication through the HMD.
- the transceiver 212 facilitates wireless communication between the processing system 206 and remote devices, systems or networks.
- the AR device may communicate with remote servers through the transceiver 212 for purposes of remote processing, such as on-line searches through remote search engines.
- the AR device 200 allows a user to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs.
- the scene camera 208 may capture an image of the scene and send the image to the on-board processing system 206 .
- the processing system 206 may process the image and output AR content 302 for display on the HMDs 202 , 204 .
- the content 302 may provide information describing what the user is seeing.
- the processing system 206 may transmit the image through the transceiver 212 to a remote processor (not shown) for processing.
- the processing system 206 may also display one or more application icons 304 , 306 , 308 on the HMDs 202 , 204 and output application content, such as e-mails, documents, web pages, or media content such as video games, movies or electronic books, in response to user interaction with the icons.
- application content such as e-mails, documents, web pages, or media content such as video games, movies or electronic books
- User interaction with the AR device 200 is provided by one or more user input mechanisms, such as a gesture tracking module or an eye-gaze tracking module.
- Gesture tracking is provided by the scene camera 208 in conjunction with a gesture tracking module of the processing system 206 .
- a user may attempt to activate an application by placing his finger on an application icon 304 , 306 , 308 in the field of view of the AR device.
- the scene camera 208 captures an image of the finger and sends the image to the gesture tracking module.
- the gesture tracking module processes the image and determines coordinates of a gesture point corresponding to where the user is pointing.
- the processing system 206 compares the coordinate location of the gesture point to the coordinate location of the icon on the display.
- the processing system 206 determines that the user has selected the icon 304 , 306 , 308 and accordingly, launches the application. Often, such eye-gaze based launching is coupled with another form of input, e.g., gesture, to confirm the user's intention of launching the application.
- another form of input e.g., gesture
- Eye-gaze tracking is provided by the eye tracking components (not visible) in conjunction with an eye tracking module of the processing system 206 .
- a user may attempt to activate an application by gazing at an application icon 304 , 306 , 308 in the field of view of the AR device.
- the eye tracking components capture images of the eyes, and provide the images to the eye tracking module.
- the eye tracking module processes the images and determines coordinates of an eye-gaze point corresponding to where the user is looking.
- the processing system 206 compares the coordinate location of the eye-gaze point to the coordinate location of the icon on the display. If the locations match, or are within a threshold distance of each other, the processing system 206 determines that the user has selected the icon 304 , 306 , 308 and accordingly, launches the application.
- FIG. 4 is a diagram illustrating elements of an example AR device 400 with optical see-through HMDs 402 .
- the AR device 400 may include one or more sensing devices, such as infrared (IR) diodes 404 facing toward the wearer of the AR device and eye cameras 406 facing toward the wearer.
- IR infrared
- a scene camera 408 facing away from the wearer captures images of the field of view seen by the user through the HMD 402 .
- the cameras 406 , 408 may be video cameras. While only one IR diode 404 and one eye camera 406 are illustrated, the AR device 400 typically includes several diodes and cameras for each of the left eye and right eye. A single scene camera 408 is usually sufficient. For ease of illustration only one of each sensor type is shown in FIG. 4 .
- the AR device 400 includes an on-board processing system 410 , which in turn includes one or more of an eye tracking module 412 and a gesture tracking module 414 .
- An object selection module 416 processes the outputs of the one or more tracking modules to determine user interactions and tracking module accuracy.
- a tracking calibration module 418 calibrates the one or more tracking modules if the tracking module is determined to be inaccurate.
- the on-board processing system 410 may also include a scene camera calibration module 420 , a graphical user interface (GUI) adjustment module 422 , a perception feedback module 424 , and a search module 436 .
- the scene camera calibration module 420 calibrates the AR device so that the AR content is aligned with real world objects.
- the GUI adjustment module 422 may adjust the parameters of GUI objects displayed on the HMD to compensate for eye-tracking or gesture-tracking inaccuracies detected by the object selection module 416 . Such adjustments may precede, supplement, or substitute for the actions of the tracking calibration module 418 .
- the feedback module 424 controls one or more feedback devices 426 to provide perception feedback to the user in response to one or more types of user interactions.
- the feedback module may command a feedback device 426 to output sound when a user selects an icon in the field of view using a gesture or eye gaze.
- the search module 436 identifies and captures a portion of a real world object seen through the optical see-through HMD for purposes of conducting online searches. The portion to be captured is identified by the search module 436 based on real world scene data provided by the scene camera 408 and one of gaze coordinates provided by the eye tracking module 412 , and gesture coordinates provided by the gesture tracking module 414 .
- the AR device 400 further includes memory 428 for storing program code to implement the foregoing features of the on-board processing system 410 .
- a communications module 430 and transceiver 432 facilitate wireless communications with remote devices, systems and networks. For example, in one implementation, an image of a real-world object may be captured by the search module 436 and transmitted by the communications module 430 and the transceiver 432 to a remote search engine, with subsequent search results being received by the transceiver.
- the diodes 404 and eye cameras 406 together with the eye tracking module 412 , provide eye tracking capability as generally described above.
- the eye tracking capability is based on known infrared technology.
- One such known technology uses infrared light emitting diodes and infrared sensitive video camera for remotely recording images of the eye.
- Infrared light output by the diode 404 enters the eye and is absorbed and re-emitted by the retina, thereby causing a “bright eye effect” that makes the pupil brighter than the rest of the eye.
- the infrared light also gives rise to an even brighter small glint that is formed on the surface of the cornea.
- the eye tracking module 412 acquires a video image of the eye from the eye camera 406 , digitizes it into a matrix of pixels, and then analyzes the matrix to identify the location of the pupil's center relative to the glint's center, as well as a vector between these centers. Based on the determined vector, the eye tracking module 412 outputs eye gaze coordinates defining an eye gaze point (E).
- the scene camera 408 together with the gesture tracking module 414 , provide gesture tracking capability using a known technology as generally described above.
- the gesture tracking capability is based on gesture images captured by the scene camera 408 .
- the gesture images are processed by the gesture tracking module 414 by comparing captured images to a catalog of images to determine if there is a match.
- the user may be pointing at an icon in the field of view.
- the gesture tracking module 412 may detect a match between the gesture image and a cataloged image of pointing and thereby recognize the gesture as pointing.
- the gesture tracking module 414 processes the captured image further to determine the coordinates of a relevant part of the gesture image. In the case of finger pointing, the relevant part of the image may correspond to the tip of the finger.
- the gesture tracking module 414 outputs gesture coordinates defining a gesture point (G).
- the object selection processor 416 functions to determine whether interactions of the user, as characterized by one or more of the eye tracking module 412 and the gesture tracking module 414 , correspond to a selection of an object, e.g., application icon, displayed on the HMD 402 and visible in the field of view. If an interaction does correspond to a selection by the user, for example, a selection of an icon to launch an application 434 , the object selection processor 416 outputs a command to the application.
- an object e.g., application icon
- the AR device 410 may provide for online searching related to objects seen through the optical see-through HMD.
- Components of the AR device that facilitate such searching include the scene camera 408 , one or more of the user interaction tracking modules, e.g., the eye tracking module 412 and the gesture tracking module 414 , the object portion identification module, the search module 436 , and/or communication module 430 .
- FIG. 5 is an illustration 500 of a real-world scene 502 visible to a user through an optical see-through HMD, including an object 504 of interest to be searched on line.
- FIG. 6 is an illustration of an augmented reality scene visible to a user that includes the real-world scene of FIG. 5 and the augmented reality search result.
- the real-world scene 502 includes an object of interest 504 to be searched on line.
- the object 504 is a book.
- the eye tracking module 412 is configured to track movement of the user's eye and to detect when the user is gazing at a portion (e.g., portion 508 ) of the object 504 . In this case, the user is determined to be gazing at a point 506 overlaying the portion 508 of the object (e.g., a book) where the barcode is located.
- the search module 436 receives data from the scene camera 408 corresponding to a captured scene (herein referred to as “scene data”) and data from the eye tracking module 412 corresponding to the eye gaze of the user (herein referred to as “gaze coordinates” or “eye gaze data”).
- the search module 436 processes the scene data and the eye gaze data to identify the portion 508 of the object 504 and to provide an image of the portion.
- the search module 436 determines the portion 508 of the object, and then determines a boundary 510 around the portion.
- the image of the portion corresponds to the scene that lies within the boundary 510 . Accordingly, the size of the image is less than the size of the object 504 as well as the scene data 502 in the field of view.
- Data corresponding to the image of the portion is provided to the communications module 430 , which in turn, transmits the image data to the online visual search engine 106 over the communications network 104 .
- the communications module 430 may be, for example, a Bluetooth device. Search results are transmitted back to the AR device, received by the communications module 430 and provided to the search module 436 for display. With reference to FIG. 6 , search results 604 are displayed on the HMD screen in the vicinity of the object 504 .
- FIG. 7 is a flowchart of a method of conducting an online visual search through an AR device having an optical see-through HMD.
- the AR device identifies a portion 508 of an object 504 in a field of view of the HMD based on user interaction with the HMD.
- the user interaction may be an eye gaze 506 , in which case the AR device identifies a portion 508 of an object by tracking the eye gaze 506 of the user to determine a location of the eye gaze, locating a region 512 of the object corresponding to the location of the eye gaze, and detecting the portion 508 within the region.
- the user interaction may be a gesture
- the AR device identifies a portion of an object by tracking the gesture of the user to determine a location of the gesture, locating a region of the object corresponding to the location of the gesture, and detecting the portion within the region.
- the AR device may display an augmented reality 510 on the HMD in the vicinity of the detected portion 508 .
- the augmented reality 510 may be a boundary around the portion 508 .
- the AR device may receive an indication to initiate an online visual search of the object 504 using content contained the portion 508 .
- the user may request that the AR device initiate an online search of the object based on the portion.
- Such a request may be input by the user through a physical search activation button on the AR device or through user interaction with the HMD screen.
- the AR device may display a search icon on the HMD screen and the search may be initiated through user interaction, e.g., eye gaze or gesture, with the search icon.
- the AR device captures an image of the portion 508 .
- the image may correspond to the area of the object 504 within the boundary 510 . Accordingly, the size of the image is less than the size of the object in the field of view.
- the AR device transmits the image to a search engine.
- the AR device receives search results from the search engine, and at step 714 , displays the search results as an augmented reality image 604 on the HMD screen in the vicinity of the object 504 .
- FIG. 8 is a diagram 800 illustrating elements of an AR device 802 that conducts online visual searches.
- the AR device 802 includes a scene camera module 804 , a HMD module 806 , an identification/capture module 808 , a user interaction module 810 and a communications module 812 .
- These modules may be referred to by different names, as indicated parenthetically, and may correspond to one or more of the modules of FIG. 4 .
- the user interaction module 810 may be the eye tracking module 412 or the gesture tracking module 414
- the identification/capture module 808 may be the search module 436 .
- the identification/capture module 808 identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD.
- the scene camera module 804 provides scene data to the identification/capturing module 808
- the user interaction module 810 provides user interaction data to the identification/capture module 808 .
- the scene data corresponds to image data of the real world scene visible in the field of view of the HMD module 806 and captured by the camera 804 .
- the user interaction data may be eye gaze data or gesture data. In either case, the user interaction data includes coordinate data that defines a user interaction point, eye-gaze point (E), in terms of the coordinate system corresponding to the HMD screen.
- E eye-gaze point
- the identification/capture module 808 identifies a portion 508 of an object 504 by tracking the eye gaze of the user to determine a location of the eye gaze, locating a region 512 of the object corresponding to the location of the eye gaze, and detecting the portion 508 within the region.
- the identification/capture module 808 identifies a portion of an object by track the gesture of the user to determine a location of the gesture, locating a region of the object corresponding to the location of the gesture, and detecting the portion within the region.
- the identification/capture module 808 Upon detecting the portion 508 , the identification/capture module 808 captures an image of the portion. To this end, the identification/capture module 808 may determine a boundary around the portion and display the boundary as an augmented reality on the HMD. In one configuration, the image of the portion includes only image data within the boundary 510 . This is beneficial in that it reduces the amount of image data to be transmitted to the search engine and also provides a more focused search by reducing the amount of extraneous searchable content from the search request.
- the communication module 812 transmits the image to a search engine. Subsequently, the communication module 812 receives data corresponding to search results from the search engine.
- the search results may be provided to the identification/capturing module 808 and displayed as an augmented reality image on the HMD screen in the vicinity of the object.
- the AR devices may include additional modules that perform each of the steps of the algorithm in the aforementioned flow chart of FIG. 7 .
- each step in the aforementioned flow chart of FIG. 7 may be performed by a module and the apparatus may include one or more of those modules.
- the modules may be one or more hardware components specifically configured to carry out the stated processes/algorithm, implemented by a processor configured to perform the stated processes/algorithm, stored within a computer-readable medium for implementation by a processor, or some combination thereof.
- FIG. 9 is a diagram 900 illustrating an example of a hardware implementation for an apparatus 802 ′ employing a processing system 914 .
- the processing system 914 may be implemented with a bus architecture, represented generally by the bus 924 .
- the bus 924 may include any number of interconnecting buses and bridges depending on the specific application of the processing system 914 and the overall design constraints.
- the bus 924 links together various circuits including one or more processors and/or hardware modules, represented by the processor 904 , the modules 804 , 806 , 808 , 810 , 812 and the computer-readable medium/memory 906 .
- the bus 924 may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.
- the processing system 914 includes a processor 904 coupled to a computer-readable medium/memory 906 .
- the processor 904 is responsible for general processing, including the execution of software stored on the computer-readable medium/memory 906 .
- the software when executed by the processor 904 , causes the processing system 914 to perform the various functions described supra for any particular apparatus.
- the computer-readable medium/memory 906 may also be used for storing data that is manipulated by the processor 904 when executing software.
- the processing system further includes at least one of the modules 804 , 806 , 808 , 810 and 812 .
- the modules may be software modules running in the processor 904 , resident/stored in the computer readable medium/memory 906 , one or more hardware modules coupled to the processor 904 , or some combination thereof.
- the apparatus 802 / 802 ′ includes means for identifying a portion of an object in a field of view of the HMD based on user interaction with the HMD, means for capturing an image of the portion, wherein the size of the image is less than the size of the object in the field of view, and means for transmitting the image to a search engine.
- the apparatus 802 / 802 ′ also includes means for displaying an augmented reality on the HMD in the vicinity of the portion, means for receiving an indication to initiate an online visual search of the object using the portion, means for receiving search results from the search engine; and means for displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- the aforementioned means may be one or more of the aforementioned modules of the apparatus 802 and/or the processing system 914 of the apparatus 802 ′ configured to perform the functions recited by the aforementioned means.
- a visual search using a HMD with augmented reality and eye tracking works as follows: A user looks at the object of interest in a scene visible through the HMD and initiates a visual search via other input techniques such as a gesture.
- An eye tracking system included in the HMD system determines a location of a gaze point corresponding to the point of the scene at which the user is gazing.
- the eye tracking system identifies the object of interest based on the gaze point. For example, with reference to FIG. 5 , based on the location of the eye gaze, the eye tracking system identifies the object of interest as the bar code. Such identification may be made by color recognition, wherein the system detects a change in color between the edge of the bar code and the back cover of the book.
- the eye tracking system determines the object of interest, it identifies an area of the scene containing the object and highlights the area with an augmented reality, such as an augmented reality boundary surrounding the object.
- the selected part of the scene corresponding to the area within the boundary is extracted and sent to the cloud (online visual search engine e.g. Google Goggles) for further analysis.
- the cloud online visual search engine e.g. Google Goggles
- the user's view is augmented with the retrieved search result.
- the benefits of this approach are: The user is not required to move to the object of interest. Only relevant part of the image is sent to the cloud; thus saves redundant data transfer. Minimum workload on the search engine since it does not need to analyze all objects presented in the user's scene, only a subset which contains object of interest.
- a method, an apparatus, and a computer program product provide for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user.
- a method includes tracking an eye gaze of the user to locate the object, capturing an image of the object, and transmitting the image of the object to a search engine. The size of the image is less than the size of the scene.
- the method may also include receiving an indication to initiate an online visual search of an object.
- the method may further include receiving search results from the search engine, and displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- a corresponding apparatus for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user includes means for tracking an eye gaze of the user to locate the object, means for capturing an image of the object, and means for transmitting the image of the object to a search engine.
- the size of the image is less than the size of the scene.
- the apparatus may also include means for receiving an indication to initiate an online visual search of an object.
- the apparatus may further include means for receiving search results from the search engine, and means for displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- Another corresponding apparatus for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user includes a memory, and at least one processor coupled to the memory and configured to track an eye gaze of the user to locate the object, capture an image of the object, and transmit the image of the object to a search engine.
- the size of the image is less than the size of the scene.
- the processor may be configured to receive an indication to initiate an online visual search of an object.
- the processor may be further configured to receive search results from the search engine, and display the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- a computer program product for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user includes a computer-readable medium comprising code for tracking an eye gaze of the user to locate the object, for capturing an image of the object, and for transmitting the image of the object to a search engine.
- the size of the image is less than the size of the scene.
- the product may include code for receiving an indication to initiate an online visual search of an object.
- the product may further include code for receiving search results from the search engine, and displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- Combinations such as “at least one of A, B, or C,” “at least one of A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and/or C, and may include multiples of A, multiples of B, or multiples of C.
- combinations such as “at least one of A, B, or C,” “at least one of A, B, and C,” and “A, B, C, or any combination thereof” may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Mathematical Physics (AREA)
- Library & Information Science (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Optics & Photonics (AREA)
- Computational Linguistics (AREA)
- User Interface Of Digital Computer (AREA)
- Position Input By Displaying (AREA)
Abstract
Techniques can conduct online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD). An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. The portion includes searchable content, such as a barcode. The user interaction may be an eye gaze or a gesture. A user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region. The apparatus captures an image of the portion. The identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view. The apparatus transmits the image to a visual search engine.
Description
- This application claims the benefit of U.S. Provisional Application Ser. No. 61/867,579, entitled “Improving Visual Search in Real World Using Optical See-Through HMD with Augmented Reality and Eye Tracking” and filed on Aug. 19, 2013, which is expressly incorporated by reference herein in its entirety and for all purposes.
- This application is a continuation of U.S. patent application Ser. No. 17/304,739, filed Jun. 24, 2021, which is a continuation of U.S. patent application Ser. No. 16/414,667, filed May 16, 2019, which is a continuation of U.S. patent application Ser. No. 16/181,175, filed Nov. 5, 2018, which is a continuation of U.S. patent application Ser. No. 14/151,664, filed Jan. 9, 2014, all of which are expressly incorporated by reference in their entirety and for all purposes.
- The present disclosure relates generally to augmented reality (AR) devices, e.g., AR eyeglasses, having optical see-through head mounted displays (HMD) and user interaction capability, and more particularly, to conducting online visual searches in real world using such AR devices. AR is a technology in which a user's view of the real world is enhanced with additional information generated from a computer model. The enhancements may include labels, 3D rendered models, or shading and illumination changes. AR allows a user to work with and examine the physical real world, while receiving additional information about the objects in it.
- AR devices typically include an optical see-through HMD and one or more user input mechanisms that allow users to simultaneously see and interact with their surroundings while interacting with applications, such as e-mail and media players. User input mechanisms may include one or more of gesture recognition technology, eye tracking technology, and other similar mechanisms.
- In AR devices, also referred to herein as HMD systems, virtual objects displayed in the HMD screen augment real world scenes viewed by a user through the HMD. For example, a person in the field of view of a user may be augmented with her name, an artwork may be augmented with descriptive information, and a book may be augmented with its price and so on.
- With such an HMD system, a user can perform a visual search to obtain more information about a specific object visible in a real-world scene, i.e., within the field of view of the user. During the visual search, the HMD system generally takes a snapshot of the scene in the field of view, conducts a search using an online visual search engine (e.g. Google Goggles), retrieves the results, and displays the results. The results are displayed on the HMD screen to provide an augmented reality scene to the user that includes the real-world scene, with the object just searched visible through the HMD, and the augmented reality search result.
- A problem with this visual search approach is that other objects may be present in the real-world scene resulting in inaccurate search results and redundant or extraneous data transfer to the online visual search engine. To work around this issue, a user may have to move closer to the object of interest so that the object of interest approximately covers her entire field of view. Although this may work for visual searching with a Smartphone, wherein a user zooms in on the object to be searched and takes a picture, this approach is not desirable for optical see-through HMD systems in that its requires the user to move and thus may not provide a good user experience.
- In an aspect of the disclosure, a method, an apparatus, and a computer program product for conducting online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD) are disclosed. An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. The portion includes searchable content, such as a barcode. The user interaction may be an eye gaze or a gesture. A user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region. The apparatus captures an image of the portion. The identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view. The apparatus transmits the image to a visual search engine.
-
FIG. 1 is a diagram illustrating an online visual search system for conducting online searches of objects visible through an AR device having an optical see-through HMD. -
FIG. 2 is an illustration of an AR device in the form of a pair of eyeglasses. -
FIG. 3 is an illustration of a real-world scene through an optical see-through HMD with augmented reality. -
FIG. 4 is a diagram illustrating elements of an AR device. -
FIG. 5 is an illustration of a real-world scene visible to a user through an optical see-through HMD, including an object of interest to be searched on line. -
FIG. 6 is an illustration of an augmented reality scene visible to a user that includes the real-world scene ofFIG. 5 and the augmented reality search result. -
FIG. 7 is a flow chart of a method of conducting online searches of objects visible through an AR device having an optical see-through HMD. -
FIG. 8 is a diagram illustrating elements of an AR device that conducts online searches of objects visible through an optical see-through HMD. -
FIG. 9 is a diagram illustrating an example of a hardware implementation for an AR device employing a processing system. - The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well known structures and components are shown in block diagram form in order to avoid obscuring such concepts.
- Several aspects of visual searching through an AR device having an optical see-through HMD and user interaction tracking will now be presented with reference to various apparatus and methods. These apparatus and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). These elements may be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.
- By way of example, an element, or any portion of an element, or any combination of elements may be implemented with a “processing system” that includes one or more processors. Examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. One or more processors in the processing system may execute software. Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.
- Accordingly, in one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes CD, laser disc, optical disc, digital versatile disc (DVD), and floppy disk where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
-
FIG. 1 is a diagram illustrating a visualonline search system 100 for conducting online searches of objects visible through an optical see-through HMD with augmented reality, herein referred to as an AR device or an HMD system. Theonline search system 100 includes anAR device 102, acommunications network 104, and avisual search engine 106. TheAR device 102 captures an image of an object seen through a HMD and transmits the image to thevisual search engine 106 over thecommunications network 104. Thevisual search engine 106 conducts an online search of the object captured in the image using known technology and sends the results to theAR device 102 over thecommunications network 104. TheAR device 102 displays the search results as an augmented reality in the HMD. -
FIG. 2 is an illustration of anexample AR device 200 in the form of a pair of eyeglasses. TheAR device 200 is configured such that the user of the device is able to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs, including both two-dimensional (2D) and three-dimensional (3D) AR content. TheAR device 200 may also be configured to allow the user to interact with the content and possibly with remote devices, systems or networks through wireless communication. The AR device may also provide feedback to the user as a result of such interactions, including for example, audio, video or tactile feedback. To these ends, theexample AR device 200 includes a pair of optical see-throughHMDs board processing system 206, one or more sensors, such as ascene camera 208, one or more eye tracking components (not visible) for each of the right eye and left eye, one or more user-interaction feedback devices 210 and atransceiver 212. - The
processing system 206 and the eye tracking components provide eye tracking capability. Depending on the eye tracking technology being employed, eye tracking components may include one or both of eye cameras and infra-red emitters, e.g. diodes. Theprocessing system 206 and thescene camera 208 provide gesture tracking capability. - The
feedback devices 210 provide perception feedback to the user in response to certain interactions with the AR device.Feedback devices 210 may include a speaker or a vibration device. Perception feedback may also be provided by visual indication through the HMD. - The
transceiver 212 facilitates wireless communication between theprocessing system 206 and remote devices, systems or networks. For example, the AR device may communicate with remote servers through thetransceiver 212 for purposes of remote processing, such as on-line searches through remote search engines. - As mentioned above, the
AR device 200 allows a user to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs. For example, with reference toFIG. 3 , as a user is viewing a real-world scene 300 through the optical see-throughHMDs scene camera 208 may capture an image of the scene and send the image to the on-board processing system 206. Theprocessing system 206 may process the image andoutput AR content 302 for display on theHMDs content 302 may provide information describing what the user is seeing. In some cases, theprocessing system 206 may transmit the image through thetransceiver 212 to a remote processor (not shown) for processing. Theprocessing system 206 may also display one ormore application icons HMDs - User interaction with the
AR device 200 is provided by one or more user input mechanisms, such as a gesture tracking module or an eye-gaze tracking module. Gesture tracking is provided by thescene camera 208 in conjunction with a gesture tracking module of theprocessing system 206. With gesture tracking, a user may attempt to activate an application by placing his finger on anapplication icon scene camera 208 captures an image of the finger and sends the image to the gesture tracking module. The gesture tracking module processes the image and determines coordinates of a gesture point corresponding to where the user is pointing. Theprocessing system 206 compares the coordinate location of the gesture point to the coordinate location of the icon on the display. If the locations match, or are within a threshold distance of each other, theprocessing system 206 determines that the user has selected theicon - Eye-gaze tracking is provided by the eye tracking components (not visible) in conjunction with an eye tracking module of the
processing system 206. A user may attempt to activate an application by gazing at anapplication icon processing system 206 compares the coordinate location of the eye-gaze point to the coordinate location of the icon on the display. If the locations match, or are within a threshold distance of each other, theprocessing system 206 determines that the user has selected theicon -
FIG. 4 is a diagram illustrating elements of anexample AR device 400 with optical see-throughHMDs 402. TheAR device 400 may include one or more sensing devices, such as infrared (IR)diodes 404 facing toward the wearer of the AR device andeye cameras 406 facing toward the wearer. Ascene camera 408 facing away from the wearer captures images of the field of view seen by the user through theHMD 402. Thecameras IR diode 404 and oneeye camera 406 are illustrated, theAR device 400 typically includes several diodes and cameras for each of the left eye and right eye. Asingle scene camera 408 is usually sufficient. For ease of illustration only one of each sensor type is shown inFIG. 4 . - The
AR device 400 includes an on-board processing system 410, which in turn includes one or more of aneye tracking module 412 and agesture tracking module 414. Anobject selection module 416 processes the outputs of the one or more tracking modules to determine user interactions and tracking module accuracy. A trackingcalibration module 418 calibrates the one or more tracking modules if the tracking module is determined to be inaccurate. - The on-
board processing system 410 may also include a scenecamera calibration module 420, a graphical user interface (GUI)adjustment module 422, aperception feedback module 424, and asearch module 436. The scenecamera calibration module 420 calibrates the AR device so that the AR content is aligned with real world objects. TheGUI adjustment module 422 may adjust the parameters of GUI objects displayed on the HMD to compensate for eye-tracking or gesture-tracking inaccuracies detected by theobject selection module 416. Such adjustments may precede, supplement, or substitute for the actions of the trackingcalibration module 418. Thefeedback module 424 controls one ormore feedback devices 426 to provide perception feedback to the user in response to one or more types of user interactions. For example, the feedback module may command afeedback device 426 to output sound when a user selects an icon in the field of view using a gesture or eye gaze. Thesearch module 436 identifies and captures a portion of a real world object seen through the optical see-through HMD for purposes of conducting online searches. The portion to be captured is identified by thesearch module 436 based on real world scene data provided by thescene camera 408 and one of gaze coordinates provided by theeye tracking module 412, and gesture coordinates provided by thegesture tracking module 414. - The
AR device 400 further includesmemory 428 for storing program code to implement the foregoing features of the on-board processing system 410. Acommunications module 430 andtransceiver 432 facilitate wireless communications with remote devices, systems and networks. For example, in one implementation, an image of a real-world object may be captured by thesearch module 436 and transmitted by thecommunications module 430 and thetransceiver 432 to a remote search engine, with subsequent search results being received by the transceiver. - With further respect to eye tracking capability, the
diodes 404 andeye cameras 406, together with theeye tracking module 412, provide eye tracking capability as generally described above. In the example implementation ofFIG. 4 , the eye tracking capability is based on known infrared technology. One such known technology uses infrared light emitting diodes and infrared sensitive video camera for remotely recording images of the eye. Infrared light output by thediode 404 enters the eye and is absorbed and re-emitted by the retina, thereby causing a “bright eye effect” that makes the pupil brighter than the rest of the eye. The infrared light also gives rise to an even brighter small glint that is formed on the surface of the cornea. Theeye tracking module 412 acquires a video image of the eye from theeye camera 406, digitizes it into a matrix of pixels, and then analyzes the matrix to identify the location of the pupil's center relative to the glint's center, as well as a vector between these centers. Based on the determined vector, theeye tracking module 412 outputs eye gaze coordinates defining an eye gaze point (E). - The
scene camera 408, together with thegesture tracking module 414, provide gesture tracking capability using a known technology as generally described above. In the example implementation ofFIG. 4 , the gesture tracking capability is based on gesture images captured by thescene camera 408. The gesture images are processed by thegesture tracking module 414 by comparing captured images to a catalog of images to determine if there is a match. For example, the user may be pointing at an icon in the field of view. Thegesture tracking module 412 may detect a match between the gesture image and a cataloged image of pointing and thereby recognize the gesture as pointing. Upon detection of a recognized gesture, thegesture tracking module 414 processes the captured image further to determine the coordinates of a relevant part of the gesture image. In the case of finger pointing, the relevant part of the image may correspond to the tip of the finger. Thegesture tracking module 414 outputs gesture coordinates defining a gesture point (G). - The
object selection processor 416 functions to determine whether interactions of the user, as characterized by one or more of theeye tracking module 412 and thegesture tracking module 414, correspond to a selection of an object, e.g., application icon, displayed on theHMD 402 and visible in the field of view. If an interaction does correspond to a selection by the user, for example, a selection of an icon to launch anapplication 434, theobject selection processor 416 outputs a command to the application. - As previously mentioned, the
AR device 410 may provide for online searching related to objects seen through the optical see-through HMD. Components of the AR device that facilitate such searching include thescene camera 408, one or more of the user interaction tracking modules, e.g., theeye tracking module 412 and thegesture tracking module 414, the object portion identification module, thesearch module 436, and/orcommunication module 430. -
FIG. 5 is anillustration 500 of a real-world scene 502 visible to a user through an optical see-through HMD, including anobject 504 of interest to be searched on line.FIG. 6 is an illustration of an augmented reality scene visible to a user that includes the real-world scene ofFIG. 5 and the augmented reality search result. InFIG. 5 , the real-world scene 502 includes an object ofinterest 504 to be searched on line. In the example ofFIG. 5 , theobject 504 is a book. Theeye tracking module 412 is configured to track movement of the user's eye and to detect when the user is gazing at a portion (e.g., portion 508) of theobject 504. In this case, the user is determined to be gazing at apoint 506 overlaying theportion 508 of the object (e.g., a book) where the barcode is located. - The
search module 436 receives data from thescene camera 408 corresponding to a captured scene (herein referred to as “scene data”) and data from theeye tracking module 412 corresponding to the eye gaze of the user (herein referred to as “gaze coordinates” or “eye gaze data”). Thesearch module 436 processes the scene data and the eye gaze data to identify theportion 508 of theobject 504 and to provide an image of the portion. With reference toFIG. 5 , thesearch module 436 determines theportion 508 of the object, and then determines aboundary 510 around the portion. The image of the portion corresponds to the scene that lies within theboundary 510. Accordingly, the size of the image is less than the size of theobject 504 as well as thescene data 502 in the field of view. - Data corresponding to the image of the portion is provided to the
communications module 430, which in turn, transmits the image data to the onlinevisual search engine 106 over thecommunications network 104. Thecommunications module 430 may be, for example, a Bluetooth device. Search results are transmitted back to the AR device, received by thecommunications module 430 and provided to thesearch module 436 for display. With reference toFIG. 6 , search results 604 are displayed on the HMD screen in the vicinity of theobject 504. -
FIG. 7 is a flowchart of a method of conducting an online visual search through an AR device having an optical see-through HMD. Atstep 702, the AR device identifies aportion 508 of anobject 504 in a field of view of the HMD based on user interaction with the HMD. In one configuration, the user interaction may be aneye gaze 506, in which case the AR device identifies aportion 508 of an object by tracking theeye gaze 506 of the user to determine a location of the eye gaze, locating aregion 512 of the object corresponding to the location of the eye gaze, and detecting theportion 508 within the region. In another configuration, the user interaction may be a gesture, in which case the AR device identifies a portion of an object by tracking the gesture of the user to determine a location of the gesture, locating a region of the object corresponding to the location of the gesture, and detecting the portion within the region. - At
step 704, the AR device may display anaugmented reality 510 on the HMD in the vicinity of the detectedportion 508. Theaugmented reality 510 may be a boundary around theportion 508. - At
step 706, the AR device may receive an indication to initiate an online visual search of theobject 504 using content contained theportion 508. For example, upon seeing theaugmented reality 510 that identifies theportion 508 of the object, the user may request that the AR device initiate an online search of the object based on the portion. Such a request may be input by the user through a physical search activation button on the AR device or through user interaction with the HMD screen. In the latter case, the AR device may display a search icon on the HMD screen and the search may be initiated through user interaction, e.g., eye gaze or gesture, with the search icon. - At
step 708, the AR device captures an image of theportion 508. The image may correspond to the area of theobject 504 within theboundary 510. Accordingly, the size of the image is less than the size of the object in the field of view. Atstep 710, the AR device transmits the image to a search engine. - At
step 712, the AR device receives search results from the search engine, and atstep 714, displays the search results as anaugmented reality image 604 on the HMD screen in the vicinity of theobject 504. -
FIG. 8 is a diagram 800 illustrating elements of anAR device 802 that conducts online visual searches. TheAR device 802 includes ascene camera module 804, aHMD module 806, an identification/capture module 808, auser interaction module 810 and acommunications module 812. These modules may be referred to by different names, as indicated parenthetically, and may correspond to one or more of the modules ofFIG. 4 . For example, theuser interaction module 810 may be theeye tracking module 412 or thegesture tracking module 414, and the identification/capture module 808 may be thesearch module 436. - The identification/
capture module 808 identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. To this end, thescene camera module 804 provides scene data to the identification/capturing module 808, and theuser interaction module 810 provides user interaction data to the identification/capture module 808. The scene data corresponds to image data of the real world scene visible in the field of view of theHMD module 806 and captured by thecamera 804. The user interaction data may be eye gaze data or gesture data. In either case, the user interaction data includes coordinate data that defines a user interaction point, eye-gaze point (E), in terms of the coordinate system corresponding to the HMD screen. - In the case where the user interaction is an eye gaze, the identification/
capture module 808 identifies aportion 508 of anobject 504 by tracking the eye gaze of the user to determine a location of the eye gaze, locating aregion 512 of the object corresponding to the location of the eye gaze, and detecting theportion 508 within the region. In the case where the user interaction is a gesture, the identification/capture module 808 identifies a portion of an object by track the gesture of the user to determine a location of the gesture, locating a region of the object corresponding to the location of the gesture, and detecting the portion within the region. - Upon detecting the
portion 508, the identification/capture module 808 captures an image of the portion. To this end, the identification/capture module 808 may determine a boundary around the portion and display the boundary as an augmented reality on the HMD. In one configuration, the image of the portion includes only image data within theboundary 510. This is beneficial in that it reduces the amount of image data to be transmitted to the search engine and also provides a more focused search by reducing the amount of extraneous searchable content from the search request. - The
communication module 812 transmits the image to a search engine. Subsequently, thecommunication module 812 receives data corresponding to search results from the search engine. The search results may be provided to the identification/capturing module 808 and displayed as an augmented reality image on the HMD screen in the vicinity of the object. - The AR devices, as illustrated in
FIGS. 4 and 8 may include additional modules that perform each of the steps of the algorithm in the aforementioned flow chart ofFIG. 7 . As such, each step in the aforementioned flow chart ofFIG. 7 may be performed by a module and the apparatus may include one or more of those modules. The modules may be one or more hardware components specifically configured to carry out the stated processes/algorithm, implemented by a processor configured to perform the stated processes/algorithm, stored within a computer-readable medium for implementation by a processor, or some combination thereof. -
FIG. 9 is a diagram 900 illustrating an example of a hardware implementation for anapparatus 802′ employing aprocessing system 914. Theprocessing system 914 may be implemented with a bus architecture, represented generally by thebus 924. Thebus 924 may include any number of interconnecting buses and bridges depending on the specific application of theprocessing system 914 and the overall design constraints. Thebus 924 links together various circuits including one or more processors and/or hardware modules, represented by theprocessor 904, themodules memory 906. Thebus 924 may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further. - The
processing system 914 includes aprocessor 904 coupled to a computer-readable medium/memory 906. Theprocessor 904 is responsible for general processing, including the execution of software stored on the computer-readable medium/memory 906. The software, when executed by theprocessor 904, causes theprocessing system 914 to perform the various functions described supra for any particular apparatus. The computer-readable medium/memory 906 may also be used for storing data that is manipulated by theprocessor 904 when executing software. The processing system further includes at least one of themodules processor 904, resident/stored in the computer readable medium/memory 906, one or more hardware modules coupled to theprocessor 904, or some combination thereof. - In one configuration, the
apparatus 802/802′ includes means for identifying a portion of an object in a field of view of the HMD based on user interaction with the HMD, means for capturing an image of the portion, wherein the size of the image is less than the size of the object in the field of view, and means for transmitting the image to a search engine. Theapparatus 802/802′ also includes means for displaying an augmented reality on the HMD in the vicinity of the portion, means for receiving an indication to initiate an online visual search of the object using the portion, means for receiving search results from the search engine; and means for displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object. The aforementioned means may be one or more of the aforementioned modules of theapparatus 802 and/or theprocessing system 914 of theapparatus 802′ configured to perform the functions recited by the aforementioned means. - In summary, methods and apparatus disclosed herein allow a user to select only a subset of scene with an eye gaze that contains an object of interest. With reference to
FIGS. 5 and 6 , a visual search using a HMD with augmented reality and eye tracking works as follows: A user looks at the object of interest in a scene visible through the HMD and initiates a visual search via other input techniques such as a gesture. An eye tracking system included in the HMD system determines a location of a gaze point corresponding to the point of the scene at which the user is gazing. The eye tracking system identifies the object of interest based on the gaze point. For example, with reference toFIG. 5 , based on the location of the eye gaze, the eye tracking system identifies the object of interest as the bar code. Such identification may be made by color recognition, wherein the system detects a change in color between the edge of the bar code and the back cover of the book. - With continued reference to
FIG. 5 , once the eye tracking system determines the object of interest, it identifies an area of the scene containing the object and highlights the area with an augmented reality, such as an augmented reality boundary surrounding the object. The selected part of the scene corresponding to the area within the boundary is extracted and sent to the cloud (online visual search engine e.g. Google Goggles) for further analysis. As shown inFIG. 6 , the user's view is augmented with the retrieved search result. - The benefits of this approach are: The user is not required to move to the object of interest. Only relevant part of the image is sent to the cloud; thus saves redundant data transfer. Minimum workload on the search engine since it does not need to analyze all objects presented in the user's scene, only a subset which contains object of interest.
- In an aspect of the disclosure, a method, an apparatus, and a computer program product provide for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user. A method includes tracking an eye gaze of the user to locate the object, capturing an image of the object, and transmitting the image of the object to a search engine. The size of the image is less than the size of the scene. The method may also include receiving an indication to initiate an online visual search of an object. The method may further include receiving search results from the search engine, and displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- A corresponding apparatus for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user, includes means for tracking an eye gaze of the user to locate the object, means for capturing an image of the object, and means for transmitting the image of the object to a search engine. The size of the image is less than the size of the scene. The apparatus may also include means for receiving an indication to initiate an online visual search of an object. The apparatus may further include means for receiving search results from the search engine, and means for displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- Another corresponding apparatus for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user, includes a memory, and at least one processor coupled to the memory and configured to track an eye gaze of the user to locate the object, capture an image of the object, and transmit the image of the object to a search engine. The size of the image is less than the size of the scene. The processor may be configured to receive an indication to initiate an online visual search of an object. The processor may be further configured to receive search results from the search engine, and display the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- A computer program product for conducting an online visual search of an object present in a scene visible through a head mounted display worn by a user, includes a computer-readable medium comprising code for tracking an eye gaze of the user to locate the object, for capturing an image of the object, and for transmitting the image of the object to a search engine. The size of the image is less than the size of the scene. The product may include code for receiving an indication to initiate an online visual search of an object. The product may further include code for receiving search results from the search engine, and displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.
- It is understood that the specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged. Further, some steps may be combined or omitted. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.
- The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.” Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “at least one of A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and/or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “at least one of A, B, and C,” and “A, B, C, or any combination thereof” may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed as a means plus function unless the element is expressly recited using the phrase “means for.”
Claims (64)
1. An apparatus for processing one or more images, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
receive a first input from a user of the apparatus, wherein the first input comprises an eye gaze of the user;
determine a portion of interest within a scene based on the first input;
in response to the first input, cause content associated with the portion of interest to be displayed on a display;
receive a second input from the user of the apparatus, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and
initiate an action based on the second input.
2. The apparatus of claim 1 , wherein the gesture includes a hand gesture.
3. The apparatus of claim 2 , wherein the hand gesture comprises a physical interaction with the apparatus.
4. The apparatus of claim 3 , wherein the physical interaction with the apparatus comprises an interaction with a physical button of the apparatus.
5. The apparatus of claim 2 , wherein the at least one processor is configured to:
receive a third input from the user of the apparatus; and
in response to the third input, cause additional content to be displayed.
6. The apparatus of claim 5 , wherein the third input comprises an interaction with a physical interface of the apparatus.
7. The apparatus of claim 6 , wherein the physical interface is a physical button.
8. The apparatus of claim 6 , wherein, to cause the content associated with the portion of interest to be displayed, the at least one processor is configured to cause displayed content to be adjusted based on the first input.
9. The apparatus of claim 2 , wherein, to initiate the action, the at least one processor is configured to launch an application.
10. The apparatus of claim 2 , wherein the first input includes an interaction with a search icon.
11. The apparatus of claim 10 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene.
12. The apparatus of claim 2 , wherein, to receive the first input, the at least one processor is configured to track the eye gaze of the user to determine a location of the eye gaze relative to the display, wherein the portion of interest within the scene corresponds to the location of the eye gaze.
13. The apparatus of claim 1 , wherein, to receive the second input, the at least one processor is configured to track a gesture of the user to determine a location of the gesture.
14. The apparatus of claim 1 , wherein the content comprises a boundary.
15. The apparatus of claim 1 , wherein the action comprises a visual search process associated with the portion of interest.
16. The apparatus of claim 2 , wherein the at least one processor is configured to obtain, from an image capture apparatus, at least one image of the scene.
17. The apparatus of claim 16 , further comprising the image capture apparatus.
18. The apparatus of claim 16 , wherein the at least one processor is configured to process the at least one image of the scene to generate data associated with the portion of interest.
19. The apparatus of claim 18 , wherein the at least one processor is configured to cause the data to be transmitted to a remote device configured to generate visual search results.
20. The apparatus of claim 19 , wherein the at least one processor is configured to:
receive the visual search results from the remote device; and
cause the visual search results to be displayed on the display.
21. The apparatus of claim 16 , wherein the apparatus is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device.
22. The apparatus of claim 1 , wherein the apparatus is a head mounted device.
23. The apparatus of claim 1 , further comprising the display configured to display the content.
24. An apparatus for processing one or more images, the apparatus comprising:
at least one memory;
at least one eye tracking camera;
at least one scene camera;
at least one infrared (IR) camera;
at least one display;
at least one battery;
at least one transceiver; and
at least one processor coupled to the at least one memory and configured to:
receive a first input from a user of the apparatus, wherein the first input comprises an eye gaze of the user;
determine a portion of interest within a scene based on the first input;
in response to the first input, cause content associated with the portion of interest to be displayed on the at least one display;
receive a second input from the user of the apparatus, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and
initiate an action based on the second input.
25. The apparatus of claim 24 , wherein the gesture includes a hand gesture.
26. The apparatus of claim 25 , wherein the hand gesture comprises a physical interaction with the apparatus.
27. The apparatus of claim 26 , wherein the physical interaction with the apparatus comprises an interaction with a physical button of the apparatus.
28. The apparatus of claim 25 , wherein the at least one processor is configured to:
receive a third input from the user of the apparatus; and
in response to the third input, cause additional content to be displayed on the at least one display.
29. The apparatus of claim 28 , wherein the third input comprises an interaction with a physical interface of the apparatus.
30. The apparatus of claim 29 , wherein the physical interface is a physical button.
31. The apparatus of claim 29 , wherein, to cause the content associated with the portion of interest to be displayed, the at least one processor is configured to cause displayed content to be adjusted based on the first input.
32. The apparatus of claim 25 , wherein, to initiate the action, the at least one processor is configured to launch an application.
33. The apparatus of claim 25 , wherein the first input includes an interaction with a search icon.
34. The apparatus of claim 33 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene.
35. The apparatus of claim 25 , wherein, to receive the first input, the at least one processor is configured to track the eye gaze of the user to determine a location of the eye gaze relative to the at least one display, wherein the portion of interest within the scene corresponds to the location of the eye gaze.
36. The apparatus of claim 24 , wherein, to receive the second input, the at least one processor is configured to track a gesture of the user to determine a location of the gesture.
37. The apparatus of claim 24 , wherein the content comprises a boundary.
38. The apparatus of claim 24 , wherein the action comprises a visual search process associated with the portion of interest.
39. The apparatus of claim 25 , wherein the at least one processor is configured to obtain, from the at least one scene camera, at least one image of the scene.
40. The apparatus of claim 39 , wherein the at least one processor is configured to process the at least one image of the scene to generate data associated with the portion of interest.
41. The apparatus of claim 40 , wherein the at least one processor is configured to cause the data to be transmitted, via the at least one transceiver, to a remote device configured to generate visual search results.
42. The apparatus of claim 41 , wherein the at least one processor is configured to:
receive the visual search results from the remote device; and
cause the visual search results to be displayed on the at least one display.
43. The apparatus of claim 39 , wherein the apparatus is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device.
44. The apparatus of claim 24 , wherein the apparatus is a head mounted device.
45. A method of processing one or more images, the method comprising:
receiving a first input from a user of a device, wherein the first input comprises an eye gaze of the user;
determining a portion of interest within a scene based on the first input;
displaying, in response to the first input, content associated with the portion of interest on a display;
receiving a second input from the user of the device, wherein the second input comprises a gesture, and wherein the second input is associated with the displayed content; and
initiating an action based on the second input.
46. The method of claim 45 , wherein the gesture includes a hand gesture.
47. The method of claim 46 , wherein the hand gesture comprises a physical interaction with the device.
48. The method of claim 47 , wherein the physical interaction with the device comprises an interaction with a physical button of the device.
49. The method of claim 46 , further comprising:
receiving a third input from the user of the device; and
displaying, in response to the third input, additional content.
50. The method of claim 49 , wherein the third input comprises an interaction with a physical interface of the device.
51. The method of claim 50 , wherein the physical interface is a physical button.
52. The method of claim 50 , wherein displaying the content associated with the portion of interest comprises adjusting displayed content based on the first input.
53. The method of claim 46 , wherein initiating the action comprises launching an application.
54. The method of claim 46 , wherein the first input includes an interaction with a search icon.
55. The method of claim 54 , wherein, based on the interaction with the search icon, a search is performed based on a boundary around a portion in the scene.
56. The method of claim 46 , wherein receiving the first input comprises tracking the eye gaze of the user to determine a location of the eye gaze relative to the display, wherein the portion of interest within the scene corresponds to the location of the eye gaze.
57. The method of claim 45 , wherein receiving the second input comprises tracking a gesture of the user to determine a location of the gesture.
58. The method of claim 45 , wherein the content comprises a boundary.
59. The method of claim 45 , wherein the action comprises a visual search process associated with the portion of interest.
60. The method of claim 46 , further comprising obtaining, from an image capture device, at least one image of the scene.
61. The method of claim 60 , further comprising processing the at least one image of the scene to generate data associated with the portion of interest.
62. The method of claim 61 , further comprising transmitting the data to a remote device configured to generate visual search results.
63. The method of claim 62 , further comprising:
receiving the visual search results from the remote device; and
displaying the visual search results on the display.
64. The method of claim 60 , wherein the device is a head mounted device, and wherein the at least one image of the scene corresponds to a field of view observed by the user through a head mounted device.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/344,512 US20230342388A1 (en) | 2013-08-19 | 2023-06-29 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201361867579P | 2013-08-19 | 2013-08-19 | |
US14/151,664 US10152495B2 (en) | 2013-08-19 | 2014-01-09 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/181,175 US10372751B2 (en) | 2013-08-19 | 2018-11-05 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/414,667 US11068531B2 (en) | 2013-08-19 | 2019-05-16 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US17/304,739 US11734336B2 (en) | 2013-08-19 | 2021-06-24 | Method and apparatus for image processing and associated user interaction |
US18/344,512 US20230342388A1 (en) | 2013-08-19 | 2023-06-29 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/304,739 Continuation US11734336B2 (en) | 2013-08-19 | 2021-06-24 | Method and apparatus for image processing and associated user interaction |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230342388A1 true US20230342388A1 (en) | 2023-10-26 |
Family
ID=52466534
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/151,664 Active 2034-02-01 US10152495B2 (en) | 2013-08-19 | 2014-01-09 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/181,175 Active US10372751B2 (en) | 2013-08-19 | 2018-11-05 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/414,667 Active US11068531B2 (en) | 2013-08-19 | 2019-05-16 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US17/304,739 Active 2034-01-31 US11734336B2 (en) | 2013-08-19 | 2021-06-24 | Method and apparatus for image processing and associated user interaction |
US18/344,512 Pending US20230342388A1 (en) | 2013-08-19 | 2023-06-29 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
Family Applications Before (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/151,664 Active 2034-02-01 US10152495B2 (en) | 2013-08-19 | 2014-01-09 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/181,175 Active US10372751B2 (en) | 2013-08-19 | 2018-11-05 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US16/414,667 Active US11068531B2 (en) | 2013-08-19 | 2019-05-16 | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US17/304,739 Active 2034-01-31 US11734336B2 (en) | 2013-08-19 | 2021-06-24 | Method and apparatus for image processing and associated user interaction |
Country Status (2)
Country | Link |
---|---|
US (5) | US10152495B2 (en) |
WO (1) | WO2015047599A2 (en) |
Families Citing this family (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10152495B2 (en) | 2013-08-19 | 2018-12-11 | Qualcomm Incorporated | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
US20150145887A1 (en) * | 2013-11-25 | 2015-05-28 | Qualcomm Incorporated | Persistent head-mounted content display |
EP3062218A1 (en) * | 2015-02-25 | 2016-08-31 | BAE Systems PLC | Interactive information system |
WO2016135471A1 (en) * | 2015-02-25 | 2016-09-01 | Bae Systems Plc | Interactive information system |
US9829976B2 (en) * | 2015-08-07 | 2017-11-28 | Tobii Ab | Gaze direction mapping |
US9870051B2 (en) | 2015-08-07 | 2018-01-16 | Tobii Ab | Gaze direction mapping |
EP3182328A1 (en) * | 2015-12-17 | 2017-06-21 | Nokia Technologies Oy | A method, apparatus or computer program for controlling image processing of a captured image of a scene to adapt the captured image |
US10372289B2 (en) * | 2015-12-31 | 2019-08-06 | Beijing Pico Technology Co., Ltd. | Wraparound interface layout method, content switching method under three-dimensional immersive environment, and list switching method |
CN106997241B (en) * | 2016-01-22 | 2020-04-21 | 宏达国际电子股份有限公司 | Method for interacting with real world in virtual reality environment and virtual reality system |
JP2017182441A (en) * | 2016-03-30 | 2017-10-05 | 富士通株式会社 | Operation actual condition processing device, method, and program |
US10981060B1 (en) | 2016-05-24 | 2021-04-20 | Out of Sight Vision Systems LLC | Collision avoidance system for room scale virtual reality system |
US11468111B2 (en) * | 2016-06-01 | 2022-10-11 | Microsoft Technology Licensing, Llc | Online perspective search for 3D components |
CN106155315A (en) * | 2016-06-28 | 2016-11-23 | 广东欧珀移动通信有限公司 | The adding method of augmented reality effect, device and mobile terminal in a kind of shooting |
US10530972B2 (en) * | 2016-09-21 | 2020-01-07 | Htc Corporation | Control method for optical tracking system |
US11003868B2 (en) | 2016-11-07 | 2021-05-11 | ' Rockwell Automation Technologies, Inc. | Filtering display data |
US10185848B2 (en) * | 2016-11-07 | 2019-01-22 | Rockwell Automation Technologies, Inc. | Emphasizing equipment based on an equipment tag |
US20180150997A1 (en) * | 2016-11-30 | 2018-05-31 | Microsoft Technology Licensing, Llc | Interaction between a touch-sensitive device and a mixed-reality device |
US11164378B1 (en) * | 2016-12-08 | 2021-11-02 | Out of Sight Vision Systems LLC | Virtual reality detection and projection system for use with a head mounted display |
TWI603227B (en) | 2016-12-23 | 2017-10-21 | 李雨暹 | Method and system for remote management of virtual message for a moving object |
US10262036B2 (en) | 2016-12-29 | 2019-04-16 | Microsoft Technology Licensing, Llc | Replacing pronouns with focus-specific objects in search queries |
CN106933364B (en) * | 2017-03-15 | 2019-09-27 | 京东方科技集团股份有限公司 | Characters input method, character input device and wearable device |
US20180275751A1 (en) * | 2017-03-21 | 2018-09-27 | Microsoft Technology Licensing, Llc | Index, search, and retrieval of user-interface content |
US10319151B2 (en) | 2017-07-07 | 2019-06-11 | Motorola Solutions, Inc. | Device and method for hierarchical object recognition |
KR102586170B1 (en) * | 2017-08-01 | 2023-10-10 | 삼성전자주식회사 | Electronic device and method for providing search result thereof |
CN107908328A (en) * | 2017-11-15 | 2018-04-13 | 百度在线网络技术(北京)有限公司 | Augmented reality method and apparatus |
US20190155560A1 (en) * | 2017-11-23 | 2019-05-23 | Mindtronic Ai Co.,Ltd. | Multi-display control apparatus and method thereof |
CN107992814A (en) * | 2017-11-28 | 2018-05-04 | 北京小米移动软件有限公司 | Object finding method and device |
US10580215B2 (en) * | 2018-03-29 | 2020-03-03 | Rovi Guides, Inc. | Systems and methods for displaying supplemental content for print media using augmented reality |
US10204160B1 (en) | 2018-04-10 | 2019-02-12 | Rovi Guides, Inc. | Methods and systems for disambiguating user input based on detection of ensembles of items |
US11169601B2 (en) | 2018-07-12 | 2021-11-09 | Toyota Research Institute, Inc. | Methods and systems for determining teleoperating user intent via eye tracking |
CN111309142A (en) * | 2018-12-11 | 2020-06-19 | 托比股份公司 | Method and device for switching input modality of display device |
US11175803B2 (en) | 2019-02-07 | 2021-11-16 | International Business Machines Corporation | Remote guidance for object observation |
US11645561B2 (en) * | 2019-03-18 | 2023-05-09 | International Business Machines Corporation | Question answering system influenced by user behavior and text metadata generation |
TWI719483B (en) * | 2019-05-17 | 2021-02-21 | 雅得近顯股份有限公司 | Convenient memo operating system |
KR102128894B1 (en) * | 2019-10-10 | 2020-07-01 | 주식회사 메디씽큐 | A method and system for eyesight sensing of medical smart goggles |
US11106914B2 (en) | 2019-12-02 | 2021-08-31 | At&T Intellectual Property I, L.P. | Method and apparatus for delivering content to augmented reality devices |
CN111563432A (en) * | 2020-04-27 | 2020-08-21 | 歌尔科技有限公司 | Display method and augmented reality display device |
CN111766947A (en) * | 2020-06-30 | 2020-10-13 | 歌尔科技有限公司 | Display method, display device, wearable device and medium |
CN111770254A (en) * | 2020-07-13 | 2020-10-13 | 深圳市中科先见医疗科技有限公司 | AR (augmented reality) glasses-based vision-aiding rehabilitation aid, system and display method |
CN112315753B (en) * | 2020-10-26 | 2022-11-25 | 杭州集视智能科技有限公司 | Visual field expansion training device based on glaucoma late-stage disease and training method thereof |
US11709553B2 (en) | 2021-02-25 | 2023-07-25 | International Business Machines Corporation | Automated prediction of a location of an object using machine learning |
Family Cites Families (49)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7680324B2 (en) * | 2000-11-06 | 2010-03-16 | Evryx Technologies, Inc. | Use of image-derived information as search criteria for internet and other search engines |
US8553949B2 (en) * | 2004-01-22 | 2013-10-08 | DigitalOptics Corporation Europe Limited | Classification and organization of consumer digital images using workflow, and face detection and recognition |
DK1607840T3 (en) * | 2004-06-18 | 2015-02-16 | Tobii Technology Ab | Eye control of a computer device |
US8838562B1 (en) * | 2004-10-22 | 2014-09-16 | Google Inc. | Methods and apparatus for providing query parameters to a search engine |
US7945099B2 (en) * | 2005-05-09 | 2011-05-17 | Like.Com | System and method for use of images with recognition analysis |
US20080268876A1 (en) * | 2007-04-24 | 2008-10-30 | Natasha Gelfand | Method, Device, Mobile Terminal, and Computer Program Product for a Point of Interest Based Scheme for Improving Mobile Visual Searching Functionalities |
US8239359B2 (en) * | 2008-09-23 | 2012-08-07 | Disney Enterprises, Inc. | System and method for visual search in a video media player |
JP5471124B2 (en) * | 2009-07-29 | 2014-04-16 | ソニー株式会社 | Image search apparatus, image search method, and image search program |
KR20110013606A (en) * | 2009-08-03 | 2011-02-10 | 엘지전자 주식회사 | Method for executing menu in mobile terminal and mobile terminal thereof |
US8670597B2 (en) * | 2009-08-07 | 2014-03-11 | Google Inc. | Facial recognition with social network aiding |
US9852156B2 (en) * | 2009-12-03 | 2017-12-26 | Google Inc. | Hybrid use of location sensor data and visual query to return local listings for visual query |
KR20110118421A (en) * | 2010-04-23 | 2011-10-31 | 엘지전자 주식회사 | Augmented remote controller, augmented remote controller controlling method and the system for the same |
US8849845B2 (en) * | 2010-11-03 | 2014-09-30 | Blackberry Limited | System and method for displaying search results on electronic devices |
US10444979B2 (en) * | 2011-01-31 | 2019-10-15 | Microsoft Technology Licensing, Llc | Gesture-based search |
US8510166B2 (en) * | 2011-05-11 | 2013-08-13 | Google Inc. | Gaze tracking system |
US8223088B1 (en) * | 2011-06-09 | 2012-07-17 | Google Inc. | Multimode input field for a head-mounted display |
JP5830784B2 (en) * | 2011-06-23 | 2015-12-09 | サイバーアイ・エンタテインメント株式会社 | Interest graph collection system by relevance search with image recognition system |
US8558759B1 (en) * | 2011-07-08 | 2013-10-15 | Google Inc. | Hand gestures to signify what is important |
US8912979B1 (en) * | 2011-07-14 | 2014-12-16 | Google Inc. | Virtual window in head-mounted display |
WO2013023705A1 (en) * | 2011-08-18 | 2013-02-21 | Layar B.V. | Methods and systems for enabling creation of augmented reality content |
US9342610B2 (en) | 2011-08-25 | 2016-05-17 | Microsoft Technology Licensing, Llc | Portals: registered objects as virtualized, personalized displays |
US8719278B2 (en) * | 2011-08-29 | 2014-05-06 | Buckyball Mobile Inc. | Method and system of scoring documents based on attributes obtained from a digital document by eye-tracking data analysis |
US8868039B2 (en) * | 2011-10-12 | 2014-10-21 | Digimarc Corporation | Context-related arrangements |
EP2587342A1 (en) * | 2011-10-28 | 2013-05-01 | Tobii Technology AB | Method and system for user initiated query searches based on gaze data |
US9684374B2 (en) * | 2012-01-06 | 2017-06-20 | Google Inc. | Eye reflection image analysis |
US9052804B1 (en) * | 2012-01-06 | 2015-06-09 | Google Inc. | Object occlusion to initiate a visual search |
US20150193098A1 (en) * | 2012-03-23 | 2015-07-09 | Google Inc. | Yes or No User-Interface |
JP5892236B2 (en) * | 2012-03-26 | 2016-03-23 | ソニー株式会社 | Information processing apparatus, information processing method, and program |
US20130275411A1 (en) * | 2012-04-13 | 2013-10-17 | Lg Electronics Inc. | Image search method and digital device for the same |
US20130293580A1 (en) * | 2012-05-01 | 2013-11-07 | Zambala Lllp | System and method for selecting targets in an augmented reality environment |
US20130293530A1 (en) * | 2012-05-04 | 2013-11-07 | Kathryn Stone Perez | Product augmentation and advertising in see through displays |
US8893164B1 (en) * | 2012-05-16 | 2014-11-18 | Google Inc. | Audio system |
US9823742B2 (en) * | 2012-05-18 | 2017-11-21 | Microsoft Technology Licensing, Llc | Interaction and management of devices using gaze detection |
US10417289B2 (en) * | 2012-06-12 | 2019-09-17 | Oath Inc. | Systems and methods involving integration/creation of search results media modules |
US9389420B2 (en) * | 2012-06-14 | 2016-07-12 | Qualcomm Incorporated | User interface interaction for transparent head-mounted displays |
CN104322058A (en) * | 2012-06-27 | 2015-01-28 | 英特尔公司 | Multiple-frame screenshot |
US9298784B1 (en) * | 2012-07-17 | 2016-03-29 | Amazon Technologies, Inc. | Searching inside items |
US20140078174A1 (en) * | 2012-09-17 | 2014-03-20 | Gravity Jack, Inc. | Augmented reality creation and consumption |
US9996150B2 (en) * | 2012-12-19 | 2018-06-12 | Qualcomm Incorporated | Enabling augmented reality using eye gaze tracking |
US9924102B2 (en) * | 2013-03-14 | 2018-03-20 | Qualcomm Incorporated | Image-based application launcher |
EP2972678A4 (en) * | 2013-03-15 | 2016-11-02 | Interaxon Inc | Wearable computing apparatus and method |
US9690831B2 (en) * | 2013-04-19 | 2017-06-27 | Palo Alto Research Center Incorporated | Computer-implemented system and method for visual search construction, document triage, and coverage tracking |
US10152495B2 (en) | 2013-08-19 | 2018-12-11 | Qualcomm Incorporated | Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking |
KR102157313B1 (en) * | 2013-09-03 | 2020-10-23 | 삼성전자주식회사 | Method and computer readable recording medium for recognizing an object using a captured image |
GB201322873D0 (en) * | 2013-12-23 | 2014-02-12 | Tobii Technology Ab | Eye gaze determination |
KR20150133898A (en) * | 2014-05-20 | 2015-12-01 | 삼성디스플레이 주식회사 | Display device with window |
KR102337682B1 (en) * | 2014-10-01 | 2021-12-09 | 삼성전자주식회사 | Display apparatus and Method for controlling thereof |
US10262036B2 (en) * | 2016-12-29 | 2019-04-16 | Microsoft Technology Licensing, Llc | Replacing pronouns with focus-specific objects in search queries |
EP3438853A1 (en) * | 2017-08-01 | 2019-02-06 | Samsung Electronics Co., Ltd. | Electronic device and method for providing search result thereof |
-
2014
- 2014-01-09 US US14/151,664 patent/US10152495B2/en active Active
- 2014-08-19 WO PCT/US2014/051717 patent/WO2015047599A2/en active Application Filing
-
2018
- 2018-11-05 US US16/181,175 patent/US10372751B2/en active Active
-
2019
- 2019-05-16 US US16/414,667 patent/US11068531B2/en active Active
-
2021
- 2021-06-24 US US17/304,739 patent/US11734336B2/en active Active
-
2023
- 2023-06-29 US US18/344,512 patent/US20230342388A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2015047599A3 (en) | 2015-05-21 |
US11734336B2 (en) | 2023-08-22 |
US20210342387A1 (en) | 2021-11-04 |
US20150049113A1 (en) | 2015-02-19 |
US11068531B2 (en) | 2021-07-20 |
US20190278799A1 (en) | 2019-09-12 |
US10372751B2 (en) | 2019-08-06 |
US20190073386A1 (en) | 2019-03-07 |
WO2015047599A2 (en) | 2015-04-02 |
US10152495B2 (en) | 2018-12-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11734336B2 (en) | Method and apparatus for image processing and associated user interaction | |
US10914951B2 (en) | Visual, audible, and/or haptic feedback for optical see-through head mounted display with user interaction tracking | |
US10089786B2 (en) | Automatic customization of graphical user interface for optical see-through head mounted display with user interaction tracking | |
US10073518B2 (en) | Automatic calibration of eye tracking for optical see-through head mounted display | |
US10394334B2 (en) | Gesture-based control system | |
US9262780B2 (en) | Method and apparatus for enabling real-time product and vendor identification | |
US9536354B2 (en) | Object outlining to initiate a visual search | |
US9405977B2 (en) | Using visual layers to aid in initiating a visual search | |
US10248192B2 (en) | Gaze target application launcher | |
US9165381B2 (en) | Augmented books in a mixed reality environment | |
CN105393191B (en) | Adaptive event identification | |
WO2015026645A1 (en) | Automatic calibration of scene camera for optical see-through head mounted display | |
US20150049001A1 (en) | Enabling remote screen sharing in optical see-through head mounted display with augmented reality | |
US20130007668A1 (en) | Multi-visor: managing applications in head mounted displays | |
US20190212828A1 (en) | Object enhancement in artificial reality via a near eye display interface | |
WO2019214442A1 (en) | Device control method, apparatus, control device and storage medium | |
US9052804B1 (en) | Object occlusion to initiate a visual search | |
US10896545B1 (en) | Near eye display interface for artificial reality applications | |
US20230305635A1 (en) | Augmented reality device, and method for controlling augmented reality device | |
US11328187B2 (en) | Information processing apparatus and information processing method | |
KR20220058277A (en) | Method of stereo matching and image processing apparatus performing the same | |
JP2022069212A (en) | Control apparatus, program, and system | |
CN116027908A (en) | Color acquisition method, device, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: QUALCOMM INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RAHMAN, MD SAZZADUR;LIU, KEXI;RENSCHLER, MARTIN H.;SIGNING DATES FROM 20140106 TO 20140206;REEL/FRAME:064246/0575 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |