US20130044912A1 - Use of association of an object detected in an image to obtain information to display to a user - Google Patents

Use of association of an object detected in an image to obtain information to display to a user Download PDF

Info

Publication number
US20130044912A1
US20130044912A1 US13/549,339 US201213549339A US2013044912A1 US 20130044912 A1 US20130044912 A1 US 20130044912A1 US 201213549339 A US201213549339 A US 201213549339A US 2013044912 A1 US2013044912 A1 US 2013044912A1
Authority
US
United States
Prior art keywords
user
identifier
image
information
scene
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/549,339
Other languages
English (en)
Inventor
Tejas Dattatraya Kulkarni
Bocong Liu
Ankur B. Nandwani
Justin E. Taseski
Benjamin J. Yule
Dimosthenis Kaleas
Babak Forutanpour
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Priority to US13/549,339 priority Critical patent/US20130044912A1/en
Priority to PCT/US2012/046816 priority patent/WO2013028279A1/en
Assigned to QUALCOMM INCORPORATED reassignment QUALCOMM INCORPORATED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: NANDWANI, Ankur B, KULKARNI, Tejas Dattatraya, TASESKI, Justin E, LIU, Bocong, YULE, Benjamin J, FORUTANPOUR, BABAK, KALEAS, DIMOSTHENIS
Publication of US20130044912A1 publication Critical patent/US20130044912A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/142—Image acquisition using hand-held instruments; Constructional details of the instruments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3191—Testing thereof
    • H04N9/3194—Testing thereof including sensor feedback

Definitions

  • one or more cameras capture a scene that includes an object in real world that is sufficiently small to be carried by a human hand (“portable”), such as a stapler or a book. Thereafter, an image of the scene from the one or more cameras is processed to detect therein a portion corresponding to the object, which is recognized from among a set of pre-selected real world objects. An identifier of the object is then used, with a set of associations that associate object identifiers and identifiers of users, to obtain a user identifier that identifies a user at least partially among a set of users.
  • the user identifier identifies the user generically, as belonging to a particular group of users (also called “weak identification”) among several such groups.
  • the user identifier identifies a single user uniquely (also called “strong identification”), among all such users in the set.
  • the user identifier (obtained, as noted above, by use of an association and the object identifier) is thereafter used in several such embodiments, either alone or in combination with user-supplied information, to obtain and store in memory, information to be output to the user. At least a portion of the obtained information is thereafter output, for example by projection into the scene.
  • FIG. 1A illustrates a portable real world object 132 (e.g. a stapler) being imaged by a camera 121 , and its use by processor 100 in an electronic device 120 to cause a projection of information into scene 130 by performing acts illustrated in FIG. 1C , in several embodiments described herein.
  • a portable real world object 132 e.g. a stapler
  • FIG. 1B illustrates another portable real world object 142 (e.g. a tape dispenser) being imaged by another electronic device 150 in a manner similar to electronic device 120 of FIG. 1A , to display information 119 T on a screen 152 of device 150 , also by performing the acts of FIG. 1C in several embodiments described herein.
  • another portable real world object 142 e.g. a tape dispenser
  • FIG. 1C illustrates in a high-level flow chart, acts performed by one or more processors 100 (e.g. in one of electronic devices 120 and 150 of FIGS. 1A and 1B ), to use an association of an object detected in an image to obtain information to display, in some embodiments described herein.
  • processors 100 e.g. in one of electronic devices 120 and 150 of FIGS. 1A and 1B
  • FIG. 1D illustrates in a high-level block diagram, a set of associations 111 in a memory 110 used by processor 100 of FIG. 1C , in some of the described embodiments.
  • FIG. 1E illustrates association of portable real world object 132 with group 2 by user 135 making a hand gesture with two outstretched fingers adjacent to portable real world object 132 , in certain embodiments.
  • FIG. 1F illustrates use of the association of FIG. 1E to obtain and display information 188 for group 2 , in certain embodiments.
  • FIG. 1G illustrates changing the association of FIG. 1E by replacing group 1 with group 2 by user 135 making another hand gesture with index finger 136 pointing at portable real world object 132 , in such embodiments.
  • FIG. 1H illustrates use of the association of FIG. 1G to obtain and display information 189 A and 189 B for group 1 , in such embodiments.
  • FIGS. 1I and 1J illustrate colored beams 163 and 164 of blue color and green color respectively projected on to object 132 by a projector 122 in certain embodiments.
  • FIG. 2A illustrates in a high-level flow chart, acts performed by one or more processors 100 to use an association of an object detected in an image in combination with user input, to obtain information to display, in some embodiments described herein.
  • FIG. 2B illustrates in a high-level block diagram, a set of associations 211 in a memory 110 used by processor 100 of FIG. 2A , in some of the described embodiments.
  • FIG. 2C illustrates real world object 132 in scene 230 imaged by camera 121 , for use in projection of information in certain embodiments.
  • FIG. 2D illustrates processor 100 coupled to memory 110 that contains image 109 as well as an address 291 (in a table 220 ) used to obtain information for display in some embodiments.
  • FIG. 2E illustrates display of information in the form of a video 295 projected adjacent to object 132 in some embodiments.
  • FIGS. 3A and 3F illustrate in high-level flow charts, acts performed by one or more processors 100 to use an association of an object detected in an image with a single person, to obtain information to display specific to that person, in some embodiments described herein.
  • FIG. 3B illustrates in a high-level block diagram, a set of associations 311 in a memory 110 used by processor 100 of FIGS. 3A and 3F , in some of the described embodiments.
  • FIG. 3C illustrates, a user interface included in information projected adjacent to object 132 and in some embodiments.
  • FIGS. 3D and 3E illustrate, a user's personalized information included in information projected adjacent to object 132 (in FIG. 3D ) and onto object 132 (in FIG. 3E ) in some embodiments.
  • FIG. 4A illustrates, in a high-level block diagram, a processor 100 coupled to a memory 110 in a mobile device 120 of some embodiments.
  • FIG. 4B illustrates in a high-level flow chart, acts performed by processor 100 of FIG. 4A in projecting information into scene 130 in several embodiments.
  • FIGS. 4C and 4D illustrate in intermediate-level flow charts, acts performed by processor 100 in projecting information in certain embodiments.
  • FIG. 5 illustrates, in a high-level block diagram, a mobile device 120 of several embodiments.
  • one or more device(s) 120 use one or more cameras 121 (and/or sensors such as a microphone) to capture input e.g. one or more images 109 ( FIG. 1B ) from a scene 130 ( FIG. 1A ) that contains an object 132 ( FIG. 1A ).
  • object 132 can be any object in the real world (in scene 130 ) that is portable by a human, e.g. small enough to be carried in (and/or moved by) a human hand, such as any handheld object. Examples of object 132 are a stapler, a mug, a bottle, a glass, a book, a cup, etc.
  • device(s) 120 can be any electronic device that includes a camera 121 , a memory 110 and a processor 100 , such as a smartphone or a tablet (e.g. iPhone or iPad available from APPLE, Inc.).
  • a smartphone e.g. iPhone or iPad available from APPLE, Inc.
  • the following description refers to a single device 120 performing the method of FIG. 1C , although multiple such devices can be used to individually perform any one or more of steps 101 - 108 , depending on the embodiment.
  • one or more captured images 109 are initially received from a camera 121 (as per act 101 in FIG. 1C ) e.g. via bus 1113 ( FIG. 5 ) and stored in a memory 110 .
  • Processor 100 then processes (as per act 102 in FIG. 1C ) the one or more images 109 to detect the presence of an object 132 ( FIG. 1A ) e.g. on a surface of a table 131 , in a scene 130 of real world outside camera 121 (see act 102 in FIG. 1C ).
  • processor 100 may be programmed to recognize a portion of image 109 corresponding to portable real world object 132 , to obtain an identifier (“object identifier”) 1120 that uniquely identifies object 132 among a set of predetermined objects.
  • object identifier an identifier
  • Processor 100 then uses the object identifier (e.g. stapler identifier 1120 in FIG. 1D ) that is obtained in act 102 to look up a set of associations 111 in an act 103 ( FIG. 1C ).
  • the result of act 103 is an identifier of a user (e.g. user identifier 112 U) who has been previously associated with portable real world object 132 (e.g. stapler).
  • memory 110 that is coupled to processor 100 holds a set of associations 111 including for example an association 112 ( FIG.
  • association 111 may be created in different ways depending on the embodiment, and in some illustrative embodiments an association 112 in set 111 is initialized or changed in response to a hand gesture by a user, as described below.
  • user identifiers 112 U and 114 U which are used in associations 112 and 114 do not identify a single user uniquely, among the users of a system of such devices 120 . Instead, in these embodiments, user identifiers 112 U and 114 U identify corresponding groups of users, such as a first group 1 of users A, B and C, and a second group 2 of users X, Y and Z (users A-C and X-Z are not shown in FIG. 1D ). In such embodiments (also called “weak identification” embodiments), a user identifier 112 U that is obtained in act 103 (described above) is generic to several users A, B and C within the first group 1 . In other embodiments (also called “strong identification” embodiments), each user identifier obtained in act 103 identifies a single user uniquely, as described below in reference to FIGS. 3A-3F .
  • processor 100 of several embodiments uses a user identifier 112 U looked up in act 103 , to generate an address of information 119 ( FIG. 5 ) to be output to the user and then obtains and stores in memory 110 (as per act 104 in FIG. 1C ) the information 119 .
  • information 119 to be output obtained in act 104 is common to all users within that group 1 (e.g. common to users A, B and C).
  • information 119 includes the text “Score: 73” which represents a score of this group 1 , in a game being played between two groups of users, namely users in group 1 and users in group 2 .
  • Information 119 is optionally transformed and displayed or otherwise output to user 135 , in an act 105 (see FIG. 1C ) as information 119 T ( FIG. 1A ). Specifically, in some embodiments, information 119 is displayed by projection of at least a portion of the information (e.g. the string of characters “Score: 73”) into scene 130 by use of a projector 122 as illustrated in FIG. 1A . In other embodiments, information 119 (or a portion thereof) may be output by act 105 ( FIG. 1C ) in other ways, e.g. device 150 ( FIG. 1B ) displaying information 119 to user 135 directly on a screen 151 ( FIG. 1B ) that also displays a live video of scene 130 (e.g. by displaying image 109 ), thereby to provide an augmented reality (AR) display.
  • AR augmented reality
  • information 119 may be played through a speaker 1111 ( FIG. 5 ) in device 120 , or even through a headset worn by user 135 .
  • device 150 is a smartphone that includes a front-facing camera 152 ( FIG. 1B ) in addition to a rear-facing camera 121 that captures image 109 .
  • Front-facing camera 152 ( FIG. 1B ) is used in some embodiments to obtain an image of a face of user 135 , for use in face recognition in certain strong identification embodiments described below in reference to FIGS. 3A-3F .
  • Device 150 ( FIG. 1B ) may be used in a manner similar or identical to use of device 120 as described herein, depending on the embodiment.
  • Processor 100 ( FIG. 5 ) of certain embodiments performs act 103 after act 102 when associations 111 ( FIG. 1D ) have been previously formed and are readily available in memory 110 .
  • Associations 111 may be set up in memory 110 based on information input by user 135 (“user input”) in any manner, as will be readily apparent in view of this detailed description.
  • user input in the form of text including words spoken by user 135 are extracted by some embodiments of processor 100 operating as a user input extractor 141 E, from an audio signal that is generated by a microphone 1112 ( FIG. 5 ) in the normal manner.
  • user input e.g.
  • processor 100 is programmed to respond to user input sensed by one or more sensors in device 120 , (e.g. camera 121 ) that detect one or more actions (e.g. gestures) by a user 135 as follows: processor 100 associates an object 132 that is selectively placed within a field of view 121 F of camera 121 with an identifier (e.g. a group identifier) that depends on user input (e.g. hand gesture).
  • identifier e.g. a group identifier
  • any person e.g. user 135 can use their hand 138 to form a specific gesture (e.g. tapping on object 132 ), to provide user input via camera 121 to processor 100 that in turn uses such user input in any manner described herein.
  • processor 100 may use input from user 135 in the form of hand gestures (or hand shapes) captured in a video (or still image) by camera 121 , to initialize or change a user identifier that is generic and has been associated with object 132 , as illustrated in FIGS. 1E-1H , and described below.
  • some embodiments accept user input in other forms, e.g.
  • audio input such as a whistling sound, and/or a drumming sound and/or a tapping sound, and/or sound of text including words “Group Two” spoken by user 135 may be used to associate an object 132 that is imaged within image 109 with a user identifier that is generic (e.g. commonly used to identify multiple users belonging to a particular group).
  • user input extractor 141 E is designed to be responsive to images from camera 121 of a user 135 forming a predetermined shape in a gesture 118 , namely the shape of letter “V” of the English alphabet with hand 138 , by stretching out index finger 136 and stretching out middle finger 137 .
  • camera 121 images hand 138 , with the just-described predetermined shape “V” at a location in real world that is adjacent to (or overlapping) portable real world object 132 ( FIG. 1E ).
  • processor 100 is programmed to perform act 106 ( FIG. 1C ) to extract from image 109 ( FIG.
  • processor 100 responds to detection of such a hand gesture by forming an association (in the set of associations 111 ) that is thereafter used to identify person 135 (as belonging to group 2 ) every time this same hand gesture is recognized by processor 100 .
  • processor 100 for processor 100 to be responsive to a hand gesture, user 135 is required to position fingers 136 and 137 sufficiently close to object 132 so that hand gesture fingers 136 , 137 and object 132 are all imaged together within a single image 109 (which may be a still image, or alternatively a frame of video, depending on the embodiment) from camera 121 .
  • a single image 109 which may be a still image, or alternatively a frame of video, depending on the embodiment
  • processor 100 when user 135 makes the same hand gesture, but outside the field of view 121 F of camera 121 ( FIG. 5 ), processor 100 does not detect such a hand gesture and so processor 100 does not use make or change any association, even when object 132 is detected in image 109 .
  • an act 106 is performed by processor 100 after act 102 , to identify the above-described hand gesture (or any other user input depending on the embodiment), from among a library of such gestures (or other such user input) that are predetermined and available in a database 199 on a disk (see FIG. 2D ) or other non-transitory storage media.
  • processor 100 performs a look up of a predetermined mapping 116 ( FIG. 1D ) based on the hand gesture (or other such user input) detected in act 106 to obtain a user identifier from the set of associations 111 ( FIG. 5 ).
  • two-finger gesture 118 or other user input, e.g.
  • whistling or drumming is related by a mapping 116 to an identifier 114 U of group 2 , and therefore in an act 108 this identifier (looked-up from mapping 116 ) is used to form association 114 in the set 111 .
  • processor 100 of several embodiments proceeds to obtain information 119 to be output (as per act 104 , described above), followed by optional transformation and output (e.g. projection as per act 105 ), as illustrated in FIG. 1F .
  • object 132 was associated with group 1 , and therefore information 119 T which is output into scene 130 includes the text string “Score: 73” which represents a score of group 1 , in the game being played with users in group 2 .
  • the same user 135 can change a previously formed association, by making adjacent to the same portable real world object 132 , a second hand gesture 117 (e.g.
  • index finger 136 outstretched) that is different from a first hand gesture 118 (e.g. index finger 136 and middle finger 137 both stretched out).
  • the hand gesture is made by user 135 sufficiently close to object 132 to ensure that the gesture and object 132 are both captured in a common image by camera 121 .
  • Such a second hand gesture 117 ( FIG. 1D ) is detected by processor 100 in act 106 , followed by lookup of mapping 116 , followed by over-writing of a first user identifier 114 U that is currently included in association 114 with a second user identifier 112 U, thereby to change a previously formed association.
  • information including the text string “Score: 73” previously displayed (see FIG. 1F ) is now replaced with new information including the text string “Score:0” which is the score of Group 2 as shown in FIG. 1G .
  • one or more additional text strings may be displayed to identify the user(s).
  • the text string “Group 2 ” is displayed as a portion of information 188 T in FIG. 1G
  • the text string “Group 1 ” is displayed as a portion of information 188 T in FIG. 1F
  • information 188 T is optionally transformed for display relative to information 188 that is obtained for output, resulting in multiple text strings of information 188 being displayed on different surfaces, e.g. information 189 A displayed on object 132 and information 189 B displayed on table 131 as shown in FIG. 1H .
  • a mapping 116 maps hand gestures (or other user input) to user identifiers of groups
  • each hand gesture (or other user input) may be mapped to a single user 135 , thereby to uniquely identify each user (“strong identification embodiments”), e.g. as described below in reference to FIGS. 3A-3F .
  • yet another data structure (such as an array or a linked list) identifies a group to which each user belongs, and processor 100 may be programmed to use that data structure to identify a user's group when needed for use in associations 111 .
  • processor 100 of some embodiments simply uses recognition of a user's hand gesture (or other user input) to select Group 2 from among two groups, namely Group 1 and Group 2 .
  • a user's hand gesture or other user input
  • processor 100 uses recognition of a user's hand gesture (or other user input) to select Group 2 from among two groups, namely Group 1 and Group 2 .
  • the identity of object 132 is not used in some embodiments to obtain the to-be-displayed information.
  • processor 100 do use two identifiers based on corresponding detections in image 109 as described above, namely user identifier and object identifier, to obtain and store in memory 110 the to-be-displayed information 119 in act 104 ( FIG. 1C ).
  • the to-be-displayed information 119 may be obtained by processor 100 based on recognition of (1) only a hand gesture or (2) only the real world object, or (3) a combination of (1) and (2), depending on the embodiment.
  • a hand gesture is not required in certain embodiments of processor 100 that accepts other user input, such as an audio signal (generated by a microphone) that carries sounds made by a user 135 (with their mouth and/or with their hands) and recognized by processor 100 on execution of appropriate software designed to recognize such user-made sounds as user input, e.g. in signals from microphone 1112 ( FIG. 5 ).
  • a user identifier with which portable real world object 132 is associated is displayed as text, as illustrated by text string 189 A in FIG. 1 H
  • a user identifier may be displayed as color, as illustrated in FIGS. 1I and 1J .
  • object 132 in the form of a cap 132 of a bottle is selected by a user 135 to be included in an image of a scene 130 of real world being captured by a camera 121 ( FIG. 5 ).
  • identity 214 of bottle cap 132 is associated in a set of associations 211 (see FIG. 2B ) by default with three users that are identified as a group of friends by identity 215 , and this association is shown by device 120 , e.g. by projecting a beam 163 of blue color light on object 132 and in a peripheral region outside of and surrounding object 132 (denoted by the word of text ‘blue’ in FIG. 1I , as colors are not shown in a black-and-white figure). In this example, color blue has been previously associated with the group of friends of identity 215 , as the group's color.
  • a book's identity 212 is associated in the set of associations 211 (see FIG. 2B ) by default with four users that are identified as a group (of four students) by identity 213 (e.g. John Doe, Jane Wang, Peter Hall and Tom McCue).
  • a person 135 identified as a user of the group of students associates his group's identity 213 (see FIG. 2B ) with object 132 ( FIG. 1J ) by tapping table surface 131 with index finger 136 repeatedly several times in rapid succession (i.e. performs a hand gesture to which processor 100 is programmed to recognize and suitably respond), until person 135 sees a projection of beam 164 of green light on and around object 132 (denoted by the word ‘green’ in FIG. 1J , as this figure is also a black-and-white figure).
  • color green was previously associated with the group of students of identity 213 , as its group color.
  • tapping is another form of hand gesture recognized by processor 100 , e.g. on processing a camera-generated image containing the gesture and optionally user input in the form of sound, or both in some illustrative embodiments.
  • processor 100 performs acts 201 - 203 that are similar or identical to acts 101 - 103 described above in reference to FIG. 1C .
  • act 201 one or more rear-facing cameras 121 are used to capture scene 130 ( FIG. 2C ) that includes real world object 132 (in the form of a book) and store image 109 ( FIG. 2D ) in memory 110 in a manner similar to that described above, although in FIGS. 2C and 2D , the object 132 being imaged is a book.
  • processor 100 In performing acts 202 - 206 in FIG. 2A processor 100 not only recognizes object 132 as a book in image 109 ( FIG. 2D ) but additionally recognizes a text string 231 A therein ( FIG. 2C ), which is identified by a hand gesture.
  • processor 100 operates as a user-input extractor 141 E ( FIG. 5 ) that obtains input from user 135 for identifying information to be obtained for display in act 205 (which is similar to act 105 described above).
  • user input is received in processor 100 by detection of a gesture in image 109 ( FIG. 2D ) in which object 132 has also been imaged by camera 121 ( FIG. 2C ).
  • user 135 makes a predetermined hand gesture, namely an index finger hand gesture 117 by stretching finger 136 of hand 138 to point to text string 231 A in portable real world object 132 .
  • This index finger hand gesture 117 is captured in one or more image(s) 109 ( FIG. 2D ) in which object 132 is also imaged (e.g. finger 136 overlaps object 132 in the same image 109 ).
  • the imaged gesture is identified by use of a library of gestures, and an procedure triggered by the index finger hand gesture 117 is performed in act 204 , including OCR of an image portion identified by the gesture, to obtain as user input, the string of characters “Linear Algebra.”
  • processor 100 operates as an object-user-input mapper 141 M ( FIG. 5 ) that uses both: (1) the user group identified in act 203 from the presence of object 132 and (2) the user input identified from a text string 231 A (e.g. “Linear Algebra”) detected by use of a gesture identified in act 204 ( FIG. 2A ), to generate an address 291 (in a table 220 in FIG. 2B ).
  • a user group identified by act 203 may be first used by object-user-input mapper 141 M ( FIG. 5 ) to identify table 220 ( FIG. 2B ) from among multiple such tables, and then the identified table 220 is looked up with the user-supplied information, to identify an address 291 , which may be accessible on the Internet.
  • Such an address 291 is subsequently used (e.g. by information retriever 141 R in FIG. 5 ) to prepare a request for fetching from Internet, a video that is associated with the string 231 A.
  • processor 100 in act 205 ( FIG. 2A ) processor 100 generates address 291 as http://ocw.mit.edu/courses/ mathematics/18-06-linear-algebra-spring-2010/video-lectures/ which is then used to retrieve information 119 ( FIG. 5 ).
  • Use of table 220 as just described enables a query that is based on a single common text string 231 A to be mapped to different addresses, for information to be displayed to different groups of users.
  • a processor for one user A retrieves an address of the website of Stanford Distance learning course (namely http://scpd.stanford.edu/coursesSeminars/seminarsAndWebinars.jsp) from one table (e.g. customized for user A) while processor 100 for another user B retrieves another address 291 for MIT's OpenCourseware website (namely http://videolectures.net/mit_ocw/) from another table 220 (e.g. customized for user B).
  • Stanford Distance learning course namely http://scpd.stanford.edu/coursesSeminars/seminarsAndWebinars.jsp
  • processor 100 for another user B retrieves another address 291 for MIT's OpenCourseware website (namely http://videolectures.net/mit_ocw/) from another table 220 (e.g. customized for user B).
  • Such an address 291 that is retrieved by processor 100 using a table 220 , in combination with one or words in text string 231 A may be used in some embodiments with an Internet-based search service, such as the website www.google.com to identify content for display to user 135 .
  • processor 100 issues a request to address 291 in accordance with http protocol and obtains as information to be output, a video stream from the Internet, followed by optional transformation and projection of the information, as described below.
  • FIG. 2E A result of performing the just-described method of FIG. 2A is illustrated in FIG. 2E by a video 295 shown projected (after any transformation, as appropriate) on a surface of table 131 adjacent to object 132 .
  • video 295 has been automatically selected by processor 100 and is being displayed, based at least partially on optical character recognition (OCR) to obtain from one or more images (e.g. in video 295 ) of object 132 , a text string 231 A that has been identified by an index finger hand gesture 117 .
  • OCR optical character recognition
  • no additional input is needed by processor 100 from user 135 , after the user makes a predetermined hand gesture to point to text string 231 A and before the video is output, e.g.
  • no further user command is needed to invoke a video player in device 120 , as the video player is automatically invoked by processor 100 to play the video stream retrieved from the Internet.
  • Other such embodiments may require user input to approve (e.g. confirm) that the video stream is to be displayed.
  • text string 231 A is recognized from among many such strings that are included in image 109 , based on the string being located immediately above a tip of index finger 136 which is recognized by processor 100 in act 204 as a portion of a predetermined hand gesture.
  • human finger 136 is part of a hand 138 of a human user 135 and in this example finger 136 is used in gesture 117 to identify as user input a string of text in scene 130 , which is to trigger retrieval of information to be output.
  • index finger hand gesture 117 as illustrated in FIGS. 2C and 2D
  • user 135 makes a circling motion with finger 136 around text string 231 A as a different predetermined gesture that is similarly processed.
  • processor 103 completes recognition of real world object 132 (in the form of a book in FIG. 2C ), in this example by recognizing string 231 A. Thereafter, processor 103 generates a request to a source on the Internet to obtain information to be projected in the scene for use by person 135 (e.g. based on a generic user identifier of person 135 as belonging to a group of students).
  • user interfaces in certain embodiments of the type described above in reference to FIGS. 2A-2E automatically project information 119 on real world surfaces using a projector 122 embedded in a mobile device 120 .
  • user interfaces of the type shown in FIGS. 2A-2E reverse the flow of information of prior art user interfaces which require a user 135 to explicitly look for information, e.g. prior art requires manually using a web browser to navigate to a web site at which a video stream is hosted, and then manually searching for and requesting download of the video stream.
  • user interfaces of the type shown in FIGS. 2A-2E automatically output information that is likely to be of interest to the user, e.g. by projection on to surfaces of objects in real world, using an embedded mobile projector.
  • mobile device 120 may be any type of electronic device with a form factor sufficiently small to be held in a human hand 138 (similar in size to object 132 ) which provides a new way of interacting with user 135 as described herein.
  • user 135 may use such a mobile device 120 in collaboration with other users, with contextual user interfaces based on everyday objects 132 , wherein the user interfaces overlap for multiple users of a specific group, so as to provide common information to all users in that specific group (as illustrated for group 1 in FIG. 1F and group 2 in FIG. 1G ).
  • processor 100 may be programmed in some embodiments to automatically contextualize a user interface, by using one or more predetermined techniques to identify and obtain for display information that a user 135 needs to view, when interacting with one or more of objects 132 .
  • processor 120 in response to user 135 opening book (shown in FIG. 2C as object 132 ) to a specific page (e.g. page 20), processor 120 automatically processes images of real world to identify therein a text string 231 A based on its large font size relative to the rest of text on page 20. Then, processor 100 of several embodiments automatically identifies a video on “Linear Algebra” available on the Internet e.g. by use of a predetermined website (as described above in reference to act 205 ), and then seeks confirmation from user 135 that the identified video should be played. The user's confirmation may be received by processor 100 in a video stream that contains a predetermined gesture, e.g. user 135 waving of index finger in a motion to make a check mark (as another predetermined gesture that identifies user input).
  • a predetermined gesture e.g. user 135 waving of index finger in a motion to make a check mark
  • Processor 100 is programmed in some embodiments to implement strong identification embodiments, by performing one or more acts 301 - 309 illustrated in FIG. 3A , by use of user identifiers that uniquely identify a single user X from among all users A-Z.
  • information 319 FIGS. 3D and 3E
  • FIGS. 3D and 3E information 319 that is obtained for display may be specific to that single user X, for example email messages that are specifically addressed to user X.
  • processor 100 performs acts 301 and 302 that are similar or identical to acts 101 and 102 described above in reference to FIG. 1C .
  • processor 100 uses an identifier of the portable real world object with a set of associations 311 ( FIG. 3B ), to obtain an identifier that uniquely identifies a user of the portable real world object.
  • act 303 is similar to act 103 except for the set of associations 311 being used in act 303 .
  • an association 314 maps an object identifier 1140 (such as a bottle cap ID) to a single user 314 U (such as Jane Doe), as illustrated in FIG. 3B .
  • user 314 U is uniquely associated with object 132 in the form of a bottle cap ( FIG. 1E ), i.e. no other user is associated with a bottle cap.
  • object 132 in the form of a bottle cap ( FIG. 1E )
  • other users may be associated with other such portable real world objects (e.g. book in FIG. 2C or stapler in FIG. 1A ), but not with a bottle cap as it has already been uniquely associated with user 314 U ( FIG. 3B ).
  • a user identifier which is obtained in act 303 is used in act 304 ( FIG. 3A ) to obtain and store in memory 110 , information 319 that is specific to user 314 U (of the portable real world object 132 ).
  • This information 319 is then displayed, in act 305 , similar to act 105 described above, except that the information being displayed is specific to user 314 U as shown in FIGS. 3D and 3E .
  • the to-be-displayed information 319 may be obtained in act 304 from a website www.twitter.com, specific to the user's identity.
  • the to-be-displayed information 319 received by processor 100 is personalized for user 135 , based on user name and password authentication by the website www.twitter.com.
  • the personalized information 319 illustrated in one example is from www.twitter.com other websites can be used, e.g. an email website such as http://mail.yahoo.com can be used to obtain other such information personalized for user 135 .
  • user-supplied text (for use in preparing associations 311 ) is received by processor 100 via an authentication (also called login) screen.
  • an act 306 is performed by processor 100 after act 302 , to display an authentication screen.
  • authentication screen 321 is projected on to table 131 adjacent to object 132 as part of information 322 as shown in FIG. 3C .
  • FIG. 3C In the example illustrated in FIG.
  • processor 100 obtains the authentication screen to be displayed in act 306 from a computer (not shown) accessible on the Internet, such as a web server at the website www.twitter.com, and this screen is of a user interface such as a dialog box that prompts the user to enter their user name and password.
  • processor 100 automatically includes adjacent to such a dialog box, a layout image 333 of a keyboard in information 322 that is projected into scene 130 .
  • a layout image 333 of a keyboard in information 322 that is projected into scene 130 .
  • processor 100 recognizes additional images (similar to image 109 described above) and generates user input by performing Optical Character Recognition (OCR), and such user input in the form of text string(s) is stored in memory 110 and then sent to the website www.twitter.com.
  • OCR Optical Character Recognition
  • the same user input is also used in act 308 by some embodiments of processor 100 to identify the user (i.e.
  • a user name and password received as the user-supplied text may be locally checked against table 391 by processor 100 .
  • processor 100 performs act 309 .
  • processor 100 prepares an association 314 in set 311 , to associate an identifier 114 U of object 132 with an identifier 314 U (e.g. name) of the user identified in the authentication screen 321 (projected adjacent to object 132 ), in response to user input being authenticated.
  • acts 304 and 305 are performed as described above. Note that in other embodiments, a different user name and password may be used locally by processor 100 .
  • the user is authenticated two times, once by processor 100 locally (when the user enters their user name and password to login, into device 120 ), and another time by a remote computer (or web server) that supplies information 119 ( FIG. 5 ) to be output (e.g. at the website www.twitter.com).
  • a single authentication is sufficient, e.g. the user name and password that were used to log into device 120 are automatically used (directly or indirectly via a table lookup) to communicate with the remote computer (or web server), to obtain information 119 ( FIG. 5 ).
  • a user name that was used to log into device 120 is also used to identify a table 220 (among multiple such tables) used in identifying address 291 , for obtaining information 119 ( FIG. 5 )
  • User-specific information 319 that is obtained in act 304 is typically displayed as per act 305 at a location adjacent to object 132 ( FIG. 3D ) or alternatively on object 132 itself ( FIG. 3E ) in order to reduce the likelihood of snooping by users other than user 135 with whom object 132 is uniquely associated.
  • Prior to display (e.g. by projection) in act 305 such information may be optionally transformed.
  • a specific technique for transformation that is selected for projection of user-specific personalized information can depend on a number of factors, such as the smoothness and/or shape and size and/or a dynamically computed surface normal (or gradient) of object 132 , and/or resolution and legibility of information 119 that is to be projected, etc.
  • a transformation technique that is selected may also edit information 119 to be output, e.g. truncate or abbreviate the information, omit images, or down-scale images etc. depending on the embodiment.
  • FIG. 3F Although certain embodiments to implement strong identification described above use an authentication screen, other embodiments use two cameras to perform a method of the type illustrated in FIG. 3F , wherein one camera is embedded with a projector 122 in device 120 and other camera is a normal camera included in a mobile device such as a smart phone (or alternatively external thereto, in other form factors). Hence, in the method of FIG. 3F , two cameras are operated to synchronize (or use) hand gestures with a person's face, but otherwise many of the acts are similar or identical to the acts described above in reference to FIG. 3A .
  • a back-facing camera 121 in mobile device 150 captures an image 109 of scene 130 . Detection in such an image 109 of a portion that corresponds to the specific hand gesture (as per act 396 in FIG. 3F ) triggers processor 100 to perform act 397 to operate a front-facing camera 152 ( FIG. 1B ). Front-facing camera 152 ( FIG. 1B ) then captures an image including a face of the user 135 , and the image is then segmented (e.g. by module 141 S in FIG. 5 ) to obtain a portion corresponding to the face which is used (in module 141 S) by processor 100 performing act 398 to determine a user identifier (e.g. perform authentication).
  • a user identifier e.g. perform authentication
  • processor 100 of some embodiments compares the image portion corresponding to the face to a database 199 ( FIG. 2D ) of faces, and on finding a match obtains from the database an identifier of the user (selected from among user identifiers of faces in the database).
  • the user's face 120 is received by processor 100 from a front facing camera 152 and is detected as such in act 398 , thereby resulting in a unique identifier for the user that may be supplied to user input extractor 141 E for use in preparing an association, to associate the user identifier with an object identifier, in response to detecting a predetermined gesture adjacent to the object.
  • Illustrative embodiments of a segmentation module 141 S FIG.
  • act 309 is performed as described above in response to detection of the gesture, to prepare an association so that object 132 (e.g. bottle cap) is identified as belonging to person 135 .
  • object 132 e.g. bottle cap
  • an object identifier of portable real world object 132 in image 109 is automatically identified by processor 100 using data 445 (also called “object data”, see FIG. 4A ) on multiple real world objects in a database 199 that is coupled to processor 100 in the normal manner.
  • processor 100 recognizes object 132 to be a bottle cap (which is identified by an identifier ID 1 ), based on attributes in data 441 D ( FIG. 4A ) in database 199 matching attributes of a portion 132 I of image 109 ( FIG. 4A ) received as per act 431 ( FIG. 4B ).
  • processor 100 is programmed with software to operate as an object extractor 141 O (see FIG. 5 ) which determines feature vectors from an image 109 , and compares these feature vectors to corresponding feature vectors of objects that are previously computed and stored in a database 199 , to identify an object. Comparison between feature vectors can be done differently depending on the embodiment (e.g. using Euclidean distance).
  • object extractor 141 O identifies from database 199 an object 132 that most closely matches the feature vectors from image 109 , resulting in one or more object identifiers 112 O, 112 U ( FIG. 5 ).
  • processor 100 is programmed with software to identify clusters of features that vote for a common pose of an object (e.g. using the Hough transform).
  • bins that accumulate a preset minimum number of votes are identified by object extractor 141 O, as object 132 .
  • object extractor 141 O extract SIFT features as described in the preceding paragraph, while other embodiments use a method described by Viola and Jones in a 25-page article entitled “Robust Real-time Object Detection,” in the Second International Workshop On Statistical And Computational Theories Of Vision—Modeling, Learning, Computing, And Sampling, Vancouver, Canada, Jul. 13, 2001 that is incorporated by reference herein in its entirety.
  • features that are determined from an image 109 as described above are used in some embodiments by object extractor 141 O ( FIG. 5 ) to generate a geometric description of items in image 109 that is received from a camera, such as object 132 .
  • Similar or identical software may be used by processor 100 to extract from image 109 , a blob 136 I ( FIG. 2D ) of a finger in a hand gesture (and/or to recognize a user's face as described herein).
  • processor 100 use Haar features which consist of vector windows that are used to calculate edges, line features and center-surrounded features in an image 109 .
  • vector windows are run by processor 100 across portions of image 109 ( FIG.
  • processor 100 uses such features (also called “feature vectors”) differently depending upon the item to be recognized (object, or face, or hand gesture). Hence, depending on the embodiment, processor 100 uses vector windows that are different for objects, hand gestures, face recognition etc.
  • Use of Haar features by processor 100 in certain embodiments has limitations, such as robustness and low fps (frames per second) due to dependency on scaling and rotation of Haar vector windows.
  • processor 100 may be programmed with software to operate as object extractor 141 O ( FIG. 5 ) that uses other methods, such as a method described in an 8-page article entitled “Object Recognition from Local Scale-Invariant Features” by David G. Lowe, in Proceedings of the International Conference on Computer Vision, Corfu (September 1999), which is incorporated by reference herein in its entirety.
  • object extractor 141 O of the type described above
  • three dimensional (3D) surfaces of the object(s) are segmented by processor 100 into local regions with a curvature (or other such property) within a predetermined range, so that the regions are similar, relative to one another.
  • processor 100 may be optionally programmed to operate as information transformer 141 T ( FIG. 5 ) to truncate or otherwise manipulate information 119 , so that the information fits within local regions of object 132 identified by segmentation. Truncation or manipulation of content in information 119 by processor 100 reduces or eliminates the likelihood that projection of information 119 on to object 132 will irregularly wrap between local regions which may have surface properties different from one another.
  • Processor 100 of some embodiments segments a 3D surface of object 132 to identify local regions therein as described in a 14-page article entitled “Partitioning 3D Surface Meshes Using Watershed Segmentation” by Alan P. Mangan and Ross T. Whitaker, in IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 5, NO. 4, OCTOBER-DECEMBER 1999 which is incorporated by reference herein in its entirety.
  • processor 100 operating as information transformer 141 T calibrates camera 121 ( FIG. 5 ) using any calibration method, such as the method described in an a 4-page article entitled “Camera Calibration Toolbox for Matlab” at “http://www.vision.caltech. edu/ bouguetj/calib_doc/” as available on Apr. 10, 2012, which is incorporated by reference herein in its entirety.
  • information transformer 141 T FIG. 5
  • FIG. 5 are programmed to determine shapes and/or surface normals of surfaces of object 132 in image 109 , using one of the following two methods described in the next paragraph.
  • a first method uses a projection of light, e.g. as described in an 8-page article entitled “Dynamic scene shape reconstruction using a single structured light pattern” by Hiroshi Kawasaki et al, IEEE Conference on Computer Vision and Pattern Recognition 2008, which is incorporated by reference herein in its entirety.
  • a second method also uses a projection of light, e.g. as described in a 13-page article entitled “Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming” by Li Zhang et al, in Proc. Int. Symposium on 3D Data Processing Visualization and Transmission (3DPVT), 2002, which is incorporated by reference herein in its entirety.
  • certain other data 442 D and 443 D in object data 445 may include attributes to be used by object extractor 141 O ( FIG. 5 ) in identifying various other objects such as a object 132 ( FIG. 2C ) having identifier ID 2 ( FIG. 4A ) or a cup (not shown) having identifier ID 3 , in an image 109 analyzed by processor 100 in act 432 ( FIG. 4B ).
  • one or more of object identifiers ID 1 , ID 2 and ID 3 uniquely identify within processor 100 , corresponding portable real world objects, namely a bottle cap, a book and a cup when these objects are imaged in an image 109 of a scene 130 by a camera of device 120 .
  • image 109 illustrated in FIG. 4A includes a portion 132 I that corresponds to the entirety of object 132
  • another image that captures only a portion of object 132 may be sufficient for processor 100 to recognize object 132 (in act 432 of FIG. 4B ).
  • processor 100 may be programmed to operate as object extractor 141 O to perform act 432 to recognize additional information captured from object 132 , as described above.
  • each of associations 448 , 449 contains a single user's name as the user identifier associated with a corresponding object identifier.
  • certain embodiments may use both forms of identification in processor 100 operating as an information retriever 141 R ( FIG. 5 ) with different types of portable real world objects and/or different information display software 141 ( FIG. 4A ), depending on the programming of processor 100 ( FIG. 4B ).
  • associations 447 may be set up in different ways in database 199 , prior to their use by processor 100 , depending on the embodiment.
  • processor 100 is programmed with software to operate as information identifier 141 I ( FIG. 5 ) that extracts user input (in user input extractor 141 E) and uses associations to generate an address 291 of information to be output (in object-user-input mapper 141 M).
  • processor 100 is programmed with software to operate as user input extractor 141 E to process an image 109 received in act 431 or to process additional image(s) received in other such acts or to process other information captured from scene 130 (e.g.
  • processor 100 may return to act 432 (described above).
  • image 109 may include image portions corresponding to one or more human fingers, e.g. index finger 136 ( FIG. 1E ) and middle finger 137 are parts of a human hand 138 , of a person 135 .
  • Processor 100 is programmed to operate as user input extractor 141 E ( FIG. 5 ) in act 439 A to use predetermined information (not shown in FIG. 4A ) in database 199 to recognize in image 109 ( FIG. 4A ) in memory 110 certain user gestures (e.g. index and middle finger images 136 I and 137 I outstretched in human hand image 138 I), and then use a recognized gesture to identify person 135 (e.g. as belonging to a specific group), followed by identification of information to be output.
  • predetermined information not shown in FIG. 4A
  • predetermined information not shown in FIG. 4A
  • database 199 to recognize in image 109 ( FIG. 4A ) in memory 110 certain user gestures (e.g. index and middle finger images 136 I and 137 I
  • processor 100 may perform different acts after act 439 A, to identify user 135 as per act 433 , e.g. in user input extractor 141 E.
  • processor 100 is programmed to perform an act 439 D to recognize a face of the user 135 in another image from another camera.
  • mobile device 120 includes two cameras, namely a rear camera 121 that images object 132 and a front camera 152 that images a user's face.
  • face features feature vectors for human faces
  • gesture features feature vectors for hand gestures
  • processor 100 is programmed (by instructions in one or more non-transitory computer readable storage media, such as a disk or ROM) to compare a portion of an image segmented therefrom and corresponding to a face of user 138 , with a database of faces. A closest match resulting from the comparison identifies to processor 100 a user identifier, from among user identifiers of multiple faces in the database. Processor 100 then associates (e.g. in act 439 C in FIG. 4B ) this user identifier with an object identifier e.g. by user input extractor 141 E ( FIG. 5 ) preparing an association, in response to detection of a predetermined hand gesture adjacent to object 132 .
  • processor 100 is programmed (by instructions in one or more non-transitory computer readable storage media, such as a disk or ROM) to compare a portion of an image segmented therefrom and corresponding to a face of user 138 , with a database of faces. A closest match resulting from the comparison identifies
  • An act 439 D may be followed by processor 100 performing another additional act, such as 439 E to synchronize (or otherwise use) recognition of the user's face with the user gesture.
  • a user gesture recognized in act 439 A may be used to identify person 135 by user input extractor 141 E looking up a table 451 of FIG. 4A (similar to mapping 116 in FIG. 1D ) as per act 439 B ( FIG. 4B ).
  • processor 100 performs act 439 C (see FIG. 4B ) to associate a user identifier of person 135 with portable real world object 132 .
  • a user identifier that is used in act 439 C depends on whether strong or weak identification is implemented by processor 100 , for real world object 132 .
  • processor 100 is optionally programmed to operate as information transformer 141 T to perform act 435 .
  • processor 100 of mobile device 120 identifies from among a group of predetermined techniques 461 , 462 , a specific technique 461 to transform the obtained information 188 for projection into scene 130 .
  • transformation technique 461 FIG. 4A
  • transformation technique 462 is to project adjacent to object 132
  • one of these techniques is identified prior to projection of information 188 .
  • the specific technique 461 is selected (and therefore identified) automatically based on one or more surface properties of real world object 132 as determined by processor 100 , such as surface roughness (or smoothness), orientation of surface normal, color, opacity, etc, whereas in other embodiments processor 100 uses user input (e.g. in the form of spoken words) that explicitly identify a specific technique to be used.
  • processor 100 uses user input (e.g. in the form of spoken words) that explicitly identify a specific technique to be used.
  • information transformer 141 T uses the specific technique that was identified (e.g. on-object technique 461 ) to transform information 188 (or 189 ) in memory 110 , and then supply transformed information 188 T (or 189 T) resulting from use of the specific technique to a projector 122 ( FIG. 1E ).
  • projector 122 of mobile device 120 projects on object 132 in scene 130 the transformed information 188 T (or 189 T), which is thereby displayed on object 132 as illustrated in FIG. 1G .
  • information transformer 141 T may perform any steps using one or more transformation techniques 461 and 462 described above.
  • any other transformation technique can be used to prepare information 188 for output to a user.
  • information 189 may be transformed by another technique into a first component 189 A of transformed information 189 T that is projected onto object 132 (namely the text string “Group 1 ”), and a second component 189 B of the transformed information 189 T that is projected adjacent to object 132 (namely the text string “Score 0”).
  • processor 100 operates projector 122 ( FIG. 5 ) to project the transformed information 188 T (or 189 T) on to object 132 that is identified by the object identifier that was obtained in act 432 (described above).
  • processor 100 operating as user input extractor 141 E receives and processes additional images in a manner similar to that described above, e.g. to receive user input 482 and store it in memory 110 ( FIG. 4A ), by recognizing one or more parts of an additional image that includes transformed information 188 T, 189 T projected into scene 130 .
  • the parts of the additional image that are recognized may be, for example, another hand gesture 117 in which only one finger namely index finger 136 is outstretched as illustrated in FIG. 1H .
  • processor 100 operating as user input extractor 141 E may determine that the user is now part of Group 1 , and therefore now obtains information 189 of Group 1 (see FIG. 4A ) in act 434 ( FIG. 4B ), followed by output to the user (e.g. by transformation and projection) as per one or more of acts 435 - 437 described above.
  • processor 100 uses recognition of a bottle cap as the portable real world object 132 to invoke execution of information transformer 141 T (e.g. instructions to perform acts 431 - 438 and 439 A- 439 E) from among multiple such softwares 141 O, 141 I, 141 R and 141 T.
  • software 141 is generic to multiple objects, although in other embodiments software 141 (also called information display software) is customized for and individually associated with corresponding objects, such as, for example, a book software 442 S ( FIG. 4A ) associated with a book identified by ID 2 and a cup software 443 S ( FIG. 4A ) associated with a cup identified by ID 3 as described above in reference to FIG. 4A .
  • table 451 is described above for use with user input in the form of a hand gesture to identify a user, such a table 451 can alternatively be looked up by processor 100 using an identifier of object 132 , depending on how table 451 is set up, in various embodiments described herein.
  • Use of table 451 with object identifiers enables “strong” identification in some embodiments of information identifier 141 I, wherein a person 135 identifies to processor 100 (ahead of time), an object 132 that is to be uniquely associated with his/her identity.
  • Other embodiments of processor 100 use both an object identifier as well as user input to look up another table 220 , which enables “weak” identification as described herein.
  • Some embodiments implement one or more acts of FIG. 4B by performing one or more acts of the type described below in reference to FIGS. 4C and 4D . Note that the acts of FIGS. 4C and 4D described below can alternatively be performed in other embodiments that do not perform any act of FIG. 4B .
  • Some embodiments of processor 100 are programmed to operate as object extractor 141 O to track portable real world objects that may be temporarily occluded from view of rear-facing camera 121 that captures image 109 , by performing an act 411 ( FIG. 4C ) to check the image 109 for presence of each object in a set of objects (in database 199 of FIG. 4A ) and to identify a subset of these objects as being initially present in image 109 .
  • processor 100 of some embodiments adds an identifier of an object 132 in the subset to a list 498 ( FIG. 4A ), and starts a timer for that identifier, and the timer starts incrementing automatically from 0 at a preset frequency (e.g. every millisecond). Therefore, if there are multiple identifiers in list 498 for multiple objects, then correspondingly multiple timers are started, by repeated performance of act 412 for each object in the subset.
  • processor 100 checks if list 498 is empty. As list 498 was just populated, it is not empty at this stage and therefore processor 100 performs act 414 ( FIG. 4C ) and then returns to act 413 . Whenever list 498 becomes empty, processor goes from act 413 via the yes branch to act 411 (described above).
  • additional images are captured into memory 110 , and processed by processor 100 in the manner described herein.
  • processor 100 scans through list 498 to check if each object identified in the list 498 is found in the additional image.
  • processor 100 resets the timer (started in act 412 as noted above) which starts incrementing automatically again from 0.
  • processor 100 removes the identifier from the list and stops the corresponding timer.
  • an object 132 when an object 132 is absent from view of camera 121 for more than the preset limit, the object 132 is no longer used in the manner described above, to retrieve and display information.
  • use of a timer as illustrated in FIG. 4C and described above reduces the likelihood that a display of information (e.g. the user's emails) is interrupted when object 132 that triggered the information display is accidentally occluded from view of camera 121 , e.g. for the period of time identified in the preset limit.
  • the preset limit of time in some embodiments is set by processor 100 , based on one or more input(s) from user 135 , and hence a different value can be set by user 135 depending on location e.g. a first limit used at a user's home (or office) and a second limit (lower than the first limit) in a public location.
  • processor 100 are optionally programmed to operate as information transformer 141 T to perform act 421 ( FIG. 4D ) to compute a value of a property of object 132 identified in image 109 , such as size of a surface, shape of the surface, orientation of surface normal, surface smoothness, etc. Then in act 422 , processor 100 checks if the property's value satisfies a predetermined test on feasibility for projection on to object 132 .
  • processor 100 may check if the surface area of a surface of object 132 is large enough to accommodate the to-be-projected information, and/or if object 132 has a color that is sufficiently neutral for use as a display, and/or a normal at the surface of object 132 is oriented within a preset range relative to a projector 122 .
  • Such feasibility tests are designed ahead of time, and programmed into processor 100 to ensure that the to-be-projected information is displayed in manner suitable for user 135 e.g. font size is legible. Numerous other feasibility tests, for information projection on to an object 132 in real world, will be readily apparent, in view of this detailed description.
  • processor 100 goes to act 423 and uses a first technique 461 ( FIG. 4A ) to generate transformed information 119 T ( FIG. 5 ) for projection on to object 132 , followed by act 437 ( FIG. 4B ).
  • first technique 461 may transform information 119 based on an orientation (in the three angles, pitch, yaw and roll) of a surface of object 132 relative to orientation of projector 122 to ensure legibility when information 119 T is rendered on object 132 .
  • a specific manner in which information 119 is transformed can be different in different embodiments, and in some embodiments there is no transformation e.g. when the information 119 is to be displayed on a screen of mobile device 120 (as shown in FIG.
  • processor 100 goes to act 424 and uses a second technique 462 ( FIG. 4A ) to generate transformed information 119 T for projection adjacent to (but not on to) object 132 . After performing one of acts 423 and 424 , processor 100 then goes to act 437 (described above in reference to FIG. 4B ).
  • one or more of acts 421 - 424 described above are performed as described in U.S. application Ser. No. ______, Attorney Docket No. Q111570U2os, filed concurrently herewith, and entitled “Dynamic Selection of Surfaces In Real World For Projection of Information Thereon” which has been incorporated by reference above.
  • processor 100 may be partially or wholly included in one or more other processor(s) and/or other computer(s) that interoperate(s) with such a mobile device 120 , e.g. by exchanging information therewith via a cellular link or a WiFi link.
  • processor(s) and/or other computer(s) that interoperate(s) with such a mobile device 120 , e.g. by exchanging information therewith via a cellular link or a WiFi link.
  • one camera 121 is shown in FIG. 1E , depending on the embodiment, one or more cameras (see FIG. 5 ) may be used.
  • FIG. 4B While certain acts illustrated in FIG. 4B are described for some embodiments as being performed by mobile device 120 , some or all of acts in FIG. 4B may be performed by use of one or more computers and/or one or more processors and/or one or more cameras. Therefore, it is to be understood that several such embodiments use one or more devices to perform such act(s), either alone or in combination with one another.
  • Mobile device 120 may be any device that includes a projector 122 and/or a camera 121 , and device 120 may include additional parts that are normally used in any hand held device, e.g. sensors, such as accelerometers, gyroscopes or the like, which may be used in one or more acts described above, e.g. in determining the pose of mobile device 120 relative to object 132 in the real world.
  • sensors such as accelerometers, gyroscopes or the like, which may be used in one or more acts described above, e.g. in determining the pose of mobile device 120 relative to object 132 in the real world.
  • User input 482 that is generated from images captured by camera 121 ( FIG. 1E ) allows a user to reach into scene 130 and manipulate real world object 132 directly, as opposed to on-screen based interaction, where users interact by directly touching a screen 151 of mobile device 150 ( FIG. 1B ).
  • image-based user supplied information is obtained as input
  • methods of the type described above in reference to FIG. 4B enable a user to use his hands in scene 130 with information projected into the real world, as the user is supplying input which changes the information being projected into scene 130 .
  • projected user interfaces in information projected into a scene as described herein can have a broad range of applications.
  • projected user interfaces can be used to generate user input 482 ( FIG. 4A ) by projecting information 322 including screen 321 and keyboard image 333 ( FIG. 3C ) similar to real world typing using a real keyboard.
  • a projected user interface allows a user to supply input to select between different software for execution and display of projected information and/or select between different groups of users to play games, and in general to specify various parameters to software being executed by a processor 100 that generates the information which is projected into scene 130 (e.g. see FIG. 1G ).
  • mobile device 120 as described herein reverse a flow of information between (A) user interfaces and (B) user input (relative to a conventional flow).
  • device 120 instead of users explicitly looking for information to be displayed, several embodiments of device 120 automatically obtain and display interactive information, e.g. by projection on real world surfaces using an embedded mobile projector.
  • Other embodiments may display information as described herein on a screen that is supported on an eye-glass frame worn by a user, for example.
  • Still other embodiments may display information as described herein on a screen that forms an integral portion of a smart phone (such as a touch screen), in the normal manner.
  • mobile device 120 may be used to receive input from a user, e.g. an IR camera may be used to receive user input in the form of hand gestures.
  • IR camera may be used to receive user input in the form of hand gestures.
  • hand gesture recognition systems may be implemented in several embodiments of a mobile device 120 as described herein.
  • an embedded projector in mobile device 120 projects a cell phone's normal display on everyday surfaces such as a surface of a wall or a surface of a desk, with which a user interacts using hand gestures.
  • mobile device 120 may be any electronic device that is portable by hand, such as a cellular or other wireless communication device, personal communication system (PCS) device, personal navigation device (PND), Personal Information Manager (PIM), Personal Digital Assistant (PDA), laptop, tablet, or an eye glass frame that supports a display to be worn on a person's face, a headset, a camera, or other suitable mobile device that is capable of imaging scene 130 and/or projecting information into scene 130 .
  • a single device 120 includes both camera 121 and projector 122 whereas in other embodiments one such device includes camera 121 and another such device includes projector 122 and both devices communicate with one another either directly or via a computer (not shown).
  • a prototype of mobile device 120 is built with custom hardware (PCB) board taped onto the back of a smartphone (e.g. GALAXY NEXUS available from Samsung Electronics Co. Ltd).
  • PCB custom hardware
  • One such approach performs computation in the infrared (IR) spectrum.
  • IR infrared
  • hand and body tracking is robust and very accurate, although additional hardware may be integrated within such a smartphone 120 used for display of information.
  • Some embodiments of device 120 use IR sensors (e.g. in an IR camera) that have been proven to work on commercially successful platforms, such as the Xbox Kinect.
  • Certain embodiments of device 120 implement augmented reality (AR) applications using marker patterns, such as checkerboard for camera calibration and detection of objects within a scene of real world, followed by use of object identifiers to display information, as described herein.
  • AR augmented reality
  • mobile device 120 may be programmed with software 141 that uses a mobile-projector system in combination with a camera.
  • An embedded projector 122 is used in such embodiments to display information 119 T on everyday surfaces such as a wall, with which users interact using hand gestures.
  • mobile device 120 is operatively coupled to an external IR camera 1006 that tracks an IR laser stylus (not shown), or gloves with one or more IR LEDs 1121 , 1122 ( FIG. 5 ) mounted at the finger tips (also called IR gloves).
  • External IR camera 1006 is used in some embodiments in a manner similar or identical to receipt of IR images and tracking of objects within the images by use of an IR camera in a remote control device (also called “Wiimote” or “Wii Remote”) for gaming console Wii, available from Nintendo Co. Ltd. So, IR camera 1006 may be used in some embodiments as described in a section entitled “Tracking Your Fingers with the Wiimote” in a 2-page article available at http://johnnylee.net/projects/ wii/ as available on Apr. 9, 2012, which is incorporated by reference herein in its entirety. Alternatively, some non-IR embodiments of device 120 use one or more normal RGB (red-green-blue) CMOS cameras 121 ( FIG. 5 ) to capture an image of scene 130 including object 132 .
  • RGB red-green-blue
  • An object extractor 141 O in a mobile device 120 of the type described herein may use any known object recognition method, based on “computer vision” techniques.
  • a mobile device 120 may also include means for controlling operation of a real world object 132 (that may be electronic) in response to user input of the type described above such as a toy equipped with an IR or RF transmitter or a wireless a transmitter enabled to receive and/or transmit one or more signals over one or more types of wireless communication networks such as the Internet, WiFi, cellular wireless network or other network.
  • mobile device 120 may additionally include a graphics engine 1004 to generate information 119 to be output, an image processor 1005 to process image(s) 109 and/or transform information 119 , and a read only memory (ROM) 1007 to store firmware and/or constant data.
  • Mobile device 120 may also include a disk 1008 to store software and/or database 199 for use by processor 100 .
  • Mobile device 120 may further include a wireless transmitter and receiver 1010 and/or any other communication interfaces 1009 , sensors 1003 , a touch screen 1001 or other screen 1002 , a speaker 1111 and a microphone 1112 .
  • Some embodiments of user input extractor 141 E sense user input in the form of hand gestures in images generated by an infra-red (IR) camera that tracks an IR laser stylus or gloves with IR LEDs (also called IR gloves), while certain other embodiments of user input extractor 141 E sense hand gestures using an existing camera (e.g. in a normal cell phone) that captures images of a user's fingers.
  • IR infra-red
  • an existing camera e.g. in a normal cell phone
  • an external PCB board 1130 FIG. 5
  • an ARM Cortex processor (not shown) is interfaced with an IR camera 1006 ( FIG. 5 ) and a Bluetooth module (not shown).
  • Hand tracking data from IR camera 1006 is sent via Bluetooth to a smartphone 1140 in device 120 that has a touch screen 1001 (e.g. HTC Explorer available from HTC Corporation).
  • PCB board 1130 mounted on and operatively coupled to a smartphone 1140 ( FIG. 5 ).
  • PCB board 1130 includes IR camera 1006 , such as mbed LPC1768 available from Foxconn, e.g. Mbed as described at http://mbed.org/nxp/lpc1768/ and a Bluetooth chipset e.g. BlueSMiRF Gold as described at http://www.sparkfun.com/products/582.
  • Mbed ARM processor
  • user input captured in images by a camera is extracted therefrom by the smartphone in device 120 performing gesture recognition on data received from an infra-red (IR) sensor, as described in an article entitled “iGesture: A General Gesture Recognition Framework” by Signer et al, In Proc. ICDAR '07, 5 pages which is incorporated by reference herein in its entirety.
  • IR infra-red
  • some embodiments of user input extractor 141 E operate with a user wearing infra-red (IR) gloves that are identified in another image generated by an IR camera.
  • An IR camera of such embodiments may be externally coupled to a smartphone in mobile device 120 in some embodiments while in other embodiments the IR camera is built into the smartphone.
  • Some embodiments operate with the user 135 using an IR laser stylus 1135 whose coordinates are detected by device 120 in any manner known in the art.
  • Still other embodiments of user input extractor 141 E ( FIG. 5 ) receive user input in other forms as noted above, e.g. as audio input from microphone 1112 .
  • user input extractor 141 E processes a frame of video captured by a camera to obtain user input in the form of hand gestures, by segmenting each image into one or more areas of interest, such as a user's hands. Any known method can be modified for use in user input extractor 141 E as described herein, to remove background noise, followed by identification of a portion of the image which contains the user's hand, which is then used to generate a binary image (also called a “blob”). A next step in some embodiments of user input extractor 141 E is to calculate locations (e.g. coordinates) of the user's fingers within the blob.
  • An IR camera 1006 is not used in certain embodiments wherein a normal RGB camera is used instead to generate one or more images 109 which contain user input.
  • the user input is extracted from images 109 by user input extractor 141 E ( FIG. 5 ) performing one of two methods as follows.
  • a first method is of the type described in a 4-page article entitled “HandVu: Vision-based Hand Gesture Recognition and User Interface” at http://www.movesinstitute.org/ ⁇ kolsch /HandVu/HandVu.html” as available on Apr. 9, 2012, which is incorporated by reference herein in its entirety.
  • a second method is of the type described in another 4-page article entitled “A Robust Method for Hand Gesture Segmentation and Recognition Using Forward Spotting Scheme in Conditional Random Fields” by Mahmoud Elmezain, Ayoub Al-Hamadi, and Bernd Michaelis, in International Conference on Pattern Recognition, 2010, which is incorporated by reference herein in its entirety.
  • Such embodiments that use an existing RGB camera in a normal smartphone may use a combination of skin segmentation, graph cut and recognition of hand movement, to detect hand gestures.
  • user input extractor 141 E For recognition of hand gestures, some embodiments of user input extractor 141 E ( FIG. 5 ) are designed to use a supervised learning approach in an initialization phase of device 120 .
  • supervised learning approach user input extractor 141 E learns different gestures from input binary images (e.g. consisting of a user's hands) during initialization, and generates a mathematical model to be used to identify gestures in images generated during normal operation (after the initialization phase) by using Support Vector Machines (SVM) of the type known in the prior art.
  • SVM Support Vector Machines
  • IR camera 1006 uses an infrared (IR) camera 1006 ( FIG. 5 ) to extract portions of an image 109 that correspond to a user's fingers as blobs.
  • a user holds an IR light source, such as a laser pointer or alternatively the user wears IR gloves.
  • a user wears on a hand 138 ( FIG. 1E ) a glove (not shown) with an IR LED 1121 , 1122 ( FIG. 5 ) on each finger 136 , 137 ( FIG. 1E ). Detection of position of one IR LED 1121 ( FIG. 5 ) on left index finger 136 ( FIG. 1E ) by IR camera 1006 ( FIG.
  • gesture recognition is performed by processor 100 executing software to operate as user input extractor 141 E ( FIG. 5 ) as described herein.
  • processor 100 executing software to operate as user input extractor 141 E ( FIG. 5 ) as described herein.
  • co-ordinates of IR LEDs 1121 and 1122 generated by IR camera 1006 are used by processor 100 to identify a blob (e.g. human hand image 138 I in FIG.
  • Some embodiments extract blobs in two-dimensional (2D) space due to limitations inherent in design (for a similar setup, see the 2-page article described above in reference to http://johnnylee.net/projects/wii/).
  • Certain embodiments of user input extractor 141 E ( FIG. 5 ) perform blob detection in 3D space, using a depth camera.
  • images from an IR camera 1006 are used to translate hand movements to specific co-ordinates within native applications running on a smartphone included in device 120 .
  • Several embodiments implement a simple camera calibration technique, similar to the techniques described in the 2-page article at http://johnnylee.net/projects/wii/.
  • mobile device 120 generate a depth map of a scene by use of a 3D Time-of-Flight (TOF) camera of the type known in the prior art.
  • TOF Time-of-Flight
  • a Time-of-Flight camera is used in certain embodiments, to measure a phase difference between photons coming onto a sensor, which in turn provides a distance between the sensor and objects in the scene.
  • device 120 also use Time-of-flight cameras, e.g. as described by Freedman in US Patent Publication 2010/0118123 entitled “Depth Mapping using Projected Patterns” which is incorporated by reference herein in its entirety.
  • Such embodiments of device 120 may use projector 122 ( FIG. 5 ) to shine an infrared (IR) light pattern on object 132 in the scene.
  • IR infrared
  • a reflected light pattern is observed in such embodiments by a depth camera 121 , which generates a depth map.
  • a depth map is used to enhance segmentation of image to identify areas that contain a user's face, the user's hand and/or one or more objects in an image received from an RGB camera 121 (which is thereby additionally used).
  • a device 120 of some embodiments uses projector 122 to project information 119 T onto a surface of object 132 , followed by capture of a user's finger movements as follows.
  • a laser pointer or gloves with IR LEDs on fingers
  • the IR camera implements a motion capture system.
  • an IR camera 1006 is calibrated. After camera calibration, certain embodiments generate a one-to-one mapping between the screen resolution of device 120 and the user's hand movements.
  • the IR camera 1006 captures the brightest IR point.
  • the coordinates of such points are processed inside an application layer or kernel layer. Based on the processed data, the user input is determined by processor 100 .
  • one of the methods is based on tracking IR light sources, e.g. IR LEDs 1121 and 1122 ( FIG. 5 ) mounted at finger tips of a glove.
  • IR light sources e.g. IR LEDs 1121 and 1122 ( FIG. 5 ) mounted at finger tips of a glove.
  • One limitation of some embodiments is a need for external hardware (i.e. hardware not present in a conventional smartphone), such as an IR camera 1006 ( FIG. 5 ).
  • Certain embodiments of mobile device 120 use one or more Infrared time-of-flight (TOF) camera(s) 1006 instead of or in addition to a CMOS infrared camera 1006 .
  • TOF Infrared time-of-flight
  • background noise may be present in images 109 being captured and filtered by device 120 .
  • Such embodiments may utilize a frame buffer of a screen in mobile device 120 and perform stereo correspondence to reduce such noise.
  • Several embodiments of device 120 implement any known techniques to reduce background noise arising from use of stereo cameras (e.g. 3D cameras) 121 .
  • Mobile device 120 of several described embodiments may also include means for remotely controlling a real world object which may be a toy, in response to user input e.g. by use of transmitter in transceiver 1010 , which may be an IR or RF transmitter or a wireless a transmitter enabled to transmit one or more signals over one or more types of wireless communication networks such as the Internet, WiFi, cellular wireless network or other network.
  • transceiver 1010 which may be an IR or RF transmitter or a wireless a transmitter enabled to transmit one or more signals over one or more types of wireless communication networks such as the Internet, WiFi, cellular wireless network or other network.
  • mobile device 120 may include other elements, such as a read-only-memory 1007 which may be used to store firmware for use by processor 100 .
  • any one or more of object extractor 141 O, information identifier 141 I, information retriever 141 R, information transformer 141 T and segmentation module 141 S illustrated in FIG. 5 and described above can, but need not necessarily include, one or more microprocessors, embedded processors, controllers, application specific integrated circuits (ASICs), digital signal processors (DSPs), and the like.
  • processor is intended to describe the functions implemented by the system rather than specific hardware.
  • memory refers to any type of computer storage medium, including long term, short term, or other memory associated with the mobile platform, and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.
  • methodologies described herein may be implemented by various means depending upon the application. For example, these methodologies may be implemented in firmware in ROM 1007 ( FIG. 5 ) or software, or hardware or any combination thereof.
  • the processing units may be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.
  • ASICs application specific integrated circuits
  • DSPs digital signal processors
  • DSPDs digital signal processing devices
  • PLDs programmable logic devices
  • FPGAs field programmable gate arrays
  • processors controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.
  • the methodologies may be implemented with modules (e.g., procedures, functions
  • software 141 may include program codes stored in memory 110 and executed by processor 100 .
  • Memory 110 may be implemented within or external to the processor 100 . If implemented in firmware and/or software, the functions may be stored as one or more computer instructions or code on a computer-readable medium. Examples include nontransitory computer-readable storage media encoded with a data structure (such as a sequence of images) and computer-readable media encoded with a computer program (such as software 141 that can be executed to perform the method of FIGS. 1C , 2 A, 3 A, 3 F, and 4 B- 4 D).
  • Computer-readable media includes physical computer storage media.
  • a storage medium may be any available medium that can be accessed by a computer.
  • such computer-readable media can comprise RAM, ROM, Flash Memory, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store program code in the form of software instructions (also called “processor instructions” or “computer instructions”) or data structures and that can be accessed by a computer;
  • disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
  • item 120 shown in FIG. 5 of some embodiments is a mobile device
  • item 120 is implemented by use of form factors that are different, e.g. in certain other embodiments item 120 is a mobile platform (such as a tablet, e.g. iPad available from Apple, Inc.) while in still other embodiments item 120 is any electronic device or system.
  • a mobile platform such as a tablet, e.g. iPad available from Apple, Inc.
  • item 120 is any electronic device or system.
  • Illustrative embodiments of such an electronic device or system 120 may include multiple physical parts that intercommunicate wirelessly, such as a processor and a memory that are portions of a stationary computer, such as a lap-top computer, a desk-top computer, or a server computer communicating over one or more wireless link(s) with sensors and user input circuitry enclosed in a housing that is small enough to be held in a hand.
  • a processor and a memory that are portions of a stationary computer, such as a lap-top computer, a desk-top computer, or a server computer communicating over one or more wireless link(s) with sensors and user input circuitry enclosed in a housing that is small enough to be held in a hand.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Library & Information Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Transforming Electric Information Into Light Information (AREA)
  • Image Analysis (AREA)
US13/549,339 2011-08-19 2012-07-13 Use of association of an object detected in an image to obtain information to display to a user Abandoned US20130044912A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US13/549,339 US20130044912A1 (en) 2011-08-19 2012-07-13 Use of association of an object detected in an image to obtain information to display to a user
PCT/US2012/046816 WO2013028279A1 (en) 2011-08-19 2012-07-14 Use of association of an object detected in an image to obtain information to display to a user

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161525628P 2011-08-19 2011-08-19
US13/549,339 US20130044912A1 (en) 2011-08-19 2012-07-13 Use of association of an object detected in an image to obtain information to display to a user

Publications (1)

Publication Number Publication Date
US20130044912A1 true US20130044912A1 (en) 2013-02-21

Family

ID=47712374

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/549,339 Abandoned US20130044912A1 (en) 2011-08-19 2012-07-13 Use of association of an object detected in an image to obtain information to display to a user
US13/549,388 Active 2033-07-24 US9245193B2 (en) 2011-08-19 2012-07-13 Dynamic selection of surfaces in real world for projection of information thereon

Family Applications After (1)

Application Number Title Priority Date Filing Date
US13/549,388 Active 2033-07-24 US9245193B2 (en) 2011-08-19 2012-07-13 Dynamic selection of surfaces in real world for projection of information thereon

Country Status (7)

Country Link
US (2) US20130044912A1 (ja)
EP (1) EP2745237B1 (ja)
JP (2) JP2014531787A (ja)
KR (1) KR101575016B1 (ja)
CN (1) CN103875004B (ja)
IN (1) IN2014MN00316A (ja)
WO (2) WO2013028279A1 (ja)

Cited By (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120051625A1 (en) * 2010-08-23 2012-03-01 Texas Instruments Incorporated Method and Apparatus for 2D to 3D Conversion Using Scene Classification and Face Detection
US20140075349A1 (en) * 2012-09-10 2014-03-13 Samsung Electronics Co., Ltd. Transparent display apparatus and object selection method using the same
US20140125580A1 (en) * 2012-11-02 2014-05-08 Samsung Electronics Co., Ltd. Method and device for providing information regarding an object
US20140185872A1 (en) * 2012-12-28 2014-07-03 Hyundai Motor Company Method and system for recognizing hand gesture using selective illumination
WO2014153120A1 (en) * 2013-03-14 2014-09-25 George Martin Methods and apparatus for message playback
US20140325459A1 (en) * 2004-02-06 2014-10-30 Nokia Corporation Gesture control system
US20150022444A1 (en) * 2012-02-06 2015-01-22 Sony Corporation Information processing apparatus, and information processing method
US20150062046A1 (en) * 2013-09-03 2015-03-05 Samsung Electronics Co., Ltd. Apparatus and method of setting gesture in electronic device
US20150092981A1 (en) * 2013-10-01 2015-04-02 Electronics And Telecommunications Research Institute Apparatus and method for providing activity recognition based application service
US20150193088A1 (en) * 2013-07-15 2015-07-09 Intel Corporation Hands-free assistance
US20150208244A1 (en) * 2012-09-27 2015-07-23 Kyocera Corporation Terminal device
US20150219500A1 (en) * 2012-09-11 2015-08-06 Barco N.V. Projection system with safety detection
US20150227198A1 (en) * 2012-10-23 2015-08-13 Tencent Technology (Shenzhen) Company Limited Human-computer interaction method, terminal and system
US20150234477A1 (en) * 2013-07-12 2015-08-20 Magic Leap, Inc. Method and system for determining user input based on gesture
US20150339860A1 (en) * 2014-05-26 2015-11-26 Kyocera Document Solutions Inc. Article information providing apparatus that provides information of article, article information providing system,and article information provision method
US9245193B2 (en) 2011-08-19 2016-01-26 Qualcomm Incorporated Dynamic selection of surfaces in real world for projection of information thereon
US20160116983A1 (en) * 2014-10-23 2016-04-28 Samsung Electronics Co., Ltd. User input method for use in portable device using virtual input area
US9344615B1 (en) * 2015-01-26 2016-05-17 International Business Machines Corporation Discriminating visual recognition program for digital cameras
CN105993038A (zh) * 2014-02-07 2016-10-05 皇家飞利浦有限公司 操作控制系统的方法及因此的控制系统
US9560272B2 (en) 2014-03-24 2017-01-31 Samsung Electronics Co., Ltd. Electronic device and method for image data processing
US9563955B1 (en) * 2013-05-15 2017-02-07 Amazon Technologies, Inc. Object tracking techniques
US9612403B2 (en) 2013-06-11 2017-04-04 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US20170126691A1 (en) * 2014-09-18 2017-05-04 International Business Machines Corporation Dynamic multi-user computer configuration settings
US9671566B2 (en) 2012-06-11 2017-06-06 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US20170278304A1 (en) * 2016-03-24 2017-09-28 Qualcomm Incorporated Spatial relationships for integration of visual images of physical environment into virtual reality
US9886769B1 (en) * 2014-12-09 2018-02-06 Jamie Douglas Tremaine Use of 3D depth map with low and high resolution 2D images for gesture recognition and object tracking systems
US20180039479A1 (en) * 2016-08-04 2018-02-08 Adobe Systems Incorporated Digital Content Search and Environmental Context
US9913117B2 (en) * 2013-06-27 2018-03-06 Samsung Electronics Co., Ltd Electronic device and method for exchanging information using the same
US10096166B2 (en) 2014-11-19 2018-10-09 Bae Systems Plc Apparatus and method for selectively displaying an operational environment
US10139961B2 (en) * 2016-08-18 2018-11-27 Microsoft Technology Licensing, Llc Touch detection using feature-vector dictionary
US10216273B2 (en) * 2015-02-25 2019-02-26 Bae Systems Plc Apparatus and method for effecting a control action in respect of system functions
US20190076731A1 (en) * 2017-09-08 2019-03-14 Niantic, Inc. Methods and Systems for Generating Detailed Datasets of an Environment via Gameplay
US10262465B2 (en) 2014-11-19 2019-04-16 Bae Systems Plc Interactive control station
US10277945B2 (en) * 2013-04-05 2019-04-30 Lenovo (Singapore) Pte. Ltd. Contextual queries for augmenting video display
US10430559B2 (en) 2016-10-18 2019-10-01 Adobe Inc. Digital rights management in virtual and augmented reality
US10506221B2 (en) 2016-08-03 2019-12-10 Adobe Inc. Field of view rendering control of digital content
US10521967B2 (en) 2016-09-12 2019-12-31 Adobe Inc. Digital content interaction and navigation in virtual and augmented reality
US10825254B1 (en) * 2019-05-30 2020-11-03 International Business Machines Corporation Augmented reality book selection-assist
US10896219B2 (en) * 2017-09-13 2021-01-19 Fuji Xerox Co., Ltd. Information processing apparatus, data structure of image file, and non-transitory computer readable medium
CN112673393A (zh) * 2018-08-29 2021-04-16 天阶金融科技有限公司 使用增强现实显示提供一个或多个服务的系统和方法
US11073901B2 (en) * 2015-07-07 2021-07-27 Seiko Epson Corporation Display device, control method for display device, and computer program
CN113577766A (zh) * 2021-08-05 2021-11-02 百度在线网络技术（北京）有限公司 对象处理方法及装置
US11221819B2 (en) * 2014-03-31 2022-01-11 Amazon Technologies, Inc. Extendable architecture for augmented reality system
US20220075856A1 (en) * 2016-05-19 2022-03-10 Payfone Inc., D/B/A Prove Identifying and authenticating users based on passive factors determined from sensor data
US11461820B2 (en) 2016-08-16 2022-10-04 Adobe Inc. Navigation and rewards involving physical goods and services
US11474671B2 (en) * 2020-01-31 2022-10-18 Salesforce.Com, Inc. Neutralizing designs of user interfaces
US11543933B2 (en) * 2014-05-14 2023-01-03 Purdue Research Foundation Manipulating virtual environment using non-instrumented physical object
US12032668B2 (en) * 2021-11-15 2024-07-09 Prove Identity, Inc. Identifying and authenticating users based on passive factors determined from sensor data

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9456187B1 (en) 2012-06-01 2016-09-27 Amazon Technologies, Inc. Edge-based pose detection
US10528853B1 (en) * 2012-06-29 2020-01-07 Amazon Technologies, Inc. Shape-Based Edge Detection
US20170026636A1 (en) * 2013-12-12 2017-01-26 Testo Ag Method for the positionally accurate projection of a mark onto an object, and projection apparatus
US20150193915A1 (en) * 2014-01-06 2015-07-09 Nvidia Corporation Technique for projecting an image onto a surface with a mobile device
US9207780B2 (en) * 2014-01-27 2015-12-08 Fuji Xerox Co., Ltd. Systems and methods for hiding and finding digital content associated with physical objects via coded lighting
US20170223321A1 (en) * 2014-08-01 2017-08-03 Hewlett-Packard Development Company, L.P. Projection of image onto object
US9715865B1 (en) * 2014-09-26 2017-07-25 Amazon Technologies, Inc. Forming a representation of an item with light
US20180013998A1 (en) 2015-01-30 2018-01-11 Ent. Services Development Corporation Lp Relationship preserving projection of digital objects
JP6711284B2 (ja) * 2015-02-03 2020-06-17 ソニー株式会社 情報処理装置、情報処理方法およびプログラム
CN111857332A (zh) 2015-02-12 2020-10-30 北京三星通信技术研究有限公司 获取笔记信息的方法和装置
CN106033257B (zh) * 2015-03-18 2019-05-31 联想(北京)有限公司 一种控制方法和装置
WO2016151869A1 (en) * 2015-03-23 2016-09-29 Nec Corporation Information processing apparatus, information processing method, and program
CN104796678A (zh) * 2015-04-28 2015-07-22 联想(北京)有限公司 信息处理方法及电子设备
WO2017072008A1 (en) * 2015-10-30 2017-05-04 Unilever Plc Hair curl measurement
CN108351970B (zh) * 2015-10-30 2023-04-04 联合利华知识产权控股有限公司 毛发直径测量
US9799111B2 (en) * 2016-02-11 2017-10-24 Symbol Technologies, Llc Methods and systems for highlighting box surfaces and edges in mobile box dimensioning
US10249084B2 (en) 2016-06-10 2019-04-02 Microsoft Technology Licensing, Llc Tap event location with a selection apparatus
US10720082B1 (en) * 2016-09-08 2020-07-21 Ctskh, Llc Device and system to teach stem lessons using hands-on learning method
KR102048674B1 (ko) * 2017-07-31 2019-11-26 코닉오토메이션 주식회사 조명램프형 멀티미디어 장치
EP3669260A4 (en) 2017-12-04 2021-03-24 Hewlett-Packard Development Company, L.P. PERIPHERAL DISPLAY DEVICES
DE102018203349A1 (de) * 2018-03-07 2019-09-12 BSH Hausgeräte GmbH Interaktionsmodul
TWI724858B (zh) * 2020-04-08 2021-04-11 國軍花蓮總醫院 基於手勢動作的混合實境評量系統

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090237328A1 (en) * 2008-03-20 2009-09-24 Motorola, Inc. Mobile virtual and augmented reality system
US20100082629A1 (en) * 2008-09-29 2010-04-01 Yahoo! Inc. System for associating data items with context
US20100083373A1 (en) * 2008-09-29 2010-04-01 Scott White Methods and apparatus for determining user authorization from motion of a gesture-based control unit
US20100153457A1 (en) * 2008-12-15 2010-06-17 Grant Isaac W Gestural Interface Device and Method
US20110154266A1 (en) * 2009-12-17 2011-06-23 Microsoft Corporation Camera navigation for presentations
US20110213664A1 (en) * 2010-02-28 2011-09-01 Osterhout Group, Inc. Local advertising content on an interactive head-mounted eyepiece
US20120162254A1 (en) * 2010-12-22 2012-06-28 Anderson Glen J Object mapping techniques for mobile augmented reality applications
US20120242800A1 (en) * 2011-03-23 2012-09-27 Ionescu Dan Apparatus and system for interfacing with computers and other electronic devices through gestures by using depth sensing and methods of use
US20130004016A1 (en) * 2011-06-29 2013-01-03 Karakotsios Kenneth M User identification by gesture recognition
US20130050069A1 (en) * 2011-08-23 2013-02-28 Sony Corporation, A Japanese Corporation Method and system for use in providing three dimensional user interface
US20130050258A1 (en) * 2011-08-25 2013-02-28 James Chia-Ming Liu Portals: Registered Objects As Virtualized, Personalized Displays
US20130174213A1 (en) * 2011-08-23 2013-07-04 James Liu Implicit sharing and privacy control through physical behaviors using sensor-rich devices
US20130207962A1 (en) * 2012-02-10 2013-08-15 Float Hybrid Entertainment Inc. User interactive kiosk with three-dimensional display
US20130272574A1 (en) * 2000-11-06 2013-10-17 Nant Holdings Ip, Llc Interactivity Via Mobile Image Recognition
US20130285894A1 (en) * 2012-04-27 2013-10-31 Stefan J. Marti Processing image input to communicate a command to a remote display device
US20130311329A1 (en) * 2012-03-29 2013-11-21 Digimarc Corporation Image-related methods and arrangements
US8646000B2 (en) * 2009-12-04 2014-02-04 Lg Electronics Inc. Augmented remote controller and method for operating the same

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE3578241D1 (de) 1985-06-19 1990-07-19 Ibm Verfahren zum identifizieren dreidimensionaler objekte mittels zweidimensionaler bilder.
JP3869897B2 (ja) 1997-01-28 2007-01-17 キヤノン株式会社 カメラ制御システムおよび映像受信装置および制御方法および記憶媒体
JP2001211372A (ja) 2000-01-27 2001-08-03 Nippon Telegr & Teleph Corp <Ntt> 映像投影装置
JP4009851B2 (ja) 2002-05-20 2007-11-21 セイコーエプソン株式会社 投写型画像表示システム、プロジェクタ、プログラム、情報記憶媒体および画像投写方法
JP4591720B2 (ja) 2002-05-20 2010-12-01 セイコーエプソン株式会社 投写型画像表示システム、プロジェクタ、プログラム、情報記憶媒体および画像投写方法
US6811264B2 (en) * 2003-03-21 2004-11-02 Mitsubishi Electric Research Laboratories, Inc. Geometrically aware projector
US7515756B2 (en) 2003-06-23 2009-04-07 Shoestring Research, Llc. Region segmentation and characterization systems and methods for augmented reality
JP2005313291A (ja) 2004-04-30 2005-11-10 Mitsubishi Heavy Ind Ltd ロボット行動とリンクした映像表示方法とその装置
US20070050468A1 (en) 2005-08-09 2007-03-01 Comverse, Ltd. Reality context menu (RCM)
JP2007142495A (ja) 2005-11-14 2007-06-07 Nippon Telegr & Teleph Corp <Ntt> 平面投影装置、及び平面投影プログラム
US7905610B1 (en) 2006-08-29 2011-03-15 Nvidia Corporation Graphics processor system and associated method for projecting an image onto a three-dimensional object
KR100775123B1 (ko) 2006-09-15 2007-11-08 삼성전자주식회사 영상 객체 인덱싱 방법 및 이를 이용한 영상 객체 인덱싱시스템
US8493496B2 (en) 2007-04-02 2013-07-23 Primesense Ltd. Depth mapping using projected patterns
US8228170B2 (en) 2008-01-10 2012-07-24 International Business Machines Corporation Using sensors to identify objects placed on a surface
JP5258399B2 (ja) 2008-06-06 2013-08-07 キヤノン株式会社 画像投影装置及びその制御方法
US7954953B2 (en) * 2008-07-30 2011-06-07 Microvision, Inc. Scanned beam overlay projection
US8385971B2 (en) 2008-08-19 2013-02-26 Digimarc Corporation Methods and systems for content processing
JP2010072025A (ja) 2008-09-16 2010-04-02 Nikon Corp プロジェクタ付電子機器
US9569001B2 (en) * 2009-02-03 2017-02-14 Massachusetts Institute Of Technology Wearable gestural interface
KR20110071349A (ko) * 2009-12-21 2011-06-29 삼성전자주식회사 휴대단말의 외부 출력 제어 방법 및 장치
US8549418B2 (en) * 2009-12-23 2013-10-01 Intel Corporation Projected display to enhance computer device use
US8631355B2 (en) * 2010-01-08 2014-01-14 Microsoft Corporation Assigning gesture dictionaries
CN101907459B (zh) * 2010-07-12 2012-01-04 清华大学 基于单目视频的实时三维刚体目标姿态估计与测距方法
CN102959616B (zh) 2010-07-20 2015-06-10 苹果公司 自然交互的交互真实性增强
WO2012020410A2 (en) * 2010-08-10 2012-02-16 Pointgrab Ltd. System and method for user interaction with projected content
US9560314B2 (en) * 2011-06-14 2017-01-31 Microsoft Technology Licensing, Llc Interactive and shared surfaces
US20130044912A1 (en) 2011-08-19 2013-02-21 Qualcomm Incorporated Use of association of an object detected in an image to obtain information to display to a user

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130272574A1 (en) * 2000-11-06 2013-10-17 Nant Holdings Ip, Llc Interactivity Via Mobile Image Recognition
US20090237328A1 (en) * 2008-03-20 2009-09-24 Motorola, Inc. Mobile virtual and augmented reality system
US20100082629A1 (en) * 2008-09-29 2010-04-01 Yahoo! Inc. System for associating data items with context
US20100083373A1 (en) * 2008-09-29 2010-04-01 Scott White Methods and apparatus for determining user authorization from motion of a gesture-based control unit
US20100153457A1 (en) * 2008-12-15 2010-06-17 Grant Isaac W Gestural Interface Device and Method
US8646000B2 (en) * 2009-12-04 2014-02-04 Lg Electronics Inc. Augmented remote controller and method for operating the same
US20110154266A1 (en) * 2009-12-17 2011-06-23 Microsoft Corporation Camera navigation for presentations
US20110213664A1 (en) * 2010-02-28 2011-09-01 Osterhout Group, Inc. Local advertising content on an interactive head-mounted eyepiece
US20120162254A1 (en) * 2010-12-22 2012-06-28 Anderson Glen J Object mapping techniques for mobile augmented reality applications
US20120242800A1 (en) * 2011-03-23 2012-09-27 Ionescu Dan Apparatus and system for interfacing with computers and other electronic devices through gestures by using depth sensing and methods of use
US20130004016A1 (en) * 2011-06-29 2013-01-03 Karakotsios Kenneth M User identification by gesture recognition
US20130174213A1 (en) * 2011-08-23 2013-07-04 James Liu Implicit sharing and privacy control through physical behaviors using sensor-rich devices
US20130050069A1 (en) * 2011-08-23 2013-02-28 Sony Corporation, A Japanese Corporation Method and system for use in providing three dimensional user interface
US20130050258A1 (en) * 2011-08-25 2013-02-28 James Chia-Ming Liu Portals: Registered Objects As Virtualized, Personalized Displays
US20130207962A1 (en) * 2012-02-10 2013-08-15 Float Hybrid Entertainment Inc. User interactive kiosk with three-dimensional display
US20130311329A1 (en) * 2012-03-29 2013-11-21 Digimarc Corporation Image-related methods and arrangements
US20130285894A1 (en) * 2012-04-27 2013-10-31 Stefan J. Marti Processing image input to communicate a command to a remote display device

Cited By (94)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140325459A1 (en) * 2004-02-06 2014-10-30 Nokia Corporation Gesture control system
US20120051625A1 (en) * 2010-08-23 2012-03-01 Texas Instruments Incorporated Method and Apparatus for 2D to 3D Conversion Using Scene Classification and Face Detection
US8718356B2 (en) * 2010-08-23 2014-05-06 Texas Instruments Incorporated Method and apparatus for 2D to 3D conversion using scene classification and face detection
US9245193B2 (en) 2011-08-19 2016-01-26 Qualcomm Incorporated Dynamic selection of surfaces in real world for projection of information thereon
US20150022444A1 (en) * 2012-02-06 2015-01-22 Sony Corporation Information processing apparatus, and information processing method
US10401948B2 (en) * 2012-02-06 2019-09-03 Sony Corporation Information processing apparatus, and information processing method to operate on virtual object using real object
US9671566B2 (en) 2012-06-11 2017-06-06 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US20140075349A1 (en) * 2012-09-10 2014-03-13 Samsung Electronics Co., Ltd. Transparent display apparatus and object selection method using the same
US9965137B2 (en) * 2012-09-10 2018-05-08 Samsung Electronics Co., Ltd. Transparent display apparatus and object selection method using the same
US20150219500A1 (en) * 2012-09-11 2015-08-06 Barco N.V. Projection system with safety detection
US9677945B2 (en) * 2012-09-11 2017-06-13 Barco N.V. Projection system with safety detection
US9801068B2 (en) * 2012-09-27 2017-10-24 Kyocera Corporation Terminal device
US20150208244A1 (en) * 2012-09-27 2015-07-23 Kyocera Corporation Terminal device
US20150227198A1 (en) * 2012-10-23 2015-08-13 Tencent Technology (Shenzhen) Company Limited Human-computer interaction method, terminal and system
US9836128B2 (en) * 2012-11-02 2017-12-05 Samsung Electronics Co., Ltd. Method and device for providing information regarding an object
US20140125580A1 (en) * 2012-11-02 2014-05-08 Samsung Electronics Co., Ltd. Method and device for providing information regarding an object
US20140185872A1 (en) * 2012-12-28 2014-07-03 Hyundai Motor Company Method and system for recognizing hand gesture using selective illumination
US9373026B2 (en) * 2012-12-28 2016-06-21 Hyundai Motor Company Method and system for recognizing hand gesture using selective illumination
US10540863B2 (en) 2013-03-14 2020-01-21 Martigold Enterprises, Llc Delayed message playback methods and apparatus
US11568717B2 (en) 2013-03-14 2023-01-31 Martigold Enterprises, Llc Varied apparatus for message playback
WO2014153120A1 (en) * 2013-03-14 2014-09-25 George Martin Methods and apparatus for message playback
US9626843B2 (en) 2013-03-14 2017-04-18 Martigold Enterprises, Llc Varied message playback methods and apparatus
US10277945B2 (en) * 2013-04-05 2019-04-30 Lenovo (Singapore) Pte. Ltd. Contextual queries for augmenting video display
US10671846B1 (en) 2013-05-15 2020-06-02 Amazon Technologies, Inc. Object recognition techniques
US9563955B1 (en) * 2013-05-15 2017-02-07 Amazon Technologies, Inc. Object tracking techniques
US11412108B1 (en) 2013-05-15 2022-08-09 Amazon Technologies, Inc. Object recognition techniques
US9612403B2 (en) 2013-06-11 2017-04-04 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US9913117B2 (en) * 2013-06-27 2018-03-06 Samsung Electronics Co., Ltd Electronic device and method for exchanging information using the same
US10591286B2 (en) 2013-07-12 2020-03-17 Magic Leap, Inc. Method and system for generating virtual rooms
US10288419B2 (en) 2013-07-12 2019-05-14 Magic Leap, Inc. Method and system for generating a virtual user interface related to a totem
US10866093B2 (en) 2013-07-12 2020-12-15 Magic Leap, Inc. Method and system for retrieving data in response to user input
US9651368B2 (en) 2013-07-12 2017-05-16 Magic Leap, Inc. Planar waveguide apparatus configured to return light therethrough
US11221213B2 (en) 2013-07-12 2022-01-11 Magic Leap, Inc. Method and system for generating a retail experience using an augmented reality system
US10295338B2 (en) 2013-07-12 2019-05-21 Magic Leap, Inc. Method and system for generating map data from an image
US10767986B2 (en) 2013-07-12 2020-09-08 Magic Leap, Inc. Method and system for interacting with user interfaces
US11656677B2 (en) 2013-07-12 2023-05-23 Magic Leap, Inc. Planar waveguide apparatus with diffraction element(s) and system employing same
US10641603B2 (en) 2013-07-12 2020-05-05 Magic Leap, Inc. Method and system for updating a virtual world
US10352693B2 (en) 2013-07-12 2019-07-16 Magic Leap, Inc. Method and system for obtaining texture data of a space
US20150234477A1 (en) * 2013-07-12 2015-08-20 Magic Leap, Inc. Method and system for determining user input based on gesture
US9857170B2 (en) 2013-07-12 2018-01-02 Magic Leap, Inc. Planar waveguide apparatus having a plurality of diffractive optical elements
US10408613B2 (en) 2013-07-12 2019-09-10 Magic Leap, Inc. Method and system for rendering virtual content
US11060858B2 (en) 2013-07-12 2021-07-13 Magic Leap, Inc. Method and system for generating a virtual user interface related to a totem
US10473459B2 (en) 2013-07-12 2019-11-12 Magic Leap, Inc. Method and system for determining user input based on totem
US9952042B2 (en) 2013-07-12 2018-04-24 Magic Leap, Inc. Method and system for identifying a user location
US10228242B2 (en) * 2013-07-12 2019-03-12 Magic Leap, Inc. Method and system for determining user input based on gesture
US10571263B2 (en) 2013-07-12 2020-02-25 Magic Leap, Inc. User and object interaction with an augmented reality scenario
US11029147B2 (en) 2013-07-12 2021-06-08 Magic Leap, Inc. Method and system for facilitating surgery using an augmented reality system
US10533850B2 (en) 2013-07-12 2020-01-14 Magic Leap, Inc. Method and system for inserting recognized object data into a virtual world
US10495453B2 (en) 2013-07-12 2019-12-03 Magic Leap, Inc. Augmented reality system totems and methods of using same
US20150193088A1 (en) * 2013-07-15 2015-07-09 Intel Corporation Hands-free assistance
CN105308535A (zh) * 2013-07-15 2016-02-03 英特尔公司 无需用手的协助
US20150062046A1 (en) * 2013-09-03 2015-03-05 Samsung Electronics Co., Ltd. Apparatus and method of setting gesture in electronic device
KR20150039252A (ko) * 2013-10-01 2015-04-10 한국전자통신연구원 행동 인식 기반의 응용 서비스 제공 장치 및 그 방법
US20150092981A1 (en) * 2013-10-01 2015-04-02 Electronics And Telecommunications Research Institute Apparatus and method for providing activity recognition based application service
US9183431B2 (en) * 2013-10-01 2015-11-10 Electronics And Telecommunications Research Institute Apparatus and method for providing activity recognition based application service
KR102106135B1 (ko) 2013-10-01 2020-05-04 한국전자통신연구원 행동 인식 기반의 응용 서비스 제공 장치 및 그 방법
CN105993038A (zh) * 2014-02-07 2016-10-05 皇家飞利浦有限公司 操作控制系统的方法及因此的控制系统
US20170177073A1 (en) * 2014-02-07 2017-06-22 Koninklijke Philips N.V. Method of operating a control system and control system therefore
US10191536B2 (en) * 2014-02-07 2019-01-29 Koninklijke Philips N.V. Method of operating a control system and control system therefore
US9560272B2 (en) 2014-03-24 2017-01-31 Samsung Electronics Co., Ltd. Electronic device and method for image data processing
US11221819B2 (en) * 2014-03-31 2022-01-11 Amazon Technologies, Inc. Extendable architecture for augmented reality system
US11543933B2 (en) * 2014-05-14 2023-01-03 Purdue Research Foundation Manipulating virtual environment using non-instrumented physical object
US9626804B2 (en) * 2014-05-26 2017-04-18 Kyocera Document Solutions Inc. Article information providing apparatus that provides information of article, article information providing system,and article information provision method
US20150339860A1 (en) * 2014-05-26 2015-11-26 Kyocera Document Solutions Inc. Article information providing apparatus that provides information of article, article information providing system,and article information provision method
US10320799B2 (en) * 2014-09-18 2019-06-11 International Business Machines Corporation Dynamic multi-user computer configuration settings
US20170126691A1 (en) * 2014-09-18 2017-05-04 International Business Machines Corporation Dynamic multi-user computer configuration settings
US20160116983A1 (en) * 2014-10-23 2016-04-28 Samsung Electronics Co., Ltd. User input method for use in portable device using virtual input area
US9727131B2 (en) * 2014-10-23 2017-08-08 Samsung Electronics Co., Ltd. User input method for use in portable device using virtual input area
US10262465B2 (en) 2014-11-19 2019-04-16 Bae Systems Plc Interactive control station
US10096166B2 (en) 2014-11-19 2018-10-09 Bae Systems Plc Apparatus and method for selectively displaying an operational environment
US9886769B1 (en) * 2014-12-09 2018-02-06 Jamie Douglas Tremaine Use of 3D depth map with low and high resolution 2D images for gesture recognition and object tracking systems
US9497376B2 (en) * 2015-01-26 2016-11-15 International Business Machines Corporation Discriminating visual recognition program for digital cameras
US9344615B1 (en) * 2015-01-26 2016-05-17 International Business Machines Corporation Discriminating visual recognition program for digital cameras
US10216273B2 (en) * 2015-02-25 2019-02-26 Bae Systems Plc Apparatus and method for effecting a control action in respect of system functions
US11073901B2 (en) * 2015-07-07 2021-07-27 Seiko Epson Corporation Display device, control method for display device, and computer program
US11301034B2 (en) 2015-07-07 2022-04-12 Seiko Epson Corporation Display device, control method for display device, and computer program
US10665019B2 (en) * 2016-03-24 2020-05-26 Qualcomm Incorporated Spatial relationships for integration of visual images of physical environment into virtual reality
US20170278304A1 (en) * 2016-03-24 2017-09-28 Qualcomm Incorporated Spatial relationships for integration of visual images of physical environment into virtual reality
US20220075856A1 (en) * 2016-05-19 2022-03-10 Payfone Inc., D/B/A Prove Identifying and authenticating users based on passive factors determined from sensor data
US10506221B2 (en) 2016-08-03 2019-12-10 Adobe Inc. Field of view rendering control of digital content
US20180039479A1 (en) * 2016-08-04 2018-02-08 Adobe Systems Incorporated Digital Content Search and Environmental Context
US11461820B2 (en) 2016-08-16 2022-10-04 Adobe Inc. Navigation and rewards involving physical goods and services
US10139961B2 (en) * 2016-08-18 2018-11-27 Microsoft Technology Licensing, Llc Touch detection using feature-vector dictionary
US10521967B2 (en) 2016-09-12 2019-12-31 Adobe Inc. Digital content interaction and navigation in virtual and augmented reality
US10430559B2 (en) 2016-10-18 2019-10-01 Adobe Inc. Digital rights management in virtual and augmented reality
US11110343B2 (en) 2017-09-08 2021-09-07 Niantic, Inc. Methods and systems for generating detailed datasets of an environment via gameplay
US10300373B2 (en) * 2017-09-08 2019-05-28 Niantic, Inc. Methods and systems for generating detailed datasets of an environment via gameplay
US20190076731A1 (en) * 2017-09-08 2019-03-14 Niantic, Inc. Methods and Systems for Generating Detailed Datasets of an Environment via Gameplay
US10896219B2 (en) * 2017-09-13 2021-01-19 Fuji Xerox Co., Ltd. Information processing apparatus, data structure of image file, and non-transitory computer readable medium
CN112673393A (zh) * 2018-08-29 2021-04-16 天阶金融科技有限公司 使用增强现实显示提供一个或多个服务的系统和方法
US10825254B1 (en) * 2019-05-30 2020-11-03 International Business Machines Corporation Augmented reality book selection-assist
US11474671B2 (en) * 2020-01-31 2022-10-18 Salesforce.Com, Inc. Neutralizing designs of user interfaces
CN113577766A (zh) * 2021-08-05 2021-11-02 百度在线网络技术（北京）有限公司 对象处理方法及装置
US12032668B2 (en) * 2021-11-15 2024-07-09 Prove Identity, Inc. Identifying and authenticating users based on passive factors determined from sensor data

Also Published As

Publication number Publication date
JP6273334B2 (ja) 2018-01-31
JP2017038397A (ja) 2017-02-16
EP2745237A2 (en) 2014-06-25
CN103875004A (zh) 2014-06-18
US9245193B2 (en) 2016-01-26
CN103875004B (zh) 2017-12-08
WO2013028280A3 (en) 2013-04-18
US20130044193A1 (en) 2013-02-21
WO2013028280A2 (en) 2013-02-28
IN2014MN00316A (ja) 2015-09-11
EP2745237B1 (en) 2022-09-07
WO2013028279A1 (en) 2013-02-28
KR101575016B1 (ko) 2015-12-07
JP2014531787A (ja) 2014-11-27
KR20140047733A (ko) 2014-04-22

Similar Documents

Publication Publication Date Title
US20130044912A1 (en) Use of association of an object detected in an image to obtain information to display to a user
US20220334646A1 (en) Systems and methods for extensions to alternative control of touch-based devices
US10761612B2 (en) Gesture recognition techniques
US10936080B2 (en) Systems and methods of creating a realistic displacement of a virtual object in virtual reality/augmented reality environments
US10120454B2 (en) Gesture recognition control device
US9160993B1 (en) Using projection for visual recognition
US9658695B2 (en) Systems and methods for alternative control of touch-based devices
US8549418B2 (en) Projected display to enhance computer device use
US20180292907A1 (en) Gesture control system and method for smart home
US20180218545A1 (en) Virtual content scaling with a hardware controller
US20170255450A1 (en) Spatial cooperative programming language
US20130010207A1 (en) Gesture based interactive control of electronic equipment
US8938124B2 (en) Computer vision based tracking of a hand
US20200142495A1 (en) Gesture recognition control device
US11709593B2 (en) Electronic apparatus for providing a virtual keyboard and controlling method thereof
EP2702464B1 (en) Laser diode modes
KR20210033394A (ko) 전자 장치 및 그의 제어 방법
US11054941B2 (en) Information processing system, information processing method, and program for correcting operation direction and operation amount
US12032746B2 (en) Systems and methods of creating a realistic displacement of a virtual object in virtual reality/augmented reality environments

Legal Events

Date Code Title Description
AS Assignment

Owner name: QUALCOMM INCORPORATED, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KULKARNI, TEJAS DATTATRAYA;LIU, BOCONG;NANDWANI, ANKUR B;AND OTHERS;SIGNING DATES FROM 20120801 TO 20120807;REEL/FRAME:028761/0604

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION