US20200301502A1 - Multiparty object recognition - Google Patents
Multiparty object recognition Download PDFInfo
- Publication number
- US20200301502A1 US20200301502A1 US16/899,518 US202016899518A US2020301502A1 US 20200301502 A1 US20200301502 A1 US 20200301502A1 US 202016899518 A US202016899518 A US 202016899518A US 2020301502 A1 US2020301502 A1 US 2020301502A1
- Authority
- US
- United States
- Prior art keywords
- party
- object data
- data
- modified
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004048 modification Effects 0.000 claims abstract description 38
- 238000012986 modification Methods 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000004422 calculation algorithm Methods 0.000 claims description 41
- 238000004590 computer program Methods 0.000 claims description 14
- 238000004891 communication Methods 0.000 claims description 10
- 230000003993 interaction Effects 0.000 claims description 7
- 230000000153 supplemental effect Effects 0.000 claims description 6
- 239000000654 additive Substances 0.000 claims description 4
- 230000000996 additive effect Effects 0.000 claims description 4
- 238000013475 authorization Methods 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 238000003909 pattern recognition Methods 0.000 claims description 4
- 230000002123 temporal effect Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 2
- 230000003190 augmentative effect Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000001514 detection method Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 230000002085 persistent effect Effects 0.000 description 6
- 101100126955 Arabidopsis thaliana KCS2 gene Proteins 0.000 description 5
- 241000132023 Bellis perennis Species 0.000 description 5
- 235000005633 Chrysanthemum balsamita Nutrition 0.000 description 5
- 210000001525 retina Anatomy 0.000 description 5
- 230000005672 electromagnetic field Effects 0.000 description 4
- 239000002516 radical scavenger Substances 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 239000000126 substance Substances 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012804 iterative process Methods 0.000 description 2
- 230000007774 longterm Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/213—Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
- A63F13/215—Input arrangements for video game devices characterised by their sensors, purposes or types comprising means for detecting acoustic signals, e.g. using a microphone
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
- A63F13/65—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor automatically by game devices or servers from real world data, e.g. measurement in live racing competition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04845—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation, expansion or change of colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
- H04L65/403—Arrangements for multi-party communication, e.g. for conferences
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/024—Multi-user, collaborative environment
Definitions
- the invention relates to virtual representation technologies, in particular as they relate to modification of virtual objects and object representation attributes.
- the inventive subject matter provides apparatuses, systems and methods in which data related to real-world objects can be manipulated in a cooperative manner by a plurality of users.
- Data related to a first real-world object and a second real-world object can be stored in a database that is accessible by two or more users via a portal engine.
- the portal engine permits a user to access and modify data related to a first object, and further supports presentation of the modified first object data and data related to a second object to a different user.
- the data related to the first object can be modified in the context of data related to the second object.
- interactively manipulating virtual object data wherein an object database is configured to store first party object data, associated with a first party, that corresponds to a first real-world object and is further configured to store second party object data, associated with a second party, that corresponds to a second real-world object, includes obtaining the first party object data and the second party object data for storage within the object database.
- the first party object data and the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data.
- Access to the object database is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party.
- Modification of the first party object data by the second party is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data, and the modified first party object data is communicated to the first party.
- the at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, and a geometric context of the second party object data.
- the modified first party object data may be usable for presentation within a collaborative user environment, and the modified first party object data may be communicated to the first party in real-time.
- the first party and the second party may include at least one of a person, a group of people, and an automated device.
- the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device.
- At least one of the first party object data and second party object data is analyzed, and analyzing at least one of the first party object data and second party object data may include matching features of the first party object data and the second party object data using at least one of an image recognition algorithm, a pattern recognition algorithm, a speech recognition algorithm, a user matching algorithm, and a location recognition algorithm.
- At least one of the first party object data and second party object data may comprise metadata.
- the metadata may comprise at least one of first party-specific data or second party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data.
- At least one of the first party object data and the second party object data may be obtained by using an image recognition algorithm on a digital representation of at least one of the first real-world object and the second real-world object.
- At least one of the first party object data and the second party object data may be obtained based on at least one of an identification of object data by a user, and an instruction from another device.
- the first real-world object may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture.
- the visible object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object.
- the audible object may comprise at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal.
- the first real-world object and second real-world object may comprise different aspects of a single real-world object.
- the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data, creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data.
- the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data.
- the modified first party object data may comprise at least one of the second party object data, and object data from a source other than the first party and the second party.
- the object database may be accessible to at least the first party and the second party via a user portal, and the user portal may comprise at least one of a mobile device, an appliance, a vehicle, a robot, a kiosk, a television, and a game console.
- a user interface comprising at least one of a graphical user interface, a voice-controlled interface, and a motion-controlled interface, may be presented for access to the object database.
- access to the object database may be controlled such that the object database is accessible to a party other than the first party and second party. In some embodiments, access to the object database may be controlled based on at least one of a classification of a real-world object, a proximity metric, a subscription status, and an authorization.
- the modified first party object data may be stored within the object database, wherein the modified first object data may be further modifiable to generate nth-order modified first object data.
- the modified first party object data may be communicated to the second party.
- FIG. 1 illustrates an exemplary network environment that may be used for multiparty object recognition systems and methods in accordance with various embodiments.
- FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment.
- FIG. 3 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.
- FIG. 4A illustrates obtaining first party object data in accordance with an embodiment.
- FIG. 4B illustrates obtaining second party object data in accordance with an embodiment.
- FIG. 5 illustrates modification of the first party object data by the second party in accordance with an embodiment.
- FIG. 6 illustrates communicating the modified first party object data to the first party in accordance with an embodiment.
- FIG. 7 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.
- FIG. 8 illustrates a block diagram of a distributed computer system that can be used for implementing one or more aspects of the various embodiments.
- Coupled to is intended to include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements). Therefore, the terms “coupled to” and “coupled with” are used synonymously. Within the context of a networked environment where two or more components or devices are able to exchange data, the terms “coupled to” and “coupled with” are also used to mean “communicatively coupled with”, possibly via one or more intermediary devices.
- inventive subject matter provides many example embodiments of the inventive subject matter. Although some of the various embodiments presented herein constitute a single combination of inventive elements, it should be appreciated that the inventive subject matter is considered to include all possible combinations of the disclosed elements. As such, if one embodiment comprises elements A, B, and C, and another embodiment comprises elements B and D, then the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly discussed herein.
- any language directed to a computer should be read to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing device structures operating individually or collectively.
- computing devices comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, FPGA, PLA, solid state drive, RAM, flash, ROM, etc.).
- the software instructions configure or program the computing device to provide the roles, responsibilities, or other functionality as discussed below with respect to the disclosed apparatus.
- the disclosed technologies can be embodied as a computer program product that includes a non-transitory computer readable medium storing the software instructions that causes a processor to execute the disclosed steps associated with implementations of computer-based algorithms, processes, methods, or other instructions.
- the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods.
- Data exchanges among devices can be conducted over a packet-switched network, such as the Internet, a LAN, WAN, VPN, or other type of packet-switched network, a circuit-switched network, cell-switched network; or other type of network.
- the numbers expressing quantities of ingredients, properties such as concentration, reaction conditions, and so forth, used to describe and claim certain embodiments of the invention are to be understood as being modified in some instances by the term “about.” Accordingly, in some embodiments, the numerical parameters set forth in the description and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable. The numerical values presented in some embodiments of the invention may contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.
- a computing device is configured to allow data related to real-world objects to be manipulated in a cooperative manner by two or more users.
- the computing device is able to process substantial amounts of digital data well beyond the capability of a human being.
- FIG. 1 illustrates an exemplary network environment that may be used for multiparty object recognition systems and methods in accordance with various embodiments.
- a multiparty object recognition system for interactively manipulating object data can be implemented within a networked computing environment 100 (e.g., a cloud-computing environment).
- Networked computing environment 100 comprises at least one network 102 .
- network 102 is the Internet.
- network 102 may comprise one or more of a number of different types of networks, such as, for example, an intranet, a local area network (LAN), a wide area network (WAN), a wireless network, or other type of packet-switched network, circuit-switched network or cell-switched network. Other networks may be used. Alternatively, network 102 may comprise a combination of different types of networks.
- networks such as, for example, an intranet, a local area network (LAN), a wide area network (WAN), a wireless network, or other type of packet-switched network, circuit-switched network or cell-switched network.
- LAN local area network
- WAN wide area network
- wireless network or other type of packet-switched network, circuit-switched network or cell-switched network.
- Other networks may be used.
- network 102 may comprise a combination of different types of networks.
- one or more servers 106 A-B and/or client devices 104 A-E can implement a multiparty object recognition system.
- client devices 104 A-E can include mobile devices (such as laptop computers, tablets, or smartphones), appliances, vehicles, robots, kiosks or other types of public terminals, televisions, digital media players, game consoles, or other types of computing devices.
- a client device may be associated with a party (user) such that, for example, client devices 104 A and 104 B may be associated with a first party and a second party, respectively.
- one or more client devices 104 A-E and/or servers 106 A-B can host or be in communication with apparatuses, e.g., portal engine 110 and object database 120 , to implement a multiparty object recognition system that is accessible to client devices 104 A-E.
- object database 120 may be configured to store first party object data, associated with a first party, that corresponds to a first real-world object and be further configured to store second party object data, associated with a second party, that corresponds to a second real-world object.
- Portal engine 110 may be configured to obtain the first party object data and the second party object data from any of client devices 104 A-E and servers 106 A-B for storage within object database 120 .
- portal engine 110 may be configured to obtain object data, including object data associated with a first party (i.e., first party object data) and a second party (i.e., second party object data), that corresponds to real-world objects depicted in a digital representation (e.g., a digital image or a video stream) using one or more image recognition algorithms, or based on identification instructions provided by a user or an external device.
- object data including object data associated with a first party (i.e., first party object data) and a second party (i.e., second party object data), that corresponds to real-world objects depicted in a digital representation (e.g., a digital image or a video stream) using one or more image recognition algorithms, or based on identification instructions provided by a user or an external device.
- Portal engine 110 may be further configured to control access to object database 120 such that the first party object data and the second party object data is accessible to the first party (e.g., at client device 104 A) and the second party (e.g., at client device 104 B), facilitate modification of the first party object data by the second party to generate modified first party object data that is in accordance with at least one context parameter of the second party object data, and communicate the modified first party object data to the first party, e.g., at client device 104 A or another client device accessible by the first party.
- object database 120 such that the first party object data and the second party object data is accessible to the first party (e.g., at client device 104 A) and the second party (e.g., at client device 104 B), facilitate modification of the first party object data by the second party to generate modified first party object data that is in accordance with at least one context parameter of the second party object data, and communicate the modified first party object data to the first party, e.g., at client device 104 A or another client
- object database 120 is communicatively coupled to portal engine 110 .
- portal engine 110 is configured to generate a user portal to object database 120 that may be accessible to multiple parties via client devices 104 A-E.
- the user portal can be programmed or configured by the portal engine 110 to provide access to data stored in object database 120 .
- object database 120 and portal engine 110 can be remote from each other as well as from the one or more client devices 104 A-E used to access object database 120 , e.g., via the user portal.
- one or more of object database 120 and portal engine 110 can be housed in one or more server computers 106 A-B located remotely from client devices 104 A-E.
- the server(s) (or other computing device(s)) hosting portal engine 110 can be located remotely from server computers or other non-transitory computer-readable storage media hosting object database 120 .
- one or more of object database 120 and portal engine 110 can be integral to one or more of client devices 104 A-E.
- one of the client devices may host portal engine 110 and/or object database 120 , such that the other client devices 104 B-E are clients to the host client device.
- various functions described herein as being performed by portal engine 110 and object database 120 may be distributed between a plurality of client devices 104 A-E, a plurality of servers 106 A-B, or any combination of client devices and servers.
- FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment.
- elements for multiparty object recognition include portal engine 110 , object database 120 , and client devices 104 A-B in communication with portal engine 110 via network 102 .
- portal engine 110 portal engine 110
- object database 120 object database 120
- client devices 104 A-B client devices 104 A-B in communication with portal engine 110 via network 102 .
- FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment.
- elements for multiparty object recognition include portal engine 110 , object database 120 , and client devices 104 A-B in communication with portal engine 110 via network 102 .
- FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment.
- elements for multiparty object recognition include portal engine 110 , object database 120 , and client devices 104 A-B in communication with portal engine 110 via network 102 .
- FIG. 2 illustrates apparatuses that may be used for multiparty
- interactively manipulating virtual object data wherein object database 120 is configured to store first party object data, associated with a first party, that corresponds to a first real-world object and is further configured to store second party object data, associated with a second party, that corresponds to a second real-world object
- portal engine 110 being configured to obtain the first party object data and the second party object data for storage within object database 120 .
- the first party object data and the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data.
- the first party object data and second party object data may comprise metadata.
- the metadata may comprise at least one of first party-specific data or second-party-specific object data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data.
- portal engine 110 may be configured to obtain at least one of the first party object data and the second party object data by using an image recognition algorithm on a digital representation of at least one of the first real-world object and the second real-world object.
- the feature detection algorithm may include at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.
- Portal engine 110 also may be configured to obtain at least one of the first party object data and the second party object data based on at least one of an identification of object data by a user, and an instruction
- the first real-world object and second real-world object may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture.
- the visible object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object
- the audible object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal.
- the first real-world object and second real-world object may comprise different aspects of a single real-world object.
- the first real-world object may be one side of a real-world object and the second real-world object may be another side of the real-world object.
- Portal engine 110 is further configured to control access to object database 120 such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., via client devices 104 A and 1046 , respectively.
- the first party and the second party may include at least one of a person, a group of people, and an automated device.
- Portal engine 110 is further configured to facilitate modification of the first party object data by the second party (e.g., via client device 104 B) to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data.
- the at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, and a geometric context of the second party object data.
- the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data, creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data.
- the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data.
- the modified first party object data may comprise at least one of the second party object data, and object data from a source other than the first party and the second party.
- the Portal engine 110 is further configured to communicate the modified first party object data to the first party.
- the modified first party object data may be usable for presentation within a collaborative user environment, and the modified first party object data may be communicated to the first party in real-time.
- the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device.
- the modified first party object data also may be communicated to the second party.
- portal engine 110 is further configured to analyze at least one of the first party object data and second party object data.
- portal engine 110 may be configured to analyze at least one of the first party object data and second party object data by matching features of the first party object data and the second party object data using at least one of an image recognition algorithm, a pattern recognition algorithm, a speech recognition algorithm, a user matching algorithm, and a location recognition algorithm.
- portal engine 110 may be further configured to provide at least the first party and the second party access to the object database via a user portal.
- the user portal may comprise at least one of a mobile device (e.g., client devices 104 A-B), an appliance, a vehicle, a robot, a kiosk, a television, and a game console.
- portal engine 110 may be configured to present a user interface, comprising at least one of a graphical user interface, a voice-controlled interface, and a motion-controlled interface, to the first party and the second party for access to the object database.
- the user portal and/or user interface may comprise a content source control system (e.g., such as GitHub, the Concurrent Versions System (CVS), etc.), so that the content from the first party and the second party can be synchronized once it is published, e.g., by portal engine 110 .
- a content source control system e.g., such as GitHub, the Concurrent Versions System (CVS), etc.
- portal engine 110 may be further configured to control access to object database 120 such that the object database is accessible to a party other than the first party and second party.
- access to object database 120 may be controlled based on at least one of a classification of a real-world object, a proximity metric, a subscription status, and an authorization.
- portal engine 110 may be further configured to store the modified first party object data within object database 120 , wherein the modified first object data may be further modifiable to generate nth-order modified first object data.
- FIG. 3 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.
- FIG. 3 presents an overview of a multiparty object recognition method 300 executed by a portal engine (e.g., computer server, computer client, tablet, gaming console, etc.), such as portal engine 110 in FIG. 2 .
- FIGS. 4-6 below present additional details regarding the various steps of method 300 , and are referenced seriatim with corresponding steps of method 300 .
- FIG. 4A illustrates obtaining first party object data in accordance with an embodiment.
- an image of first real-world object 402 is captured by image capture device 404 and presented as digital representation 406 (e.g., a digital image) within user interface 408 .
- a client device such as client device 104 A, may comprise image capture device 404 (e.g., a digital camera or video camera) and user interface 408 (e.g., a visual display).
- First real-world object 402 may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture.
- a visible real-world object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object.
- An audible real-world object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal.
- portal engine 110 is configured to receive digital representation 406 and obtain first party object data by using an image recognition algorithm on digital representation 406 .
- the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.
- SIFT scale-invariant feature transform
- FREAK Fast Retina Keypoint
- HOG Histograms of Oriented Gradient
- SURF Speeded Up Robust Features
- DAISY Binary Robust Invariant Scalable Keypoints
- portal engine 110 may be configured to obtain the first party object data based on at least one of an identification of object data by a user (e.g., the first party), and an instruction from another device.
- the first party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data.
- the first party object data may comprise metadata.
- the metadata may comprise at least one of first party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data.
- Portal engine 110 is then configured to store the first party object data in object database 120 .
- FIG. 4B illustrates obtaining second party object data in accordance with an embodiment. Similar to block diagram 400 A above, in block diagram 400 B an image of second real-world object 410 is captured by image capture device 412 and presented as digital representation 414 (e.g., a digital image) within user interface 416 .
- a client device associated with a second user such as client device 104 B, may comprise image capture device 412 (e.g., a digital camera or video camera) and user interface 416 (e.g., a visual display).
- Second real-world object 410 may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture.
- a visible real-world object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object.
- An audible real-world object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal.
- portal engine 110 is configured to receive digital representation 414 and obtain second party object data by using an image recognition algorithm on digital representation 414 .
- the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.
- SIFT scale-invariant feature transform
- FREAK Fast Retina Keypoint
- HOG Histograms of Oriented Gradient
- SURF Speeded Up Robust Features
- DAISY Binary Robust Invariant Scalable Keypoints
- portal engine 110 may be configured to obtain the second party object data based on at least one of an identification of object data by a user (e.g., the second party), and an instruction from another device.
- the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data.
- the second party object data may comprise metadata.
- the metadata may comprise at least one of second party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data.
- the first real-world object and second real-world object may comprise different aspects of a single real-world object.
- the first real-world object may be one side of a real-world object and the second real-world object may be another side of the real-world object.
- access to object database 120 is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., via client devices 104 A and 104 B, respectively.
- the first party and the second party may include at least one of a person, a group of people, and an automated device.
- modification of the first party object data by the second party is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data.
- FIG. 5 illustrates modification of the first party object data by the second party in accordance with an embodiment.
- the second party may modify the first party object data (e.g., via user interface 416 ) in accordance with a context parameter of the second party object data (e.g., modifications 504 and 506 ), thereby generating modified first party object data.
- the second party can modify the first party object data in any suitable fashion.
- the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data (e.g., as shown by modifications 405 and 506 ), creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data.
- the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data.
- Such modified first party object data can include second party object data.
- data representing the first real-world object 402 supplied by the first party can be modified in accordance with a context parameter of data representing the second real-world object 410 supplied by the second party such that, when displayed, a representation of the first real-world object 402 can appear to be manipulated or otherwise modified by a representation of the second real-world object 410 , as shown in representation 502 .
- modified first party object data can include object data from a source other than the first party and/or second party, for example object data from a third party.
- the at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, a geometric context of the second party object data, or a combination of context parameters.
- the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- the modified first party object data is communicated to the first party.
- FIG. 6 illustrates communicating the modified first party object data to the first party in accordance with an embodiment.
- portal engine 110 can be configured to retrieve the modified first party object data (e.g., from local memory and/or object database 120 ) and communicate the modified first party object data (e.g., via network 102 using an output module) for presentation 602 to the first party, such as at user interface 408 .
- the modified first party object data also may be communicated to the second party, e.g., for presentation at client device 104 B.
- the modified first party object data may be usable for presentation within a collaborative user environment (e.g., for presentation at user interface 408 and user interface 416 ), and the modified first party object data may be communicated at least to the first party in real-time.
- the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device, e.g., client device 104 A.
- the first party can also modify the second party object data in a similar fashion, thereby generating modified second party object data in the context of first party object data.
- portal engine 110 can permit the first user to access object database 120 and modify first party object data in the context of second party object data in order to generate modified first party object data.
- modified first party object data and modified second party object data can be made prior to (long-term) storage of the unmodified first party object data and/or second party object data. It should be appreciated that in some embodiments modified first party object data and modified second party object data can be further modified repeatedly in an iterative process.
- portal engine 110 is configured to facilitate modification, by a user, of object data provided by a different user. For example, utilizing a user portal, a second party User B can modify object data that was supplied by a first party User A (i.e. first party object data) to produce modified first party object data. Subsequently, portal engine 110 can present this modified first party object data to User A and/or User B, along with second party object data associated with User B. In some embodiments, the portal engine and/or the object database can be used to store such modified first party object data.
- Modifications of the first party object data can include the addition of supplemental information, changing one or more spatial aspects of first party object data, changing an attribute of a first party object data, combining first party object data with second party object data, creating a relationship between the first party object data and second party object data, changing a relationship between the first party object data and second party object data, and adding contextual data to the first party object data.
- a real-world object from which first party object data, second party object data, or both first party object data and second party object data is derived can include one or more properties that trigger or are amenable to modification.
- properties can include a chemical signature, a temperature, a pressure, an electromagnetic field, a polarity, a static charge, a radiofrequency, a shape or configuration, a color, a texture, a taste, a smell, a mechanical property, a biometric property, and/or an electrical property.
- a real-world object can include a number of parts. For example, each part may exhibit one or more of such modifiable properties, and can be modified independently.
- modification of data related to one property can trigger an automated modification of data related to a different property.
- modification of data related to shape or configuration of a real-world object by a user can trigger an automated change in data related to a mechanical property that reflects a result of the user-directed change.
- object database 120 is configured to store data related to real-world objects and/or modified data related to real-world objects.
- a database can be supported by any computing device that supports data storage (such as a mainframe, server, desktop computer, and/or laptop computer), and may be located remotely from other system components.
- object database 120 can be composed of two or more secondary databases that are in communication with each other. Such secondary databases can reside in multiple and/or separate computing devices.
- the object database 120 is in communication with portal engine 110 , which can provide access to object database 120 for two or more users.
- the secondary object databases can be in communication with each other and/or with portal engine 110 .
- Portal engine 110 may be located at a distance from object database 120 , and communication between object database 120 and portal engine 110 can be through a wired connection, a wireless connection, or via an intermediary data system such as a data cloud or cloud storage.
- portal engine 110 can be located in a portable device, such as a laptop computer, tablet, and/or smart phone. At least two (and in some embodiments, three or more) parties, may be provided access to object database 120 by portal engine 110 .
- First party object data which corresponds to a first real world object and is obtained from a first user or party
- second party object data which corresponds to a second real world object and is obtained from a second user or party
- object database 120 is stored in an object database.
- the first party and second party (and in some embodiments, additional users) are provided with access to object database 120 via portal engine 110 .
- portal engine 110 can be located in or on a user device that is remote or distal to the object database. Examples of user devices include a desktop computer, a laptop computer, a tablet, and/or a smart phone.
- access to object database 120 can be controlled by physical access to a terminal or similar controller, a password, biometric data, possession of a hardware and/or software key, the class or other characteristic of the real world object, a proximity metric (ex: a location), a subscription status, and/or an authorization.
- a portal engine can include a user interface. Suitable user interfaces include a graphical user interface, a voice-controlled interface, and/or a motion controlled interface.
- modified first party object data can be further modified to form secondary, tertiary, and nth-order modified first party object data.
- Such a series of data modifications can be displayed, for example, in an order that creates an illusion of movement or a passage of time.
- portal engine 110 can be configured to perform an analysis on the first party object data, the second party object data, and/or both the first party object data and the second party object data. For example, portal engine 110 can match features of the first party object data and the second party object data. Alternatively, the portal engine can utilize time coding, image recognition, pattern recognition, speech recognition, user matching, user relationship, and or location or geographic recognition in the analysis of object data.
- some or all of the information provided by the user devices to the portal engine and/or other user devices can be provided more than once.
- location data of the user devices can be provided to the portal engine and/or each other multiple times (e.g., continuously, repeatedly according to a desired frequency), and as such the object data contributed by/corresponding to each user device can be updated to account for this changed location.
- information about a particular scene as determined by each user device can be shared frequently between the user devices themselves as well as the portal engine such that the portal engine and the other devices can receive updated regarding changes in a scene.
- the updated version of the object data is shared with other user devices in the virtual space.
- a first user device can transmit video data about a physical structure, whereby the virtual version of the structure can begin with only the initial view of the structure as provided by the first user device, and whereby the virtual version of the structure is updated as the user continues to transmit video data showing different parts of the structure.
- Other users can use their own object data (e.g., annotations for or about a structure that is being worked on, their own virtual tools; etc.) on the virtual structure as it becomes more developed.
- any modifications to object data by other users in the collaborative environment can be updated, removed, deleted, modified, or otherwise changed based on updated versions of the source object data.
- the updating of object data by the source device or other user devices can be used in a dynamic recognition system.
- object data is updated over time with more information, the object data becomes a more “fleshed out” representation of the real-world object.
- the recognition of the real-world object and its properties can be updated to recognize new or different aspects of the real-world object, to further refine the recognized real-world object (such as from a general recognition of a car in an image as a generic ‘car’ to increasingly specific recognitions of ‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certain make and model of a particular year’).
- portal engine 110 can be configured to resolve redundancies in provided information, such as redundancies in data objects or redundancies in interactions with or modifications to data objects, such that a synchronized, synthesized version of the object data is shared among all participants.
- the object data representing real-world objects, scenes, etc. are updated over time by one or more user devices and collaborations
- the physical aspects of the real-world objects, scenes, etc. e.g., rigidity, static/dynamic properties, opacity, specularity, etc.
- Changes to the object data within the portal engine can also be tracked as they affect the physical aspects represented by the virtual version of the real-world object.
- the portal engine can track the changes and resolve the physical changes over time.
- FIG. 7 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.
- FIG. 7 presents an overview of a multiparty object recognition method 700 executed by a portal engine (e.g., computer server, computer client, tablet, gaming console, etc.), such as portal engine 110 in FIG. 2 .
- a portal engine e.g., computer server, computer client, tablet, gaming console, etc.
- multiparty object recognition includes receiving digital representations from a first party and a second party.
- portal engine 110 may receive one or more digital representations of a first real-world object (e.g., as captured by the first party using an image capture device, as illustrated in FIG. 4A ) and one or more digital representations of a second real-world object (e.g., as captured by the second party using an image capture device, as illustrated in FIG. 4B ).
- first party object data and second party object data may be stored (e.g., in object database 120 ) at step 708 . If object data is not included with the digital representations received from either the first party or the second party, first party object data and/or second party object data may be generated using an image recognition algorithm at step 706 .
- the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.
- SIFT scale-invariant feature transform
- FREAK Fast Retina Keypoint
- HOG Histograms of Oriented Gradient
- SURF Speeded Up Robust Features
- DAISY Binary Robust Invariant Scalable Keypoints
- BRISK Binary Robust Independent Elementary Features
- Harris Corners Edges
- GLOH Gradient Location and Orientation His
- the first party object data and second party object data may be obtained based on at least one of an identification of object data by a user (e.g., the first party), and an instruction from another device.
- the generated or obtained first party object data and/or second party object data may then be stored (e.g., in object database 120 ) at step 708 .
- access to object database 120 is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., via client devices 104 A and 104 B, respectively.
- the first party and the second party may include at least one of a person, a group of people, and an automated device.
- modification of the first party object data by the second party is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data.
- modifications of the first party object data can include the addition of supplemental information, changing one or more spatial aspects of first party object data, changing an attribute of a first party object data, combining first party object data with second party object data, creating a relationship between the first party object data and second party object data, changing a relationship between the first party object data and second party object data, and adding contextual data to the first party object data.
- step 714 if the modified first party object data is identified as a known object, additional object data associated with the known object is retrieved at step 718 , e.g., from object database 120 and/or from a third-party source. If the modified first party object data is not identified as a known object, an image recognition search is performed on the modified first party object data at step 716 .
- the image recognition search may be performed using an image recognition algorithm, wherein the modified first party object data is used as a query image that is matched to one or more document images of known objects. Once a match is found, additional object data associated with the identified known object may be retrieved at step 718 .
- the recognition of the real-world object matching the modified first party object data can be updated to recognize new or different aspects of the real-world object, and to further refine the recognized real-world object (such as from a general recognition of a car in an image as a generic ‘car’ to increasingly specific recognitions of ‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certain make and model of a particular year’).
- the image recognition algorithm used for the image recognition search may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.
- SIFT scale-invariant feature transform
- FREAK Fast Retina Keypoint
- HOG Histograms of Oriented Gradient
- SURF Speeded Up Robust Features
- DAISY Binary Robust Invariant Scalable Keypoints
- BRISK Binary Robust Independent Elementary Features
- Harris Corners Edges
- GLOH Gradient
- the modified first party object data and the additional object data is communicated to the first party.
- portal engine 110 can be configured to retrieve the modified first party object data (e.g., from local memory and/or object database 120 ) and communicate the modified first party object data (e.g., via network 102 using an output module) for presentation to the first party, as shown in FIG. 6 .
- the modified first party object data also may be communicated to the second party.
- the modified first party object data may be usable for presentation within a real-time collaborative user environment.
- the first party can also modify the second party object data in a similar fashion, thereby generating modified second party object data in the context of first party object data.
- portal engine 110 can permit the first user to access object database 120 and modify first party object data in the context of second party object data in order to generate modified first party object data.
- modifications to the first party object data and/or the second party object data can be made prior to (long-term) storage of the unmodified first party object data and/or second party object data.
- modified first party object data and modified second party object data can be further modified repeatedly in an iterative process.
- portal engine 110 may be a 3-D graphical game engine.
- the 3-D graphical game engine can incorporate virtual visual elements in an augmented reality display of a user's device.
- the recognized physical attributes of the scene can be used to construct interaction rules with virtual elements of the augmented reality environment.
- the augmented reality functions can be associated with gameplay, whereby the virtual elements of the game (e.g., characters, menus, goals, interactions with other game elements, game rules) can be generated based on the scene in which the user device is located, and the physical attributes thereof.
- two devices in a same general location can both contribute to the generation of the game elements based on their individual perceptions of the scene and real-world objects within the scene.
- User devices in separate locations can also collaborate in game play, whereby users in environments providing different scenes with different physical attributes can generate different virtual game content for the augmented reality gameplay. For example, if a first player is in a hot-weather environment and a second player is in a cold-weather environment, the augmented reality content generated by the hot-weather scene can include heat-based weapons which would be common in a hot-weather environment, but with limited effectiveness against hot-weather enemies. By collaborating with the cold-weather player, the cold-weather player can add cold-weather attributes to the hot-weather player's weapon, making the weapon more effective against the hot-weather enemies.
- Roy uses the camera on his cellular phone to take a picture of the engine of his car.
- Roy can access a portal engine that provides access to a virtual garage, and provide the image data from his cellular phone to the portal engine.
- the portal engine uses image recognition techniques, the portal engine recognizes the engine in the image data, and obtains object data that includes information about the make and model of Roy's car, the engine type, and a 3-D graphic mock-up of the engine.
- Roy can upload a video of the engine of his car, including audio of the engine as it is running, and diagnostic data received from the engine controller.
- the video and controller data can be communicated instead of or in addition to the still images.
- Roy can record an audio clip of the engine, such as by using his phone, and upload the sound in addition to the still images and controller data.
- Frank in turn, can take photos of his tools in his own garage, and provide the image data to the portal engine.
- the portal engine recognizes the tools in the images submitted by Frank, and obtains object data related to the tools, including information about each of the tools (e.g., specification, sizes, gauges, etc.), and a virtual representation of the tools that is integrated into the virtual garage.
- Frank can utilize the virtual tools and the object data obtained from Roy to simulate a repair on the virtual engine. For example, Frank may determine whether the (real-world) tools he has are the right sizes/gauges for use on the parts of the engine that need to be worked on, such as whether the wrenches he has will fit the bolts of the engine, and whether the tools will accommodate the hard-to-reach corners of the engine. Using the virtual tools, Frank may service the virtual engine as he would with the real engine to get to the suspected problem area.
- the object data for a particular engine in a database can include representations of the engine, along with corresponding audio, depicting the engine as affected by a variety of known problems typically encountered by the engine (e.g., the sources of rattling, pings, and other issues that can result in an unusually-sounding engine).
- the portal engine can employ sound recognition techniques to match the sound provided by Roy to the proper representation of the engine recognized from the visual images.
- the portal engine can be used to obtain data objects about the specific parts or engine components that are at fault.
- the data objects can include suitable replacements, their cost at various sources, and instructions on installation.
- Frank can further use the virtual tools to install a virtual representation of the replacement part.
- the portal engine can present an augmented reality overlay of some or all of the virtual representation of the car's engine over the real-world engine, such as via the display on Roy's phone.
- Roy can see the changes Frank makes to the virtual engine using the virtual tools, overlaid on the real engine.
- the augmented reality function can be used as a virtual guide that walks Roy through the work he can do on the real-world engine on his own.
- a clue in a scavenger hunt can require a combination of elements by players that are playing together while being physically far apart.
- a clue in the scavenger hunt can require a piece of paper retrieved from a location close to Player A and the use of a pencil retrieved from a location close to teammate Player B, who in real life is across the country from Player A.
- Player A can use their camera-equipped phone to take a picture of the piece of paper at the proper location.
- the portal engine can recognize the paper as the correct one for the scavenger hunt, and obtain object data associated with the use of the paper in the game.
- Player B Upon arriving at the appropriate location, Player B can use their camera-equipped phone to take a photo of a pencil.
- the image, as well as location information, is similarly provided to the portal engine, which similarly confirms the pencil as a correct pencil for the game, and obtains object data associated with the pencil.
- the background content of images can be used to recognize the location of the photos of Player A and/or Player B instead of GPS or other location information.
- the object data of the paper and the pencil may include virtual representations of each, which can be communicated for presentation on the display of each player's phone.
- the object data can also include instructions regarding the clue, such as instructing Player B to write a particular password on the virtual paper using the virtual pencil (such as via touch input using the touch-screen of Player B's phone). After the correct phrase is written, the next clue in the scavenger hunt can be presented to both players.
- Player A may be required to explore a dark room.
- Player A may be asked to take an image of a candle, e.g., using a mobile device, which is provided to the portal engine.
- the portal engine may be configured to recognize the candle, and generate a virtual representation of the candle (e.g., utilizing an image recognition algorithm) for display on the devices of Player A and Player B.
- Player B may then be asked to find an object capable of producing fire, such as a set of matches or a lighter.
- Player B may take an image of the lighter using a mobile device camera and communicate the image to the portal engine.
- the portal engine may be configured to recognize the lighter and generate a virtual representation of the lighter that Player B is able to interact with. Player B can then use the virtual representation of the lighter to light the virtual representation of the candle.
- game functions associated with the lit candle may provide instructions to Player A's mobile device to turn on the camera's flash or to turn the display of the Player A's device white and at maximum brightness, so as to act as a source of light to search the dark room.
- the darkness of the room can be a virtual darkness in addition to a real-world darkness, where the goal of exploring the dark room is to find an augmented reality item.
- the real-world component of the augmented reality item might be found with any other light source, only the use of the lit virtual candle would reveal the virtual aspects of the augmented reality item.
- Systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable storage device, for execution by a programmable processor; and the method steps described herein, including one or more of the steps of FIGS. 3 and 7 may be implemented using one or more computer programs that are executable by such a processor.
- a computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result.
- a computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
- Apparatus 800 comprises a processor 810 operatively coupled to a persistent storage device 820 and a main memory device 830 .
- Processor 810 controls the overall operation of apparatus 800 by executing computer program instructions that define such operations.
- the computer program instructions may be stored in persistent storage device 820 , or other computer-readable medium, and loaded into main memory device 830 when execution of the computer program instructions is desired.
- portal engine 110 and object database 120 may comprise one or more components of apparatus 800 .
- Apparatus 800 can be defined by the computer program instructions stored in main memory device 830 and/or persistent storage device 820 and controlled by processor 810 executing the computer program instructions.
- the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform an algorithm defined by the method steps of FIGS. 3 and 7 .
- the processor 810 executes an algorithm defined by the method steps of FIGS. 3 and 7 .
- Apparatus 800 also includes one or more network interfaces 880 for communicating with other devices via a network.
- Apparatus 800 may also include one or more input/output devices 890 that enable user interaction with apparatus 800 (e.g., display, keyboard, mouse, speakers, buttons, etc.).
- Processor 810 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of apparatus 800 .
- Processor 810 may comprise one or more central processing units (CPUs), for example.
- CPUs central processing units
- Processor 810 , persistent storage device 820 , and/or main memory device 830 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs).
- ASICs application-specific integrated circuits
- FPGAs field programmable gate arrays
- Persistent storage device 820 and main memory device 830 each comprise a tangible non-transitory computer readable storage medium.
- Persistent storage device 820 , and main memory device 830 may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices.
- DRAM dynamic random access memory
- SRAM static random access memory
- DDR RAM double data rate synchronous dynamic random access memory
- Input/output devices 890 may include peripherals, such as a printer, scanner, display screen, etc.
- input/output devices 890 may include a display device such as a cathode ray tube (CRT), plasma or liquid crystal display (LCD) monitor for displaying information (e.g., a plurality of image transformations for selection) to a user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to apparatus 800 .
- a display device such as a cathode ray tube (CRT), plasma or liquid crystal display (LCD) monitor for displaying information (e.g., a plurality of image transformations for selection) to a user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to apparatus 800 .
- CTR cathode ray tube
- LCD liquid crystal display
- portal engine 110 and object database 120 may be performed by, and/or incorporated in, an apparatus such as apparatus 800 .
- FIG. 8 is a high level representation of some of the components of such a computer for illustrative purposes.
Abstract
Description
- This application claims the benefit of U.S. Provisional Application No. 62/024,825, filed Jul. 15, 2014. The entire content of that application is hereby incorporated herein by reference.
- The invention relates to virtual representation technologies, in particular as they relate to modification of virtual objects and object representation attributes.
- The background description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.
- Recent improvements in processing power have led to the possibility of mainstream image recognition and object recognition applications. For example, gaming systems have been developed that incorporate elements of image recognition and object recognition to implement novel or simplified control systems, where, for example, a user's movements are captured, digitized, and used to control movements or other actions within a game. Despite considerable potential, however, such image recognition and object recognition technology has not seen widespread adoption.
- Various attempts have been made to incorporate elements of image recognition and object recognition systems in applications that allow for the manipulation of digital images (or other digital analogs) of real-world objects. One such attempt is disclosed in United States Patent Application No. 2012/0306924 to Willoughby et al., which describes the use of “skeletons” that are mapped onto both a user and an object to allow a user to control movements of a representation of the object via digitized images of the user. Another approach is described in U.S. Pat. No. 7,564,469 to Cohen, which discloses the use of an image captured by a mobile device to control a mixed reality display via a command set. In another approach, United States Patent Application No. 2012/0256954 to Soon-Shiong, describes an augmented reality system that modifies a scene that is presented to a user based on interactions between the represented objects.
- While such approaches have utility within certain narrow contexts, they are too limited to be useful in many collaborative engineering and creative applications, where efforts to portray augmented reality, mixed reality or virtual reality interactions between real-world objects are frequently limited to previously prepared animations, the manipulation of physical models, and static whiteboard depictions. As such, there is still a need for systems, apparatuses and methods for interactively manipulating virtual object data in augmented reality, mixed reality or virtual reality environments, such as in real-time.
- The inventive subject matter provides apparatuses, systems and methods in which data related to real-world objects can be manipulated in a cooperative manner by a plurality of users. Data related to a first real-world object and a second real-world object can be stored in a database that is accessible by two or more users via a portal engine. The portal engine permits a user to access and modify data related to a first object, and further supports presentation of the modified first object data and data related to a second object to a different user. The data related to the first object can be modified in the context of data related to the second object.
- In an embodiment, interactively manipulating virtual object data, wherein an object database is configured to store first party object data, associated with a first party, that corresponds to a first real-world object and is further configured to store second party object data, associated with a second party, that corresponds to a second real-world object, includes obtaining the first party object data and the second party object data for storage within the object database. The first party object data and the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data. Access to the object database is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party. Modification of the first party object data by the second party is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data, and the modified first party object data is communicated to the first party. The at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, and a geometric context of the second party object data. The modified first party object data may be usable for presentation within a collaborative user environment, and the modified first party object data may be communicated to the first party in real-time. The first party and the second party may include at least one of a person, a group of people, and an automated device.
- In some embodiments, the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device.
- In some embodiments, at least one of the first party object data and second party object data is analyzed, and analyzing at least one of the first party object data and second party object data may include matching features of the first party object data and the second party object data using at least one of an image recognition algorithm, a pattern recognition algorithm, a speech recognition algorithm, a user matching algorithm, and a location recognition algorithm.
- In some embodiments, at least one of the first party object data and second party object data may comprise metadata. The metadata may comprise at least one of first party-specific data or second party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data.
- In some embodiments, at least one of the first party object data and the second party object data may be obtained by using an image recognition algorithm on a digital representation of at least one of the first real-world object and the second real-world object.
- In some embodiments, at least one of the first party object data and the second party object data may be obtained based on at least one of an identification of object data by a user, and an instruction from another device.
- In some embodiments, the first real-world object may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture. The visible object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object. The audible object may comprise at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal.
- In some embodiments, the first real-world object and second real-world object may comprise different aspects of a single real-world object.
- In some embodiments, the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- In some embodiments, the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data, creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data. In some embodiments, the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data.
- In some embodiments, the modified first party object data may comprise at least one of the second party object data, and object data from a source other than the first party and the second party.
- In some embodiments, the object database may be accessible to at least the first party and the second party via a user portal, and the user portal may comprise at least one of a mobile device, an appliance, a vehicle, a robot, a kiosk, a television, and a game console. A user interface, comprising at least one of a graphical user interface, a voice-controlled interface, and a motion-controlled interface, may be presented for access to the object database.
- In some embodiments, access to the object database may be controlled such that the object database is accessible to a party other than the first party and second party. In some embodiments, access to the object database may be controlled based on at least one of a classification of a real-world object, a proximity metric, a subscription status, and an authorization.
- In some embodiments, the modified first party object data may be stored within the object database, wherein the modified first object data may be further modifiable to generate nth-order modified first object data.
- In some embodiments, the modified first party object data may be communicated to the second party.
- Various objects, features, aspects and advantages of the inventive subject matter will become more apparent from the following detailed description of preferred embodiments, along with the accompanying drawing figures in which like numerals represent like components.
-
FIG. 1 illustrates an exemplary network environment that may be used for multiparty object recognition systems and methods in accordance with various embodiments. -
FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment. -
FIG. 3 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment. -
FIG. 4A illustrates obtaining first party object data in accordance with an embodiment. -
FIG. 4B illustrates obtaining second party object data in accordance with an embodiment. -
FIG. 5 illustrates modification of the first party object data by the second party in accordance with an embodiment. -
FIG. 6 illustrates communicating the modified first party object data to the first party in accordance with an embodiment. -
FIG. 7 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment. -
FIG. 8 illustrates a block diagram of a distributed computer system that can be used for implementing one or more aspects of the various embodiments. - While the invention is described with reference to the above drawings, the drawings are intended to be illustrative, and other embodiments are consistent with the spirit, and within the scope, of the invention.
- The various embodiments now will be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific examples of practicing the embodiments. This description may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this description will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Among other things, this description may be embodied as methods or devices. Accordingly, any of the various embodiments herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. The following description is, therefore, not to be taken in a limiting sense.
- Throughout the description and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise:
- The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, though it may. Thus, as described below, various embodiments of the invention may be readily combined, without departing from the scope or spirit of the invention.
- As used herein, the term “or” is an inclusive “or” operator, and is equivalent to the term “and/or,” unless the context clearly dictates otherwise.
- The term “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise.
- As used herein, and unless the context dictates otherwise, the term “coupled to” is intended to include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements). Therefore, the terms “coupled to” and “coupled with” are used synonymously. Within the context of a networked environment where two or more components or devices are able to exchange data, the terms “coupled to” and “coupled with” are also used to mean “communicatively coupled with”, possibly via one or more intermediary devices.
- In addition, throughout the description, the meaning of “a,” “an,” and “the” includes plural references, and the meaning of “in” includes “in” and “on.”
- The following discussion provides many example embodiments of the inventive subject matter. Although some of the various embodiments presented herein constitute a single combination of inventive elements, it should be appreciated that the inventive subject matter is considered to include all possible combinations of the disclosed elements. As such, if one embodiment comprises elements A, B, and C, and another embodiment comprises elements B and D, then the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly discussed herein.
- As used in the description herein and throughout the claims that follow, when a system, engine, server, device, module, or other computing element is described as being configured to perform or execute functions on data in a memory, the meaning of “configured to” or “programmed to” is defined as one or more processors or cores of the computing element being programmed by a set of software instructions stored in the memory of the computing element to execute the set of functions on target data or data objects stored in the memory.
- It should be noted that any language directed to a computer should be read to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, controllers, modules, or other types of computing device structures operating individually or collectively. One skilled in the art should appreciate the computing devices comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, FPGA, PLA, solid state drive, RAM, flash, ROM, etc.). The software instructions configure or program the computing device to provide the roles, responsibilities, or other functionality as discussed below with respect to the disclosed apparatus. Further, the disclosed technologies can be embodied as a computer program product that includes a non-transitory computer readable medium storing the software instructions that causes a processor to execute the disclosed steps associated with implementations of computer-based algorithms, processes, methods, or other instructions. In some embodiments, the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods. Data exchanges among devices can be conducted over a packet-switched network, such as the Internet, a LAN, WAN, VPN, or other type of packet-switched network, a circuit-switched network, cell-switched network; or other type of network.
- The following description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.
- In some embodiments, the numbers expressing quantities of ingredients, properties such as concentration, reaction conditions, and so forth, used to describe and claim certain embodiments of the invention are to be understood as being modified in some instances by the term “about.” Accordingly, in some embodiments, the numerical parameters set forth in the description and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of some embodiments of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as practicable. The numerical values presented in some embodiments of the invention may contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.
- Unless the context dictates the contrary, all ranges set forth herein should be interpreted as being inclusive of their endpoints and open-ended ranges should be interpreted to include only commercially practical values. Similarly, all lists of values should be considered as inclusive of intermediate values unless the context indicates the contrary.
- As used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.
- The recitation of ranges of values herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range. Unless otherwise indicated herein, each individual value is incorporated into the description as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g. “such as”) provided with respect to certain embodiments herein is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention otherwise claimed. No language in the description should be construed as indicating any non-claimed element essential to the practice of the invention.
- Groupings of alternative elements or embodiments of the invention disclosed herein are not to be construed as limitations. Each group member can be referred to and claimed individually or in any combination with other members of the group or other elements found herein. One or more members of a group can be included in, or deleted from, a group for reasons of convenience and/or patentability. When any such inclusion or deletion occurs, the description herein is deemed to contain the group as modified, thus fulfilling the written description of all Markush groups used in the appended claims.
- One should appreciate that the disclosed techniques provide many advantageous technical effects including facilitating and streamlining cooperative design and editing efforts. Thus, a computing device is configured to allow data related to real-world objects to be manipulated in a cooperative manner by two or more users. The computing device is able to process substantial amounts of digital data well beyond the capability of a human being.
- A multiparty object recognition as described herein allows for the manipulation of virtual objects by multiple users. Methods and systems for a multiparty object recognition can be implemented in a variety of ways.
FIG. 1 illustrates an exemplary network environment that may be used for multiparty object recognition systems and methods in accordance with various embodiments. In one exemplary embodiment, a multiparty object recognition system for interactively manipulating object data can be implemented within a networked computing environment 100 (e.g., a cloud-computing environment).Networked computing environment 100 comprises at least onenetwork 102. In an exemplary embodiment,network 102 is the Internet. In other embodiments,network 102 may comprise one or more of a number of different types of networks, such as, for example, an intranet, a local area network (LAN), a wide area network (WAN), a wireless network, or other type of packet-switched network, circuit-switched network or cell-switched network. Other networks may be used. Alternatively,network 102 may comprise a combination of different types of networks. - Within the
environment 100, one ormore servers 106A-B and/orclient devices 104A-E can implement a multiparty object recognition system. Examples ofclient devices 104A-E can include mobile devices (such as laptop computers, tablets, or smartphones), appliances, vehicles, robots, kiosks or other types of public terminals, televisions, digital media players, game consoles, or other types of computing devices. Further, a client device may be associated with a party (user) such that, for example,client devices - In an embodiment, one or
more client devices 104A-E and/orservers 106A-B can host or be in communication with apparatuses, e.g.,portal engine 110 andobject database 120, to implement a multiparty object recognition system that is accessible toclient devices 104A-E. For example,object database 120 may be configured to store first party object data, associated with a first party, that corresponds to a first real-world object and be further configured to store second party object data, associated with a second party, that corresponds to a second real-world object.Portal engine 110 may be configured to obtain the first party object data and the second party object data from any ofclient devices 104A-E andservers 106A-B for storage withinobject database 120. For example,portal engine 110 may be configured to obtain object data, including object data associated with a first party (i.e., first party object data) and a second party (i.e., second party object data), that corresponds to real-world objects depicted in a digital representation (e.g., a digital image or a video stream) using one or more image recognition algorithms, or based on identification instructions provided by a user or an external device. -
Portal engine 110 may be further configured to control access toobject database 120 such that the first party object data and the second party object data is accessible to the first party (e.g., atclient device 104A) and the second party (e.g., atclient device 104B), facilitate modification of the first party object data by the second party to generate modified first party object data that is in accordance with at least one context parameter of the second party object data, and communicate the modified first party object data to the first party, e.g., atclient device 104A or another client device accessible by the first party. - As shown,
object database 120 is communicatively coupled toportal engine 110. In an embodiment,portal engine 110 is configured to generate a user portal to objectdatabase 120 that may be accessible to multiple parties viaclient devices 104A-E. The user portal can be programmed or configured by theportal engine 110 to provide access to data stored inobject database 120. - Alternatively,
object database 120 andportal engine 110 can be remote from each other as well as from the one ormore client devices 104A-E used to accessobject database 120, e.g., via the user portal. For example, one or more ofobject database 120 andportal engine 110 can be housed in one ormore server computers 106A-B located remotely fromclient devices 104A-E. Likewise, the server(s) (or other computing device(s)) hostingportal engine 110 can be located remotely from server computers or other non-transitory computer-readable storage media hostingobject database 120. In some embodiments, one or more ofobject database 120 andportal engine 110 can be integral to one or more ofclient devices 104A-E. For example, one of the client devices, e.g.,client device 104A, may hostportal engine 110 and/orobject database 120, such that theother client devices 104B-E are clients to the host client device. In other embodiments, various functions described herein as being performed byportal engine 110 andobject database 120 may be distributed between a plurality ofclient devices 104A-E, a plurality ofservers 106A-B, or any combination of client devices and servers. -
FIG. 2 illustrates apparatuses that may be used for multiparty object recognition in accordance with an embodiment. In block diagram 200, elements for multiparty object recognition includeportal engine 110,object database 120, andclient devices 104A-B in communication withportal engine 110 vianetwork 102. However, it should be noted that the elements inFIG. 2 , and the various functions attributed to each of the elements, while exemplary, are described as such solely for the purpose of ease of understanding. One skilled in the art will appreciate that one or more of the functions ascribed to the various elements may be performed by any one of the other elements, and/or by an element (not shown) configured to perform a combination of the various functions. - In an embodiment, interactively manipulating virtual object data, wherein
object database 120 is configured to store first party object data, associated with a first party, that corresponds to a first real-world object and is further configured to store second party object data, associated with a second party, that corresponds to a second real-world object, includesportal engine 110 being configured to obtain the first party object data and the second party object data for storage withinobject database 120. The first party object data and the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data. In some embodiments, the first party object data and second party object data may comprise metadata. The metadata may comprise at least one of first party-specific data or second-party-specific object data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data. - In some embodiments,
portal engine 110 may be configured to obtain at least one of the first party object data and the second party object data by using an image recognition algorithm on a digital representation of at least one of the first real-world object and the second real-world object. In some embodiments, the feature detection algorithm may include at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm.Portal engine 110 also may be configured to obtain at least one of the first party object data and the second party object data based on at least one of an identification of object data by a user, and an instruction from another device. - In some embodiments, the first real-world object and second real-world object may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture. For example, the visible object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object, and the audible object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal. In some embodiments, the first real-world object and second real-world object may comprise different aspects of a single real-world object. For example, the first real-world object may be one side of a real-world object and the second real-world object may be another side of the real-world object.
-
Portal engine 110 is further configured to control access toobject database 120 such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., viaclient devices 104A and 1046, respectively. The first party and the second party may include at least one of a person, a group of people, and an automated device. -
Portal engine 110 is further configured to facilitate modification of the first party object data by the second party (e.g., viaclient device 104B) to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data. The at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, and a geometric context of the second party object data. - In some embodiments, the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- In some embodiments, the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data, creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data. In some embodiments, the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data.
- In some embodiments, the modified first party object data may comprise at least one of the second party object data, and object data from a source other than the first party and the second party.
-
Portal engine 110 is further configured to communicate the modified first party object data to the first party. The modified first party object data may be usable for presentation within a collaborative user environment, and the modified first party object data may be communicated to the first party in real-time. In some embodiments, the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device. In some embodiments, the modified first party object data also may be communicated to the second party. - In some embodiments,
portal engine 110 is further configured to analyze at least one of the first party object data and second party object data. For example,portal engine 110 may be configured to analyze at least one of the first party object data and second party object data by matching features of the first party object data and the second party object data using at least one of an image recognition algorithm, a pattern recognition algorithm, a speech recognition algorithm, a user matching algorithm, and a location recognition algorithm. - In some embodiments,
portal engine 110 may be further configured to provide at least the first party and the second party access to the object database via a user portal. For example, the user portal may comprise at least one of a mobile device (e.g.,client devices 104A-B), an appliance, a vehicle, a robot, a kiosk, a television, and a game console. In some embodiments,portal engine 110 may be configured to present a user interface, comprising at least one of a graphical user interface, a voice-controlled interface, and a motion-controlled interface, to the first party and the second party for access to the object database. For example, the user portal and/or user interface may comprise a content source control system (e.g., such as GitHub, the Concurrent Versions System (CVS), etc.), so that the content from the first party and the second party can be synchronized once it is published, e.g., byportal engine 110. - In some embodiments,
portal engine 110 may be further configured to control access toobject database 120 such that the object database is accessible to a party other than the first party and second party. In some embodiments, access toobject database 120 may be controlled based on at least one of a classification of a real-world object, a proximity metric, a subscription status, and an authorization. - In some embodiments,
portal engine 110 may be further configured to store the modified first party object data withinobject database 120, wherein the modified first object data may be further modifiable to generate nth-order modified first object data. -
FIG. 3 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.FIG. 3 presents an overview of a multipartyobject recognition method 300 executed by a portal engine (e.g., computer server, computer client, tablet, gaming console, etc.), such asportal engine 110 inFIG. 2 .FIGS. 4-6 below present additional details regarding the various steps ofmethod 300, and are referenced seriatim with corresponding steps ofmethod 300. - At
step 302, interactively manipulating virtual object data, wherein an object database is configured to store first party object data, associated with a first party, that corresponds to a first real-world object and is further configured to store second party object data, associated with a second party, that corresponds to a second real-world object, includes obtaining the first party object data and the second party object data for storage within the object database.FIG. 4A illustrates obtaining first party object data in accordance with an embodiment. In block diagram 400A, an image of first real-world object 402 is captured byimage capture device 404 and presented as digital representation 406 (e.g., a digital image) withinuser interface 408. For example, a client device, such asclient device 104A, may comprise image capture device 404 (e.g., a digital camera or video camera) and user interface 408 (e.g., a visual display). - First real-
world object 402 may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture. For example, a visible real-world object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object. An audible real-world object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal. - In an embodiment,
portal engine 110 is configured to receivedigital representation 406 and obtain first party object data by using an image recognition algorithm ondigital representation 406. For example, the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm. Alternatively,portal engine 110 may be configured to obtain the first party object data based on at least one of an identification of object data by a user (e.g., the first party), and an instruction from another device. In an embodiment, the first party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data. In some embodiments, the first party object data may comprise metadata. The metadata may comprise at least one of first party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data. -
Portal engine 110 is then configured to store the first party object data inobject database 120. -
FIG. 4B illustrates obtaining second party object data in accordance with an embodiment. Similar to block diagram 400A above, in block diagram 400B an image of second real-world object 410 is captured byimage capture device 412 and presented as digital representation 414 (e.g., a digital image) withinuser interface 416. For example, a client device associated with a second user, such asclient device 104B, may comprise image capture device 412 (e.g., a digital camera or video camera) and user interface 416 (e.g., a visual display). - Second real-
world object 410 may include at least one of a visible object, an audible object, a biometric object, a motion, and a gesture. For example, a visible real-world object may comprise at least one of a living object, a landmark, an architectural structure, a conveyance, a consumer product, a tool, a printed or displayed object, a toy, a musical instrument, an educational item, an art object, or a natural inanimate object. An audible real-world object may comprise, e.g., at least one of a voice, a sound of a musical instrument, a sound of a living object, a sound of an inanimate object, a sound of a vehicle, and an electronically generated audible signal. - In an embodiment,
portal engine 110 is configured to receivedigital representation 414 and obtain second party object data by using an image recognition algorithm ondigital representation 414. For example, the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm. Alternatively,portal engine 110 may be configured to obtain the second party object data based on at least one of an identification of object data by a user (e.g., the second party), and an instruction from another device. In an embodiment, the second party object data may include at least one of digital image data, digital video data, digital audio data, digital text data, known virtual object data, biometric data, financial data, medical data, and transaction data. In some embodiments, the second party object data may comprise metadata. The metadata may comprise at least one of second party-specific data, location data, time data, user identification data, annotation data, social media data, product data, security data, and contextual data. -
Portal engine 110 is then configured to store the second party object data inobject database 120. In some embodiments, the first real-world object and second real-world object may comprise different aspects of a single real-world object. For example, the first real-world object may be one side of a real-world object and the second real-world object may be another side of the real-world object. - At
step 304, access toobject database 120 is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., viaclient devices - At
step 306, modification of the first party object data by the second party (e.g., viaclient device 104B) is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data.FIG. 5 illustrates modification of the first party object data by the second party in accordance with an embodiment. In block diagram 500, once access is granted to objectdatabase 120 the second party may modify the first party object data (e.g., via user interface 416) in accordance with a context parameter of the second party object data (e.g.,modifications 504 and 506), thereby generating modified first party object data. The second party can modify the first party object data in any suitable fashion. For example, the modification of the first party object data may comprise one or more of adding supplemental information, changing a spatial aspect of the first party object data, changing an attribute of the first party object data, combining the first party object data with the second party object data (e.g., as shown by modifications 405 and 506), creating or changing a relationship between the first party object data and the second party object data, and adding contextual data to the first party object data. In some embodiments, the modification of the first party object data may include at least one of filtering, moving, manipulating, combining, editing, annotating, or animating the first party object data. Such modified first party object data can include second party object data. For example, data representing the first real-world object 402 supplied by the first party can be modified in accordance with a context parameter of data representing the second real-world object 410 supplied by the second party such that, when displayed, a representation of the first real-world object 402 can appear to be manipulated or otherwise modified by a representation of the second real-world object 410, as shown inrepresentation 502. Alternatively, modified first party object data can include object data from a source other than the first party and/or second party, for example object data from a third party. The at least one context parameter may correspond to at least one of a temporal context, a spatial context, a thematic context, an additive context, a subtractive context, a geometric context of the second party object data, or a combination of context parameters. - In some embodiments, the first real-world object may comprise at least one property that triggers the modification of the first party object data, the at least one property comprising at least one of a chemical signature, temperature, pressure, electromagnetic field, radiofrequency, shape, color, texture, taste, smell, mechanical, biometric, or electrical property.
- At
step 308, the modified first party object data is communicated to the first party.FIG. 6 illustrates communicating the modified first party object data to the first party in accordance with an embodiment. In block diagram 600,portal engine 110 can be configured to retrieve the modified first party object data (e.g., from local memory and/or object database 120) and communicate the modified first party object data (e.g., vianetwork 102 using an output module) forpresentation 602 to the first party, such as atuser interface 408. In some embodiments, the modified first party object data also may be communicated to the second party, e.g., for presentation atclient device 104B. As such, the modified first party object data may be usable for presentation within a collaborative user environment (e.g., for presentation atuser interface 408 and user interface 416), and the modified first party object data may be communicated at least to the first party in real-time. In some embodiments, the modified first party object data may be communicated as one of machine-readable image data or raw data which is convertible at a receiving device, e.g.,client device 104A. In various embodiments, the first party can also modify the second party object data in a similar fashion, thereby generating modified second party object data in the context of first party object data. For example,portal engine 110 can permit the first user to accessobject database 120 and modify first party object data in the context of second party object data in order to generate modified first party object data. Such modifications to the first party object data and/or the second party object data can be made prior to (long-term) storage of the unmodified first party object data and/or second party object data. It should be appreciated that in some embodiments modified first party object data and modified second party object data can be further modified repeatedly in an iterative process. - Therefore,
portal engine 110 is configured to facilitate modification, by a user, of object data provided by a different user. For example, utilizing a user portal, a second party User B can modify object data that was supplied by a first party User A (i.e. first party object data) to produce modified first party object data. Subsequently,portal engine 110 can present this modified first party object data to User A and/or User B, along with second party object data associated with User B. In some embodiments, the portal engine and/or the object database can be used to store such modified first party object data. Modifications of the first party object data can include the addition of supplemental information, changing one or more spatial aspects of first party object data, changing an attribute of a first party object data, combining first party object data with second party object data, creating a relationship between the first party object data and second party object data, changing a relationship between the first party object data and second party object data, and adding contextual data to the first party object data. - Towards this end, a real-world object from which first party object data, second party object data, or both first party object data and second party object data is derived can include one or more properties that trigger or are amenable to modification. For example, such properties can include a chemical signature, a temperature, a pressure, an electromagnetic field, a polarity, a static charge, a radiofrequency, a shape or configuration, a color, a texture, a taste, a smell, a mechanical property, a biometric property, and/or an electrical property. In some embodiments, a real-world object can include a number of parts. For example, each part may exhibit one or more of such modifiable properties, and can be modified independently. In other embodiments, modification of data related to one property can trigger an automated modification of data related to a different property. For example, modification of data related to shape or configuration of a real-world object by a user can trigger an automated change in data related to a mechanical property that reflects a result of the user-directed change.
- Further,
object database 120 is configured to store data related to real-world objects and/or modified data related to real-world objects. Such a database can be supported by any computing device that supports data storage (such as a mainframe, server, desktop computer, and/or laptop computer), and may be located remotely from other system components. In some embodiments,object database 120 can be composed of two or more secondary databases that are in communication with each other. Such secondary databases can reside in multiple and/or separate computing devices. Theobject database 120 is in communication withportal engine 110, which can provide access toobject database 120 for two or more users. In some embodiments that utilize two or more secondary object databases, the secondary object databases can be in communication with each other and/or withportal engine 110.Portal engine 110 may be located at a distance fromobject database 120, and communication betweenobject database 120 andportal engine 110 can be through a wired connection, a wireless connection, or via an intermediary data system such as a data cloud or cloud storage. In some embodiments,portal engine 110 can be located in a portable device, such as a laptop computer, tablet, and/or smart phone. At least two (and in some embodiments, three or more) parties, may be provided access toobject database 120 byportal engine 110. - The various embodiments allow interactive manipulation of a virtual object using a common or shared platform. First party object data, which corresponds to a first real world object and is obtained from a first user or party, and second party object data, which corresponds to a second real world object and is obtained from a second user or party, is stored in an object database. The first party and second party (and in some embodiments, additional users) are provided with access to
object database 120 viaportal engine 110. Such a portal engine can be located in or on a user device that is remote or distal to the object database. Examples of user devices include a desktop computer, a laptop computer, a tablet, and/or a smart phone. In some embodiments, access toobject database 120 can be controlled by physical access to a terminal or similar controller, a password, biometric data, possession of a hardware and/or software key, the class or other characteristic of the real world object, a proximity metric (ex: a location), a subscription status, and/or an authorization. To facilitate communication and usability, a portal engine can include a user interface. Suitable user interfaces include a graphical user interface, a voice-controlled interface, and/or a motion controlled interface. - In the various embodiments, modified first party object data can be further modified to form secondary, tertiary, and nth-order modified first party object data. Such a series of data modifications can be displayed, for example, in an order that creates an illusion of movement or a passage of time.
- In some embodiments,
portal engine 110 can be configured to perform an analysis on the first party object data, the second party object data, and/or both the first party object data and the second party object data. For example,portal engine 110 can match features of the first party object data and the second party object data. Alternatively, the portal engine can utilize time coding, image recognition, pattern recognition, speech recognition, user matching, user relationship, and or location or geographic recognition in the analysis of object data. - In some embodiments, some or all of the information provided by the user devices to the portal engine and/or other user devices can be provided more than once. For example, location data of the user devices can be provided to the portal engine and/or each other multiple times (e.g., continuously, repeatedly according to a desired frequency), and as such the object data contributed by/corresponding to each user device can be updated to account for this changed location. Likewise, information about a particular scene as determined by each user device can be shared frequently between the user devices themselves as well as the portal engine such that the portal engine and the other devices can receive updated regarding changes in a scene.
- As the object data in the object database is updated by a corresponding user device, the updated version of the object data is shared with other user devices in the virtual space. For example, a first user device can transmit video data about a physical structure, whereby the virtual version of the structure can begin with only the initial view of the structure as provided by the first user device, and whereby the virtual version of the structure is updated as the user continues to transmit video data showing different parts of the structure. Other users, then can use their own object data (e.g., annotations for or about a structure that is being worked on, their own virtual tools; etc.) on the virtual structure as it becomes more developed. In some embodiments, any modifications to object data by other users in the collaborative environment can be updated, removed, deleted, modified, or otherwise changed based on updated versions of the source object data.
- In a further aspect, the updating of object data by the source device or other user devices can be used in a dynamic recognition system. As object data is updated over time with more information, the object data becomes a more “fleshed out” representation of the real-world object. As the object data becomes more “fleshed out”, the recognition of the real-world object and its properties can be updated to recognize new or different aspects of the real-world object, to further refine the recognized real-world object (such as from a general recognition of a car in an image as a generic ‘car’ to increasingly specific recognitions of ‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certain make and model of a particular year’).
- In situations where multiple users are working in a collaborative environment, it is possible that their contributions can be duplicitous, redundant, or overlapping. In such cases,
portal engine 110 can be configured to resolve redundancies in provided information, such as redundancies in data objects or redundancies in interactions with or modifications to data objects, such that a synchronized, synthesized version of the object data is shared among all participants. - As the object data representing real-world objects, scenes, etc., are updated over time by one or more user devices and collaborations, the physical aspects of the real-world objects, scenes, etc., (e.g., rigidity, static/dynamic properties, opacity, specularity, etc.) can be retained in
object database 120. Changes to the object data within the portal engine, such as by other users interacting with the object data in a collaborative environment, can also be tracked as they affect the physical aspects represented by the virtual version of the real-world object. The portal engine can track the changes and resolve the physical changes over time. -
FIG. 7 illustrates a flow diagram of example operations for multiparty object recognition in accordance with an embodiment.FIG. 7 presents an overview of a multipartyobject recognition method 700 executed by a portal engine (e.g., computer server, computer client, tablet, gaming console, etc.), such asportal engine 110 inFIG. 2 . - At
step 702, multiparty object recognition includes receiving digital representations from a first party and a second party. For example,portal engine 110 may receive one or more digital representations of a first real-world object (e.g., as captured by the first party using an image capture device, as illustrated inFIG. 4A ) and one or more digital representations of a second real-world object (e.g., as captured by the second party using an image capture device, as illustrated inFIG. 4B ). - At
step 704, if object data is included with the digital representations received from the first party and the second party, the first party object data and second party object data may be stored (e.g., in object database 120) atstep 708. If object data is not included with the digital representations received from either the first party or the second party, first party object data and/or second party object data may be generated using an image recognition algorithm atstep 706. For example, the image recognition algorithm may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm. Alternatively, the first party object data and second party object data may be obtained based on at least one of an identification of object data by a user (e.g., the first party), and an instruction from another device. The generated or obtained first party object data and/or second party object data may then be stored (e.g., in object database 120) atstep 708. - At
step 710, access toobject database 120 is controlled such that the first party object data and the second party object data is accessible to at least the first party and the second party, e.g., viaclient devices - At
step 712, modification of the first party object data by the second party (e.g., viaclient device 104B) is facilitated to generate modified first party object data, the modified first party object data being in accordance with at least one context parameter of the second party object data. For example, modifications of the first party object data can include the addition of supplemental information, changing one or more spatial aspects of first party object data, changing an attribute of a first party object data, combining first party object data with second party object data, creating a relationship between the first party object data and second party object data, changing a relationship between the first party object data and second party object data, and adding contextual data to the first party object data. - At
step 714, if the modified first party object data is identified as a known object, additional object data associated with the known object is retrieved atstep 718, e.g., fromobject database 120 and/or from a third-party source. If the modified first party object data is not identified as a known object, an image recognition search is performed on the modified first party object data atstep 716. For example, the image recognition search may be performed using an image recognition algorithm, wherein the modified first party object data is used as a query image that is matched to one or more document images of known objects. Once a match is found, additional object data associated with the identified known object may be retrieved atstep 718. In some embodiments, as object data is updated over time with more information, the recognition of the real-world object matching the modified first party object data can be updated to recognize new or different aspects of the real-world object, and to further refine the recognized real-world object (such as from a general recognition of a car in an image as a generic ‘car’ to increasingly specific recognitions of ‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certain make and model of a particular year’). As above, the image recognition algorithm used for the image recognition search may be a feature detection algorithm including at least one of a scale-invariant feature transform (SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust Invariant Scalable Keypoints (BRISK), FAST, Binary Robust Independent Elementary Features (BRIEF), Harris Corners, Edges, Gradient Location and Orientation Histogram (GLOH), Energy of image Gradient (EOG) or Transform Invariant Low-rank Textures (TILT) feature detection algorithm. - At
step 720, the modified first party object data and the additional object data is communicated to the first party. For example,portal engine 110 can be configured to retrieve the modified first party object data (e.g., from local memory and/or object database 120) and communicate the modified first party object data (e.g., vianetwork 102 using an output module) for presentation to the first party, as shown inFIG. 6 . In some embodiments, the modified first party object data also may be communicated to the second party. As such, the modified first party object data may be usable for presentation within a real-time collaborative user environment. In various embodiments, the first party can also modify the second party object data in a similar fashion, thereby generating modified second party object data in the context of first party object data. For example,portal engine 110 can permit the first user to accessobject database 120 and modify first party object data in the context of second party object data in order to generate modified first party object data. Such modifications to the first party object data and/or the second party object data can be made prior to (long-term) storage of the unmodified first party object data and/or second party object data. It should be appreciated that in some embodiments modified first party object data and modified second party object data can be further modified repeatedly in an iterative process. - In one illustrative example,
portal engine 110 may be a 3-D graphical game engine. Based on the recognition of a particular scene, including the physical attributes of a scene and the real-world objects within a scene, the 3-D graphical game engine can incorporate virtual visual elements in an augmented reality display of a user's device. The recognized physical attributes of the scene can be used to construct interaction rules with virtual elements of the augmented reality environment. For example, the augmented reality functions can be associated with gameplay, whereby the virtual elements of the game (e.g., characters, menus, goals, interactions with other game elements, game rules) can be generated based on the scene in which the user device is located, and the physical attributes thereof. In further embodiments, two devices in a same general location can both contribute to the generation of the game elements based on their individual perceptions of the scene and real-world objects within the scene. User devices in separate locations can also collaborate in game play, whereby users in environments providing different scenes with different physical attributes can generate different virtual game content for the augmented reality gameplay. For example, if a first player is in a hot-weather environment and a second player is in a cold-weather environment, the augmented reality content generated by the hot-weather scene can include heat-based weapons which would be common in a hot-weather environment, but with limited effectiveness against hot-weather enemies. By collaborating with the cold-weather player, the cold-weather player can add cold-weather attributes to the hot-weather player's weapon, making the weapon more effective against the hot-weather enemies. - The following is an illustrative example of a use case involving functions and processes associated with the inventive subject matter. Consider a first user, Roy, whose car suddenly starts making strange noises while he is driving. Roy calls a second user, his friend Frank, who is a car enthusiast with experience working with cars and engines, for help with the problem. Frank wants to help Roy out, and they plan for Frank to gather his tools from his garage, and go to Roy's house. However, before Frank goes through the trouble of gathering all of his tools and driving across town, they both want to know whether they'll have the necessary tools to do the job, as well as the necessary parts for any necessary repairs.
- Roy uses the camera on his cellular phone to take a picture of the engine of his car. Roy can access a portal engine that provides access to a virtual garage, and provide the image data from his cellular phone to the portal engine. Using image recognition techniques, the portal engine recognizes the engine in the image data, and obtains object data that includes information about the make and model of Roy's car, the engine type, and a 3-D graphic mock-up of the engine.
- In a variation of this example, Roy can upload a video of the engine of his car, including audio of the engine as it is running, and diagnostic data received from the engine controller. The video and controller data can be communicated instead of or in addition to the still images. In another variation of the example, Roy can record an audio clip of the engine, such as by using his phone, and upload the sound in addition to the still images and controller data.
- Frank, in turn, can take photos of his tools in his own garage, and provide the image data to the portal engine. The portal engine recognizes the tools in the images submitted by Frank, and obtains object data related to the tools, including information about each of the tools (e.g., specification, sizes, gauges, etc.), and a virtual representation of the tools that is integrated into the virtual garage.
- Within the virtual garage, Frank can utilize the virtual tools and the object data obtained from Roy to simulate a repair on the virtual engine. For example, Frank may determine whether the (real-world) tools he has are the right sizes/gauges for use on the parts of the engine that need to be worked on, such as whether the wrenches he has will fit the bolts of the engine, and whether the tools will accommodate the hard-to-reach corners of the engine. Using the virtual tools, Frank may service the virtual engine as he would with the real engine to get to the suspected problem area.
- In embodiments where audio (or video with accompanying audio), Frank can use this audio information to assist in diagnosing potential problems with the engine. For example, the object data for a particular engine in a database can include representations of the engine, along with corresponding audio, depicting the engine as affected by a variety of known problems typically encountered by the engine (e.g., the sources of rattling, pings, and other issues that can result in an unusually-sounding engine). In these embodiments, the portal engine can employ sound recognition techniques to match the sound provided by Roy to the proper representation of the engine recognized from the visual images.
- Once the virtual engine diagnostic has been performed, and Frank has arrived at the suspected cause of the trouble, the portal engine can be used to obtain data objects about the specific parts or engine components that are at fault. The data objects can include suitable replacements, their cost at various sources, and instructions on installation. Frank can further use the virtual tools to install a virtual representation of the replacement part.
- In a variation of this example, the portal engine can present an augmented reality overlay of some or all of the virtual representation of the car's engine over the real-world engine, such as via the display on Roy's phone. As such, Roy can see the changes Frank makes to the virtual engine using the virtual tools, overlaid on the real engine. If Roy has some of the same tools Frank has introduced into the virtual garage, the augmented reality function can be used as a virtual guide that walks Roy through the work he can do on the real-world engine on his own.
- The following is another illustrative example of a use case involving the inventive subject matter. Consider a scavenger-hunt game played by participants that are remotely located from one another.
- In this example, a clue in a scavenger hunt can require a combination of elements by players that are playing together while being physically far apart. For example, a clue in the scavenger hunt can require a piece of paper retrieved from a location close to Player A and the use of a pencil retrieved from a location close to teammate Player B, who in real life is across the country from Player A. To be able to retrieve the clue, Player A can use their camera-equipped phone to take a picture of the piece of paper at the proper location. Based on the image of the paper and GPS or other location-identifying data provided by the phone, the portal engine can recognize the paper as the correct one for the scavenger hunt, and obtain object data associated with the use of the paper in the game. Upon arriving at the appropriate location, Player B can use their camera-equipped phone to take a photo of a pencil. The image, as well as location information, is similarly provided to the portal engine, which similarly confirms the pencil as a correct pencil for the game, and obtains object data associated with the pencil. In an alternative scenario, the background content of images can be used to recognize the location of the photos of Player A and/or Player B instead of GPS or other location information.
- The object data of the paper and the pencil may include virtual representations of each, which can be communicated for presentation on the display of each player's phone. The object data can also include instructions regarding the clue, such as instructing Player B to write a particular password on the virtual paper using the virtual pencil (such as via touch input using the touch-screen of Player B's phone). After the correct phrase is written, the next clue in the scavenger hunt can be presented to both players.
- In another game-related example, Player A may be required to explore a dark room. To aid in this part of the game, Player A may be asked to take an image of a candle, e.g., using a mobile device, which is provided to the portal engine. The portal engine may be configured to recognize the candle, and generate a virtual representation of the candle (e.g., utilizing an image recognition algorithm) for display on the devices of Player A and Player B. Player B, may then be asked to find an object capable of producing fire, such as a set of matches or a lighter. Upon finding a lighter, Player B may take an image of the lighter using a mobile device camera and communicate the image to the portal engine. The portal engine may be configured to recognize the lighter and generate a virtual representation of the lighter that Player B is able to interact with. Player B can then use the virtual representation of the lighter to light the virtual representation of the candle. Upon lighting the candle, game functions associated with the lit candle may provide instructions to Player A's mobile device to turn on the camera's flash or to turn the display of the Player A's device white and at maximum brightness, so as to act as a source of light to search the dark room.
- In a variation of this example, the darkness of the room can be a virtual darkness in addition to a real-world darkness, where the goal of exploring the dark room is to find an augmented reality item. Thus, while the real-world component of the augmented reality item might be found with any other light source, only the use of the lit virtual candle would reveal the virtual aspects of the augmented reality item.
- Systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable storage device, for execution by a programmable processor; and the method steps described herein, including one or more of the steps of
FIGS. 3 and 7 may be implemented using one or more computer programs that are executable by such a processor. A computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. - A high-level block diagram of an exemplary apparatus that may be used to implement systems, apparatus and methods described herein is illustrated in
FIG. 8 .Apparatus 800 comprises aprocessor 810 operatively coupled to apersistent storage device 820 and amain memory device 830.Processor 810 controls the overall operation ofapparatus 800 by executing computer program instructions that define such operations. The computer program instructions may be stored inpersistent storage device 820, or other computer-readable medium, and loaded intomain memory device 830 when execution of the computer program instructions is desired. For example,portal engine 110 andobject database 120 may comprise one or more components ofapparatus 800. Thus, the method steps ofFIGS. 3 and 7 can be defined by the computer program instructions stored inmain memory device 830 and/orpersistent storage device 820 and controlled byprocessor 810 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform an algorithm defined by the method steps ofFIGS. 3 and 7 . Accordingly, by executing the computer program instructions, theprocessor 810 executes an algorithm defined by the method steps ofFIGS. 3 and 7 .Apparatus 800 also includes one ormore network interfaces 880 for communicating with other devices via a network.Apparatus 800 may also include one or more input/output devices 890 that enable user interaction with apparatus 800 (e.g., display, keyboard, mouse, speakers, buttons, etc.). -
Processor 810 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors ofapparatus 800.Processor 810 may comprise one or more central processing units (CPUs), for example.Processor 810,persistent storage device 820, and/ormain memory device 830 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and/or one or more field programmable gate arrays (FPGAs). -
Persistent storage device 820 andmain memory device 830 each comprise a tangible non-transitory computer readable storage medium.Persistent storage device 820, andmain memory device 830, may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices. - Input/
output devices 890 may include peripherals, such as a printer, scanner, display screen, etc. For example, input/output devices 890 may include a display device such as a cathode ray tube (CRT), plasma or liquid crystal display (LCD) monitor for displaying information (e.g., a plurality of image transformations for selection) to a user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input toapparatus 800. - Any or all of the systems and apparatus discussed herein, including
portal engine 110 andobject database 120 may be performed by, and/or incorporated in, an apparatus such asapparatus 800. - One skilled in the art will recognize that an implementation of an actual computer or computer system may have other structures and may contain other components as well, and that
FIG. 8 is a high level representation of some of the components of such a computer for illustrative purposes. - The foregoing specification is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the specification, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the invention.
Claims (30)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/899,518 US20200301502A1 (en) | 2014-07-15 | 2020-06-11 | Multiparty object recognition |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201462024825P | 2014-07-15 | 2014-07-15 | |
US14/800,554 US10719123B2 (en) | 2014-07-15 | 2015-07-15 | Multiparty object recognition |
US16/899,518 US20200301502A1 (en) | 2014-07-15 | 2020-06-11 | Multiparty object recognition |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/800,554 Continuation US10719123B2 (en) | 2014-07-15 | 2015-07-15 | Multiparty object recognition |
Publications (1)
Publication Number | Publication Date |
---|---|
US20200301502A1 true US20200301502A1 (en) | 2020-09-24 |
Family
ID=55074569
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/800,554 Active 2036-09-01 US10719123B2 (en) | 2014-07-15 | 2015-07-15 | Multiparty object recognition |
US16/899,518 Pending US20200301502A1 (en) | 2014-07-15 | 2020-06-11 | Multiparty object recognition |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/800,554 Active 2036-09-01 US10719123B2 (en) | 2014-07-15 | 2015-07-15 | Multiparty object recognition |
Country Status (1)
Country | Link |
---|---|
US (2) | US10719123B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11514690B2 (en) | 2020-06-30 | 2022-11-29 | Sony Interactive Entertainment LLC | Scanning of 3D objects with a second screen device for insertion into a virtual environment |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10230780B2 (en) * | 2016-12-28 | 2019-03-12 | Intel Corporation | Methods and apparatus for collaborative content rendering |
KR20210056404A (en) * | 2018-09-10 | 2021-05-18 | 류와인드르, 엘엘씨 | Image management using area-based metadata indexing |
US11474678B2 (en) * | 2019-12-20 | 2022-10-18 | Bayerische Motoren Werke Aktiengesellschaft | Method, an apparatus, and a machine-readable medium for displaying information |
US11138803B1 (en) * | 2020-04-30 | 2021-10-05 | At&T Intellectual Property I, L.P. | System for multi-presence interaction with extended reality objects |
WO2022025908A1 (en) * | 2020-07-31 | 2022-02-03 | Siemens Aktiengesellschaft | Sharing world model objects between multiple autonomous systems |
US11256402B1 (en) * | 2020-08-12 | 2022-02-22 | Facebook, Inc. | Systems and methods for generating and broadcasting digital trails of visual media |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120264510A1 (en) * | 2011-04-12 | 2012-10-18 | Microsoft Corporation | Integrated virtual environment |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AU2802097A (en) | 1997-04-15 | 1998-11-11 | Criticom Corporation | Computer games having optically acquired images which are combined with computergenerated graphics and images |
JP4288843B2 (en) * | 2000-10-25 | 2009-07-01 | 沖電気工業株式会社 | Remote work support system |
US20070265089A1 (en) | 2002-05-13 | 2007-11-15 | Consolidated Global Fun Unlimited | Simulated phenomena interaction game |
US20060223635A1 (en) | 2005-04-04 | 2006-10-05 | Outland Research | method and apparatus for an on-screen/off-screen first person gaming experience |
KR101019569B1 (en) | 2005-08-29 | 2011-03-08 | 에브릭스 테크놀로지스, 인코포레이티드 | Interactivity via mobile image recognition |
JP5792499B2 (en) * | 2011-04-07 | 2015-10-14 | シャープ株式会社 | Electronic device, display method, and display program |
US8810598B2 (en) | 2011-04-08 | 2014-08-19 | Nant Holdings Ip, Llc | Interference based augmented reality hosting platforms |
US8332424B2 (en) | 2011-05-13 | 2012-12-11 | Google Inc. | Method and apparatus for enabling virtual tags |
US9724600B2 (en) | 2011-06-06 | 2017-08-08 | Microsoft Technology Licensing, Llc | Controlling objects in a virtual environment |
US9122321B2 (en) * | 2012-05-04 | 2015-09-01 | Microsoft Technology Licensing, Llc | Collaboration environment using see through displays |
US9250721B2 (en) * | 2012-12-17 | 2016-02-02 | Disney Enterprises, Inc. | Wireless stylus device with interchangeable tips and eraser |
US20140354553A1 (en) * | 2013-05-29 | 2014-12-04 | Microsoft Corporation | Automatically switching touch input modes |
US9113068B1 (en) * | 2014-05-15 | 2015-08-18 | Camera Slice, Inc. | Facilitating coordinated media and/or information capturing and aggregation |
-
2015
- 2015-07-15 US US14/800,554 patent/US10719123B2/en active Active
-
2020
- 2020-06-11 US US16/899,518 patent/US20200301502A1/en active Pending
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120264510A1 (en) * | 2011-04-12 | 2012-10-18 | Microsoft Corporation | Integrated virtual environment |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11514690B2 (en) | 2020-06-30 | 2022-11-29 | Sony Interactive Entertainment LLC | Scanning of 3D objects with a second screen device for insertion into a virtual environment |
Also Published As
Publication number | Publication date |
---|---|
US10719123B2 (en) | 2020-07-21 |
US20160018886A1 (en) | 2016-01-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200301502A1 (en) | Multiparty object recognition | |
US11074758B2 (en) | Collaborative augmented reality | |
US11747634B1 (en) | Augmented reality content creation | |
Bekele et al. | A comparison of immersive realities and interaction methods: Cultural learning in virtual heritage | |
CN109034397B (en) | Model training method and device, computer equipment and storage medium | |
US20240062081A1 (en) | Private artificial intelligence (ai) model of a user for use by an autonomous personal companion | |
CN106030457B (en) | Tracking objects during a process | |
TW202220438A (en) | Display method, electronic device and computer readable storage medium in augmented reality scene | |
US20180330549A1 (en) | Editing interactive motion capture data for creating the interaction characteristics of non player characters | |
US9925461B2 (en) | Character model animation using stored recordings of player movement interface data | |
CN105593856B (en) | Adapt to the video-game of the location drawing | |
Qian et al. | Scalar: Authoring semantically adaptive augmented reality experiences in virtual reality | |
CN105122304A (en) | Real-time design of living spaces with augmented reality | |
US11501658B2 (en) | Augmented reality platform for collaborative classrooms | |
Zhang | Design of mobile augmented reality game based on image recognition | |
Scott et al. | Towards an interaction blueprint for mixed reality experiences in glam spaces: the augmented telegrapher at porthcurno museum | |
Zhang et al. | Vrgit: A version control system for collaborative content creation in virtual reality | |
US20190378335A1 (en) | Viewer position coordination in simulated reality | |
Solah et al. | Mood-driven colorization of virtual indoor scenes | |
Yoo et al. | Evaluation of an Augmented Reality for Historical Context Experiences of 3D Restored Court Paintings | |
US20240062497A1 (en) | Method and system for generating virtual content | |
Kelly | evoExplore: Multiscale Visualization for Evolutionary Design Histories | |
CN117853622A (en) | System and method for creating head portrait | |
Monteiro et al. | Enabling rapid prototyping of tangible augmented reality experiences | |
KR20230160105A (en) | Real-time video chat rendering technology using character IP |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: NANT HOLDINGS IP, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NANT VISION, INC.;REEL/FRAME:053992/0259 Effective date: 20150925 Owner name: NANT VISION, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SIDDIQUI, MATHEEN;REEL/FRAME:053992/0232 Effective date: 20151007 Owner name: NANT VISION, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MCKINNON, DAVID;REEL/FRAME:053992/0241 Effective date: 20150401 Owner name: NANTMOBILE, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WIACEK, JOHN;REEL/FRAME:053992/0247 Effective date: 20150827 Owner name: NANT HOLDINGS IP, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NANTWORKS, LLC;REEL/FRAME:053992/0262 Effective date: 20150925 Owner name: NANT HOLDINGS IP, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NANTMOBILE, LLC;REEL/FRAME:053992/0256 Effective date: 20150925 Owner name: NANTWORKS, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SONG, BING;REEL/FRAME:053992/0244 Effective date: 20150828 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |