WO2018017353A1 - Réseautage social à réalité augmentée - Google Patents

Réseautage social à réalité augmentée Download PDF

Info

Publication number
WO2018017353A1
WO2018017353A1 PCT/US2017/041320 US2017041320W WO2018017353A1 WO 2018017353 A1 WO2018017353 A1 WO 2018017353A1 US 2017041320 W US2017041320 W US 2017041320W WO 2018017353 A1 WO2018017353 A1 WO 2018017353A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
information
article
mobile device
articles
Prior art date
Application number
PCT/US2017/041320
Other languages
English (en)
Inventor
Joel M. FOLGELSON
Thomas Edward Horlander
Chad Andrew Lefevre
William G. Redmann
Original Assignee
Thomson Licensing
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Thomson Licensing filed Critical Thomson Licensing
Publication of WO2018017353A1 publication Critical patent/WO2018017353A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0282—Rating or review of business operators or products
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0283—Price estimation or determination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0623—Item investigation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0641—Shopping interfaces
    • G06Q30/0643—Graphical representation of items or shoppers

Definitions

  • Embodiments described herein relate generally to augmented reality (AR) and, more particularly, to using augmented reality to identify articles that a person is wearing or carrying and to provide other information pertinent to the articles, such as purchase information.
  • AR augmented reality
  • a multicast system as used herein is a system in which a server transmits the same data to multiple receivers simultaneously, where multiple receivers form a subset of all the receivers up to and including all of the receivers.
  • a broadcast system is a system in which a server transmits the same data to all of the receivers simultaneously. That is, a multicast system by definition can include a broadcast system.
  • an AR device such as the HoloLensTM that
  • HoloLensTM includes an untethered, see-through head-mounted display (HMD) driven by an integrated computer. Such devices enable a person (user) to interact with 3D models displayed so as to appear to be in the user's world (that is, the models appear to be outside the HMD).
  • HoloLensTM differs from Google Glass (an HMD historically marketed by Google), which merely superimposed a screen into the field-of-view of one eye.
  • Still another AR device is a tablet computer, for example the iPad by Apple, which can be aimed at a scene by the user and which uses a camera to capture the scene and presents it on the display of the tablet with annotations graphically overlaid.
  • a smartphone can provide this same kind of interaction and by itself be considered an AR device.
  • the present invention can be used with any such AR device
  • Traditional mobile devices such as smartphones have short-range wireless capabilities such as BluetoothTM (BT) or Near Field Communication (NFC) technology which may be used to communicate between the smartphone and an HMD, such as the HoloLensTM.
  • BT and NFC each include a set of communication protocols that enable two electronic devices, one of which is usually a portable device such as a smartphone, to establish communication. While BT can operate over a distance of several meters, NFC operates by bringing the two devices within 4 cm (2 in) of each other.
  • the combination of the smartphone device and HMD can also include a cellular data phone connection that can be used to obtain information from a communication network such as the Internet using a cell phone connection (e.g., 3G/4G).
  • Various exemplary embodiments described herein may relate to, include, or take the form of a method for using augmented reality and articles accompanying a user.
  • accompanying articles include a purse, backpack, books, recent purchases (e.g., in shopping bags), or cellular telephone carried by a user, a shirt, shoes, hat, sunglasses, contact lenses, or other clothing or items worn by the user, makeup, such as lipstick worn by a user; a baby stroller or automobile being operated by the user, and pet or the pet's accessories (e.g., a leash or dog sweater).
  • clothing includes but is not limited to hats, caps, scarfs, ear muffs, sweaters, vests, sweatshirts, shirts, pullovers, pants, jeans, skirts, dresses, shoes, boots, slippers, coats, jackets, gloves, and eyewear. Further, clothing is not limited to apparel worn visibly, but also may include underwear such as undershirts, thermal underwear, briefs, boxers, panties, bras, lingerie.
  • the computer program product may include a first set of instructions stored in the non-transitory storage medium executable by a processing unit to determine articles of clothing worn by the user.
  • the computer program product may further include a second set of instructions stored in the non-transitory storage medium executable by the processing unit to determine relationships among users.
  • the computer program product may additionally include a third set of instructions stored in the non-transitory storage medium executable by the processing unit to use augmented reality with the identified articles clothing worn by the user and relationships among users to forward the list of articles worn by the user and related purchase information to some or all users with whom the user has a relationship.
  • the computer program can be tangibly embodied in a non-transitory storage medium which may be included in a mobile device.
  • a mobile device includes but is not limited to a cellular phone, smartphone, laptop computer, personal digital assistant (PDA), tablet computer, augmented reality glasses or any other equivalent device.
  • PDA personal digital assistant
  • Fig. 1 shows an exemplary HMD with an integrated computer suitable for augmented reality.
  • Fig. 2 shows an exemplary smartphone having wireless communication with the
  • Fig. 3A shows a male and a female, each having accompanying articles (here, clothing items) on or about their person.
  • Fig. 3B is a system overview showing a male and a female.
  • Fig. 4 depicts some fabric care tags and an exemplary QR code.
  • Fig. 5 is a flowchart of the operation of an augmented reality system (e.g., smartphone or HMD and integrated computer) in accordance with the principles described herein.
  • an augmented reality system e.g., smartphone or HMD and integrated computer
  • Fig. 6 is a block diagram of a mobile device in accordance with the principles described herein.
  • Fig. 7 is a flowchart of the operation of a device of the second user.
  • processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage.
  • DSP digital signal processor
  • ROM read only memory
  • RAM random access memory
  • any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
  • any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements that performs that function or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function.
  • the disclosure as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which the claims call for. It is thus regarded that any means that can provide those functionalities are equivalent to those shown herein.
  • Augmented reality is a live view (direct, as with an HMD, or indirect, as with the display of a tablet computer or smartphone) of a physical, real world space whose elements are augmented (or supplemented) by a computing device, where the augmentation is based, at least in part, on sensor input.
  • augmentation may include visual information, such as, images, video, computer graphics and the like, presented on the HMD or display, or audio presented over a speaker or earphone.
  • Such sensor input may include any of audio captured from a microphone, images or video captured by a camera, position information as determined by a localization service (e.g., GPS or wireless triangulation or ranging technique), and orientation as can be determined from one or more of an accelerometer, flux gate, gyroscope, and camera.
  • computing device generated visual information may be displayed on (and/or projected onto) the transparent screen of an HMD, through which a user can see a physical space.
  • an electronic display of a tablet computer or smartphone may present live video of a physical space that is combined with additional computing device generated visual information.
  • augmented reality may enhance a user's perception of a physical space, in contrast to virtual reality (VR), which replaces a physical space with a simulated space.
  • VR virtual reality
  • a user's cell phone is configured with a program used to identify the articles, for example, what that user is wearing on a specific day.
  • the smartphone can read, using a camera, a Quick Response (QR) code provided on an article, a Universal Product Code (UPC) associated with the article, another type of barcode or machine- readable code associated with the article, or the smartphone can accept a manually entered (that is, typed in) description of the items, or provide another user interface (UI) that permits the user to provide information about the article.
  • QR Quick Response
  • UPC Universal Product Code
  • the descriptive information may also indicate color, a logo, and/or a thumbnail image of the corresponding article.
  • the UI permits the user to select from a list of articles previously entered, to reduce burden on the user.
  • the user can also specify where a particular article is being worn (e.g., pants on the lower body, T-shirt on the upper body).
  • the manufacturer or retailer of an article may have the incentive to include a machine-readable tag (e.g., barcode or radio frequency identification (RFID)) that identifies the article as a means of advertising.
  • a machine-readable tag e.g., barcode or radio frequency identification (RFID)
  • RFID radio frequency identification
  • the identifying tag can provide a link to, or otherwise include information about the clothing article to allow the wearer of the clothes to sync the clothes to the mobile device rather than manually entering information about the clothes.
  • the wearer of the clothes would have the option of blocking the emission of the clothes identifying information, except as allowed by the social networking rules mentioned elsewhere in this description.
  • Information representing the user's defined inventory of articles can be stored within the memory of the mobile device so that the information can be made available to other individuals using a short range wireless technology.
  • Bluetooth, NFC, Zigbee and WiFi are examples of short range wireless technologies that could form the communication channel between the clothing and the mobile device.
  • NFC technology is characterized by a very short range on the order of several centimeters, which would only be suitable for very close encounters. It would be preferable to use a short range wireless technology or a personal area network (PAN).
  • PAN personal area network
  • the user can also define where an article was purchased, or how much was paid.
  • the user when specifying an inventory of what the user is wearing on a particular day, the user can also specify that this item is available from a specific retailer (or manufacturer) for purchase.
  • Such associations can be made by using a user interface where a person can make such associations by inputting information into a form/user interface/and the like.
  • the retailer when a user has identified the retailer of an accompanying article (a particular item being worn today), the retailer can be notified, that is, the retailer can be informed that a particular (or anonymous) user has annotated a particular article (e.g., a shirt) as being available from that retailer.
  • Such notification may be the basis for a reward program, or to track the success of certain fashion items, to detect trends, and otherwise keep the retailer better informed.
  • visual recognition software/device can be used to recognize different items (clothing, apparel, attire) that a person may wear or put on for a day.
  • items clothing, apparel, attire
  • Amazon has a service where a user can take a picture of an item and information about the item (whether done through a live person via the Amazon Mechanical Turk service or through machine vision techniques) and the Amazon service resolves the picture to determine the attributes of the item(s). If multiple identifications of an item are returned (i.e., the result offers several different identifications for the same article), the alternatives can be provided to a user and the user can select which one is correct.
  • the mobile device can be configured to remember the source from which a user has purchased specific items. Hence, when a user inputs that they are wearing a specific item, the device can reference memory to see if that item is listed on a purchase history log that resides within the device or remotely. If the device recognizes that the item was previously purchased from a retailer, the device can offer an option to a user to see if they want to make this information available to others using the short range wireless technology referenced above.
  • a gatekeeper between determining whether or not a user's information is made available to others relationships set up through a social networking service such as Facebook, Google+, Instagram, Snapchat, Linkedln, and the like to determine if two people are "friends" or have “friends” in common.
  • a social networking service such as Facebook, Google+, Instagram, Snapchat, Linkedln, and the like to determine if two people are "friends" or have “friends” in common.
  • an address book or contact list such as used for an email, could be used, such that finding another user in the address book or contact list indicates a relationship with that other user.
  • contact lists include the nature of a relationship for each entry, or the nature of a relationship might be indicated in general notes.
  • two users can be linked together by one user sending another user an invitation to be linked together through a social networking platform. The user receiving the invitation must accept in order for the social networking to associate the two users as "friends", allowing them to be networked together.
  • These associations can be stored in the local
  • social networking sites can be pinged (checked) periodically from the service to generate a list of contacts that are associated with a particular user.
  • This information can be downloaded to a mobile device of the particular user and can be referenced when another person approaches the particular user.
  • permission to share specific classes of information e.g., description of an article, source for an article, date of purchase, or price paid for an article, may be set by independent user-selectable policies.
  • a user might select settings to allow any user to see a description of an article (e.g., "Levi's 501 jeans"), only users who are friends might learn where a user purchased the article (e.g., "gap.com”, and perhaps only selected users (e.g., your sister) is allowed to see what you paid for the article (e.g., "$39.99") and when (e.g., "Spring Sale, March 2016”). Perhaps nobody is allowed to see the size (e.g., "size 14 slim”).
  • policies may be compounded and may provide exclusions, as well: For example a policy setting might be directed to "friends, but not co-workers" so that while someone classified as a friend might be able to access certain information, nobody a user knows from work (even those classified as friends) would see such information.
  • the first user is the one wearing or carrying articles that are registered and described in the inventory.
  • the second user is the one employing AR techniques to observe descriptions of the articles being worn or carried by the first user.
  • the second user having an AR-capable device based on any of an HMD, tablet computer, or smartphone and including a second mobile device approaches a first user having a first mobile device.
  • the first mobile device has been previously supplied with the inventory of articles being worn or carried by the first user.
  • the AR-capable second mobile device represents the "querying user", while the first mobile device represents the "queried user".
  • querying and “queried” are relative terms:
  • “querying user” or “querying device” refers to the user or mobile device that makes an inquiry to another (i.e., first) user and/or mobile device in proximity to the querying user.
  • “queried user” or “queried device” refers to the user or mobile device that receives an inquiry from another (i.e., second) user and/or mobile device, the queried (first) user being in corresponding proximity to the querying (second) user.
  • the querying user will have line-of-sight to the queried user, so that the AR-capable device is able display information in conjunction with the view of the queried user.
  • the queried mobile device receives a query from the querying device of the second user in order to verify that there is a social connection between the first and second users.
  • a second user identifying information is transmitted by the querying mobile device to the queried mobile device.
  • the existence of a social connection can be verified on the basis of the second user identifying information being found within the social information of the queried mobile device.
  • the nature of a relationship is also determined, for example whether the first and second users are friends, or friends of friends.
  • the query from the second device to the first device may be facilitated through a short range wireless technology, or in other embodiments may rely on communication through the Internet.
  • the queried mobile device can verify the existence of a social relationship with the second user by consulting a remote server (e.g., a social networking service) using an account of the queried (first) user, to determine whether or not a relationship exists based on the second user identifying information.
  • a remote server e.g., a social networking service
  • the relationship is verified by using a data communication channel to query a social service about whether or not the second user' s identifying information is associated within that social service with the first user's account (i.e., has a relationship to the first user). If a social connection between the queried (first) user and the querying (second) user is affirmed based on the second user's identifying information, a determination is made as to what information is to be returned to the querying user.
  • This determination may depend upon the nature of the relationship between the two users.
  • what information is to be returned to the querying user is based on the degrees of separation between the two: Friends (1 degree), friend of a friend (2 degrees) etc. and further based on policies regarding what information about articles of the queried user are to be communicated to other users based on their particular relationship(s). This can be defined in the listing on what is to be presented. Only the information that should be presented to the querying user is then transmitted from the queried mobile device to the querying device.
  • the querying mobile device may supply parameters so as to be more selective about the information to be obtained regarding the queried user's inventory of articles.
  • Such parameters may represent information directed to a particular article or articles of the inventory: For example, the location of the article on the queried user's body may be supplied, such that a querying mobile device might provide "upper torso" as a location parameter, so as to obtain further information about the queried user's shirt.
  • a querying mobile device might request a particular kind of information about an article, e.g., information about where the item can be purchased. In combination, two such parameters may mean, "where can I buy that shirt?".
  • the information to be returned may be determined, at least in part, from parameters received from the querying mobile device.
  • the querying mobile device can request additional information about the article(s) from an online store or other resource over the Internet using a data connection, as when the response from the queried mobile device includes article-identifying information such as a UPC code, model number, or text description (e.g., "Levi's 501"), or the response includes a hyperlink (i.e., a URL).
  • article-identifying information such as a UPC code, model number, or text description (e.g., "Levi's 501"
  • the response includes a hyperlink (i.e., a URL).
  • Rendering of the descriptive information obtained about the article is performed so as to present the descriptive information to the querying (second) user in apparent proximity to the queried user, thereby creating an AR presentation.
  • the AR-capable device will use information from the previous step to render descriptive information (item name, price, and source of where purchased) in display positions relative to different locations of the first user. That is, the AR-capable device of the second user generically identifies the head of the first user, the arm or arms of the first user, the chest of the first user, and/or the leg or legs of the first user (many devices can do this using well known principles).
  • the first user can be identified before querying by noting the unique identifiers of the mobile devices that are in short range wireless communication and consulting a database of smartphone identifiers and the users known to be associated with those smartphones. For example, if the querying device wirelessly detects and recognizes a unique identifier (e.g., the Bluetooth address, i.e., the BD_ADDR, of a smartphone previously known to be associated with the first user), then the querying mobile device can prepare to find the first user in proximity. In this case, the unique identifier of the queried smartphone will correspond to a first user identification information. If identification of the first user has not been made before the first mobile device is queried, then the reply from the queried device can include the first user identification information.
  • a unique identifier e.g., the Bluetooth address, i.e., the BD_ADDR
  • Receipt of the first user identification information by the querying mobile device can make recognition of the first user easier because the querying mobile device can be prepared to find the first user in proximity.
  • the identity of the first user can be determined by identifying a head of some person in proximity and in the field of view of a camera of the AR-capable device, and then attempting to match the face of that person, as captured by the camera, by facial recognition. If the first user identification information has been determined, then the attempt at facial recognition is easier and faster, as it is performed looking for a match to a particular person on the first user identification information, but even without that simplifying parameter, facial recognition can be attempted by matching against all individuals related to the querying user on the social networking site. Detection of a successful facial recognition, that is finding a match to an associated reference facial information, results in the identification of the head in the field of view as being the head of the first user.
  • the device can then display the descriptive information about the articles so as to overlay or otherwise indicate their association with the first (queried) user, or more particularly, to the identified body regions of the queried user using AR techniques.
  • information about the first user's pants could be rendered over or near the first user's "legs” with pricing and information about where to purchase such pants.
  • the pants information can be rendered to appear beside the first user with a callout line pointing to the first user, or the legs of the first user.
  • information about the shirt the queried user is wearing could be rendered over or near the first user's "chest" (or rendered with a line pointing to the first user's chest) using the techniques described above.
  • identification of body parts or articles by the AR-capable device may be informed by information associated with the articles. For example, if the first mobile device has replied to a query by returning an inventory that includes a pink shirt, then identification in the field-of-view of the camera of the AR-capable second mobile device can make use of that information by detecting pink regions.
  • information can be searched against an online commercial database such as Amazon, to determine if similar alternatives to an identified item are available. This can include alternatives that are less expensive, similar in size, color, shape, material, and the like. It is even possible to offer alternatives that are the "most selected" by others or most popular as selected via by ratings, crowdsourced data, or the like.
  • Fig. 1 shows a head-mounted display (HMD) as may be worn by a querying second user, wherein the HMD is suitable for augmented reality applications either independently or in conjunction with a querying second smartphone, where Fig. 2 shows an exemplary queried first smartphone, as might be carried by a queried first user, the queried smartphone having communication with either the HMD of Fig. 1 which may perform the querying function without need of the second smartphone, or with the second smartphone, wherein the first smartphone is the queried mobile device.
  • HMD head-mounted display
  • Fig. 2 shows an exemplary queried first smartphone, as might be carried by a queried first user, the queried smartphone having communication with either the HMD of Fig. 1 which may perform the querying function without need of the second smartphone, or with the second smartphone, wherein the first smartphone is the queried mobile device.
  • Fig. 3A shows a male and a female having various articles about their respective persons.
  • the male is wearing a shirt, pants and shoes (sneakers) and is carrying a book.
  • the female is wearing a shirt (blouse), pants and shoes and is carrying several shopping bags.
  • the articles e.g., of clothing
  • the articles may have QR or other machine-readable codes on the articles themselves or on tags attached to the articles, e.g., in the manner of fabric care tags such as shown in Fig. 4.
  • the articles may have, or may have had, UPC codes on tags or packaging.
  • Fig. 3B is an exemplary system overview having a male figure and a female figure.
  • the male figure has a smartphone and is wearing an AR device.
  • the male figure's smart phone is in communication with a store server through the cloud.
  • the male figure is wearing a green shirt or jacket on his torso and blue pants from the waist down.
  • the female figure has a smartphone in her red purse and is wearing a pink shirt or jacket on her torso and a brown skirt from the waist down.
  • the female figure's smart phone is in communication with a social network server through the cloud.
  • the male figure is interested in the female figure's brown skirt. Perhaps he would like to purchase on for his wife or girlfriend.
  • the two smartphones communicate, the smartphone carried by the male provides and identification of the male to the female's smartphone so that the female's smartphone can determine (alone or with the social network server) the relationship between the male and female, if any. Based on that relationship, the proposed method and apparatus allow the male figure to determine that the skirt is a maxi-skirt and available from LongRiver.com. If the information about the skirt as supplied by the female's smartphone is only identification of the articles she has (and optionally the articles' placements and/or colors), then the male's smartphone can interrogate the remote store server for more details.
  • Fig. 5 is a flowchart of the operation of the queried (first) mobile device, e.g., a smartphone or AR-capable device (such as HoloLensTM), in accordance with the principles of the proposed method.
  • the smartphone or AR-capable device scans the first user's articles (e.g., clothing) and the corresponding description tags. This scanning of the first user's articles defines an inventory of articles to be worn or carried by the first user and is the preferred method by which the mobile device identifies the articles.
  • the smartphone can utilize QR codes that are on such articles (e.g., the first user's clothing) or UPCs associated with the articles. It is also possible for the mobile device, at this point, to accept a manually entered (typed in or, with voice recognition, dictated) description of the articles through a UI. Alternatively, the UI can accept a manual selection by the first user of the articles from a list of articles historically entered. The first user can also specify where an article is being worn or carried (pants on the lower body, T-shirt on an upper body, etc.). Optionally, the first user can also define where particular articles can be (or were) purchased.
  • the first user when defining the inventory of what the first user is wearing on a particular day, the first user can also define that this item is available from a specific retailer for purchase. Such associations (between an article and information about the article's availability) can be made through a user interface where the first user makes such associations by inputting information into a form or other UI.
  • an image recognition module can be used to recognize different articles (e.g., each specific item of clothing) that a user may wear or carry today.
  • Amazon has a service where a customer can take a picture of an article and information about the article is returned. Whether done through a live person via the Amazon Mechanical Turk service or through machine vision techniques, the Amazon service resolves the picture to determine the identity and attributes of the article(s).
  • a mobile device can be configured to remember (retrieve from memory) where the first user has purchased specific items, e.g., through an online retailer such as Amazon, or based on automatic or manual entry of a shopping receipt.
  • the device can reference memory to see if that article is listed on a purchase history log that resides within the mobile device, or in remote storage accessible by the mobile device (e.g., server, gateway or cloud storage).
  • the mobile device can automatically import, or offer to the first user the option of importing, some or all of this information to be made available to others through the present invention, using the short range wireless technology referenced above.
  • the inventory of articles is stored along with the associated description information.
  • the inventory and corresponding descriptions can be stored within the memory accessible to the mobile device so that the information can be made available to other individuals using short range wireless technology.
  • the mobile device is queried by another mobile device regarding a social connection with a second user in close proximity of the first user, the second user typically having line of sight to the first user.
  • the mobile device performs a lookup and responds to the query.
  • the mobile device acts as a gatekeeper to determine whether or not the first user's information is made available to other individuals (e.g., the second user).
  • Relationships set up through a social networking service such as Facebook, Google+, Instagram, Snapchat, Linkedln, and the like are used to see if two people (the first and second users) have a relationship, e.g., that they are "friends" or have “friends” that are in common.
  • Such relationships can be stored in the memory accessible to the mobile device, allowing the relationships to be stored in local memory, or on a remote server (e.g, "in the cloud”).
  • Social networking sites can be checked periodically by the mobile device to maintain a list of contacts that have a relationship with the first user.
  • This information can be downloaded to a mobile device of the first user and can be referenced when a second user approaches or queries the mobile device through a short range wireless technology.
  • the queried mobile device can verify a social relationship with a user identifying information that corresponds to the second user.
  • the relationship of the users is verified by using a data communication channel to query a social networking service about whether or not the user identifying information corresponding to the second user is associated with the account corresponding to the first user on the social network service.
  • the result of the lookup indicates whether there is a relationship between the two users, and if so, may further specify the nature of that relationship.
  • the mobile device determines what information can be disclosed and transmitted to the second mobile device (that of the second user), based on the results of the lookup. If it is affirmed that there is a social relationship between the first and second users (as verified by the social information), a determination is made as to what information is to be represented, that is, is allowed to be disclosed or transmitted to the second user. This determination may depend upon the nature of the relationship between the first and second users (for example, the degrees of separation between the two users: Friends (1 degree), friend of a friend (2 degrees) etc.) and what information about the articles present with the user are to be communicated to the other users based on the nature of the relationship between them. What is to be shared given a particular nature of relationship can be defined explicitly per article in the inventory or can be more generally represented as policies (e.g., only share pricing information about articles with friends, but not friends of friends).
  • the appropriate information of what can or should be disclosed is then transmitted from the first user's mobile device to that of the second user at 530.
  • the mobile device of the second user may perform a lookup and reference more information about articles in the inventory of the first user, including a name of the article, a location of the article on or about the first user's body (e.g., a shirt is worn on the upper torso), any historical purchase information about the article, and information about where the article can be purchased now or was purchased. This information can be wholly or partially determined from information received from the first user's mobile device.
  • the AR-capable mobile device of the second user can request additional information about such items from a store or other online resource over the Internet using a data connection.
  • information can be searched against a commercial database such as an Amazon et al., to determine whether alternatives for (i.e., products similar to) an identified article are available. This can include alternatives that are less expensive, similar in size, color, shape, material, and the like. It is even possible to offer alternatives that are the "most selected" by others or most popular as selected via by ratings, review sites, or other crowdsourced data.
  • Fig. 6 is a block diagram of a mobile device in accordance with the principles described herein.
  • the mobile device is an AR-capable device such as HoloLensTM.
  • the block diagram configuration includes a bus-oriented 650 configuration interconnecting a processor 620, and a memory 645.
  • the configuration of Fig. 6 also includes a network interface 605 to a gateway, such as router or gateway.
  • the router or gateway may utilize either a wired or a wireless interface to the mobile device.
  • the network interface also provides a short range wireless interface.
  • Processor 620 provides computation functions for the mobile device.
  • bus 650 provides a communication path between the various elements of the embodiment and that other point-to-point interconnection options (e.g. non- bus architecture) are also feasible.
  • the mobile device is queried by another mobile device regarding a social connection with a second user in close proximity of the first user, the second user typically having line of sight to the first user.
  • the processor performs the query by accessing memory and/or via the network interface in order to access social connection information residing in a server or cloud storage.
  • the mobile device acts as a gatekeeper to determine whether or not the first user's information is made available to other individuals (e.g., the second user).
  • Relationships set up through a social networking service such as Facebook, Google+, Instagram, Snapchat, Linkedln, and the like are used to see if two people (the first and second users) have a relationship, e.g., that they are "friends” or have “friends” that are in common.
  • Such relationships can be stored in the memory accessible to the mobile device, allowing the relationships to be stored in local memory, or on a remote server (e.g, "in the cloud”). This enables the mobile device to perform a lookup to determine whether a relationship exists between the first user and the second user.
  • Social networking sites can be checked periodically by the mobile device to maintain a list of contacts that have a relationship with the first user.
  • This information can be downloaded to a mobile device of the first user and can be referenced when a second user approaches or queries the mobile device through a short range wireless technology.
  • the queried mobile device can verify a social relationship with a user identifying information that corresponds to the second user.
  • the relationship of the users is verified by using a data communication channel to query a social networking service about whether or not the user identifying information corresponding to the second user is associated with the account corresponding to the first user on the social network service.
  • the result of the lookup indicates whether there is a relationship between the two users, and if so, may further specify the nature of that relationship.
  • the mobile device determines what information can be disclosed and transmitted to the second mobile device (that of the second user), based on the results of the lookup. If it is affirmed that there is a social relationship between the first and second users (as verified by the social information), a determination is made as to what information is to be represented, that is, is allowed to be disclosed or transmitted to the second user. This determination may depend upon the nature of the relationship between the first and second users (for example, the degrees of separation between the two users: Friends (1 degree), friend of a friend (2 degrees) etc.) and what information about the articles presently with the user are to be communicated to the other users based on the nature of the relationship between them. What is to be shared given a particular nature of relationship can be defined explicitly per article in the inventory or can be more generally represented as policies (e.g., only share pricing information about articles with friends, but not friends of friends).
  • the appropriate information of what can or should be disclosed is then transmitted from the first user's mobile device to that of the second user via the network interface.
  • the mobile device of the second user may perform a lookup and reference more information about articles in the inventory of the first user, including a name of the article, a location of the article on or about the first user's body (e.g., a shirt is worn on the upper body), any historical purchase information about the article, and information about where the article can be purchased now or was purchased. This information can be wholly or partially determined from information received from the first user's mobile device.
  • the AR-capable mobile device of the second user can request additional information about such items from a store or other online resource over the Internet using a data connection.
  • information can be searched against a commercial database such as an Amazon et al., to determine whether alternatives for (i.e., products similar to) an identified article are available. This can include alternatives that are less expensive, similar in size, color, shape, material, and the like. It is even possible to offer alternatives that are the "most selected" by others or most popular as selected via by ratings, review sites, or other crowdsourced data.
  • Memory 645 is accessible to processor 620 and can act as a repository for memory related to any of the methods that incorporate the functionality of the media device. Memory 645 can provide the repository for storage of information such as program memory, downloads, uploads, or scratchpad calculations. The inventory of articles is stored along with the associated description information. The inventory and corresponding descriptions can be stored within the memory accessible to the mobile device so that the information can be made available to other individuals using short range wireless technology. One or more external databases can be accessed via the database interface 635. Those of skill in the art will recognize that memory 645 may be incorporated all or in part of processor 620. Portions of memory 645 may be remote from the mobile device (not shown) such that access by processor 620 is wireless.
  • Network interface 605 has both receiver and transmitter elements for communication as known to those of skill in the art.
  • User interface and display 610 is driven by interface circuit 615.
  • the interface 610 is used as a multimedia interface having both audio and video capability to display streamed and/or downloaded audio and/or video obtained via network interface 605 connected to a network.
  • user interface and display 610 includes a head-mounted display, including the sensors as previously discussed. It is also possible for the mobile device, at this point, to accept a manually entered (typed in or, with voice recognition, dictated) description of the articles through a UI. Alternatively, the UI can accept a manual selection by the first user of the articles from a list of articles historically entered and available via the memory described above.
  • the first user can also specify where an article is being worn or carried (pants on the lower body, T-shirt on an upper body, etc.).
  • the first user can also define where particular articles can be (or were) purchased. For example, when defining the inventory of what the first user is wearing on a particular day, the first user can also define that this item is available from a specific retailer for purchase.
  • Such associations can be made through a user interface where the first user makes such associations by inputting information into a form or other UI.
  • an image recognition module can be used to recognize different articles (e.g., each specific item of clothing) that a user may wear or carry today.
  • Amazon has a service where a customer can take a picture of an article and information about the article is returned. Whether done through a live person via the Amazon Mechanical Turk service or through machine vision techniques, the Amazon service resolves the picture to determine the identity and attributes of the article(s). When such a service returns multiple alternative articles as matching the image, the user can select which of the alternatives is correct.
  • a mobile device can be configured to remember where the first user has purchased specific items, e.g., through an online retailer such as Amazon, or based on automatic or manual entry of a shopping receipt.
  • the device can reference memory to see if that article is listed on a purchase history log that resides within the mobile device, or in remote storage accessible by the mobile device (e.g., server, gateway or cloud storage). If the mobile device recognizes that the article was previously purchased from a retailer, the mobile device can automatically import, or offer to the first user the option of importing, some or all of this information to be made available to others through the present invention, using the short range wireless technology referenced above.
  • Database interface 635 is used to connect the mobile device to a database of information about known articles (e.g., to retailers of clothing items for clothing item descriptions, costs, and any other information, for example to facilitate comparison shopping by the second user).
  • the databases can be accessed via a public or private network and the database interface 635 represents the network interface connecting to the databases.
  • the database interface 635 is used to connect to a local database (not shown).
  • User interface and display 610 and its respective display driver 615 may be used to manage local data in memory 645, in particular the inventory created as described in conjunction with step 505.
  • Camera 655 is configured to scan the user's articles and the corresponding description tags. .
  • This scanning of the first user's articles defines an inventory of articles to be worn or carried by the first user and is the preferred method by which the mobile device identifies the articles.
  • the smartphone can utilize QR codes that are on such articles (e.g., the first user's clothing) or UPC codes associated with the articles.
  • the processor 620 is shown to have access to the camera 655 via the communication bus 650, though communication with the camera may incorporate a wireless link (not shown), for example if the camera 655 is located on an HMD remote from the processor 620.
  • Fig. 7 is a flowchart of the operation of a device of the second user.
  • the device of the second user receives a request from the second user for information about an article of a first user.
  • this can be achieved by the second user centering his field of view in the AR device on the article of the second user and speaking a trigger phrase such as "What is that?" which is recognized by a speech detection module, where both gaze direction determination and speech detection modules are components of the user interface 610.
  • Identification of the first user as being in or near the second user's field- of-view can be achieved as described above.
  • the device of the second user sends a query to the device of the first user regarding social connection information, the query including identifying information about the second user.
  • Information about the article should not be returned to the device of the second user until the degrees of relationship (social connection) between the first user and the second user is determined, if any.
  • the social connection (relationship) determination is performed by the device of the first user using either locally stored relationship information, or by accessing social networking sites such as Facebook, Linkedln etc.
  • the device of the first user performs the steps 515, 520, 525, and 530, as described in conjunction with Fig. 5.
  • the selected information about the article is received from the device of the first user by the device of the second user.
  • the selected information about the article may be enhanced by the second device by a query to additional resources (e.g., the store server of Fig. 5).
  • the selected information about the article is displayed for the second user by the second user's device.
  • the article information includes availability of the article including information about the retailer and pricing information.
  • the device of the second user may be a smartphone or AR device or a combination of the smartphone and the AR device.
  • the mobile device of the second user includes components similar to those of the first user described above and shown in Fig. 6.
  • the processor 620 of the mobile device e.g., AR-capable device
  • the processor 620 of the mobile device of the second user receives the information about the article of the first user via the network interface 605.
  • the received information about the article of the first user is forwarded to the interface and display driver 615 which forwards the information about the article of the first user to the user interface and display 610.
  • the proposed method and apparatus may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof.
  • Special purpose processors may include application specific integrated circuits (ASICs), reduced instruction set computers (RISCs) and/or field programmable gate arrays (FPGAs).
  • ASICs application specific integrated circuits
  • RISCs reduced instruction set computers
  • FPGAs field programmable gate arrays
  • the proposed method and apparatus is implemented as a combination of hardware and software.
  • the software is preferably implemented as an application program tangibly embodied on a program storage device.
  • the application program may be uploaded to, and executed by, a machine comprising any suitable architecture.
  • the machine is implemented on a computer platform having hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s).
  • the computer platform also includes an operating system and microinstruction code.
  • the various processes and functions described herein may either be part of the microinstruction code or part of the application program (or a combination thereof), which is executed via the operating system.
  • various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device. It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof.
  • these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input/output interfaces.
  • general-purpose devices which may include a processor, memory and input/output interfaces.
  • the phrase "coupled” is defined to mean directly connected to or indirectly connected with through one or more intermediate components. Such intermediate components may include both hardware and software based components.

Landscapes

  • Business, Economics & Management (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Development Economics (AREA)
  • Strategic Management (AREA)
  • Engineering & Computer Science (AREA)
  • Marketing (AREA)
  • Economics (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Game Theory and Decision Science (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

La présente invention concerne un procédé et un appareil permettant de faire fonctionner un dispositif consistant à balayer un article d'un premier utilisateur et à stocker dans une liste au moins une information concernant ledit article balayé. Le procédé et l'appareil consistent en outre à recevoir une demande concernant une connexion sociale avec un second utilisateur, à effectuer une consultation pour déterminer ladite connexion sociale entre le premier utilisateur et le second utilisateur et à déterminer que ladite information concernant l'article balayé peut être présentée au second utilisateur sur la base de la connexion sociale. L'invention concerne également un procédé et un appareil permettant de faire fonctionner un dispositif consistant à demander, par un second utilisateur, des informations concernant un article d'un premier utilisateur, à recevoir, par le second utilisateur, les informations concernant l'article du premier utilisateur et à afficher pour le second utilisateur les informations concernant l'article du premier utilisateur.
PCT/US2017/041320 2016-07-21 2017-07-10 Réseautage social à réalité augmentée WO2018017353A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201662365182P 2016-07-21 2016-07-21
US62/365,182 2016-07-21

Publications (1)

Publication Number Publication Date
WO2018017353A1 true WO2018017353A1 (fr) 2018-01-25

Family

ID=59366543

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2017/041320 WO2018017353A1 (fr) 2016-07-21 2017-07-10 Réseautage social à réalité augmentée

Country Status (1)

Country Link
WO (1) WO2018017353A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10755487B1 (en) * 2018-05-22 2020-08-25 Facebook, Inc. Techniques for using perception profiles with augmented reality systems

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130044130A1 (en) * 2011-08-17 2013-02-21 Kevin A. Geisner Providing contextual personal information by a mixed reality device
US20150126226A1 (en) * 2012-06-06 2015-05-07 Donde Fashion Inc Wearable articles identification
US20150279117A1 (en) * 2014-04-01 2015-10-01 Hallmark Cards, Incorporated Augmented Reality Appearance Enhancement

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130044130A1 (en) * 2011-08-17 2013-02-21 Kevin A. Geisner Providing contextual personal information by a mixed reality device
US20150126226A1 (en) * 2012-06-06 2015-05-07 Donde Fashion Inc Wearable articles identification
US20150279117A1 (en) * 2014-04-01 2015-10-01 Hallmark Cards, Incorporated Augmented Reality Appearance Enhancement

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10755487B1 (en) * 2018-05-22 2020-08-25 Facebook, Inc. Techniques for using perception profiles with augmented reality systems

Similar Documents

Publication Publication Date Title
US12008619B2 (en) Methods and systems for virtual fitting rooms or hybrid stores
US11232512B2 (en) Method and device for combining an avatar image with a portion of a user's image
CN110826528B (zh) 时尚偏好分析
US10282772B2 (en) Systems and methods for wardrobe management
US8743145B1 (en) Visual overlay for augmenting reality
US20150095228A1 (en) Capturing images for financial transactions
KR102318952B1 (ko) 인공지능을 활용한 의류 추천 및 구매 방법, 장치 및 시스템
JP6069565B1 (ja) レコメンド装置、レコメンド方法、およびプログラム
US20220327747A1 (en) Information processing device, information processing method, and program
CA2934526C (fr) Cabines d'essayage intelligente
WO2015200617A1 (fr) Avatars numériques dans des marchés en ligne
KR101620938B1 (ko) 의류 제품 판매 지원 장치와 이와 통신 가능한 의류 제품 정보 관리 서버, 의류 연관 제품 추천 서버 및 의류 제품 정보 제공 방법
JP2017054283A (ja) アイテム販売システム、アイテム選択装置及びアイテム販売方法
WO2015066675A2 (fr) Système et procédé de contrôle et de partage en ligne d'images de marchandises
KR102289452B1 (ko) 병행수입 플랫폼을 운용하는 시스템
KR20190070452A (ko) 증강현실 의류 쇼핑을 위한 웨어러블 글래스 및 증강현실 의류 쇼핑 방법
WO2018017353A1 (fr) Réseautage social à réalité augmentée
KR102614684B1 (ko) 증강 현실을 이용한 의류 구매 지원 방법 및 이에 사용되는 서비스 제공 서버
KR100928347B1 (ko) 무선인식을 이용한 스타일 코디 추천 시스템
US20240071019A1 (en) Three-dimensional models of users wearing clothing items
KR102527398B1 (ko) 화상미팅 프로그램 기반의 가상 피팅 방법 및 그 시스템
KR20140085630A (ko) 인터넷을 이용한 패션 정보 제공 시스템 및 그 방법
CN104718540A (zh) 多种环境中的基于结构化照明的内容交互
KR102555445B1 (ko) 통관 edi 폼을 관리하는 해외직구 플랫폼을 운용하는 시스템
KR102140151B1 (ko) 온라인 의류 쇼핑몰에서의 의류 구입 방법 및 이를 위한 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17740879

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17740879

Country of ref document: EP

Kind code of ref document: A1