US20160180402A1

US20160180402A1 - Method for recommending products based on a user profile derived from metadata of multimedia content

Info

Publication number: US20160180402A1
Application number: US14/627,264
Authority: US
Inventors: Mohammad Sabah; Mohammad Iman SADREDDIN; Shafaq Abdullah
Original assignee: Honest Company Inc; Insnap Inc
Current assignee: Honest Company Inc
Priority date: 2014-12-17
Filing date: 2015-02-20
Publication date: 2016-06-23
Also published as: US9805098B2; US20160203137A1; US20180082200A1; US20160180235A1; US20180052855A1; US20160203141A1; US9798980B2

Abstract

Techniques disclosed herein describe identifying one or more products to recommend to a plurality of users based on metadata of digital multimedia files. A product feed extractor extracts a product feed. The product feed lists one or more items. The product feed extractor identifies, for each item in the product feed, one or more attributes describing the item. Each item is mapped to concepts of an interest taxonomy based on the identified one or more attributes for the item. One or more users are associated with each concept in the interest taxonomy based on the metadata of the digital multimedia files. Each item is associated to one or more of the users based on the mapping.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 62/093,372, filed Dec. 17, 2014. The content of the aforementioned application is incorporated by reference in its entirety.

BACKGROUND

1. Field
Embodiments of the present disclosure generally relate to data analytics. More specifically, to recommending products based on a user profile derived from metadata of digital multimedia (e.g., images, videos, etc.).
2. Description of the Related Art
Individuals take images and videos to capture personal experiences and events. The images and videos can represent mementos of various times and places experienced in an individual's life.
In addition, mobile devices (e.g., smart phones, tablets, etc.) allow individuals to easily capture digital multimedia. For instance, cameras in mobile devices have steadily improved in quality and are can capture high-resolution images. Further, mobile devices now commonly have a storage capacity that can store thousands of images. And because individuals can easily carry smart phones around with them, they can take a greater number of images in many places.
All of this has resulted in an explosion of images, and metadata describing images, as virtually anyone can capture and share digital images via text message, image services, social media, and the like. This volume of digital images, now readily available, provides variety of information valuable to third parties, such as advertisers, marketers, and the like.

SUMMARY

One embodiment presented herein describes a method for identifying one or more products to recommend to a plurality of users based on metadata of digital multimedia files. The method generally includes, extracting a product feed. The product feed lists one or more items. The method also includes identifying, for each item in the product feed, one or more attributes describing the item. Each item is mapped to concepts of an interest taxonomy based on the identified one or more attributes for the item. One or more users are associated with each concept in the interest taxonomy based on the metadata of the digital multimedia files. Each item is associated to one or more of the users based on the mapping.
Other embodiments include, without limitation, a computer-readable medium that includes instructions that enable a processing unit to implement one or more aspects of the disclosed methods as well as a system having a processor, memory, and application programs configured to implement one or more aspects of the disclosed methods.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only exemplary embodiments and are therefore not to be considered limiting of its scope, may admit to other equally effective embodiments.

FIG. 1 illustrates an example computing environment, according to one embodiment.

FIG. 2 further illustrates the mobile application described relative to FIG. 1, according to one embodiment.

FIG. 3 further illustrates the analysis tool described relative to FIG. 1, according to one embodiment.

FIG. 4 further illustrates the product feed extractor described relative to FIG. 1, according to one embodiment.

FIG. 5 illustrates a method for building an interest taxonomy across a userbase, according to one embodiment.

FIG. 6 illustrates a method for inferring user interests from concepts derived based on image metadata, according to one embodiment.

FIG. 7 illustrates a method for recommending products based on inferred interests derived from image metadata, according to one embodiment.

FIG. 8 illustrates an application server computing system, according to one embodiment.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation.

DETAILED DESCRIPTION

Embodiments presented herein describe techniques for recommending products to users based on user interests inferred from image metadata. Digital images provide a wealth of information valuable to third parties (e.g., advertisers, marketers, and the like). For example, assume an individual takes pictures at a golf course using a mobile device (e.g., a smart phone, tablet, etc.). Further, assume that the pictures are the only indication the individual was at the golf course (e.g., because the individual made only cash purchases and signed no registers). Metadata associated with this image can place the individual at the golf course at a specific time. Further, event data could be used to correlate whether there was going on at that time (e.g., a specific tournament). Such information may be useful to third parties, e.g., for targeted advertising and recommendations.
However, an advertiser might not be able to identify an effective audience for targeting a given product or service based on such information alone. Even if image metadata places an individual at a golf course at a particular point of time, the advertiser might draw inaccurate inferences about the individual. For example, the advertiser might assume that because the metadata places the individual at a high-end golf course, the individual is interested in high-end golf equipment. The advertiser might then recommend other high-end equipment or other golf courses to that individual. If the individual rarely plays golf or does not usually spend money at high-end locations. Such recommendations may lead to low conversion rates for the advertiser. Historically, advertisers have been generally forced to accept low conversation rates, as techniques for identifying individuals likely to be receptive to or interested in a given product or service are often ineffective.
Embodiments presented herein describe techniques for recommending products based on user interests inferred from metadata of digital multimedia (e.g., images and videos). In one embodiment, a multimedia service platform provides a mobile application which allows users to upload digital multimedia files and metadata to the platform from a mobile device. Further, the multimedia service platform may identify patterns from metadata extracted from images and videos. The metadata may describe where and when a given multimedia file was taken. Further, in many cases, embodiments presented herein can identify latent relationships between user interests from collections of metadata from multiple users. For example, if many users who take pictures at golf courses also take pictures at an unrelated event (e.g., take pictures of a traveling museum exhibit) then the system disclosed herein can discover a relationship between the interests. Thereafter, advertising related to golfing products and services could be targeted to individuals who publish pictures of the travelling museum exhibit, regardless of any other known interest in golf.
In one embodiment, the multimedia service platform evaluates metadata corresponding to each image or video submitted to the platform against a knowledge graph. The knowledge graph provides a variety of information about events, places, dates, times, etc. that may be compared with metadata of the image or video. For example, the knowledge graph may include weather data, location data, event data, and online encyclopedia data. For instance, attributes associated with an event may include a name, location, start time, end time, price range, etc. The multimedia service platform correlates spatiotemporal metadata from a digital image or video with a specific event in the knowledge graph. That is, the knowledge graph is used to impute attributes related to events, places, dates, times, etc., to a given digital image or video based on the metadata provided with that image or video.
In one embodiment, the analysis tool represents attributes imputed to digital multimedia from a user base in a user-attribute matrix, where each row of the matrix represents a distinct user and each column represents an attribute from the knowledge graph that can be imputed to a digital multimedia file. The analysis tool may add columns to the user-attribute matrix as additional attributes are identified. The cells of a given row indicate how many times a given attribute has been imputed to a digital multimedia file published by a user corresponding to that row. Accordingly, when the analysis tool imputes an attribute to a digital multimedia file (based on the file metadata), a value for that attribute is incremented in the user-attribute matrix. Doing so allows the multimedia service platform to identify useful information about that user. For instance, the analysis tool may identify that a user often attends sporting events, movies, participates in a particular recreational event (e.g., skiing or golf), etc. In addition, the analysis tool may identify information about events that the user attends, such as whether the events are related to a given sports team, whether the events are related to flights from an airport, a range specifying how much the event may cost, etc.
In one embodiment, the multimedia service platform may learn concepts. A concept is a collection of one or more identified attributes. The multimedia service platform may perform machine learning techniques to learn concepts from the attributes of the user-attribute matrix. For example, the multimedia service platform may score an attribute to each respective concept. The multimedia service platform may associate attributes that satisfy specified criteria (e.g., the top five scores per concept, attributes exceeding a specified threshold, etc.) to a given concept.
Further, the analysis tool may generate an interest taxonomy based on the user-attribute matrix. In one embodiment, an interest taxonomy is a hierarchical representation of user interests based on the concepts. For example, the interest taxonomy can identify general groups (e.g., sports, music, and travel) and sub-groups (e.g., basketball, rock music, and discount airlines) of interest identified from the concepts.
The multimedia service platform may use the interest taxonomy to discover latent relationships between concepts. For example, the multimedia service platform may build a predictive learning model using the interest taxonomy. The multimedia service platform could train the predictive learning model using existing user-to-concept associations. Doing so would allow the multimedia service platform use the model to predict associations for users to other concepts that the user is not currently associated with.
Further, the multimedia service platform may map distinct product and service feeds of third parties (e.g., retailers, travel services, venues, etc.) to the user interest taxonomy to identify products and services to recommend to a given user. Generally, a product feed is a listing of items that are provided commercially. For example, a product feed of a clothing retailer may list items such as shirts, pants, shoes, and accessories. Further, each item may contain various information about the item, such as a name of the item, type of the item, price of the item, size information for the item, description of the item, and the like. The product feed may be hosted on a website of the third party or be provided by the third party to the multimedia service platform.
In one embodiment, a product feed extractor of multimedia service platform retrieves a product feed from a third party system, such as from a web server of a retailer. The product feed extractor evaluates each item in the product feed to identify item attributes. The product feed extractor may build an item-attribute matrix, where rows represent items and columns represent attributes. Each cell includes a bit representing whether a given item has a given attribute. The product feed extractor determines a mapping for each product to a concept, if available, based on the item-attribute matrix. The product feed extractor may then identify users that may be interested in a given item based on whether a user is associated with a corresponding concept.
Note, the following description relies on digital images captured by a user and metadata as a reference example of determining product recommendations based on a user profile derived from image metadata. However, one of skill in the art will recognize that the embodiments presented herein may be adapted to other digital multimedia that include time and location metadata, such as digital videos captured on a mobile device. Further, an analysis tool may be able to extract additional metadata features from such videos, such as the length of the video, which can be used relative to the techniques described herein.
FIG. 1 illustrates an example computing environment 100, according to one embodiment. As shown, the computing environment 100 includes one or more mobile devices 105, an extract, transform, and load (ETL) server 110, an application server 115, and one or more third party systems 125, connected to a network 130 (e.g., the Internet).
In one embodiment, the mobile devices 105 include a mobile application 106 which allows users to interact with a multimedia service platform (represented by the ETL server 110 and the application server 115). In one embodiment, the mobile application 106 is developed by a third-party organization (e.g., a retailer, social network provider, fitness tracker developer, etc.). The mobile application 106 may send images 108 and associated metadata to the multimedia service platform, e.g., through a software development kit (SDK) provided by the multimedia service platform.
In another embodiment, the mobile application 106 may access a social media service (application service 116) provided by the multimedia service platform. The social media service allows users to capture, share, and comment on images 108 as a part of existing social networks (or in junction) with those social networks. For example, a user can link a social network account to the multimedia service platform through application 106. Thereafter, the user may capture a number of images and submit the images 108 to the social network. In turn, the application 106 retrieves the metadata from the submitted images. Further, the mobile application 106 can send images 108 and metadata to the multimedia service platform. The multimedia service platform uses the metadata to infer latent interests of the user.
In any case, the mobile application 106 extracts Exchangeable Image Format (EXIF) metadata from each image 108. The mobile application 106 can also extract other metadata (e.g., PHAsset metadata in Apple iOS devices) describing additional information, such as GPS data. In addition, the mobile application 106 may perform extract, transform, and load (ETL) operations on the metadata to format the metadata for use by components of the multimedia service platform. For example, the mobile application 106 may determine additional information based on the metadata, such as whether a given image was taken during daytime or nighttime, whether the image was taken indoors or outdoors, whether the image is a “selfie,” etc. Further, the mobile application 106 also retrieves metadata describing application use. Such metadata includes activity by the user on the mobile application 106, such as image views, tagging, etc. Further, as described below, the mobile application 106 provides functionality that allows a user to search through a collection of images by the additional metadata, e.g., searching a collection of images that are “selfies” and taken in the morning.
In one embodiment, the ETL server 110 includes an ETL application 112. The ETL application 112 receives streams of image metadata 114 (e.g., the EXIF metadata, PHAsset metadata, and additional metadata) from mobile devices 105. Further, the ETL application 112 cleans, stores, and indexes the image metadata 114 for use by the application server 115. Once processed, the ETL application 112 may store the image metadata 114 in a data store (e.g., such as in a database) for access by the application server 115. In one embodiment, the ETL server 110 may be a physical computing system or a virtual machine computing instance in the cloud. Although depicted as a single server, the application server 110 may comprise multiple servers configured as a cluster (e.g., via the Apache Spark framework on top of Hadoop-based storage). This architecture allows the application servers 110 to process large amounts of images and image metadata sent from mobile applications 106.
In one embodiment, an application service 116 communicates with the mobile application 106. In one embodiment, the application server 115 may be a physical computing system or a virtual machine computing instance in the cloud. Although depicted as a single server, the application server 115 may comprise multiple servers configured as a cluster (e.g., via the Apache Spark framework on top of a Hadoop-based storage architecture). This architecture allows the application servers 115 to process large amounts of images and image metadata sent from mobile applications 106.
As shown, the application server 115 includes an analysis tool 117, a knowledge graph 118, and a user interest taxonomy 119. In one embodiment, the analysis tool 117 generates the user interest taxonomy 119 based on image metadata 114 from image collections of multiple users. As described below, the user interest taxonomy 119 represents interests inferred from image attributes identified from the knowledge graph 118.
In one embodiment, the knowledge graph 118 includes a collection of attributes which may be imputed to an image. Examples of attributes include time and location information, event information, genres, price ranges, weather, subject matter, and the like. The analysis tool 117 builds the knowledge graph 118 using weather data, location data, events data, encyclopedia data, and the like from a variety of data sources.
In one embodiment, the analysis tool 117 imputes attributes from the knowledge graph 118 to the images 108 based on the metadata 114. That is, the analysis tool 117 may correlate time and location information in image metadata 114 to attributes in the knowledge graph 118. For example, assume that a user captures an image 108 of a baseball game. Metadata 114 for that image 108 may include a GPS, a date, and a time when the image 108 was captured. The analysis tool 117 can correlate this information to attributes such as weather conditions at that time and location (e.g., “sunny”), an event name (e.g., “Dodgers Game”), teams playing at that game (e.g., “Dodgers” and “Cardinals”), etc. The analysis tool 117 associates the imputed attributes with the user who took the image. As noted, e.g., a row in a user attribute matrix may be updated to reflect the imputed attributes of each new image taken by that user. Further, the analysis tool 117 may perform machine learning techniques, such as latent Dirichlet analysis (LDA), to decompose the user-attribute matrix into sub-matrices. Doing so allows the analysis tool 117 to identify concepts, i.e., clusters of attributes.
As described further below, the product feed extractor 120 may use the user interest taxonomy 119 to identify commercial products and services of a third party (e.g., a retailer, airlines company, health and fitness organization, etc.) that may be of interest to a user.
For example, the product feed extractor 120 may retrieve information from a product feed 127 of a third party system 125. In one embodiment, the product feed 127 is a listing of commercial products or services of a third party, such as those of a retailer. For example, a product feed 127 of a shoe retailer may list items such as dress shoes, casual shoes, sports shoes, etc. Further, each item may contain various information about the item, such as a name of the item, type of the item, price of the item, size information for the item, description of the item, and the like. The product feed extractor 120 may identify, from the product feed 127, one or more attributes describing each product. For example, a product of a shoe retailer may have attributes such as “shoe,” “running,” “menswear,” and so on. The product feed extractor 120 can map the attributes of the product feed 127 with concepts in the interest taxonomy 119. Doing so allows the analysis tool 117 to identify products and services from the feed 127 that align with certain user interests identified in the interest taxonomy. As a result, third parties can target users who may be interested in the identified products and services.
FIG. 2 illustrates mobile application 106, according to one embodiment. As shown, mobile application 106 includes a SDK component 200 used to send image and metadata information to the multimedia service platform. The SDK component 200 further includes an extraction component 205, a search and similarity component 210, and a log component 215. In one embodiment, the extraction component 205 extracts metadata (e.g., EXIF metadata, PHAsset metadata, and the like) from images captured using a mobile device 105. Further, the extraction component 205 may perform ETL preprocessing operations on the metadata. For example, the extraction component 205 may format the metadata for the search and similarity component 210 and the log component 215.
In one embodiment, the search and similarity component 210 infers additional metadata from an image based on the metadata (e.g., spatiotemporal metadata) retrieved by the extraction component 205. Examples of additional metadata include whether a given image was captured at daytime or nighttime, whether the image was captured indoors or outdoors, whether the image was edited, weather conditions when the image was captured, etc. Further, the search and similarity component 210 generates a two-dimensional image feature map from a collection of images captured on a given mobile device 105, where each row represents an image and columns represent metadata attributes. Cells of the map indicate whether an image has a particular attribute. The image feature map allows the search and similarity component 210 to provide analytics and search features for the collection of images captured by a mobile device. For example, a user of the mobile application 106 may search for images on their mobile device which have a given attribute, such as images taken during daytime or taken from a particular location. In turn, the search and similarity component 210 may evaluate the image map to identify photos having such an attribute.
In one embodiment, the log component 215 evaluates the image metadata. For example, the log component 215 records metadata sent to the ETL server 110. Once received, the application 112 performs ETL operations, e.g., loading the metadata into a data store (such as a database). The metadata is accessible by the analysis tool 117.
FIG. 3 further illustrates the analysis tool 117, according to one embodiment. As shown, the analysis tool 117 includes an aggregation component 305, a knowledge graph component 310, a taxonomy component 320, and a user interest inference component 325.
In one embodiment, the aggregation component 305 receives streams of image metadata corresponding to images captured by users of application 106 by users from the ETL server 110. Once received, the aggregation component 305 organizes images and metadata by user. The metadata may include both raw image metadata (e.g., time and GPS information) and inferred metadata (e.g., daytime or nighttime image, indoor or outdoor image, “selfie” image, etc.). To organize metadata by user, the aggregation component 305 evaluates log data from the ETL server 110 to identify image metadata from different devices (and presumably different users) and metadata type (e.g., whether the metadata corresponds to image metadata or application usage data).
In one embodiment, the knowledge graph component 310 (and later maintains) the knowledge graph 118 using any suitable data source, such as local news and media websites, online event schedules for performance venues, calendars published by schools, government, or private enterprises, online schedules and ticket sales. The knowledge graph component 310 determines a set of attributes related to each event to store in the knowledge graph 118.
In one embodiment, to impute attributes from the knowledge graph 118 to a given image, the knowledge graph component 310 evaluates time and location metadata of the image against the knowledge graph 118. The knowledge graph component 310 determines whether the image metadata matches a location and/or event in the knowledge graph. The information may be matched using a specified spatiotemporal range, e.g., within a time period of the event, within a set of GPS coordinate range, etc. In one embodiment, the knowledge graph component 310 may further match the information based on a similarity of metadata of other user photos that have been matched to that event.
In one embodiment, the taxonomy component 320 evaluates the user-attribute matrix to determine concepts associated with a given user. As stated, a concept is a cluster of related attributes. The interest taxonomy generation component 320 may perform machine learning techniques, such as Latent Dirichlet Analysis (LDA), Non-Negative Matrix Factorization (NNMF), Deep Learning algorithms, and the like, to decompose the user-attribute matrix into sub-matrices. The taxonomy component 320 evaluates the sub-matrices to identify latent concepts from co-occurring attributes.
Further, the taxonomy component 320 may determine a score distribution for each attribute over each concept. The taxonomy component 320 may populate a concept-attribute matrix, where the concepts are rows and attributes are columns. Each cell value is the membership score of the respective attribute to the respective concept. The taxonomy component 320 may perform further machine learning techniques (e.g., LDA, NNMF, Deep Learning algorithms, etc.) to identify relationships and hierarchies between each concepts.
In one embodiment, the interest inference component 325 builds a learning model based on the identified concepts and the users. To do so, the interest inference component 325 may train Support Vector Machine (SVM) classifiers for each concept to determine user association in one or more concepts. Doing so results in each user in the platform being assigned an interest score per concept.
Once trained, the interest inference component 325 may predict user interests using the learning model. As the multimedia service platform receives image metadata from new users, the interest inference component 325 can assign the new users with scores for each concept based on the metadata and the learning model. A user having a high membership score in a given concept may indicate a high degree of interest for that concept. The interest inference component 325 may build a user-concept matrix, where rows represent users and columns represent concepts. A cell in the matrix represent a score for a given user-concept combination.
FIG. 4 further illustrates the product feed extractor 120, according to one embodiment. As shown, the product feed extractor 120 includes a retrieval component 405, an evaluation component 410, a mapping component 415, and an identification component 420.
In one embodiment, the retrieval component 405 extracts a product feed from a system of a third party organization, such as a website of a retailer, fitness organization, or travel company. An example of the product feed is a product inventory provided on a website of a sports clothing retailer. As stated, a product feed is a listing of commercial products or services provided by the organization. Continuing the previous example, the product feed of a sports clothing retailer includes items such as running shoes, basketball shorts, baseball caps, etc. Further, each item in the product feed may include information associated with the item, such as a name of the item, a price of the item, an average rating of the item by consumers, price, a type of the item, a description of the item, and the like.
In one embodiment, the transformation component 410 determines one or more attributes of each item to associate with the item. Continuing the previous example of a sports clothing retailer, attributes of a given item may include “shoes,” “black,” “running,” “menswear,” and so on. The transformation component 410 may perform NLP techniques such as tokenization, lexical analysis, semantic analysis, and pattern matching, to identify attributes. In one embodiment, the transformation component 410 builds an item-attribute matrix, where rows represent evaluated items and columns represent attributes. If a given item is associated with a given attribute, the transformation component 410 flags the corresponding cell value as 1 (and 0 if the attribute is not present).
In one embodiment, the mapping component 415 associates item attributes to concepts of the interest taxonomy. For example, the mapping component 415 may perform NLP and Machine Learning techniques to determine word space model distance of a given item attribute from a concept. The mapping component 415 can determine a score based on such distances. The mapping component 415 may associate an attribute having score that exceeds a given threshold for a given concept with that concept. The mapping component 415 may build an item-concept matrix, where rows represent items and columns represent concepts. Cells represent a concept score for a given item-concept combination.
In one embodiment, the identification component 420 determines one or more products that may be of interest to a given user. To do so, the identification component 420 may evaluate a dot product between a user vector in the user-concept matrix and the product-concept matrix. The identification component 420 may determine that products exceeding a threshold score for that user indicates that a user may have interest in that product. A third party may use such information to target specific recommendations for that product to the user.
FIG. 5 illustrates a method 500 for building an interest taxonomy across a userbase, according to one embodiment. Method 500 begins at step 505, where the aggregation component 305 segments images by users. Doing so allows the analysis tool 107 to evaluate collections of image metadata for each user individually.
At step 510, the knowledge graph component 310 imputes attributes from the knowledge graph 118 onto the images based on the image metadata. To do so, the graph component 310 correlates time and location metadata of a given image to information provided in the knowledge graph, such as events, that coincide with the time and location metadata (with a degree of allowance). As a result, each image is associated with a set of attributes.
At step 515, the knowledge graph component 310 builds a user-attribute matrix based on the imputed attributes to the images. The knowledge graph component 310 further imputes attributes associated with each image to the respective user. Each cell in the user-attribute matrix is an incremental value that represents a count of images in which the corresponding attribute is present.
At step 520, the interest taxonomy generation component 320 decomposes the user-attribute matrix to identify concepts from the attributes. As stated, a concept may include one or more attributes. The interest taxonomy generation component 320 may evaluate the attributes using machine learning techniques to identify the concepts. Further, the interest taxonomy generation component 320 may generate an attribute-concept matrix, where the cell values represent membership scores of each attribute to a given concept. Attributes having a qualifying score may be associated with the concept.
FIG. 6 illustrates a method 600 for inferring user interests from concepts derived based on image metadata, according to one embodiment. Method 600 begins at step 605, where the analysis tool 117 determines, for each user, an interest score for each concept relative to other concepts. To do so, the analysis tool 117 may calculate a dot product between a user vector in the user-attribute matrix to the attribute-concept matrix.
At step 610, the analysis tool 117 assigns each user to one or more concepts based on the interest scores. To do so, the analysis tool 117 may determine whether a given interest score exceeds a threshold for that concept. And if so, the analysis tool 117 associates the user with that concept.
At step 615, the analysis tool 117 trains multiple one-versus-all predictive models for inferring user interests. The analysis tool 117 may use associations between a user and a concept as positive examples for association to that concept. The analysis tool 117 may also use lack of associations between a user and a concept as negative examples.
FIG. 7 illustrates a method 700 for recommending products based on inferred interests derived from image metadata, according to one embodiment. At step 705, the retrieval component 405 extracts a product feed of a third party system. For example, assume the retrieval component 405 extracts a product feed from a website of sports clothing retailer. The product feed includes items such as running shoes, basketball shorts, baseball caps, etc. Further, each item in the product feed includes information associated with the item (e.g., a name of the item, a price of the item, an average rating of the item by consumers, price, a type of the item, a description of the item, etc.).
At step 710, the transformation component 410 determines a set of attributes for each item. The transformation component 410 performs NLP techniques over the raw text associated with a given item, such as tokenization, lexical and semantic analysis, pattern matching, and so on. Doing so results in a set of attributes for each item (e.g., “outerwear,” “shoes,” “Mercury 7,” “menswear,” “running,” etc.). Further, the transformation component 410 builds an item-attribute matrix, where rows represent evaluated items and columns represent attributes. As stated, if a given item is associated with a given attribute, the transformation component 410 flags the corresponding cell value as 1 (and 0 if the attribute is not present).
At step 715, the mapping component 415 associates product feed attributes with learned concepts of the interest taxonomy. To do so, the mapping component 415 determines a word space model distance of a given item attribute from a concept. Further, the mapping component 415 determines a score based on such distances. The mapping component 415 associates an attribute having score that exceeds a given threshold for a given concept with that concept. The mapping component 415 populates an item-concept matrix, where rows represent items and columns represent concepts. Cells represent a concept score for a given item-concept combination.
At step 720, the identification component 420 determines which users to target for a given product based on the associations. To do so, the identification component 420 evaluates a dot product of a user vector of the user-concept matrix and the product-concept matrix. The identification component 420 may determine that products exceeding a threshold score for that user indicates that a user may have interest in that product. As a result, a third party may use such information to target specific recommendations for that product to the user.
FIG. 8 illustrates an application server computing system 800, according to one embodiment. As shown, the computing system 800 includes, without limitation, a central processing unit (CPU) 805, a network interface 815, a memory 820, and storage 830, each connected to a bus 817. The computing system 800 may also include an I/O device interface 810 connecting I/O devices 812 (e.g., keyboard, mouse, and display devices) to the computing system 800. Further, in context of this disclosure, the computing elements shown in computing system 800 may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executing within a computing cloud.
The CPU 805 retrieves and executes programming instructions stored in the memory 820 as well as stores and retrieves application data residing in the memory 820. The interconnect 817 is used to transmit programming instructions and application data between the CPU 805, I/O devices interface 810, storage 830, network interface 815, and memory 820. Note, CPU 805 is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, and the like. And the memory 820 is generally included to be representative of a random access memory. The storage 830 may be a disk drive storage device. Although shown as a single unit, the storage 830 may be a combination of fixed and/or removable storage devices, such as fixed disc drives, removable memory cards, or optical storage, network attached storage (NAS), or a storage area-network (SAN).
Illustratively, the memory 820 includes an application service 822, an analysis tool 824, and a product feed extractor 826. The storage 830 includes a knowledge graph 834, and a user interest taxonomy 836. The application service 822 provides access to various services of a multimedia service platform to mobile devices. The analysis tool 824 generates a user interest taxonomy 836 based on metadata of images taken by users.
Further, the analysis tool 824 builds the knowledge graph 834 from external data sources. To do so, the analysis tool 824 performs NLP techniques on the raw text obtained from the data sources to identify relevant terms related to events, moments, weather, etc. Further, the analysis tool 824 may impute information from the knowledge graph 834 images submitted to the multimedia service platform. In addition, the analysis tool 824 generates a user interest taxonomy 836 of concepts inferred from the attributes. To do so, the analysis tool 824 may perform machine learning techniques to identify concepts based on co-occurring attributes. In addition, the analysis tool 824 may determine a membership score for each attribute to each identified concept. The analysis tool 824 may associate attributes to a given concept based on the membership score. Further, the analysis tool 824 may identify hierarchical relationships between the concepts through machine learning.
Further, the product feed extractor 826 identifies commercial products and services of a third party that may be of interest to a user, based on the user interest taxonomy 836. For example, the product feed extractor 823 may retrieve information from a product feed of a third party system (e.g., of a retailer). The product feed extractor 836 may identify, from the product feed 127, one or more attributes describing each product. The product feed extractor 836 can map the attributes of the product feed with concepts in the interest taxonomy 836. Doing so allows the analysis tool 824 to identify products and services from the feed that align with certain user interests identified in the interest taxonomy. As a result, third parties can target users who may be interested in the identified products and services.
While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

What is claimed is:

1. A method for identifying one or more products to recommend to a plurality of users based on metadata of digital multimedia files, the method comprising:

extracting a product feed, wherein the product feed lists one or more items;

identifying, for each item in the product feed, one or more attributes describing the item;

mapping each item to concepts of an interest taxonomy based on the identified one or more attributes for the item, wherein one or more users are associated with each concept in the interest taxonomy based on the metadata of the digital multimedia files; and

associating each item to one or more of the users based on the mapping.

2. The method of claim 1, wherein the identified attributes include at least one of a name of the item, a price of the item, a description of the item, and a type of the item.

3. The method of claim 1, further comprising:

for each attribute identified for the item, updating an item-attribute matrix to reflect the attribute identified for the item.

4. The method of claim 3, wherein mapping each item to the concepts of the interest taxonomy comprises:

evaluating the item-attribute matrix to determine at least a first concept to associate with the item.

5. The method of claim 4, further comprising:

updating an item-concept matrix to reflect the attribute identified for the first concept.

6. The method of claim 4, wherein the first concept is determined based on a co-occurrence between one or more attributes in the item-attribute matrix.

7. The method of claim 1, wherein each of the digital multimedia files is one of either an image or a video.

8. A non-transitory computer-readable storage medium storing instructions, which, when executed on a processor, performs an operation for identifying one or more products to recommend to a plurality of users based on metadata of digital multimedia files, the operation comprising:

extracting a product feed, wherein the product feed lists one or more items;

associating each item to one or more of the users based on the mapping.

9. The non-transitory computer-readable storage medium of claim 8, wherein the identified attributes include at least one of a name of the item, a price of the item, a description of the item, and a type of the item.

10. The non-transitory computer-readable storage medium of claim 8, wherein the operation further comprises:

11. The non-transitory computer-readable storage medium of claim 10, wherein mapping each item to the concepts of the interest taxonomy comprises:

12. The non-transitory computer-readable storage medium of claim 11, wherein the operation further comprises:

13. The non-transitory computer-readable storage medium of claim 11, wherein the first concept is determined based on a co-occurrence between one or more attributes in the item-attribute matrix.

14. The non-transitory computer-readable storage medium of claim 8, wherein each of the digital multimedia files is one of either an image or a video.

15. A system, comprising:

a processor; and

a memory storing one or more application programs configured to perform an operation for identifying one or more products to recommend to a plurality of users based on metadata of digital multimedia files, the operation comprising:

extracting a product feed, wherein the product feed lists one or more items,

identifying, for each item in the product feed, one or more attributes describing the item,

mapping each item to concepts of an interest taxonomy based on the identified one or more attributes for the item, wherein one or more users are associated with each concept in the interest taxonomy based on the metadata of the digital multimedia files, and

associating each item to one or more of the users based on the mapping.

16. The system of claim 15, wherein the identified attributes include at least one of a name of the item, a price of the item, a description of the item, and a type of the item.

17. The system of claim 15, wherein the operation further comprises:

18. The system of claim 17, wherein mapping each item to the concepts of the interest taxonomy comprises:

19. The system of claim 18, wherein the operation further comprises:

20. The system of claim 18, wherein the first concept is determined based on a co-occurrence between one or more attributes in the item-attribute matrix.

21. The system of claim 15, wherein each of the digital multimedia files is one of either an image or a video.