KR102244748B1 - Systems and methods for image classification by correlating contextual cues with images - Google Patents

Abstract

A sample set of images is received. Each image in the sample set may be associated with one or more social cues. The correlation of each image in the sample set based on one or more social cues associated with the image is scored as an image class. Based on the scoring, a training set of images training the classifier is determined from the sample set. In one embodiment, the degree to which an evaluation set of images is correlated with an image class is determined. Determining may include ranking a subset of the highest score among the evaluation set of images.

Description

System and method for classifying images by correlating context queues with images {SYSTEMS AND METHODS FOR IMAGE CLASSIFICATION BY CORRELATING CONTEXTUAL CUES WITH IMAGES}

The technical field is about the field of social networks. More particularly, the technical field relates to image classification techniques in social networks.

Social networks can provide an online community rich in interactions and content that connects members with each other. Members of the social network can indicate how they relate to each other. For example, members of a social network may indicate that they are friends, family members, business colleagues, or followers of each other, and members may assign some other relationship to each other. Social networks can allow members to message each other or post messages to an online community.

Social networks can also allow members to share content with each other. For example, members may create or use one or more pages containing interactive feeds that can be viewed across multiple platforms. Pages may contain images, videos, and other content that a member wishes to share with certain members of the social network or generally wish to publish on the social network. Members can also share content with social networks in different ways. For example, in the case of images, members may publish images on an image board or make the images available for search by an online community.

It is included in the content of the present invention.

The system may include at least one processor and a memory storing instructions configured to instruct the processor to perform reception of a sample set of images, each image in the sample set associated with one or more social cues. . The correlation for the image class of each image in the sample set is scored based on one or more social cues associated with the image. Based on the score, a training set of images to train the classifier may be determined from the sample set.

In some embodiments, an image class may be specified. In some embodiments, the determination may include ranking each image from the set of samples based on a score. Determining may include selecting a subset of the highest score among the sample set of images. The subset of the highest score may be the training set of images.

In various embodiments, the classifier may be trained based on a training set of images. A visual pattern template related to the image class may be created. The classifier may be configured to use a bag of visual words image classification technique or a neural network image classification technique, in some embodiments.

In some embodiments, the degree to which an evaluation set of images correlates with an image class may be determined. The evaluation set of images may be different from the sample set of images. The evaluation set of images may comprise a set of images that are larger than the sample set of images.

In various embodiments, a correlation of each image of a rating set of images to a visual pattern template associated with an image class may be scored. Each image in the rating set may be ranked based on the scored correlation of each image in the rating set of images. A subset of the highest score among the evaluation set of images may be associated with the image class.

In some embodiments, one or more social cues may include one or more image tags. The number of instances of a specific image tag may be determined from the total number of one or more image tags related to an image.

In some embodiments, the one or more social cues may include: location data associated with an image of a sample set of images; Alternatively, one or more of the image uploader, tagger, or the identity of the owner may be included in the sample set of images. In various embodiments, one or more social cues may be received by a social networking system.

The computer-implemented method may include receiving a sample set of images by a computer system, wherein each image in the sample set is associated with one or more social cues. The method may include scoring, by a computer system, a correlation of an image class with each image in the sample set based on one or more social cues associated with the image. The method may also include determining, by the computer system, a training set of images that train the classifier based on the score, from the sample set.

The computer storage medium stores computer-executable instructions that, when executed, cause a computer system to perform a computer-implemented method comprising receiving a sample set of images, wherein each image in the sample set includes one or more social cues and It is related. The method may include scoring a correlation between the image class and each image in the sample set based on one or more social cues associated with the image. The method may also include determining from the sample set a training set of images that train the classifier based on the score.

Other features and embodiments will become apparent from the accompanying drawings and the detailed description below.

It is included in the content of the present invention.

1 shows an example of a contextual image classification system, according to some embodiments.
2A shows an example of an image classification module, according to some embodiments.
2B shows an example of an image classification module, according to some embodiments.
3 shows an example of an image classification training module, according to some embodiments.
4 illustrates an example of an image classification evaluation module, according to some embodiments.
5 shows an example of a classifier, according to some embodiments.
6 shows an example of a process for classifying an image, according to some embodiments.
7 shows an example of a process for training a classifier, in accordance with some embodiments.
8 shows an example of a process for classifying an image, according to some embodiments.
9 shows an example of a preview of contextually generated image filters applied to a group of images, according to some embodiments.
10 shows an example of a preview of a contextually generated image filter applied to a group of images, according to some embodiments.
11 shows an example of a network diagram of a context image classification system in a social networking system, in accordance with some embodiments.
12 shows an example of a computer system that may be used to implement one or more embodiments described herein, in accordance with some embodiments.
The drawings show various embodiments of the present invention for illustrative purposes only, and the drawings use the same reference numerals to identify the same elements. One of ordinary skill in the art will readily appreciate from the following description that alternative embodiments of structures and methods shown in the figures may be used without departing from the principles of the invention described herein.

Image classification by correlating context queues

The social networking system can provide users with the ability to create content and share it with friends. Users of the social networking system's photo-sharing service can enjoy capturing images (e.g., still images, memes), videos, or interactive content from their mobile phones and sharing the content with their online friends. . Similarly, users can enjoy sharing content with their friends, for example by updating an interactive feed on their home page.

The social networking system may also provide or support the ability to display, identify, categorize, label, describe, or provide information about an item of content or an attribute about the content. One way to display this information is through tags that can identify or relate to the object of the content or its attributes. Another way to display this information is through the location of the upload, or through global positioning system (GPS) coordinates of the content that the user is uploading to identify where the content was captured. As will be described in more detail herein, there are many different methods for presenting information about content in a social networking system. Many of these indicators, including tags (eg, hashtags or other metadata tags) and GPS system coordinates, are not visual and are not based on automated analysis of visual data in the content.

In certain circumstances, non-visual indicators may be subjective or potentially misleading. For example, tags selected by the content creator to apply to his or her own content may describe the object of the content from the content creator's point of view, but the tags may be misrepresented or even irrelevant from the perspective of others. . For example, a user who posts a picture of himself dressed as a catwoman on Halloween can tag the picture as "#cat" even though the picture does not include a house cat at home. A user posting a photo of a dog named "Paris" can tag the photo with the tag "#paris" even though the photo does not depict Paris, France. A user posting images of his family captured in the Super Bowl in New Orleans on Super Bowl Sunday may have GPS coordinates and/or timestamps indicating that the images were captured in the Super Bowl, but the content of the images themselves is football. It may not be related to the game.

The subjectivity of non-visual indicators helps users of social networking systems to creatively express and share a wide variety of content, but the subjectivity of non-visual indicators often makes it difficult to search for user-uploaded images such as photographs. For example, an attempt to search for images posted on a social networking system for a "cat" may reveal an image of a user dressed as a catwoman on Halloween. An attempt to conduct a graphical search for an image of the Eiffel Tower in Paris, France, could lead to an image of a dog named "Paris". Attempts to search for Super Bowl photos may reveal personal photos of a fan's family that may not be highly relevant to anyone looking for a primary description of a football match. To some extent, non-visual indicators associated with images in these examples become "noisy" in that they may not accurately reflect the content of the associated image. It may be desirable to accurately search for user-uploaded content in a social networking system.

1 shows an example of a context image classification system 102, in accordance with some embodiments. The context image classification system 102 may be integrated into a social networking system, an example of which is provided in FIG. 11. In the example of FIG. 1, the context image classification system 102 may include an image classification module 104 and an image application module 106.

The image classification module 104 may recognize an object in the content based on the content and a context queue related to the visual attribute of the content. Content may include, for example, images, memes, videos, interactive audio-visual materials, and the like. The visual attribute may include an image reflecting the characteristic attribute of the object depicted in the content or a visual pattern in the image segment. The visual attributes may be based on, for example, one or a combination of appearance, color, shape, layout, and the like.

The context cue may include a visual indicator of an object depicted in the content. The context queue may reflect or suggest at least some objects of the content. In some embodiments, the context queue may include a content tag. The context queue may also include other types of non-visual indicators of objects in the content, such as a social queue. For example, without limitation, the context cue may include: GPS coordinates of an image or a user's digital device, the number of tags other than a specified tag, the degree to which a specified tag occurs in a series of tags, and the order of tags specified in a series of tags. , The identity of the content's tagger (e.g., an entity that associates a character string with the content), the identity of the content's uploader (e.g., an entity providing content for storage in a data store of a social networking system), of the content The identity of the owner, the time of uploading the content, the connection of the tagger (or uploader or owner) and the type of connection (e.g., friend), the status or profile of the tagger (or uploader or owner), metadata related to the content, given type It can include the identity of people you like or browse the contents of, EXIF (Exchangeable Image File) information, etc.

In some embodiments, the image classification module 104 may train the classifier to recognize a visual attribute of an image class based on a context cue collected from a sample set of images. The sample set of images may comprise a group of images from which a training set for training the classifier is selected. A sample set of images may contain a sufficiently large number of images to ensure accurate results by the classifier. The classifier may allocate a statistical score corresponding to the degree to which the content belongs to a specific image class to each item of the content. In some embodiments, the classifier may incorporate a hierarchical classifier, a linear classifier, or other classifier. An example of a classifier is provided in FIG. 5. In some embodiments, the classifier may be initially trained based on a selected subset of images managed by the social networking system. Classifiers can be maintained in a variety of situations. For example, the classifier may be held periodically at a selected frequency, or it may be held non-periodically as the image becomes available to the classifier. As another example, the classifier may be maintained when certain events occur, such as events with a high probability of generating a large number of images to be uploaded to the social networking system (eg, Super Bowl). As another example, the classifier may be maintained when the social networking system receives a threshold number of new images. Keeping in these and other situations can further refine the ability of the classifier to recognize the visual properties of the image classes.

Image classes include, for example, objects (e.g., cats, cars, people, wallets, etc.), objects associated with a brand or brand (e.g., Coca-Cola®, Ferrari®), professional sports teams (e.g., the Golden State Warriors®), Location (eg, Mount Everest), activity (eg, swimming), phrases or concepts (eg, red dress, happiness), and any other things, actions or ideas that may be associated with the content. While many examples provided herein mean a single "image class", an image class may mean a plurality of image classes or one or more image classes including a combination of an object, brand, professional sports team, location, etc. I can.

In some embodiments, the image classification module 104 compares the visual attribute of the evaluation set of images with the visual attribute of the image class, and whether the visual attribute in the evaluation set of images can be sufficiently correlated with the visual attribute of the image class. We can use the trained classifier to determine. The evaluation set of images may include a group of images selected for classification by a classifier. In various embodiments, a rating set of images may include all or part of the images in a data store, or all or part of the images in a social networking system. In one embodiment, the classifier may be trained by any suitable technique, such as machine learning.

In various embodiments, the image classification module 104 may provide content classified by the image application module 106. The classified content may include content ranked and/or scored by a classifier. On the other hand, raw content or unclassified content may include content that is not ranked and/or scored by a classifier or is not related to one or more image classes. The classified content may have a score indicating the degree to which the classified content matches an image class. Items with higher scores of classified content may have higher degrees of correlation with the visual properties of the image class. As a result, in various embodiments, the image classification module 104 may effectively search for classified content based on a score.

Using a classifier during analysis of an evaluation set of images for classification can occur at various times. For example, the classifier may analyze the evaluation set of images at a selected frequency, or non-periodically as the images become available as a classifier. The classifier may also analyze a rating set of images when a specific event occurs, such as events with a high probability of generating a large number of images that are uploaded to the social networking system. The classifier may analyze a rating set of images when the social networking system receives a threshold number of new images. As another example, the classifier may analyze an evaluation set of images for classification prior to performing an expected image search.

The image application module 106 may adapt the classified content for use in a social networking system. In some embodiments, the image application module 106 may interface with search application programming interfaces (APIs) to make each item of classified content searchable according to its image class. For example, the image application module 106 may interface with a search module that searches for classified images uploaded by a user to a social networking system. As another example, the image application module 106 may interface with a search module that retrieves a feed of a social networking system for classified images or memes that users have posted in their feed. The image application module 106 may also provide classified content in response to a search query. In some embodiments, the image application module 106 may extract topics related to the classified images provided by the image classification module 104 using a subject dictionary, category tree, and topic tag technology. I can.

2A shows an example of an image classification module 104, in accordance with some embodiments. The image classification module 104 may include an unclassified image datastore 202, an image classification training module 204, and a classifier 208. In addition to the components shown in FIG. 2A, the image classification module 104 may also include the components shown in FIG. 2B. It should be noted that the same components in FIGS. 2A and 2B may have the same reference numerals.

The unclassified image datastore 202 may be connected to the image classification training module 204 and the image classification evaluation module 206. The unclassified image datastore 202 may include unclassified images. Uncategorized images can have context cues associated with them. A datastore may contain an organization of any data including tables, comma-separated values (CSV) files, traditional databases (eg, SQL), or other known or conventional organizational formats. In some embodiments, unclassified datastore 202 may also store a set of context cues associated with an image, such as a tag or other cues. In some embodiments, the unclassified image datastore 202 may represent some or all of the unclassified images in a social networking system.

The image classification training module 204 may be connected to the unclassified image datastore 202 and the classifier 208. In some embodiments, the image classification training module 204 may implement a training phase. The training step may include the step of the image classification module 104 where the image classification training module 204 trains the classifier 208 to recognize the visual properties of images selected from a sample set of images. During the training phase, the image classification training module 204 may obtain a sample set of images from the unclassified image datastore 202. The image classification training module 204 may also collect a set of context cues associated with each obtained unclassified image. Based on the set of context cues, a training set of images selected from a sample set of images may be used to train the classifier 208 to recognize a visual pattern. Unclassified images and/or context cues may be obtained by querying the unclassified image datastore 202 for relevant information to determine a training set of images from a sample set of images.

During the training phase, the image classification training module 204 may be configured to specify one or more image classes for which the classifier 208 is to be trained. To specify the image class, the image classification training module 204 may receive an automated input defining the image class. The specification of the image class may also involve manual input from a person, such as an administrator who is assigned the task of classifying images as well.

During the training phase, the image classification training module 204 may be configured to identify and select context cues corresponding to the image class. In various embodiments, the image classification training module 204 may evaluate properties of a particular image class, and may determine if certain contextual cues were possibly associated with that image class. For example, the image classification training module 204 says that one type of tag has a high probability of carrying a picture of a house cat at home, while another type of tag has a high probability of carrying a picture of a user who is a catwoman's outfit on Halloween. You can decide. In this case, the image classification training module 204 may select a type of tags that are likely to carry a picture of a domestic cat corresponding to the cat's image class. As discussed in more detail herein, the determination of whether to apply context cues to a specific image class is, for example, a tag (eg, tag "#cat", tag "#Halloween", etc.), the order of tags, and a specific tag. Are carried by other specific tags (eg, whether the tag “#cat” is carried by the tag “#animal” or whether the tag “#cat” is carried by the tag “#Halloween”), etc. Can be based. The image classification training module 204 may also be configured to rank and/or score the degree to which context cues associated with a particular image correspond to a particular image class.

Context cues are analyzed to identify a training set of images from a sample set of images. The training set of images represents the images that correlate most closely with the image class. During the training phase, the image classification training module 204 may provide a training set of images to the classifier 208 to identify visual attributes associated with the image class. In some embodiments, the image classification training module 204 may instruct the classifier 208 to generate a template of a visual pattern corresponding to a specific image class. In some embodiments, the image classification training module 204 may store classified images and/or associated visual pattern templates, if desired, in a location such as the classified image datastore 210. The image classification training module 204 may also use a manual annotator to help select a training set of images. The image classification training module 204 is further discussed in the context of FIGS. 3 and 7.

The classifier 208 may be connected to the image classification training module 204 and may be connected to a classified image datastore (eg, the classified image datastore 210 shown in FIG. 2B ). The classifier may receive images from the image classification training module 204. In the training phase, the classifier 208 may evaluate the training set of images for the presence of a particular visual pattern. The classifier 208 may associate a specific visual pattern with an image class, generate a visual pattern template, and allow the visual pattern template to be stored. In the training phase, the classifier 208 may return the images used for training to the image classification training module 204. In some embodiments, the classifier 208 may include a connection back to the image classification training module 204 (eg, via a feedback loop). As a result, the classifier 208 may assist the accuracy of the image classification training module 204. These revolving connections can help improve classification and training in the future. The classifier 208 is further discussed in the context of FIG. 5.

2B shows an example of an image classification module 104, according to some embodiments. The image classification module 104 may include an unclassified image data store 212, an image classification evaluation module 206, a classifier 208, and a classified image data store 210. The unclassified image data store 212 may be connected to the image classification evaluation module 206. The unclassified image data store 212 may store unclassified images. The unclassified image data store 212 may be the same as the unclassified image data store 202 shown in FIG. 2A, but it is not necessary.

The image classification evaluation module 206 may be connected to the unclassified image data store 212 and may be connected to the classifier 208. In some embodiments, the image classification evaluation module 206 may implement an evaluation phase. The evaluating step may include the step of the image classification module 104 in which the image classification evaluation module 206 recognizes a visual pattern in the evaluation set of images using the classifier 208. In one embodiment, the evaluation set of images may be selected from the unclassified image datastore 212.

During the evaluation phase, the image classification evaluation module 206 may provide an evaluation set of images from the unclassified image datastore 212 to the classifier 208. In various embodiments, the image classification training module 204 may have trained, in the training phase, the classifier 208 to recognize the visual properties of the training set of images associated with the image class. The evaluation set of images may include a set of images that are different from the sample set of images and the training set of images. In some embodiments, the image classification evaluation module 206 may also provide an image class to the classifier 208 for the classifier 208 to compare to the evaluation set of images. In various embodiments, the image classification evaluation module 206 may instruct the classifier 208 to rank and/or score the evaluation set of images based on a correlation with the image class. The image classification evaluation module 206 may also store a ranking and score of images (ie, images scored based on a correlation with an image class) in the classified image data store 210. The image classification evaluation module 206 is further discussed in the context of FIGS. 4 and 8.

The classifier 208 may be connected to the image classification training module 204 and the image classification evaluation module 206. The classifier 208 may receive images from the image classification evaluation module 206. In the evaluation step, the classifier 208 may perform visual pattern recognition on the evaluation set of images to score a correlation between each image and the visual pattern template associated with the image class of interest. In the evaluation step, the classifier 208 may return the images used in the evaluation to the image classification evaluation module 206. The classifier 208 is further discussed in the context of FIG. 5.

The classified image datastore 210 may be connected to the classifier 208 and the image application module 106. The classified image data store 210 may store information including classified images, image classes, visual pattern templates, and other information. In some embodiments, the classified image datastore 210 may be indexed to facilitate efficient retrieval of classified images by APIs that find access to the classified image. For example, the classified image data store 210 may be configured to be compatible with a search module connected to the image application module 106 to access the classified images.

3 shows an example of an image classification training module 204, in accordance with some embodiments. The image classification training module 204 may include a training image selection module 301, a training image data store 309, and a classifier training module 310.

The training image selection module 301 may be connected to the unclassified image data store 202 and may be connected to the training image data store 309. The training unknown selection module 301 may identify a training set of images from a sample set of images. The training image selection module 301 may also store a training set of images in the training image datastore 309. The training image selection module 301 may include a training image collection module 302, a context queue extraction module 304, an image class specification module 306, and an image class correlation module 308.

The training image collection module 302 may be connected to other modules of the training image selection module 301. In some embodiments, the training image collection module 302 may collect a sample set of images along with a context queue associated with the sample set of images. A sample set of images and an associated context queue may be retrieved from the unclassified image datastore 202.

The context queue extraction module 304 may be connected to other modules of the training image selection module 301. The context queue extraction module 304 may be configured to extract context queues associated with a sample set of images. As discussed herein, the context cue may include a non-visual indicator of the context of an image. Examples of context cues for an image are the image tag for the image, the GPS coordinates of the device capturing the image, the tagger of the image, the uploader, and the identity of the owner, the identity of the owner of the image, and other information directly or indirectly related to the image And the like. In some embodiments, the context queue extraction module 304 may provide the image class correlation module 308 with a set of context queues to cause the image class correlation module 308 to correlate the context queue with the image class.

The image class specification module 306 may be connected to other modules of the training image selection module 301. The image class specification module 306 may be configured to specify an image class that the classifier 208 has trained to recognize. In some embodiments, the image class specifying module 306 may receive a command specifying an image class from an administrator that may be human or automated. In various embodiments, specifying an image class may involve creating an image class if the image class does not exist, or designating an image class if such an image class exists.

The image class correlation module 308 may be connected to other modules of the training image selection module 301. The image class correlation module 308 may receive a sample set of one or more images with an associated context queue from the context queue extraction module 304, and may receive a specified image class from the image class specific module 306. . The image class correlation module 308 may determine an extent to which context cues of a specific image are correlated with a specific image class. More specifically, the image class correlation module 308 may assign to each image a score or value indicating a probability that a given image is correlated with the image class. In some embodiments, the image class correlation module 308 may also rank each of the sample sets of images based on the score of each image. In some embodiments, the image class correlation module 308 may select from a sample set of images to provide a training set, such as the highest scored images of the sample set of images, to the classifier training module 310. Advantageously, the image class correlation module 308 does not need to perform any visual recognition of the content in the sample set of images.

The following description provides an example of how the image class correlation module 308 can determine the content of likely images based on the context queue associated with the images. As provided in the following description, the image class correlation module 308 may analyze the tag itself, the syntax of the tag, or other types of analysis in the context queue extracted by the context queue extraction module 304 You can do it. The image class correlation module 308 may also provide any combination of the following examples of correlating a sample set of images with a specified image class based on context cues to determine a training set of images.

In some embodiments, the image class correlation module 308 may analyze the syntax of an image tag of a sample of images. The image class correlation module 308 may determine how likely a particular syntax is correlated with a given image class. In some embodiments, syntax analysis of an image tag may involve assigning weights to the exact language of the image tag. That is, the image class correlation module 308 may determine that correct wording of the tag associated with the image indicates that the tag should be correlated with the image class. For example, the image may be tagged with the image tag "#domestic housecat". The image class correlation module 308 may determine that the tag "#domestic housecat" correlates to a high degree with the image class for the image of the domestic house cat. As another example, the image class correlation module 308 may determine that the tag "#domestic house market" correlates to a low degree with the image class for the domestic cat.

The image class correlation module 308 may also analyze the order of a particular image tag in a series of image tags associated with the image. For example, a person who wants to tag a picture of a home cat may include the following series of tags: "#cat, #athome, #Sunday, #animal". The image class correlation module 308 may identify that the tag "#cat" is a first tag in a series of tags, and thus, the image is likely to be an image of a domestic cat. The image class correlation module 308 may assign a weight in correlating an image to an image class to a series of tags. It should be noted that, to correlate an image with an image class, the image class correlation module 308 may also take into account tags at locations other than the first location in the series of tags.

The image class correlation module 308 may also analyze whether a plurality of image tags are synonymous with each other. For example, suppose the first image was tagged as: "#cat, #athome, #Halloween, #Catwoman, #costume, #DC Comics®". Also, assume that the second image is tagged as: "#cat, #mammal, #animal, #housecat, #tomcat, #feline, #cute". The image class correlation module 308 may determine that, based on some of the tags of the first image, the series of tags are not synonymous with each other, and that the first image is not likely to include an image of a domestic cat. . The image class correlation module 308 also allows a series of tags to be synonymous (e.g., "cat", "tomcat", and "feline"), based on a portion of the tags of the second image, and the second image It may be determined that the probability of including the image of the house cat is higher than that of the first image. Accordingly, the image class correlation module 308 may allocate a score higher than the score of the first image to the second image in relation to the image class of the domestic cat.

In some embodiments, the image class correlation module 308 may evaluate a plurality of image tags for antonym or non-presence of another meaning. For example, assume that the first image is tagged as: "#blackcar," "#whitecar," "#luxurycar," "#My Mercedes". Assume the second image was tagged as: "#blackcar," "#darkcar," "#luxurycar," "#My Mercedes". The image class correlation module 308 may determine that the second image correlates to a high degree with an image class corresponding to a black car, based on the fact that the second series of tags lack an antonym for “blackcar”. The image class correlation module 308 may allocate a score for the second image that is greater than the score for the first image in relation to the image class of the black car.

The image class correlation module 308 may, in some embodiments, evaluate the relationship of image tags based on an ontology or language hierarchy. For example, the image class correlation module 308 may develop an ontology of one or more words from an online source (eg, WordNet), and words with emotional influence (eg, “happy,” “sad,” red dress,” "black car") As another example, suppose the image is tagged as: "#cat, #mammal, #animal, #housecat, #tomcat, #feline, #cute". Image class correlation module 308 may determine that “cat” is part of the family of items identified by “mammal”, and eventually is part of the family of items identified by “animal.” As a result, the image class correlation module 308 In this example, the tag can be identified as providing a reliable indicator of the domestic house cat depicted in the image. The image class correlation module 308 then retrieves the images in relation to the image class of the household house cat. You can score accordingly.

In various embodiments, the image class correlation module 308 may perform other natural language analysis of tagged words and phrases. In some embodiments, the image class correlation module 308 may allow for misspelling of tag words. The image class correlation module 308 may also take into account non-English languages, and the presence of non-English words (eg, “#cat” and “#gato” in the same series of tags) in association with the English equivalent. ). In each of these examples, the image class correlation module 308 may appropriately score a set of images with respect to a particular image class.

In addition to analyzing the syntax of an image tag, the image class correlation module 308 may analyze social cues related to image generation and/or tagging. For example, the image class correlation module 308 may analyze positional data associated with a sample set of images. More specifically, the image class correlation module 308 may evaluate the GPS coordinates of the image. For example, GPS coordinates may be obtained from a GPS transceiver on the user's mobile device, or from geographical tagging of the image when or after the image is uploaded. Using the location data of a particular image, the image class correlation module 308 may assign a score related to the particular image class to the particular image. For example, the image class correlation module 308 may determine from location data that the image was taken near the Golden Gate Bridge in San Francisco. The image class correlation module 308 may then assign a score related to the image class of the bridge to the image.

In some embodiments, the image class correlation module 308 may analyze a social cue that includes the identity of a creator of a particular image among a sample set of images. The image class correlation module 308 may allocate a score according to whether certain entities are likely to generate images in a certain image class. For example, if an entity has generated many images from a particular image class in the past, the image class correlation module 308 may score a particular image of the entity to reflect a high degree of correlation with the image class.

In some embodiments, the image class correlation module 308 may analyze a social cue indicating whether the owner of one of the sample set of images was the creator of the image or the tagger of the image. In some embodiments, the image class correlation module 308 may determine whether the entity owns the image or owns intellectual property rights for the created or tagged image. For example, with respect to the image class of a trademark (e.g., Coca-Cola® logo), the image class correlation module 308 may be used if the image was created or tagged by an entity that owns the trademark (e.g., Coca-Cola®). , A score reflecting a high degree of correlation can be provided to the image. This weighting technique can prove to be particularly useful for recognizing trademarked images or logos.

In some embodiments, the image class correlation module 308 may analyze a social queue indicating a person's status or profile in a social networking system. A person may be the creator and/or tagger of a particular image of a sample set of images. For example, the image class correlation module 308 may include human activities (eg, past posts, status updates, friend relationships, message history, past tag history, past image creation history, history browsing, online profile, etc.), or relationships. Based on their friends (eg, friends), one can determine whether a person is likely to be a reliable image creator and/or tagger. The image class correlation module 308 may assign various scores to one or more images based on whether the person's status or profile indicates that the person is a reliable image creator and/or predecessor in general or particularly with respect to the image class. have.

In various embodiments, the image class correlation module 308 may analyze social cues indicating the presence or absence of other image classes. For example, the image class correlation module 308 may determine that the image includes a boat, and as a result, it is highly likely that the image is not indoors. The presence of context cues (eg, hashtag “#indoors”) may indicate that the image is unlikely to contain a boat. As another example, the image class correlation module 308 may determine that the existence of one object in the image may mean that another object may or may not exist in the image. For example, the presence of context cues representing cows in an image is that, given the fact that the image is unlikely to contain both a cow and an airplane, the image class correlation module 308 is negatively correlated with the airplane. You can try to provide a negative correlation. As another example, the existence of one context queue representing one chicken in the image, in light of the fact that the image containing one chicken is likely to contain other chickens, the image class correlation module 308 Can be made to provide positive correlations for other chickens. Likewise, the presence of a sheep can indicate a positive correlation for a sheepdog.

The image class correlation module 308 includes, but is not limited to, those discussed above, one or more: whether the uploader is the owner of the image, the file type of the image, the metadata of the image or related to the image, people who liked the image. Analyze various social cues including the identity of the user, the skill level or experience of users who viewed the tag or image, whether the image was previously classified in the image class (and, if so, what the score of the image associated with that image class), etc. can do. Social cues other than those explicitly discussed herein may be analyzed by the image class correlation module 308 to correlate and score images with the image class. The image class correlation module 308 may assign various scores to various images based on the degree to which the context queue of these images correlates to one or more image classes to a high degree.

In addition to assigning a score to an image, the image class correlation module 308 may also rank a sample set of images with respect to one or more image classes. For example, the image class correlation module 308 may rank each image in a sample set of images based on the score of each image associated with the image class. The ranking may reflect the relative correlation of each image to the image class. Image class correlation module 308 may use weights, scores, and/or rankings to generate a training set of images.

The training image datastore 309 may receive a training set of images from the training image selection module 301. The training image datastore 309 may store a training set of images. In some embodiments, the training image datastore 309 may allow the classification training module 310 to access a training set of images.

The classifier training module 310 may be connected to the training image datastore 309 and may be connected to the classifier 208. In various embodiments, the classifier training module 310 may provide a training set of images to train the classifier 208 to recognize an image class. The training set of images may be limited to a threshold number of topmost images of the sample set with respect to a particular image class. The threshold number of top-level images can be represented by the value K, where K is an arbitrary integer value. For example, assume that the training image collection module 302 ultimately collected 1000 images with a sample set of images to train the classifier 208 to recognize the image class for “domestic housecats”. In addition, it is assumed that the image class correlation module 308 has assigned scores higher than the scores assigned to 800 images remaining in 200 of these images. In this case, the classifier training module 310 may provide the classifier 208 with 200 images of the high score as a training set of images, so that the classifier 208 Enables efficient determination of visual properties.

4 shows an example of an image classification evaluation module 206, according to some embodiments. The image classification evaluation module 206 includes an evaluation image collection module 402, a classifier input module 404, a visual pattern template collection module 405, a classifier score receiving module 406, a visual attribute ranking module 408, and It may include an evaluated image providing module 410.

The evaluation image collection module 402 may be connected to other modules of the image classification evaluation module 206. In some embodiments, the evaluation image collection module 402 may collect an evaluation set of images for image classification from the unclassified image data store 202. The evaluation set of images may be the same or different from the sample set of images and the training set of images. The evaluation set of images may include a different number of images (eg, a greater number of images) than the sample set of images and the training set of images. In various embodiments, the evaluation set of images may be obtained randomly or selectively from unclassified image datastore 202.

The visual pattern template collection module 405 may be connected to other modules of the image classification evaluation module 206 and may be connected to the image classification training module 204. In some embodiments, the visual pattern template collection module 405 may receive a visual pattern template corresponding to a specific image class from the image classification training module 204. The visual pattern template collection module 405 may further provide a visual pattern template to the classifier input module 404.

The classifier input module 404 may be connected to other modules of the image classification evaluation module 206. In some embodiments, the classifier input module 404 may receive a rating set of images from the rating image collection module 402. The classifier input module 404 may also receive a visual pattern template corresponding to a specific image class from the visual pattern template collection module 405. The classifier input module 404 may instruct the classifier 208 to attempt to recognize the visual pattern template in each evaluation set of images.

The classifier score receiving module 406 may be connected to other modules of the image classification evaluation module 206. In some embodiments, the classifier score receiving module 406 may receive scores from the classifier 208 indicating a degree to which specific images of the evaluation set of images are correlated with the visual pattern template.

The visual attribute ranking module 408 may be connected to other modules of the image classification evaluation module 206. In various embodiments, the visual attribute ranking module 408 may rank the evaluation set of images based on the degree to which the score of each image in the evaluation set of images correlates with the visual pattern template. In some embodiments, the visual attribute ranking module 408 may provide a re-ranked set of images that make up a classified set of images for indexing or search.

The evaluated image providing module 410 may be connected to other modules of the image classification evaluation module 206. In some embodiments, the evaluated image providing module 410 may receive re-ranked or classified images from the visual attribute ranking module 408. The evaluated image providing module 410 stores the re-ranked image together with an index or other information reflecting the degree to which each re-ranked image is correlated with the specified image class in the classified image data store 210. Can provide.

5 shows an example of a classifier 208, in accordance with some embodiments. The classifier 208 may include a visual pattern generation module 502, a visual pattern recognition module 504, and a classified image interface module 506.

The visual pattern generation module 502 may be connected to the visual pattern recognition module 504. The visual pattern generation module 502 may receive a training set of images from the image classification training module 204, during the training phase of the classifier 208, and visualize features common to the training set of images associated with the image class. You can create a pattern template. To create a visual pattern template, the visual pattern generation module 502 can be used, such as a word bag model in a computer vision or technique that counts the occurrence of a vocabulary of local image features in each of the training sets of images. Visual pattern recognition algorithm can be implemented. In one embodiment, the visual pattern generation module 502 may break up the image into segments and evaluate each segment of the image for the presence of a visual feature. The visual pattern generation module 502 may also extract visual features identified in each segment of the image, and express the visual features as a vector. Using the vector, the visual pattern generation module 502 can generate a visual pattern template of features common to the training set of images.

In various embodiments, the visual pattern generation module 502 may generate a visual pattern template based on features most commonly found in a training set of images. For example, in these embodiments, the image classification training module 204 may provide an integer “K” images to the visual pattern generation module 502, and the visual pattern generation module 502 is the most Many commonly found features can be recognized.

The visual pattern recognition module 504 may be connected to the visual pattern generation module 502. The visual pattern recognition module 504 may receive an evaluation set of images from the image classification evaluation module 206 during the evaluation step, and determine the degree to which each of the evaluation sets of images is correlated with a specific visual pattern template associated with the image class Can be identified. In some embodiments, the visual pattern recognition module 504 may use visual pattern recognition, such as a word bag algorithm. Visual pattern recognition may include neural network image classification techniques, in some embodiments. The visual pattern recognition module 504 may determine various visual features within the evaluation set of images, and for each image, may express the extracted visual features as a set of vectors. The visual pattern recognition module 504 may also compare vectors for each of the evaluation sets of images to various visual pattern templates of features. In some embodiments, the visual pattern recognition module 504 may score the degree to which each of the evaluation set of images correlates with various visual pattern templates generated during the training phase.

The classified image interface module 506 may be connected to the visual pattern generation module 502 and the visual pattern recognition module 504. In some embodiments, the classified image interface module 506 may receive the classified images from the visual pattern recognition module 504 along with their scores. The classified image interface module 506 may provide classified images and/or scores to the classified image data store 210. The classified image interface module 506 may also provide a visual pattern template to the classified image data store 210.

6 shows an example of a process 600 for classifying an image, in accordance with some embodiments. The process 600 is discussed in conjunction with the image classification module 104 shown in FIG. 2A. Process 600 may include a training step 600a and an evaluation step 600b. At block 602 of process 600, image classification training module 204 may collect a sample set of images from unclassified image datastore 202. At block 604, the image classification training module 204 may collect a context queue associated with a sample set of images. At block 606, the image classification training module 204 scores and ranks the images based on their correlation to the image class of interest to generate the training set of images discussed herein. You can use them. Based on scoring and ranking, a training set of images can be determined. At block 607, the image classification training module 204 may train the classifier 208 based on the training set. At block 608, the image classification evaluation module 206 may collect a rating set of images from the unclassified image datastore 202. At block 610, classifier 208 may compare the visual attribute of the evaluation set of images to a set of visual pattern templates associated with the image class. At block 612, the classifier 208 may determine whether each image in the evaluation set of images belongs to an image class based on the comparison.

7 shows an example of a process 700 for training a classifier, in accordance with some embodiments. The process 700 is discussed in conjunction with the image classification training module 204 shown in FIG. 3. At block 701, the image class specification module 306 may specify an image class for training to be recognized by the classifier 208. At block 702, training image collection module 302 may receive a sample set of images, each of which has an associated context cue. At block 704, the context queue extraction module 304 may extract the context queue from the sample set of images. At block 708, the image class correlation module 308 may score a correlation between each image and the image class in the sample set of images based on one or more context cues associated with the image. At block 710, the image class correlation module 308 may rank a sample set of images based on the score of each image. At block 712, the image class correlation module 308 may determine a training set of images to train the classifier 208 from the sample set. In some embodiments, determining the training set may include ranking each image in the sample set based on scoring. The determination may also include selecting a subset of the highest score among the sample set of images. The subset of the top score may include a training set of images. At block 714, the classification training module 310 can train the classifier 208 to identify common visual patterns in the training set of images.

8 shows an example of a process 800 for classifying images, in accordance with some embodiments. The process 800 is discussed with the image classification evaluation module 206 shown in FIG. 4 and the classifier 208 shown in FIG. 5. At block 802, the evaluation image collection module 402 may collect a rating set of images from the unclassified image datastore 202. At block 804, the evaluation image collection module 402 may determine an image class for evaluating the visual attribute of the evaluation set of images. In some embodiments, the classifier input module 404 may provide an image class and a rating set of images to the classifier 208. At block 806, the visual pattern recognition module 504 may score a correlation of each image in the evaluation set of images to the visual pattern template associated with the image class. At block 808, the visual pattern recognition module 504 may rank each image in the evaluation set of images based on the scored correlation of each image in the evaluation set of images. At block 810, the visual pattern recognition module 504 may associate a subset of the highest score among the evaluation set of images with the image class. In some embodiments, the classified image interface module 506 may then provide a subset of the highest score to various other modules in the social networking system.

9 shows an example of a preview 900 of a contextually generated classification of a group of images by the image classification module 104, in accordance with some embodiments. The preview 900 includes an unclassified group 902 of images, a classified group 904 of first images, a classified group 906 of second images, a classified group 908 of third images, and a fourth image. Includes a classified group 910 of images. In the example of FIG. 9, the unclassified group 902 of images includes a group of images with an associated context queue. The context queue may contain image tags and other context information. To create a classified group 904 of first images, a classified group 906 of second images, a classified group 908 of third images, and a classified group 910 of fourth images, one The unclassified group 902 of the above images was provided to the image classification training module 204. During the training phase, the image classification training module 204 includes four image classes, namely: a first image class of images with a synthesized look/feel, a second image class of close-up images, and outdoor. In order to train the classifier 208 to recognize the visual properties associated with the third image class of images taken at and the fourth image class of images depicting water, the unclassified group 902 of images classified based on the context cue. Used sets of. The image classification evaluation module 206 provides an unclassified group 902 of images to a classifier 208 trained to compare the visual properties of the unclassified group 902 of images with visual pattern templates associated with the four image classes during the evaluation phase. ) Was provided. The output of the classifier 208 corresponded to the four image classes that the classifier 208 was trained to recognize. More specifically, the classifier 208 includes a classified group 904 of first images corresponding to the first image class; A classified group 906 of second images corresponding to a second image class; A classified group 908 of third images corresponding to the third image class; And a classified group 910 of fourth images corresponding to the fourth image class.

10 shows an example of a preview 1000 of a contextually generated image filter classification of a group of images by the image classification module 104, according to some embodiments. The preview 1000 includes an unclassified group 1002 of images and a classified group 1004 of images. The unclassified group 1002 of images may represent some of the unclassified images in a social networking system. A sample set of images was associated with tags and other context cues. Based on its context cues, each image in the sample set of images was scored based on correlation with the image class of interest. In this example, "cat" is the image class of interest. Images of the sample set of images were then ranked based on their score. In this example, the top 200 images of the sample set of images have been designated as the training set of images. The training set of images was then applied to train the classifier to recognize a common visual pattern depicted in the images. The visual pattern template was created based on the training set and was associated with the "cat" image class. The unclassified group 1002 of images was used as an evaluation set of images. A rating set of images was applied to a classifier to score their correlation with the visual pattern template associated with the "cat" image class. The scores of the rating set of images were ranked, and the topmost images in the rating set of images were selected as a classified group 1004 of images.

Social Networking System-Example Implementation

11 is a network diagram of an exemplary social networking system 1100 that implements a context image classification system 102, in accordance with some embodiments. The social networking system 1100 includes one or more user devices 1110, one or more external systems 1120, a social networking system 1130, and a network 1150. In one embodiment, the social networking system discussed in connection with the above-described embodiments may be implemented as the social networking system 1130. For illustrative purposes, the embodiment of the social networking system 1100 shown in FIG. 11 includes a single external system 1120 and a single user device 1110. However, in other embodiments, the social networking system 1100 may include more user devices 1110 and/or more external systems 1120. In a specific embodiment, the social networking system 1130 is operated by a social networking system provider, while the external system 1120 is separate from the social networking system 1130 in that it can be operated by different entities. . However, in various embodiments, social networking system 1130 and external system 1120 operate together to provide social networking services to users (or members) of social networking system 1130. In this respect, social networking system 1130 provides a platform or backbone through which other systems, such as external systems 1120, can be used to provide social networking services and functions to users over the Internet.

User device 1110 includes one or more computing devices capable of receiving input from a user and transmitting and receiving data through network 1150. In one embodiment, the user device 1110 is a conventional computer system running, for example, a Microsoft Windows compatible operating system (OS), Apple OS X, and/or a Linux distribution. In another embodiment, the user device 1110 may be a device having a computer function such as a smart phone, a tablet, a personal digital assistant (PDA), a mobile phone, or the like. User device 1110 is configured to communicate over network 1150. User device 1110 may run an application such as a browser application that allows a user of user device 1110 to interact with social networking system 1130. In another embodiment, the user device 1110 interacts with the social networking system 1130 through an application programming interface (API) provided by the native operating system of the user device 1110 such as iOS and ANDROID. User device 1110 may include external systems 1120 and social networking systems 1130 via network 1150, which may include any combination of local area and/or wide area networks using wired and/or wireless communication systems. ) To communicate with.

In one embodiment, network 1150 uses standard communication technologies and protocols. Accordingly, the network 1150 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 3G, 4G, CDMA, GSM, LTE, and digital subscriber line (DSL). Similarly, networking protocols used in the network 1150 include multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), User Datagram Protocol (UDP), hypertext transport protocol (HTTP), and simple mail (SMTP). transfer protocol), FTP (file transfer protocol), and the like. Data exchanged on the network 1150 may be expressed using a technology and/or format including hypertext markup language (HTML) and extensible markup language (XML). Additionally, all or some of the links may be encrypted using conventional encryption techniques such as secure sockets layer (SSL), transport layer security (TLS), and Internet Protocol security (IPsec).

In one embodiment, the user device 1110 uses the browser application 1112 to process the markup language document 1114 received from the external system 1120 and the social networking system 1130 to allow the external system 1120 to And/or display content from the social networking system 1130. The markup language document 1114 identifies the content and one or more instructions describing the format or presentation of the content. By executing the instructions contained in markup language document 1114, browser application 1112 displays content identified using the format or presentation described by markup language document 1114. For example, markup language document 1114 includes instructions for creating and displaying a web page having multiple frames containing text and/or image data retrieved from external systems 1120 and social networking system 1130. do. In various embodiments, the markup language document 1114 includes a data file including extensible markup language (XML) data, extensible hypertext markup language (XHTML) data, or other markup language data. Additionally, the markup language document 1114 facilitates the exchange of JavaScript Object Notation (JSON) data, JSON (JSONP) with padding, and data between the external system 1120 and the user device 1110. You can include JavaScript data that you want. The browser application 1112 on the user device 1110 may use a JavaScript compiler to decode the markup language document 1114.

The markup language document 1114 may also include or link to an application or application framework such as FLASH™ or Unity™ applications, SilverLight™ application framework, or the like.

In one embodiment, the user device 1110 also includes one or more cookies 1116 containing data indicating whether a user of the user device 1110 is logged into the social networking system 1130, which It may enable modification of data communicated from the networking system 1130 to the user device 1110.

External system 1120 includes one or more web servers including one or more web pages 1122a and 1122b that are communicated to user device 1110 using network 1150. External system 1120 is separate from social networking system 1130. For example, external system 1120 is associated with a first domain, while social networking system 1130 is associated with a separate social networking domain. Web pages 1122a and 1122b included in external system 1120 include a markup language document 1114 that identifies content and includes instructions that specify the format or presentation of the identified content.

Social networking system 1130 includes one or more computing devices that provide users of the social networking system the ability to communicate and interact with other users of the social networking system for a social networking system including a plurality of users. In some examples, a social networking system may be represented as a graph, that is, a data structure including edges and nodes. Other data structures, including but not limited to databases, objects, classes, meta elements, files, or any other data structure, may also be used to represent social networking systems. The social networking system 1130 may be managed, maintained, or controlled by an operator. The operator of the social networking system 1130 may be a person, an automated application, or a set of applications for managing content, regulating policies, and collecting usage metrics within the social networking system 1130. Any type of operator can be used.

Users may subscribe to social networking system 1130 and then add connections to any number of other users of social networking system 1130 to which they wish to connect. The term “friend” as used herein refers to any other user of the social networking system 1130 to which the user has connected, associated, or established a relationship through the social networking system 1130. For example, in an embodiment, if users in the social networking system 1130 are represented as nodes in a social graph, the term “friend” may mean an edge formed between directly connecting two user nodes.

The connection may be explicitly added by the user or may be automatically created by the social networking system 1130 based on common characteristics of the user (eg, users who are graduates of the same educational institution). For example, the first user specifically selects a specific other user as a friend. The connection in social networking system 1130 is usually two-way, but need not, so the terms “user” and “friend” are subject to a frame of reference. The connections between users of the social networking system 1130 are usually two-way ("two-way"), or "interactive," but the connections may also be one-way, or "one-way." For example, if Bob and Joe are both users of the social networking system 1130 and are connected to each other, then Boe and Joe are their connections. On the other hand, if Bob wishes to connect to Joe to view the data communicated by Joe to the social networking system 1130, but Joe does not wish to establish an interconnection, a one-way connection may be established. Connections between users may be direct connections, but some embodiments of social networking system 1130 allow indirect connections through one or more connection levels or degrees of separation.

In addition to establishing and managing connections between users and allowing interaction between users, social networking system 1130 provides users with information on various types of items supported by social networking system 1130. It provides the ability to perform an action. These items include groups or networks to which users of social networking system 1130 may belong (i.e., social networks of people, entities, and concepts), events or calendar entries that the user may be interested in, and users. Computer-based applications available through the social networking system 1130, allowing a user to buy or sell items provided by the social networking system 1130 or through services through the social networking system 1130. Transactions may include interactions with advertisements that a user may perform within or outside the social networking system 1130. These are just a few examples of items a user can act on for the social networking system 1130, and many other examples are possible. A user may interact with anything that may be represented in the social-networking system 1130 or in an external system 1120 that is separate from the social-networking system 1130 or connected to the social-networking system 1130 through the network 1150. I can.

The social networking system 1130 may also link various entities. For example, social networking system 1130 allows users to interact with each other as well as interact with external systems 1120 or other entities via APIs, web services, or other communication channels. The social networking system 1130 creates and manages a “social graph” comprising a plurality of nodes interconnected by a plurality of edges. Each node in the social graph may act on another node and/or may represent an entity on which the other node may act. The social graph can include various types of nodes. Examples of types of nodes include users, non-human entities, content items, web pages, groups, activities, messages, concepts, and any other that may be represented by an object in the social networking system 1130. . An edge between two nodes in a social graph may represent a certain type of connection or association between the two nodes, and an edge may arise from a node connection or from an action one of the nodes performs on another node. In some cases, the edges between nodes may be weighted. The weight of the edge may represent an attribute associated with the edge, such as the strength of a connection or association between nodes. Different weights may be provided for different types of edges. For example, one weight may be given to an edge created when one user “likes” another user, while a different weight may be given to an edge created when a user makes another user a friend.

As an example, when a first user identifies a second user as a friend, an edge in the social graph is created to connect a node representing the first user and a second node representing the second user. As the various nodes are associated and interact with each other, the social networking system 1130 modifies the edges connecting the various nodes to reflect the relationships and interactions.

Social networking system 1130 also includes user-generated content that enhances user interaction with social networking system 1130. User-generated content may include anything that a user can add, upload, transmit, or “post” to social networking system 1130. For example, a user communicates a post from user device 1110 to social networking system 1130. Posts may include status updates or other textual data, location information, images such as photos, videos, links, music, or other similar data and/or data such as media. Content may also be added to the social networking system 1130 by a third party. Content “items” may be represented as objects in the social networking system 1130. In this manner, users of the social networking system 1130 are encouraged to communicate with each other by posting various types of text and content items of media over various communication channels. Such communication increases the interaction of users with each other and increases the frequency with which users interact with the social networking system 1130.

The social networking system 1130 includes a web server 1132, an API request server 1134, a user profile store 1136, a connection store 1138, an action logger 1140, an activity log 1142, an authentication server 1144. , An image classification module 1146, and an image application module 1148. In one embodiment, social networking system 1130 may include additional, fewer, or different components for various applications. Other components such as network interfaces, security mechanisms, load balancers, failover servers, management and network operation consoles, etc. are not shown in order not to obscure the details of the system.

The user profile store 1136 contains personal, demographic, and other types of technical information such as career, education, hobbies or preferences, location, etc. declared by the user or inferred by the social networking system 1130. Manage information about your account. This information is stored in the user profile store 1136 so that each user is uniquely identified. The social networking system 1130 also stores data describing one or more connections between different users in the connection store 1138. The connection information may indicate users with similar or common careers, group memberships, hobbies, or educational backgrounds. Additionally, social networking system 1130 includes user-defined connections between different users, allowing users to specify their relationships with other users. For example, user-defined connections allow the user to create real-life relationships of the user, such as relationships with other users that rival friends, colleagues, partners, and the like. Users can choose from predefined types of connections or define their own connection types as needed. Connections to other nodes in the social networking system 1130, such as non-human entities, buckets, cluster centers, images, interests, pages, external systems, concepts, etc. are also stored in the connection store 1138.

The social networking system 1130 manages data about objects with which a user can interact. To manage this data, user profile store 1136 and connection store 1138 store instances of corresponding types of objects managed by social networking system 1130. Each object type has an information field suitable for storing information appropriate to the type of object. For example, user profile store 1136 includes a data structure with fields suitable for describing the user's account and information related to the user's account. When a new object of a particular type is created, the social networking system 1130 initializes a new data structure of the corresponding type, assigns a unique object identifier to it, and adds data to the object as necessary. This, for example, allows the user to become a user of the social networking system 1130, the social networking system 1130 creates a new instance of the user profile in the user profile store 1136, assigns a unique identifier to the user account, and This can occur when starting to fill in the fields of the user account with the information provided by.

The connection store 1138 contains a data structure suitable for describing a user's connection to other users, a connection to an external system 1120, or a connection to other entities. The connection store 1138 can also associate a connection type with the user's connection, and the connection type can be used in conjunction with the user's privacy settings to regulate access to information about the user. In one embodiment, the user profile store 1136 and the connection store 1138 may be implemented as a federated database.

The data stored in the connection store 1138, the user profile store 1136, and the activity log 1142 identifies the edges that connect the various objects and nodes for the social networking system 1130 to identify relationships between different objects. Lets you create a social graph that uses nodes to do so. For example, if a first user establishes a connection with a second user in the social networking system 1130, the user accounts of the first user and the second user from the user profile store 1136 will serve as nodes in the social graph. I can. The connection between the first user and the second user stored by the connection store 1138 is the edge between the nodes associated with the first user and the second user. Continuing with this example, the second user may then transmit a message to the first user within the social networking system 1130. The act of sending this message, which can be stored, is another edge between the two nodes in the social graph representing the first user and the second user. Additionally, the message itself may be identified and included in the social graph as another node connected to nodes representing the first user and the second user.

As another example, the first user may tag the second user in an image managed by the social networking system 1130 (or alternatively, an image managed by another system outside the social networking system 1130 ). The image itself may be represented as a node in social networking system 1130. This tagging action may not only create an edge between the first user and the second user, but also create an edge between the images and respective users who become nodes in the social graph. As another example, if the user confirms attending an event, the user and the event are nodes obtained from the user profile store 1136, where the attendance of the event is the edge between the nodes that can be retrieved from the activity log 1142 . By creating and managing social graphs, the social networking system 1130 contains data describing different types of objects and interactions and the connections between those objects, and provides a rich source of socially relevant information. to provide.

Web server 1132 links social networking system 1130 to one or more user devices 1110 and/or one or more external systems 1120 via network 1150. The web server 1132 serves web pages as well as other web-associated content such as Java, JavaScript, Flash, XML, and the like. The web server 1132 may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 1130 and one or more user devices 1110. The messages may be instant messages, queued messages (eg, email), text and SMS messages, or any other suitable message format.

The API request server 1134 causes one or more external systems 1120 and user devices 1110 to call access information from the social networking system 1130 by calling one or more API functions. The API request server 1134 may also cause the external system 1120 to send information to the social networking system 1130 by calling APIs. In one embodiment, the external system 1120 transmits an API request to the social networking system 1130 through the network 1150, and the API request server 1134 receives the API request. The API request server 1134 processes the request by calling an API associated with the API request to generate an appropriate response for the API request server 1134 to communicate to the external system 1120 via the network 1150. For example, in response to an API request, the API request server 1134 collects data associated with the user, such as a user's connection logged to the external system 1120, and communicates the collected data to the external system 1120. In another embodiment, the user device 1110 communicates with the social networking system 1130 through APIs in the same manner as the external system 1120.

The behavior logger 1140 may receive communications from the web server 1132 regarding user behavior in and/or outside the social networking system 1130. The behavior logger 1140 populates the activity log 1142 with information about user behavior so that the social networking system 1130 can perform various actions performed by its users within the social networking system 1130 and outside the social networking system 1130. Make the action discoverable. Any action a particular user has done with respect to other nodes on social networking system 1130 may be associated with each user's account through activity log 1142 or information managed in a similar database or other data store. Examples of actions performed by a user within the identified and stored social networking system 1130 include, for example, adding a connection to another user, sending a message to another user, reading a message from another user, and viewing content associated with another user. It may include viewing, attending an event posted by another user, posting an image, attempting to post an image, or other actions of interacting with another user or other object. When a user performs an action within the social networking system 1130, the action is recorded in an activity log 1142. In one embodiment, social networking system 1130 manages activity log 1142 as a database of entries. When an action is performed within the social networking system 1130, an entry for the action is added to the activity log 1142. The activity log 1142 may be referred to as an activity log.

Additionally, user actions may be associated with concepts and actions occurring within entries external to the social networking system 1130, such as an external system 1120 separate from the social networking system 1130. For example, the behavior logger 1140 may receive data describing a user interaction with the external system 1120 from the web server 1132. In this example, external system 1120 reports user interactions according to structured actions and objects in the social graph.

Other examples of actions in which the user interacts with the external system 1120 include the user expressing interest in the external system 1120 or other entity, the user's Posting a comment discussing page 1122a to the social networking system 1130, the user posting a Uniform Resource Locator (URL) or other identifier associated with the external system 1120 to the social networking system 1130, The user attends an event associated with the external system 1120, or any other action by the user associated with the external system 1120. Accordingly, the activity log 1142 may include actions that describe interactions between a user of the social networking system 1130 and an external system 1120 that is separate from the social networking system 1130.

The authentication server 1144 forces one or more personal information settings of users of the social networking system 1130. The user's privacy settings determine how certain information associated with the user can be shared. The personal information setting includes a specification of specific information associated with a user and a specification of entities or entities to which the information can be shared. Examples of entities to which information may be shared may include other users, applications, external systems 1120, or any entity that may potentially have access to the information. Information that can be shared by a user includes user account information, such as profile picture, phone number associated with the user, connection of the user, actions taken by the user, such as adding a connection, changing the user's profile information, etc. do.

Personal information setting specifications can be provided at different levels of granularity. For example, privacy settings can identify specific information to be shared with other users; The privacy settings identify a work phone number or a specific set of related information, such as a profile picture, home phone number, and status. Alternatively, privacy settings can be applied to all information associated with the user. The specification of the set of entities that can access certain information can also be specified at various levels of granularity. The various sets of entities to which information may be shared may include, for example, all friends of the user, all friends of friends, all applications, or all external systems 1120. One embodiment allows a specification of a set of entities to include a list of entities. For example, a user may provide a list of external systems 1120 that are allowed to access certain information. Another embodiment allows the specification to include a set of entities with exceptions that are not allowed to access information. For example, the user may allow all external systems 1120 to access the user's job information, but specify a list of external systems 1120 that are not allowed to access the job information. Certain embodiments may refer to a list of exceptions that are not allowed to access certain information as a “block list”. The external system 1120 belonging to the block list specified by the user is blocked from accessing the information specified in the personal information setting. Various combinations of the granularity of the specification of the information and the granularity of the specification of the entities to which the information is shared are possible. For example, all personal information can be shared with friends, while all work information can be shared with friends of friends.

The authentication server 1144 includes logic to determine whether certain information associated with the user can be accessed by the user's friends, external systems 1120, and/or other applications and entities. The external system 1120 may require authentication from the authentication server 1144 in order to access the user's more personal and sensitive information, such as the user's work phone number. Based on the user's personal information settings, the authentication server 1144 may allow other users, external systems 1120, applications, or other entities to access information associated with the user, including information about actions taken by the user. Decide whether it is allowed or not.

In the example of FIG. 11, the social networking system 1130 may include an image classification module 1146 and an image application module 1148 described in more detail herein. In one embodiment, the image classification module 1146 may collect context hints for a sample set of images and may use the context hints to generate a training set of images. The training set of images can be used to train the classifier to create a visual pattern template of the image class. The classifier may score a rating set of images based on correlation with the visual pattern template. The images with the highest scores among the evaluation set of images may be considered to be most closely related to the image class. In one embodiment, the image classification module 1146 may be implemented as the image classification module 104. The image application module 1148 may interface with other applications to allow searching of classified images. In one embodiment, the image application module 1148 may be implemented as the image application module 106.

Hardware implementation

The processes and features described above may be implemented by a wide variety of machine and computer system architectures in a wide variety of network and computing environments. 12 shows an illustration of a computer system 1200 that may be used to implement one or more embodiments described herein, according to one embodiment. Computer system 1200 includes sets of instructions that cause computer system 1200 to perform the processes and features described herein. Computer system 1200 may be connected (eg, networked) to other machines. In a networked deployment, computer system 1200 may operate at the capacity of a server machine or client machine in a client-server network environment or as a peer machine in a peer-to-peer (or distributed) network environment. In one embodiment, computer system 1200 may be a social networking system 1130, a user device 1110, an external system 1120, or a component thereof. In one embodiment, computer system 1200 may be one of a number of servers that make up all or part of social networking system 1130.

Computer system 1200 includes a processor 1202, a cache 1204, and one or more executable modules and drivers related to the processes and features described herein and stored on a computer-readable medium. Additionally, computer system 1200 includes a high performance input/output (I/O) bus 1206 and a standard I/O bus 1212. The host bridge 1210 connects the processor 1202 to the high performance I/O bus 1206, while the I/O bus bridge 1212 connects the two buses 1206 and 1212 to each other. System memory 1214 and network interface 1216 are connected to a high performance I/O bus 1206. The computer system 1200 may further include a video memory and a display device (not shown) connected to the video memory. Mass storage 1218 and I/O ports 1220 are connected to a standard I/O bus 1212. Computer system 1200 may optionally include a keyboard, pointing device, display device, or other input/output devices (not shown) connected to a standard I/O bus 1212. Overall, these components include an x126-compatible processor manufactured by Intel Corporation of Santa Clara, CA, and an x126-compatible processor manufactured by Advanced Micro Devices (AMD), Inc. of Sunnydale, CA, as well as any other suitable processor. It is intended to represent a broad category of computer hardware systems including, but not limited to, computer systems based on.

The operating system manages and controls the operation of the computer system 1200, including input and output of data to and from a software application (not shown). The operating system provides an interface between the software applications running on the system and the hardware components of the system. Any suitable operating system may be used, such as the LINUX operating system, the Apple Macintosh operating system available from Apple Computer Inc. of Cupertino, Calif., the UNIX operating system, the Microsoft® Windows® operating system, the BSD operating system, and the like. Other implementations are possible.

The components of computer system 1200 are described in more detail below. In particular, network interface 1216 provides communication between computer system 1200 and any wide range of networks such as Ethernet (eg, IEEE 1202.3) networks, backplanes, and the like. Mass storage 1218 provides permanent storage for programming instructions and data to perform the processes and features implemented by each of the computing systems identified above, while system memory 1214 (e.g., DRAM ) Provides temporary storage for data and programming instructions when executed by processor 1202. The I/O port 1220 may be one or more serial and/or parallel communication ports that provide communication between additional peripheral devices that may be connected to the computer system 1200.

Computer system 1200 may include various system structures, and various components of computer system 1200 may be rearranged. For example, the cache 1204 may be an on-chip having a processor 1202. Alternatively, the cache 1204 and the processor 1202 may be packed together into a "processor module" in which the processor 1202 is referred to as a "processor core." Moreover, certain embodiments may neither require nor include all of the above components. For example, peripheral devices connected to the standard I/O bus 1212 may be connected to the high performance I/O bus 1206. Additionally, in some embodiments, only a single bus may exist, and components of computer system 1200 may be connected to a single bus. Moreover, computer system 1200 may include additional components such as additional processors, storage devices, or memory.

In general, the processes and features described herein may be implemented as part of an operating system or a specific application, component, program, object, module, or series of instructions referred to as a “program”. For example, one or more programs may be used to execute certain processes described herein. Programs typically contain one or more instructions in computer system 1200 that, when read and executed by one or more processors, cause the computer system 1200 to perform operations that perform the processes and features described herein. It can be included in the device. The processes and features described herein may be implemented in software, firmware, hardware (eg, application specific integrated circuit (ASIC), or any combination thereof).

In one implementation, the processes and features described herein may be implemented individually or collectively in a distributed computing environment as a series of executable modules executed by the computer system 1200. The above-described modules may be realized by hardware, a computer-readable medium (or a machine-readable medium), or a combination of both. For example, a module may include a plurality or series of instructions executed by a processor in a hardware system such as processor 1202. Initially, a series of instructions may be stored in a storage device such as mass storage 1218, for example. However, the series of instructions may be stored on any suitable computer-readable storage medium. Moreover, the series of commands need not be stored locally, but may be received from a remote storage device, such as a server on a network, via network interface 1216. Instructions are copied from a storage device, such as mass storage 1218 to system memory 1214, and then accessed and executed by processor 1202. In various implementations, a module or modules may be executed by a processor or multiple processors in one or multiple locations, such as multiple servers in a parallel processing environment.

Examples of computer-readable media include, for example, media of a recordable type such as volatile and non-volatile memory devices; Solid state memory (SSD); Floppy and other removable disks; Hard disk drives, magnetic media; Optical disks (eg, Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks (DVDs)); Other similar non-transitory (or transient), tangible (or non-substantial) storage media; Or any type of medium suitable for storing, encoding, and carrying a series of instructions for execution by computer system 1200 performing any one or more processes and features described herein. It doesn't work.

For illustrative purposes, a number of specific details are referred to to provide a thorough understanding of the description of the invention. However, it will be apparent to those skilled in the art that the embodiments of the present specification may be implemented without these specific details. In some instances, modules, structures, processes, features, and devices are shown in block diagram form in order to avoid obscuring the description of the invention. In another example, functional block diagrams and flow diagrams are shown to represent the flow of data and logic. The components of the block diagram and flow diagram (e.g., modules, blocks, structures, devices, features, etc.) may be variously combined, separated, removed, rearranged, and other ways than those explicitly described and illustrated herein. , Can be replaced.

References to "one embodiment", "example", "some embodiments", "various embodiments", "specific embodiments", "other embodiments", "series of embodiments" and the like in the present specification are, It means that a specific feature, design, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present specification. The appearances of phrases such as "in one embodiment" or "in an embodiment" in various places in the specification refer to the same embodiment or are mutually exclusive with other embodiments and refer to separate or alternative embodiments. It is not necessarily mentioned. Moreover, regardless of whether there is an explicit reference to “an embodiment” or the like, various features that can be variously combined and included in some embodiments are described, but these are also variously omitted in other embodiments. Similarly, various features are described that may be a preference or requirement for some embodiments, but not for other embodiments.

The terms used herein are in principle selected for readability and instructional purposes, and are not selected to describe or limit the subject of the invention. Accordingly, the scope of protection is not intended to be limited by the detailed description of the present invention, but is intended to be defined by the submitted claims on which the application is based. Therefore, the disclosure of the embodiments is intended to be illustrative rather than limiting the scope of protection defined in the following claims.

Claims (20)

At least one processor; And
Receiving a sample set of images selected for training an image classifier associated with a plurality of image classes, wherein at least one image in the sample set is associated with one or more social cues, The one or more social cues comprising the reliability of the at least one creator or the tagger of at least one image based on the conversation of the at least one creator or tagger on a social networking system to receive a sample set of images. that;
Scoring a correlation for an image class of each image in the sample set based on one or more social cues associated with the image; And
And a memory storing instructions configured to instruct at least one processor to perform determining a training set of images to train a classifier from the sample set based on the scoring. The method of claim 1,
The system further comprising specifying the image class. The method of claim 1,
Wherein the determining includes ranking each image in the sample set of images based on the scoring. The method of claim 1,
Wherein the determining includes selecting a subset of the highest score among the sample set of images. The method of claim 4,
The subset of the top score is a training set of the images. The method of claim 1,
Further comprising training the classifier based on the training set of images. The method of claim 1,
The system further comprising creating a visual pattern template associated with the image class. The method of claim 1,
Wherein the classifier is configured to use a bag of visual words image classification technique or a neural network image classification technique. The method of claim 1,
The system further comprising determining an extent to which an evaluation set of images correlates to the image class. The method of claim 9,
The evaluation set of images is different from the sample set of images. The method of claim 9,
The evaluation set of images comprising a set of images that are larger than the sample set of images. The method of claim 9,
And scoring a correlation of each image of the rating set of images by a visual pattern template associated with the image class. The method of claim 12,
And ranking each image of the rating set based on scoring a correlation of each image of the rating set of images. The method of claim 12,
And associating a subset of the highest score of the rating set of images to the image class. The method of claim 1,
The one or more social cues comprising one or more image tags. The method of claim 15,
Further comprising determining a number of instances of a particular image tag in the total number of the one or more image tags associated with an image. The method of claim 1,
The one or more social cues may include: location data associated with an image of the sample set of images; Or one or more of the identity of an uploader, tagger, or owner of the image of the sample set of images. The method of claim 1,
The one or more social queues are received by the social networking system. Receiving, by a computer system, a sample set of images selected for training an image classifier associated with a plurality of image classes, wherein at least one image in the sample set is associated with one or more social cues, the one Receiving a sample set of images, wherein the social cues include a credibility of the at least one creator or tagger of the at least one image based on a conversation of at least one creator or tagger on a social networking system;
Scoring, by the computer system, a correlation for an image class of each image in the sample set based on one or more social cues associated with the image; And
Determining, by the computer system, a training set of images to train a classifier from the sample set based on the scoring. Receiving a sample set of images selected for training an image classifier associated with a plurality of image classes, wherein at least one image is associated with one or more social cues, wherein the one or more social cues are social Receiving a sample set of images comprising the reliability of the at least one creator or tagger of the at least one image based on the conversation of at least one creator or tagger on a networking system;
Scoring a correlation for an image class of each image in the sample set based on one or more social cues associated with the image; And
A computer-implemented method comprising determining a training set of images to train a classifier from the sample set based on the scoring, comprising: computer storage storing computer-executable instructions that, when executed, cause a computer system to perform. media.

Images