DE102016013631A1 - Modeling semantic concepts in an embedding space as distributions - Google Patents

Abstract

The modeling of semantic concepts in an embedding space is described as distributions. In the embedding room both images and text labels are displayed. The text labels describe semantic concepts that are present in the image content. In the embedding space, the semantic concepts described by the text labels are modeled as distributions. Using distributions, each semantic concept is modeled as a continuous cluster that can overlap with other clusters that model other semantic concepts. For example, a distribution for the semantic concept "apple" may overlap with distributions for the semantic concepts "fruit" and "tree," as it may affect both a fruit and a tree. In contrast to conventionally configured embedding spaces, the embedding spaces described herein are created to model semantic concepts as distributions, such as Gaussian distributions, Gaussian mixed distributions, and the like.

Description

background

A fundamental problem in the computer field is image classification. In general, image classification attempts to extract semantic information from an image so that the image can be labeled to describe the content of the image. For example, semantic information may include objects represented in an image (and locations of the image on which the objects are depicted), scenes represented in an image (eg, whether the image is a beach or a sunset), moods, which are associated with human faces or facial expressions represented in an image, image aesthetics (eg, good composition, poor composition, third-rule rule, and the like), image feeling (eg, fear, anger, and the like), and the like.

Some conventional image classification techniques use visual-semantic embedding spaces in which both text labels that describe semantic concepts and images that exemplify the semantic concepts are embedded. The visual-semantic embedding spaces created according to conventional techniques usually represent semantic concepts described by text labels as individual points in the embedding space. Because of the complicated relationships between semantic concepts and the multitude of meanings that carry a single semantic concept However, representing semantic concepts as individual points in an embedding space may not reflect the way they are understood in the real world. Conventionally configured visual-semantic embedding spaces may therefore present semantic concepts inaccurate. Because these inaccuracies may be adopted when used with text labels, such embedding spaces may not be suitable for accurately labeling images with text labels to describe their contents.

Summary

The modeling of semantic concepts in an embedding space is described as distributions. In the embedding room both images and text labels are displayed. The text labels describe semantic concepts that are present in the image content. In the embedding space, the semantic concepts described by the text labels are modeled as distributions. Using distributions, each semantic concept in the embedding space is modeled as a continuous cluster that can overlap with other clusters that model other semantic concepts. For example, a distribution for the semantic concept "apple" may overlap with distributions for the semantic concepts "fruit" and "tree", which may affect both a fruit and a tree. Compared to the use of distributions, conventionally configured embedding spaces represent a semantic concept as a single point. In contrast to conventionally configured embedding spaces, the embedding spaces described herein are generated to use semantic concepts as distributions (eg, Gaussian distributions, Gaussian mixed distributions, and the like) such more), resulting in a more detailed representation of the semantic concepts, more opportunities to annotate images with text labels that more accurately describe their displayed content, and the like.

Once an embedding space is trained, the embedding space can be used to locate text labels to describe the content of an image. To use a trained embedding space to locate text labels that describe the content of an image, multiple semantically meaningful areas of the image may be determined, and corresponding text labels may be located in the trained embedding space for each of the areas. The text labels that are located for the multiple regions of the image may then be associated with the image, for example, for use in annotating the image.

The present Summary introduces a selection of concepts in simplified form, which are further described below in the Detailed Description. As such, the present summary is not intended to identify the essential characteristics of the claimed subject matter, nor should it be taken as an aid in determining the scope of the claimed subject matter.

Brief description of the drawing

The detailed description is based on the accompanying figures. In the figures, the leftmost digit / denote the leftmost digits of a reference numeral the figure in which the numeral first appears. The use of the same reference numerals in various contexts in the description and figures may refer to similar or identical objects. Entities depicted in the figures may indicate one or more entities, and therefore reference to a single or multiple forms of the entities may be equivalently made in the discussion.

1 FIG. 10 is an illustration of a digital environment in an exemplary implementation operable to employ the techniques described herein.

2 FIG. 10 is an exemplary implementation for displaying an image with multiple text labels and regions of the image corresponding to the multiple text labels.

3 is an exemplary implementation of an embedding space configured to embed both text labels and image areas mapped to the text labels.

4 is another exemplary implementation of the embedding space wherein semantic concepts described by the textual labels are modeled as distributions in the embedding space.

5 FIG. 10 is a flowchart illustrating a procedure in an exemplary implementation in which portions of a training image associated with multiple text labels are mapped to the corresponding text labels in an embedding space and tagging techniques are applied to a query image to provide text labels representing portions of the text Query image correspond to locate in the embedment space.

6 FIG. 10 is a flowchart illustrating a procedure in another exemplary implementation in which a trained embedding space is used to associate multiple text labels with respective portions of an image to be commented.

7 Figure 4 is a flow chart illustrating a procedure in another exemplary implementation in which an embedding space is created to model semantic concepts described by text labels as distributions in the embedding space.

8th is an exemplary system that includes various components of an exemplary device that may be employed for one or more implementations of techniques for modeling semantic concepts in an embedding space as distributions as described herein.

detailed description

Overview

Conventional visual-semantic embedding techniques use semantic information from uncommented textual data to gain semantic relationships between textual labels and explicitly map images into a rich semantic embedding space. Traditionally configured visual-semantic embedding spaces, however, usually present semantic concepts as individual points in the embedding space. Because of the complicated relationships between semantic concepts and the multitude of meanings that a single semantic concept can support, representing semantic concepts as individual points reflects the nature How they are understood in the real world, but may not be reflected. As a result, conventionally configured visual-semantic embedding spaces may present inaccurate semantic concepts. Further, if such embedding spaces are used to label images, these inaccuracies may be adopted. As such, conventionally configured visual-semantic embedding spaces may be unsuitable for accurately labeling images with text labels to describe their contents.

The modeling of semantic concepts in an embedding space is described as distributions. In particular, an embedding space is created in which both images and text labels being represented. Compared to conventional techniques, creating the embedding space implies modeling semantic concepts described by textual lables of a textual vocabulary as distributions, such as Gaussian distributions, Gaussian mixed distributions, and the like. Consider the semantic concepts described by the words "tree", "apple" and "fruit". At least there is a certain overlap between the semantic concepts, since an apple is a fruit and also a kind of tree to assign. The presentation of such semantic concepts as distributions, rather than individual points, allows the semantic concepts to overlap such that the distribution for the semantic concept of an apple overlaps the distributions for the semantic concepts of both a tree and a fruit.

The embedding space is configured as a common image text embedding space in which both text labels and representative images (eg, areas of images) are represented. In the embedding space, the distributions model semantic concepts that may exist in images and are described by text labels of a text vocabulary. When modeling the semantic concepts as distributions, semantic relationships between these text labels are also considered. In other words, the semantic concepts are modeled so that distributions representing semantic concepts such as "sun" and "sunset" are correlated more closely than distributions to represent semantic concepts such as "sun" and "bird".

Once the embedding space has been trained as described herein, it can be used to annotate pictures with text labels. Consider an example where an image not yet linked to text labels is received as input for commenting. Label discovery techniques may be applied to the input image to locate text labels from the trained embedding space to describe the content of the input image. For this purpose, semantically meaningful areas of the input image are proposed, for example using a geodetic object proposal. The term "geodetic object suggestion" refers to a technique in which a set of candidate objects in a given image is identified according to critical level sets in geodetic distance transforms computed for seeds placed in the image. To place the seeds, the geodetic object proposal uses trained classifiers that are optimized to locate objects. Once the proposal is made, the semantically meaningful regions proposed for the input image are embedded with the image regions embedded in the embedding space as distributions relative to the semantic concepts with modeling as distributions, for example by comparing vectors to indicate the visual properties of the embedded image regions and suggested areas of the input image compared. Based on the comparison, the proposed areas of the image can be mapped to visually similar embedded image areas and thus also to those text labels that describe the semantic concepts corresponding to the similar embedded image areas. The input image can then be commented on with the text labels found in this process.

The visual-semantic embedding spaces described here (for example, for representing semantic concepts described by text labels as distributions) represent semantic concepts and the relationships between them more precisely than conventionally configured visual-semantic embedding spaces that represent semantic concepts as individual points the techniques described herein train a common image text embedding space using training images with multiple text labels and annotating an image with multiple text labels using the common image text embedding space. The techniques described herein also enable semantically meaningful areas to be found for each of the multiple text labels associated with an image. Moreover, the techniques described herein offer an advantage in zero-shot learning over conventional techniques, for example in classifying images in new categories that have not previously been described by the embedding room text labels.

In the following discussion, an exemplary environment that can employ the techniques described herein will first be described. Next, exemplary implementation details and procedures that may be used in the exemplary environment as well as in other environments will be described. as a result, the performance of the example procedures is not limited to the exemplary environment, and the example environment is not limited to the performance of the example procedures.

Exemplary environment

1 is a representation of an environment 100 in an exemplary implementation operable to employ the techniques described herein. The illustrated environment 100 includes a computing device 102 that is a processing system 104. comprising the one or more processing devices (eg, processors) and one or more computer-readable storage media 106 includes. The illustrated environment 100 also includes image data 108 , Text labels 110 , a multi-instance embedding model 112 , a multi-instance embedding module 114 ( "MIE module 114 ") And area-based multi-tagged images 116 on the computer-readable storage media 106 embodied and through the processing system 104. are operable to implement the corresponding functionality described herein. In at least some implementations, the computing device includes 102 functionality for accessing various types of web-based resources (content and services) for interacting with online providers and the like, as will be described in more detail below.

The computing device 102 is configurable as any suitable type of computing device. The computing device can be configured 102 for example, as a server, desktop computer, laptop computer, mobile device (eg, assuming a hand configuration, such as a tablet or mobile phone), tablet, device configured to receive gesture input, device configured to receive three-dimensional (3D) gestures as input, device with a configuration for receiving a voice input, a device configured to receive a pen-based input, a device configured to receive a combination of said inputs, and the like. Therefore, the computing device 102 from full-resource devices with significant memory and processor resources (e.g., servers, personal computers, game consoles) to a low-resource device with limited storage and processing resources (e.g., mobile devices). In addition, although a single computing device can 102 shown is the computing device 102 Also, for a variety of different devices, be representative of operations "over the cloud," as detailed below 8th is described to perform.

The environment 100 also provides one or more service providers 118 which is to communicate with the computing device 102 over a network 120 For example, the Internet is configured to provide a "cloud-based" computing environment. Generally, this means that the service provider 118 are configured for different resources 122 over the network 120 accessible to clients. In some scenarios, users set up accounts that can be used to access appropriate provider resources. The provider authenticates credentials (credentials) of a user (for example username and password) before having access to an account and corresponding resources 122 granted. Other resources 122 are made freely accessible (for example, without authentication or account-based access). The resources 122 may include any suitable combination of services and / or content that are typically accessed via a network by one or more providers. Some examples of services include, among others, image storage and / or sharing services (eg Flickr ^®), social networking services (such as Facebook ^®, Twitter ^®, Instagram ^®, hyperlapse ^® and the like), search engine services that can output images as search results, and more.

These services serve as sources of significant amounts of image data. The image data 108 represent image data that may be formatted in a variety of image formats including, but not limited to, JPEG, TIFF, RAW, GIF, BMP, PNG, and the like. The image data made available by the Services may be posted by users who have established accounts with these Services. For example, a user who has set up an account with an image storage and / or sharing service uploads images, such as those taken with his digital camera or those sent to the user via electronic means. A user of the image storage and / or sharing service may then share the uploaded images with others, such as by providing a link to albums of images or to a profile of the user.

In addition, the images can be linked with additional information describing the images. The of the image data 108 Images displayed may include metadata describing the images. In one example, the metadata of an image may describe the pixel size of the image, the memory size of the image, the camera device used to capture the image, the date the image was taken, the date on which the image was last modified, and more. In addition, those from the image data 108 images associated with metadata describing the content displayed in the images. The metadata can be used as text labels, for example be configured and the images can be annotated with those text labels describing their content. Such text labels may describe objects represented in an image (and locations of the image on which the objects are depicted), scenes represented in an image (eg, whether the image is a beach or a sunset), moods that are associated with human faces or facial expressions represented in an image, image aesthetic characteristics (for example, good composition, poor composition, respect of the rule of thirds and the like), image feeling (such as fear, anger and the like) and the like. Generally, this means that the text labels 110 a variety of text labels with which the images can be commented represent.

The MIE module 114 provides functionality for implementing techniques for modeling semantic concepts in an embedding space as distributions as described herein. As part of this, the MIE module provides 114 an embedding space configured as a common image text embedding space. A "common picture-text embedding room" here means that both the text labels 110 as well as representative images (for example, areas of images) from the image data 108 can be embedded in it. The multi-instance embedding model 112 is representative of such an embedding space. Furthermore, the MIE module provides 114 a functionality that represents the multi-instance embedding model 112 such that for a given image area the multi-instance embedding model 112 for finding a text label describing the content displayed in the image area can be used.

To train the multi-instance embedding model 112 correlates the MIE module 114 the text labels used in the multi-instance embedding model 112 embedded semantically according to the semantic concepts described by the textual labels. The MIE module 114 For example, the text labels are semantically correlated such that the text labels "Sun" and "Sunset" are closer in the multi-instance embedding model 112 when the text labels "Sun" and "Bird" are positioned.

The MIE module 114 also trains the multi-instance embedding model 112 using training images as part of image data 108 can be included. The training images are already linked before each workout with several text labels that describe the content presented. The MIE module processes for every training image 114 the image for generating a set of regions of the image corresponding to the plurality of text labels of the training image. This forms the MIE module 114 an area-to-label correspondence for each training image such that an area of the training image is selected to represent each of the plurality of textual labels. The MIE module 114 then embeds the set of areas in the multi-instance embedding model 112 one. In particular, the MIE module embeds 114 an area in the multi-instance embedding model 112 based on a position where the corresponding label of the area is embedded. If a training image is linked, for example, to the text label "sun", an area of the training image corresponding to the "sun" is selected. The MIE module 114 then at least partially embeds the region of the training image corresponding to the "sun" based on a position in the multi-instance embedding model 112 with the text label "Sun" embedded. The MIE module 114 can perform this process for each training image in a set of training images.

One or more implementations is the multi-instance embedding model 112 configured to model the semantic concepts described by the text labels and represented by the representative images as distributions, such as Gaussian distributions, Gaussian mixed distributions and the like. This is in contrast to techniques in which the semantic concepts are represented as individual points. Embedding images associated with multiple text labels at individual points in a visual-semantic embedding space can cause visual-semantic embedding functions to be easily confused, for example, causing the techniques to predict the content displayed in an image are less accurate.

In addition, the MIE module provides 114 a functionality for commenting images. "Commenting" means that text labels that describe the content displayed in an image can be selected and linked to the image. To comment on an image, the MIE module determines 114 a set of areas of the image. In particular, the MIE module extracts 114 Regions of the image that are semantically meaningful, for example using a geodetic object suggestion, as it covers both foreground and background areas. The MIE module 114 then applies one or more label discovery techniques to the set of areas to locate labels for the areas. This implies comparing each area with those in the multi-instance embedding model 112 embedded representative images. The MIE module 114 can then determine which one in the multi-instance embedding model 112 embedded images to one is similar to the commenting area, and the text label corresponding to similar embedded images is linked to the area to be commented.

In this way, an image may initially be associated with a large number of text labels, such as one for each of the suggested regions. However, the number of text labels can be reduced so that an image can be annotated with the labels that are determined to be most representative of the image content. So can the MIE module 114 For example, select an image with a predetermined number of text labels. In this scenario, the MIE module chooses 114 the predetermined number of text labels to which the areas of the image belong in the multi-instance embedding model 112 most similar to embedded images. Alternatively, the MIE module selects 114 a variable number of text labels for images to be annotated. In this alternative scenario, the number of selected text labels may be based on a threshold similarity such that any portions of the image that resemble an embedded image above the threshold similarity will use the retrieved text label to annotate the image. In any case, the area-based multi-tagged images represent 116 Images representing the MIE module 114 annotated with multiple text labels by defining image areas and the multi-instance embedding model 112 used to locate labels for the image areas.

In one or more implementations, the MIE module is 114 as a software module, hardware device, or using a combination of software, hardware, firmware, fixed logic circuitry, and the like. Furthermore, the MIE module 114 as an independent component of the computing device 102 as shown, be implementable. Additionally or alternatively, the MIE module 114 as a component of a web service, as an application or app, as the operating system of the computing device 102 , as a plug-in module or as another device application or app, as shown below 8th be configured.

Having considered an exemplary environment, a discussion of some exemplary details of the techniques for modeling semantic concepts in an embedding space as distributions according to one or more implementations now follows.

Modeling semantic concepts in an embedding space as distributions

This section describes exemplary details of techniques for modeling semantic concepts in an embedding space as distributions according to one or more implementations. The discussion is based on 2 which represents a picture with multiple text labels and the concepts relevant to the functionality described herein.

In particular, presents 2 generally included 200 a picture 202 with text labels 204 . 206 . 208 . 210 . 212 represents. 2 also provides image areas 214 . 216 . 218 . 220 dar. The picture 202 can represent a training image already with the text labels 204 . 206 . 208 . 210 . 212 and for that the MIE module 114 the image areas 214 . 216 . 218 . 220 such that a correspondence to the text labels is provided such that, for example, the image area 214 the text label 208 corresponds to the image area 216 the text label 206 corresponds to the image area 218 the text label 204 corresponds to the image area 220 the text label 210 corresponds to, and the image area, the image 202 as a whole, the text label 212 equivalent. Alternatively, the picture 202 represent an image that is from the MIE module 114 so commented that the MIE module 114 suggested areas for the picture 202 generates text labels for the proposed areas using the multi-instance embedding model 112 locates and the text labels 204 . 206 . 208 . 210 . 212 that the image areas 214 . 216 . 218 . 220 (as well as an area corresponding to the image as a whole) among the retrieved text labels for describing the content of the image 202 selects.

3 generally indicates 300 an exemplary embedding space in which both text labels and image areas mapped to the text labels are embedded. In particular, includes 3 a common image-text embedding space 302 (the multi-instance embedding model 112 from 1 can correspond), an image database 304 (which contains images from the MIE module 114 to train the common image-text embedding space 302 used) and a text vocabulary 306 (the text labels 110 from 1 which can be used to describe semantic concepts represented in the images).

The common image text embedding room 302 is configured to have both text labels and pictures (pictures as a whole and parts of pictures) embedded therein. The image database 304 and the text vocabulary 306 represent the images or text labels that make up the MIE module 114 to train the common image-text embedding space 302 used to allow the common image text embedding space 302 for example, to locate text labels that describe the content of an image. The text labels that make up the text vocabulary 306 are configured to describe semantic concepts that may be present in the image content. The images of the image database 304 represent a picture content, which through the text labels of the text vocabulary 306 exemplifies described semantic concepts.

When training, the MIE module correlates 114 semantically the text labels within the text vocabulary 306 , Take, for example, the text label 308 for "tree". The MIE module 114 provides functionality for semantically correlating the text label 308 for "tree" with the text label 310 for "woman" and the text label 312 ready for "building" by, for example, the text labels 308 . 310 . 312 at locations in the common image text embedding space 302 be embedded. The positions where the text labels in the common image text embedding space 302 depending on how closely the semantic concepts described by the textual labels are correlated. In one example, the text label 308 for "tree" in the common image text embedding space 302 be embedded so that it is closer to the text labels for "pine", "forest" and "plants" (not shown) than to the text label 310 for "woman" and the text label 312 is positioned for "building".

In 3 is each of the text labels 308 . 310 . 312 shown near a filled form denoting the respective text label. Such is the text label 308 for "tree", for example, shown near a solid star, the text label 310 for "woman" is shown near a filled cross, and the text label 312 for "building" is shown near a solid X-shape. Each of these filled forms is also in the common image text embedding space 302 involves embedding the text labels 308 . 310 . 312 display. Note that the illustrated form of the common image-text embedding space 302 and the illustrated positions of the filled shapes represent the actual shape of the common image text embedding space 302 or the positions where the MIE module 114 the text labels 308 . 310 . 312 actually embeds, may not accurately reproduce. Instead, the illustrated form of the common image-text embedding space 302 and the illustrated positions of the filled shapes will be useful in explaining the techniques described herein.

The MIE module 114 also correlates the images in the image database 304 with the text labels of the text vocabulary 306 , As stated above, the training images of the image database 304 represent a picture content that matches that through the text labels of the text vocabulary 306 exemplifies described semantic concepts. To correlate the images with the text labels, the MIE module embeds 114 each image at a location in the common image text embedding space 302 partly based on a position of the corresponding text label. In other words, the MIE module 114 maps each image to the text label that represents the image content for the image and embeds the image based on the position of this text label.

The images of the image database 304 that represent a tree are in the common image text embedding space 302 for example, based on a position embedded at the text label 308 is embedded for "tree". This is how the MIE module forms 114 For example, those images that represent trees on the text label 308 for "tree". Similarly, images representing a woman are embedded based on a position at which the text label 310 is embedded for "woman", and images representing a building are embedded based on a position at which the text label 312 is embedded for "building". In other words, an image will be in the common image text embedding space 302 embedded at a position closer to a text label corresponding to the picture element than to other text labels. An image is also in the common image text embedding space 302 closer to images that correspond to closely correlated semantic concepts than to images that correspond to less closely correlated semantic concepts. In one example, the image of a tree is in the common image text embedding space 302 generally closer to pictures of pines and forests than to pictures of buildings or pictures of women.

In 3 represents the image set 314 the images of the image database 304 representing a tree, the image set 316 Represents the images of the image database 304 representing a woman, and the image set 318 Represents the images of the image database 304 representing a building. The pictures of each picture set 314 . 316 . 318 are represented as unfilled shapes that designate the images. Such are the pictures of the picture set 314 (representing the trees) are shown as non-filled stars, the images of the image set 316 (representing women) are shown as unfilled crosses, and the images of the image set 318 (representing the buildings) are shown as non-filled X-entities.

The unfilled shapes are also in the common image text embedding space 302 includes and represent the embedding of images from the image sets 314 . 316 . 318 whose positions are based on where the corresponding text labels 308 . 310 . 312 are embedded. As a result, the unfilled stars representing the images that show trees are depicted around the filled star surrounding the text label 308 for "tree", the unfilled crosses that represent the images that show women are depicted around the filled cross, which is the text label 310 for "Ms." and the unfilled X-shapes that represent the images that depict "buildings" are shown around the solid X-shape that contains the text label 312 represents for "building".

Regarding the respective pictures, in the common picture text embedding room 302 embedded, the MIE module extracts 114 these from the training images as areas. Therefore, the set of images 304 that represent women, areas (for example, portions of images that are smaller than the entire image) that are extracted from training images. More generally, this means that to train the common image-text embedding space 302 the MIE module 114 begins with the training of images, each associated with multiple text labels. In one example, the MIE module begins 114 with the training of pictures, so for example with the picture 202 that already with the text labels 204 . 206 . 208 . 210 . 212 is linked.

The MIE module generates for each training image 114 a set of areas corresponding to the text labels of the training image. Consider the example of 2 in which the picture 202 with the text labels 204 . 206 . 208 . 210 . 212 is linked. In this example, the MIE module can 114 Create an area corresponding to each of the text labels 204 . 206 . 208 . 210 . 212 such that corresponds to the image area 214 the text label 208 for "woman" corresponds to the image area 216 the text label 206 for "Eiffel Tower", the image area 218 the text label 204 for "bag" corresponds to the image area 220 the text label 210 corresponds to "building" and the image area containing the image 202 as a whole, the text label 212 for "city scene".

The MIE module uses to create the areas corresponding to the text labels of the training image 114 initially a range suggestion technique for the training image. The MIE module 114 For example, it uses at least one of a geodetic object suggestion technique, arbitrary range proposition technique, and the like. Semantically meaningful areas of an image contain not only objects that are shown in the foreground of the image. Rather, background portions of an image may also be semantically meaningful. In contrast to general object suggestion techniques, the range suggestion techniques covered by the MIE module cover 114 Both foreground and background areas of images are used to suggest image areas.

Once a set of candidate regions has been suggested for a training image, the MIE module discards 114 at least some of the proposed candidate areas. The number of candidate areas, initially from the object suggestion techniques, which is the MIE module 114 may include thousands, so using each of the proposed candidate areas for training would consume significantly the computational resources. Accordingly, the MIE module 114 employ one or more post-processing techniques to discard at least some of the proposed candidate areas. The MIE module 114 For example, it may employ one or more post-processing techniques to weed out candidate candidate regions that do not meet predefined size criteria or aspect ratio criteria. By way of example and not by way of limitation, a predefined size criterion may specify that to use a proposed candidate area, the pages of the proposed candidate area should be at least 30% of the length of a corresponding page of the image. For example, for an image having a size of 400 pixels by 500 pixels, the MIE module resets 114 the criterion by discarding a suggested candidate area having a size of less than 120 pixels by 150 pixels. With regard to the exemplary aspect ratio, a predefined aspect ratio criterion may specify that to use a suggested candidate range, the aspect ratio of the proposed candidate range should be between 1: 4 and 4: 1.

Of the remaining proposed candidate areas, the MIE module indicates 114 then the best fit candidate area is shared with each text label of the training image to build an area-to-label correspondence. Here is the MIE module 114 configured for one Use single label embedding model. This has the effect of discarding an even larger number of proposed candidate areas such that the number of remaining areas corresponds to the number of text labels associated with the image. The MIE module 114 then embeds these areas in the common image-text embedding space 302 one. As in the example of 2 and in the scenario where the picture 202 to train the common image-text embedding space 302 is used, the MIE module may apply thousands of the candidate areas proposed by the area proposal techniques to the image areas 214 . 216 . 218 . 220 as well as an area corresponding to the picture 202 decrease as a whole.

To embed the areas of the training images in the common image-text embedding space 302 calculates the MIE module 114 d-dimensional feature vectors for the ranges. The MIE module 114 For example, the feature vectors for the regions may be computed by extracting image features with a Convolutional Neural Network (CNN), such as one that includes convolution and pooling, as well as inception layers. For a specific example, the MIE module extracts 114 the image features using the RCNN technique (fast region-based CNN). In other words, given a training image and areas in the common image text embedding space 302 to embed, directs the MIE module 114 the training image once through the CNN. The MIE module 114 may then pool each area of the generated set of areas in a fixed size feature map to determine a final feature vector.

Consider an example where the MIE module 114 the common image-text embedding space 302 is trained using a set of training images, each associated with multiple text labels. Such a set of training images can be considered D ≡ {(x _i , y _i )} N / i = 1 be formulated. Here, x _i denotes the i-th image in the sentence, while N denotes the number of training images in the sentence.

dargestellt werden, der gemäß vorstehender Diskussion berechnet werden kann, und zwar beispielsweise für das i-te Bild des Satzes x_i ∊

Des Weiteren ist jedes Trainingsbild derart mit einem Satz von Textetiketten verknüpft, dass für das i-te Bild x_i der Satz von Textetiketten y_i als (y 1 / i, ..., y t / i) definiert ist, wobei t die Anzahl von Textetiketten, die mit dem i-ten Bild verknüpft sind, bezeichnet. Man beachte, dass diese Definition für den Satz von Textetiketten ermöglicht, dass die Anzahl von Etiketten, die mit den Trainingsbildern verknüpft sind, von Bild zu Bild variiert.Furthermore, each image may be represented by a d-dimensional feature vector

can be calculated, which can be calculated according to the above discussion, for example, for the ith image of the set x _i ε

Furthermore, each training image is linked to a set of text labels such that for the i-th image x _i the set of text labels y _i as (y 1 / i, ..., yt / i) where t is the number of text labels associated with the ith image. Note that this definition for the set of text labels allows the number of labels associated with the training images to vary from image to image.

bezeichnet. Herkömmliche Techniken formulieren das Klassifikations- oder Kommentierungsproblem für Bilder, die mit mehreren Textetiketten verknüpft sind, als Mehr-Klassen-Klassifikation. Im Gegensatz zu den hier beschriebenen Techniken definiert diese herkömmliche Formulierung vorab einen festen Satz von Klassenetiketten, so beispielsweise y 1 / i ∊ Y ≡ {1, ..., m}, gewinnt einen m-Wege-Klassifikator oder m Einer-gegen-Alle-Klassifikatoren und verwendet den gewonnenen Klassifikator oder die gewonnenen Klassifikatoren zum Klassifizieren des Satzes von Trainingsbildern X als festen Satz von Klassenetiketten Y derart, dass (X → Y) gilt. Die Skalierbarkeit derartiger herkömmlicher Techniken ist jedoch mit Blick auf die Anzahl von Etiketten beschränkt, da herkömmliche Techniken die Modelle, die sie verwenden, neutrainieren, wenn neue Etiketten auftauchen.For the sake of simplicity, the set of training images to be embedded will be included

designated. Conventional techniques formulate the classification or annotation problem for images associated with multiple text labels as a multi-class classification. In contrast to the techniques described herein, this conventional formulation pre-defines a fixed set of class labels, such as y 1 / i ε Y ≡ {1, ..., m}, obtains an m-way classifier or m one-to-all classifiers and uses the obtained classifier or classifiers to classify the set of training images X as a fixed set of class labels Y such that (X → Y) holds. However, the scalability of such conventional techniques is limited in view of the number of labels since conventional techniques re-train the models they use when new labels appear.

Instead of defining text labels as a discrete sentence, as is the case with conventional techniques, visual semantic models gain a continuous semantic space S that captures semantic relationships between text labels and a mapping function (X → S) from the set of training images X. wins the continuous semantic space S explicitly. Therefore, the MIE module is 114 configured to construct an image tag training data pair {(x _i , s (y _i ))} to provide a mapping function f: X → S for the common image text embedding space 302 to win. The MIE module 114 the embedding function wins the i-th image x _{i of} the set of training images onto the common image text embedding space 302 map. The MIE module 114 obtains this as a function s (y _i ) of the set of text labels y _{i associated} with the ith image x _i . The function s (y _i ) is obtained such that a set of labels Y used is mapped to the continuous semantic space S such that s: Y → S.

In one or more implementations, the MIE module is 114 configured to use the continuous semantic tag space S (for example, the relationships between the text labels) using text modeling architecture technique provided with uncommented text data, such as For example, with uncommented text data from the Internet, is trained to win. The text modeling architecture allows the MIE module 114 similar embedment positions for semantically correlated text labels gain because synonyms are more likely to occur in similar semantic contexts. In one example, the MIE module uses 114 a glove model to construct a 300-dimensional text label space S that embodies the semantic relationships between text labels.

With respect to the recovery of embedding functions, some conventional visual-semantic solutions win the mapping function f: X → S for embedding space having an L ₂ -Verlustfunktion (L ₂ loss function) or a rank order loss function (ranking loss function). In general, such functions promote a projected embedding point for a given image closer to its corresponding label in the embedding space. The following is an example of such a rank order loss function used by conventional visual-semantic approaches. In particular, it corresponds to a pair-based hinge-ranking loss function:

bezeichnet positive Etikettenvektoren, und

bezeichnet negative Etikettenvektoren. Der Term

bezeichnet einen Abstand zwischen einem Einbettungsvektor

für das i-te Bild x_i und einem Vektor y_i für die Etiketten des Bildes.Here, the term f (·) represents an embedding function to be obtained, while the term m represents an edge. For ease of notation, y is used instead of s (y) to designate the tag vector in the continuous semantic tag space S. Furthermore, τ ₊ denotes a set of positive text labels, τ _- denotes a set of negative text labels,

denotes positive label vectors, and

denotes negative label vectors. The term

denotes a distance between an embedding vector

for the i-th image x _i and a vector y _i for the labels of the image.

derart, dass das i-te Bild x_i an einer Position in dem Einbettungsraum nahe an den mehreren Textetiketten (die an Positionen relativ weit entfernt voneinander eingebettet sind) eingebettet ist, Fehler bei der Einbettungsfunktion bewirken. Führen kann eine derartige Vorgehensweise zusätzlich oder alternativ zum Abbilden des Bildes x_i auf eine durchschnittliche Position der Etikettenvektoren derart, dass die durchschnittliche Position einem anderen semantischen Konzept als die durch die einzelnen Textetiketten beschriebenen Konzepte entspricht.However, conventional visual-semantic approaches using the pair-based articular function as described above have disadvantages. For example, although each image x _i may correspond to multiple labels in the set of text labels y _i , one or more pairs of those labels may be embedded at positions that are relatively far apart in the continuous semantic label space S. Therefore, moving the embedding of an entire image may be a difficult task

such that the i-th image x _{i is} embedded at a position in the embedding space near the plural text labels (which are embedded at positions relatively far apart from each other), causing errors in the embedding function. Such an approach may additionally or alternatively map the image x _i to an average position of the label vectors such that the average position corresponds to a different semantic concept than the concepts described by the individual text labels.

In contrast to conventional visual-semantic embedding techniques, the MIE module is 114 configured to use the subsequent loss function in one or more implementations to obtain a multi-instance embedding for the common image text embedding space 302 to use:

die positiven Etikettenvektoren,

bezeichnet die negativen Etikettenvektoren, und

bezeichnet einen Abstand zwischen der Einbettung des Bildbereiches f(x c / i) und dem entsprechenden Textetikett y_j, für das der Bildbereich erzeugt wird. Bei einer oder mehreren Implementierungen berechnet das MIE-Modul 114 den Abstand als quadrierten Euklidischen Abstand. Des Weiteren kann, wie in der Verlustgleichung gezeigt ist, zum Gewinnen der Multi-Instanzen-Einbettung das MIE-Modul 114 Bereich-zu-Etikett-Entsprechungen durch Anwenden einer Minimaloperation (beispielsweise des min-Terms) auf die Abstände von einem Etikett zu jedem Bereich des Satzes von Bereichen modellieren. Hierbei wählt das MIE-Modul 114 wenigstens einen Bereich eines Trainingsbildes aus, um das durch jedes Textetikett des Trainingsbildes beschriebene semantische Konzept zu erläutern. Des Weiteren wählt das MIE-Modul 114 den Bereich für ein Textetikett mit einer eingebetteten Position am nächsten an dem Textetikett in dem Einbettungsraum wahrscheinlicher aus.Here, the term denotes xc / i a certain area c of the i-th image x _i , that is, for example, a region generated in the manner described above in correspondence with that of the text labels linked to the ith image x _i . The term C denotes the set of regions which are generated for the ith image x _i . Further referred to

the positive label vectors,

denotes the negative label vectors, and

denotes a distance between the embedding of the image area f (xc / i) and the corresponding text label y _j for which the image area is created. In one or more implementations, the MIE module calculates 114 the distance as a squared Euclidean distance. Furthermore, as shown in the loss equation, to obtain the multi-instance embedding, the MIE module may be used 114 Model area-to-label correspondences by applying a minimum operation (e.g., the min-term) to the distances from a label to each area of the set of areas. Here, the MIE module selects 114 at least a portion of a training image to explain the semantic concept described by each text label of the training image. Furthermore, the MIE module chooses 114 the area for a text label with an embedded position closest to the text label in the embedding space is more likely.

In one or more implementations, the MIE module optimizes 114 the ranking of the predicted text labels for the areas of a training image. For this purpose, the MIE module uses 114 a customized multi-instance embedding loss formula that allows positive text labels (for example, those associated with the training image) to have smaller min increments than most negative text labels. In this way, the MIE module maps 114 ranked the positive text labels at the top of a list of predicted text labels for a given area of a training image. In other words, the MIE module 114 Punishes false forecasts that rank positive text labels at the bottom of the list. In one example, the MIE module 114 be configured to optimize the ranking of the predicted text labels according to the following fitted formula:

Here, the term w (·) is a weighting function, while the term r _{j is} a ranking of a positive text label in a list of predicted text labels for a given area of a training image. In one or more implementations, the MIE module determines 114 the term r _j is as follows:

eine Indikatorfunktion, die das MIE-Modul 114 für das i-te Bild x_i zum Rangordnen eines vorhergesagten Textetiketts y_t entsprechend dessen Abstand (beispielsweise in dem gemeinsamen Bild-Text-Einbettungsraum 302) zu jedem Bereich in dem Satz von Bereichen C verwendet. Insbesondere berechnet das MIE-Modul 114 die Rangordnung als Funktion des Minimalabstandes zu jedem Bereich, was durch den folgenden Term dargestellt wird:

Here is the expression

an indicator function, which is the MIE module 114 for the i-th image x _i for ranking a predicted text label y _t according to its spacing (for example in the common image text embedding space 302 ) to each area in the set of areas C. In particular, the MIE module calculates 114 the ranking as a function of the minimum distance to each area, which is represented by the following term:

Furthermore, the MIE module 114 be configured to use a weighting function defined as follows:

Here, the term # (τ ₊ ) denotes the number of text labels associated with the ith image x _i . If one of the text labels associated with the i-th image x _i is one of the top # (τ ₊ ) text labels in the prediction list, the MIE module will punish 114 the loss by assigning a smaller number to the weighting. For example, if the i-th image x _i is associated with five text labels and one of the linked text labels is one of the top five ranked labels in the list of text labels predicted for the image, the MIE penalizes -Module 114 the calculated loss by assigning a smaller number to the weighting. Conversely, if one of the labels associated with the i-th image x _{i is} not one of the topmost # (τ ₊ ) text labels in the prediction list, the MIE module will punish 114 the loss by assigning a larger number to the weighting.

Further, using an example where the i-th image x _{i is associated} with five text labels, consider a scenario in which one of the linked text labels does not include any of the top five ranked labels in the list for the Image is predicted text labels. In this case, the MIE module punishes 114 the calculated loss by assigning a larger number to the weighting. By applying abstraction to the loss function in this way, the MIE module forms 114 semantically meaningful areas closer to their corresponding labels in the common image-text embedding space 302 from. After the areas of the training images (for example, those in the Image Database 304 includes) on the text labels (for example, those in the text vocabulary 306 are included) in the common image text embedding space 302 may be used in that the MIE module 114 Find text labels for uncommented images.

Consider an example where in the MIE module 114 an image for commentary x 'is input, for example, in connection with the indexing of images for the search. In a manner similar to that of creating areas for the trained images, the MIE module generates 114 a set of areas C 'for the picture to be commented x'. This is how the MIE module applies 114 For example, an area proposal technique (eg, a geodetic object proposal or an arbitrary area proposal) is applied to the image to be commented and then applies post-processing techniques to the suggested areas to reduce the number of suggested areas for the image to be commented.

The MIE module 114 then directs the image to be commented and the set of areas created therefrom to the common image text embedding space 302 to determine embed vectors for each of the areas. If the embedding function explained above is given, the MIE module calculates 114 Embedding vectors for a set of regions of the image to be commented such that an embedment vector f (x ^'c ) is determined for a given region c. For any of the text labels y 'in the common image text embedding space 302 calculates the MIE module 114 then the distance between the image to be commented x 'and positions of the text labels y' in the common image text embedding space 302 by the following term:

This determines the MIE module 114 for each area of the picture to be commented, a text label to which the area in the common picture text embedding space 302 for example, based on the distance between the area-determined embedding vector and a position of the text label in the common image text embedding space 302 ,

The MIE module 114 then arranges the text labels intended for the picture to be commented by rank. In particular, the MIE module orders 114 the particular text labels ranked according to the distances between the embedding vectors of the areas and the respective text labels such that the text labels for which the distance to the respective area is smaller are ranked more favorably. Likewise, the particular text label for which the distance to the particular area is the lowest is the top ranked text label. Using the Ranking creates the MIE module 114 a ranked list of text labels that are determined to correspond to the areas of the image to be commented. The MIE module 114 can then comment on the image to be commented with a predetermined number of labels from the list. By way of example and not by way of limitation, the MIE module 114 Comment on the image to be commented with the top five ranked text labels from the list of text labels corresponding to the areas. Regardless of the predetermined number, the MIE module is 114 configured to use the common image text embedding space 302 to comment on images with multiple text labels.

As briefly described above, the common image text embedding space is 302 also configured to model semantic concepts described by the text labels and onto which the respective images are mapped as distributions, such as Gaussian distributions, Gaussian mixed distributions, and the like.

4 shows 400 an example of an embedding space that represents semantic concepts described by text labels as distributions. In particular, the example shows 400 a common image-text embedding space 402 , the distributions 404 . 406 . 408 includes. The distributions 404 . 406 . 408 For example, they may represent the semantic concepts of a tree, an apple, or a fruit. Note that there is a certain overlap between the distributions 404 . 406 . 408 which expresses that the semantic concepts represented by the text labels can overlap. In other words, the semantic concepts represented by the textual labels for "tree", "apple" and "fruit" may overlap partly because an apple may correspond to a tree and also to a fruit. Accordingly, semantic concepts become distributions in the common image-text embedding space 402 and not modeled as single points.

The MIE module 114 Wins the common image-text embedding space 402 to create it by modeling each semantic concept described by the text labels. In one example, the MIE module models 114 each of the semantic concepts as Gaussian distribution as follows: t _k ~ N (μ _k, Σ _k) ∀ _k ∈ (1, ..., M)

Here, the term M denotes the number of text labels in a text vocabulary used to train the common image text embedding space 402 is used. The term t _k denotes the k-th text label t in the text vocabulary, the term μ _k denotes a vector of the k-th text label t (for example a text label glove feature, which is determined from a glove model for the text label representation), the term Σ _k denotes an identity matrix that points to the common image text embedding space 402 is initialized, and N (·) denotes a distribution modeling function (for example, a standard Gaussian distribution, although other distributions may be used). In one or more implementations, the MIE module wins 114 the distributions for the common image-text embedding space 402 through a deep neural network. The MIE module 114 may represent a given image I in a set of training images through a Convolutional Neural Network (CNN) and may initialize an image embedding weight for the image corresponding to a loss function, such as a rank order loss. With regard to the term Σ _k , the MIE module wins 114 by minimizing a loss for each image I in the set of training images as follows:

As this equation makes clear, the MIE module is 114 configured to model the positive text labels t _i for a given image I as N (μ _i , F _i ) and to model the negative text labels t _n for the given image as N (μ _n , Σ _n ). In the above equations, m denotes a boundary term.

Although implementations of modeling semantic concepts as distributions in an embedding space have been described primarily with reference to Gaussian distributions, it should be appreciated that the MIE module 114 is configured to model the semantic concepts described by the text labels in an embedding space using a variety of other distributions, without departing from the spirit or scope of the techniques described herein. By way of example and not by way of limitation, the MIE module 114 also model the semantic concepts described by the text labels in an embedding space as Gaussian mixtures, log normal distributions, Pareto distributions, and the like.

By modeling the semantic concepts described by the text labels as distributions rather than individual points, the techniques described herein allow the semantic concepts to overlap. This leads to the creation of an embedding space, which represents the relationships between semantic concepts in more detail than is the case with embedding spaces, which model semantic concepts as single points, for example because many of the semantic concepts overlap. For example, the semantic concepts of a sun and a sunset overlap, as do the semantic concepts of a sun and a sunrise. In any case, the MIE module 114 configured to create embedding spaces that model the semantic concepts described by the textual labels of a textual vocabulary as distributions.

Having described exemplary details of the techniques for modeling semantic concepts in an embedding space as distributions, some exemplary procedures for illustrating additional aspects of the techniques will now be considered.

Exemplary procedures

This section describes exemplary procedures for modeling semantic concepts in an embedding space as distributions in one or more implementations. Aspects of the procedures may be implemented in hardware, firmware or software or even a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and not necessarily those Sequences that are shown to perform the operations by the respective blocks are limited. In at least some implementations, the procedures are performed by a suitably configured device, such as the example computing device 102 from 1 , which is the MIE module 114 served.

5 shows an exemplary procedure 500 in which areas of a training image associated with multiple text labels are mapped to the corresponding text labels in an embedding space to train the embedding space, and label exploitation techniques are applied to a query image to locate text labels in the embedding space that correspond to areas of the query image , A training image associated with multiple text labels is processed to produce a set of image areas corresponding to the respective plurality of text labels (Block 502 ).

For example, suppose that the picture 202 corresponds to a training image and already with the text labels 204 . 206 . 208 . 210 . 212 is linked. In this scenario, the MIE module processes 114 the picture 202 to the image areas 214 . 216 . 218 . 220 according to the text labels to produce such that the image area 214 according to the text label 208 is generated, the image area 216 according to the text label 206 is generated, the image area 218 according to the text label 204 is generated, the image area 220 according to the text label 210 is generated and the image area for displaying the image 202 as a whole according to the text label 212 is produced. As described in detail above, the MIE module generates 114 the image regions by applying a region proposal technique and performing post-processing techniques in the proposed candidate regions. The MIE module 114 also creates the corresponding areas by selecting a best fit area for each of the text labels 204 . 206 . 208 . 210 . 212 for example, based on the distance between positions where the areas are embedded in the embedding space and a position of the text label in the embedding space.

The set of areas is embedded within an embedding space configured to embed both text labels and image areas mapped to the text labels (Block 504 ). Furthermore, the set of areas is embedded at least partially based on positions at which the multiple text labels corresponding to the image areas of the training image are embedded in the embedding space. In one example, the MIE module embeds 114 the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, in the common image-text embedding space 302 one. The MIE module 114 This is at least partly based on positions where the text labels 204 . 206 . 208 . 210 . 212 in the common image-text embedding space 302 are embedded.

As described in detail above, the MIE module calculates 114 Feature vectors for each of the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole corresponds to these in the common image text embedding space 302 for example, by extracting image features with a convolutional neural network (CNN), such as a Fast Region-based CNN (RCNN) technique. In addition, if the MIE module forms 114 the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, in the common image-text embedding space 302 embeds, the MIE module 114 For example, such that images and image areas without text labels are placed on a location in the common image text embedding space 302 can be displayed.

Once trained, the embedding room is useful for finding text labels for images, such as image tagging, multi-text image search (for identifying images as searches), image captioning, and the like more. Label retrieval techniques are applied to a query image to map image areas of the query image to the text labels embedded in the embedding space to locate text labels corresponding to the image areas (Block 506 ).

Now assume that the picture 202 corresponds to the query image and the text labels 204 . 206 . 208 . 210 . 212 not yet with the picture 202 have been linked. In this scenario, at least one label locator technique may be present in the image 202 be applied to the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, regarding the text labels in the common image text embedding space 302 corresponds to depict.

The applied label finding technique can implicate a comparison of the image areas 214 . 216 . 218 . 220 and the image area that corresponds to the image 202 as a whole corresponds with the Image areas residing in the common image text embedding space 302 are embedded. The comparison is performed in one or more implementations by comparing feature vectors that apply to the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, have been extracted with feature vectors of the image areas included in the common image text embedding space 302 embedded, as part of the training. Based on the comparison, the MIE module determines 114 the embedded areas leading to the image areas 214 . 216 . 218 . 220 are similar, and the image area of the image 202 as a whole corresponds. The MIE module 114 assigns the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, to the embedded text labels to which the similar embedded image areas are mapped. That's how the MIE module does it 114 the text labels 204 . 206 . 208 . 210 . 212 in the common image text embedding space 302 for the image areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, locate.

The query image is annotated with the text labels located to describe the content of the query image (block 508 ). For one example, the MIE module comments 114 the picture 202 with the text labels 204 . 206 . 208 . 210 . 212 for example, by incorporating the same with the image 202 as metadata or by otherwise linking it to the image 202 ,

The areas of the query image corresponding to the text labels are presented (block 510 ). In one example, a user request is received to the areas of the image 202 that match the text labels that match the image 202 is commented on, present. The MIE module 114 presents indications (for example, bouling boxes) of one or more of the image areas 214 . 216 . 218 . 220 and the image area, the image 202 as a whole corresponds, for example via a display of the image 202 , It should be clear that the MIE module 114 Present information on image areas in a variety of ways without departing from the spirit or scope of the techniques described herein. By way of example and not by way of limitation, indications of the image areas may be presented along with corresponding text labels, one area may be visually altered to look different (eg, sharp, while other portions of the image are blurry) compared to other portions of the image. and more.

6 shows an exemplary procedure 600 in which a trained embedding space is used to associate multiple text labels with respective areas of an image to be commented. An embedding room is trained in which both images and text are embedded (Block 602 ). Training involves semantically correlating text labels configured to describe semantic concepts that are present in the image content and mapping representative images that have image content that exemplifies the semantic concepts to respective text labels.

In one example, the MIE module trains 114 the common image-text embedding space 302 , For this purpose, the MIE module correlates 114 the text labels of the text vocabulary 306 semantically using, for example, textual data available on the Internet, scalable and lexically distributed representations of words to capture the semantic meaning among the textual labels of the textual vocabulary 306 to win. In one or more implementations, the MIE module takes effect 114 to this end, rely on one or more text modeling architecture techniques, such as the word2vec model, the glove model, and the like. As can further be demonstrated by this example, the MIE module trains 114 the common image-text embedding space 302 by mapping the representative images of the image database 304 on the respective text labels of the text vocabulary 306 in the common image-text embedding space 302 , The MIE module 114 forms in the common image text embedding space 302 the pictures of the picture set 314 on the text label 308 for "tree", forms the images of the image set 316 on the text label 310 for "woman" and forms the images of the image set 318 on the text label 312 for "building".

Once the common image text embedding room 302 trained, it can be used to comment on pictures. An image to be commented is related (Block 604 ). In one example, the picture becomes 202 based. In this example, assume that the image 202 uncommented and not yet with the text labels 204 . 206 . 208 . 210 . 212 is linked.

A set of areas of the image is created (Block 606 ). In one or more implementations, generating implies the use of a scoping technique that suggests areas that can be mapped to corresponding text labels in the embedding space. In one example, the MIE module generates 114 a set of areas for the image 202 that the picture areas 214 . 216 . 218 . 220 and the image area corresponding to the image 202 as a whole, includes. The MIE module 114 generates the set of areas for the image 202 in the manner detailed above. The generated areas are semantically meaningful and can be mapped to an embedded text label in the embedding room.

At least one label discovery technique is used in the set of areas to map the set of areas to corresponding text labels in the embedding space for describing semantic concepts existing in the set of areas (Block 608 ). In one example, the MIE module applies 114 a label locator technique at the set of areas that is responsible for the image 202 has been generated, see block 606 , In particular, the MIE module forms 114 the set of areas to those in the common image-text embedding space 302 embedded text labels that describe the semantic concepts that exist in each of the image areas 214 . 216 . 218 . 220 and the image area, the image 202 as a whole, are present. Shown by the MIE module 114 especially the image area 214 pointing a woman at the text label 208 for "woman", the image area 216 pointing the Eiffel Tower on the text label 206 for "Eiffel Tower", the image area 218 pointing shopping bags on the text label 204 for "bag", the image area 220 pointing building on the text label 210 for "building" and the picture area, the picture 202 as a whole, on the text label 212 for "city scene".

The picture is commented on with at least two of the corresponding text labels (block 610 ). The MIE module 114 comments the picture 202 for example with the text labels 204 . 206 . 208 . 210 . 212 based on the figure in block 608 , As described above, a larger number of image areas than those in FIG 2 shown and created in the common image text embedding space 302 be imaged. As a result, a list of text labels corresponding to the image 202 be generated. The picture 202 however, can be commented on with a selected group of text labels from the list. By way of example and not by way of limitation, the list of text labels may be ranked, as previously described in detail, with the MIE module 114 a predetermined number of the best ranking labels for annotating the image 202 selects.

In addition to using a ranked list, the labels that are selected to annotate an image can also be selected in many other ways. For example, based on a threshold difference of the corresponding area, the labels may become representative images stored in the common image text embedding space 302 are embedded. For this purpose, the MIE module 114 a distance (e.g., a Euclidean distance) between each area generated for an image and embedded representative images in the common image text embedding space 302 for example, by extracting feature vectors of the image areas and the embedded representative images and comparing the feature vectors. The MIE module 114 may then select text labels for which the image area has a difference less than a threshold difference to the embedded representative images. The MIE module 114 Also, text labels may be selected in other ways to annotate an image from the text labels determined according to the areas of the image, without departing from the spirit or scope of the techniques described herein.

7 shows an exemplary procedure 700 in which an embedding space for representing semantic concepts in an embedding space is generated as distributions. An embedding space is created that represents both images and text labels of a text vocabulary (block 702 ). As part of creating the embedding space, distributions are computed to represent semantic concepts. The semantic concepts for which the distributions are generated are described by respective textual labels of a textual vocabulary and can be represented in an image content. In addition, representative images are mapped to the distributions of the embedding space as part of the creation. The respective images, which are mapped as part of the rendering, show image content that exemplifies the corresponding semantic concepts of the distributions.

In one example, that generates MIE module 114 the common image-text embedding space 402 by calculating distributions (e.g., Gaussian distributions, Gaussian mixed distributions, and the like) to represent semantic concepts described by textual labels from a textual vocabulary. The MIE module 114 creates the common image-text embedding space 402 to allow image and text label embedding relative to the distributions representing the semantic concepts. Furthermore, the MIE module trains 114 the common image-text embedding space 402 by embedding representative image areas having image content exemplifying the semantic concepts with respective distributions formed for the semantic concepts.

The embedding space, which represents the semantic concepts as distributions, is used to annotate an image (Block 704 ). In one example, the MIE module uses 114 the common image-text embedding space 402 to comment the picture 202 with text labels representing the content of the image 202 describe. The MIE module 114 uses the common image text embedding space 402 for example, to locate the text labels 204 . 206 . 208 . 210 . 212 and then annotating the image 202 with the labels found.

Having described exemplary procedures in accordance with one or more implementations, an exemplary system and apparatus will now be considered that may be used to implement the various techniques described herein.

Exemplary system and exemplary device

8th generally indicates 800 an exemplary system that includes an exemplary computing device 802 which represents one or more computing systems and / or one or more devices implementing various of the techniques described herein. This is by including the MIE module 114 , which works according to the above description. The computing device 802 For example, a server of a service provider, a device associated with a client (eg, a client device), an on-chip system, and / or any other suitable computing device or system may be included.

The exemplary computing device 802 includes a processing system 804 , one or more computer-readable media 806 and one or more I / O interfaces 808 that are communicatively coupled with each other. Although not shown, the computing device may 802 further include a system bus or other data and command transfer system that couples the various components together. A system bus may include any bus structure or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus employing any of a variety of bus architectures. A variety of other examples are included, such as control and data lines.

The processing system 804 provides functionality to perform one or more operations using hardware. Accordingly, the processing system 804 such that it is hardware elements 810 which may be more configured as processors, functional blocks, and the like. This includes an implementation in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors. The hardware elements 810 are not limited by the materials of which they are formed or the processing mechanisms employed therein. For example, the processors may be formed of a semiconductor / semiconductors and / or transistors (eg, electronic integrated circuits (ICs)). In this context, processor executable instructions may also be electronically executable instructions.

The computer-readable storage medium 806 is shown as having a memory / shelf 812 having. The storage / storage 812 provides storage / storage capacity associated with one or more computer-readable media. The storage / storage component can be included 812 volatile media (such as a random access memory (RAM)) and / or nonvolatile media (such as a read only memory (ROM), a flash memory, optical disks, magnetic disks, and the like). This may include the storage / storage component 812 fixed media (e.g., RAM, ROM, a fixed hard disk drive, and the like) as well as removable media (eg, a flash memory, a removable hard disk drive, an optical disk, and the like). The computer-readable media 806 can be configured in a variety of other ways as described below.

The input / output interface 808 provides / the input / output interfaces 808 provide functionality that allows a user to send commands and information to the computing device 802 and that also allows information to be presented to the user and / or other components or devices using input / output devices. Examples of input devices include a keyboard, a cursor control device (eg, a mouse), a microphone, a scanner, a touch functionality (eg, capacitive or with other sensors configured to detect a physical touch), a camera (e.g. such, which use visible or invisible wavelengths, such as infrared frequencies, to detect movements as gestures that do not imply touch) and the like. Examples of output devices include a display device (eg, a monitor or projector), speakers, a printer, a network card, a tactile response device, and the like. The computing device 802 may be configured in a variety of ways, as described below, to assist user interaction.

Various techniques have been described herein in the general context of software and hardware elements or program modules. In general, such modules include routines, programs, objects, elements, components, data structures, and the like that perform particular tasks or implement particular abstract data types. The terms "module," "functionality," and "component" generally refer to software, firmware, hardware, or a combination thereof as used herein. The features of the techniques described herein are platform independent, meaning that the techniques may be implemented on a variety of off-the-shelf computing platforms having a plurality of processors.

An embodiment of the described modules and techniques may be stored on or transmitted through any form of computer-readable medium. The computer-readable medium may include a variety of media that may be accessed by the computing device 802 can be accessed. By way of example and not by way of limitation, the computer-readable media includes "computer-readable storage media" and "computer-readable signal media."

"Computer-readable storage media" refers to media and / or devices that enable permanent and / or non-temporary storage of information, as opposed to mere signal transmission, to carrier waves or signals as such. Therefore, computer readable storage media does not include signals or signal carrying media as such. The computer readable storage media include hardware, such as volatile and nonvolatile, removable and non-removable media, and / or storage devices for implementation in a method or technology suitable for storage of information, such as computer readable instructions, data structures, program modules, logical elements. Circuits or other data. Examples of computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD or other optical storage, hard drives, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or others A storage device, physical media or an article of manufacture adapted to store the desired information so that a computer can access it.

"Computer readable signal media" refers to a signal bearing medium configured to provide instructions to the computing device hardware 802 for example, to transmit over a network. Signal media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also includes any type of information distribution media. The term "modulated data signal" refers to a signal in which one or more of its characteristics are altered to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or a direct-wired connection, as well as wireless media, such as acoustic, radio frequency-based, infrared, and other wireless media.

As described above, hardware elements 810 and computer-readable media 806 Modules, programmable device logic, and / or fixed device logic implemented in the form of hardware that is employed in some implementations to implement at least some aspects of the techniques described herein, such as when executing one or more instructions. The hardware may include components of an integrated circuit or on-chip system, an application specific integrated circuit (ASIC), field programmable gate array (FPGA), complex programmable logic device (CPLD) and other implementations in silicon or other hardware. In this context, hardware acts as a processing device executing program tasks defined by instructions and / or logic embodied by the hardware, as well as hardware used to store instructions for execution, such as the computer-readable ones described above storage media.

Combinations of the foregoing may also be used to implement various of the techniques described herein. Accordingly, software, hardware or executable modules are implemented as one or more instructions and / or logic that may be in any form of computer-readable storage media and / or by one or more hardware elements 810 are embodied. The computing device 802 may be configured to implement certain instructions and / or functions corresponding to the software and / or hardware modules. Accordingly, the software implementation of a module is that of the computing device 802 is executable to realize at least partially by hardware, for example, using a computer-readable storage medium and / or hardware elements 810 of the processing system 804 , The instructions and / or functions are by one or more manufactured products (for example, one or more computing devices 802 and / or processing systems 804 ) executable / operable to perform techniques, modules, and examples described herein.

The techniques described herein are provided by various configurations of the computing device 802 support and should not limit the specific examples of the techniques described here. This functionality can also be implemented in whole or in part using a distributed system, for example by means of a "cloud". 814 over a platform 816 as described below.

The cloud 814 includes a platform 816 for resources 818 and / or represents such. The platform 816 abstracts an underlying hardware functionality (such as servers) and software resources of the cloud 814 , The resources 818 may include applications and / or data that may be used while computer processing is performed on servers that are hosted by the computing device 802 are removed. The resources 818 may also include services provided over the Internet and / or through a subscriber network, such as a cell-based or Wi-Fi network.

The platform 816 abstracts resources and functions for connecting the computing device 802 with other computing devices. The platform 816 It also provides abstract scaling of resources to provide an appropriate scale level for existing resource demands 818 that over the platform 816 are implemented. Accordingly, in one embodiment with inter-connected devices, the implementation of a functionality described herein is particularly in the system 800 distributed. The functionality is partly on the computing device, for example 802 as well as the platform 816 that the functionality of the cloud 814 abstracted, implemented.

concluding remark

While the invention has been described in language specific to structural features and / or methodological acts, it is to be understood that the invention as defined in the appended claims is not necessarily limited to the particular features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementation of the claimed invention.

Claims (20)

By a computing device ( 102 ; 802 ) implemented method for annotating images ( 202 ; 314 ) with certain text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) to describe the content of the images ( 202 ; 314 ), the method comprising: creating an embedding space ( 302 ; 402 ), both images ( 202 ; 314 ) as well as text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of a text vocabulary, wherein the generating includes: calculating distributions to represent semantic concepts in the embedding space ( 302 ; 402 ), whereby the semantic concepts for which the distributions are calculated are indicated by respective text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of the text vocabulary and can be shown in an image content, and imaging of representative images ( 202 ; 314 ) on the distributions of the embedding space ( 302 ; 402 ), the image represented by the representative images ( 202 ; 314 ) shows corresponding semantic concepts of the distributions exemplarily, using the embedding space ( 302 ; 402 ) determining at least one of the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) for describing at least one semantic concept shown in the image content of an input image ( 202 ; 314 ); and Comment the input image ( 202 ; 314 ) by linking the specific text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) with the input image ( 202 ; 314 ). The method of claim 1, wherein the distributions are calculated as Gaussian distributions to represent the semantic concepts. The method of claim 1, wherein the distributions are calculated as Gaussian mixed distributions to represent the semantic concepts. The method of claim 1, wherein generating the embedding space ( 302 ; 402 ) further includes: determining semantic relationships between the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of the text vocabulary; Position the distributions in the embedding space ( 302 ; 402 ) on the basis of the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) certain semantic relationships; Processing a plurality of training images ( 202 ; 314 ), each training image ( 202 ; 314 ) several text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ), wherein the processing comprises generating sets of image areas ( 214 . 216 . 218 . 220 ), which correspond to the respective several text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ), includes; and selecting the sets of image areas ( 214 . 216 . 218 . 220 ) as representative pictures ( 202 ; 314 ) for mapping to the distributions of the embedding space ( 302 ; 402 ). The method of claim 4, wherein processing the plurality of training images ( 202 ; 314 ) for each training image ( 202 ; 314 ) includes: determining candidate image areas ( 214 . 216 . 218 . 220 ) for a respective set of image areas ( 214 . 216 . 218 . 220 ) of the training image ( 202 ; 314 ); and reducing a number of the determined candidate image areas ( 214 . 216 . 218 . 220 ) using at least one post-processing technique. Method according to claim 5, wherein the candidate image areas ( 214 . 216 . 218 . 220 ) using a geodetic object proposal. The method of claim 5, wherein the at least one post-processing technique comprises enforcing a size criterion by discarding candidate image areas ( 214 . 216 . 218 . 220 ) is implied with less than a threshold size. The method of claim 5, wherein the at least one post-processing technique comprises enforcing an aspect ratio by discarding candidate image areas ( 214 . 216 . 218 . 220 ) with aspect ratios outside predefined allowable aspect ratios. The method of claim 5, wherein the at least one post-processing technique assigns a single candidate image area to each of the respective plurality of text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of the training image ( 202 ; 314 ) based on a single label embedding model. The method of claim 1, wherein determining the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) for describing the at least one semantic concept shown in the image content of the input image ( 202 ; 314 ) includes: determining a set of semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ); Discarding at least one of the semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ) using a post-processing technique; and mapping remaining semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ) on the distributions of the embedding space ( 302 ; 402 ) for determining the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ). The method of claim 1, wherein determining the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) calculating distances in the embedding space ( 302 ; 402 ) between embedding of semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ) and the distributions. The method of claim 11, wherein the distances are calculated using vectors representing respective semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ), wherein the vectors from the semantically meaningful image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ) with a convolutional neural network (CNN). The method of claim 11, further comprising selecting the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) for linking to the input image ( 202 ; 314 ) based on the distances. The method of claim 1, further comprising presenting information about image areas ( 214 . 216 . 218 . 220 ) of the input image ( 202 ; 314 ) containing at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) correspond. System for commenting images ( 202 ; 314 ) with certain text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) to describe the content of the images ( 202 ; 314 ), the system comprising: one or more processors; and computer readable storage media having stored thereon instructions executable by the one or more processors to perform operations comprising: processing a training image ( 202 ; 314 ) with several text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ), wherein the processing comprises generating a set of image areas ( 214 . 216 . 218 . 220 ), which correspond to the respective several text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ), includes; Embedding the set of image areas ( 214 . 216 . 218 . 220 ) within an embedding room ( 302 ; 402 ), which presents semantic concepts as distributions, where the presented semantic concepts are represented by text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of a text vocabulary and can be shown in the image content, the set of image regions ( 214 . 216 . 218 . 220 ) with the distributions corresponding to those in the image content of the set of image areas ( 214 . 216 . 218 . 220 ), is embedded; Determining the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ), the semantic concepts of a query image ( 202 ; 314 ) by mapping image areas ( 214 . 216 . 218 . 220 ) of the query image ( 202 ; 314 ) on the distributions of the embedding space ( 302 ; 402 ); and commenting on the query image ( 202 ; 314 ) with at least two of the specific text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ). The system of claim 15, further comprising semantically correlating the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of the text vocabulary before embedding the image areas ( 214 . 216 . 218 . 220 ) a plurality of training images ( 202 ; 314 ) in the embedding room ( 302 ; 402 ). The system of claim 16, wherein the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of the text vocabulary are semantically correlated based on the glove model. One or more computer-readable storage media comprising instructions stored thereon responsive to execution by the computing device (10). 102 ; 802 ) Perform operations that include: preserving an image ( 202 ; 314 ) with at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) for describing at least one respective semantic concept that is present in the image content of the image ( 202 ; 314 ) is commented on, the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) for the picture ( 202 ; 314 ) using an embedding room ( 302 ; 402 ), which presents semantic concepts as distributions, rather than representing the semantic concepts as individual points, with the semantic concepts represented by the text labels ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) of a text vocabulary and can be present in the image content. One or more computer-readable storage media according to claim 18, wherein said image ( 202 ; 314 ) with the at least one text label ( 110 ; 204 . 206 . 208 . 210 . 212 ; 308 . 310 . 312 ) as part of the indexing of the image ( 202 ; 314 ) is commented on for a search. One or more computer-readable storage media according to claim 18, wherein the distributions representing the semantic concepts are at least one of Gaussian distributions and Gaussian mixed distributions.

Images