DE102016010910A1 - Structured modeling and extraction of knowledge from images - Google Patents

Abstract

Techniques and systems for modeling and extracting knowledge from images are described. A digital media environment is configured to learn and use a model to compute a descriptive summary of an input image automatically and without user intervention. Training data is obtained to learn a model using machine learning to create a structured image representation that serves as the descriptive summary of an input image. The images and associated text are processed to extract structured semantic knowledge from the text, which is then associated with the images. The structured semantic knowledge is processed along with corresponding images to train a model using machine learning so that the model describes a relationship between textual features in the structured semantic knowledge. After the model has been learned, the model can be used to process input images to produce a structured image representation of the image.

Description

Related applications

This application claims the benefit of US Provisional Application No. 62 / 254,143, filed on Nov. 11, 2015, entitled "Structured Knowledge Modeling and Extraction from Images," the disclosure of which is incorporated herein by reference in its entirety.

background

Image searches often involve the challenge of matching text in a search query with text associated with the image, such as tags, and so forth. For example, a creative professional may capture an image and associate text-containing tags used to locate the image. On the other hand, a user attempting to locate the image in an image search inputs one or more keywords. Accordingly, this requires that the working creative and the user come to an understanding of how to describe the image with text so that the user can find the image of the working creative that can make the image available to the user. Conventional tag and keyword searching techniques may be prone to errors, misunderstandings and different interpretations due to the need for the creative and the user to come to a common understanding to locate the images.

Further, conventional image search techniques do not support high-precision semantic image searches due to limitations of conventional tagging and image searching. This is because conventional techniques only associate tags with the pictures, but neither define relationships between the tags nor with the pictures themselves. As such, conventional search techniques can not provide accurate search results for complex queries, such as "a man feeding a baby in a high chair while the baby is holding a toy." Accordingly, these conventional search techniques force users to navigate through tens, hundreds, and even thousands of images, often using multiple search queries to find a desired image, caused by forcing the user as part of the search in conventional techniques to gain an understanding of how the working creative has described the picture to find it.

Summary

Techniques and systems are described for extracting and modeling structured knowledge from images. In one or more implementations, a digital media environment is configured to learn and use a model to compute a descriptive summary of an input image automatically and without user intervention. Training data (e.g., image and unstructured text such as image captions) are first obtained to train a model using machine learning to produce a structured image representation that serves as the descriptive summary of an input image.

The images and associated text are then processed to extract structured semantic knowledge from the text, which is then associated with the images. Structured semantic knowledge can take on a variety of forms, such as tuple <subject, attribute> and tuple <subject, predicate, object>, which act as an expression linking the subject to the object via the predicate. This may involve an association with the image as a whole and / or with objects in the image, by a process called "localization".

The structured semantic knowledge is then processed with corresponding images to train a model using machine learning so that the model establishes a relationship between textual features within the structured semantic knowledge (e.g., subjects and objects) and image features of images, for example Parts of the image defined in bounding rectangles containing the subjects or objects.

Once the model is learned, the model can then be used to process input images to produce a structured image representation of the image. The structured image representation may include text that is structured in a manner that describes relationships between objects in the image and the image itself. The structured image representation can be used to create a Support for a variety of functionalities, including image browsing, automatic generation of image captions and metadata, object tags and so on.

This summary introduces a selection of concepts in a simplified manner, which are further described below in the detailed description. As such, this summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Brief description of the drawings

The detailed description will be described with reference to the accompanying drawings. In the figures, the leftmost digit (s) identify the figure in which the reference number appears for the first time. The use of the same reference numeral in different instances in the description and figures may indicate similar or identical elements. Entities depicted in the figures may be representative of one or more entities, and therefore, in the discussion may be synonymously referenced to singular forms or plural forms of the entities.

1 FIG. 12 is an illustration of an environment in an exemplary implementation configured to apply techniques for extracting knowledge from images as described herein.

2 FIG. 11 shows another example of an image from which knowledge is extracted using a knowledge extraction system of FIG 1 ,

3 shows a system using the knowledge extraction system of 1 in more detail shows.

4 FIG. 12 shows an exemplary implementation that includes an extraction module of FIG 3 in more detail shows.

5 shows an exemplary system in which an extraction module of 4 is shown as containing a localization functionality as part of the knowledge extraction.

6 shows an example of the localization of structured semantic knowledge to parts of images.

7 FIG. 12 shows an exemplary implementation that includes a model training module of FIG 3 in more detail than containing a machine learning module to model a relationship between the structured semantic knowledge and images.

8th FIG. 12 shows an exemplary implementation showing the training of a model using a two-module machine learning system.

9 FIG. 10 is a flowchart illustrating a procedure in an exemplary implementation in which a digital media environment is used to automatically extract knowledge from an input image without user intervention.

10 FIG. 10 is a flowchart illustrating a procedure in an exemplary implementation in which a digital media environment is used to extract knowledge and locate textual features into image features of an input image.

11 shows a system for structured fact picture embedding.

12 shows Model 1 and Model 2 as part of machine learning.

13 FIG. 12 shows an example system including various components of an exemplary device that may be implemented as a type of computing device as described and / or used with reference to FIG 1 - 12 to implement embodiments of the techniques described herein.

Detailed description

Overview

Techniques and systems are described which support knowledge extraction from an image to produce a descriptive summary of the image, which can then be used to support image searching, automatic creation of image captions and metadata for the image, and a variety of other uses. For example, the descriptive summary may describe qualities of the image as a whole, as well as attributes, objects and interaction of the objects with each other in the image as further described below. Accordingly, although examples will be described below that include image searches, these techniques are equally applicable to a variety of other examples, such as automated structured image tagging, image captioning, and so forth.

Training data is first obtained to train a model using machine learning to produce a structured image representation. Techniques are described herein in which training data is obtained which images and associated text (e.g., captions of the images containing any type of text configuration describing a scene captured by the image) that are easily obtained from a variety of sources can be. The images and associated text are then processed automatically and without user intervention to extract semantic knowledge structured from the text which is then associated with the images. This may involve an association with the image as a whole and / or with objects in the image through a process referred to hereinafter as "localization". The use of this training data differs from conventional crowd-sourcing techniques, where people manually label images, which can be expensive, error-prone, and inefficient.

In one example, structured semantic knowledge is extracted from the text using natural language processing. Structured semantic knowledge can take on a variety of forms, such as tuple <subject, attribute> and <subject, predicate, object>, which act as a statement linking the subject to the object via the predicate. The structured semantic knowledge is then processed with the corresponding images to train a model using machine learning so that the model describes a relationship between textual features in the structured semantic knowledge (e.g., subjects and objects) and image features of images, for example Example Parts of the image defined in bounding rectangles containing the subjects or objects. In one example, the model is a common probabilistic model that is built without requiring a reduction of a large vocabulary of individual words to a small predefined set of concepts, and as such, the model can directly address this large vocabulary, using conventional techniques not possible.

For example, localization techniques may be used so that the structured semantic knowledge is mapped to the corresponding object in an image. For example, a tuple <baby, holds, toy> can explicitly map the subject "baby" in an image using the predicate "holds" to the object "toy" in the image and therefore provides a structure to describe what in the picture "going on". This is not possible in conventional unstructured tagging techniques, which techniques were not explicit in that correspondence between a particular object in the image and the tag was not possible, so that when multiple objects were included in the image and no distinction was made between them Objects, for example, several babies, was made. The structured knowledge provided by the techniques described herein can therefore be used in a manner that can be searched by a computing device.

For example, when looking for images of a "red flower", a conventional bag-of-words approach looks at "red" and "flower" separately, which may return images of flowers that are not red but they are red elsewhere in the picture. However, the use of the techniques described herein knows that a user searches for the <flower, red> concept of a search query structure, which is then used to find images having a corresponding structure. In this way, the model may have increased accuracy over techniques based on a description of the image as a whole, as further described below with reference to FIG 5 and 6 ,

This mapping can further exploit a common vector space that punctuates differences so that similar semantic concepts in this space are close to each other. This can be done, for example, for feature vectors for text so that "curvy road" and "winding road" are relatively close to each other in the vector space. Similar techniques can be used to favor similar concepts for image vectors as well as to match the image and text vectors. A variety of machine learning techniques can be used to train the model to perform this mapping. In such an example, a two-column deep network is used to determine the correlation between the structured semantic information and an image or part of an image, for example a bounding rectangle, an example of which is shown in FIG 8th shown is to learn.

Once the model has been learned, the model can then be used to process input images to produce a structured image representation of the image by calculating a confidence value to describe which text best suits the image. For example, the model may iterate over bounding rectangles of parts of the image to determine which structured text (for example, <flower, red>) is likely to describe the part of the image, such as objects, attributes, and relationships between them, by computing probabilities (that is, the confidence values) that the structured text describes a similar concept as the image features in the image. In this way, the structured image representation provides a descriptive summary of the image that uses structured text to describe the images and parts of the image. The structured image representation can therefore be calculated for an image to contain text structured in a manner that includes relationships between objects in the image (for example flower), attributes of the object (for example red), relationships between them (for example < Flower, red> <baby, holds, toy>) and the picture itself describes as described above. The structured image representation can be used to support a variety of functionalities, including image browsing, automatic creation of image captions and metadata, automated object tagging, and so forth. Further discussion of these and other examples is included in the following sections.

In the following discussion, an exemplary environment that can use the knowledge extraction techniques described herein will first be described. Thereafter, exemplary procedures that may be performed in the example environment and other environments will be described. Accordingly, executing the example procedures is not limited to the example environment, and the example environment is not limited to performing the example procedures.

Exemplary environment

1 is a representation of an environment 100 in an example implementation arranged to apply knowledge extraction techniques described herein. The illustrated environment 100 includes a computing device 102 that can be configured in a variety of ways.

The computing device 102 For example, as a desk top computer, a laptop computer, a mobile device (adopting, for example, a hand-held configuration, such as a desk or a mobile phone as illustrated), portable devices, and so on. Therefore, the computing device 102 from full resource devices with significant memory resources and processor resources (e.g., PCs, game consoles) to a low resource device with limited memory resources and / or processor resources (e.g., mobile devices). In addition, even if only a single computing device 102 is shown, the computing device 102 be representative of a variety of different devices, such as multiple servers used by a company to perform "cloud-over" operations, as further described with reference to FIG 13 ,

The computing device 102 is a knowledge extraction system 104. comprising, which is representative of a functionality, from an image 108 a structured picture representation 106 to form the picture 108 summarized in a descriptive way. The structured image representation 106 can be used to support a variety of functionalities, such as from an image search engine 110 is to use a database 112 of pictures 114 search based on corresponding structured image representations. As previously described, other uses of the structured image representation are also 106 such as the automatic creation of image captions and metadata for images such as an image captioning system 118 represents. Although the knowledge extraction system 104. and the image search engine 110 and the database 112 than using the computing device 102 implemented, this functionality can also be further split "across the cloud" over a network 116 as further described with respect to 13 ,

The structured image representation 106 provides a set of concepts with a structure that describes a relationship between entities contained in the concepts. This allows the structured image representation as an intermediary representation of the image 108 using text to not only describe "what's included" in the image 108 but also a relationship of entities and concepts among themselves, in the picture 108 are included. This can be used to support a higher degree of semantic precision in an image search that is not possible using conventional techniques based on unstructured tags.

For example, a high-precision semantic image search involves finding images with the specific content requested in a textual search query. For example, a user may enter a search query into a rendering service for a "man feeding a baby in a high chair while the baby is holding a toy" to retrieve an image of interest that is available for licensing. However, traditional techniques based on unstructured tags are unable to accurately satisfy this query. Due to this lack of structure, images provided in practice by conventional image browsing satisfy some but not all of the elements in the query, such as a man feeding a baby, but the baby does not hold a toy, a baby in a high chair, but it there is no man in the picture, a picture of a woman feeding a baby holding a toy and so on.

A structured picture representation 106 however, provides an explicit representation of what an image is 108 is known. This supports an ability to determine which concepts are missing from a search query in a searched database image and therefore improves the accuracy of the search results. Accordingly, a measure of similarity between the search query and a picture 114 in a database 112 consider which and how many concepts are missing. Moreover, if there is an image that comes close to satisfying the query but lacks a concept, techniques can be used to synthesize a new image using the approximate image and content from another image containing the image contains missing concept, as further described below.

Das Bildbeschriftungserzeugungssystem 118 ist konfiguriert, dieses extrahierte Wissen zu verwenden, um eine Bildbeschriftung wie folgt zu erzeugen:
„Ein Mann füttert ein lächelndes Baby, während das Baby ein Spielzeug hält. Das Baby sitzt in einem Hochstuhl. Der Mann ist auch fröhlich. Es ist wahrscheinlich ein Vater, der seinen Sohn füttert. Der Vater und sein Sohn haben zusammen Spaß, während die Mutter weg ist”.It is an example of using the structured image representation 106 considered in which the extracted knowledge of the image 108 the following contains:

The image captioning system 118 is configured to use this extracted knowledge to generate an image label as follows:
"A man is feeding a smiling baby while the baby is holding a toy. The baby is sitting in a high chair. The man is also happy. It's probably a father feeding his son. The father and his son have fun together while the mother is away ".

The explicit representation of knowledge of structured image representation 106 allows a description in several sentences of the scene of the picture 108 as an image caption in this example that is formed automatically and without user intervention. The first two sentences are a simple inclusion of the concepts <man, feeding, baby>, <baby, holding, toy> and <baby, sitting in, high chair>. The third sentence involves reasoning based on the concepts of <man, smiling> and <baby, smiling> to go through the image captioning system 118 to conclude that the man is cheerful and to add the "too" because both the baby and the man are smiling. The fourth sentence also uses a conclusion based on the extracted concept that the baby is wearing blue to conclude that the baby is a boy.

The image captioning system 118 can also use external statistical knowledge, for example, that in most cases when a man is feeding a male baby, it is a father feeding his son. The above fourth sentence produced is tempered with "It's likely ..." because statistics can indicate a reasonable degree of uncertainty in this conclusion that the baby is a boy because the baby is wearing blue clothes. Because the structured image representation 106 be used In order to extract all relevant information about the scene, the absence of information may also be used as part of conclusions drawn by the image captioning system 118 be executed. In this case, the structured image representation mentions 106 no woman as in the picture 108 present. Therefore, the image capturing generating system 118 Conclude that the "mother is gone", and combined with the concepts that the man and the baby smile, form the final sentence "The father and his son have fun together while the mother is away".

It should be noted that an image capturing system 118 can avoid using some of the extracted information. In this case, the picture caption does not mention that the man is sitting at the table because the image captioning system 118 has rated the concept as uninteresting or unimportant for describing the scene, or that it is likely to be inferred from another concept, such as the baby sitting in a high chair. This conclusion is made possible by using the structured image representation 106 as a set of structured knowledge that acts as a descriptive summary of the image 106 using text acts.

The structured image representation 106 can also use Part-of-Speech (POS) tags, such as a single noun, adjective, adverb, and so on, for the extracted subjects, predicates, actions, attributes, and objects. The part-of-speech tags may be used as part of the reasoning as described above, as well as for position fills in a grammatical-based image captioning approach, and to ensure that a valid sentence is generated, as further described below.

In addition, the explicit extraction of knowledge from images allows 108 at the level of the objects in the picture 108 and corresponding attributes and interactions further conclusions about scene properties at middle and higher levels. The inferences that the baby is a boy, the man is happy and that father and son have fun while the mother is gone, are examples.

Das Vorhandensein eines Fußballs zeigt an, dass die Leute Fußball spielen, was weiter durch das Wissen gestützt wird, dass einer der Leute den Fußball tritt. Dass es nur zwei Farben von Hemden gibt, zeigt an, dass zwei Mannschaften ein Spiel spielen. Dies wird durch das Wissen gestützt, dass eine Person in Rot tatsächlich eine Person in blau verfolgt, welche den Ball tritt, und dass andere Leute auf einem Feld rennen. Aus diesem extrahierten Wissen auf Objektebene können Szenenebeneneigenschaften von dem Bildbeschriftungserzeugungssystem 118 mit verbesserten Objektebenenbeschreibungen abgeleitet werden, wie etwa „Ein Fußballspiel zwischen einer Mannschaft in rot und einer Mannschaft in blau”. 2 shows another example of an image 200 , In this example, the structured image representation 106 have the following knowledge, that from the picture 200 extracted is:

The presence of a football indicates that people are playing football, which is further supported by the knowledge that one of the people is playing football. That there are only two colors of shirts indicates that two teams play a game. This is supported by the knowledge that a person in red is actually following a person in blue kicking the ball and that other people are running in a field. From this extracted object level knowledge, scene level properties can be obtained from the image captioning system 118 derived with improved object-level descriptions, such as "A football game between a team in red and a team in blue".

Further closing and reasoning about scenes and their constituent objects and actions can also be achieved by building a knowledge base about the content of images, where the knowledge base is then used by a conclusion work. The establishment of a knowledge base may take as input, for example, structured knowledge describing images such as <subject, attribute, ->, <subject, predicate, object>, <subject, -, ->, <-, action, -> , Input data for building the knowledge base can be taken from existing image caption databases and from image captions and surrounding text in documents. The ability of the techniques described herein to extract such knowledge from any image allows the image knowledge base to capture much more data from non-captioned images and untagged images, as is the case with most images. The image-knowledge base and the corresponding inference work can make conclusions, such as those necessary in the above example of the man feeding the baby. The Image knowledge base can also provide the statistics to support the probabilistic reasoning used in the example that concludes that the man is probably the baby's father. If the example contained an attribute such as "old man," then a more likely conclusion might be that the man is probably the baby's grandfather.

After examples of an environment have been described in which a structured image representation 106 is used to take pictures 114 In the following, a more detailed discussion of operations of the knowledge extraction system can be summarized in a descriptive way 104. to create and use a model as part of the extraction of knowledge from images.

3 shows a system 300 an exemplary implementation of the knowledge extraction system 104. from 1 in more detail shows. In this example, the knowledge extraction system uses 104. an approach of machine learning to the structured image representation 106 to create. Accordingly, from the knowledge extraction system 110 first training data 302 which are to be used to train the model that will be used afterwards for the structured image representation 106 to build. Conventional techniques used to train models in similar scenarios (e.g., image understanding problems) are based on users manually tagging the images to the training data 302 which can be inefficient, expensive, time consuming and error prone. However, in the techniques described herein, the model is trained using techniques that can be performed automatically and without user intervention.

In the illustrated example, the training data includes 302 images 304 and associated text 306 such as picture captions or metadata associated with the pictures 304 are associated. An extraction module 308 is then used to structured semantic knowledge 310 For example, "<Subject, Attribute>Image" and "<Subject, Predicate, Object>Image" using natural language processing as described with reference to FIG 4 further described. Extraction can also be a localization of structured semantic knowledge 310 to objects in the image as related to 5 and 6 further described.

The pictures 304 and the corresponding structured semantic knowledge 310 then go to a model training module 312 to hand over. The model training module 312 is as a machine learning module 314 comprising, which is representative of functionality, to use machine learning (e.g., neural networks, convolutional neural networks, and so on) to model 316 using the pictures 304 and structured semantic knowledge 310 to train. The model 316 is trained, a relationship between textual features in the structured semantic knowledge 310 are included, and define image features in the images as related to 7 further described.

The model 316 is then from a structured logic determination module 318 used to create a structured image representation 106 for an input image 108 to create. The structured image representation 106 may, for example, include text that is structured in such a way that concepts of the image 108 to define, even in cases where the picture 108 has no text. Rather, the model can 316 used this text as part of the structured image representation 106 which is then generated by the structured image representation usage module 320 is used to control, regulate and / or control a variety of functionality automatically and without user intervention, such as image browsing, image captioning and metadata generation and so on. Having exemplary modules and functionalities of the knowledge extraction system 110 Generally, the following provides a description of these modules in more detail.

4 shows an exemplary implementation 400 that the extraction module 308 from 3 in more detail shows. The extraction module 308 contains a module 402 natural language processing representative of a functionality of using natural language processing (NLP) for semantic knowledge extraction from freeform (i.e., unstructured) text 306 who with pictures 304 in the training data 302 is associated. Such free-form descriptions are readily available in existing image captioning databases and in documents with images, such as web pages and PDF documents, and therefore the module may 402 natural language processing takes advantage of this availability, which is not possible using conventional manual techniques. However, manual techniques may also be used where a worker captions texts 306 for pictures 304 created the pictures 304 to describe.

The structured semantic knowledge 310 can be configured in a variety of ways, as previously described, such as tuple "<Subject, Attribute>, Image" 406 and / or "<subject, predicate, object>, image" 408 , Examples of image captions and tuples of structured knowledge, such as those provided by the extraction module 308 "A boy strokes a dog while watching TV", which is then extracted as "<boy, caresses, dog>, <boy, sees, TV>". In another example, a caption "A brown horse eats grass in a large, green field" is then extracted as "<horse, brown>, <field, green>, <horse, eats, grass>, <horse, in, field >".

A variety of tuple extraction solutions may be available from the module 402 used for processing natural language. In addition, in some cases, a variety of tuple extraction techniques can be applied to the same image caption, and consensus among the techniques used to correct errors in tuples, remove bad tuples and identify high confidence tuples or assign confidences to tuples. A similar technique can be used in which an extraction technique is used to collect tuple extraction on a set of image captions for the same image, and to consensus to correct for errors in tuples, remove bad tuples, and high tuples Identifying confidence or assigning confidences to tuples. These data are readily available from existing databases, as images often have multiple image captions. Additionally, crowdsourcing input can also be used to confirm good tuples and remove bad tuples.

In one or more implementations, abstract meaning representation (AMR) techniques are used by the module 402 used to process natural language to aid tuple extraction. AMR aims to obtain a deeper semantic understanding from freeform text. Although it does not explicitly extract knowledge tuples of the form <subject, attribute> or <subject, predicate, object>, a tuple representation can be extracted from an AMR output. In addition, knowledge tuples can be extracted from a scene graph (for example, a Stanford scene graph data set), which is a type of image representation for capturing object attributes and relationships for use in semantic image retrieval.

5 shows an exemplary system 500 in which the extraction module 308 from 4 is shown as containing a localization functionality as part of the knowledge extraction. In addition to the extraction of structured semantic knowledge 310 to describe a picture as a whole as part of the training data 302 , that can be structured semantic knowledge 310 be localized in an image to promote efficient and correct machine learning.

For example, if there is a complex scene, such as a man walking a dog, then this may be structured semantic knowledge 310 be configured as "<man, walk, dog>, image data", whereby the image data refers to a part of the image 304 which contains the man who guides the dog Gassi, which is hereinafter referred to as a limiting rectangle 504 referred to as. Therefore, tuples of structured semantic knowledge can become 310 refer to parts in the image, examples of which are represented as "<subject, attribute>, part" 506 and "<subject, predicate, object>, part" 508 ,

Accordingly, this may promote accuracy in training and subsequent use for images having multiple entities and corresponding actions. For example, if an entirety of an image provided with an image caption containing multiple concepts, for example, a woman jogging or a boy climbing a tree, then any machine learning that is performed is confronted with it to determine which part of the picture actually correlates with <man, walk, dog>. All the more the structured semantic knowledge 310 is located, the easier it will be through the model training module 312 to adapt a high quality model that correlates images and structured text. The problem of associating parts of a textual description with parts of an image is also called grounding.

The grounding and localization module 502 can use a variety of techniques to perform the localization. In one example, object recognition and classification modules configured to recognize particular objects and / or to classify objects are used to portions of images 304 to process. An area CNN (convolutional neural network) or a semantic segmentation technique may also be used to locate objects in an image.

In another example, tuples are structured semantic knowledge 310 how <subject, attribute> and <subject, predicate, object> and localized objects identified by calculate how many class occurrences have been localized for the subject and object classes, as further described below. This may also include identifying subjects or objects that indicate that the tuple describes an entire scene, in which case the entire training image 304 with the tuple of structured semantic knowledge 310 is associated. An external list of scene types, such as bathrooms, is used for this purpose.

Before the grounding and localization module 502 If the bounding rectangles for an object class called in the subject or object of a tuple can be looked up, the text used for the subject or object is mapped to a predefined subset of database objects, since bounding rectangles are typically according to those class specifiers get saved. The mapping problem can be solved, for example, from the subject or object text "guy" to a predefined class such as "man" using a hierarchy to perform the mapping.

Once a lot of limiting rectangles 504 in a picture 304 was obtained for the subject and object classes in a triple <subject, predicate, object> or bounding rectangle 504 for a doublet <subject, attribute>, then from the grounding and localization module 502 Rules and heuristics are applied to a tuple of structured semantic knowledge 310 in the training picture 304 to locate. In a first such example, for a tuple <subject, attribute>, if there is only a single occurrence of a subject class in the image 304 gives (for example, only one car), the tuple associated with the single bounding rectangle for that tuple, as the bounding rectangle 504 contains the subject and the attribute describes the subject in that rectangle, for example, "<car glossy>".

For a tuple <subject, predicate, object> with only a single occurrence of the subject class and an occurrence of the object class, the tuple is associated with the smallest rectangular image area covering the smallest rectangular image area, the bounding rectangle for the subject and the bounding rectangle covering the object, that is, the bounding rectangle of the two bounding rectangles. For example, if there is a single person and a single dog in the picture, then <lead person, walk dog, dog> will be located to the rectangles bounding the person and the dog. This probably includes the leash that connects the person and the dog. In general, the tacit assumption here is that the predicate that relates the subject and the object is visible near the subject and the object.

For a tuple <subject, predicate, object> having a single subject and a single object ("automobile" not "automobiles") and more than one occurrence of either the subject class or the object class, the following is determined. If a next pair of bounding rectangles 504 with one of the subject class and one of the object class within a threshold distance, then that tuple is associated with the bounding rectangle of the next pair of bounding rectangles. The assumption here is that the relationship between a subject and an object can be visually well located. The distribution of the distances between each of the pairs can also be used to determine if there is any uncertainty in this selection because of a second pair or third pair that also has a small distance.

The above heuristics give examples of types of information considered in the localization. Other techniques may also be used to assist in locating that of the grounding and locating module 502 is performed. An example of this is a text semantic module 510 which is representative of a functionality to use text understanding to aid in grounding subjects and objects in the image. In one example, positional attributes associated with a subject are used to select the correct bounding rectangle for the subject or to narrow down the selection. For example, if there are multiple automobiles in a scene, but the caption says "A child is sitting on the hood of the leftmost automobile," then the text semantics module can 510 helping to select the limiting rectangle with the smallest horizontal coordinate to earth as the leftmost automobile in this caption and in the tuple <child sitting on, automobile> extracted from it. Instead of using the bounding rectangle of all automotive bounding rectangles in the example above, only the bounding rectangle of the anchored automobile alone or the subset of automobiles satisfying the leftmost criterion may be used. This determination can be generalized to other criteria that can be measured, such as color.

Grounding a tuple reduces the ground and localization module 502 First, a set of bounding rectangles for the subject and the object, using their attributes, around bounding rectangles 504 filter out those attributes that do not. Such attributes include location, color, and proximity to other identifiable areas, for example, for "the automobile on the grass," the grass area may be recognized using a semantic segmentation algorithm.

Relative position information is also used to select the correct pair of subject class and object class bounding rectangles for a positional relationship. For example, if the caption is "A baby is sitting on a table," then the baby and the table are grounded to rectangles in the image, with the baby's rectangle over the rectangle of the table. As such, this uniquely identifies the image area to be associated with this tuple when there are multiple babies and / or multiple tables in the scene.

For a tuple <subject, predicate, object> in which the subject and the object in the image are grounded, the tuple has a smallest rectangular image area covering the bounding rectangle for the subject and the bounding rectangle for the object. A variety of other examples are also contemplated, such as adding a set of context to the bounding rectangles by trapping a larger area than would otherwise be contained within a "narrow" bounding rectangle.

6 shows an exemplary implementation 600 the localization between parts of an image 108 and structured semantic knowledge 310 , As shown, contains a bounding rectangle 602 for "<man, sitting on, chair>" the man and the chair. A limiting rectangle 604 for "<husband, feeds, baby>" contains both the man and the baby. A limiting rectangle 606 for "<baby, holds, toy>" contains the baby and the toy. After the extraction of structured semantic knowledge 310 In the following, the use of this extracted, structured semantic knowledge will be described 310 through the model training module 312 to train a model 316 discussed.

7 shows an exemplary implementation 700 which the model training module 312 in more detail than a machine learning module 314 to model a relationship between the structured semantic knowledge 310 that from the test 306 was extracted, and the pictures 304 , used. In this example, the machine learning module is 314 configured a relationship 702 between text features 704 of structured semantic knowledge 310 with picture features of the picture 304 the training data 302 to model the model 316 to train.

The model 316 can be formed, for example, as a multivariate model, with a "P (<Subject, Attribute>, Image I), P (<Subject, Predicate, Object>, Image I)". The model 316 is constructed in this example to output a probability that the image "I" and the structured text <subject, attribute> or <subject, predicate, object> represent the same concept of the real world visually and textually. The model 316 in this example, is configured to generalize well to unseen or infrequent combinations of subjects, attributes, predicates, and objects, and does not require an explicit reduction of a large vocabulary of individual words to a small, predefined set of concepts through the use of the identification latent concepts and matching of this structure, as further described below.

The model 316 After being trained, it is configured to locate images based on structured text by calculating probabilities that images correspond to the structured text. For example, a text-based image search involves associating a text query (for example, represented as a set of structured knowledge using a natural language tuple extraction technique) into an image. This is supported by a common model, as further described with respect to 8th by traversing and checking a loop over pictures "I" which provides a high probability "P (structured text <S, P, O>, picture I)" for a given concept <S, P, O>. Knowledge extraction / tagging is assisted by looping through and examining possible concepts <S, P, O>, which has a high probability of "P (structured text <S, P, O>, image I)" for a given image or image provides a given image portion "I".

For making the model 316 through the model training module 312 there are two parts. The first is a feature representation for the structured text "<S, P, O>", "<S, A, ->";"<S, -, ->" (where "-" indicates an unused position to represent all concepts as triples) and for images. The second part is to correlate the feature representation of the structured text with image features, for example the text feature "t" 704 and the image feature "x: P (t, x)" 706 to correlate. For example, a relationship 702 of text features (for example, <man, holds, ball>) with image features 706 of a picture showing a ball being held by a man. These two parts are described in greater detail below.

The tuples "<S, P, O>" and "<S, A>" of the structured semantic knowledge 310 are configured such that similar concepts of structured knowledge have nearby and related representations, for example, as vectors in a vector space. This supports the generalization and use of a large vocabulary. For example, text feature 704 Representations of "<road, curvy>" and "<road, writhing>" are configured to be similar, and the representations between "<dog, walk-in>" and "<person, walk" are interrelated Relationship through the split action of gassi walking or gassi guiding. This can be done so that similar words in the space are close to each other and the vector space captures some relationships between words. For example vec ("man") + (vec ("queen") - vec ("woman")) = vec ("king").

The model training module 312 can also be configured to build on semantic vector representations of individual words to develop a vector representation of a knowledge tuple that captures the relationship between two concepts "<S1, P1, O1>" and "<S2, P2, O2>". More specifically, a feature vector is formed for a triple "<S, P, O>" as a function of single-word representations "vec (S)", "vec (P)" and "vec (O)". The "vec (<S, P, O>)" is formed as a concatenation of the individual word vectors "vec (<S, P, O>) = [vec (S) vec (P) vec (O)]".

If an "<S, P, O>" element is missing, such as the object "O", if a "<subject, attribute>" is represented, or both a predicate "P" and an object "O" are missing, then a "<subject>" is represented, the corresponding vector locations are or are filled with zeros. Therefore, the vector representation for a single subject lies along the "S" axis in "S, P, O" space. Visual attributes may be addressed as modifiers for a plain subject moving the representation of "<S, P>" to the "SP" level of "S, P, O" space. Another option involves summing the vector representations of the individual words.

For a composite "S" or "P" or "O", the vector representation for each word in the phrase is averaged to place a single vector in a target location in a "[vec (S) vec (P) vec (O)] "To use representation. For example, "vec (" runs towards ") is" equal to "0.5 * (vec (" runs ") + vec (" toward "))". An average of unequal weights may also be used if some words in the phrase carry more meaning than others. In one implementation, a semantic representation (for example, vector or probability distribution) for compound phrases such as "run towards" or "run away from" is learned directly by treating these phrases atomically as new vocabulary elements in an existing semantic word embedding model.

There are a variety of techniques available that can be used to obtain semantics of image features 706 capture. In such an example, a deep machine learning network is used that has a plurality of feature levels that are learned directly from the data. In particular, convolutional neural networks (CNNs) with folding, pooling, and activation layers (for example, rectified linear units, which form an activation threshold) have proven useful for image classification. Examples include AlexNet, VGGNet and GoogLeNet.

In addition, it has been shown that classification features of deep classification networks give high quality results for other tasks (eg, segmentation), especially after fine-tuning these features for the other task. Therefore, starting from features that have been learned for classification, and then fine-tuning these features for another image understanding task, can show increased efficiency in terms of training rather than starting the training from scratch for a new task. For the above reasons, CNN features are assumed to be fixed features in a linear CCA baseline model. The machine learning module 314 then performs a fine-tuning of the model 316 from a CNN in a deep network to text and image features 704 . 706 to correlate.

The machine learning module 316 is configured, text features "t" 704 and image features "x" 706 to map into a common vector space and to penalize differences in the mapped features if the same concept or similar concepts are represented by "t" and "x".

One technique that can be used for this purpose involves a linear mapping called Canonical Correlation Analysis (CCA) based on text and image features 704 . 706 is applied. In the CCA, matrices "T" and "X" are detected, which map feature vectors "t" and "x" into a common vector space "t '= Tt" and "x' = Xx", respectively. If the mapping is performed in a common space of dimension "D" and "t" is a vector in "D_t-dimensional space" and "x" is a vector in "D_x-dimensional space", then "T" is a "(" D times D_t) "matrix," X "is a" (D times D_x) "matrix, and the mapped representations t 'and x' are D-dimensional vectors.

Loss functions may also be used for model matching using training pairs "(t, x)" based on quadratic Euclidean distance "|| t '- x' || - 2 ^ 2", or a cosine similarity "scalar product (t ', x'). ) ", Or the" angle_between (t ', x') "which removes the vector length from the cosine similarity measure. If the scalar product is used then the CCA correlation function is expressed as follows: f (t, x) = f_CCA_dp (t, x) = tr (Tt) * Xx = tr (t) * M * x = sum_ {i, j} t_i M_ {ij} x_j, where "tr" stands for transpose and "M = tr (T) * X (D_t by D_x)", and indices indicate vector components. This form supports a quicker than exhaustive search for images or text when the other is given. For example, in a text-based image search, images having feature vectors "x" are found such that the "scalar product (v, x)" is large, where "v = tr (t) * M".

For a quadratic Euclidean loss, the CCA correlation function can be expressed as follows: f (t, x) = f_CCA_E (t, x) = || Tt - Xx || _2 ^ 2. The above simple closed form of the correlation function, in turn, may support a faster than exhaustive search for images or text if the other is given. For example, in a text-based image search, images having feature vectors "x" are found such that "f_CCA_E (t, x)" is small for a given text vector "t". Given "(T, X)" of fitting the CCA model and the query "t", linear algebra yields a set of vectors that minimize "f (t, x)", and images with a feature vector "x "Found near this amount.

8th shows an example of a deep network 800 for correlating text and images as part of machine learning. The deep network 800 includes a machine text module 802 (for example, column) and a machine image learning module 804 (For example, column that is separate from the column that the machine text module 802 implemented) configured to learn the correlation "f (<S, P, O>, I)" between structured semantic knowledge "<S, P, O>" of an image or image part "I" by non-linear mapping in a common room.

The machine text module 802 starts with a semantic text vector representation "t", which vec (S) 806 , vec (P) 808 and vec (O) 810 which then passes through sets of fully connected and activation layers 812 to output a non-linear map t-> t 'as a feature vector for the text 814 ,

The machine-learning module 804 is as a deep folding neural network 814 configured (for example, as AlexNet or VGGNet or GoogLeNet, with the final layers that map to class probabilities are removed), that of image pixels of the image 816 goes out and a feature vector x 'for the image 814 outputs. The image module is initialized as the training result of an existing CNN, and the image features are fine-tuned to correlate images with structured text, capturing image attributes and interactions, rather than merely object-class discrimination, as in the existing CNN.

matching layers 822 . 824 in the machine text and image learning modules 802 . 804 adjust the representations according to a non-linear function to map them into a common space with image features representing the same concept. A loss layer 828 connects the modules and pairs differences in the outputs t 'and x' of the machine text and image learning modules 802 . 804 to promote a picture in a common space for the same concept.

A discriminating loss function, such as a ranking loss function, can be used to ensure that misallocated text and images have a smaller correlation or distance than correctly mapped text and images. For example, a simple ranking loss Correlations require "scalar product (t_i ', x_i')> scalar product (t_j ', x_i')" for a training example "(t_i, x_i)" and where the original tuple for training the tuple t_j does not match the training image "x_i". A ranking loss function may also use a semantic text similarity or an external object hierarchy, such as ImageNet, to formulate the loss to differentiate different misalignments.

Other loss functions and architectures are possible, for example, with fewer or more matching layers between the semantic text representation "t = [vec (S), vec (P), vec (O)]" and the embedding space t 'or with connections between Text and image layers in front of the common embedding room. In one example, a wildcard loss is also possible that ignores the object part of second-order facts <S, P>, and the predicate part and the object part of first-order facts <S> of depository factors.

Back to 3 becomes structured semantic knowledge at this point 310 through the model training module 312 to solve the problem of extracting a concept relevant to an image area. The above modeling is now applied to "P (Concept <S, P, O>, Image I)" to extract all highly probable concepts about a part of an image. This can be done without selecting the concept that is most likely. For example, consider an image area containing a smiling man wearing a blue shirt. Image pixel data "I" for this region will have a higher correlation with both "<man, smiling>" and with "<man, wears, blue shirt>" and therefore both of these concepts can be extracted for the same image area.

The knowledge extraction task can be solved by applying the above model to image pixel data from areas identified by an object suggestion algorithm, or object areas identified by the R-CNN algorithm, or even in a sliding window approach that more densely scans image areas. To capture object interactions, bounding rectangles are formed from pairs of object suggestions or pairs of R-CNN object areas. One approach is to sample all pairs of object areas to test for possible interactions. Another approach is to use some heuristics to be more selective, such as not examining couples that are distant in the picture. Since the model can be applied to not extract one, or more highly probable concepts over an image area, the extracted <S, P, O> concepts can be located to image areas providing the corresponding visual data.

Example procedures

The following discussion describes knowledge extraction techniques that can be implemented using the systems and devices described above. Aspects of each of the procedures may be implemented in hardware, firmware or software, or a combination thereof. The procedures are shown as a set of blocks that specify operations performed by one or more devices and that are not necessarily limited to the illustrated sequences to perform the operations through the respective blocks. In parts of the following discussion will focus on 1 - 8th based.

9 shows a procedure 900 in an exemplary implementation in which a digital media environment is used to automatically extract knowledge from an input image without user intervention. A digital media environment is described to learn a model that can be used to compute a descriptive summary of an input image automatically and without user intervention. Training data is obtained containing images and associated text (Block 902 ). The training data 320 can, for example, pictures 304 and unstructured text 306 included with the pictures 304 is associated, for example, as image captions, metadata and so on.

Structured semantic knowledge is extracted from the associated text using natural language processing by the at least one computing device, wherein the structured semantic knowledge describes textual features (Block 904 ). The structured semantic knowledge 310 For example, it may be extracted using natural language processing to generate tuples, such as <subject, attribute>, <subject, predicate, object> and so on.

A model is trained using structured semantic knowledge and images as part of machine learning (Block 906 ). A model training module 312 can for example a neural network using images 304 and structured semantic knowledge 310 work out. This knowledge can also be localized, as related in more detail 10 described.

The model is used to form a structured image representation of the input image that explicitly correlates at least a portion of the text features with image features of the input image as the descriptive summary of the input image (Block 908 ). For example, the structured image representation may correlate concepts in the text with portions of the images, along with addressing a structure of the knowledge to describe "what happens" in the images as a description summary. This description summary can be used in a variety of ways, such as finding images as part of an image search, automatically generating image captions, and so on.

10 shows a procedure 1000 in an exemplary implementation in which a digital media environment is used to extract knowledge and to locate text features to image features of an input image. A digital media environment is described to learn a model that can be used to compute a descriptive summary of an object within an input image automatically and without user intervention. Structured semantic knowledge is extracted from text associated with images, wherein processing using natural language processing by the at least one computing device (Block 1002 ). Image features of objects in each of the images are localized as corresponding to the textual features of the structured semantic knowledge (Block 1004 ). As before, structured semantic knowledge 310 extracted. In this case, however, this knowledge is localized to certain parts of the image and therefore can improve the accuracy of subsequent modeling by potentially distinguishing between several concepts in an image, for example the baby holding the toy and the man holding the toy Baby feeds as in 1 shown.

A model is trained using the localized image and text features as part of machine learning (Block 1006 ). A variety of different techniques can be used, such as to perform probabilistic modeling. The model is used to form a structured image representation of the input image that explicitly correlates at least a portion of the text features with at least one image feature of the object contained in the input image (Block 1008 ). For example, the structured logic determination module 318 an input image 108 take and a structured image representation 106 especially in cases where the input image 108 does not include any associated text. Next, the structured image representation 106 be located to correlate concepts contained in the text and image. As before, the structured image representation 106 can be used to support a variety of functionalities, such as image browsing, automated generation of image captions and so on.

implementation example

11 shows an exemplary system 1100 which can be used to perform structured fact image embedding. This system 1100 supports properties such as an ability to do so (1) can be continuously fed with new facts without changing the architecture, (2) being able to learn with wildcards to support all kinds of facts, (3) can not work on (4) allows two-way retrieval, such as retrieving relevant facts in a voice view when an image is given, and retrieving relevant images when a fact is given in a voice view. This system 1100 aims to model structured knowledge in images as a problem, with views in the visual domain V and in the speech domain L. Let "f" denote a structured "fact" (ie concept) and "f _l ε L" the view of "f" in the language domain. For example, an annotated fact, with voice view "f _l = <S: girl, P: drive, O: bike>", would have a corresponding visual view "f _v " as an image in which the fact occurs, as in 11 shown.

The system is configured to learn a representation of which first order facts <S> (objects), second order facts <S, P> (actions and attributes) and third order facts <S, P, O> (interaction and positional facts ) covers. These types of facts are represented as an embedding problem in a "structured fact space". The structured fact is configured as a learning representation of three hyperdimensions, denoted as follows:

The embedding function from a visual view of a fact "f _v " are each designated as follows: φ _S , φ _P and φ _O as φ V / S (f _v ), φ V / P (f _v ) and φ V / O (f _v )

Similarly, the embedment function is from a language view of a fact f _l : φ _S , φ _P and φ _O denotes as each of the following: φ L / S (f _l ), φ L / P (f _l ) and φ L / O (f _l )

The concatenation of the hyperdimensions of the visual view is referred to as: φ ^V (f _v )

Linking of embedding of hyperdimensions of language view is called: φ ^L (f _l ) the above being the visual embedding of "f", thus forming: φ ^V (f _v ) = [φ V / S (f _v ), φ V / P (f _v ), φ V / O (f _v )], φ ^L (f _l ) = [φ L / S (f _l ), φ L / P (f _l ), φ L / O (f _l )]

As can be seen from the above, the third-order facts <S, P, O> can thus be embedded directly in the structured fact space by: φ ^V (f _v ) for the picture view and: φ ^L (f _l ) for the voice view.

First order facts are facts that indicate an object, such as <S: Person>. Second order facts are more specific about the subject, for example <S: Person, P: plays>. Third order facts are even more specific, for example <S: Person, P: plays, O: piano). In the following, higher order facts are defined as lower order facts to which an additional modifier is applied. For example, adding the modifier "P: eats" to the fact <S: child> constructs the fact <S: child, P: eats>. Further, adding the modifier "O: ice cream" to the fact <S: child, P: eats> constructs the fact <S: child, P: eats O: ice cream>. Similarly, attributes can be addressed as modifiers for a subject, for example, applying "P: smiles" to the fact <S: Baby> constructs the fact <S: baby, P: smiles>.

Based on the above-noted fact modification, both first order and second order facts can be represented as placeholders, as illustrated in the following equations for first order and second order facts respectively. φ ^V (f _v) = [φ V / S (f _v), φ V / P (f _v) = *, V / O φ (f _v) = *], ^L φ (f _l) = [φ L / S (f _l ), φ L / P (f _l ) = *, φ L / O (f _l ) = *] φ ^V (f _v ) = [φ V / S (f _v ), φ V / P (f _v ), φ V / O (f _v ) = *], φ ^L (f _l ) = [φ L / S (f _l ), φ L / P (f _l ), φ L / O (f _l ) = *] The setting of "φ _P " and "φ _O " to "*" for first order facts is interpreted as meaning that the modifiers "P" and "O" are not of interest for first order facts. Similarly, setting "φ ₀ " to "*" for second order facts indicates that the modifier "0" is not of interest for frame actions and attributes.

First and second order facts are called placeholder facts. Since modeling structured facts in visual data potentially allows logical reasoning about facts from images, the problem described below is also referred to as a "Sherlock" problem.

To train a machine learning model that links the structured factual language view in L to its visual view in V, data is collected in the form of pairs (f _v , f _l ). The collection of data for large scale problems is becoming increasingly challenging, especially in the examples below, since the model is based on a localized association of a structured linguistic feature "f _l " with an image "f _v " when such facts occur. In particular, it is a complex task to collect annotations especially for second-order facts <S, P> and third-order facts <S, P, O>. Also, multiple structured linguistic facts may be assigned to the same image, for example, <S: man, P; smiling> and <S: man, P: wears, O: glass>. If these facts refer to the same man, the same picture example could be used to learn both facts.

As discussed above, techniques are discussed in which factual annotations are automatically collected from datasets that are in the form of image / caption pairs. For example, a large amount of high quality facts can be obtained from image caption databases using natural language processing. Since the writing of image captions is freeform, these are typically readily available, for example, from social networks, preconfigured databases, and so forth.

In the following example, a two-step automatic annotation process is described (i) fact extraction from image captions comprising any text associated with an image describing the image; and (ii) Fact location in pictures. First, the image labels associated with the given image are analyzed to extract sets of clauses considered to be candidates of facts <S, P> and <S, P, O> in the image. Clauses are facts, but are not necessarily facts themselves.

Image captions can provide large amounts of information for image comprehension systems. However, developing natural language processing systems to extract accurate and fully structured knowledge from freeform text is challenging because of (1) spelling and punctuation errors; (2) word sense ambiguity within clauses; and (3) a dictionary of spatial prepositions that may contain hundreds of terms, such as "off," "on," and aggregate phrase adjectives, such as "group of," "bundles of," and so on.

The process of locating facts in an image is limited by information in the dataset. For example, a database may contain object annotations for different objects through training and validation sets. This allows first-order facts to be located for objects using bounding rectangle information. To locate facts of higher order in images, visual entities are defined as any noun that is either a dataset object or a noun in a predefined ontology that is a direct or indirect generic term of one of the objects. It is expected that visual entities will appear in either the S or the O part, if any, for a candidate factor "f _l ", allowing the location of facts for images. For a given third order fact, it is first tried to assign each "S" and "O" to one of the visual entities. If "S" and "O" are not visual entities, then the clause is ignored. Otherwise, the clauses are processed by different heuristics. For example, the heuristics may consider whether the subject or object is singular or plural or a scene. For example, in the fact that the techniques described herein may identify <S: Men, P: Hunt, O: Football>, "Men" may involve an association of multiple limiting candidate rectangles, while "Football" is expected to only gives a single bounding rectangle.

A direct approach to modeling facts in images is to learn a classifier for each separate fact. However, there is a clear limitation in scalability in this technique because the number of facts is significant, for example | S | × | P | × | O |, where | S |, | P | and | O | the number of subjects, predicates or objects are. Therefore, this number could reach millions for possible facts in the real world. In addition to these scalability issues, this technique ignores semantic relationships between facts, which are a significant feature that allows a generalization to unseen facts or figures with few examples. For example, during training, a second order fact may be <S: boy, P: plays> and a fact of first order like <S: girl>, <S: boy>. At runtime, the model trained using the techniques described herein understands Image titled <Girl, plays> even though this fact was not seen during practice which is clearly missed by learning a model for each fact in the workout.

Accordingly, this example describes an embedding problem with two views that is used to model structured facts. For example, a structured fact embedding model may include (1) bidirectional retrieval (that is, retrieving relevant facts in a voice view when an image is given and retrieving relevant images when a fact is given in a voice view) and (2) supporting facts are supported means facts of first and second order.

The first property is fulfilled in this example by using a generative model p (f _v , f _l ) connecting the visual view and the language view of "f". This technique first models the following: p (f _v , f _l ) α s (φ ^V (f _v ), φ ^L (f _l )) where "s (·, ·)" is a similarity function defined over the structured fact space designated by "S", which is a discriminating space of facts. This is done so that two views of the same fact are embedded close to each other.

To model and train "φ ^V (f _v )", a CNN encoder is used, and to model and train "φ ^L (f _l )", an RNN encoder is used. In an exemplary implementation 1200 from 12 Two models are proposed to learn facts that are referred to as Model 1 and Model 2. Models 1 and 2 have the same structured fact-language embedding and the same encoder, but differ in the structured fact-image encoder.

This process begins by defining an activation operator "ψ (θ, α)", where "α" is an input and "θ" is a series of one or more neural network layers that may include different types of layers, such as four convolutional layers, a pooling layer, and another folding and pooling layer. The operator "ψ (θ, α)" applies parameter "θ" layer by layer to calculate the subnetwork "θ" active for a given "α". An operator "ψ (·, ·)" is used to define model 1 and model 2 structured fact image encoders.

In Model 1, a structured fact is visually coded by dividing convolutional layer parameters (denoted by θ v / c) and fully connected layer parameters (denoted by θ u / c). Then transformation matrices "W v / S", "W v / P" and "W v / O" applied to "Φ v / S (f _v ), φ v / P (f _v ), φ v / O (f _v )" to produce as follows: φ V / S (f _v ) = WS / vψ (θ u / v, ψ (θ c / v, | f _v )), φ V / P (f _v ) = WP / vψ (θ u / v, ψ (θ c / v, f _v )), φ V / O (f _v) = WO / vψ (θ u / v, ψ (θ c / v, f _v))

In contrast to model 1, in model 2 for "S", different convolution layers are used than for "P" and "O", consistent with the above discussion, that "P" and "O" are modifiers for "S", as previously described , Starting from "f _v " there is a common set of convolutional layers, denoted by "Θ c0 / v", then the network splits into two branches, with two sets of convolutional layers "Θ cs / v" and "Θ cp0 / v" generated, followed by two fully connected layers "Θ us / v" and "θ uP0 / v" Finally "Φ v / S (f _v ), φ v / P (f _v ), φ v / O (f _v )" through transformation matrices "W v / S", "W v / P" and "fv / O", calculated as follows:

In both models, a structured linguistic feature is encoded using RNN speech embedding vectors for "S, P and O". Therefore, in the case Φ L / S (f _L ) = RNN _θ L (f L / S), φ L / P (f _L ) = RNN _{θ L} (f P / S), φ L / O (f _L ) = RNN _{θ L} (f O / L) ", in which "F L / S", "f P / S" and "f O / L" are the subject, predicate and object part of "f _L ε L". For each of these literals are dropped, and when one of "F L / S", "f P / S" and "f O / L" contains several words, the average vector is calculated as the representation of that part. The RNN language encoder parameters are referred to as "θ ^L ". In one or more implementations, "θ ^L " is fixed to a pre-trained word vector embedding model for "F L / S", "f P / S" and "f O / L".

One way to train "p (f _v , f _l )" for model 1 and model 2 is to assume that "p (f _v , f _l ) ∞ = exp (-loss _w (f _v , f _l )) And minimize the distance loss "loss _w (f _v , f _l )" defined as follows: loss _w (f _v , f _l ) = wf / S · || φ V / S (f _v ) - φ L / S (f _l ) || ² + wf / P · || φ V / P (f _v ) - φ L / P (f _l ) || ² + wf / O · || φ V / O (f _v ) - φ L / O (f _l ) || ² which minimizes the distances between the embedding of the visual view and the voice view. One solution to palerize placeholder facts is to ignore the placeholder modifiers in the loss by using a weighted Euclidean distance, the weighting of which is based on whether corresponding parts of the feature vector are present, which is called a "placeholder" loss. Here "Wf / S = 1", "wf / P = 1" and "wf / O = 1" for facts <S, P, O>, "Wf / S = 1", "wf / P = 1" and "wf / O = 0" For facts <S, P> and "Wf / S = 1", "wf / P = 0" and "wf / O = 0" for facts <S>. Therefore, "loss _w " does not penalize the "O" second order factor modifier or the "P" and "O" first order factor modifiers, which follows the above definition of a wildcard modifier.

Accordingly, this example describes a problem of association of high-order visual and linguistic facts. A neural network approach is described to associate visual facts and linguistic facts in a common, continuous space of structured facts that allow natural language facts to be associated with an image, and images associated with structured descriptions of natural language.

Exemplary system and device

13 shows an exemplary system in general 1300 , which is an exemplary computing device 1302 which is representative of one or more computing devices and / or devices that can implement the various techniques described herein. This is illustrated by the knowledge extraction system 104. is included. The computing device 1302 For example, a server of a service provider, a device associated with a client (eg, a client device), an on-chip system, and / or other suitable computing device or other suitable computing system may be.

The illustrated computing device 1302 includes a processing system 1304 , one or more computer-readable media 1306 and one or more input / output interfaces 1308 that are communicatively connected. Although not shown, the computing device 1302 further include a system bus or other data and command transfer system interconnecting the various components. A system bus may include any or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a USB bus, and / or a processor or local bus using any of a variety of bus architectures. A variety of other examples are also contemplated, such as control and data lines.

The processing system 1304 is representative of functionality for performing one or more operations using hardware. Accordingly, the processing system 1304 as a hardware element 1310 containing, which may be configured as processors, functional blocks and so on. This may involve a hardware implementation as an application specific circuit or as another logic device formed using one or more semiconductors. The hardware elements 1310 are not limited by the materials by which they are formed or the processing mechanisms used therein. For example, processors may include semiconductors and / or transistors (eg, integrated circuits (ICs)). In such a context, instructions executable by a processor may be electronically executable instructions.

The computer-readable storage medium 1306 is with a memory 1312 shown. The memory 1312 represents a storage capacity associated with one or more computer-readable media. The storage component 1312 may include volatile media (such as a random access memory (RAM)) and / or nonvolatile media (such as a read only memory (ROM), flash memory, optical disks, magnetic disks, and so on). The storage component 1312 can fixed built-in media (for Example RAM, ROM, built-in hard disk and so on) as well as removable media (eg flash memory, a removable hard disk, an optical disk and so on). The computer-readable medium 1306 can be configured in several other ways, as described further below.

Input / output interface (s) is representative of a functionality that allows a user, the computing device 1302 Entering commands and information as well as allowing information to be presented to the user and / or other components or devices using input / output devices. Examples of input devices include a keyboard, a cursor control device (eg, a mouse), a microphone, a scanner, touch functionality (eg, capacitive or other sensors configured to detect a physical touch), a camera (which may be used for the purpose of FIG Example, use visible or non-visible wavelengths, such as infrared frequencies, to detect motion as gestures that do not involve touch), and so on. Exemplary output devices include a display device (eg, a monitor or projector), speakers, a printer, a network card, tactile feedback devices, and so on. The computing device 1302 Therefore, it can be configured in various ways, as further described below, to support interactions

Various techniques may be described herein in the general context of software, hardware elements or program modules. In general, such modules include routines, programs, objects, elements, components, and so forth that perform certain tasks or implement particular abstract data types. The terms "module", "functionality" and "component" as used herein generally represent software, firmware, hardware or a combination thereof. The features of the techniques described herein are platform independent, which means that the techniques can be implemented on a variety of commercial computing platforms with a variety of processors.

An implementation of the described modules and techniques may be stored or transmitted on some form of computer-readable media. The computer readable media may include a variety of media to which the computing device 1302 can access. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

"Computer-readable storage media" may refer to media and / or devices that enable persistent and / or non-transitory storage of information, as opposed to mere signal transmission, carrier waves, or signals as such. Computer-readable storage media therefore refer to media that carry no signal. The computer-readable storage media include hardware such as volatile and nonvolatile, removable and non-removable media, and / or memory devices implemented with a method or technology that may be used to store information such as computer readable instructions, data structures, program modules, logical Elements / circuits or other data are suitable. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD other optical storage, hard disks, magnetic cassettes, magnetic tapes, magnetic storage disks or other magnetic storage devices, or other storage device , tangible media, or other manufactured article capable of storing the desired information and allowing access by a computer.

"Computer readable signal media" may refer to a signal carrying medium configured to provide instructions to the hardware of the computing device 1302 to transmit, such as over a network. Signal media may typically embody computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves, data signals, or other transport mechanism. Signal media also includes all information delivery media. The term "modulated data signal" means a signal in which one or more of its characteristics are adjusted or changed in such a way as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or a direct-wired connection, and wireless media such as acoustic, wireless, and other wireless media.

As stated previously, the hardware elements are 1310 and computer-readable media 1306 representative of modules, programmable device logic, and / or fixed device logic implemented in a hardware form that may be used in some embodiments to implement at least some aspects of the techniques described herein, such as to execute one or more instructions. Hardware may include integrated circuit components or a one-chip system, application specific integrated circuit (ASIC), field programmable gate logic (FPGA), complex programmable logic device (CPLD) and other implementations in silicon or other hardware. In this context, hardware may function as a processing device that performs program tasks that are represented by instructions and / or logic embodied by the hardware, as well as hardware used to store instructions to be executed, such as the The computer-readable storage medium described above.

Combinations of the above may also be used to implement various techniques described herein. Accordingly, software, hardware, or executable modules may be implemented as one or more instructions and / or logic stored on a form of computer-readable storage medium and / or one or more hardware elements 1310 are embodied. The computing device 1302 may be configured to implement particular instructions and / or functions corresponding to the software and / or hardware modules. Accordingly, an implementation of a module implemented by the computing device 1302 be executed as software, at least partially achieved in hardware, for example by the use of computer-readable storage media and / or hardware elements 1310 of the processing system 1304 , The instructions and / or functions may be executable by one or more products (for example, one or more computing devices 1302 and / or processing systems 1304 ) to implement techniques, modules and examples described herein.

The techniques described herein may be achieved by various configurations of the computing device 1302 and are not limited to the specific examples of the techniques described herein. This functionality may also be implemented in whole or in part by using a distributed system, such as a cloud. 1314 over a platform 1316 as described below.

The cloud 1314 includes and / or is representative of a platform 1316 for resources 1318 , The platform 1316 abstracts the underlying functionality of hardware resources (such as servers) and software resources of the cloud 1314 , The resources 1318 may include applications and / or data that may be used while computer processing is performed on servers that are the computing device 1302 are away. The resources 1318 may also include services provided over the Internet and / or subscriber network, such as a cellular network or wireless network.

The platform 1316 can abstract resources and functions to the computing device 1302 to connect with other computing devices. The platform 1316 may also serve to abstract the scaling of resources to a corresponding level of scaling to meet demanded resources 1318 to offer that over the platform 1316 are implemented. Accordingly, in an environment of interconnected devices, implementation of functionality described herein may be via the system 1300 be distributed across. For example, the functionality may be partially on the computing device 1302 and partly by means of the platform 1316 implements the functionality of the cloud 1314 abstracted.

conclusion

Although the invention has been described in language specific to structural features and / or method actions, it should be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed invention.

Claims (20)

In a digital media environment for learning a model ( 316 ) that can be used to automatically and without user intervention explicitly image features ( 706 ) of an input image ( 108 ) with text features ( 704 ), a method by at least one computing device ( 102 ; 1302 ), comprising: obtaining training data ( 302 ) by the at least one computing device ( 102 ; 1302 ), whereby the training data ( 302 ) Images ( 304 ) and associated text ( 306 ) include; Extracting Text Features ( 704 ) as structured semantic knowledge ( 310 ) from the associated text ( 306 ) of the training data ( 302 ) using natural language processing by the at least one computing device ( 102 ; 1302 ); Train a model ( 316 ) using the structured semantic knowledge ( 310 ) and images ( 304 ) as part of machine learning by the at least one computing device ( 102 ; 1302 ), where the model ( 316 ) after being trained, a structured image representation ( 106 ) to explicitly define the image features ( 706 ) of the input image ( 108 ) with at least one of the extracted text features ( 704 ) to correlate. The method of claim 1, further comprising generating the structured image representation ( 106 ) as a descriptive summary of the input image ( 108 ) by associating structured tags with the input image ( 108 ) using the trained model ( 316 ), wherein associating includes calculating probabilities that the extracted text features ( 704 ) and the image features ( 706 ) of the input image ( 108 ) describe a similar concept. Method according to claim 1 or claim 2, wherein the associated text ( 306 ) is free-form and unstructured. The method of claim 3, wherein the associated text ( 306 ) an image caption or metadata of each one of the image ( 304 ). Method according to one of the preceding claims, wherein the structured semantic knowledge ( 310 ) in a form of a tuple <subject, attribute> or <subject, predicate, object>. The method of any one of the preceding claims, wherein extracting includes: extracting a plurality of the text features ( 704 ) of the structured semantic knowledge using a variety of different tuple extraction techniques; and based on at least partial consensus in the plurality of textual features ( 704 ) of structured semantic knowledge ( 310 ) extracted using the plurality of different tuple extraction techniques: determine whether at least one of the textual features ( 704 ) of structured semantic knowledge ( 310 ) should be corrected; Removing the at least one of the textual features ( 704 ) of structured semantic knowledge ( 310 ), or identifying a degree of confidence in extracting each of the textual features ( 704 ) of structured semantic knowledge ( 310 ) by the at least one computing device ( 102 ; 1302 ). Method according to one of the preceding claims, wherein the extraction comprises: jointly extracting the structured semantic knowledge ( 310 ) for a plurality of individual textual features ( 704 ) associated with a single respective image ( 304 ); and based on at least partial consensus of the plurality of individual ones of the textual features ( 704 ) of the text ( 306 ), with the individual respective image ( 304 ) is associated; Determining whether at least one of the plurality of individual text features ( 704 ) of structured semantic knowledge ( 310 ) is to be corrected; Removing the at least one of the plurality of individual text features ( 704 ) of structured semantic knowledge ( 310 ); or identifying a degree of confidence in extracting the respective text feature ( 704 ) of structured semantic knowledge ( 310 ) by the at least one computing device ( 102 ; 1302 ). Method according to one of the preceding claims, wherein the extracting comprises locating at least part of the structured semantic knowledge ( 310 ) as respective respective subjects, predicates or objects in respective ones of the images ( 304 ) of the training data ( 302 ). Method according to one of the preceding claims, wherein the training includes adapting the text features ( 704 ) or image features ( 706 ) of the image information, so that similar structured knowledge concepts represent nearby and related representations in a vector space as part of the structured semantic knowledge ( 310 ) exhibit. Method according to one of the preceding claims, wherein the model ( 316 ) the image features ( 706 ) of the input image ( 108 ) explicitly with the text features ( 704 ), so that at least one of the Image characteristics ( 706 ) explicitly with a first of the text characteristics ( 704 ), but not with a second of the text features ( 704 ) is correlated. Method according to one of the preceding claims, wherein the structured image representation ( 106 ) of the input image ( 108 ) structured semantic knowledge ( 310 ), that of the trained model ( 316 ), explicitly with the entered image ( 108 ) is correlated and executed without text ( 306 ), otherwise using the input picture ( 108 ) is associated. In a digital media environment to use a model ( 316 ) to explicitly use text features ( 704 ) with image features ( 706 ) of an input image ( 108 ) to correlate automatically and without user intervention, a system of at least one computing device ( 102 ; 1302 ), comprising: an extraction module for extracting structured semantic knowledge ( 310 ) from text ( 306 ), with pictures ( 304 ) in training data ( 302 ) using natural language processing, wherein the structured semantic knowledge ( 310 ) the text characteristics ( 704 ) describes; a model training module ( 312 ) for training a model ( 316 ) using the structured semantic knowledge ( 310 ) and image features ( 706 ) of the pictures ( 304 ) in the training data ( 302 ) as part of machine learning by the at least one computing device ( 102 ; 1302 ), where the model ( 316 ) is configured to determine probabilities to predict how well image features ( 706 ) of the input image ( 108 ) with the extracted text features ( 704 ) correlate. The system of claim 12, wherein the associated text ( 306 ) is free-form and unstructured. The system of claim 12 or claim 13, wherein the structured semantic knowledge ( 310 ) in a form of a tuple <subject, attribute> or <subject, predicate, object>. The system of any one of claims 12 to 14, wherein the extraction module is configured to include at least a portion of the structured semantic knowledge ( 310 ) as respective objects in respective ones of the images ( 304 ) to locate accordingly. The system of any of claims 12 to 15, further comprising a module for using the structured image representation ( 106 ) that is configured, the structured image representation ( 106 ) to use the input image ( 108 ) as part of a picture search as part of a determination of how well the structured image representation ( 106 ) corresponds to a search query the image search. The system of any one of claims 12 to 16, further comprising a module for using the structured image representation ( 106 ), which is configured based on the structured image representation ( 106 ) an image caption for the image ( 304 ) to create. The system of any one of claims 12 to 17, further comprising a module for using the structured image representation ( 106 ), which is configured based on the structured image representation ( 106 ) Scene properties of the input image ( 108 ). In a digital media environment to use a model ( 316 ) as part of an image search to display an input image ( 108 ) automatically and without user intervention to locate a method, the at least one computing device ( 102 ; 1302 ), comprising: extracting structured semantic knowledge ( 310 ) from text ( 306 ), with pictures ( 304 ) in training data ( 302 ) using natural language processing by the at least one computing device ( 102 ; 1302 ), where the structured semantic knowledge ( 310 ) Text features ( 704 ) describes; Train a model ( 316 ) using the structured semantic knowledge ( 310 ) and images ( 304 ) from the training data ( 302 ) as part of machine learning by the at least one computing device ( 102 ; 1302 ); Forming a structured image representation ( 106 ) of the input image ( 108 ), by the at least one computing device ( 102 ; 1302 ) which explicitly include at least some of the textual features ( 704 ) of the training data ( 302 ) with image features ( 706 ) of the input image ( 108 ) correlates; and locate the input image ( 108 ) as part of an image search by the at least one computing device ( 102 ; 1302 ), using the structured image representation ( 106 ). The method of claim 19, wherein the text ( 306 ) is a picture caption.

Images