DE102017008030A1 - USE OF A DIGITAL CANVAS TO PERFORM A ROOM-SEMANTIC SEARCH FOR DIGITAL VISUAL MEDIA - Google Patents

Abstract

The present disclosure includes methods and systems for searching digital visual media based on semantic and spatial information. In particular, one or more embodiments of the disclosed systems and methods identify or identify digital visual media that represents a searched visual content in a searched area based on a query term and an interrogation area provided by a digital canvas. In particular, the disclosed systems and methods may receive user input of a query term and an interrogation area and provide the query term and polling area for a neural polling network to generate a polling feature set. In addition, the disclosed systems and methods may compare the query feature set with digital visual media feature sets. Further, the disclosed systems and methods may identify, based on the comparison, digital visual media that represents a searched visual content corresponding to the query term within a searched range that corresponds to the query range.

Description

This application claims the priority of U.S. Provisional Patent Application No. 62 / 414,140 , filed October 28, 2016, entitled "Use of a Digital Canvas to Perform a Spatial-Semantic Search for Digital Visual Media," which is incorporated herein by reference in its entirety.

BACKGROUND

In recent years, there has been a rapid technological development in the field of search in digital visual media. As a result of the proliferation of personal computer devices and digital cameras, individuals and businesses now routinely manage large memory contents with digital images and digital video. Accordingly, a search for digital visual media has become a widespread need for individuals and businesses in various scenarios ranging from occasional users seeking to locate specific moments from a personal photo collection to professional graphic designers who browse image collections to enhance creative projects ,

In response, developers have created various digital search systems that can search digital visual media. These conventional digital search systems fall largely under two main search paradigms: search through text-based searches (ie, systems that use a keyword to search a memory content with digital images) and search through a similar image (ie, systems that use an existing digital image to search for use similar digital pictures). Although these conventional digital search systems can identify digital visual media that reflect a particular content, they also have a number of deficiencies. For example, although conventional digital search systems may identify content in digital images, such conventional digital search systems are unable to efficiently identify digital visual content that reflects a particular spatial configuration.

As an example, users often want to find digital images with a specific visual arrangement of objects. For example, a professional designer may need a digital image showing a specific item in a particular location for a creative project. Existing digital systems allow users to search for digital images that reflect specific content but can not accurately identify digital images based on the spatial arrangement.

To illustrate this point show the 1A and 1B the results of conventional search systems for a picture of a person holding a tennis racket to the left. The 1A shows the results of a conventional text-based search, while the 1B shows the results of a conventional similar image-based search. As it is shown, the shows 1A in that a word query 102 is limited in its ability to render spatial features in a search. In particular, the word query 102 may describe a desired content (ie, "tennis racket") but may not provide a way for accurate spatial restrictions. Although the word query 102 includes the text describing a particular configuration (ie, "left"), such a term can not be translated into a meaningful search result. Thus, the word search results 102a represent digital images that include tennis rackets; however, the spatial configuration of the tennis rackets shown within the digital images is random. Accordingly, a user searching an image of a person holding a tennis racket to the left will see the word search results 102a in one Attempt to find a digital image that matches the desired spatial arrangement.

Accordingly, as shown, the image query 104 is limited in its ability to render spatial information in a search. As an original item, the picture retrieval unit 104 requires an image of a person holding a tennis racket to the left to search for a digital image of a person holding a tennis racket to the left. Of course, this represents a significant inconvenience to the user, inasmuch as the lack of a digital example image is primarily the reason for conducting the search. Even if it is assumed that a user already has an image of a person holding a tennis racket to the left to produce the image query 104, the image query 104 may not adequately include spatial concepts in the search. Although the image search results 104a generally include tennis rackets and tennis players, the image search results reflect 104 tennis rackets in many different spatial configurations. Thus, a user seeking an image of a person holding a tennis racket to the left will have to search the image search results 104a in an attempt to find a digital image that matches the desired spatial arrangement.

As shown, conventional digital search systems generally lack the ability to output accurate search results for images having a particular spatial arrangement of objects.

SHORT SUMMARY

One or more embodiments of the present disclosure provide or provide a benefit and / or solves or solves one or more of the above or other problems in the art with systems and methods based on spatial and environmental principles Search for and identify semantic information about digital visual media. In particular, in one or more embodiments, the disclosed systems and methods utilize user interaction with a digital canvas to determine both a spatial and a semantic search subject (eg, a query term indicating a searched visual content and a query range that includes a searched Indicates the visual content area). Moreover, the disclosed systems and methods perform a search based on the particular spatial and semantic search subject for finding digital images that represent the searched visual content within the searched area. In particular, in one or more embodiments, the disclosed systems and methods develop a deep learning model that generates a representation of semantic and spatial features of one or more query terms and one or more query ranges. In addition, the disclosed systems and methods utilize the features of the in-depth learning model to search for corresponding digital visual media assets having similar characteristics. In particular, the disclosed systems and methods compare a feature representation of a query region and a query term with digital image feature sets representing a plurality of digital images. In this way, the disclosed systems and methods identify digital visual media objects that render searched visual content within a searched area.

Additional features and advantages of one or more embodiments of the present disclosure are set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of such example embodiments.

list of figures

• 1A die Ergebnisse einer Suche nach einer Person, die einen Tennisschläger nach links hält, mittels einer herkömmlichen Textsuche zeigt;
• 1B die Ergebnisse einer Suche nach einer Person, die einen Tennisschläger nach links hält, mittels einer herkömmlichen entsprechenden Bildsuche zeigt;
• 1C die Ergebnisse einer Suche nach einer Person, die einen Tennisschläger nach links hält, mittels einer räumlich-semantischen Suche gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 2 eine Darstellung der Identifizierung von digitalen visuellen Medien mittels eines trainierten neuronalen Abfragenetzwerks und eines Schritts zur Erzeugung eines Abfragemerkmalssatzes mittels eines neuronalen Abfragenetzwerks gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 3 eine Darstellung der Erzeugung einer Darstellung von Abfragebegriffen und Abfragebereichen in einer digitalen Leinwand gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 4A und 4B ein Training eines neuronalen Abfragenetzwerks gemäß einer oder mehrerer Ausführungsform(en) zeigen;
• 5A bis 5C eine Rechenvorrichtung und eine graphische Nutzerschnittstelle zur Durchführung einer Suche nach digitalen Bildern unter Verwendung einer digitalen Leinwand gemäß einer oder mehrerer Ausführungsform(en) zeigen;
• 6 eine Darstellung einer Mehrzahl von beispielhaften räumlich-semantischen Suchen und Suchergebnissen gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 7 ein schematisches Diagramm, das ein räumlich-semantisches Mediensuchsystem darstellt, gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 8 ein schematisches Diagramm, das eine beispielhafte Umgebung, in der ein räumlich-semantisches Mediensuchsystem implementiert werden kann, gemäß einer oder mehrerer Ausführungsform(en) zeigt;
• 9 ein Flussdiagramm einer Reihe von Vorgängen in einem Verfahren des Identifizierens eines digitalen Bilds mittels räumlicher Informationen und semantischer Informationen gemäß einer oder mehrerer Ausführungsform(en) zeigt; und
• 10 ein Blockdiagramm einer beispielhaften Rechenvorrichtung gemäß einer oder mehrerer Ausführungsform(en) zeigt.

The detailed description refers to the attached drawings, wherein:

• 1A showing the results of a search for a person holding a tennis racket to the left by a conventional text search;
• 1B showing the results of a search for a person holding a tennis racket to the left by means of a conventional corresponding image search;
• 1C showing the results of a search for a person holding a tennis racket to the left by means of a spatial-semantic search according to one or more embodiments;
• 2 Figure 12 shows an illustration of the identification of digital visual media by means of a trained neural interrogation network and a step of generating a query feature set by means of a neural interrogation network according to one or more embodiments;
• 3 Figure 12 shows an illustration of the generation of a representation of query terms and query ranges in a digital canvas according to one or more embodiments;
• 4A and 4B show training of a neural interrogation network according to one or more embodiments;
• 5A to 5C show a computing device and graphical user interface for performing a digital image search using a digital canvas in accordance with one or more embodiments;
• 6 depicting a plurality of exemplary spatially-semantic searches and search results according to one or more embodiments;
• 7 12 is a schematic diagram illustrating a spatial-semantic media search system, according to one or more embodiments;
• 8th 12 is a schematic diagram illustrating an example environment in which a spatial semantic media search system may be implemented, according to one or more embodiments;
• 9 FIG. 3 shows a flowchart of a series of operations in a method of identifying a digital image using spatial information and semantic information, in accordance with one or more embodiments; and
• 10 a block diagram of an exemplary computing device according to one or more embodiments (s) shows.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure include or include a spatial-semantic media search system that identifies digital visual media, including spatial and semantic properties. In particular, in one or more embodiments, the spatial semantic media search system identifies digital images that render searched visual content within a searched area. For example, in one or more embodiments, the spatial semantic media search system utilizes a neural network to generate a query feature set from a query term and a query scope. Further, the spatial semantic media search system utilizes the query feature set to search a storage content of digital visual media. In particular, the spatial semantic media search system generates feature sets of digital images corresponding to digital images using another neural network and compares the query feature set with the feature sets of digital images. Based on this comparison, the spatial semantic media search system identifies digital visual media objects that reflect searched content that corresponds to a query term within a searched range that corresponds to a query scope.

The spatial semantic media search system provides a number of advantages over conventional digital search systems. As an original item, the spatial semantic media search system determines the user's intent to search for both semantic and spatial features, and provides digital visual media search results representing the desired semantic and spatial features. Accordingly, the spatial semantic media search system quickly and easily searches for and identifies digital visual media objects that represent desired spatial and semantic features.

Further, through the use of in-depth learning techniques, the spatial semantic media search system analyzes high level and low level features in identifying digital visual media of query terms and query ranges. For example, the spatial semantic media search system analyzes in-depth features that have been learned by a trained neural network and that effectively capture high-level concepts as well as low-level pixel similarities. This results in more accurate and reliable results compared to conventional digital search systems.

For example, conventional digital search systems often have the problem of a semantic gap. A semantic gap refers to the difference in meaning between presentation systems, such as the difference in meaning between digital low-level representations of visual media (e.g., pixels in a digital image) and high-level concepts rendered by digital visual media (e.g., an item or an environment rendered through a digital image). Conventional digital search systems are generally more accurate in identifying digital visual media items having similar low level features (e.g., red color pixels), but have difficulties in identifying similarity of high level features (e.g., a distinction between a red sock and red pants). Utilizing deep learning techniques to train a neural interrogation network, the spatial semantic media search system compares both high and low level features and bridges the semantic gap.

Further, by utilizing a neural interrogation network, the spatial semantic media search system may directly generate a feature representation (e.g., a feature set) from a query using a neural interrogation network and compare the generated feature representation to a memory content of digital visual media objects. Because the spatial semantic media search system provides a trainable end-to-end framework, the spatial semantic media search system can work more flexibly and more easily generalize searches and concepts rendered in digital visual media.

For example, some conventional systems rely on a text-based search in databases (eg, a text search of tagged items rendered in a sample image database with corresponding feature sets) to identify visual features. Such an approach limits the reliability of the resulting search. In fact, such an approach is limited to the particular set of samples within the sample image database. In contrast, the spatial semantic media search system uses a neural interrogation network that directly generates a feature set. The result is a more flexible approach that can generalize high-level concepts and features when performing a search, and that does not depend on specific samples identified in a database.

Further, as described in more detail below, the spatial semantic media search system can directly optimize the retrieval performance through the use of a neural interrogation network. This improves both the user experience and the performance of computer devices that operate the spatial semantic media search system. In particular, in one or more embodiments, the spatial semantic media search system trains the neural interrogation network using an objective loss function that optimizes the performance of the neural interrogation network. In particular, the spatial semantic media search system utilizes a loss function to train the neural interrogation network to generate feature sets that reduce similarity loss and increase differentiation with respect to irrelevant visual media and terms. The result is a neural interrogation network that can identify sought-after visual content within searched areas more accurately, faster, and with fewer computational resources (e.g., fewer resources to train and use the neural interrogation network).

With reference to the figures, additional details regarding a search for digital visual media according to one or more embodiments are provided. As used herein, the term "digital visual media" (or "digital visual media objects") refers to any digital object that can provide a visual representation. For example, the term "digital visual media article" includes digital images and digital video. As used herein, the term "digital image" includes any digital symbol, image, icon or representation. For example, the term "digital image" includes digital files with the following file extensions: JPG, TIFF, BMP, PNG, RAW, or PDF. Accordingly, the term "digital video" as used herein refers to a digital sequence of images. For example, the term "digital video" includes digital files with the following file extensions: FLV, GIF, MOV, QT, AVI, WMV, MP4, MPG, MPEG, or M4V. Although many examples are described herein with respect to digital images, the disclosed embodiments may also be implemented with respect to any digital visual media objects.

The 1C shows an example of search results of the spatial semantic media search system of the present disclosure. In particular, the shows 1C according to the 1A and 1B an embodiment in which a user wants to find a picture of a person holding a tennis racket to the left. As it is in the 1C is shown, a user generates a spatial-semantic query 106 containing both spatial information and semantic information for searching for a digital image that includes searched visual content within searched areas. To illustrate, the 1C that the spatially-semantic query 106 a digital canvas 108 , a first query area 110 , the first query term 110a (ie, one person) and a second polling area 112 , the second query term 112a (ie, a tennis racket) includes. The digital canvas 108 allows users to have a search interest both in terms of semantic information (ie, the query terms 110a . 112a ) as well as spatial information (ie, the query areas 110 . 112 ).

As used herein, the phrase "searched digital image" (or searched visual media) refers to a digital image (or visual media object) that meets search parameters. In particular, a "searched digital image" includes a desired digital image that a user is looking for (ie, that satisfies the user's desired search parameters). For example, a user searches in relation to the 1C an image showing a person holding a tennis racket to the left. Accordingly, a searched digital image with respect to the 1C a digital image showing a person holding a tennis racket to the left.

As used herein, the phrase "sought-after visual content" refers to a desired representation that is displayed in digital visual media. In particular, the phrase "sought-after visual content" refers to a visual representation that a user seeks in a searched digital image. For example, a sought-after visual content may include a desired item, a desired action, or any other desired visual representation. By way of illustration, the visual content sought is in terms of the 1C a person and a tennis racket. With respect to the spatial semantic media search system, a query term may indicate a searched visual content.

As used herein, "query term" refers to a word or phrase that is used to express a desired concept. In particular, "query term" includes a word or phrase used to express a desired visual content in a searched digital image. In other words, a query term in one or more examples refers to an item in an image to be identified. A query term may include any word or sentence, including, for example, nouns, verbs, adverbs or adjectives. Thus, a "query term" may include a term indicating an item (eg, the term "car"), an action (eg, the term "pitch"), a descriptor (eg, the term "red"), a query criterion (eg, the term "Dangerous") or any combination thereof (eg the phrase "red car that is racing dangerously"). For example, the query term includes 110a in terms of the 1 "Person" and the query term 112a includes "tennis rackets".

As used herein, the phrase "searched area" refers to an area of a digital image. In particular, the term "searched area" includes an area of a digital image that includes a searched visual content. For example, the "searched area" for the spatial-semantic search of 1C after the person the center of the digital image. With respect to the spatial-semantic media search system, a user can express a searched area by means of a polling area.

As used herein, the term "query area" refers to a specified area of a digital asset. In particular, the term "interrogation area" refers to an area of a digital screen. For example, the "query area" includes an area of a digital canvas that indicates a searched area in a searched digital image. In particular, the term "query area" includes an area of a digital canvas that indicates a searched area that reflects a searched visual content. A query scope can include many different shapes or scope types. For example, a query region may include a circle, square, rectangle, triangle, or other shape. Similarly, a query area may include a sketch, a drawing, or other irregular border or shape. Regarding the 1C is the query area 110 a rectangle inside the digital canvas 108 indicating the area in which the user wishes to have a person there.

Moreover, as used herein, the term "digital canvas" refers to a digital area in which a user can specify or enter a query area and / or a query term. In particular, the term "digital canvas" includes a graphical user interface element that includes a visual representation of a searched digital image for inputting a query region indicating a search region and a search term indicating a searched visual content. For example, a digital canvas includes a digital, two-dimensional representation of a field with which a user may interact to provide user input of a query area and / or a query term. Consequently, the digital canvas includes 108 in terms of the 1C a field for entering the query terms 110a . 112a and the query areas 110 . 112 , In one or more embodiments, a digital canvas may be used 108 have a size / shape corresponding to a digital image or other digital visual media.

As it is in the 1C is shown, the spatial-semantic media search system performs a search on the basis of the digital screen 108 by. In particular, the spatial-semantic media search system performs a search based on the first one query range 110 , the first query term 110a , the second query range 112 and the second query term 112a through to identify digital images. In particular, the spatial-semantic media search system performs a search to identify digital images that contain searched visual content corresponding to the first query term 110a within a searched range corresponding to the first query range 110 and to present a searched visual content corresponding to the second query term 112a within a searched range corresponding to the second query range 112 equivalent. Thus, as shown, the spatial-semantic search results include 106a digital pictures with a person holding a tennis racket to the left. In particular, the digital images in the spatial-semantic search results 106a a person within a first searched area that corresponds to the first query area 110 and a tennis racket within a second searched area corresponding to the second query area 112 corresponds, again.

Accordingly, a user designing a particular page layout (ie, a page layout that requires a tennis player holding a tennis racket to the left) may simply be a user input of the first polling area 110 , the first query term 110a , the second query range 112 and the second query term 112a by means of the digital screen 108 provide. In response, the spatial semantic media search system may identify and provide an indication of a plurality of digital images that match the requested semantic and spatial features.

As mentioned above, in one or more embodiments, the spatial semantic media search system selects digital images corresponding to an interrogation area and a query term of a digital canvas through the use of a neural interrogation network and a neural digital image network. In particular, the spatial semantic media search system generates a query feature set based on a digital canvas by providing the digital canvas for a neural interrogation network. Accordingly, the spatial semantic media search system generates digital image feature sets by providing digital images for a neural digital image network. In addition, the spatial semantic media search system may compare the query feature set and the digital image feature sets to select digital images corresponding to the digital canvas. For example, the shows 2 4 is an illustration of identifying digital images based on a neural interrogation network and a neural digital image network in accordance with one or more embodiments.

As used herein, the term "neural interrogation network" refers to a neural network that generates a feature set based on spatial and semantic information. In particular, the term "neural interrogation network" includes a neural convolution network that generates a query feature set based on a query term and an interrogation area. Additional details regarding example embodiments of a neural interrogation network are provided below.

As used herein, the term "neural digital image network" refers to a neural network that generates a feature set based on a digital image. In particular, the term "neural digital image network" includes a neural convolution network that generates a digital image feature set based on a digital image as input to the neural convolution network. The spatial semantic media search system may also utilize a neural digital media network that generates a feature set based on a digital media item (e.g., a feature set based on one or more representative frames of a digital video). Additional details regarding example embodiments of a neural digital image network are given below.

As used herein, the term "query feature set" refers to a digital object that has been generated by a neural interrogation network based on a query term and an interrogation area. In particular, the term "query feature set" may include one or more feature vectors that have been or are being generated by a neural convolution network and that represent or reproduce spatial and semantic information. For example, a query feature set may include a feature set generated by a level of a neural convolution network that represents a representation of features that correspond to a query term and a query scope. For example, the query feature set may include a collection of feature vectors representing a query term and a query scope, the query feature set having the same dimensionality as a digital image feature set.

As used herein, moreover, the term "digital image feature set" refers to a digital object that has been generated by a digital image based neural digital image network. In particular, the term "digital image feature set" includes one or more feature vectors that are or have been generated by a neural convolution network and that render or reproduce features of the digital image. For example, a query feature set may include a feature set generated by a plane of a neural convolution network that reproduces semantic information and spatial information from the digital image (e.g., a feature set at a high level of a convolutional neural network as opposed to a fully connected plane). For example, the digital image feature set may comprise a collection of feature vectors representing a digital image, wherein the digital image feature set has the same dimensionality as a query feature set.

To illustrate, the 2 a representation of identifying digital images that corresponds to a digital canvas, using a neural interrogation network and a neural digital image network. In particular, the shows 2 a digital canvas 202 containing a query area 202a and a corresponding query term 202b includes. In addition, the shows 2 a memory content for digital pictures 208 that has a majority of digital pictures 208a to 208n includes. On the basis of the digital screen 202 The spatial-semantic media search system performs a search of the storage contents for digital images 208 according to digital images, the query term 202b within the query range 202a play.

In particular, the spatial-semantic media search system, as described in the 2 shown is a step 220 for generating a query feature set from the query area and the query term by means of a neural interrogation network. As it is in the 2 and the description below, the spatial semantic media search system performs the step 220 by providing the digital canvas 202 for a neural interrogation network 204 for generating a query feature set 206 by.

The spatial-semantic media search system provides the digital images 208a to 208n the memory contents for digital pictures 208 for a neural digital image network 210 for generating a plurality of digital image feature sets 212a to 212n ready. Further, the spatial semantic media search system compares the query feature set 206 and the plurality of digital image feature sets 212a to 212n for identifying digital images 214a to 214c that the digital canvas 202 correspond.

With respect to the embodiment of 2 includes the digital canvas 202 the query area 202a and the query term 202b that represent a searched area or visual content. As mentioned above, the digital canvas can 202 include any number or type of query areas and corresponding query terms. For example, the digital canvas 202 instead of a single query term and a single query area, comprise a plurality of query terms and a plurality of query areas.

As it is in the 2 is shown, the spatial-semantic media search system is the digital canvas 202 for the neural interrogation network 204 ready (ie, as part of the step 220 for generating a query feature set from the query area and the query term using a neural interrogation network). In particular, the spatial-semantic media search system represents the digital screen 202 for the neural interrogation network 204 by generating a representation of the query area 202a and the query term 202b ready. In particular, the spatial-semantic media search system represents the digital screen 202 as a three-dimensional grid and provides the three-dimensional grid as input to the trained neural interrogation network 204 ready.

For example, the spatial-semantic media search system converts the query term 202b using a word-to-vector algorithm into a query term vector. In addition, the spatial-semantic media search system then populates elements (eg, spatial locations) of the three-dimensional grid, the query area 202a corresponds to the query term vector. Although the spatial semantic media search system may generate a three-dimensional grid of many different sizes, the spatial-semantic media search system for illustration in one or more embodiments generates a three-dimensional 31 x 31 x 300 grid that represents the digital canvas. Additional details relating to generating a three-dimensional grid are given below with respect to FIGS 3 specified.

As it is in the 2 shown uses the spatial-semantic media search system as part of the step 220 the neural interrogation network 204 for generating the query feature set 206 on the basis of the digital screen 202 , In particular, the neural interrogation network 204 from 2 a generative convolution model (ie, a neuronal convolution network). In particular, the neural interrogation network 204 a convolutional neural network that utilizes the input of a three-dimensional grid (eg, a three-dimensional 31 x 31 x 300 grid) and generates a query feature set as output. In particular, the spatial semantic media search system generates a query feature set having a similar dimensionality as the digital image feature sets (eg, a 7x7x832 feature set).

The spatial semantic media search system uses a neural interrogation network of various forms to generate query feature sets. With respect to the embodiment of 2 and as part of the step 220 includes the neural interrogation network 204 a generative convolution model with three levels of convolution, which are nested by two max-pooling and two subsampling levels. In particular, Table 1 includes the detailed architecture of the neural interrogation network 204 with respect to the embodiment of 2 , TABLE 1 level Type Number of features Receptive field step 1 folding 256 3x3 1 2 Max pooling 256 2x2 2 3 folding 512 3x3 1 4 Max Poolinq 512 2x2 2 5 folding 832 2x2 1

In addition to using the neural interrogation network 204 to generate the query feature set 206 as it is in the 2 is shown, the spatial-semantic media search system also uses the neural digital image network 210 for generating the digital image feature sets 212a to 212n , In particular, the spatial semantic media search system provides the plurality of digital images 208a to 208n for the neural digital image network 210 for generating the digital image feature sets 212a to 212n ready.

Regarding the 2 includes the neural digital image network 210 a deep neural folding network. In particular, the neural digital image network includes 210 a deep neural convolution network with a plurality of levels (ie, high level levels, max pool levels, and fully connected levels). For example, the neural digital image network includes 210 in one or more embodiments, the neural convolution network GoogLeNet, which is incorporated in C. Szegedy et al., "Going deeper with convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pages 1-9 , the entire contents of which are incorporated herein by reference.

GoogLeNet is a neural folding network with a specific architecture. In particular, GoogLeNet generally includes a root, a plurality of starting modules and an output classifier. The strain comprises a successive chain of processes of folding, pooling and normalizing a local response. The initial modules each comprise a set of convolutions and poolings of different scales, each being executed in parallel and then linked together. For example, GoogLeNet uses 22 startup modules. The output classifier performs an average pooling followed by a fully connected level.

In one or more embodiments, the spatial semantic media search system utilizes the neural digital image network 210 for generating the digital image feature sets 212a to 212n by using a feature set that is at a high level within the neural digital image network 210 has been generated. For example, the neural digital image network becomes 210 in one or more embodiments for predicting item classifications represented in a digital image. In one or more embodiments, instead of receiving a classification from the neural digital image network, the spatial semantic media search system obtains a feature set from a level of the neural digital image network and uses the feature set as one of the digital image feature sets 212a to 212n ,

In particular, as described above, the neural digital image network includes 210 in one or more embodiments, a plurality of high level setup levels, max pooling levels, and fully connected levels. The high level circuitry levels within the neural digital image network tend to retain both spatial information (ie, information regarding the location of objects in the digital image) and semantic information (ie, information regarding the classification or marking of objects in the digital image) as opposed to fully connected layers that focus on the semantic information for classifying the objects shown in the digital image. Accordingly, in one or more embodiments, the spatial semantic media search system utilizes a feature set determined at a high-level setup level that preserves both spatial information and semantic information. By way of illustration, the spatio-semantic uses Media search system in relation to the 2 a feature set from the fourth initial module of the GoogLeNet architecture for generating the digital image feature sets 212a to 212n ,

In one or more embodiments, the spatial semantic media search system also trains a neural digital image network. For example, the spatial-semantic media search system trains with respect to the 2 the neural digital image network 210 by means of a set of digital training images with one or more known classifications (eg known objects shown in the digital image). The spatial semantic media search system provides the set of digital training images for the neural digital image network 210 ready and the neural digital image network 210 generates predicted classifications of the digital training images (eg, an attempt to classify objects shown in the digital training images). The spatial-semantic media search system trains the neural digital image network 210 by comparing the predicted classification with the actual classification of the items shown in the digital image. Although the spatial semantic media search system may utilize any variety of digital training images or storage contents of digital training images, the spatial semantic media search system is practicing with respect to the embodiment of FIG 2 the neural digital image network 210 by providing the neural digital image network with the ImageNet image data set - cf. J. Deng, W. Dong, R. Socher, Li Li, Kai Li and Li Fei-Fei, "ImageNet: A large-scale hierarchical image database", Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, Miami, FL, 2009, pages 248-255 the entire contents of which are incorporated herein by reference.

As it is in the 2 3, the spatial-semantic media search system compares after generating the digital image feature sets 212a to 212n and the picture feature set 206 the query feature set and the digital image feature sets 212a to 212n to identify digital images, the digital canvas 202 correspond. In particular, the spatial semantic media search system compares the feature sets by determining a distance between the query feature set and the digital image feature sets 212a to 212n , In particular, the feature sets in relation to the embodiment of FIG 2 Feature vectors. Accordingly, the spatial semantic media search system compares the feature sets by determining a cosine distance between the feature vectors.

Based on the comparison between the query feature set 206 and the plurality of digital image feature sets 212a to 212n The spatial-semantic media search system identifies the digital images 214a to 214c that the digital canvas 202 correspond. In particular, the spatial semantic media search system identifies the digital images 214a to 214c that show a visual content that matches the query term 202b within an area corresponding to the query area 202a equivalent. By way of illustration, the spatial semantic media search system classifies digital images based on the comparison between the query feature set 206 and the plurality of digital image feature sets 212a to 212n (ie, classifies the digital images based on the distance between the query feature set and the corresponding digital image feature sets). In addition, the spatial semantic media search system provides the best classified digital images 214a to 214c ready for an ad (ie, the best percentage or the best number of digital images). Additional details regarding a graphical user interface for providing digital image search results for display are described below with respect to FIGS 5A to 5C specified.

With reference to the 3 Additional details regarding the generation of a representation of a digital canvas according to one or more embodiments are provided. As discussed above, the spatial semantic media search system generates a representation of a query term and an interrogation area to provide a neural interrogation network. The 3 shows the generation of a query area and a query term from a digital canvas. In particular, the shows 3 a digital canvas 300 containing a first query area 302 with a first query term 302a (ie, "window"), a second polling area 304 with a second query term 304a (ie, "wall") and a third polling area 306 with a third query term 306a (ie, "bed").

As shown, the spatial semantic media search system extracts the query terms 302a . 304a . 306a from the digital screen 300 and applies a word-to-vector algorithm 308 at. A word-to-vector algorithm generates a vector representation relating to a linguistic context of a term or terms. In particular, a word-to-vector algorithm for generating a vector from words or sentences is trained, the resulting vector indicating a linguistic context of words or sentences. For example, a word-to-vector algorithm training input may use a memory content of text and associate each word with a multidimensional space. In particular, the word-to-vector algorithm assigns each word in the training memory content of text a corresponding vector in the multidimensional space. The word-to-vector algorithm positions the word vectors in the space such that words having a corresponding linguistic context / meaning in the space are located very close to each other. Accordingly, a word-to-vector algorithm may generate vectors that represent a linguistic meaning of one or more input terms. The spatio-semantic Media search system may use any of various word-to-vector algorithms. Regarding the 3 uses the spatial-semantic media search system "Word2vec" as a word-to-vector algorithm 308 , as in Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Jeff Dean ( 2013 ), "Distributed representations of words and phrases and their compositionality", NIPS 2013 , the entire contents of which are incorporated herein by reference.

As shown, the spatial semantic media search system applies the word-to-vector algorithm 308 on the query terms 302a . 304a . 306a and generates a first query term vector 310 , a second query term vector 312 and a third query term vector 314 , As used herein, the term "query term vector" refers to a vector representation of a word. In particular, the term "query term vector" includes a vector representation of a linguistic meaning of a query term. Accordingly, the first query term vector comprises 310 a vector representation of the linguistic meaning of the query term 302a , Accordingly, the second query term vector comprises 312 a vector representation of the linguistic meaning of the query term 304a and the third query term vector 314 includes a vector representation of the linguistic meaning of the query term 306a ,

After generating the query term vectors 310 to 314 The Spatial Semantic Media Search System encodes the query term vectors in a three-dimensional grid 320 , As used herein, the term "three-dimensional grid" refers to a digital object that represents three variables. Accordingly, the term "three-dimensional grid" includes a matrix, a database, or a table that includes data that represents three variables. For example, a three-dimensional grid includes a matrix of data representing a position in a first direction, a position in a second direction, and a query term vector. By way of illustration, a three-dimensional grid may comprise a matrix of query term vector values embedded with respect to a spatial location of a digital canvas.

For example, the shows 3 the three-dimensional grid 320 in the form of a three-dimensional matrix, wherein two dimensions of the x-position and the y-position (ie, a spatial position) of the digital canvas 300 and a third dimension matches values of query term vectors. In particular, the spatial-semantic media search system generates the three-dimensional grid 320 by associating the query term vectors 310 to 314 to the spatial positions of the corresponding query areas 302 to 306 , As a result, the spatial-semantic media search system orders the first query term vector 310 spatial locations corresponding to the first query area 302 match, assigns the second query term vector 312 the second query area 304 and assigns the third query term vector 314 the third query area 306 to. Accordingly, the three-dimensional grid 320 a digital representation of both the query areas 302 to 306 as well as the query terms 302a to 306a ,

Although the 3 the three-dimensional grid 320 Having a particular structure (eg, seven columns and five rows), the spatial-semantic media search system can create a three-dimensional mesh that has many different structures. For example, in one or more embodiments, the spatial semantic media search system generates a three-dimensional grid that includes a 31x31x300 canvas image. In addition, as described above, the spatial semantic media search system may provide the 31x31x300 screen representation for a neural interrogation network to produce a 7x7x832 feature set that is dimensionality of feature sets generated by GoogLeNet to display digital images.

As mentioned above, in addition to using a neural interrogation network in one or more embodiments, the spatial semantic media search system also trains a neural interrogation network. In particular, the spatial semantic media search system trains a neural interrogation network to generate a query feature set based on a representation of a digital canvas. In particular, in one or more embodiments, the spatial semantic media search system trains a neural interrogation network by providing a plurality of training terms and training areas corresponding to a plurality of digital training images. The neural interrogation network generates predicted feature sets based on the training terms and the training areas. In addition, the spatial semantic media search system trains the neural interrogation network using a training structure that compares the predicted feature sets to real feature sets that correspond to the digital training images.

As used herein, the term "digital training image" refers to a digital image used to train a neural network. In particular, a digital training image includes a digital image that displays known visual content in a particular area of the digital image (eg, an identified object that has a known subject mask within the digital image). As described below, the spatial semantic media search system may utilize the known visual content and the neural network training area. In particular, the spatial semantic media search system may identify a training term and a training area and provide the training term and training area for the neural interrogation network. In addition, the spatial semantic media search system may utilize a feature set corresponding to the digital training image to train the neural network to predict more accurate feature sets.

As used herein, the term "training area" refers to an area provided for a neural network to train the neural network. In particular, the term "training area" refers to an area of a digital training image that contains a known visual content. For example, the term "training area" includes an object mask or other border that corresponds to a visual content (e.g., an item) shown in a digital training image.

In addition, the term "training term" refers to a word or phrase that describes a visual content of a digital training image. In particular, the term "training term" includes a word or phrase that describes a visual content that is within a training range of a digital training image. Consequently, e.g. with respect to a digital training image showing an automobile, the spatial semantic media search system identifies a training term (i.e., "car") and training range (i.e., an area of the digital training image that includes the car).

For example, the show 4A and 4B training a neural interrogation network according to one or more embodiments of the spatial semantic media search system. In particular, the shows 4A a first training area 402a , the first training concept 402b corresponds to a second training area 404a , the second training concept 404b corresponds, and a third training area 406a , the third training concept 406b equivalent. Each of the training areas 402a to 406a and the training terms 402b to 406b corresponds to a visual content of the digital training images 402 to 406 , For example, the training area corresponds 402a and the training concept 402b the digital training image 402 Being a person within the training area of the digital training image 402 shows.

The spatial-semantic media search system receives the digital training images 402 to 406 from a memory content of digital training images. In particular, the spatial semantic media search system accesses a memory content of digital training images and selects the digital training images 402 to 406 along with information from, the items included in the digital training images 402 to 406 are shown, and the place in the digital training images 402 to 406 specify items shown. The spatial-semantic media search system then generates the training terms 402b to 406b and the training areas 404a to 406a on the basis of the information indicating the objects shown in the digital images and the location of the objects shown.

In addition, each of the digital training images has a corresponding digital training image feature set. For example, the first training area correspond 402a and the first concept of training 402b (ie, "person") a first feature set 402c the first digital training image 402 , Accordingly, the second training area correspond 404a and the second concept of training 404b (ie, "car") a second feature set 404c a second digital training image 404 ,

The spatial semantic media search system obtains or generates the digital training image feature sets 402c to 406c , For example, in one or more embodiments, the spatial semantic media search system accesses a memory content of digital images that already contains the feature sets 402c to 406c includes. In other embodiments, the spatial semantic media search system generates the feature sets 402c to 406c (eg by providing the digital training images 402 to 406 for a digital image interrogation network).

As it is in the 4A is shown, the spatial-semantic media search system sets the training areas 402a to 406a and the training terms 402b to 406b for a neural interrogation network 410 ready (eg the neural interrogation network 204 which has been described above). As with the use of a neural interrogation network to generate a query feature set, the spatially-semantic media search system may provide a representation of a neural interrogation network when training a neural interrogation network Create a training area and a training term and provide the representation of the training area and the training concept for the neural interrogation network. For example, the spatial-semantic media search system may generate a three-dimensional grid (eg, the three-dimensional grid 320 ), which corresponds to each training area and training term, and provide the three-dimensional grid for a neural interrogation network.

For example, the spatial semantic media search system generates with respect to the 4A a three-dimensional grid that is the first training area 402a and the first concept of training 402b equivalent. In particular, the spatial-semantic media search system transforms the first concept of training 402b using a word-to-vector algorithm into a training term vector. The spatial-semantic media search system then encodes the training term vector to a spatial area of a three-dimensional grid that corresponds to the training area 402a equivalent. The spatial-semantic media search system then provides the three-dimensional grid for the neural interrogation network 410 ready. The spatial semantic media search system may generate and provide representations of the remaining training area and training terms accordingly.

Furthermore, the spatial-semantic media search system, as used in the 4A after obtaining a training area and a training term (ie, a representation of the training area and the training term), the neural interrogation network is shown 410 for generating predicted feature sets. In particular, the spatial semantic media search system utilizes the neural interrogation network 410 for generating a first predicted feature set 412 , a second predicted feature set 414 and a third predicted feature set 416 that the training areas 402a to 406a and the training terms 402b to 406b correspond. In particular, the spatial semantic media search system generates the predicted feature sets 412 to 416 such that they have the same dimensionality as the feature sets 402c to 406c that the digital training images 402 to 406 correspond. By way of illustration, the spatial semantic media search system generates the predicted feature sets 412 to 416 with a dimensionality of 7 x 7 x 832, the feature sets 402c to 406c corresponds to the digital training images, which also have a dimensionality of 7 x 7 x 832.

After generating the predicted feature sets 412 to 416 the spatial semantic media search system uses the feature sets 402c to 406c that the digital training images 402 to 406 to train the neural interrogation network. In particular, the shows 4A a training structure 420 which the predicted feature sets 412 to 416 for generating a trained neural interrogation network 430 uses. In particular, the training structure includes 420 a first loss function 422 , a second loss function 424 and a third loss function 426 ,

As shown, every loss function compares 422 to 426 predicted feature sets with the actual feature sets corresponding to each digital training image. For example, the spatial semantic media search system compares with respect to the first loss function 422 the predicted first feature set 412 and the first feature set 402c that the first digital training image 402 correspond. Accordingly, the spatial-semantic media search system compares with respect to the second loss function 424 the predicted second feature set 414 and the second feature set 404c that the second digital training image 404 correspond.

The spatial-semantic media search system compares the predicted feature sets 412 to 416 with the feature sets 402c to 406c by means of different loss functions. For example, in one or more embodiments, the spatial semantic media search system utilizes a loss function (ie, minimizes a loss function) that determines the distance between the predicted feature sets 412 to 416 and the feature sets 402c to 406c compares. To illustrate with respect to the embodiment of 4A include the predicted feature sets 412 to 416 and the feature sets 402c to 406c Feature vectors; accordingly, the spatial semantic media search system compares the predicted feature sets 412 to 416 and the feature sets 402c to 406c by determining a distance (eg, a cosine distance) between feature vectors.

Although the 4A the spatial-semantic media search system shows that three loss functions 422 to 426 , the three different digital images 402 to 406 and three different feature sets 402c to 406c Simultaneously analyzed, the spatial-semantic media search system can analyze digital images simultaneously or sequentially. For example, the spatially-semantic media search system trains with respect to the embodiment of FIG 4A the neural interrogation network 410 by means of the loss function 422 , the feature set 402c and the predicted feature set 412 in a first training query. Thereafter, the spatial-semantic media search system trains the neural interrogation network 410 by means of the second loss function 424 , the second feature set 404c and the predicted feature set 414 in a second training query. In particular, the spatial semantic media search system utilizes the embodiment of FIG 4A a "Stochastic Gradient Descend" algorithm for training the neural interrogation network 410 by minimizing the accumulated stochastic loss set for each training query.

Training structures and the loss functions may also be described as pseudocode and / or equations implemented by a computing device to minimize the accumulated stochastic loss. For example, the spatial semantic media search system minimizes a similarity loss function that reduces the cosine distance between a predicted feature set, F _q (eg, the predicted feature set 412 ), with a known feature set, F _{I q} (eg the feature set 402c ), the digital training image, I _q (eg the digital training image 402 ), compares. In particular, in one or more embodiments, the spatial semantic media search system utilizes the following similarity loss function: $$L_s\left(F_q\right)=1-\text{cos}\left(F_q.F_{1_q}\right)$$

Minimizing this loss function supports a neural interrogation network (eg, the neural interrogation network 410 ) in predicting feature set F _q as closely as possible to the true ("ground-truth") digital training image (eg, the digital training image 402 ) on which the training area (eg the training area 402a ) and the training concept (eg the training concept 402b ). In other words, minimizing the loss function (using a Stochastic Gradient Descend algorithm) creates a trained neural network that will produce feature sets that are closer to the searched visual content within searched ranges of searched digital images.

The spatial semantic media search system may use various digital training images (with corresponding training terms and training areas) to form the trained neural interrogation network 430 to create. For example, in one or more embodiments, the spatial semantic media search system utilizes a memory content of digital images that includes thousands of searched digital images having known visual content and known regions (ie, digital images with known objects and meshes). By way of illustration, in one or more embodiments, the spatial semantic media search system utilizes a combination of MS COCO and visual genome data sets. The spatial semantic media search system may also utilize other storage contents of digital images, such as digital images managed by the ADOBE STOCK® software and a digital image database.

In one or more embodiments, the spatial semantic media search system also utilizes a spatial mask when training a neural interrogation network. In particular, the spatial semantic media search system may apply a spatial mask to feature sets of digital training images to focus training of the neural interrogation network on a training area. In particular, in one or more embodiments, the spatial semantic media search system determines an article boundary of an object shown in a digital training image and applies a spatial mask to an area outside the article boundary to produce a masked feature set. The spatial semantic media search system may then use the masked feature set to train the neural interrogation network.

For example, the shows 4A applying a spatial mask to a feature set to generate a masked feature set according to one or more embodiments. In particular, the spatial semantic media search system uses a spatial mask 432 on the second feature set 404c at. In particular, the spatial-semantic media search system determines an item boundary 428 of an object (eg the car) in the digital training image 404 shown is the second feature set 404c equivalent. In addition, the spatial-semantic media search system uses the spatial mask 432 to an area outside the subject line 428 so that the second feature set 404c only features within the item boundary 428 includes. In this way, the spatial semantic media search system removes another visual content contained in the digital training image 402 shown to the neural interrogation network 410 Focuss faster on features that are within the second training range 404a contained (rather than on irrelevant features outside of the searched Range). By applying a spatial mask, the spatial semantic media search system can significantly improve the speed of training a neural interrogation network.

Although in the 4A just applying a spatial mask to the digital training image 404 shown is the second feature set 404c Similarly, when training a neural interrogation network, the Spatial Semantic Media Search System may apply a spatial mask to additional (or all) digital images. In addition, although the 4A a certain subject line 428 shows (ie, an object mask that follows very closely the contours of a car, in the digital training image 404 shown) and the spatial mask 432 to an area outside the subject line 428 applies, the spatial-semantic media search system uses many different subject boundaries. For example, in one or more embodiments, the spatial-semantic media search system uses an article boundary based on a searched area (eg, the subject boundary 428 equivalent to the training area 404a and the spatial semantic media search system applies a spatial mask to an area outside of the training area 404a at).

As mentioned above, in one or more embodiments, the spatial semantic media search system further trains a neural interrogation network to support optimal retrieval performance. In particular, the spatial semantic media search system may train a neural interrogation network to not only generate query feature sets that are similar to searched digital images, but also to distinguish between irrelevant digital images or queries. In particular, in one or more embodiments, the spatial semantic media search system not only utilizes a similarity loss function (as described above) but also utilizes an image-based rating loss function and / or a query-based rating loss function in training a neural interrogation network.

For example, the shows 4B training a neural interrogation network using a similarity loss function, an image-based rating loss function, and a query-based rating loss function according to one or more embodiments of the spatial-semantic media search system. In particular, the shows 4B a training area 440a and a training concept 440b , In particular, the training area correspond 440a and the training concept 440b a digital training image 440 that has a feature set 440c (ie, the digital training image 440 shows the training area term 440b , a person, within the training area 440a as it is in the training feature set 440c is shown)

As shown, the spatial semantic media search system provides the training area 440a and the training concept 440b for the neural interrogation network 442 ready. In addition, training trains the neural interrogation network 442 a predicted feature set 444 , The spatial-semantic media search system then uses a training structure 446 for generating a trained neural interrogation network 460 , In particular, the training structure includes 446 a query-based rating loss function 448 , a similarity loss function 450 and an image-based rating loss function 452 ,

As described above, the similarity loss function includes 450 a comparison between the predicted feature set 444 and the feature set 440c that corresponds to the digital training image. The similarity loss function 450 therefore gives a measure of similarity between the digital training image 440 that for creating the training area 440a and the training concept 440b is used and the predicted feature set 444 again. In addition, minimization has the similarity loss function 450 the effect of learning to the neural interrogation network how to create feature sets that are similar to feature sets of digital training images.

In addition to the similarity loss function 450 includes the training structure 446 also the query-based rating loss function 448 , The spatial-semantic media search system uses the query-based ranking loss function 448 to support an appropriate ranking via a set of queries with a referenced digital image. In other words, the spatial semantic media search system uses the query-based rating loss function 448 to train the neural interrogation network 442 for generating a query feature set that is not only similar to searched digital images but is also different from irrelevant digital images. As shown, the query-based rating loss function includes 448 a comparison between the predicted feature set 444 and a negative digital image feature set 454 based on a negative digital image.

As used herein, the term "negative digital image" refers to a digital image that is different from a digital training image. In particular, the term "negative digital image" includes a digital image representing a visual content other than a training term describing a visual content shown in a digital training image. For example, if a digital training image shows a cat, a negative digital training image would include a digital image showing a dog (ie, not a cat).

As used herein, the term "negative digital image feature set" refers to a feature set that has been generated from a negative digital image. In particular, the term "negative digital image feature set" includes one or more feature vectors that render or reproduce a negative digital image. For example, a negative digital image feature set includes a feature set that has been generated by a neural digital image network based on input of a negative digital image.

In one or more embodiments, the spatial semantic media search system generates, determines, and / or identifies a negative digital image and / or a negative digital image feature set. For example, the spatially-semantic media search system searches with respect to the embodiment of FIG 4B a memory content of digital training images for identifying the negative digital image feature set 454 , In particular, the spatial semantic media search system performs a search of a memory content of digital training images based on the training term 440b to identify digital training images that do not show visual content, the training concept 440b equivalent. For example, there the training concept 440b "Person", the spatial-semantic media search system performs a search for a digital training image that shows visual content that does not include a person. After identifying a negative digital training image, the spatial semantic media search system also identifies or generates a corresponding negative digital training image feature set (eg, from the memory content of digital training images or by using a neural digital image network).

In one or more embodiments, the spatial semantic media search system may also select a negative digital image based on a query region. For example, the spatial semantic media search system performs in relation to the 4B a search for digital training images (ie, from a memory content of digital training images) that provides visual content within the training area 440a show, wherein the visual content does not include a person. The spatial-semantic media search system can thus identify a negative digital image that includes visual content at a spatial location that belongs to the training area 440a equivalent. In this way, the spatial semantic media search system may have some concept overlap (ie, spatial similarity) between the negative digital training image feature set and the feature set 440c of the digital training image 440 introduce. In addition, the spatial-semantic media search system can use the neural interrogation network 442 train to discriminate between digital images containing different visual content at the same (or similar) spatial location.

The spatial semantic media search system may also use a neural interrogation network to select negative digital images. For example, in one or more embodiments, the spatial semantic media search system determines the training of digital images that comprise a neural interrogation network (eg, the neural interrogation network 442 ), and then uses these digital training images to further train the neural interrogation network. In particular, the spatial-semantic media search system, as already described in relation to the 2 has been described using a neural interrogation network to generate a query feature set. In addition, the spatial semantic media search system may use the query feature set to select digital images from a memory content of digital training images. The spatial-semantic media search system may then analyze the digital training images selected by the neural interrogation network (eg, the best one hundred selected digital images) to determine irrelevant digital images that the neural interrogation network has selected (eg, low-training digital training images) Similarity, which nevertheless identified the neural interrogation network as similar). The spatial semantic media search system may then use the digital training images selected by the neural interrogation network in an inappropriate manner as negative digital training images.

For example, the neural interrogation network identifies 442 by means of the approach used in the 2 described, an irrelevant digital training image as the most relevant digital image. The spatial-semantic media search system then uses the irrelevant digital training image to further train the neural interrogation network 442 , In this way, the spatial semantic media search system identifies digital training images that comprise the neural interrogation network 442 difficult to distinguish accurately and then uses these digital training images to the neural interrogation network 442 to learn to generate query feature sets more accurately and efficiently.

After identifying the negative digital image feature set, the spatial semantic media search system in one or more embodiments compares the negative digital image feature set and a predicted feature set. For example, the spatial semantic media search system uses a loss function that measures a distance (eg, a cosine distance) between the predicted feature set and the negative digital image feature set. As it is in the 4B 3, the spatial semantic media search system utilizes the query-based rating loss function 448 containing the negative digital image feature set 454 and the predicted feature set 444 compares.

In addition to comparing a negative digital image feature set and a predicted feature set, the spatial semantic media search system may also compare the difference between a negative digital image feature set and a predicted feature set with the difference between a feature set of a digital training image and a predicted feature set. In this way, the spatial-semantic media search system supports the neural interrogation network in distinguishing between digital images and negative digital images.

For example, the spatial-semantic media search system compares with respect to the 4B a difference between the negative digital image feature set 454 and the predicted feature set 444 with a difference between the feature set 440c of the digital training image 440 and the predicted feature set 444 , In other words, the query-based ranking loss function 448 measures the fortune of the neural interrogation network 442 for discriminating between the negative digital image feature set and the feature set of the digital training image. By minimizing the query-based rating loss function 448 The spatial-semantic media search system supports the neural interrogation network 442 for generating query feature sets that are not only similar to searched digital images but are also not similar to irrelevant digital images.

wobei F_{I q} den negatives digitales Bild-Merkmalssatz bezeichnet (z.B. den negatives digitales Bild-Merkmalssatz 454), der von einem negativen digitalen Trainingsbild, I_q̅, extrahiert worden ist. Das Minimieren dieses Verlusts unterstützt die geeignete Einstufung über einen Satz von Abfragen bezüglich eines referenzierten digitalen Bilds.A query-based rating loss function may also be described in terms of a pseudocode and / or equations implemented by a computing device to minimize query-based rating loss. For example, in one or more embodiments, the spatial semantic media search system uses the following query-based rating loss function: $$L_{r_q}\left(F_q\right)=\text{Max}\left(0\alpha -COs\left(F_q.F_{1_q}\right)+COs\left(F_q.F_{1_{\overline{q}}}\right)\right)$$

where F _{I q} denotes the negative digital image feature set (eg, the negative digital image feature set 454 ) extracted from a negative digital training _image , I _q̅ . Minimizing this loss assists the proper ranking over a set of queries regarding a referenced digital image.

As it is in the 4B In addition to the query-based rating loss function, the spatial semantic media search system utilizes 448 also an image-based rating loss function 452 , The spatial-semantic media search system uses the image-based ranking loss function 452 to support appropriate grading of images relative to a query feature set. In other words, the image-based ranking loss function supports the generation of query feature sets that not only accurately represent digital images that represent particular terms but that also distinguish between images that represent irrelevant query terms. As shown, the image-based grading loss function includes 452 a comparison between the predicted feature set 444 and a negative training term feature set 458 that of a negative training term 462 has been generated.

As used herein, the term "negative training term" refers to a word or phrase that differs from a training term. In particular, the term "negative training term" includes a word or sentence having a linguistic meaning different from a training term. For example, if the spatial-semantic media search system uses the term "hot" would use a negative term "cold" (or a term or phrase that is different from hot).

As used herein, the term "negative training term feature set" refers to a feature set that has been generated from a negative training term. In particular, the term "negative training term feature set" includes one or more feature vectors that represent or reproduce a negative training term. For example, a negative training term feature set includes a feature set that has been generated by a neural interrogation network with a negative training term as input.

In one or more embodiments, the spatial semantic media search system generates, determines, and / or identifies a negative training term. By way of illustration, the spatial-semantic media search system chooses with respect to the 4B the negative training concept 462 (ie, "candy") based on the training concept 440b (ie, "person"). In particular, the spatial-semantic media search system randomly selects a term that differs from the training term 440b different.

In addition, in one or more embodiments, the spatial semantic media search system utilizes a neural interrogation network (eg, the neural interrogation network 442 ) for selecting a negative training item. For example, in one or more embodiments, the spatial semantic media search system determines negative training terms that are difficult for a neural interrogation network to distinguish, and then uses these negative training terms to further train the neural interrogation network.

In particular, as it regards the 2 has been discussed, the spatial-semantic media search system uses a neural interrogation network to generate a query feature set. In addition, the spatial semantic media search system may use the query feature set to select digital images from a memory content of digital training images. The spatial semantic media search system may then analyze the digital training images selected by the neural interrogation network to determine irrelevant visual concepts shown in the selected digital training images. The spatial-semantic media search system may then use the irrelevant visual concepts selected by the neural interrogation network as negative training terms.

For example, the neural interrogation network 442 by means of the approach associated with the 2 in a memory content of digital training images search for digital images showing a person. In addition, the neural query network 442 identify an image of a candy as the best resulting digital image. The spatial semantic media search system may determine that the item candy shown in the resulting digital image is irrelevant. In response, the spatio-semantic media search system can use the irrelevant training concept "candy" as a negative training concept when training the neural interrogation network 442 use.

After identifying a negative training term, the spatial semantic media search system may also generate a negative training term feature set. For example, as it is in the 4B the spatial semantic media search system shows the neural interrogation network 442 for generating the negative training term feature set 458 use. In particular, the spatial semantic media search system makes reference to the embodiment of FIG 4B the negative training concept 462 for the neural interrogation network 442 ready. In particular, the spatial-semantic media search system provides the negative concept of training 462 for the neural interrogation network 442 along with the training area 440a ready (eg as a three-dimensional grid that includes a vector containing the negative training subject 462 represents the training area 440a is assigned).

As it is in the 4B 3, the spatial semantic media search system may then use the neural interrogation network 442 use the negative training term feature set 458 to create. In addition, the spatial semantic media search system may then use the negative training term feature set 458 in the image-based classification loss function 452 use. In particular, the spatial semantic media search system may use the negative training term feature set 458 with the predicted feature set 444 to compare. For example, the spatial semantic media search system determines a distance between the negative training term feature set 458 and the predicted feature set 444 ,

In addition, in one or more embodiments, the spatial semantic media search system uses a loss function that compares the difference between the negative training term feature set and the predicted feature set with a difference between the digital training image feature set and the predicted feature set. For example, the spatial-semantic media search system uses in relation to the 4B the image-based rating loss function 452 representing a difference between the negative training term feature set 458 and the predicted feature set 444 with a difference between the feature set 440c and the predicted feature set 444 compares. Using this approach, the spatial semantic media search system measures the capacity of the neural interrogation network 442 for differentiating between the negative training term feature set and the feature set of the digital training image. In particular, the spatial semantic media search system assists the neural interrogation network by minimizing the image-based loss function 552 When generating query feature sets that are not only similar to query terms, but dissimilar to irrelevant query terms.

wobei F_q̅ einen negativer Trainingsbegriff-Merkmalssatz (z.B. den negativer Trainingsbegriff-Merkmalssatz 458) darstellt, der auf der Basis eines negativen Trainingsbegriffs erzeugt worden ist (z.B. auf der Basis des negativen Trainingsbegriffs 462 und des Trainingsbereichs 440a erzeugt worden ist). Das Minimieren dieses Verlusts unterstützt bei der geeigneten Einstufung von Bildern in Bezug auf einen vorhergesagten Abfragemerkmalssatz.An image-based grading loss function may also be described as pseudocode and / or equations implemented by a computing device to minimize image-based grading loss. For example, in one or more embodiments, the spatial semantic media search system utilizes the following image-based ranking loss function: $$L_{r_1}\left(F_q\right)=\text{Max}\left(0\alpha -COs\left(F_q.F_{1_q}\right)+COs\left(F_{\overline{q}}.F_{1_q}\right)\right)$$

where F _{q̅ is} a negative training term feature set (eg, the negative training term feature set 458 ), which has been generated on the basis of a negative training concept (eg on the basis of the negative training concept 462 and the training area 440a has been generated). Minimizing this loss helps in properly ranking images for a predicted query feature set.

Insbesondere nutzt das räumlich-semantische Mediensuchsystem einen „Stochastic Gradient Descent“-Algorithmus zum Trainieren des neuronalen Abfragenetzwerks 442 zum Minimieren des akkumulierten stochastischen Verlusts der drei Verlustfunktionen, L_s(F_q), L_{r I} (F_q), L_{r q} (F_q), über jeder Trainingsabfrage.As mentioned above, the spatial semantic media search system in one or more embodiments together minimize loss functions. In particular, the spatial semantic media search system together minimizes a similarity loss function, an image-based rating loss function, and a query-based rating loss function. For example, the shows 4B the training structure 446 , which together form the query-based ranking loss function 448 , the similarity loss function 450 and the image-based grading loss function 452 minimized. In particular, the spatial semantic media search system minimizes with respect to the 4B the following common loss function: $$L\left(F_q\right)=L_s\left(F_q\right)+L_{r_1}\left(F_q\right)+L_{r_q}\left(F_q\right)$$

In particular, the spatial semantic media search system utilizes a "Stochastic Gradient Descent" algorithm to train the neural interrogation network 442 to minimize the accumulated stochastic loss of the three loss functions, L _s (F _q ), L _{r I} (Q _q ), L _{r q} (F _q ), above each training query.

Further, as discussed above, by minimizing the common loss function, the spatial semantic media search system assists the trained model in optimizing retrieval performance. In particular, as mentioned above, the spatial semantic media search system trains the neural interrogation network 442 to generate query feature sets that represent closely similar digital images as they differentiate between irrelevant digital images and irrelevant query terms. By jointly minimizing the three individual losses becomes the neural interrogation network 442 training (ie, the spatio-semantic media search system becomes the trained neural interrogation network 460 to optimize the similarity of its predicted features while supporting the appropriate ranking between the queries and the images for their relevance.

Although the 4B a single negative training term 462 and a single negative training term feature set 458 2, the spatial semantic media search system may identify and generate a plurality of negative training terms and negative training term feature sets. In addition, the spatial semantic media search system may provide an image-based rating loss function determine the difference between a plurality of negative training digital images and the predicted feature set 444 analyzed.

Although the 4B a single negative digital image feature set 454 3, the spatial semantic media search system may accordingly select a plurality of negative digital images and negative digital image feature sets. In addition, the spatial semantic media search system may calculate a query-based rating loss function that measures the difference between a plurality of negative digital image feature sets and the predicted feature set 444 analyzed.

Although the 4A and 4B training areas 402a to 406a As rectangles, the spatial-semantic media search system can define training areas in many different forms. For example, instead of using rectangles defining training areas around a visual content shown in a digital training image, the spatial semantic media search system may define training areas that include circles or polygons. Accordingly, the spatial-semantic media search system may use object masks that closely follow the contours of visual content shown in a digital training image to create a training area.

As mentioned above, in addition to identifying a digital image based on a query term and a query scope in one or more embodiments, the spatial semantic media search system searches for digital images based on iterative user input of query terms and query ranges. For example, the show 5A to 5C a user interface for an iterative user input of query terms and query ranges and for iteratively displaying searched digital images based on the query terms and query ranges according to one or more embodiments.

In particular, the shows 5A a computing device 500 with a screen 502 , which is a user interface 504 shows. As shown, the user interface includes 504 a digital canvas 506 , In addition, the user interface includes 504 a plurality of user interface elements 508a to 508N to create, edit and modify the digital canvas 506 , Furthermore, the user interface includes 504 a search results area 512 to display digital images resulting from a search based on the digital canvas 506 result. As it is shown, a user can use the digital screen 506 by means of the user interface 504 interact to a query area 510 provide. For example, the shows 5A a user input of the query area 510 by means of a selection event (eg with a mouse or a touch screen), a drag event and a release event.

In addition to a user input of the query area 510 a user may also provide a user input of a query term. For example, a user may be based on a user interaction (eg, a selection) of the query area 510 provide a user input of a query term. For example, the shows 5B the user interface 504 after entering a query term. As it is shown, the digital canvas includes 506 a query area 510 and a query term 520 , In addition, the spatio-semantic media search system identifies based on the query range 510 and the query term 520 a first plurality of resulting digital images 522a to 522n and shows the plurality of resulting digital images 522a to 522n in the search results area 512 at.

In particular, the spatial-semantic media search system provides the query area 510 and the query term 520 ready for a neural interrogation network. In response, the neural interrogation network generates a query feature set and compares the query feature set with a memory content of digital images. In particular, the spatial semantic media search system compares the query feature set with digital image feature sets that correspond to the memory contents of digital images. Based on the comparison, the spatial semantic media search system identifies the first plurality of resulting digital images 522a to 522n , The spatial semantic media search system identifies the first plurality of resulting digital images 522a to 522n that provide a searched visual content that matches the query term 520 matches within a searched range that corresponds to the query range 510 equivalent.

The spatial semantic media search system may also receive additional user input of additional query terms and query ranges and identify additional resulting digital images. For example, the shows 5C the user interface 504 after the user input of a second query area 530 , a second query term 532 , a third query area 534 and a third query term 536 , Based on the query range 510 , the query term 520 , the second query range 530 , the second query term 532 , the third query area 534 and the third query term 536 The spatial semantic media search system identifies a second plurality of resulting digital images 538a to 538n , Thus, as shown, users may progressively examine different search results by iteratively adding concepts to the canvas rather than specifying all the elements at once.

The spatial-semantic media search system may include the second plurality of digital images 538a to 538n identify in various ways. In one or more embodiments, the spatial semantic media search system provides the query area 510 , the query term 520 , the second query area 530 , the second query term 532 , the third query area 534 and the third query term 536 for the neural interrogation network to generate a second query feature set. By way of illustration, the spatio-semantic media search system may generate a three-dimensional grid representing the query term 520 to a spatial location corresponding to the query area 510 corresponds to the second query term 532 to a spatial location that is the second query range 530 corresponds, and the third query term 536 encoded to a spatial location corresponding to the third query area 534 equivalent. The spatial semantic media search system may provide the three-dimensional grid for a neural query network to provide a second query feature set. In addition, the spatial semantic media search system may use the second query feature set to identify the second plurality of resulting digital images 538a to 538n use.

Instead of generating a second query feature set by providing the query area 510 , the query term 520 , the second query range 530 , the second query term 532 , the third query area 534 and the third query term 536 For the neural interrogation network, the spatial semantic media search system uses the original query feature set (ie, the one related to the 5B has been generated). In particular, the spatial semantic media search system may include the original query feature set along with a representation (eg, a three-dimensional grid) of the second query region 530 , the second query term 532 , the third query area 534 and the third query term 536 for the neural interrogation network. The neural interrogation network may then retrieve the second query feature set based on the original query feature set along with the representation of the second query range 530 , the second query term 532 , the third query area 534 and the third query term 536 produce.

Although the 5A to 5C the user interface 504 show the resulting digital images in the search results area 512 provides, the spatial semantic media search system may provide one or more digital images in other locations or by other means. For example, in one or more embodiments, the spatial semantic media search system provides a resulting digital image on the digital canvas 506 Thus, after obtaining the first query area and the first query term, the spatial-semantic media search system may obtain a resulting digital image using the digital canvas 506 provide. In addition, the spatial semantic media search system can create a second resulting digital image using the digital canvas 506 after obtaining additional query terms and / or query ranges. In this way, the spatial-semantic media search system can enable a user by means of the digital screen 506 to see how the query areas and the query terms correspond to resulting digital images.

Although the 5A to 5C In addition, when performing a search for searched digital images based on query terms and query ranges, it should be noted that the spatial semantic media search system may also perform searches by other inputs. For example, in one or more embodiments, the spatial semantic media search system may perform a search for digital images based on a background tag (eg, a query term for the background of a searched digital image). By way of illustration, the spatial semantic media search system may receive a user input of a term as a background tag and perform a search for images showing the background tag. In particular, the spatial-semantic media search system may provide the background tag for a neural network (eg, in a three-dimensional grid, with the entire spatial area of the three-dimensional grid coded with a vector corresponding to the background tag), and the spatial-semantic media search system may use the neural network Use a network to generate a query feature set. The spatial semantic media search system may then use the query feature set to identify searched for digital images that show searched visual content corresponding to the background day.

Although the above example describes performing a search by means of a background tag as such, the spatial semantic media search system may also perform a search based on a background tag and one or more additional search term (s) and search range (s). For example, the spatial semantic media search system may receive user input of a background tag and a query scope and query term. The spatial-semantic media search system may provide the background tag, query area and query term for a neural network (eg in the form of a three-dimensional grid having a first spatial area corresponding to the query area defined by a vector corresponding to the query term). and wherein the remainder of the three-dimensional grid is defined by a vector corresponding to the background concept). The spatial-semantic media search system may use the neural network to generate a query feature set and to identify digital images that represent visual content corresponding to the first query term within a searched area that corresponds to the query while also displaying visual content. which corresponds to the background tag in the background of the digital image.

In addition to background tags, the spatial semantic media search system may also perform a search based on an existing digital image. For example, a user may have an existing digital image that has various desired characteristics (e.g., an image of a beach) but the existing digital image is missing a desired element (e.g., the image lacks a beach ball on the left side of the image). The spatial-semantic media search system may perform a search based on the existing digital image and a query region and a query term.

By way of illustration, a user may provide input of the existing digital image (e.g., select the existing digital image) and a query term (e.g., "beach ball") and an interrogation area (e.g., a searched area on the left). The spatial semantic media search system may generate a feature set based on the existing digital image (e.g., by using a neural digital image network) and a query feature set based on the query term and the query region. The spatial semantic media search system may then perform a search based on both the feature set based on the existing digital image and the query feature set based on the query term and the query range. For example, the spatial semantic media search system may analyze digital image feature sets and determine a distance between the digital image feature sets and the feature set based on the existing digital image and the query feature set based on the query term and the query region. In this way, the spatial semantic media search system can identify searched digital images that are similar to the existing digital image and that display visual content corresponding to the query term within a searched range that corresponds to the query region.

Accordingly, the spatial semantic media search system may also perform searches based on one or more modifiers. For example, the spatial semantic media search system may support color modifiers. For example, in one or more embodiments, the spatial semantic media search system may train the neural network to convert color terms to a color feature set and to combine the color feature set with other query feature sets. The spatial semantic media search system may then identify searched digital images based on the color feature set and the other query feature sets. Accordingly, in one or more embodiments, the spatial semantic media search system may combine a color modifier with other query terms in generating a query feature set (eg, transform the color modifier along with other query terms using a word-to-vector algorithm and query terms for the neural Provide a query network). In this way, the spatial-semantic media search system can identify searched digital images that match query terms and query ranges, while also displaying certain colors.

As mentioned above, the spatial semantic media search system allows users to quickly and easily identify a plurality of searched digital images that render visual content within a searched area. To illustrate, the 6 three sample queries with the best ten resulting digital images. In particular, the shows 6 a first digital canvas 602 , a second digital canvas 604 and a third digital canvas 606 , Furthermore, the shows 6 a first plurality of search results 602a that's the first digital canvas 602 correspond to a second plurality of search results 604a that the second digital canvas 604 correspond, and a third plurality of search results 606a , the third digital canvas 606 correspond. As shown, the spatial semantic media search system can identify digital visual media that Show a searched visual content that matches query terms within searched ranges that match the query ranges you want.

With reference to the 7 additional details regarding components and capabilities of one embodiment of the spatial semantic media search system are provided. In particular, the shows 7 an embodiment of an exemplary spatial semantic media search system 700 (eg the aforementioned spatial-semantic media search system). As shown, the spatial-semantic media search system 700 a user input detector 702 , a user interface management 704 , a digital canvas management 706 , a feature set generator 708 , a digital visual media search engine 710 , a neural network training facility 712 and a memory manager 714 (which is a neural interrogation network 714a , a neural digital visual media network 714b , a digital visual media storage content 714c , digital visual training media 714d and search results 714E includes), but is not limited thereto.

As it has been mentioned immediately above and as it is in the 7 can be shown, the spatial-semantic media search system 700 the user input detector 702 include. The user input detector 702 can capture, identify, monitor, obtain, process, capture, and / or record various types of user input. For example, the user input detector 702 capture one or more user interaction (s) related to a user interface and / or a digital canvas. In particular, the user input detector 702 capture a user input of a query term and / or a query area using a digital canvas.

The user input detector 702 may work in conjunction with any number of user input devices or computing devices (individually or in combination) including personal computers, laptops, smartphones, smart watches, tablets, touch screen devices, televisions, personal digital assistants, mouse devices, keyboards, trackpads, or pen devices. The user input detector 702 detects and identifies various types of user interactions with user input devices, such as push events, drag events, scrolling events, enable events, etc. For example, the user input detector detects 702 in the case where a client device belonging to the spatial semantic media search system 700 A touch screen includes one or more touch gestures (eg, swipe gestures, knock gestures, merge gestures, and swipe gestures) from a user performing a user interaction.

As it has been mentioned above and as it is in the 7 is shown, includes the spatial-semantic media search system 700 also the user interface management 704 , The user interface management 704 provides, manages and / or controls a graphical user interface (or simply "user interface") for use with the spatial semantic media search system 700 , In particular, the user interface management 704 the representation of information by means of an external component of a client device (eg a computing device 500 ) facilitate. For example, the user interface management 704 display a user interface by means of a display screen associated with a client device. The user interface may be composed of a plurality of graphical components, objects, and / or elements that enable a user to perform a function. The user interface management 704 represents various types of information by means of a client device, including text, images, videos, audio content, characters or other information. In addition, the user interface management 704 various user interfaces ready (eg the user interface 504 ) specific to any of a variety of functions, programs, applications, plug-ins, devices, operating systems, and / or components of a client device. In addition, the user interface management 704 provide various elements for a display, including a digital canvas, query terms, query ranges, and / or other fields or selectable elements.

It also includes, as in the 7 is shown, the spatial-semantic media search system 700 also the digital screen management 706 , The digital canvas management 706 can identify, obtain, determine, capture, extract, and / or manage a user input via a digital canvas. In particular, digital canvas management 706 receive one or more query terms and one or more query ranges that match or correspond to a digital canvas. Accordingly, the digital canvas management 706 determine a background tag and existing digital images (eg, background tags or existing digital images that a user wants to use in searching for a searched digital image).

It also includes, as in the 7 is shown, the spatial-semantic media search system 700 also the feature set generator 708 , The feature set generator 708 can generate, generate, occupy, determine and / or identify a feature set or multiple feature sets. For example, as described above, the feature set generating means 708 use a digital canvas to generate a query feature set. Accordingly, the feature set generating means 708 use a digital image to generate a digital image feature set.

By way of illustration, the feature set generator 708 create a representation of a query term and a query scope. For example, the feature set generator 708 create a three-dimensional grid that generates a query term and a query area from a digital canvas. In addition, the feature set generation kit may provide the representation of the query term and query space for a neural interrogation network (eg, the neural interrogation network 714a ) to generate a query feature set.

Accordingly, the feature set generating means 708 generate a digital image feature set. For example, the feature set generator 708 provide a digital image for a neural digital image network (eg, the neural digital visual media network 714b ) to generate a digital image feature set.

As it is in the 7 can be shown, the spatial-semantic media search system 700 also the digital visual media search module 710 include. The digital visual media search engine 710 can identify, select and / or determine visual media corresponding to a digital canvas. In particular, the digital visual media search module 710 Identify digital images that show searched visual media that match a query term within a searched range that corresponds to a query scope. As described, the digital visual media search engine identifies 710 In one or more embodiments, digital images are obtained by comparing one or more query feature sets and a digital image feature set or a plurality of digital image feature sets (eg, from the feature set generator) 708 ).

As it is in the 7 can be shown, the spatial-semantic media search system 700 in addition, the neural network training facility 712 include. The neural network training device 712 can direct, learn, promote and train a neural network to produce a desired output. In particular, as described above, the neural network training device 712 train a neural interrogation network to generate a query feature set (eg, by using training terms, training areas, a similarity loss function, a query-based loss function, and / or an image-based loss function). Furthermore, the neural network training device 712 also train a neural digital image network (eg, by using digital training images and a similarity loss function).

It also includes, as in the 7 is shown, the spatial-semantic media search system 700 also the memory management 714 , The memory management 714 preserves data for performing the functions of the spatial semantic media search system 700 , The memory management 714 may include one or more memory or storage devices for preserving data for the spatial semantic media search system 700 include. As shown, memory management includes 714 the neural interrogation network 714a (eg the trained neural interrogation network 430 or the trained neural interrogation network 460 ), the neural digital image network 714b (eg the neural digital image network 210 ), the digital image visual media storage content 714c (eg, a plurality of digital images that a user wants to search), digital visual training media 714d (eg, a plurality of digital training images showing known visual content, digital training image feature sets, training terms, training areas, and / or subject boundaries) and search results 714E (eg, identified digital images and / or query feature sets).

Each of the components 702 to 714 of the spatial-semantic media search system 700 (as it is in the 7 shown) may be associated with another component by any suitable communication technologies. It should be noted that, although the components 702 to 714 of the spatial-semantic media search system 700 in the 7 are shown separately, any of the components 702 to 714 may be combined into fewer components, such as a single device or a single module, divided into more components, or may be formed into various components that may serve as a specific embodiment.

The components 702 to 714 of the spatial-semantic media search system 700 may include software, hardware, or both. For example, the components 702 to 714 comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the spatial semantic media search system may be used 700 cause a client device and / or a server device to execute or execute the methods described herein. Alternatively, the components can 702 to 714 and their corresponding elements comprise hardware, such as a special processing device for performing a particular function or group of functions. In addition, the components can 702 to 714 a combination of computer-executable instructions and hardware.

Furthermore, the components can 702 to 714 for example, as one or more operating systems, as one or more standalone application (s), as one or more modules of an application, as one or more plugins, as one or more library function (s) or functions that can be invoked by other applications and / or implemented as a cloud computing model. Consequently, the components can 702 to 714 as a stand-alone application, such as a desktop application or mobile application. Furthermore, the components can 702 to 714 be implemented as one or more web-based application (s) installed on a remote server. The components 702 to 714 can also be implemented in a bundle of mobile device applications or "apps". By way of illustration, the components may be 702 to 714 are implemented in an application, including, but not limited to, ADOBE PHOTOSHOP software, ADOBE STOCK software and image memory content, or ADOBE LIGHTROOM software. "ADOBE", "PHOTOSHOP", "STOCK" and "LIGHTROOM" are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and / or other countries.

The 8th FIG. 12 is a schematic diagram of one embodiment of an example environment. FIG 800 in which the spatial-semantic media search system 700 can be operated. In one or more embodiments, the example environment includes 800 one or more client device (s) 802a . 802b , ... 802n, a network 804 and server 806 , The network 804 may be any suitable network over which the computing devices can communicate. Examples of networks will be described below with reference to FIGS 10 discussed in more detail.

As it is in the 8th shown includes the environment 800 the client devices 802a - 802N , The client devices 802a - 802N may include any computing device. For example, one or more of the client device (s) comprises 802a - 802N in one or more embodiments, one or more computing devices described below with respect to FIGS 10 are described.

In addition, the environment can 800 also the server (s) 806 include. The server or servers 806 can generate, store, receive and transmit any type of data, including the neural interrogation network 714a , the neural digital visual media network 714b , the digital visual media storage content 714c , the digital visual training media 714d and the search results 714E , For example, the server (s) may or may not 806 Transfer data to a client device, such as the client device 802a , The server or servers 806 may or may also be electronic messages between one or more users of the environment 800 transfer. In an example embodiment, the server (s) includes or includes 806 a content server. The server or servers 806 may or may also include a communication server or a web hosting server. Additional details regarding server (s) 806 will be described below with respect to FIGS 10 discussed.

As it is shown, the server (s) can or can 806 in one or more embodiments, the entire spatially-semantic media search system 700 or part of it. In particular, the spatial-semantic media search system 700 comprise an application running on the server (s) 806 or comprise part of a software application that may be downloaded from the server (s) 806. For example, the spatial semantic media search system 700 include a web hosting application that allows the client devices 802a - 802N allows to interact with content residing on the server (s) 806. By way of illustration, in one or more embodiments, the example environment may 800 one or more client device (s) 802a - 802N access a web page that resides on the server (s) 806. In particular, the client device 802a run an application that allows a user to access, view, and / or interact with a web page or web site located on the server (s) 806.

Although the 8th a special arrangement of client devices 802a - 802N , the network 804 and the server (s) 806, various additional arrangements are possible. For example, while the 8th a plurality of separate client devices 802a - 802N shows that with the server (s) 806 over the network 804 In one or more embodiments, a single client device may communicate directly with the server (s) 806, the network 804 is bypassed.

Accordingly, although the environment 800 from 8th is shown to have different components, the environment 800 have additional or alternative components. For example, the spatial semantic media search system 700 be implemented on a single computing device. In particular, the spatial-semantic media search system 700 as a whole by the client device 802a be implemented or the spatio-semantic media search system 700 can be as a whole by the server or servers 806 be implemented. Alternatively, the spatial-semantic media search system 700 be implemented over several devices or components (eg by means of the client devices 802a - 802N and server (s) 806).

For example, in one or more embodiments, the client device receives 802a a user input (eg by means of the user input detector 702 ) of a query term and a query area by means of a digital screen (eg by means of digital screen management 706 ). In addition, the client device sends 802a the query term and the query scope to the server (s) 806 , The server or servers 806 provides or sets the query term and the query range (eg by means of the feature set generator 708 ) a neural interrogation network (eg the neural interrogation network 714a ) for generating a query feature set. It also compares or compares the server (s) 806 (eg by means of the digital visual media search module 710 ) retrieves the query feature set with a plurality of learned digital media feature sets (eg, by the feature set generator) 708 ) of a plurality of digital images using a neural digital image network (eg, the neural digital image network 714b ). Based on the comparison, identify or identify the server (s) 806 (eg by means of the digital visual media search module 710 ) a digital image showing a searched visual content corresponding to the query term within a searched visual area corresponding to the query area. It also provides or sets the server or servers 806 the identified digital image for a display to the client device 802a ready (eg by means of user interface management 704 ).

Further, in one or more embodiments, the server (s) train or train 806 also a neural interrogation network and / or a neural digital image network (eg by means of the neural network training device 712 ). As has been discussed above, this constitutes or constitutes the servers 806 in one or more embodiments, provide a neural interrogation network having a training area and a training term corresponding to a digital training image, and train or train the neural interrogation network by comparing a predicted feature set to an actual feature set corresponding to the digital training image. Furthermore, the server (s) may or may not 806 together, minimize similarity loss functions, query-based rating loss functions, and image-based rating loss functions to train the neural interrogation network. Accordingly, the server (s) may or may not 806 also train a neural digital image network by providing digital training images to the neural digital image network and comparing a predicted feature set to an actual feature set corresponding to the digital training image.

As an additional example, the environment includes 800 in one or more embodiments, one or more memories (eg, at the server (s) 806 and / or the client devices 802a - 802N ). The one or more memories may include a plurality of feature sets, each feature set corresponding to a digital image of a plurality of digital images and extracted from a level of a neural digital image network that preserves semantic and spatial information from the corresponding digital image. Further, the one or more memories may also include a neural interrogation network trained to generate query feature sets from representations of query areas and query terms, the query feature sets having a dimensionality of the feature sets of the plurality of digital images.

In addition, stores or stores the server or servers 806 In one or more embodiments, instructions that, when executed by the server or servers, cause the system (eg, the client devices 802a - 802N and / or the server (s) 806 ): generates a representation of a query area and a query term that encodes the query term in a spatial location corresponding to the query term Query Area, where the query term specifies a searched visual content and the query specifies a searched area to represent the searched visual content; using the neural interrogation network, generates a query feature set from the representation of the query area and the query term; and from the plurality of digital images, identify a digital image representing the searched visual content within the searched area by comparing the query feature set with the plurality of feature sets. The server or servers 806 may or may also store instructions as they pass through the server or servers 806 which are described below in relation to 9 perform the steps described.

The 1A to 8th , the related text and examples provide a number of different systems and apparatus for providing digital images of a virtual environment via full path space learning. In addition to the foregoing, embodiment (s) may also be described by way of flowcharts including acts and steps in a method to achieve a particular result. For example, the show 9 Flowcharts of example methods according to one or more embodiments. The in terms of the 9 described methods may be performed with fewer or more steps / operations or the steps / operations may be performed in different orders. In addition, the steps / operations described herein may be repeated or performed in parallel or in parallel with different instances of the same or similar steps / operations.

The 9 shows a flowchart of a series of operations in a method 900 the use of spatial and semantic information to search for digital images according to one or more embodiments. In one or more embodiments, the method becomes 900 performed in a digital media environment, the spatial-semantic media search system 700 includes. The procedure 900 is intended to be representative of one or more methods in accordance with the present disclosure and is not intended to limit potential embodiments. Alternative embodiments may include additional, fewer, or different steps than those described in the 9 are shown.

As it is in the 9 is shown, the method comprises 900 a process 910 getting a user input of a query area and a query term. In particular, the process can 910 receiving a user input of a query area and a query term using a digital canvas, the query term indicating a searched visual content, and the query area indicating a searched area for displaying the searched content.

As it is in the 9 is shown, the method comprises 900 also a process 920 determining a query feature set based on the query term and the query range by generating a representation and providing the representation for a neural query network. In particular, the process can 920 determining a query feature set based on the query term and the query range, wherein determining the query feature set comprises: generating a representation of the query term and the query range; and providing the representation of the query term and the query range for a neural query network. For example, in one or more embodiments, the neural interrogation network includes a neural convolution network having three convolution levels, two max-pooling levels, and two subsampling levels. In addition, the process includes 920 in one or more embodiments, further converting the query term into a query term vector using a word-to-vector algorithm; and generating a three-dimensional grid by associating the query term vector with the query area of the digital canvas.

As it is in the 9 The procedure can be shown 900 also a process 930 identifying a digital image showing a searched visual content within a searched area based on the query feature set. In particular, the process can 930 identifying a digital image from a plurality of digital images showing the searched visual content within the searched area by comparing the query feature set with feature sets learned from the plurality of digital images using a neural digital image network. For example, in one or more embodiments, the feature set includes feature vectors having a dimensionality of the feature sets that have been learned from the plurality of digital images using the neural digital image network. In addition, the process includes 930 in one or more embodiments, extracting the feature sets from a level of the neural digital image network that preserves the semantic and spatial information from the digital images.

In addition, the procedure can 900 also include a process of training a neural interrogation network. In particular, the method comprises 900 In one or more embodiments, training the neural interrogation network by: providing a training term and training range as input to the neural interrogation network, wherein the training term and training range correspond to an item shown in a digital training image, wherein the digital training image has a corresponding feature set; Generating a predicted feature set by the neural interrogation network based on the training term and training range; and comparing the predicted feature set generated by the neural interrogation network with the feature set corresponding to the digital training image. Furthermore, training the neural interrogation network may also involve identifying a negative training term that is different from the training term; generating a negative training term feature set based on the negative training term; and comparing the negative training term feature set, the predicted feature set, and the feature set corresponding to the digital training image.

The training of the neural interrogation network may also include identifying a negative digital image that shows an item other than the training term; generating a negative digital image feature set from the negative digital image; and comparing the predicted feature set, the negative digital image feature set, and the feature set corresponding to the digital training image. In addition, training the neural interrogation network may also include constructing a training structure that includes a similarity loss function, an image-based rating loss function, and a query-based rating loss function, wherein the similarity loss function compares a similarity between the predicted feature set and the feature set that corresponds to the digital training image; the image-based ranking loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative training term feature set; and the query based rating loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative digital image feature set. In one or more embodiments, the neural interrogation network collectively minimizes the similarity loss function, the image-based rating loss function, and the query-based rating loss function.

The training of the digital image may further include generating the feature set corresponding to the digital training image. In particular, generating the feature set corresponding to the digital training image may include identifying an item shown in the digital training image and an item boundary corresponding to the item shown in the digital training image; and applying a spatial mask to an area of the digital training image outside the subject boundary.

The procedure 900 can also receive a user input of a second query term and a second query range by means of the digital canvas in addition to the query term and the query area, the query term indicating a second searched visual content and the second query area indicating a second searched area for showing the searched visual content; generating a second query feature set using the neural query network by providing the second query term, the second query range and the query feature set for the neural query network; and identifying at least one digital image of the plurality of digital images showing the searched visual content within the searched area and the second searched visual content within the second searched area by comparing the second query feature set and the feature sets that are among the plurality of digital images digital images have been learned by means of the neural digital image network.

In addition, the procedure can 900 also receiving a second query term and a second query range using the digital canvas; modifying the query feature set using the neural query network to render the second query term and the second query scope; and identifying at least one digital image by comparing the modified query feature set and the digital feature sets corresponding to the plurality of digital images.

• Erhalten einer Nutzereingabe eines Abfragebereichs und eines Abfragebegriffs mittels einer digitalen Leinwand, wobei der Abfragebegriff einen gesuchten visuellen Inhalt angibt und der Abfragebereich einen gesuchten Bereich zum Wiedergeben des gesuchten visuellen Inhalts angibt;
• einen Schritt zum Erzeugen eines Abfragemerkmalssatzes aus dem Abfragebereich und dem Abfragebegriff mittels eines neuronalen Abfragenetzwerks; und
• Identifizieren eines digitalen Bilds von einer Mehrzahl von digitalen Bildern, das den gesuchten visuellen Inhalt innerhalb des gesuchten Bereichs wiedergibt, durch Vergleichen des Abfragemerkmalssatzes mit Merkmalssätzen, die von der Mehrzahl von digitalen Bildern mittels eines neuronalen digitalen Bildnetzwerks gelernt worden sind.

Embodiments may also be described by computer-implemented methods or systems. For example, one or more embodiments include (s) in a digital Media Environment A computer-implemented method of searching for and identifying digital images based on semantic and spatial information, comprising:

• Obtaining a user input of a query area and a query term using a digital canvas, the query term indicating a searched visual content and the query area indicating a searched area for displaying the searched visual content;
• a step of generating a query feature set from the query area and the query term using a neural interrogation network; and
• Identifying a digital image of a plurality of digital images representing the searched visual content within the searched area by comparing the query feature set with feature sets learned from the plurality of digital images using a neural digital image network.

Further, in the method, the query feature set may include feature vectors having a dimensionality of the feature sets that have been learned from the plurality of digital images through the neural digital image network.

• Bereitstellen eines Trainingsbegriffs und eines Trainingsbereichs als Eingabe für das neuronale Abfragenetzwerk, wobei der Trainingsbegriff und der Trainingsbereich einem Gegenstand entsprechen, der in einem digitalen Trainingsbild wiedergegeben ist, wobei das digitale Trainingsbild einen entsprechenden Merkmalssatz aufweist;
• Erzeugen eines vorhergesagten Merkmalssatzes durch das neuronale Abfragenetzwerk auf der Basis des Trainingsbegriffs und des Trainingsbereichs; und
• Vergleichen des vorhergesagten Merkmalssatzes, der durch das neuronale Abfragenetzwerk mit dem Merkmalssatz erzeugt worden ist, der dem digitalen Trainingsbild entspricht, umfassen.

The method may also further train the neural interrogation network by:

• Providing a training term and training range as input to the neural interrogation network, wherein the training term and training range correspond to an item displayed in a digital training image, the digital training image having a corresponding feature set;
• Generating a predicted feature set by the neural interrogation network based on the training term and training range; and
• Comparing the predicted feature set generated by the neural interrogation network with the feature set corresponding to the digital training image.

• Identifizieren eines negativen Trainingsbegriffs, der von dem Trainingsbegriff verschieden ist;
• Erzeugen eines negativer Trainingsbegriff-Merkmalssatzes auf der Basis des negativen Trainingsbegriffs; und
• Vergleichen des negativer Trainingsbegriff-Merkmalssatzes, des vorhergesagten Merkmalssatzes und des Merkmalssatzes, der dem digitalen Trainingsbegriff entspricht.

Further, in the method, training the neural interrogation network may further include:

• Identifying a negative training term different from the training term;
• Generating a negative training term feature set based on the negative training term; and
• Comparing the negative training term feature set, the predicted feature set, and the feature set corresponding to the digital training term.

• Identifizieren eines negativen digitalen Bilds, das einen Gegenstand wiedergibt, der von dem Trainingsbegriff verschieden ist;
• Erzeugen eines negatives digitales Bild-Merkmalssatzes von dem negativen digitalen Bild; und
• Vergleichen des vorhergesagten Merkmalssatzes, des negatives digitales Bild-Merkmalssatzes und des Merkmalssatzes, der dem digitalen Trainingsbild entspricht.

In addition, in the method, training the neural interrogation network may further include:

• Identifying a negative digital image representing an item other than the training term;
• Generating a negative digital image feature set from the negative digital image; and
• Comparing the predicted feature set, the negative digital image feature set and the feature set corresponding to the digital training image.

• eine Ähnlichkeitsverlustfunktion, eine bildbasierte Einstufungsverlustfunktion und eine abfragebasierte Einstufungsverlustfunktion, wobei
• die Ähnlichkeitsverlustfunktion eine Ähnlichkeit zwischen dem vorhergesagten Merkmalssatz und dem Merkmalssatz vergleicht, der dem digitalen Trainingsbild entspricht;
• die bildbasierte Einstufungsverlustfunktion eine Ähnlichkeit zwischen dem vorhergesagten Merkmalssatz und dem Merkmalssatz, der dem digitalen Trainingsbild entspricht, und ein Maß der Unähnlichkeit zwischen dem vorhergesagten Merkmalssatz und dem negativer Trainingsbegriff-Merkmalssatz vergleicht; und
• die abfragebasierte Einstufungsverlustfunktion eine Ähnlichkeit zwischen dem vorhergesagten Merkmalssatz und dem Merkmalssatz, der dem digitalen Trainingsbild entspricht, und ein Maß der Unähnlichkeit zwischen dem vorhergesagten Merkmalssatz und dem negatives digitales Bild-Merkmalssatz vergleicht.

In the method, training the neural interrogation network may also include constructing a training structure comprising:

• a similarity loss function, an image-based ranking loss function and a query-based rating loss function, wherein
• the similarity loss function is similar comparing between the predicted feature set and the feature set corresponding to the digital training image;
• the image-based ranking loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative training term feature set; and
• the query-based rating loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative digital image feature set.

In the method, the neural interrogation network may together minimize the similarity loss function, the image-based ranking loss function and the query-based rating loss function.

The method may further comprise extracting the feature sets from a level of the neural digital image network that preserves semantic and spatial information from the digital images.

• Erhalten einer Nutzereingabe eines zweiten Abfragebegriffs und eines zweiten Abfragebereichs mittels der digitalen Leinwand zusätzlich zu dem Abfragebegriff und dem Abfragebereich, wobei der Abfragebegriff einen zweiten gesuchten visuellen Inhalt angibt und der zweite Abfragebereich einen zweiten gesuchten Bereich zum Zeigen des gesuchten visuellen Inhalts angibt;
• Erzeugen eines zweiten Abfragemerkmalssatzes mittels des neuronalen Abfragenetzwerks durch Bereitstellen des zweiten Abfragebegriffs, des zweiten Abfragebereichs und des Abfragemerkmalssatzes für das neuronale Abfragenetzwerk; und
• Identifizieren mindestens eines digitalen Bilds von der Mehrzahl von digitalen Bildern, das den gesuchten visuellen Inhalt innerhalb des gesuchten Bereichs und den zweiten gesuchten visuellen Inhalt innerhalb des zweiten gesuchten Bereichs wiedergibt, durch Vergleichen des zweiten Abfragemerkmalssatzes und der Merkmalssätze, die von der Mehrzahl von digitalen Bildern mittels des neuronalen digitalen Bildnetzwerks gelernt worden sind.

The method may also include:

• Receiving a user input of a second query term and a second query scope using the digital canvas in addition to the query term and the query scope, the query term indicating a second searched visual content and the second query scope indicating a second searched scope for showing the searched visual content;
• Generating a second query feature set using the neural query network by providing the second query term, the second query range, and the neural query network query feature set; and
• Identifying at least one digital image from the plurality of digital images representing the searched visual content within the searched area and the second searched visual content within the second searched area by comparing the second query feature set and the feature sets that are from the plurality of digital images have been learned by means of the neural digital image network.

• Erhalten einer Nutzereingabe eines Abfragebegriffs und eines Abfragebereichs, die einer digitalen Leinwand entsprechen, wobei der Abfragebegriff einen gesuchten visuellen Inhalt angibt und der Abfragebereich einen gesuchten Bereich zum Wiedergeben des gesuchten visuellen Inhalts angibt;
• Bestimmen eines Abfragemerkmalssatzes auf der Basis des Abfragebegriffs und des Abfragebereichs, wobei das Bestimmen des Abfragemerkmalssatzes umfasst:
  • Erzeugen einer Darstellung des Abfragebereichs und des Abfragebegriffs; und
  • Bereitstellen der Darstellung des Abfragebereichs und des Abfragebegriffs für ein neuronales Abfragenetzwerk; und
• Identifizieren eines digitalen Bilds von einer Mehrzahl von digitalen Bildern, das den gesuchten visuellen Inhalt innerhalb des gesuchten Bereichs wiedergibt, durch Vergleichen des Abfragemerkmalssatzes mit Merkmalssätzen, die von der Mehrzahl von digitalen Bildern mittels eines neuronalen digitalen Bildnetzwerks gelernt worden sind.

Additionally, one or more embodiments in a digital media environment also include or include a computer-implemented method of searching for and identifying digital images based on semantic and spatial information, comprising:

• Obtaining a user input of a query term and a query range corresponding to a digital canvas, the query term indicating a searched visual content, and the query scope indicating a searched range for reproducing the searched visual content;
• Determining a query feature set based on the query term and the query range, wherein determining the query feature set comprises:
  • Generating a representation of the query area and the query term; and
  • Providing the representation of the query area and the query term for a neural interrogation network; and
• Identifying a digital image of a plurality of digital images representing the searched visual content within the searched area by comparing the query feature set with feature sets learned from the plurality of digital images using a neural digital image network.

In the method, the neural interrogation network may also include a neural convolution network having three convolution levels, two max-pooling levels, and two subsampling levels.

• Umwandeln des Abfragebegriffs in einen Abfragebegriffvektor mittels eines Wort-zu-Vektor-Algorithmus; und
• Erzeugen eines dreidimensionalen Gitters durch Zuordnen des Abfragebegriffvektors zu dem Abfragebereich der digitalen Leinwand.

In the method, generating the representation of the query area and the query term may include:

• Converting the query term into a query term vector using a word-to-vector algorithm; and
• Generating a three-dimensional grid by assigning the query term vector to the query area of the digital canvas.

• Bereitstellen eines Trainingsbegriffs und eines Trainingsbereichs als Eingabe für das neuronale Abfragenetzwerk, wobei der Trainingsbegriff und der Trainingsbereich einem Gegenstand entsprechen, der in einem digitalen Trainingsbild gezeigt ist, wobei das digitale Trainingsbild einen entsprechenden Merkmalssatz aufweist;
• Erzeugen eines vorhergesagten Merkmalssatzes durch das neuronale Abfragenetzwerk auf der Basis des Trainingsbegriffs und des Trainingsbereichs; und
• Vergleichen des durch das neuronale Abfragenetzwerk erzeugten vorhergesagten Merkmalssatzes mit dem Merkmalssatz, der dem digitalen Trainingsbild entspricht, umfassen.

The method may further include training the neural interrogation network by:

• Providing a training term and training range as input to the neural interrogation network, the training term and training range corresponding to an item shown in a digital training image, the digital training image having a corresponding feature set;
• Generating a predicted feature set by the neural interrogation network based on the training term and training range; and
• Comparing the predicted feature set generated by the neural interrogation network with the feature set corresponding to the digital training image.

• Bestimmen eines negativen Trainingsbegriffs, der von dem Trainingsbegriff verschieden ist;
• Erzeugen eines negativer Trainingsbegriff-Merkmalssatzes auf der Basis des negativen Trainingsbegriffs;
• Identifizieren eines negativen digitalen Bilds, das einen Gegenstand wiedergibt, der von dem Trainingsbegriff verschieden ist, wobei das negative digitale Bild einen negatives digitales Bild-Merkmalssatz aufweist; und
• Nutzen einer Verlustfunktion zum Vergleichen des negativer Trainingsbegriff-Merkmalssatzes, des negatives digitales Bild-Merkmalssatzes, des vorhergesagten Merkmalssatzes und des Merkmalssatzes, der dem digitalen Trainingsbild entspricht.

In the method, training the neural interrogation network may further include:

• Determining a negative training term different from the training term;
• Generating a negative training term feature set based on the negative training term;
• Identifying a negative digital image representing an item other than the training term, the negative digital image having a negative digital image feature set; and
• Taking advantage of a loss function to compare the negative training term feature set, the negative digital image feature set, the predicted feature set and the feature set corresponding to the digital training image.

• Identifizieren eines Gegenstands, der in dem digitalen Trainingsbild wiedergegeben ist, und einer Gegenstandsgrenze, die dem Gegenstand entspricht, der in dem digitalen Trainingsbild wiedergegeben ist; und
• Anwenden einer räumlichen Maske auf einen Bereich des digitalen Trainingsbilds außerhalb der Gegenstandsgrenze umfassen.

The method may further include generating the feature set corresponding to the digital training image

• Identifying an item displayed in the digital training image and an item border corresponding to the item displayed in the digital training picture; and
• Applying a spatial mask to an area of the digital training image outside the subject boundary.

• Erhalten eines zweiten Abfragebegriffs und eines zweiten Abfragebereichs mittels der digitalen Leinwand;
• Modifizieren des Abfragemerkmalssatzes mittels des neuronalen Abfragenetzwerks zum Wiedergeben des zweiten Abfragebegriffs und des zweiten Abfragebereichs; und
• Identifizieren mindestens eines digitalen Bilds durch Vergleichen des modifizierten Abfragemerkmalssatzes und der digitalen Merkmalssätze, die der Mehrzahl von digitalen Bildern entsprechen.

The method may further include:

• Obtaining a second query term and a second query range using the digital canvas;
• Modifying the query feature set using the neural query network to render the second query term and the second query scope; and
• Identifying at least one digital image by comparing the modified query feature set and the digital feature sets corresponding to the plurality of digital images.

• einen oder mehrere Speicher, umfassend:
  • eine Mehrzahl von Merkmalssätzen, wobei jeder Merkmalssatz:
    • einem digitalen Bild einer Mehrzahl von digitalen Bildern entspricht, und
    • von einer Ebene eines neuronalen digitalen Bildnetzwerks extrahiert wird, das semantische und räumliche Informationen von dem entsprechenden digitalen Bild bewahrt;
  • ein neuronales Abfragenetzwerk, das zum Erzeugen von Abfragemerkmalssätzen aus Darstellungen von Abfragebereichen und Abfragebegriffen trainiert ist, wobei die Abfragemerkmalssätze eine Dimensionalität der Merkmalssätze der Mehrzahl von digitalen Bildern aufweisen;
  • mindestens einen Server, auf dem Anweisungen gespeichert sind, die, wenn sie durch den mindestens einen Server ausgeführt werden, bewirken, dass das System:
    • eine Darstellung eines Abfragebereichs und eines Abfragebegriffs erzeugt, die den Abfragebegriff an einem räumlichen Ort kodiert, der dem Abfragebereich entspricht, wobei der Abfragebegriff einen gesuchten visuellen Inhalt angibt und die Abfrage einen gesuchten Bereich zum Wiedergeben des gesuchten visuellen Inhalts angibt;
    • mittels des neuronalen Abfragenetzwerks einen Abfragemerkmalssatz aus der Darstellung des Abfragebereichs und des Abfragebegriffs erzeugt; und
    • aus der Mehrzahl von digitalen Bildern ein digitales Bild, das den gesuchten visuellen Inhalt innerhalb des gesuchten Bereichs wiedergibt, durch Vergleichen des Abfragemerkmalssatzes mit der Mehrzahl von Merkmalssätzen identifiziert.

In addition, one or more embodiments also include or include a system for identifying digital images based on semantic and spatial information, comprising:

• one or more memories comprising:
  • a plurality of feature sets, each feature set:
    • corresponds to a digital image of a plurality of digital images, and
    • is extracted from a level of a neural digital image network that preserves semantic and spatial information from the corresponding digital image;
  • a neural interrogation network trained to generate query feature sets from representations of query areas and query terms, the query feature sets having a dimensionality of the feature sets of the plurality of digital images;
  • At least one server that stores instructions that, when executed by the at least one server, causes the system to:
    • generates a representation of a query scope and a query term that encodes the query term in a spatial location that corresponds to the query scope, the query term indicating a searched visual content, and the query indicates a searched scope for displaying the searched visual content;
    • using the neural interrogation network, generates a query feature set from the representation of the query area and the query term; and
    • from the plurality of digital images, identifying a digital image representing the searched visual content within the searched area by comparing the query feature set with the plurality of feature sets.

In the system, the neural interrogation network may include a neural convolution network having three convolution levels, two max-pooling levels, and two subsampling levels.

• Umwandeln des Abfragebegriffs in einen Abfragebegriffvektor mittels eines Wort-zu-Vektor-Algorithmus; und
• Erzeugen eines dreidimensionalen Gitters durch Zuordnen des Abfragebegriffvektors zu dem räumlichen Ort, der dem Abfragebereich entspricht.

In the system, the server may further include instructions that, when executed by the at least one server, further cause the system to generate the representation of the query scope and the query term by performing operations comprising:

• Converting the query term into a query term vector using a word-to-vector algorithm; and
• Create a three-dimensional grid by associating the query term vector with the spatial location corresponding to the query area.

• eine Nutzereingabe eines zweiten Abfragebegriffs und eines zweiten Abfragebereichs zusätzlich zu dem Abfragebegriff und dem Abfragebereich erhält, wobei der zweite Abfragebegriff einen zweiten gesuchten visuellen Inhalt angibt und der zweite Abfragebereich einen zweiten gesuchten Bereich zum Wiedergeben des zweiten gesuchten visuellen Inhalts angibt;
• mittels des neuronalen Abfragenetzwerks einen zweiten Abfragemerkmalssatz durch Bereitstellen des zweiten Abfragebegriffs, des zweiten Abfragebereichs und des Abfragemerkmalssatzes für das trainierte neuronale Netzwerk erzeugt; und
• von der Mehrzahl von digitalen Bildern mindestens ein digitales Bild, das den gesuchten visuellen Inhalt innerhalb des gesuchten Bereichs wiedergibt und den zweiten gesuchten visuellen Inhalt innerhalb des zweiten gesuchten Bereichs wiedergibt, durch Vergleichen des zweiten Abfragemerkmalssatzes mit der Mehrzahl von Merkmalssätzen identifiziert.

The system may further include instructions that, when executed by the server, further cause the system to:

• a user input of a second query term and a second query range is obtained in addition to the query term and the query scope, the second query term indicating a second searched visual content and the second query scope indicating a second searched scope for rendering the second searched visual content;
• using the neural interrogation network to generate a second query feature set by providing the second query term, the second interrogation area, and the trained neural network query feature set; and
• among the plurality of digital images, at least one digital image representing the searched visual content within the searched area and reproducing the second searched visual content within the second searched area is identified by comparing the second query feature set with the plurality of feature sets.

Embodiments of the present disclosure may include or utilize a special or generic computer that includes computer hardware, such as computer hardware. one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for containing or storing computer-executable instructions and / or data structures. In particular, one or more of the acts described herein may be implemented, at least in part, as instructions residing in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (eg, a microprocessor) receives instructions from a non-transitory computer-readable medium (eg, a memory, etc.) and executes those instructions, thereby performing one or more operations, including one or more operations, here or there is or is, is or will be.

Computer readable media can be any available media that can be accessed by a general or specific computer system. Computer-readable media storing computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media containing computer-executable instructions are transmission media. Thus, by way of example and not limitation, embodiments of the disclosure may include at least two distinctly different types of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) include RAM, ROM, EEPROM, CD-ROM, SSDs (eg, based on RAM), flash memory, phase change memory ("PCM"), other types of memory, other optical disk storage, Magnetic disk storage or other magnetic storage devices, or any other medium for storing desired Program code means in the form of computer-executable instructions or data structures and which can be accessed by a general or special computer.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures may be automatically transferred from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures obtained via a network or data link may be buffered in RAM within a network interface module (eg, a "NIC") and then eventually to a computer system RAM and / or less volatile computer storage media (devices). be transmitted in a computer system. Thus, it should be noted that non-transitory computer-readable storage media (devices) can be included in computer system components that also use (or even predominantly) transmission media.

Computer-executable instructions include e.g. Instructions and data that, when executed in a processor, cause a general purpose computer, computer, or processing device to perform a particular function or set of functions. In some embodiments, computer-executable instructions are executed on a general computer so that the general computer becomes a particular computer that executes elements of the disclosure. The computer-executable instructions may be e.g. binary statements ("binaries"), intermediate format statements, such as Assembly language, or even a source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the features and processes described are disclosed as example forms for implementing the claims.

One skilled in the art will appreciate that the disclosure can be made in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, cell phones , PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be made in distributed system environments where local and remote computer systems connected through a network (either through wired data links, wireless data links, or a combination of wired and wireless data links) perform both tasks. In a distributed system environment, program modules may reside in both local and remote storage devices.

Embodiments of the present disclosure may also be implemented in cloud computing environments. In this description, "cloud computing" is defined as a model that allows network access to a common pool of configurable computing resources as needed. For example, cloud computing may be deployed in the marketplace to provide anywhere, convenient on-demand access to the common pool of configurable computing resources. The shared pool of configurable computing resources can be quickly deployed through virtualization and released with a low overhead or service provider interaction, and then scaled accordingly.

A cloud computing model may be composed of various characteristics, such as e.g. on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, etc. A cloud computing model may also include various service models, such as: Software as a Service ("SaaS"), Platform as a Service ("PaaS") and Infrastructure as a Service ("IaaS"). A cloud computing model may also be propagated by means of various distribution models, e.g. a private cloud, a community cloud, a public cloud, a hybrid cloud, etc. In this description and in the claims, a "cloud computing environment" is an environment in which cloud computing is employed.

The 10 shows in block diagram form an exemplary computing device 1000 , which may be configured to perform one or more of the above-described operations. It is clear that the spatial-semantic media search system 700 Implementations of the computing device 1000 may include. As it is in the 10 1, the computing device may be a processor 1002 , a store 1004 , a storage device 1006 , an I / O interface 1008 and a Communication Interface 1010 include. In certain embodiments, the computing device 1000 include fewer or more components than those used in the 10 are shown. Components of the computing device 1000 in the 10 are shown below in more detail.

In certain embodiments, the processor (s) comprise or include 1002 Hardware for executing instructions, such as those that make up a computer program. By way of example and not limitation, the processor (s) may or may not be 1002 the instructions from an internal register, an internal cache, a memory 1004 or a storage device 1006 retrieve (or fetch) and decode and execute.

The computing device 1000 includes a memory 1004 that is related to the processor (s) 1002 is coupled. The memory 1004 may be used to store data, metadata and programs for execution by the processor (s). The memory 1004 may include one or more of volatile and nonvolatile memories, such as random access memory ("RAM"), read only memory ("ROM"), semiconductor disk ("SSD"), flash memory, phase change memory ("PCM"), or other types of data storage. The memory 1004 can be an internal or a distributed or external memory.

The computing device 1000 comprises a storage device 1006 which includes a memory for storing data or instructions. As an example and not by way of limitation, the memory device 1006 comprise a nonvolatile storage medium as described above. The storage device 1006 may include a hard disk drive (HDD), a flash memory, a serial universal bus (USB) drive, or a combination of these or other memory devices.

The computing device 1000 Also includes one or more input or output ("I / O") devices / interfaces 1008 provided to enable a user input to (such as user touches) the computing device 1000 perform an output from the computing device 1000 or otherwise receive data to and from the computing device 1000 transferred to. These I / O devices / interfaces 1008 may be a mouse, keypad or keyboard, touch screen, camera, optical scanner, network interface, modem or other known I / O devices, or a combination of such I / O devices. Devices / interfaces 1008 include. The touch screen can be activated with a pen or a finger.

The I / O devices / interfaces 1008 may include one or more devices for presenting an output to a user, including, but not limited to, a graphics module, a display (eg, a display screen), one or more output drivers (eg Display driver), one or more speakers, and one or more audio drivers. In certain embodiments, the devices / interfaces 1008 adapted to provide graphical data for a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and / or any other graphical content that may serve as a particular implementation.

The computing device 1000 may further include a communication interface 1010 include. The communication interface 1010 may include hardware, software, or both. The communication interface 1010 may include one or more interfaces for communicating (such as with packet-based communication) between the computing device and one or more other computing devices. 1000 or one or more network (s). By way of example and not limitation, the communication interface 1010 a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as WI-FI. The computing device 1000 can also have a bus 1012 include. The bus 1012 may include hardware, software, or both, the components of the computing device 1000 coupled with each other.

In the foregoing description, the invention has been described with reference to specific exemplary embodiments thereof. Various embodiments and aspects of the invention are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The foregoing description and drawings are merely illustrative of the invention and should not be construed as limiting the invention. Numerous specific details are described so that various embodiments of the present invention will be readily understood.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with fewer or more steps / operations, or the steps / operations may be performed in a different order. In addition, the steps / operations described herein may be repeated or performed in parallel with different instances of the same or similar steps / operations. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. Any changes within the meaning and range of equivalency of the claims are intended to be within the scope of the invention.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 62414140 [0001]

Cited non-patent literature

• C. Szegedy et al., "Going deeper with convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pages 1-9 [0042]
• ImageNet image data set - cf. J. Deng, W. Dong, R. Socher, Li Li, Kai Li and Li Fei-Fei, "ImageNet: A large-scale hierarchical image database", Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, Miami, FL, 2009, pages 248-255 [0046]

Claims (20)

A computer-implemented method of searching for and identifying digital images based on semantic and spatial information in a digital media environment, comprising: Obtaining a user input of a query area and a query term using a digital canvas, the query term indicating a searched visual content and the query area indicating a searched area for displaying the searched visual content; a step of generating a query feature set from the query area and the query term using a neural interrogation network; and Identifying a digital image of a plurality of digital images representing the searched visual content within the searched area by comparing the query feature set with feature sets learned from the plurality of digital images using a neural digital image network. Method according to Claim 1 in which the query feature set comprises feature vectors having a dimensionality of the feature sets that have been learned from the plurality of digital images using the neural digital image network. Method according to Claim 2 further comprising training the neural interrogation network by: providing a training term and a training range as input to the neural interrogation network, wherein the training term and training range correspond to an item displayed in a digital training image, the digital training image having a corresponding feature set ; Generating a predicted feature set by the neural interrogation network based on the training term and training range; and comparing the predicted feature set generated by the neural interrogation network with the feature set corresponding to the digital training image. Method according to Claim 3 wherein training the neural interrogation network further comprises: identifying a negative training term different from the training term; Generating a negative training term feature set based on the negative training term; and comparing the negative training term feature set, the predicted feature set, and the feature set corresponding to the digital training term. Method according to Claim 4 wherein training the neural interrogation network further comprises: identifying a negative digital image representing an object other than the training term; Generating a negative digital image feature set from the negative digital image; and comparing the predicted feature set, the negative digital image feature set and the feature set corresponding to the digital training image. Method according to Claim 5 wherein training the neural interrogation network comprises constructing a training structure comprising: a similarity loss function, an image-based rating loss function, and a query-based rating loss function, wherein: the similarity loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image; the image-based ranking loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative training term feature set; and the query based rating loss function compares a similarity between the predicted feature set and the feature set corresponding to the digital training image and a measure of the dissimilarity between the predicted feature set and the negative digital image feature set. Method according to Claim 6 in which the neural interrogation network together minimizes the similarity loss function, the image-based ranking loss function and the query-based rating loss function. The method of any one of the preceding claims, further comprising extracting the feature sets from a level of the neural digital image network that preserves semantic and spatial information from the digital images. The method of any one of the preceding claims, further comprising: Receiving a user input of a second query term and a second query scope using the digital canvas in addition to the query term and the query scope, the query term indicating a second searched visual content and the second query scope indicating a second searched scope for showing the searched visual content; Generating a second query feature set using the neural query network by providing the second query term, the second query range, and the neural query network query feature set; and Identifying at least one digital image from the plurality of digital images representing the searched visual content within the searched area and the second searched visual content within the second searched area by comparing the second query feature set and the feature sets that are from the plurality of digital images have been learned by means of the neural digital image network, A computer-implemented method of searching for and identifying digital images based on semantic and spatial information in a digital media environment, comprising: Obtaining a user input of a query term and a query range corresponding to a digital canvas, the query term indicating a searched visual content, and the query scope indicating a searched range for reproducing the searched visual content; Determining a query feature set based on the query term and the query range, wherein determining the query feature set comprises: Generating a representation of the query area and the query term; and Providing the representation of the query area and the query term for a neural interrogation network; and Identifying a digital image of a plurality of digital images representing the searched visual content within the searched area by comparing the query feature set with feature sets learned from the plurality of digital images using a neural digital image network. Method according to Claim 10 in which the neural interrogation network comprises a three-fold neural folding network, two max-pooling levels, and two subsampling levels. Method according to Claim 10 or 11 wherein generating the representation of the query area and the query term comprises: converting the query term into a query term vector using a word-to-vector algorithm; and generating a three-dimensional grid by associating the query term vector with the query area of the digital canvas. Method according to one of Claims 10 to 12 further comprising training the neural interrogation network by: providing a training term and training range as input to the neural interrogation network, wherein the training term and training range correspond to an item shown in a digital training image, the digital training image having a corresponding feature set ; Generating a predicted feature set by the neural interrogation network based on the training term and training range; and comparing the predicted feature set generated by the neural interrogation network with the feature set corresponding to the digital training image. Method according to Claim 13 wherein training the neural interrogation network further comprises: determining a negative training term other than the training term; Generating a negative training term feature set based on the negative training term; Identifying a negative digital image representing an item other than the training term, the negative digital image having a negative digital image feature set; and using a loss function to compare the negative training term feature set, the negative digital image feature set, the predicted feature set, and the feature set corresponding to the digital training image. Method according to Claim 13 or 14 further comprising generating the feature set corresponding to the digital training image by identifying an object represented in the digital training image and an article boundary corresponding to the object rendered in the digital training image; and applying a spatial mask to an area of the digital training image outside the subject boundary. Method according to one of Claims 10 to 15 further comprising: obtaining a second query term and a second query range using the digital canvas; Modifying the query feature set using the neural query network to render the second query term and the second query domain; and identifying at least one digital image by comparing the modified query feature set and the digital feature sets corresponding to the plurality of digital images. A system for identifying digital images based on semantic and spatial information, comprising: one or more memories comprising: a plurality of feature sets, each feature set: corresponds to a digital image of a plurality of digital images, and is extracted from a level of a neural digital image network that preserves semantic and spatial information from the corresponding digital image; a neural interrogation network trained to generate query feature sets from representations of query areas and query terms, the query feature sets having a dimensionality of the feature sets of the plurality of digital images; At least one server that stores instructions that, when executed by the at least one server, causes the system to: generates a representation of a query scope and a query term that encodes the query term in a spatial location that corresponds to the query scope, the query term indicating a searched visual content, and the query indicates a searched scope for displaying the searched visual content; using the neural interrogation network, generates a query feature set from the representation of the query area and the query term; and from the plurality of digital images, identifying a digital image representing the searched visual content within the searched area by comparing the query feature set with the plurality of feature sets. System after Claim 17 in which the neural interrogation network comprises a three-fold neural folding network, two max-pooling levels, and two subsampling levels. System after Claim 17 or 18 wherein the server further comprises instructions that, when executed by the at least one server, further cause the system to generate the representation of the query area and the query term by performing operations, comprising: converting the query term into a query term vector by means of a word-to-vector algorithm; and generating a three-dimensional grid by associating the query term vector with the spatial location corresponding to the query area. System according to one of Claims 17 to 19 method further comprising instructions that, when executed by the server, further cause the system to: receive a user input of a second query term and a second query range in addition to the query term and the query scope, the second query term having a second searched visual content and the second polling area indicates a second searched area for displaying the second searched visual content; using the neural interrogation network to generate a second query feature set by providing the second query term, the second interrogation area, and the trained neural network query feature set; and from the plurality of digital images, at least one digital image representing the searched visual content within the searched area and the second searched visual content within the second is searched by comparing the second query feature set with the plurality of feature sets.

Images