KR20010041607A - Method and system for generating semantic visual templates for image and video retrieval - Google Patents

Abstract

To retrieve images / videos from a database, a semantic visual template (SVT) is a set of icons consisting of example scenes / objects that can characterize concepts such as skiing, sunsets, and so on. Semantics Visual templates provide a two-way dialogue between the user and the system. The user can provide an initial sketch or example picture as a seed that can automatically generate other representatives of the same concept to the system. The user can then select the perspectives that include what is considered desirable to represent the concept. Once the semantic visual template is established, the database can retrieve it and the user can provide relevance feedback for the returned results. Users can interact with the system at the conceptual level using established semantic visual templates. When forming new concepts, you can use existing semantic visual templates.

For queries to the system, the limited lexical range of words related to the semantic visual template can be analyzed on a per component basis.

Description

Method and system for creating semantic visual templates for image and video retrieval {METHOD AND SYSTEM FOR GENERATING SEMANTIC VISUAL TEMPLATES FOR IMAGE AND VIDEO RETRIEVAL}

As images and videos are increasingly created, disseminated, and stored in digital form, tools and systems for retrieving visual information have become important. However, while effective "search engines" have become widely available for text data, the tools for searching corresponding visual picture and video data remain ambiguous.

Generally, keyword techniques are used to index and search for pictures in the available picture and video databases. Search systems known in this form were time-consuming and subjective because they could not incorporate textual information into an image, and had to manually include captions, and textual annotations were incidental. ) Had many defects, such as the lack of unique image characteristics. For example, a textual description such as "a person is leaning against a brick wall" conveys only a small amount of image information about the person or brick wall. Such image information is often essential for searching for a particular video.

Recently, developers have begun developing new forms of image and video repository retrieval. They develop on the basis of temporal relationships between objects constituting the video, such as image properties, for example, color, texture, shape and spatial. Typically this theory specifies a query by a given example image or visual sketch.

The present invention relates to database still picture, video, and audio retrieval and, more particularly, to a technology that facilitates access to database items.

1 is a block diagram of an interactive technique for generating a library or collection of semantic visual templates according to a preferred embodiment of the present invention;

2 is a schematic diagram showing one concept with necessary and sufficient conditions,

3 is a schematic diagram showing query generation;

4 is a block diagram of a semantic visual system according to another embodiment of the present invention including audio processing;

5 shows a set of icons illustrating the "high jump" concept,

6 shows a set of icons illustrating the "sunset" concept,

7 shows a set of icons illustrating the concept of a "spinning race".

To easily retrieve pictures and videos from a database, the database can be indexed using a collection of visual templates. Preferably, in view of the present invention, the visual template represents a semantic concept or category, such as skiing, sunset, and the like. An architecture using semantic visual templates (SVT) is created, where each semantic visual template represents a concept and consists of queries that well describe the concept.

Semantic visual templates can also be established through a dialogue process between the user and the system. The user can provide an initial sketch or example picture as a seed that can automatically generate other representatives of the same concept to the system. The user can then select the perspectives that include what is considered desirable to represent the concept. Once the semantic visual template is established, the database can retrieve it and the user can provide relevance feedback for the returned results. Users can interact with the system at the conceptual level using established semantic visual templates. When forming new concepts, you can use existing semantic visual templates.

For querying the system, there is further provided a technique for component-by-component analysis of the limited lexical range of words related to the semantic visual template.

The present invention discloses a provisional patent application No. 60 / 045,637 (filed May 5, 1997) and PCT / US98 / 09124 (filed May 5, 1998). Canada, Japan, South Korea, and the United States). These applications describe object-based spatiotemporal visual retrieval techniques for retrieving pictures and videos in different categories using visual templates. The present invention can use this search technique together called VideoQ.

Video cues can be used to determine the beginning and end of a scene in a video stream. For example, the video cue may be used to further compensate for the camera motion for extracting an object from the scene. By using Video Cue, each object can be characterized with remarkable properties such as color, texture, size, shape, and motion. In this way, a video object database consists of a scene and all objects extracted from its attributes.

Visual template

Visual templates represent ideas in the form of sketches or animated sketches. Since a single visual template is not sufficient to represent a class of interest, it is possible to combine a library of visual templates containing representative templates for different semantic classes. have. For example, when retrieving a video clip of the sunset class, the user can select one or more visual templates corresponding to this class, and use similarity-based querying to find the video clip about sunset. It is available.

An important advantage of using the visual template library is the incorporation of a low-level visual feature representation into a high-level semantic concept. For example, if a user queries in a constrained natural language form as described in the previous patent application, the visual template can be used to convert this natural language query into an automatic query specified by image attributes and constraints. have. If the visual content in a collection or database is not indexed textually, the custom textual search method cannot be applied immediately.

Meaning Visual Template (SVT)

Semantic visual templates are sets of visual templates combined with individual semantics. The concept of semantic visual template (SCT) has the following key properties;

Semantics Visual templates are common in practice. For a given concept, there should be a set of visual templates that can contain it well. Examples of successful semantic visual templates include sunset, high jump, down-hill skiing, and the like.

Implications for Concepts Visual templates should be small in size, but in order to perform high-precision reproductions they must be able to contain large amounts of related images and videos in a collection.

Implications for Different Concepts The process for finding visual templates is systematic, efficient, and robust. Efficiency is related to the focus on a small set of visual templates. Robustness is evident by applying a new template library to new picture and video collection.

In the context of a video cue, a semantic visual template may also be understood as a bundle of icons or example scenes / objects representing semantics associated with it. A feature vector for a query can be extracted from the semantic visual template. The icon is a fairy tale sketch. In Video Cue, the properties associated with each object and their space-time relationships are important. Histogram, texture, and structure information are examples of generic features that can be part of this template. The choice of whether to implement based on an icon or a set of feature vectors formed from a comprehensive feature depends on the meaning it represents. For example, sunset scenes are better represented by a single waterfall or crowd than by a comprehensive feature set, so it is appropriate to represent them as a pair of objects. For this reason, each template contains various icons, such as scenes / objects, to express one meaning. The components of the set may overlap according to their coverage. It is desirable to maximize its coverage with a minimal set of templates.

Each icon for a concept, such as downhill skiing, sunset, beach crowds, etc., is an image representative of the graphical objects that resemble real objects in the scene. Each object is associated with a set of image properties such as color, shape, texture, motion, and the like. The relevance of each attribute and each object to the concept is also specified. For example, for "sunset", the color and spatial structure of objects such as the sun and the sky are more relevant. For sunset scenes, the sun object may be visual because there may be a sunset video where the sun is not visible. For the concept of "high jump", the behavioral property of the foreground object is mandatory, but the textual property is non-mandartory, and both properties are more relevant than the rest. Some concepts may require only a single object to represent the generic properties of the scene.

FIG. 5 shows some possible icons for "high jump" and FIG. 6 for "sunset". The optimal set of icons can be selected based on relevancy feedback and maximum coverage in terms of representation, which will be described in more detail below.

The present invention provides an efficient technique for generating semantic visual templates for various concepts. Each semantic concept can have several representative visual templates for positive coverage or high reproduction from a collection of materials, which can be used to retrieve important parts of pictures and videos. . Positive coverage sets for different visual templates may overlap. Therefore, it aims to find a small set of visual templates with positive coverage that can be extensive but minimally redundant.

Users can provide initial conditions for an effective visual template. For example, a user may use yellow circles (foreground) and pale red squares (background) as initial templates for searching for sunset scenes. Users can also indicate the requirements that belong to the context and the weights of different objects and attributes, depending on the answer to the dialogue questionnaire. The questionnaires are sensitive to the current query sketched by the user on a sketchpad or the like.

Depending on the relevance of a given initial visual template and all of its picture attributes, the retrieval system returns the most similar set of images / videos to the user. For the results returned, the user can provide a subjective assessment for this. The precision and positive coverage, or representation, of the results returned can be computed by computer.

The system may determine an optimal strategy for changing the initial image query and generate a modified query based on the following;

1. Relevance factor of each image attribute obtained from the user's query.

2. Perform a precise representation of the query above.

3. Information about the distribution of feature levels of pictures and videos in a collection of materials.

Techniques for implementing these features are conceptually illustrated in FIG. 1, which specifically specifies a query for the "high jump" concept. This query contains three objects, two static rectangular background regions, and one object moving downward and to the right. For each object in the query, four attributes, such as color, texture, shape, and size, shown as vertical bars in FIG. 1, are specified in relation to importance. By raising one or more attributes one step, you can form a new query, and at one point up, you can invoke a dialog with the user to decide whether to include them as icons in the template. Once a reasonable number of icons are collected in a temporary template, you can use this template to search the database. The retrieved results can be evaluated for reproduction and precision. Acceptable templates can be stored as semantic visual templates for "high jump".

Template Metric

The basic video data unit may be called a video shot and includes a plurality of segmented video objects. The lifetime of a particular video object may be less than or equal to the duration of the video shot. A similarity measure D between a part of a semantic visual template (SVT) set and a video shot may be defined as follows.

_{D = min {ω r · Σ} {i} d f (O i, O 'i) + ω s · d s}

Where O _i is the object specified in the template, O ' _i is the object matched against O _i , d _f is the feature distance between their arguments, and d _s is Similarity of the space-time structure between the space-time structure and the matched objects in the video shot, and ω _r and ω _s are the normal importance for feature distance and structural dissimilarity. The query procedure produces a candidate list for each object in the query. Then, distance D is the minimum value for all possible sets of matching objects in space and time. For example, if a semantic template has three objects, and two candidate objects are preserved in each single object query, then, considering the calculation of the minimum distance in Equation 1, there may be at most eight potential candidate object sets. Can be.

If N objects are given in the query, they appear to require a search for all sets of N objects that can appear together in the video shot. However, considering the economics of the calculation, the following more economic process can be adopted;

1. Use each video object O _i to query the entire object database to create a list of matching objects that can be kept short by the use of thresholds. Then, only objects contained in this list are considered candidate objects to match O _i .

2. Then, the candidate objects in the list are combined with each other to create the final matched object that can identify the space-time structure.

Template generation

Two-way interaction is used to create a template between the user and the system. Given an initial scenario, the technique converges into a small set of icons that provides maximum representation using relevance feedback. The user provides an initial query made up of objects with space-time constraints as a sketch of the concept that the template will create. The user also specifies whether to force the object. Each object has features for which the user will assign relevant importance.

The initial query can be thought of as a point in the high-dimensional feature space that can map all the videos in the database. In order to automatically generate a set of test icons, it is necessary to quantize each feature of each object in space and then jump. For quantization, the user can determine the size of the step with the help of a particular importance according to the initial query. The importance here may be regarded as an evaluation criterion of relevance given to the feature of the object by the user. Therefore, if the importance level is low, it is roughly quantized and vice versa. For example, it is as follows.

Δ (ω) = 1 / (a

Where Δ is the jump distance corresponding to one feature, ω is the importance associated with the feature, a and b are Δ (0) = 1, Δ (1) = d ₀ (d ₀ is the system associated with the thresholding Parameter, which is set to 0.2 in a prototype system). Using jump distances, the feature face is quantized into a hyper-rectangle. For example, a cuboid can be generated using a metric of the LUV space according to Δ (ω) for color.

In order to prevent a sudden increase in the total number of possible icons, no joint variation of features occurs. For example:

1. For each feature of the object, the user selects a preferred set of features.

2. The system then performs a combination of feature sets associated with the object.

3. The user selects the combination that best represents the change in the object to create a list of candidate icons.

In the case of multiple objects, additional combining may be included in the second step with respect to the candidate list for each object. Once the preferred scenario list is created, the system is queried using the icon selected by the user. Relevancy feedback is used to determine an icon that can be reproduced as much as possible with the user's labeling as positive or negative.

Concept Covers

There is plenty of good "covering" for a concept when searching a user's database. Each marker may be on a different feature space. For example, "sunset" can be described at the object level as well as the generic level. A comprehensive level of technology can take the form of a color or texture histogram. Object-level descriptions can be a collection of two objects, such as the sky or the sun. This object may also be quantified using a feature level descriptor.

As shown in Fig. 2, one concept (e.g. sunset) has two different kinds of conditions: requirement N and sufficient condition S. A semantic visual template is not a requirement, but a semantic visual template does not need to cover the concept entirely. Additional templates can be created manually, i. E. By the user entering additional queries. Tasks are left to each concept. Requirements can be assigned to a concept, thereby automatically creating additional templates for a given initial query template.

The user communicates with the system via a "conceptual questionnaire" to specify the requirements for the retrieved meaning. These conditions can also be inclusive, such as inclusive color distribution, relative space-time correlation. Once the requirements and sufficient conditions for the concept are established, the system moves to the feature space and creates additional templates with the user's original template as a starting point. This generation is also modified according to the relevance feedback given to the system by the user. Analysis of the relevance feedback can determine new rules that correspond to the requirements. They can also be used to modify template generation procedures. With the videos marked as appropriate by the user, a rule is created by considering the correlation between the conditions deemed necessary for the concept. The rules for determining this rule (or suggestion) are similar to the techniques found in "data mining." This is described in the aforementioned patent applications and in a paper by S.Brin et al. "Dynamic Itemset Counting and Implication Rules for Market Basket Data" (ACM SIGMOD Conference on Data Management). , 1997, pp225-246), and by S.Brin et al., "Beyond Market Baskets: Generalizing Association Rules to Correlations" (ACM SIGMOD Conference on Data Management). , 1997, pp 265-276.

Rule Generation Example

In the video queue, the query for "crowd of people" is sketchy. Users could specify image queries by attaching importance to colors and sizes to objects, but could be specified as more detailed techniques, such as forms of textures or relative spatiotemporal mobility that could characterize the concept of human clustering. There was no number. However, he can list them as requirements because he feels that the idea of "crowd" can be strongly characterized by textures and the relative space-time arrangement of people.

The feedback process allows the system to identify video clips appropriate to the concepts of interest to the user. Since the system now knows that texture and space-time placement are essential to the concept, we want to determine the matching pattern between the features that are considered necessary between the appropriate videos. This pattern is returned to the user, asking if the user matches the concepts they are searching for. If the user accepts that these patterns match the concept, they are used to form a new query template as shown in FIG. The new rule includes a double effect on query template generation, including improved search speed and increased precision of returned results.

Generating Concept Covers

The query defines the feature space in which the search is to be performed. The feature space is determined by the properties and relevance importance of the visual template. In particular, attributes define the coordinate axes of the feature space, and relevance importance increases or decreases the associated axes. Each video shot in the synthesized feature space may be represented by a point. Visual templates take up part of this space. Because visual templates vary by character and character (inclusive at the object level), the spaces defined by the templates are different and do not overlap.

Choosing several features alone may not be sufficient to determine the concept, but may be appropriately expressed, for example, by appropriately selecting different importance. Accordingly, the concept can be mapped to one feature space.

One concept is not limited to a single feature space or a single cluster. For example, with respect to the class of video sequence for sunsets, it is not possible to characterize sunsets all in a single color or a single shape. Thus, it is important to determine not only the comprehensive static features and importance associated with the concept, but also the changeable features and importance.

The search for concepts begins with specifying global constants. The contextual query determines the number of objects to retrieve and the comprehensive features required for each object. These represent constraints in the search process that should not change.

The user provides an initial query to specify the features and set their importance. In the set of requirements set by the user, there is an intersection between the sets. The requirement is left unchanged. Changes are made to templates based on changes to features that are considered sufficient. If the sets are not crossed, a rule is drawn that can characterize concepts based on requirements and reversion of relevance.

The relevance importance of each feature represents the tolerance the user desires for each feature. This tolerance is mapped to the distance threshold for each feature. For example, the equation d (ω) = 1 / (a.ω + c), which defines a hyper-ellipsoid in the retrieved feature space. The threshold determines the number of possible non-overlapping markers. The number of markers determines the size and number of jumps possible for a particular feature. The algorithm performs an instantaneous initial search and is guided according to the following three criteria.

First, a greedy algorithm that proceeds to increase reproduction;

Compute all possible initial jumps.

Convert each jump to the corresponding visual template.

Run the query and match all the results.

Relevant recall results are shown to the user, and the results that maximize incremental recall are selected as possible points of the subsequent query.

Second, a logarithmic search that increases in logarithmic series as subsequent queries are retrieved by accepting smaller jumps in the local area. This is based on the fact that the current point of query yields good results and requires careful search for additional templates. The search stops when enough visual templates have been created to cover more than 70% of the concept (ie 70% or more representations).

Third, feature level distributions are used to guide the search, because momentary initial searches often cause too many possibilities to fail at once. The distribution according to each feature is precomputed. This information is used to avoid areas where video shots are scattered and to select jumps to areas of high concentration.

Language Integration with SVT

Existing text-based queries for pictures and videos rely on the matching of keywords attached to pictures or videos. Keywords appended to the data can be generated manually or obtained by association. That is, keywords are extracted from the text to be appended (in the case of an image) or the caption attached to the video.

This approach excludes the possibility of a real system containing a large video or image database for several reasons;

It is not possible to manually create annotations on existing video databases.

Most videos do not contain captions.

There is no direct correlation between the caption and the video being added. For example, during a baseball game, the narrator may talk about the achievements of Babe Ruth who did not play in an ongoing game, where the text-based keyword was placed in a video containing "Babe Ruth" to display this video. If it is wrong.

Generating semantic content for video by analyzing only the video stream corresponds to a computer vision problem that is known to be difficult. A more practical approach is to use visual content, such as color and texture, or the behavior of an object, simultaneously with the descriptive power of natural language.

When the user enters a string, the system parses it into a video model. Video cues provide a "language" into which you can enter queries about sketches.

What exists in the Video Cue and its natural language copy simply corresponds to that shown in Table 1.

TABLE 1

Attribute NL type

Action verb

Color, texture adjectives

Shape nouns

Space / time prepositions / adjuncts

Compulsory language sets can be used with acceptable word sets. To create an action model of the video sequence, the sentence is analyzed into classes such as nouns, verbs, adjectives, and adverbs.

For example, for the phrase “Bill walked slowly towards sunset,” the system can analyze as shown in Table 2.

TABLE 2

Word NI type

Bill noun

Walked verb

Slow adverbs

Towards prepositions

Sunset celeb

For verbs, adverbs, adjectives, and prepositions, these are modifiers (or descriptors) for nouns (objects), so a small but fixed database can be used. The noun (ie scenario / object) database may initially contain a hundred or more scenes, and may be expanded by dialogue with the user.

Each object can have a shape description that is modified by various modifiers such as adjectives (color, texture), verbs (walked), adverbs (slowly), and so on. Then they need to be inserted into the VideoCue palette and further refined.

When a parser encounters a word that does not have a modifier database (i.e., a database corresponding to verbs, adverbs, prepositions, adjectives, etc.), the parser is searched for thesaurus to determine whether the word's synonyms are in that database. And use them instead. If this fails, the analyst returns a message indicating an invalid string.

If the analyst encounters a word that cannot be categorized, the user must modify the text, or if the word is a noun such as "Bill," the system can be instructed that class (in this case a noun). The word may additionally indicate that it refers to a human being. If the user tries to point to a noun that is not in the system database, after the prompt, the user quickly drags the object into the sketch pad so that the system can learn about the object. In the database, properties such as behavior, color, texture, and shape can be created at the object level, so one level to match can be that level.

As shown in Fig. 4, an audio stream appended to the video can be used as a source of information. Indeed, if audio is closely related to video, that audio can be the single most important source of semantic content for the video. For example, a set of 10 to 20 keywords can be generated per video sequence from one audio stream. You can then combine the search at the keyword level with the search at the model level. Such videos can be classified into the best positions that can be matched at the keyword (meaning) level as well as the motion model level.

Example (EXAMPLES)

Semantic visual template for searching video shots of slalom skiers

1. The system asks and the user answers questions about the context. Semantics Visual templates are classified under the label "slalom". The query is specified on an object basis that contains two objects.

2. The user sketches the initial query. Backgrounds with large margins represent ski slopes and smaller foreground objects represent skiers with characteristic zigzag motion trajectories.

3. The skier assigns the highest value of relevance to all features related to people and background.

4. The system creates one set or test icon, from which the user selects the desired feature change in the skier's color and motion trajectory.

5. Combine the four selected colors with the three selected motion trajectories to form 12 possible skiers. The list of skiers is combined into a single background to produce the 12 icons of FIG. In FIG. 7, groups of three adjacent icons can be understood to have the same color.

6. The user selects a set of candidates to query the system. The system searches for the 20 closest video shots. The user provides relevance feedback to guide the system to a set of small sized downhill skiers.

Sunsets. A database containing 72 sunsets over 1952 video shots was used. Semantics Using only initial sketches without a visual template, 10% reproduction and 35% accuracy were achieved. Semantics Using the visual template, eight icons can be created and 36 sunsets. 50% reproduction and 24% accuracy are achieved.

High Jumpers. The database contains nine high jumpers in 2589 video shots. With no semantic visual template, 44% reproduction and 20% accuracy are achieved. By using semantic visual templates, we were able to improve 56% reproduction and 25% accuracy. The system converged into a single icon different from the initial sketch provided by the user.

Claims (24)

(a) obtaining at least one initial query for the concept, (b) generating at least one additional query related to the initial query, (c) checking the additional query against the concept; (d) including the additional query in a visual template for the concept, if the additional query is appropriate Containing How to generate visual templates for concepts with a computer. In claim 1, Wherein each query is represented by an icon / example image. In claim 1, And the initial query is obtained through a sketch pad. In claim 1, The additional query generating step includes advancing the query feature by a step size that is inversely related to the importance associated with the query feature. In claim 1, The generating of the additional icon includes forming a combination of desirable feature values. In claim 1, The adequacy check is a visual template generation method confirmed through a two-way user dialogue. In the method of connecting a natural language subset to a semantic visual template to query a video database about a concept with a computer, (a) obtaining a text query, (b) generating an image property by analyzing the query; (c) generating an image query using the image property; (d) retrieving information using the image query; and (e) displaying the information Containing How to query. In claim 7, The text query is obtained from a keyboard. In claim 7, The subset of the natural language includes a set of small nouns, verbs, prepositions, adjectives, and adverbs. In claim 7, Interactively expanding the subset. In claim 7, The image property generating step (Iii) establishing a correspondence between the query and the natural language subset, (Ii) marking different parts of the query as nouns, verbs, adjectives or prepositions, and (Iii) obtaining a description of whether a word in the query is not in the natural language subset and marking the word accordingly. Containing How to query. In claim 7, The image query generating step Setting a correspondence between the natural language subset and a semantic visual template set generated by the method of claim 1, The semantic visual template is an image implementation of a noun in the query, an adjective is used to modify the image implementation of the noun, a verb is used to implement an action, and a preposition is used to determine the spatiotemporal order necessary to generate an image query. Used to set How to query. (a) means for obtaining at least one initial query for the concept, (b) means for generating at least one additional query related to the initial query, (c) means for checking the additional query against the concept and (d) means for including the additional query in a visual template for the concept, if the additional query is appropriate Containing Computer system for creating visual templates for concepts. In claim 13, Wherein each query is represented by an icon / example image. In claim 13, And the initial query is obtained through a sketchpad. In claim 13, And said additional query generating means comprises means for advancing said query feature by a step size that is inversely related to the importance associated with said query feature. In claim 13, Said additional icon generating means comprises means for forming a combination of desirable feature values. In claim 13, The suitability check is ascertained through two-way user dialogue. In a computer system that associates a semantic visual template with a semantic visual template and queries a video database for concepts, (a) means for obtaining a text query, (b) means for analyzing the query to generate image attributes, (c) means for generating an image query using the image attribute, (d) means for retrieving information using the image query and (e) means for displaying the information Containing Computer system. The method of claim 19, And the text query is obtained from a keyboard. The method of claim 19, The subset of the natural language includes a set of small nouns, verbs, prepositions, adjectives, and adverbs. The method of claim 19, And means for interactively expanding the subset. The method of claim 19, The image property generating means (Iii) means for establishing a correspondence between the query and the natural language subset, (Ii) means for marking different parts of the query, such as nouns, verbs, adjectives or prepositions; (Iii) means for obtaining a description of whether a word in said query is not in said natural language subset and for marking said word accordingly; Containing Computer system. The method of claim 19, The image query generating means Means for establishing a correspondence between the natural language subset and a set of semantic visual templates generated by the system of claim 13, The semantic visual template is an image implementation of a noun in the query, an adjective is used to modify the image implementation of the noun, a verb is used to implement an action, and a preposition is used to determine the spatiotemporal order necessary to generate an image query. Used to set Computer system.