KR100841181B1 - Motion activity description method and apparatus for video - Google Patents

Abstract

The present invention provides a method and apparatus for describing a motion activity of a video, in particular, a motion activity feature as a statistical characteristic of the size and direction of motion parameters extracted from the video. Description Method and apparatus. According to the method and apparatus of the present invention, the signal characteristics and spatiotemporal temporal characteristics of an entire video, a representative video, and a specific section in time, which are difficult to express by a conventional video motion indexing technique. Perceptual features such as distribution, degree of change and pattern can be described, and digital such as video retrieval, remote surveillance, multimedia database, broadcasting filtering, etc. It can be effectively used for video service applications.

Motion pictures, motion activity descriptors, motion parameters, magnitude, direction, video retrieval, remote surveillance, multimedia database, broadcast filtering

Description

Method and apparatus for describing motion activity features of video TECHNICAL FIELD AND APPARATUS FOR VIDEO

1 is a flowchart of a method for describing a motion activity feature of the present invention;

2 is a block diagram of one embodiment of a motion activity feature description apparatus for implementing the method of the present invention;

3 is a block diagram of an embodiment of a motion activity feature description device of FIG. 2;

4A through 4J are block diagrams of an apparatus for generating a motion descriptor using a cumulative motion histogram according to an embodiment of the present invention;

FIG. 5 is a detailed block diagram of a motion estimation divergence processing unit in FIGS. 4B, 4D, 4F, 4G, and 4J;

6 is a detailed block diagram of the motion filter in FIGS. 4C and 4D;

7 is a detailed block diagram of a motion descriptor generator of FIGS. 4A to 4J;

FIG. 8 is a diagram for explaining an inter-image motion estimation method using a block matching algorithm (BMA) in FIG. 5; FIG.

9A to 9C are diagrams for explaining the divergence of motion estimation.

FIG. 10 is a diagram showing a spatial correlation between a current area and a peripheral area for explaining the motion estimation divergence processing in the motion estimation divergence processor in FIGS. 4B, 4D, 4F, 4G, and 4J; FIG.

FIG. 11 is a diagram for describing a motion histogram of motion direction data generated by the motion histogram generator of FIGS. 4A to 4J; FIG.

FIG. 12 is a diagram for describing a cumulative motion histogram of motion direction data accumulated by the cumulative motion histogram generator in FIGS. 4A to 4J; FIG.

FIG. 13 is an exemplary diagram of a cumulative motion histogram for a video composed of 35 images accumulated by the cumulative motion histogram generator in FIGS. 4A to 4J; FIG.

14 is an exemplary diagram for explaining an index (clipping) of a cumulative motion histogram accumulated by the cumulative motion histogram generator in FIGS. 4A to 4J;

15A to 15J are flowcharts illustrating a method of generating a motion descriptor using a cumulative motion histogram according to an embodiment of the present invention.

Fig. 16 is a detailed flowchart of the motion estimation divergence processing steps in Figs. 15B, 15D, 15G, 15H, and 15J.

FIG. 17 is a flowchart illustrating a motion magnitude and direction filtering process in consideration of visual characteristics in FIGS. 15C and 15D.

18A and 18B are detailed flow charts of the motion descriptor generation step in FIGS. 15A-15J.

19 illustrates an example video retrieval system using a motion descriptor according to an embodiment of the present invention.

20A is a diagram for explaining structuring of still images.

20B is a diagram for explaining structuring of a video.

21 is a diagram illustrating video structuring in units of scenes.

* Explanation of symbols for the main parts of the drawings *

1100: means for extracting motion vectors

2100-1 to 2100-6: means for extracting magnitude characteristics of a motion vector;

3100-1 to 3100-6: direction characteristic extraction means of the motion vector;

4100: means for calculating the representative direction vector 5100: a combiner

2: motion estimation divergence processor 3: motion filter

4: motion histogram generator 5: cumulative motion histogram generator

6: motion descriptor generation unit 7: motion size calculation unit

8: motion size histogram generator

9: cumulative motion size histogram generator

10: movement direction calculation unit 11: movement direction histogram generator

12: previous image storage unit 13: motion size calculation unit

14: cumulative movement direction histogram generator

15: motion size estimation value divergence processing unit

17: motion direction estimate divergence processor 22: current image storage unit

23: motion vector comparison converter

26: motion clip descriptor generation unit

32: average value calculation unit 33: movement direction calculation unit

36: motion descriptor generator 42: absolute value calculation unit

43: motion direction quantization / inverse quantization unit 52: motion vector value conversion unit

The present invention relates to a method and apparatus for describing a motion activity of a video (in this specification, the term video is used interchangeably), and more particularly, to extract a motion activity feature from a video. The present invention relates to a method and apparatus for describing a motion activity feature of a moving picture, characterized by describing the statistical characteristics of the magnitude and direction of the motion parameters.

At present, the continuous development of expression media (means for representing information that includes text, graphics, voice, sound and more broadly images) and transmission media (communication networks, broadcast networks, storage media) and the performance of their operating systems Therefore, there is a desire to freely generate, search for, and easily use and reuse large-capacity multimedia data consisting of a plurality of monomedia rather than a monomedia consisting of a small capacity single media (e.g., a single media consisting of data only data and voice only voice). It is increasing. As the desire for the creation of free multimedia data has been satisfied by the expression media electronics, vast quantities of monomedia or multimedia data have been scattered on personal or public systems.

However, as the amount of multimedia data increases, so does the time and cost of searching to use and reuse the data. Therefore, research and development of a search technology suitable for an effective search of multimedia data having a complex information attribute, including the character-based search technology that is widely used for faster and more efficient search has been actively conducted.

Effective indexing and retrieval of multimedia data requires the reduction of the size of information attributes representing the characteristics of each media data, the simplification and real-time of the preprocessing process, the validity and polymorphism of the information attributes representing the characteristics, and the flexibility in the search. .

In addition, objective similarity and subjective similarity of search results are also important factors in evaluating the performance of the search. The importance of subjective similarity is due to the limitation of description / expression of the information attribute that expresses the characteristics of media data. Even if the objective similarity is large, the effectiveness and practicality of the search are deteriorated if the user cannot obtain the intended search result. . Therefore, in recent years, researches on techniques that can reflect subjective similarities in information attribute technology expressing media data characteristics have been actively conducted.

The main differences between text-based retrieval and multimedia data-based retrieval, which are already widely used, include the difficulty of information attribute extraction and the polymorphism of information attribute representation.

In terms of difficulty of extracting information attributes, text data can be searched by indexing several key words and sentences in a document.However, in the case of multimedia, the data itself is large and mixed with various media. Appropriate preprocessing is required to obtain valid new information attributes combined.

The main goal of the preprocessing process is to extract feature information valid for retrieval and the effectiveness of the process. In other words, even if a technique capable of detecting valid feature information is expensive in terms of hardware (H / W) and software (S / W) in the process, a large amount of multimedia data must be processed and used in a short time. This is because it cannot be put into practical use in applications or terminal systems with poor system performance.

In the case of video retrieval as an example of the polymorphic aspect of information attribute expression, video is mixed with information of various media such as video, voice, and audio, and when the characteristics of the video are expressed, only the information attribute of each monomedia data is used. After extracting a valid information attribute that can be a feature of data by performing proper preprocessing on the information attribute of multimedia data mixed with two or more media, it is possible to search for various types of data using the extracted information attribute. . For example, an information property of an image may be used for the video search, and an information attribute of an image and an audio may be organically combined to be searched. Therefore, a search using various multi-media properties is more effective than a search using a single media property.

 Currently, the most researched field in multimedia data indexing and retrieval is the field of still image which is easy to acquire data. Still images are widely used in storage systems such as digital electronic still cameras and image databases, still image transmission devices, transmission systems such as audio graphic conferences, video conferences, and even printing systems such as printers. Still image indexing and retrieval is a content-based image retrieval method whose main interests are consistent characteristics such as rotation, resizing, and movement of image color, texture, and shape information. This method extracts and indexes visible features and searches using them.

Video retrieval is not as easy as acquiring data compared to still images, and its application is limited due to the need to store and process large amounts of data. However, due to the rapid development of storage media such as disks, tapes and CD ROMs, and transmission media such as communication networks, the cost required for data acquisition becomes low, and the required devices are miniaturized. . Normally video is a generic term for a series of images that have an order of acquisition over successive times. In addition, since video acquires image data in continuous time, spatial redundancy (redundancy) in the image and redundancy between neighboring images are very large, and thus inter-image prediction is possible. Due to this redundancy, the inter-image characteristics in the video are distinguished from the still image. The similarity between the images in the video retrieval technique is important for extracting feature information.

Redundancy between images in video can be measured using the degree of motion between images. For example, when the redundancy between images is large, this indicates that the size of the overlapping region between the two images is large, which may be interpreted as a small movement between the images. On the contrary, when the redundancy between images is small, the overlapping area between the two images is small and the movement between the images is large. Currently, standardized video compression techniques employ compression techniques using BMA-Block Matching Algorithm (H.261, H.261) to minimize data redundancy to improve data compression efficiency. H.263, MPEG-1, MPEG-2, MPEG-4).

Conventional video retrieval techniques use arbitrary size temporal intervals (hereinafter referred to as clips) based on changes in color, texture, shape, and motion between images. Video Structuring selects several key-frames that can represent the semantic and signal characteristics of the images in the section, and extracts and extracts feature information on the information attributes of the selected representative images. Or used for search.

The general structure of video structuring is the basic structure unit of 'shot', which is a continuous sequence of still images that are not temporally continuous, and 'scene', which is a unit of time and space in content as a temporal continuation of 'shot'. It can have a hierarchical structure such as 'story' which is a unit of story development at the level of 'competition' composed of ')' and 'scenes'.

This is illustrated in FIG. 20B. Video structuring can be structured in the form of Event Tree based on signal characteristics. In the structure of video, both signal and semantic structural information exist based on the link.

That is, the segment tree shown on the left side of FIG. 20B and the event tree shown on the right side are linked to each other in the direction of the arrow. For example, when searching for a Clinton Case structured in the Event Tree, Sub-segment1 in Segment 1 shown on the Segment Tree is searched. The video of shot 2 of 2 and the video of shot 3 of segment 3 are linked.

Even in the case of one still picture, the structure is possible. In the case of a picture of a person in the forest, it is a person and a forest object, which is a face and a body, and a face is composed of objects such as eyes, nose, and ears. This is illustrated in FIG. 20A. FIG. 20A is a diagram illustrating the structure of a still image. A signal structure of a region tree based on signal characteristics in an image and a semantic structure of an object tree based on an object having a perceptual meaning in the image are possible. Do. In general, signal structuring is performed using semi-automatic and automatic stationary structuring techniques, and since semantic structuring is a conceptual structuring method, a manual structuring technique by a user can be used.

Structure diagram of still picture shown in FIG. 20A Similar to the structure of video shown in FIG. 20B, the Region Tree shown on the left and the Object Tree shown on the right are linked to each other in the direction of the arrow to signal There are also both semantic and semantic structural information.

In the case of an acoustic signal, it has a structure composed of background noise of the surroundings, the sound of a person talking, background music, and the like.

Such data structuring has an advantage in that the larger the number of images constituting the video, the more accurate and various indexes and faster retrieval can be supported.

21 is a diagram representing video structuring in 'scene' units.

The indexing and retrieval method using the representative image is characterized by the signal characteristics, spatiotemporal distribution, and the main pattern of change in the video, the representative image, and the temporal region which cannot be represented only by the characteristic information of the representative image. Since it is not described, there is a disadvantage that it is not suitable for applications where such a characteristic is required.

On the other hand, the conventional techniques for moving picture description methods have been used for character, key still image (key frame), representative still image features, etc. This is a disadvantage that does not effectively describe the degree of motion activity that is a unique feature of the video there was. Conventional techniques for describing motion activity features in a video include a method of using features such as a camera movement or a trajectory of an image object, but this has a disadvantage in that it does not describe an overall motion activity feature that appears in the entire video screen. In addition, although the characteristics of the motion size can be expressed by using the motion size itself in the image, a feature indicating motion activity, which is a concept of the change amount of the motion of the image, has not been proposed.

Accordingly, one object of the present invention is to provide a method and apparatus for describing a motion activity feature of a motion picture which can efficiently and effectively describe and utilize motion motion of a motion picture by statistical characteristics of the size and direction of motion parameters in the motion picture.

Another object of the present invention is to provide a method and apparatus for describing a motion activity feature of a moving picture which can effectively describe the motion activity according to the degree of change of the motion as well as the magnitude of the motion of the objects in the image.

Another object of the present invention is an apparatus and method for generating a motion descriptor using a cumulative motion histogram that reflects a user's perceptual characteristics in motion indexing and retrieval of contents in a video and supports various levels of indexing and retrieval. To provide.

The first means for achieving the object of the present invention comprises the steps of extracting a motion parameter from the video; Extracting statistical characteristics of the magnitudes of the motion parameters extracted in the previous step; And extracting statistical characteristics of the direction of the motion parameter.

The second means for achieving the object of the present invention comprises a motion parameter extraction means for extracting the motion parameters from the video; Means for extracting a statistical characteristic of the magnitude of the input motion parameter from the motion parameter extraction means; Means for extracting a statistical characteristic of the direction of the motion parameter; And a combiner for collecting the extracted statistical characteristics to define a motion activity descriptor.

The third means for achieving the object of the present invention is a motion histogram generator for generating a motion histogram for each of the input motion size data and direction data, and the motion histogram generated by the motion histogram generator in a predetermined order A cumulative motion histogram generator that generates a cumulative motion histogram according to the present invention, and the video is arbitrarily sized (layered) according to a change amount of the cumulative motion histogram generated by the cumulative motion histogram generator, and the motion characteristics of each structured unit It comprises a motion descriptor generator for generating a motion descriptor for describing.

The fourth means for achieving the object of the present invention comprises a motion histogram generation step of generating a motion histogram for the input motion size and direction data, and the motion histogram generated in the motion histogram generation step in a predetermined order A cumulative motion histogram generation step of generating a histogram, and a cumulative motion histogram generated in the cumulative motion histogram generation step, the video is structured (layered) to an arbitrary size according to an amount of change over time, and motion is performed for each structured unit. A motion descriptor generation step of generating a motion descriptor describing the characteristic.

The fifth means for achieving the object of the present invention is a motion size histogram for the motion size calculation means for calculating the magnitude (degree) of the input motion size information, and the motion size information calculated by the motion size calculation means A motion size histogram generating means for generating, a cumulative motion size histogram generating means for generating a cumulative motion size histogram according to a predetermined order of the motion size histogram generated by the motion size histogram generating means, and the cumulative motion size histogram generating means And a motion descriptor generating means for structuring (layering) the video to an arbitrary size according to the amount of change in the accumulated cumulative motion size histogram, and generating a motion descriptor for describing the motion characteristics for each structured unit.

The sixth means for achieving the object of the present invention comprises a movement direction calculation means for calculating the direction of the input movement direction information; Motion direction histogram generating means for generating a motion direction histogram with respect to the motion direction information calculated by the motion direction calculating means; Cumulative motion direction histogram generating means for generating a cumulative motion direction histogram according to a predetermined order of the motion direction histogram generated by the motion direction histogram generating means; Motion descriptor generation means for structuring (layering) the video to an arbitrary size according to the amount of change in the cumulative motion direction histogram generated by the cumulative motion direction histogram generating means, and generating a motion descriptor for describing the motion characteristics for each structured unit. It is configured to include.

The seventh means for achieving the object of the present invention comprises: motion size calculation means for calculating the size (degree) of the input motion size information; Motion magnitude histogram generating means for generating a motion magnitude histogram with respect to the motion magnitude information calculated by the motion magnitude calculating means; Cumulative motion magnitude histogram generating means for generating a cumulative motion magnitude histogram according to a predetermined order of the motion magnitude histogram generated by the motion magnitude histogram generating means; Motion direction calculation means for calculating a direction of input motion direction information; Motion direction histogram generating means for generating a motion direction histogram with respect to the motion direction information calculated by the motion direction calculating means; Cumulative motion direction histogram generating means for generating a cumulative motion direction histogram according to a predetermined order of the motion direction histogram generated by the motion direction histogram generating means; According to the cumulative motion magnitude histogram generating means and the cumulative motion direction histogram generating means, the video is structured (layered) to an arbitrary size and the motion characteristics are described for each structured unit. And a motion descriptor generating means for generating a motion descriptor.

Eighth means for achieving the object of the present invention comprises a motion size calculation step of calculating the magnitude (degree) of the input motion size information; A motion magnitude histogram generation step of generating a motion magnitude histogram with respect to the motion magnitude information calculated in the motion magnitude calculation step; A cumulative motion magnitude histogram generating step of generating a cumulative motion magnitude histogram according to a predetermined order of the motion magnitude histogram generated in the motion magnitude histogram generating step; A motion descriptor for structuring (layering) the video to an arbitrary size according to the amount of change in the cumulative motion magnitude histogram generated in the cumulative motion magnitude histogram generation step, and generating a motion descriptor for describing motion characteristics for each structured unit. It comprises a generation step.

A ninth means for achieving the object of the present invention comprises a movement direction calculation step of calculating the direction of the input movement direction information; A motion direction histogram generating step of generating a motion direction histogram with respect to the motion direction information calculated in the motion direction calculation step; A cumulative motion direction histogram generating step of generating a cumulative motion direction histogram according to a predetermined order of the motion direction histogram generated in the motion direction histogram generating step; A motion descriptor generation step of structuring (layering) the video to an arbitrary size according to the amount of change in the cumulative motion direction histogram generated in the cumulative motion direction histogram generation step, and generating a motion descriptor describing motion characteristics for each structured unit. It is configured to include.

A tenth means for achieving the object of the present invention comprises a motion size calculation step of calculating the magnitude (degree) of the input motion size information; A motion magnitude histogram generation step of generating a motion magnitude histogram with respect to the motion magnitude information calculated in the motion magnitude calculation step; A cumulative motion magnitude histogram generating step of generating a cumulative motion magnitude histogram according to a predetermined order of the motion magnitude histogram generated in the motion magnitude histogram generating step; A motion direction calculation step of calculating a direction of input motion direction information; A motion direction histogram generation step of generating a motion direction histogram with respect to the motion direction information calculated in the motion direction calculation step; A cumulative motion direction histogram generating step of generating a cumulative motion direction histogram according to a predetermined order of the motion direction histogram generated in the motion direction histogram generating step; According to the cumulative motion magnitude histogram generation step and the cumulative motion direction histogram generation step, the video is structured (layered) to an arbitrary size and the motion characteristics are described for each structured unit. And a motion descriptor generation step of generating a motion descriptor.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the present invention, the motion activity feature for the video is described by the statistical characteristics of the motion parameters appearing in the video. That is, the present invention extracts the statistical value of the Nth order from the spatial axis, and extracts the statistical value of the Mth order from the time axis of these values and describes it as a feature value for one video unit. In particular, an embodiment of the present invention is characterized by extracting the magnitude and direction of the motion parameter as a statistical feature of the motion parameter of the video and extracting the first and second statistical values thereof as the motion activity descriptor. In the present invention, the primary statistical value means an average value and the secondary statistical value means a moment of any order, and preferably a standard deviation value or a variance value. In the following description, for convenience of explanation, the standard deviation value will be described as an example of the second statistical value. A video may consist of individual images and may be characterized by primary and secondary statistics of the size and direction of motion parameters in each image, and the primary and secondary statistics of these statistics extracted from one or more images. Secondary statistics can be used as feature values.

In general, a video can be divided into an image stored in an existing analog form and a video based on a recently proposed digital image. In the case of media such as MPEG or H.263 based on digital image compression technology, motion parameters are stored in bit streams, so it is easy to describe motion activity using them. In addition, it is possible to estimate motion parameters in any analog video by any motion estimation method, to describe motion activity using this method, and to search video according to this technology value. In particular, in video compression encoding scheme such as MPEG or H.263, video is divided into several blocks or objects and encoded. When encoding a block, in order to reduce the temporal redundancy of the image information, find a reference block most similar to the block to be encoded in a temporal neighboring image, and encode the relative position on the screen of the block to be referred to as a motion parameter. The compression efficiency is increased by encoding the difference between the block to be referenced and the reference block. The motion parameter of the k-th block of one screen may be expressed by, for example, two-dimensional MV _k = (MV _xk , MV _yk ) . Where MV _xk is a horizontal motion component and MV _yk is a vertical motion component. The motion parameter may be represented by the magnitude component I _k of the motion parameter represented by Equation 1 below and the direction component φ _k of the motion parameter represented by Equation 2 below.

In the present invention, the motion activity feature is described as a statistical characteristic of the magnitude and direction of the motion parameter. When each screen of the video is composed of M blocks or objects, first and second statistical values of the size and direction of motion parameters are extracted from each screen. The first statistical value I _av of the motion parameter size on one screen is obtained by the following Equation 3, and the second statistical value (eg, I _dev ) of the motion parameter size on one screen is expressed by Equation 4 below. The primary statistical value φ _av in the motion parameter direction is obtained by the following Equation 5, and the secondary statistical value φ _dev in the motion parameter direction in one screen is obtained by Equation 6 below. Can be done.

The motion activity descriptor for a single video composed of several consecutive screens is obtained by first-order statistics of the size and direction of motion parameters on one screen obtained by Equations 3 to 6 from the respective screens constituting the video. It is calculated | required by following formula (7)-formula 15 from a value and a secondary statistical value. As described above, in indexing the degree of motion activity of the video, some or all of the motion parameter statistics expressed by Equations 7 to 15 may be used as the motion activity descriptor.

The number of screens from which motion parameters are extracted from the video is T, and the average of motion parameters of the i th screen is I _{av , I, the} second parameter of motion parameters I _{dev , i} the average of motion parameter directions is φ _{av , i} , The secondary statistical value in the direction of the motion parameter is called φ _{dev, i} .

Equation 7 is used as a statistical descriptor representing a video composed of T images as an average of average values of motion parameter sizes obtained from each of T videos.

The feature value obtained by Equation 7 is an average value of motion parameter magnitudes on the time and space axes, and is a statistical value representing the average motion of the entire video and may be used as a representative feature when searching and expressing the video.

Equation 8 is used as a statistical descriptor representing a video composed of T images as a standard deviation of average values of motion parameter sizes obtained from each of T videos.

The feature value obtained by Equation 8 is a statistical feature value representing the movement activity that changes in time by the temporal activity of the motion parameter size, and can be useful for video search. In addition, this value can represent the reliability of the first-order statistical value of Equation 7 as the standard deviation of the size of the motion parameter over time. For example, a large standard deviation value of Equation 8 means that the distribution of the size of the motion parameter is large based on the first statistical value of Equation 7. In other words, a larger value means less reliability of the primary statistics. By using this property, the weight can be given by dividing by this value.

Equation 9 is used as a statistical descriptor representing a video composed of T images as an average of standard deviation values of motion parameter sizes obtained from each of T videos.

The feature value obtained by Equation 9 is the average value of spatial motion activity in one or more images, and is a statistical feature value expressing the degree of motion activity in space. With this feature value, the video can be searched according to the degree of spatial motion activity.

Equation 10 is used as a statistical descriptor representing a video composed of T images as standard deviations of standard deviation values of motion parameter sizes obtained from each of T videos.

The feature value obtained by Equation 10 is a feature value indicating both the degree of spatial motion activity and the degree of temporal motion activity.

Equation 11 is used as a statistical descriptor representing a video composed of T images as an average of average values of motion parameter directions obtained from each of T videos.

The feature value obtained by Equation 11 is a feature value representing the motion direction of at least one video as a spatial and temporal primary statistical value in the direction of the motion parameter.

Equation 12 is used as a statistical descriptor representing a video composed of T images as a standard deviation of average values of the motion parameter directions obtained from each of the T videos.

The feature value obtained by Equation 12 is a feature value representing a temporal change in the direction of the motion parameter value, that is, an activity degree, and the video may be searched according to the change amount of the direction.

Equation 13 is used as a statistical descriptor representing a video composed of T images as an average of standard deviation values of the motion parameter directions obtained from each of the T videos.

The feature value obtained by Equation 13 represents the spatial activity of the motion parameter as an average of spatial secondary statistics in the direction of the motion parameter. The feature value can be used to search for a video based on a rate of change in spatial direction.

Equation 14 below is a standard deviation of standard deviation values of the motion parameter directions obtained from each of the T videos, and is used as a statistical descriptor representing a video composed of T images.

The feature value obtained by Equation 14 is a feature value representing the degree of spatial and temporal activity in the movement direction.

The calculation of the statistical feature values for the magnitude and direction of the motion parameters described above is not necessarily obtained by the above Equations 3 to 14 but is calculated by other methods apparent to those skilled in the art. Can be.

In another exemplary embodiment of the present invention, the frequency of the motion parameter direction quantized in M directions may be included in the entire video screen. This value is sequenced, and N pieces are sequentially extracted from the most frequent direction vector, which is called a representative direction vector. Here, the motion parameters quantized in order of highest frequency may be expressed as φ _1, φ _2, ... φ _N.

φmax = 1, φ 2, ... φN>, N ≤M

By using this feature, the presenter provides a presenter capable of searching for a video having a movement in a desired direction. Each video or partial video composed of several screens may be represented by a motion activity descriptor composed of statistical elements of motion parameter size and direction representing the motion activity characteristics obtained above. Such motion activity descriptors can be used to compare videos according to motion activities, and can be used for various applications such as video search.

1 is a flowchart of a method for describing a motion activity feature of the present invention. In the case of describing a motion activity feature of a video by one aspect of the present invention, first, motion parameters are extracted from a given video (S1100), and when each screen of the video is composed of M blocks or objects, the motion parameters are displayed on each screen. Extract the statistical characteristic of the size of (S2100), and extract the statistical characteristic of the direction of the motion parameter (S3100). In the present invention, a method of extracting a motion parameter is typically to obtain a motion vector, but is not necessarily limited to this method.

In another embodiment of the method of the present invention, a motion activity descriptor extracted by the above method may be hierarchically described to describe the motion activity characteristics of the video.

In the present invention, it is preferable to use a motion parameter encoded by digital video encoding as the motion parameter. In addition, in extracting the statistical features of the motion parameter size and direction, the spatial statistical feature values of each image can be obtained, or the temporal statistical feature values of various images can be obtained. have.

In the step of extracting the statistical characteristic of the size of the motion parameter of the present invention (S2100), after obtaining the first statistical value of the size of the motion parameter for each image, the first statistical value of these values is a statistical feature representing the entire video. As a value (e.g., I _{av, av} ), or obtain a first order statistical value of the magnitude of the motion parameter for each image, then use the second order statistics (e.g., I _{dev , av} ) of the values It can be used as a statistical feature value representing. In this case, the second statistical values I _{dev and av} extracted as statistical feature values representing the entire video may be used as the reliability of the first statistical values I _{av and av} .

On the other hand, in the step of extracting the statistical characteristic of the size of the motion parameter (S2100), after obtaining the second statistical value of the size of the motion parameter for each image, the first statistical value of these values (eg, I _{av, dev} ) is obtained. Use the statistical features representing the entire video, or obtain the second statistical value of the size of the motion parameter for each video, and then use the second statistical values (eg, I _{dev , dev} ) of these values It can be used as a representative statistical feature value. Even in this case, the second statistical values I _{dev and dev} extracted as statistical feature values representing the entire video may be used as the reliability of the first statistical values I _{av and dev} .

In the present invention, in the step of extracting statistical characteristics of the direction of the motion parameter (S3100), after obtaining the first statistical value of the direction of the motion parameter for each image, the first statistical values (eg, φ _{av, av} ) of these values are obtained. Use the statistical characteristic values representing the entire video, or obtain the first statistical value in the direction of the motion parameter for each video, and then use the secondary statistical values (e.g., φ _{dev , av} ). It can be used as a statistical feature value representing. In this case, the second statistical values φ _{dev and av} extracted as statistical feature values representing the entire video may be used as the reliability of the first statistical values φ _{av and av} .

In the present invention, in the step of extracting the statistical characteristics of the direction of the motion parameter (S3100), after obtaining the second statistical value of the direction of the motion parameter with respect to each image, the first statistical values of these values (eg, φ _{av, dev} ) are obtained. Use the statistical characteristic values representing the entire video, or obtain the second statistical value in the direction of the motion parameter for each video, and then use the second statistical value (eg, φ _{dev , dev} ) of these values. It can be used as a statistical feature value representing. In this case, the second statistical values φ _{dev and dev} extracted as statistical feature values representing the entire video may be used as the reliability of the first statistical values φ _{av and dev} .

In addition, the method of the present invention can use the frequency of the motion parameter in the moving picture as the overall average direction of the moving picture.

Referring to the specific example of the motion activity feature description method of the present invention, the motion parameter is extracted at the time of moving image input, and the primary statistical value and the secondary statistical value of the motion parameter magnitude and direction component for each screen [eg, I _av (Equation 3), I _dev (Equation 4), φ _av (Equation 5), φ _dev (Equation 6)], and then the motion activity descriptor elements I _{av , av} (Equation 7), I _{av , dev} (Equation 8) _, I _{dev , av} (Equation 9) _, I _{dev , dev} (Equation 10) _, φ _{av , av} (Equation 11) _, φ _{av, dev} (Equation 12), φ _{dev , av} (Equation 13) _, φ _{dev , dev} (Equation 14) _, φ _max Calculate Equation (15) and record them as motion activity descriptors along with video position and video temporal length information. The motion activity descriptors can optionally be used in view of the application and the precision of the motion representation.

2 is a block diagram of one embodiment of a motion activity feature description apparatus of the present invention. The motion activity feature description apparatus of the present invention comprises: motion parameter extracting means 1100 for extracting motion parameters from a video; Means (2100) for extracting a statistical characteristic of the magnitude of the input motion parameter from the motion parameter extraction means; Means (3100) for extracting a statistical characteristic of the direction of the motion parameter; And a combiner 4100 that gathers the extracted statistical characteristics to define a motion activity descriptor.

3 is a detailed block diagram of an embodiment of the motion activity feature description apparatus of FIG. The motion parameters extracted from the video input by the motion parameter extracting means 1100 are input to the devices 2100-1, 2100-2, 3100-1, and 3100-2. The device 2100-1 uses I _av from a motion parameter value input from the motion parameter extracting means 1100. (Equation 3) is obtained and output to the device 2100-2, the device 2100-3, and the device 2100-5, the device 3100-1 is φ _av (Equation 5) is calculated and output to the apparatus 3100-2, the apparatus 3100-3, and the apparatus 3100-5. Device 2100-2 is a motion parameter value input from the motion parameter extraction means 1100 and I _av input from the device 2100-1 From Equation (3) I _dev Calculate Equation 4 and output it to the devices 2100-4 and 2100-6. Device 3100-2 is the input φ _av from Device 3100-1 Φ _dev (Equation 6) is calculated from the equation (5) and the motion parameter input from the motion parameter extracting means 1100 and output to the devices 3100-4 and 3100-6. Device 2100-3 is the I _av input from device 2100-1 Equation 3 from I _{av , av} The equation (7) is calculated and output to the apparatus 2100-5 and the gathering means 4100. Device 3100-3 is input via φ _av from Device 3100-1 Φ _{av and av} (Equation 11) are calculated from Equation 5 and output to the apparatus 3100-5 and the gathering means 4100. Device 2100-4 is the I _dev input from device 2100-2 From Equation 4 I _{av , dev} (Equation 8) Is calculated and output to the apparatus 2100-6 and the gathering means 4100. The device 3100-4 calculates φ _{av , dev} (Equation 12) from φ _dev (Equation 6) inputted from the device 3100-2, and outputs the result to the device 3100-6 and the aggregation means 4100. Device 2100-5 is I _av of the apparatus 2100-1 (Equation 3) from the device 2100-3 I _{av , av} The equation (7) is input and I _{dev , av} (Equation 9) is calculated and output to the aggregation means 4100. Device 2100-6 is the device I _dev from device 2100-2 (Equation 4) from device 2100-4 I _{av , dev} Input (Equation 8) to calculate I _{dev , av} (Equation 9), and outputs I _{dev , dev} (Equation 10) to the aggregation means (4100). The device 3100-5 receives φ _av (Equation 5) from the device 3100-1, φ _{av , av} (Equation 11) from the device 3100-3, calculates φ _{dev , av} (Equation 13), and calculates the aggregation means ( 4100). Apparatus 3100-6 receives the φ _dev (Equation 6) from the apparatus unit 3100-4 3100-2 from the input φ _{av, dev} (formula 12) φ _{dev, dev} Equation 14 is calculated and output to the aggregation means 4100. The representative direction vector extracting means 5100 receives the motion parameter from the motion parameter extracting means 1100, calculates the representative direction vector φ _{max ,} and outputs it to the aggregation means 4100. The aggregation means 4100 aggregates the output from the magnitude feature extraction means 2100 of the motion vector and the direction feature extraction means 3100 of the motion vector with the position and temporal length information of the moving picture data to form a moving picture motion descriptor.

4A through 4J are block diagrams of an apparatus for generating a motion descriptor using a cumulative motion histogram according to another exemplary embodiment of the present invention. As described below, in the present invention, the motion activity feature descriptor may be obtained using a cumulative motion histogram.

As shown in FIG. 4A, the motion descriptor generation apparatus using the cumulative motion histogram according to another embodiment of the present invention includes a motion histogram generator 4 for generating a motion histogram for input motion size information and direction information, respectively. A cumulative motion histogram generator 5 generating a cumulative motion histogram according to a predetermined order of the motion histogram generated by the motion histogram generator 4, and a cumulative motion histogram generated by the cumulative motion histogram generator 5. And a motion descriptor generation unit 6 for structuring (layering) the video to an arbitrary size according to the amount of change and generating a motion descriptor for describing the motion characteristics for each structured unit.

As shown in FIG. 4B, the motion descriptor generation apparatus using the cumulative motion histogram according to the embodiment of the present invention includes a motion estimation divergence processing unit 2 which processes divergence of an estimate of motion magnitude and direction by an external frame selection mode. The motion histogram generator 4 generates a motion histogram with respect to the motion size information and the direction information diverged by the motion estimate divergence processor 2, and a motion histogram generated by the motion histogram generator 4, respectively. The video is structured (layered) to an arbitrary size according to a cumulative motion histogram generator 5 generating a cumulative motion histogram according to a predetermined order, and a change amount of the cumulative motion histogram generated by the cumulative motion histogram generator 5. For each structured unit, the motion group describes the motion characteristics. It consists of a motion descriptor generator 6 for generating a predicate.

As shown in FIG. 4C, the motion descriptor generation apparatus using the cumulative motion histogram according to an embodiment of the present invention includes a motion filter 3 for performing visual filtering on input motion size information and direction information, and the motion filter. A motion histogram generator 4 for generating a motion histogram for the motion size information and the direction information filtered in (3), and a motion histogram generated in the motion histogram generator 4 according to a predetermined order. The video is structured (layered) to an arbitrary size according to the amount of change in the cumulative motion histogram generated by the cumulative motion histogram generator 5 and the cumulative motion histogram generator 5, and for each structured unit. Create a motion descriptor that creates a motion descriptor that describes the motion characteristics Consists of a part (6).

As shown in FIG. 4D, the motion descriptor generation apparatus using the cumulative motion histogram according to an embodiment of the present invention includes a motion estimation divergence processor 2 which processes divergence of an estimate of motion size information and direction information by an external frame selection mode. ), A motion filter 3 for performing visual filtering on the motion size information and the direction information diverged by the motion estimation value divergence processing unit 2, and the motion size information and the direction information filtered by the motion filter 3. A motion histogram generating unit 4 for generating a motion histogram for the cumulative motion, a cumulative motion histogram generating unit 5 for generating a cumulative motion histogram according to a predetermined order of the motion histogram generated in the motion histogram generating unit 4 and The cumulative motion histogram generated by the cumulative motion histogram generator 5 It consists of the video according to the variation of the ram to the motion descriptor generating unit 6 for structuring (layered) into an arbitrary size, and generating a motion descriptor for describing motion characteristics for the respective structured units.

As shown in FIG. 4E, the apparatus for generating a motion descriptor using a cumulative motion histogram according to an embodiment of the present invention includes a motion size calculator 7 for calculating a magnitude (degree) of input motion size information and the motion size. The motion magnitude histogram generator 8 which generates a motion magnitude histogram with respect to the motion magnitude information calculated by the calculation unit 7 and the motion magnitude histogram generated by the motion magnitude histogram generator 8 are accumulated in a predetermined order. The video is structured (layered) to an arbitrary size according to a cumulative motion magnitude histogram generator 9 for generating a motion magnitude histogram and a change amount of the cumulative motion magnitude histogram generated by the cumulative motion magnitude histogram generator 9, A motion that generates a motion descriptor that describes the motion characteristic for each structured unit Is composed of the caster generator (6).

As shown in FIG. 4F, the motion descriptor generation apparatus using the cumulative motion histogram according to the embodiment of the present invention includes a motion direction calculator 10 for calculating a direction of input motion direction information, and the motion direction calculator ( 10. A cumulative motion direction histogram according to a predetermined order of the motion direction histogram generator 11 generating a motion direction histogram with respect to the motion direction information calculated in 10) and the motion direction histogram generated by the motion direction histogram generator 11 in a predetermined order. The video is arbitrarily structured (layered) according to the amount of change in the cumulative motion direction histogram generator 14 and the cumulative motion direction histogram generated by the cumulative motion direction histogram generator 14, and each structured angle is generated. A motion that generates a motion descriptor that describes the motion characteristics for the unit Is composed of the caster generator (6).

As shown in FIG. 4G, the motion descriptor generation apparatus using the cumulative motion histogram according to an embodiment of the present invention includes a motion size estimate divergence processor 15 which processes divergence of the estimated value of the motion size information by an external frame selection mode. The motion size calculation unit 7 for calculating the magnitude (degree) of the motion size information divergently processed by the motion size estimation value divergence processor 15, and the motion size information calculated by the motion size calculator 7 A motion magnitude histogram generator for generating a motion magnitude histogram, and a cumulative motion magnitude histogram generator for generating a cumulative motion magnitude histogram according to a predetermined order of the motion magnitude histogram generated by the motion magnitude histogram generator 8 9) and the cumulative motion generated by the cumulative motion magnitude histogram generator 9 Group consists of the video according to the variation of the histogram as a motion descriptor generating unit 6 for structuring (layered) into an arbitrary size, and generating a motion descriptor for describing motion characteristics for the respective structured units.

As shown in FIG. 4H, the motion descriptor generation apparatus using the cumulative motion histogram according to an embodiment of the present invention includes a motion direction estimation value divergence processing unit 17 which processes divergence of an estimate of motion direction information by an external frame selection mode. A motion direction histogram with respect to the motion direction calculation unit 10 for calculating the direction of the motion direction information diverged by the motion direction estimation value divergence processing unit 17 and the motion direction information calculated by the motion direction calculation unit 10. A motion direction histogram generator 11 for generating a motion direction histogram generator 11, a cumulative motion direction histogram generator 14 for generating a cumulative motion direction histogram in a predetermined order, and a motion direction histogram generated by the motion direction histogram generator 11; Cumulative motion generated by the cumulative motion direction histogram generator 14 It consists of a motion descriptor generation unit 6 for structuring (layering) the video to an arbitrary size according to the amount of change in the directional histogram, and generating a motion descriptor for describing the motion characteristic for each structured unit.

As shown in FIG. 4I, a motion descriptor generating apparatus using a cumulative motion histogram according to an embodiment of the present invention includes a motion size calculator 7 for calculating a magnitude (degree) of input motion size information, and the motion size. The motion magnitude histogram generator 8 which generates a motion magnitude histogram with respect to the motion magnitude information calculated by the calculation unit 7 and the motion magnitude histogram generated by the motion magnitude histogram generator 8 are accumulated in a predetermined order. A cumulative motion magnitude histogram generator 9 for generating a motion magnitude histogram, a motion direction calculator 10 for calculating a direction of input motion direction information, and motion direction information calculated in the motion direction calculator 10. A motion direction histogram generator 11 for generating a motion direction histogram with respect to the motion direction histogram. A cumulative motion direction histogram generator 14 for generating a cumulative motion direction histogram according to a predetermined order of the motion direction histogram generated by the togram generator 11, the cumulative motion magnitude histogram generator 9, and the cumulative motion A motion descriptor for structuring (layering) the video to an arbitrary size according to the cumulative motion magnitude generated by the direction histogram generator 14 and the amount of change in the direction histogram, and generating a motion descriptor for describing motion characteristics for each structured unit. It is comprised of the generation part 6.

As shown in FIG. 4J, the motion descriptor generation apparatus using the cumulative motion histogram according to an embodiment of the present invention includes a motion size estimate divergence processing unit 15 which processes divergence of an estimate of motion size information by an external frame selection mode. The motion size calculation unit 7 for calculating the magnitude (degree) of the motion size information divergently processed by the motion size estimation value divergence processor 15, and the motion size information calculated by the motion size calculator 7 A motion magnitude histogram generator for generating a motion magnitude histogram, and a cumulative motion magnitude histogram generator for generating a cumulative motion magnitude histogram according to a predetermined order of the motion magnitude histogram generated by the motion magnitude histogram generator 8 9) and the divergence of the estimate of the motion direction information by the external frame selection mode. The motion direction estimation value divergence processing unit 17, the motion direction calculation unit 10 that calculates the direction of the motion direction information divergent by the motion direction estimation value divergence processing unit 17, and the motion direction calculation unit 10. A motion direction histogram generator 11 generating a motion direction histogram with respect to the calculated motion direction information and a motion direction histogram generated by the motion direction histogram generator 11 according to a predetermined order to generate a cumulative motion direction histogram. The video is randomly selected according to the cumulative movement direction histogram generator 14, the cumulative motion magnitude histogram generator 9, and the cumulative motion direction histogram generator 14, respectively, according to the cumulative movement magnitude and the amount of change in the direction histogram. Structured (layered) to the size of Which consists of a motion descriptor generating unit 6 for generating a motion descriptor.

FIG. 5 shows a detailed block diagram of the motion estimation divergence processing unit 2 in FIGS. 4B, 4D, 4G, 4H, and 4J.

As shown in FIG. 5, the motion estimate divergence processor 2 includes a previous image storage unit 12 in which a previous image is stored in advance, a current image storage unit 22 in which a current image is stored in advance, and the external frame selection mode. a second area to a first area MBc having a motion estimate in the current input image and a second to fourth current area stored in the current image storage unit 22 neighboring the first area according to (frame_select_mode), or a first thereof An average value for calculating an average value of the previous area of the first area MBc stored in the previous image storage unit 12 and the second to fourth previous areas previously adjacent to the first area, respectively, in the area MBc and the previous image storage unit 12; The difference between the calculation unit 32 and the first area and the second to fourth areas calculated by the average calculation unit 32 or the first area, the previous area of the first area, and the second to fourth previous area. Calculate the difference between and calculate the absolute value of each After comparing the absolute value calculated by the absolute value calculation unit 42 and the absolute value calculated by the absolute value calculation unit 42 and the predetermined threshold value, the X and Y motion vector values input by the motion estimation unit 1 according to the comparison result. A motion vector value comparison converter 52 converts (MVx, MVy) and outputs the motion vector values MVox, MVoy.

FIG. 6 shows a detailed block diagram of the motion filter 3 in FIGS. 4C and 4D.

As shown in FIG. 6, the motion filter 3 includes a motion size calculator for calculating a motion size using the X and Y motion vector values MVox and MVoy processed by the motion estimate divergence processor 2. 13), a motion vector value comparison converter 23 for comparing the motion size calculated by the motion size calculator 13 with a preset threshold value, and converting a motion vector value according to the comparison result, and the motion vector. The motion direction calculator 33 calculates a motion direction using the motion vector value converted by the value comparison converter 23, and the motion direction calculated by the motion direction calculator 33 is quantized and inversely quantized. It consists of a motion direction quantization / dequantization unit 43 which outputs a direction value [theta] xy.

FIG. 7 shows a detailed block diagram of the motion descriptor generation unit 6 in FIGS. 4A to 4J.

As illustrated in FIG. 7, the motion descriptor generator 6 includes a motion histogram change calculator 161 for calculating a change amount of the cumulative motion histogram accumulated by the cumulative motion histogram generator 5, and the motion histogram. A clip time indexing unit 162 for generating a motion clip descriptor by indexing the motion histogram change time and the number of clips calculated by the change amount calculating unit 161, and the motion histogram change amount calculated by the motion histogram change amount calculating unit 161; A comparison unit 163 for enabling the motion histogram change calculation unit 161 or the clip time index unit 162 and the clip time index unit 162 according to the comparison result after comparing a preset threshold value; A motion descriptor generator for generating a motion descriptor using the information described by the motion clip descriptor generated in 36).

The clip time indexing unit 162 and the comparing unit 163 operate only when the cumulative motion histogram generated by the cumulative motion histogram generating unit 5 is not structured, and the motion when the cumulative motion histogram is structured. The histogram change amount calculated by the histogram change amount calculator 161 is directly provided to the motion descriptor generator 36.

An apparatus and method for generating a motion descriptor using a cumulative motion histogram according to an embodiment of the present invention configured as described above are described in detail.

First, the motion estimation process will be described with reference to BMA (Block Matching Algorithm) mainly used in the current video compression standard illustrated in FIG. 8.

The BMA requires two previous and current images to perform motion estimation between images. The motion estimation unit is an area called a macroblock (MB) composed of 16x16 pixels. Therefore, after dividing the current image into MBs, the predefined motion estimation region Sr is searched based on the position in the previous image at the same position as the MB to be estimated in the current image (hereinafter referred to as MBc). Next, the position of the image of the MB size data most similar to the current MB image data is found, and the motion is represented by the difference vectors MVx and MVy from the MB position of the current image.

In general, the motion vector for the current MB must be estimated by comparing the data of all positions in the motion estimation region, but this method has a disadvantage in that it takes a lot of time to estimate the movement. Use fast motion estimation techniques such as Search. These techniques reduce the time required for motion estimation, but do not always guarantee optimal motion estimation.

In general motion estimation techniques, when all or part of the image data in the motion estimation region is the same, multiple positions in the motion estimation region may be selected, and thus, accurate motion estimation may be hindered without proper processing. It is called divergence.

Next, the motion estimate divergence processor 2 shown in Figs. 4B and 4D, the motion magnitude estimate divergence processor 15 shown in Fig. 4G, the motion direction estimate divergence processor 17 shown in Fig. 4H, and Fig. 4J are shown in Figs. Operations of the shown motion magnitude estimate divergence processor 15 and the motion direction estimate divergence processor 17, and the motion estimate divergence processing steps S12, S32, and S32 shown in Figs. 12B and 15D, and the motions shown in Fig. 15G. 15 illustrates the magnitude estimate divergence processing step S62, the motion direction estimate divergence processing step S74 shown in FIG. 15H, the motion magnitude estimate divergence processing step S91, and the motion direction estimate divergence processing step S92 shown in FIG. 15J. 5, 9A to 9C and 16 will be described in detail as follows.

An example of divergence of motion estimation is shown in Figs. 9A to 9C. FIG. 9A illustrates a case in which the entire motion estimation region has the same pixel value, FIG. 9B illustrates a region having the same pixel value in the vertical direction, and FIG. 9C illustrates a region having the same pixel value in the horizontal direction.

In the above case, FIG. 9A has the same similarity at any position in the motion estimation region, and FIG. 9B has the same similarity in the vertical direction and FIG. 6C in the horizontal direction. As such, when there are a plurality of locations having the same similarity, accurate motion estimation cannot be performed. This is called divergence of motion estimation.

The present invention proposes the following two methods for processing the divergence for the motion estimate.

The first method uses a direct current (DC) of neighboring areas MB1, MB2, and MB3 of the area MBc having the input motion estimate present in the current image, and the second method uses the current motion estimate area. DC in the region of the previous image and its surrounding area at the same position as is used. The spatial positional relationship with respect to these is shown in FIG.

The first method cannot accurately measure the divergence of estimated motion estimates, but due to the large redundancy due to the small difference in acquisition time between neighboring images, the previous image is used instead of the previous image used for actual motion estimation. This is because some of the image characteristics can be analyzed. This method is also suitable for applications that cannot store the entire previous image because of the limited storage capacity. Since the second method uses the information of the previous image, it is possible to more accurately measure the divergence of the motion estimate than the first method.

Referring to FIG. 10, FIG. 10 illustrates a spatial correlation between a current region and a peripheral region for explaining a motion estimation divergence processing in the motion estimation divergence processor in FIGS. 4B, 4D, 4F, 4G, and 4J. Drawing. The reason for using DC of the region having corresponding motion estimates (corresponding to MB1, MB2, MB3, MBc, and MBp) for the divergence processing of the estimated motion is that the DC value is the average value of the region and represents the whole region, and the local difference of the region. This is because it is less sensitive to noise. The above DC is calculated as follows using Equation 16.

Where Pi is the i-th pixel value of the region having the motion estimate, N is the number of pixels in the region, S is the sum of all pixels in the region, and DC is the average of the pixel values in the region.

Here, MBp represents a region in a previous image having an arbitrary size having the same spatial position as MBc in the image, and MB1, MB2, and MB3 represent a spatial position in the current image or the previous image according to the frame selection mode (frame_select_mode) of FIG. 5. Represents a peripheral area neighboring MBc. When using the first divergence processing method (frame_select_mode is the current image selection mode), MB1, MB2, and MB3 correspond to the surrounding areas of MBc, and when using the second method (frame_select_mode is the previous image selection mode), MB1 , MB2, MB3 correspond to neighboring peripheral regions in the spatial position of MBc in the previous image. In the present invention, since the size (Sr) and the estimation method of the motion estimation region are different depending on the application, there is no particular limitation. However, the size of the region does not need to be limited, but using a predefined size has an advantage in calculation.

FIG. 5 is a diagram showing a detailed block diagram of the motion estimate divergence processing unit 2 that performs the motion estimation divergence processing method described so far.

Here, the frame selection mode (frame_select_mode) is an external input mode for selecting an image to be used in the first method or the second method, and MVx and MVy are estimated motion vectors in the horizontal / vertical direction of MBc in the motion estimation unit 1. When the DC values of MB1, MB2, MB3, MBc, and MBp calculated by the average calculation unit 32 are MB1_DC, MB2_DC, MB3_DC, MBc_DC, and MBp_DC, respectively, the motion estimation value divergence processor of FIG. The operation of (2) and the motion estimation value divergence processing steps (S13-S93) of FIG. 16 are as follows.

First, when the frame selection mode (frame_select_mode) indicates the current image selection mode (S33), that is, the first divergence processing method, MB1_DC, MB2_DC, MB3_DC, and MBc_DC calculated by the average value calculation unit 32 are each in the current image. The average values of the regions obtained in the regions (S43).

First, if the absolute value S53 of the difference between MBc_DC and MB2_DC and MBc_DC and MB3_DC calculated by the absolute value calculator 42 is smaller than the defined threshold THO, the motion vector value comparison converter 52 considers no motion. The motion vector values MVox and MVoy are both converted to 0 (S63).

Second, if it is not the first and the absolute value S53 of the difference between MBc_DC and MB3_DC calculated by the absolute value calculator 42 is smaller than the defined threshold THO, the motion vector value comparison converter 52 moves in the horizontal direction. Considering that there is no motion, MVox is converted to 0, and MVoy outputs MVy as it is (S63).

Third, if the absolute value S53 of the difference between MBc_DC and MB2_DC calculated by the absolute value calculator 42 is not smaller than the defined threshold THO, the motion vector value comparison converter 52 does not correspond to the first and second. Considers that there is no movement in the vertical direction, converts MVoy to 0 and outputs MVox equal to MVx (S63).

Fourth, if it does not correspond to the first to third, the motion vector value comparison converter 52 considers that the motion estimation value does not diverge and MVox and MVoy output MVx and MVy as they are (S63).

On the other hand, when the frame_select_mode indicates the previous image selection mode, that is, the second divergence processing method, the MBc_DC calculated by the average value estimating unit 32 is a region obtained from the current image and the respective regions within the MBp_DC, MB1_DC, MB2_DC, and MB3_DC previous images. Are the averages of (S73).

First, if the absolute value S83 of the difference between MBc_DC and MBp_DC calculated by the absolute value calculating unit 42 is smaller than the defined threshold THO, the motion vector value comparison and converting unit 52 considers that there is no motion, All MVoys are converted to 0 (S93).

Second, if not the first, and the absolute value (S83) of the difference between MBc_DC and MB2_DC and MBc_DC and MB3_DC calculated by the absolute value calculation unit 42 is smaller than the defined threshold THO, the motion vector value comparison converter 52 ) Considers no movement and converts both MVox and MVoy to 0 (S93).

Third, if the absolute value S83 of the difference between MBc_DC and MB3_DC calculated by the absolute value calculator 42 is smaller than the defined threshold THO, the motion vector value comparison converter 52 does not correspond to the first and second. Considers that there is no movement in the horizontal direction, converts MVox to 0, and outputs MVoy equal to MVy (S93).

Fourth, if the absolute value S83 of the difference between MBc_DC and MB2_DC calculated by the absolute value calculator 42 is not smaller than the defined threshold THO, the motion vector value comparison converter 52 does not correspond to the first to third times. Considers that there is no movement in the vertical direction, converts MVoy to 0, and outputs MVox in the same manner as MVx (S93).

Fifth, if it is not the first to fourth, the motion vector value comparison converter 52 considers that the motion estimation value does not diverge and outputs MVox and MVoy calculated as MVx and MVy as they are (S93). .

The difference between MBp_DC, MB1_DC, MB2_DC, and MB3_DC based on MBc_DC in the divergence processing methods is that when the absolute value of the DC difference is smaller than the defined threshold, the image of the motion estimation region of MVx and MVy is global and horizontal. Alternatively, since there is a high possibility that the motion vector is equal to or almost indistinguishable from MBc in the vertical direction (no motion will occur), it is regarded as a wrong motion estimate, and an appropriate one of the above two methods is selected to perform divergence processing.

Next, the operation of the motion filter 3 shown in FIGS. 4C and 4D and the motion filtering steps S22 and S33 shown in FIGS. 15C and 16D will be described in detail with reference to FIGS. 6 and 17. Same as

Motion filtering is a process for reflecting human visual feeling and extracting main movements and movements in the image to improve subjective similarity with respect to movements, and aims to reduce data expressing movements by using visual limitations appropriately. The motion filtering consists of a filtering step of the motion magnitude and a filtering step of the motion direction.

As shown in FIG. 17, the following describes the filtering of the motion magnitude (S14-S54) in detail.

When inputting the motion vector values MVox and MVoy diverged by the motion estimate divergence processor 2 (S14), the motion magnitude calculator 13 calculates the magnitude of the motion Lmv using Equation 17 below. (S11).

The motion vector comparison converter 23 determines whether the calculated motion size Lmv is smaller than the defined threshold value THL (S34), even though the motion is actually generated in the image. It is assumed that random noise is generated in the image during unacceptable size or during image acquisition or processing, and the estimated motion values, MVfx and MVfy, are all converted to 0 (S44).

On the other hand, if the calculated motion size Lmv is greater than the defined threshold value THL (S34), the motion vector value comparison and converting unit 23 converts the estimated motion values MVfx and MVfy into the motion vectors MVx and MVy. Convert to a value (S54).

This is because, as mentioned in the description of the prior art, subjective similarity in multimedia data retrieval is an important factor affecting the performance of retrieval. The threshold of magnitude filtering (THL) can be calculated using experimental or statistical methods depending on the visual characteristics of the human being or the characteristics of the application.

Next, the motion direction filtering step (S64-74) will be described in detail as follows. As with the motion size filtering step, the purpose is to reflect the limitations that a human visually feels about the movement and to reduce the size of data required to express the movement of the image by effectively using the threshold value.

In order to reflect the visual limitation on the motion, the motion direction calculator 33 uses the motion vector values MVfx and MVfy converted by the motion vector value comparison converter 23, as shown in Equation 18 below. (θxy) is calculated (S64).

The motion direction quantization / dequantization unit 43 performs quantization and inverse quantization on the direction data θxy calculated by the motion direction calculation unit 33 before generating a histogram as shown in Equation 19 (S74). )). In this case, the quantization method used may be a linear method or a nonlinear quantization method reflecting visual characteristics.

Where q is a quantization factor for the direction, p is a quantization factor for the magnitude, θxy is the direction of MVfx and MVfy, and R (x) is an integer not greater than x.

Next, the motion histogram generator 4 shown in Figs. 4A to 4D, the motion magnitude histogram generator 8 shown in Figs. 4E and 4G, and the motion direction histogram generator 11 shown in Figs. 4F and 4H. And the motion magnitude histogram generator 8 and the motion direction histogram generator 11 shown in FIGS. 4I and 4J, and the motion histogram generation steps S2 and S13 shown in FIGS. 15A to 12D, (S23), (S342), the motion magnitude histogram generation steps S43, S64 shown in Figs. 15E and 15G, the movement direction histogram generation steps S53, S74 shown in Figs. 15F and 15H, and The motion magnitude histogram generation steps S83 and S95 and the movement direction histogram generation steps S84 and S96 illustrated in FIGS. 15I and 15J will be described in detail with reference to FIG. 11 as follows.

Histogram is a method frequently used in the field of image signal processing or pattern recognition because it can express the overall statistical characteristics of data to be analyzed hierarchically in 2-D, 3-D, etc. In the present invention, the motion histogram is used to describe the statistical characteristics of the motion of the video.

The motion histogram proposed by the present invention is a quantization / dequantization method or a general method of the motion filtering technique in which motion information (parameters representing motion directions (θxy), motion magnitude (Lmv) and other motions) of contents in a video is described. We divide the area represented by the information into several groups (bins) and indicate how often information corresponding to each group occurs. A method of obtaining a motion histogram may be expressed as Equation 20.

H (MVi) = SMVi / M

here,

The SMVi is the sum of the frequencies of the i-th group of motion information to be represented by the histogram, the M is the total frequency of motion data to be represented by the histogram, and H (MVi) is the probability of the motion information of the i-th group. Indicates. The SMV is a reference to a parameter representing a motion direction, a motion size, and a motion.

* Motion histogram enables analysis and expression of the overall and characteristic movements and patterns in the image. FIG. 11 is a motion histogram of the direction of movement in one image, and the thicker the line, the greater the number of regions having motion in the corresponding direction in the image. Therefore, the 2-D motion histogram of FIG. 11 may be useful when describing the overall statistical characteristics of each representative image or a video clip of any size in the structured video. However, since the 2-D motion histogram cannot express the detailed flow of motion information in the video clip, the present invention describes the motion feature using the 3-D cumulative motion histogram shown in FIG.

Next, the cumulative motion histogram generator 5 shown in Figs. 4A to 4D, the cumulative motion magnitude histogram generator 9 shown in Figs. 4E and 4G, and the cumulative motion direction histogram shown in Figs. 4F and 4H. Operation of the generator 14 and the cumulative motion magnitude histogram generator 9 and the cumulative motion direction histogram generator 14 shown in FIGS. 4I and 4J, and the cumulative motion histogram generation step shown in FIGS. 15A to 15D. (S3), (S14), (S24), (S35), cumulative motion magnitude histogram generation steps (S44), (S65) shown in Figs. 15E and 15G, and the cumulative motion direction histograms shown in Figs. 15F and 15H. The process of generating steps S54 and S75 and the cumulative motion magnitude histogram generating steps S85 and S97 and the cumulative motion direction histogram generating steps S86 and S98 shown in FIGS. 15I and 15J are shown in FIG. 12. And it will be described in detail with reference to Figure 13 as follows.

The cumulative motion histogram generator 5, the cumulative motion magnitude histogram generator 9, and the cumulative motion direction histogram generator 14 accumulate the motion histogram in a predetermined order to generate a 3-D motion histogram, It is used to express the motion characteristics of the video. In the present invention, this is called a cumulative motion histogram. The shape of the cumulative motion histogram can be represented, as shown in FIG. 12. In FIG. 12, fmv is motion information such as motion size and direction in each image, and F = {fmv (1), ...., fmv (n)} indicates a video set having motion information of arbitrary size. . H (x) corresponds to a motion histogram value for each motion information, and is a value of a ratio of a cumulative histogram.

In the generation of the cumulative motion histogram, what motion information is used and what is used as the reference of the y-axis in the accumulation is optional depending on the application.

The cumulative motion histogram can properly reflect the human visual characteristics using the motion filtering technique proposed in the present invention, and can express the motion flow and pattern of the entire video or during a specific time by using a small amount of motion characteristic information. . For example, FIG. 13 shows 35 scenes of "a scene where people and cars pass by a quiet street, and for some reason a continuous explosion occurred on the ground and in the sky, and after the local movement of people trying to avoid it, the explosion again." When the video is configured, the cumulative motion histogram generated by applying motion filtering with q of equation 19 at 45 ° for the motion direction information is shaped.

Here, the x axis represents the temporal position of each image in the video, and the y axis represents the motion direction information for each image. It can be seen that even a scene having a complex motion as in the example described above can be expressed very clearly and clearly as shown in FIG. 13 by using a cumulative motion histogram.

As described above, since a normal video may be composed of many images, data of a motion histogram and a cumulative motion histogram expressing a motion characteristic of each image in 2-D or 3-D in proportion to an increase in the number of images. The amount will also increase. In addition, the time required for searching also increases. In order to appropriately cope with such a problem, the present invention includes a motion description method using the histogram described above, and describes a video motion feature that can cope with more various levels of search (Search / Retrieval from Coarse to Fine Level). We propose a motion descriptor generation method. The proposed motion descriptor is hierarchically structured in multiple stages, and a search stage can be selected according to a user or an application of the present invention.

Next, the operation of the motion descriptor generation unit 6 shown in Figs. 4A to 4J and the motion descriptor generation steps S4, S15, S25, S36 and S36 shown in Figs. 15A to 15J. S45, S55, S66, S76, S87, and S99 will be described in detail with reference to FIGS. 7, 14, 18A, and 18B as follows.

In general, since a video may consist of many more images than the scene, for example, from above, the amount of data in the cumulative motion histogram representing the motion feature is also increased. In order to properly cope with the increase in the time required for searching as the amount of data in the cumulative motion histogram increases, the present invention proposes a motion descriptor for an effective indexing of the cumulative motion histogram. The proposed motion descriptor analyzes the features of the cumulative motion histogram and divides the sections with similar features into clips, and the motion descriptors of the cumulative motion histogram data included in the clip are effectively and flexibly applied to the motion descriptor. present. The overall flow of the motion descriptor generation process is shown in FIG. 18A.

In detail, the motion histogram change calculation unit 161 calculates the change amount of the cumulative motion histogram when the cumulative motion histogram accumulated by the motion histogram accumulator 5 is input (S17), and the clip time indexing unit. 162 generates a motion clip descriptor by indexing the motion histogram change time and the number of clips calculated by the motion histogram change calculator 161 (S37).

The motion descriptor generator 36 generates a motion descriptor using information described by the motion clip descriptor generated by the clip time indexing unit 162 (S47).

Looking at the motion descriptor in detail as follows. The motion descriptor divides the cumulative motion histogram representing the entire video into clips according to changes in time, and describes motion characteristics for each divided clip. In this case, the clip is represented by feature information on histogram data between temporal positions whose variation amount ΔHt exceeds a defined threshold THΔH. If the determination of the threshold THΔH with respect to the change amount ΔHt used in the clipping process may vary depending on the application of the present invention, the calculation of the threshold value may be determined using an experimental method or a statistical method. The clipping process can be shown as in FIG. 18B.

In detail, the motion histogram change calculation unit 161 inputs the change amount ΔHt of the motion histogram when the motion histogram change calculation unit 161 is input as shown in Equation 21 of the cumulative motion histogram accumulated by the motion histogram accumulator 5 (S17). The calculation is performed as in Equation 22 (S271).

The comparing unit 163 compares the change amount ΔHt calculated by the motion histogram change amount calculating unit 161 with the preset threshold THΔH (S272), and then the threshold value THΔH is changed to the change amount ( If larger than ΔHt, the motion histogram change calculation unit 161 is enabled to repeat the step S271, and if the threshold THΔH is equal to or smaller than the change amount ΔHt, clip time The index unit 162 is enabled.

The clip time indexing unit 162 is enabled by the comparing unit 163 and sequentially increases the change time t and the number of clips c of the change amount calculated by the motion histogram conversion amount calculating unit 161. The clip time to be indexed (S273).

FIG. 14 is an exemplary view of clipping by the method described above with respect to FIG. 13. In FIG. 14, the cumulative motion histogram is divided into eight clips, and each clip has frequency information for each of the eight directions and characteristic information about an average of the motion sizes for each direction. Displays the duration on the time axis. The size of the clip and the interval of the cumulative histogram represented by the clip are not limited in the degree of overlap. Also in the expression, one clip can be subdivided again to express hierarchically. However, for the accuracy of retrieval and validity of representation, every motion histogram that represents a video must belong to one or more clips.

The motion descriptor described so far is generated based on the process as shown in FIG. 18A and may be expressed as follows. The xbits representing the length of each information described by the motion descriptor (MotionDescripor) is the number of bits of arbitrary size required to represent each data. The present invention is not related to an effective compression method of motion feature information. Since the number of bits required to represent each data may be different, it is not limited and defined.

The information described by the motion clip descriptor (MotionClipDescriptor) generated in the motion descriptor generator 36 shown in FIG. 7 will now be described.

A motion descriptor is a descriptor located at the top of a motion descriptor proposed by the present invention.

The MotionDescriptor is a video_id describing the video identifier VideoID, time_des which is a TimeDescriptor indicating the temporal position of the video described by the MotionDescriptor, a level of the MotionDescriptionLevel indicating the described motion feature information description level, and a 2-direction about the motion direction. Direction_des of MotionDirectionDescriptor describing feature information of D or 3-D form, intensity_des of MotionIntensityDescriptor describing feature information of 2-D or 3-D form of motion size, and directions that can be usefully used in video search. Flag_used_des, which is the selection flag information about the size, and mot_sub_des of the MotionDescriptor describing the motion characteristics of the next stage, and n of NumberOfSubDescriptor, which indicates the number of flag_exist_sub_des and mot_sub_des indicating this.

In the present invention, the motion direction represented by the MotionDescriptor to statistically describe the motion characteristics of the representative image or the structured unit for the structured unit such as video, story, scene, shot, segment and subsegment mentioned in the video structure Table 1 shows the index using the mean for the size, the Central Momonent for the mean and secondary statistics, and the 2-D / 3-D cumulative motion histogram for the motion data.

MotionDescriptor Video identifier (VideoID) Description scope identifier (TimeDescriptor) Direction Descriptor video_id time_des direction_des Size Descriptor (IntensityDescriptor) Additional Descriptor intensity_des sub_des [1]. sub_des [n]

In the description of the characteristics using the motion descriptor, the additional descriptor in Table 1 needs to hierarchically express the motion characteristics of the structured video as in the conceptual representation of the MotionDescriptor (MD) according to the structure of the video data in Table 2. A recursive construct used when there is a need to display more specific characteristics.

VideoID, which is an identifier for a video whose motion characteristics are described by the MotionDescriptor, is described as shown in Table 3 to distinguish different versions of the same name.

Video Identifier (VideoID) Video Name Credit (VideoCredits) When / Time (VideoDate) Versions (VideoVersions) title credit date version

The video identifier (VideoID) in Table 3 describes the title describing the VideoName of the video name, the credit of VideoCredits describing the creator of the video, the source and provider of the video, and the date of video production (VideoDate). It consists of a date and a version describing the version (VideoVersion).

In the present invention, each field in Table 3 may be selectively used according to a situation in using a video identifier, and an existing video identifier may be used in addition to the identifier in Table 3. In addition, since a variable situation may occur depending on the application field of the proposed motion feature descriptor in the video identifier technology, there is no limitation on the description of the video identifier in the motion descriptor. Table 4 is the syntax of the video identifier in Table 3.

Next, a description range identifier (TimeDescriptor) is classified into a sequential range description and a random range description according to whether the motion descriptor represents all the motion data in the description range.

The sequential time descriptor means that the proposed motion descriptor expresses all motion data and characteristics within the description range. The start time (Start_Time), end time (End_Time), and duration ( Duration (Duration) refers to the interval between the start and end time, it is information about the time of the total cumulative histogram data. Thus an end time or period can optionally be used.

Random Time Descriptor (Random Time Descriptor) means that only part of the motion data and characteristics within the description range is defined as shown in Table 5. Sequential range descriptors and random range descriptors can be used together for efficient representation of the description range identifier.

Description scope identifier (TimeDescriptor) Sequential TimeDescription Random TimeDescriptor Start time (Start_Time) Duration End_Time TemporalPosition start duration end s [1] ....... s [n]

The usedSequential flag field in the description range identifier syntax of Table 6 indicates whether to use a sequential range description, and the usedRandom flag field indicates whether to use a random range description. NumberOfPosition corresponds to the total number of positions represented in the random range description.

In the present invention, each field of Table 5 may be selectively used according to a situation in the use of the description range identifier in a motion descriptor, and in addition to the identifier of Table 5, an existing efficient video description range identifier may be used.

Next, the direction descriptor (DirectionDescritpor) is a descriptor that expresses the statistical characteristics of the movement direction with respect to the motion data of each image or the entire image within the technical scope intended by the technology range identifier. It consists of central moments for the dominant directions, dominant directions, cumulative movement histogram and data about directions. As described above, the two-dimensional motion information is basically composed of a direction and a magnitude, and the motion information can be expressed as MV = (MVx, MVy) of a vector (MV-Motion Vector), where MVx is a horizontal motion component. (Magnitude) and MVy is the vertical motion component (magnitude).

The direction of motion ψ in the motion vector can be obtained by the following equation.

[Equation 2]

In addition, a number of methods for calculating a direction using a motion vector may exist, and the use thereof is not limited. (MVxk, MVyk) in Equation 2 is motion information for the k-th region when one image is divided into M regions having an arbitrary size.

ρ i is an average of the moving direction of the i-th image in the video and can be obtained by Equation (9). In addition to Equation 23, there are a number of methods for calculating the average of the direction of movement.

θ1 (MeanOfClipMotionHistogram) is an average of the averages ρ of movement directions in T images within a range intended by the technical range descriptor in the motion descriptor, and can be obtained by Equation (24). In addition to the equation (24), there are a number of methods for obtaining an average of the average ρ of the movement direction. The MeanOfClipMotionHistogram is an average value of the entire cumulative motion histogram with respect to the movement direction.

Here, T may not be equal to the number of all images in the description range.

The central moment with respect to the direction of movement represents a characteristic of the temporal distribution and the degree of distortion of the direction average ρ of each image in the direction average θ1 and may have a p-dimensional moment θp. The p-dimensional moment θp can be obtained by the following equation (25). There are many methods for obtaining the p-dimensional moment θp in addition to the equation (25).

The direction standard deviation sigma i representing the characteristics of the spatial distribution of the motion direction of the i-th image in the T images within the range intended by the technical range technician can be obtained by the following equation (26). There are many methods for obtaining the direction standard deviation?

The average σθ1 of the second statistical value indicating the average of the spatial distribution of the directions for the T total images in the range may be obtained by the following equation (27). There are a number of methods for obtaining the average sigma? 1 of secondary statistical values in addition to the method shown in Equation 27.

The central moment for the secondary statistical value in the direction is characterized by the spatial distribution and the degree of distortion of the secondary statistical value (στi) of each image in the average of the secondary statistical value (σθ1) and represents the p-dimensional moment (σθp). Can have The p-dimensional moment σθp can be obtained by the following equation (28). There are a number of methods for obtaining the p-dimensional moment (? Θp) in addition to the method given by equation (28).

Β, which represents the dominant direction of motion, generates a histogram of the direction using Equation 20, and the direction having the largest histogram value corresponds to β. They are sorted according to the size of the histogram value and are determined in order of the larger value. DataOfVideoMotionLength represents the average of the motion magnitudes for each bin of the cumulative motion histogram with respect to the motion direction.

As described above, the DirectionDescriptor describing the characteristics of the movement direction of the contents in the video is summarized in the following Table 7.

Direction Descriptor Direction of movement average Movement direction temporal distribution Moving direction deviation mean Spatial direction of movement θ ₁ θ ₂ . θ _p σ _θ1 sigma _θ1 . σ _θp Dominant direction of movement Cumulative histogram DataOfVideoMotionLength β ₁ . β _k H _θ n _{1 .} n _M

The following Table 8 shows the syntax of the motion direction descriptor (DirectionDescriptor), conceptually representing a MotionDescriptor (MD) according to video data structure. The flag of flag_exist_sub_des is used when the direction characteristic using the motion direction descriptor is needed to express hierarchically the motion characteristic of the structured video as shown in Table 8, or when it is necessary to display the detailed direction characteristic. Recursive syntax.

Next, the IntensityDescritpor is a descriptor that expresses the statistical characteristics of the motion size with respect to the motion data for each image or the entire image within the technical scope intended by the technology range identifier. It consists of Central Moments for values, cumulative size histogram and data about size.

As described above, the two-dimensional motion information is basically composed of a direction and a magnitude, and the motion information can be expressed as MV = (MVx, MVy) of a vector (MV-Motion Vector), where MVx is a horizontal motion component. (Magnitude) and MVy is the vertical motion component (magnitude).

The magnitude I of the motion in the motion vector can be obtained by the following equation. In addition, it can be calculated using a method other than Equation 1, and there is no limitation on the method of calculating the motion size.

[Equation 1]

Here, (MVxk, MVyk) is motion information for the k-th region when one image is divided into M regions having an arbitrary size.

[lambda] i is an average of the motion size of the i-th image in the video, and can be obtained by the following equation (29). In addition, it may be calculated using a method other than Equation 29, and there is no limitation on the method of calculating the average of the motion magnitudes.

ω1 (MeanOfClipMotionLength) is an average of the mean λ of the motion sizes in the T images within the range intended by the technical range descriptor in the motion descriptor, and can be obtained by the following equation (30). In addition, it may be calculated using a method other than Equation 30, and there is no limitation on the method of calculating the average of the motion magnitudes. In other words, the MeanOfClipMotionLength represents an average value of the motion magnitude of the entire cumulative motion histogram.

Here, T may not be equal to the number of all images in the description range.

The central moment for the motion magnitude represents a characteristic of the temporal distribution and the degree of distortion of the magnitude average λ of each image in the magnitude average ω1 and may have a q-dimensional moment ωq. The q-time moment ωq can be obtained by the following equation (31). In addition, it can be calculated using a method other than Equation 31, and there is no limitation on the method of calculating the moment (ωq) of the q dimension.

The magnitude second statistical value (σλi) representing a feature of the spatial distribution of the motion size of the i-th image in the T images within the range intended by the technical range technician may be obtained by the following equation (32). In addition, it can be calculated using a method other than Equation 32, and there is no limitation on the method of calculating the magnitude second statistical value?

The average σω1 of the second statistical value indicating the average of the spatial distribution of the sizes of the T total images in the range may be obtained by the following equation (33). In addition, it can be calculated using a method other than Equation 33, and there is no limitation on the method of calculating the average (σω1) of the secondary statistical values.

The central moment for the second-order statistics of magnitude represents the spatial distribution and distortion of the second-order statistics (σλi) of each image from the mean of the second-order statistics (σω1), and the q-dimensional moment (σωq) It may have a. The q-dimensional moment (σωq) can be obtained by the following equation (34). In addition, it can be calculated using a method other than Equation 34, and there is no limitation on the method of calculating the q-dimensional moment σωq.

DataOfVideoMotionDirection represents the average value of the direction of motion for each bin of the cumulative motion histogram with respect to the motion size.

 As described above, IntensityDescriptor describing characteristics of the motion size of the contents in the video is summarized in the following Table 9.

IntensityDescriptor Motion size average Motion size temporal distribution Movement size deviation mean ω ₁ ω ₂ . ω _q σ _ω1 Motion size spatial distribution Motion magnitude cumulative histogram DataOfVideoMotionDirection sigma _ω2 ... σ _ωq H _ω n ₁ . n _m

Table 10 shows the syntax of the motion size descriptor, and conceptually represents a MotionDescriptor (MD) according to video data structuring. In Table 10, the flag of flag_exist_sub_des is a recursion used when size characterization using a motion size descriptor is needed when hierarchically expressing motion characteristics of a structured video or when it is necessary to display more detailed size characteristics. Is a syntactic construct.

Next, a motion histogram descriptor (MotionHistogramDescritptor) is a method of describing statistical characteristics of motion information such as direction and size, and the number of bins (NumberOfBins or NumberOfHistogramBin) displayed by the histogram and the frequency value (BinValueOfHistogram or DataOfVideoMotionHistogram) and RepresentativeValueOfBin in the actual data that the bean means.

The NumberOfMotionHistogramBin corresponds to the bin number of the cumulative histogram with respect to the motion size and direction, that is, the number of groups of motion data to be represented by the histogram. The DataOfVideoMotionHistogram means data of a cumulative motion histogram with respect to the motion size and direction.

The Motion Histogram Descriptor is summarized in Table 11 below.

In general, the number n of histograms can be selectively determined by the user in consideration of application fields, precision of characteristic values to be represented in the histogram, and overall data size.

Motion HistogramDescriptor Histogram Frequency (NumberOfBins) Histogram Value (BinValueOfHistogram) n bin_value [1]... bin_value [n] Bean's RepresentativeValueOfBin Motion sub-cumulative histogram rvalue_of_each_bin [1]... rvalue_of_each_bin [n] H ₁ . H _m

Table 12 below shows the syntax of the motion histogram descriptor.

In Table 12, NumberOfSubHistogram and MotionHistogramDescriptor are used to describe more detailed or corrected information about motion feature information in a 2-D or 3-D auxiliary histogram. For example, when using the motion direction cumulative histogram (Hθ), the MotionHistogramDescriptor can display information about the motion size according to each bin, and the motion generated by using a method different from the motion direction cumulative histogram (Hθ). It may be a cumulative histogram.

If motion feature information is described hierarchically according to general video structure using MotionDescriptor proposed in the present invention, it will be described as Table 13 and Table 14. In Table 13, MD stands for MotionDescriptor.

Table 14 conceptually represents a MotionDescriptor (MD) according to video data structure. As shown in Table 14, the MotionDescriptor can describe the entire video or structured units of arbitrary sizes such as story, scene, shot, and segment, and can describe the characteristics of video motion data in each unit. Hierarchical representation is possible with the more detailed feature description of lower level in general motion feature description of.

In addition, in the search using the video motion characteristics described by the MotionDescriptor, as shown in Table 15 below, by assigning usable flags (Flag used_direction_des, used_intensity_des) to each field in the descriptor, the user can search using the desired feature information. In addition, the exist_sub_des flag can be used to search the entire video or a specific unit section.

MotionDescriptor {VideoID video_id TimeDescriptor time_des MotionDescriptionLevel level Flag used_direction_des Flag used_intensity_des if (used_direction_des) MotionDirectionDescriptor direction_des if (used_intensity_des) MotionIntensityDescriptor intensity_des

In the case of the DirectionDescriptor and the IntensityDescriptor, the usable flag can be set for the average, the Central Moments, and the histogram.

19 is a system for video search to which the motion descriptor proposed in the present invention can be applied. The operation of the video retrieval system using the motion descriptor shown in FIG. 19 will now be described.

First, the motion descriptor is extracted by the motion descriptor extractor 102 from the query video 101 interfaced by the graphical user interface 100 or retrieved from the database DB. The extracted motion descriptor is encoded by the motion descriptor encoding apparatus 103, multiplexed via the multiplexing apparatus 104, and then transmitted to the server side via a network.

The demultiplexer 105 on the server side demultiplexes the multiplexed motion descriptors, and the motion descriptor decoder 106 decodes them. The motion descriptor similarity comparator 107 of the search engine 106 determines the similarity between the motion descriptor decoded by the motion descriptor decoder 106 and the motion descriptor stored in the multimedia database 114 of the database engine 112. After the comparison, the motion descriptors having high similarity are ranked by a predetermined number according to the comparison result.

On the other hand, the query video 101 at the client side is decoded via the multimedia decoder 109, and then the process is repeated.

On the other hand, the video 110 on the server side is stored in the multimedia database 114 via the database building device 113 after the motion descriptor is extracted by the motion descriptor extraction device 111.

As such, the technique proposed in the present invention may be applied to a motion descriptor extraction apparatus, and a brief example thereof is illustrated in FIG. 19.

The present invention uses the above-described cumulative motion histogram data and a MotionDescripor descriptor that describes the effect more effectively in video retrieval. Conventional similarity measurement methods (such as SAD-Sum Of Absolute Difference, MSE Mean Square Error) can be applied to the flow and pattern of the motion of the entire video or a specific section. In addition, when more accurate similarity measurement is required, DataOfVideoMotionHistogram or DataOfVideoMotionLength in MotionDescripor can be used. In addition, it is also possible to search video by time using Start_Time and End_Time.

Digital video application service using the present invention

The motion descriptor of the present invention can be used for the degree of movement to effectively utilize the limited bandwidth and system resources such as mobile communication environment, video conferencing, digital broadcasting, and VOD supported by the digital video service such as the Internet and IMT-2000. It can be used in various digital video service fields. Among them, video browsing, remote monitoring, retrieval, and refurbishing are described as examples.

1) Browsing Browising )

Browsing refers to information retrieval activity in which a user moves through various forms of links that exist between data through a multimedia database. One of the essential functional elements that must be supported for fast and efficient browsing of large media data such as video is the user interface in visual form of structure, summary information, representative image, and feature information of alternative video on the video being used by the user. This can be provided by means of which the user can see or hear. There is also a need for the ability to quickly browse video segments of the same or different categories by classifying them into several categories according to the degree of movement.

2) Remote monitoring Surveilance )

Remote monitoring is an application where there are expectations of commonly occurring movement characteristics that must be recorded and monitored for events occurring in confined spaces. Therefore, video segments with motion characteristics belonging to different categories that differ from the usual expectations contain important information about the event, so when using the degree of movement, you can control the amount of data to be recorded depending on the degree of movement. It can also support efficient search.

3) Video Search ( Video Retrieval )

Video retrieval is another form of search activity that finds the information needed with browsing, usually using a natural language, a structured computer language, etc. to express a query or present similar examples as additional information. Video retrieval is performed in units of video segments composed of a plurality of frames having a temporal order, and a plurality of objects of various types exist in the segments. There are areas of various colors, shapes, and textures or objects. Therefore, if the information about the general degree of movement such as the degree of movement is used in the initial stage of the search rather than the search by the specific color, shape, shape, or texture of the composition, the scope of the search can be greatly reduced, so that the faster search is possible. In addition, when information on specific colors, shapes, and textures is not available, motion information based on changes in content can be used for more efficient search. For example, a scene in which an anchor appears in a database of a news program can be found. If you want to search, it's usually done by different clothes, different studios and different people every day, so you can "find a scene with very little movement" and then use more specific features rather than using colors, shapes, and textures. It is desirable to. Also, information about the direction can be very useful when there is an object with a consistent direction in the video.

4) Video Repurposing ( Video Repurposing )

Video refurbishing refers to an application that reprocesses and services appropriately according to user requirements and user system capabilities using encoded streams on the server. In particular, when performing a digital video service for a system having a limited bandwidth and performance, such as a mobile communication terminal, the degree of motion may be used to control the number of transmission frames, that is, the bit rate, in order to use the bandwidth more efficiently. . For example, if the movement of the GOP (Group Of Picture) to be transmitted is small, it is considered that no significant change to the content has occurred. Therefore, data is not transmitted or the number of frames to be transmitted is reduced. It can be regarded as occurring and transmit a larger number of frames to provide more accurate information to the user.

The present invention is economically advantageous when the method proposed in the present invention is used for video retrieval by proposing a descriptor of motion feature information of a video to be used in multimedia data retrieval.

In addition, the motion activity descriptor of the present invention can identify the entire video, the representative image, and the temporal image that are difficult to express by the existing video motion indexing technique that uses only characteristic information of a few representative images in searching and identifying the content of the video or partial video. Signal characteristics, space-time distribution, and perceptual characteristics such as the degree of change and pattern of the interval can be described, which can be used in digital video service applications where such a degree of movement is an important feature. In addition, the feature information of the motion activity descriptor can be selectively used in consideration of the application field and the precision of the motion expression, and the video compressed using the existing video compression techniques MPEG-1, -2, -4 and H.263 In the case of the stream, the motion information in the video stream can be used directly without any additional processing, so the complexity required for feature information extraction is very small, and can be applied to applications requiring real-time processing.

Claims (22)

Extracting a motion parameter from a video; Extracting statistical characteristics of the magnitudes of the motion parameters extracted in the previous step; And Extracting a statistical characteristic of the direction of the motion parameter, Extracting the statistical characteristic of the magnitude of the motion parameter is extracting a spatial statistical feature value of the magnitude of the motion parameter of each image, and extracting the statistical characteristic of the direction of the motion parameter is the motion of each image. A method of describing a motion activity feature of a video, comprising extracting spatial statistical feature values of a direction of a parameter. The method of claim 1, wherein the method uses a motion vector encoded by digital video encoding as the motion parameter. The method of claim 1, wherein the step of extracting the spatial statistical feature value of the size of the motion parameter comprises obtaining a standard deviation value I _dev of the size of the motion parameter on each screen of a video composed of T screens from which the motion parameters are extracted. And using a mean (I _av , _dev ) of the standard deviation values of the motion parameter sizes obtained from the T-screen video as a statistical characteristic value representing the entire video. Activity feature description method. The method of claim 3, wherein the standard deviation value I _dev of the magnitude of the motion parameter is obtained by Equation 4 below, and the average values I _av and _dev of the standard deviation values are obtained by Equation 9 below. Method of describing motion activity feature of moving picture. [Equation 4]

Where I _k is the magnitude component of the motion parameter, I _av is the mean value of the magnitude of the motion parameter,       M is the number of blocks or objects constituting one screen, [Equation 9]

In the above formula, I _{dev , i} is the standard deviation value of the motion parameter size of the i-th screen,        T is the number of screens from which motion parameters are extracted from the video. 5. The method of claim 4, wherein the method calculates the standard deviation value I _{dev, av} of the mean of the motion parameters according to the following equation (7) of the mean value I _{av , av} of the mean values of the motion parameter magnitudes of Equation 8 below. The method of claim 1, further comprising the step of using reliability. [Equation 7]

In the above formula, I _{av , i} is an average of motion parameters of the i th screen, T is the number of screens from which motion parameters are extracted from the video. [Equation 8]

In the above formula, I _{av , i} is an average of motion parameters of the i th screen, I _{av , av} are the first-order statistics of the first-order statistics of the motion parameter size,       T is the number of screens from which motion parameters are extracted from the video. The method of claim 1, wherein the step of extracting the statistical characteristics of the size of the motion parameter is to calculate the standard deviation value I _dev of the size of the motion parameter by the following Equation 4 for each image, And using the standard deviation values (I _dev , _dev ) of these values as statistical feature values representative of the video as a whole. [Equation 4]

Where I _k is the magnitude component of the motion parameter, I _av is the first statistical value of the magnitude of the motion parameter,      M is the number of blocks or objects constituting one screen. [Equation 10]

In the above formula, I _{dev, i} are standard deviation values of the motion parameter size of the i-th screen, I _{av and dev} are averages of standard deviation values of the motion parameter sizes obtained from each of the T videos generated by Equation 9 below. T is the number of screens from which motion parameters are extracted from the video. [Equation 9]

In the above formula, I _{dev , i} is the standard deviation value of the magnitude of the motion parameter, T is the number of screens from which motion parameters are extracted from the video. Claim 6, wherein the method is for the standard deviation value of the standard deviation of the size of the motion parameters of the equation 10 (I _{dev, dev)} the average value of the size standard deviation value of the movement parameter of the equation (9) to (I _{av, dev} ) further comprising the step of using the reliability of the movement activity feature description method. [Equation 10]

In the above formula, I _{dev , i} are second order statistics of the motion parameter size of the i-th screen, I _{av and dev} are averages of standard deviation values of motion parameter sizes obtained from each of the T videos generated by Equation 9 below. T is the number of screens from which motion parameters are extracted from the video. [Equation 9]

In the above formula, I _{dev , i} is the second statistical value of the motion parameter size of the i-th screen, T is the number of screens from which motion parameters are extracted from the video. The method of claim 1, wherein the step of extracting the time statistical feature value in the direction of the motion parameter is to obtain a standard deviation value φ _dev of the direction of the motion parameter for each image, and then, averages the standard deviation values φ _av. and _dev ) as a statistical feature value representative of the video as a whole. The method of claim 8, wherein the standard deviation value of the direction of the motion parameter is obtained by the following equation (6), and the average of the standard deviation values (φ _av , _dev ) of the direction of the motion parameter is obtained by the following equation (13). The motion activity feature description method of a video characterized by the above-mentioned. [Equation 6]

Where _k is the direction component of the motion parameter, φ _av is the first statistical value in the direction of the motion parameter, M is the number of blocks or objects constituting one screen. [Equation 13]

In the above formula, φ _{dev , i} is a second statistical value in the direction of the motion parameter of the i-th screen, T is the number of screens from which motion parameters are extracted from the video. The method of claim 1, wherein the step of extracting the statistical characteristics of the direction of the motion parameter is obtained by calculating the standard deviation (φ _dev ) of the direction of the motion parameter by the following equation (6) for each image. And using the standard deviation values (φ _dev , _dev ) of these values obtained as the statistical feature values representative of the video as a whole. [Equation 6]

Where _k is the direction component of the motion parameter, φ _av is the first statistical value in the direction of the motion parameter, M is the number of blocks or objects constituting one screen. [Equation 14]

In the above formula, φ _{dev , i} is the standard deviation value of the direction of the motion parameter of the i-th screen, φ _{av , dev} are the average of the standard deviation values in the direction of the motion parameter, T is the number of screens from which motion parameters are extracted from the video. 12. The method of claim 10, wherein the method calculates the standard deviation values φ _{dev and dev} of standard deviation values in the direction of the motion parameter of Equation 14 below. _{av , dev} ). The method of claim 1, further comprising the step of using reliability. [Equation 14]

In the above formula, φ _{dev , i} is the standard deviation value of the direction of the motion parameter of the i-th screen, φ _av , _dev are the mean values of the standard deviations in the direction of the motion parameters, T is the number of screens from which motion parameters are extracted from the video. [Equation 13]

In the above formula, φ _{dev , i} is the standard deviation value of the direction of the motion parameter of the i-th screen, T is the number of screens from which motion parameters are extracted from the video. The method according to claim 1, wherein the method obtains a frequency of the motion parameter direction quantized in M directions in the entire moving picture screen, and several motion parameter directions (φ _1, φ ₂ ,. _. φ _N) and the number N (M≥N) motion activity to a vector characterized in that it further comprises a step of extracting a motion activity descriptor consisting of the features described method. [Equation 15] φ _max = <N, φ _1, φ _{2, ...} φ _N > Means for extracting motion parameters from a video; Means for extracting statistical characteristics of the magnitudes of the motion parameters extracted in the previous step; Means for extracting a statistical characteristic of the direction of the motion parameter: and And a combiner for gathering the extracted statistical characteristics to define a motion activity descriptor, wherein the means for extracting the statistical characteristics of the size of the motion parameter extracts the spatial statistical feature values of the size of the motion parameter of each image. And means for extracting the statistical characteristic of the direction of the motion parameter is means for extracting spatial statistical feature values of the direction of the motion parameter of each image. The apparatus of claim 13, wherein the apparatus uses the motion parameter encoded by digital video encoding as the motion parameter. The method of claim 13, wherein the means for extracting the spatial statistical feature values of the magnitude of the motion parameters obtains the standard deviation of the magnitudes of the motion parameters for each image, and then averages the values to represent the entire video. And a means for use as a value. The method according to claim 13, wherein the means for extracting the statistical characteristics of the magnitude of the motion parameter obtains the standard deviation of the magnitude of the motion parameter for each image, and then the standard deviation value of these values represents the entirety of the video. And a means for use as a value. 15. The apparatus of claim 13, comprising means for using the standard deviation of the standard deviation values of the magnitude of the motion parameter as a confidence in the mean value of the standard deviation values of the magnitude of the motion parameter. The method of claim 13, wherein the means for extracting the spatial statistical feature values in the direction of the motion parameter obtains the standard deviation of the direction of the motion parameter for each image, and then averages the standard deviation values to represent the entire video. And a means for use as a feature value. The method of claim 13, wherein the means for extracting the statistical characteristics of the direction of the motion parameter obtains the standard deviation value of the direction of the motion parameter for each image, and then represents the standard deviation value of the standard deviation values representing the entire video. A motion activity feature description apparatus for a video, comprising means for use as a statistical feature value. 14. The apparatus of claim 13, comprising means for using the standard deviation of the standard deviation values of the direction of the motion parameter as the reliability of the mean value of the standard deviation of the motion parameter. 20. The apparatus of claim 19, wherein the apparatus further comprises means for using the frequency of motion parameters in the video in the overall average direction of the video. The method of claim 13, wherein the device obtains the frequency of the motion parameter direction quantized in M directions in the entire video screen, and several motion parameter directions φ1, φ2, ... And means for extracting a vector consisting of the number N (M≥N) into a motion activity descriptor. [Equation 15] φmax = <N, φ1, φ 2, ... φ N> __

Images