EP1026601A2 - Méthode et appareil pour retrouver des données multimedia utilisant des informations concernant la forme des objets - Google Patents

Méthode et appareil pour retrouver des données multimedia utilisant des informations concernant la forme des objets Download PDF

Info

Publication number
EP1026601A2
EP1026601A2 EP00101895A EP00101895A EP1026601A2 EP 1026601 A2 EP1026601 A2 EP 1026601A2 EP 00101895 A EP00101895 A EP 00101895A EP 00101895 A EP00101895 A EP 00101895A EP 1026601 A2 EP1026601 A2 EP 1026601A2
Authority
EP
European Patent Office
Prior art keywords
eigen
shape information
difference
set forth
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP00101895A
Other languages
German (de)
English (en)
Other versions
EP1026601A3 (fr
Inventor
Jong Deuk Kim
Nam Kyu Kim
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Pantech Co Ltd
Original Assignee
Hyundai Electronics Industries Co Ltd
Hyundai Curitel Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hyundai Electronics Industries Co Ltd, Hyundai Curitel Co Ltd filed Critical Hyundai Electronics Industries Co Ltd
Publication of EP1026601A2 publication Critical patent/EP1026601A2/fr
Publication of EP1026601A3 publication Critical patent/EP1026601A3/fr
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G06V10/507—Summing image-intensity values; Histogram projection analysis

Definitions

  • the present invention relates in general to a method and apparatus for retrieving multimedia data by extracting a feature of shape information of an image using eigen vectors of a shape information covariance matrix and calculating a similarity on the basis of the extracted feature, and more particularly to a method and apparatus for retrieving multimedia data, in which the multimedia data can rapidly and accurately be retrieved by using multilayer eigen vectors capable of expressing complex shape information in detail and having a consistency against even rotation, scaling and translation of an image.
  • Multimedia data is much larger in size than data composed of only characters, and it is a combination of various types of information such as images, sounds, characters, etc. As a result, it is next to impossible to retrieve desired multimedia data using the multimedia data itself. For this reason, in order to retrieve multimedia data from a multimedia database, respective multimedia data expressible features are previously extracted through a preprocessing procedure and then compared respectively with information in the multimedia database. For example, in the case of retrieving video with a mixture of images, voice and audio, respective features of the images, voice and audio are extracted and then calculated in similarity with information in a multimedia database to be retrieved. As a result, desired information can be retrieved in accordance with the similarity calculation.
  • key points in the multimedia data retrieval are the types of features of multimedia data to be considered, how to express the features and how to compare between the features.
  • a data model expressive of each feature is called a descriptor.
  • a still image or moving image retrieval method is now most studied in multimedia data retrieval techniques.
  • features of an image such as a color, texture, shape, etc. are extracted and then measured in similarity.
  • used as descriptors expressive of the color feature may be a color histogram, correlogram, etc. [see: J. Huang, S. R. Kumar, M. Mitra, W. J. Zhu, and R. Zabih, Image indexing using color correlation, Proc, 16th IEEE Conf. on computer Vision and Pattern Recognition, pp. 762-768, 1997].
  • a wavelet coefficient, DFT coefficient, etc. may be used as descriptors expressive of the texture feature.
  • various descriptors may be used to express one feature and have both merits and demerits. In this connection, the performance of a retriever may be greatly influenced by an employed descriptor.
  • a shape information retrieval method is one of useful methods for image retrieval.
  • shape information of an object signifies information indicating which pixel of an arbitrary image belongs to the object and which pixel of the arbitrary image belongs to a background.
  • it is necessary to define a descriptor capable of appropriately expressing shape information of an object and compare a similarity of the shape information on the basis of the defined descriptor.
  • Existing descriptors used for the shape information retrieval may generally be classified into two types, or geometric feature-based descriptors and moment feature-based descriptors.
  • the geometric feature-based descriptors may generally be a parameter, area, maximum radius, minimum radius, corner, roundness, etc., and the moment feature-based descriptors be a center of mass, orientation, bounding rectangle, best-fit ellipse, eigen vector, etc.
  • the above shape information descriptors should be consistent regardless of any variation of an image such as rotation, scaling, translation, etc. of an object.
  • One of existing shape information feature extraction methods is to use eigen vectors of a covariance matrix of shape information.
  • the eigen vectors of the covariance matrix are composed of two vectors capable of expressing a distribution of the shape information.
  • the two eigen vectors have their directions signifying two axes (i.e., major and minor axes) indicative of distribution directions of the shape information, respectively, and their magnitudes representing distribution degrees of the shape information, respectively.
  • the major axis represents a main distribution direction of the shape information
  • the minor axis represents a minimum distribution direction of the shape information.
  • eigen vectors of the covariance matrix can be calculated in the following manner:
  • the components C xx and C yy of the covariance matrix indicate x-axis and y-axis distribution degrees of the shape information, respectively, and the components C xy and C yx of the covariance matrix indicate a correlation between x and y coordinates.
  • the eigen vectors A 1 and A 2 and eigen values r 1 and r 2 of the covariance matrix C can be obtained by solving the above equation 4.
  • the eigen vectors A 1 and A 2 of the covariance matrix C represent the main and minimum distribution directions of the shape information, respectively
  • the eigen values r 1 and r 2 of the covariance matrix C represent the distribution degrees of the shape information in the main and minimum distribution directions, respectively.
  • the above-mentioned shape information feature extraction method using the eigen vectors of the covariance matrix is able to express an approximate distribution of the shape information with a small amount of data and thus has advantages in that it is small in calculation amount, simple in calculation algorithm and has a consistency against translation of the shape information.
  • the above-mentioned shape information feature extraction method is disadvantageous in that it has a limitation in accurate shape information expression because it should express the entire shape information using only two eigen vectors in a single layer. In other words, eigen vectors to be calculated with respect to different types of shape information may often have the same value, resulting in a grievous situation. Further, the eigen vectors have no consistency against scaling or rotation of the shape information. As a result, the eigen vectors are insufficient to define a descriptor for expression of the shape information, leading to a reduction in the accuracy of the associated multimedia data retrieval method.
  • the present invention has been made in view of the above problems, and it is an object of the present invention to provide a method and apparatus for retrieving multimedia data, in which the multimedia data can accurately and rapidly be retrieved by adopting a shape information feature extraction method and apparatus capable of expressing complex shape information in detail using multilayer eigen vectors of a covariance matrix for the definition of a descriptor of the shape information.
  • the above and other objects can be accomplished by a provision of a method for retrieving multimedia data using shape information, comprising the first step of receiving shape information of a query image and extracting a feature of the received shape information using a shape information descriptor based on eigen vectors of a multilayer covariance matrix; the second step of extracting a feature of each image data in the same manner as the above first step; the third step of creating a multimedia database on the basis of the features extracted at the above second step; the fourth step of comparing the feature of the query image with the features of the image data in the multimedia database to calculate similarities therebetween; and the fifth step of outputting the results calculated at the above fourth step.
  • an apparatus for retrieving multimedia data using shape information which is capable of embodying the above multimedia data retrieval method.
  • a multimedia data retrieval method based on the multimedia data retrieval apparatus of Fig. 2 first receives shape information of a query image and extracts a feature of the received shape information using a shape information descriptor based on eigen vectors of a multilayer covariance matrix. Then, the multimedia data retrieval method extracts a feature of each image data in the same manner as the above step and creates a multimedia database on the basis of the extracted features. Subsequently, the multimedia data retrieval method compares the feature of the query image with the features of the image data in the created multimedia database to calculate similarities therebetween and outputs the calculated results. Therefore, the multimedia retrieval method of the present invention is characterized by the step of extracting a feature of shape information using a shape information descriptor defined in a proper manner and the step of comparing a similarity between two images on the basis of the extracted feature.
  • the shape information feature extraction step is characterized in that a shape information descriptor is defined by applying eigen vectors of a covariance matrix (referred to hereinafter as CMEVs) to every layer.
  • CMEVs covariance matrix
  • the present method partitions the shape information into a plurality of regions according to layers and obtains CMEVs for each of the partitioned regions. Because the obtained eigen vectors express shape information of the associated partitioned regions, respectively, they are increased in number in proportion to the number of the partitioned regions so that they can express the shape information in more detail. Therefore, the present invention is able to express complex shape information in more detail by applying the CMEVs to every layer.
  • Fig. 3 is a flowchart illustrating the multilayer CMEV-based shape information feature extraction step in the multimedia data retrieval method in accordance with the present invention.
  • CMEVs are obtained from the entire region of shape information at step S1 and a feature of the entire region of the shape information is extracted on the basis of the obtained CMEVs at step S2.
  • the entire region of the shape information is partitioned into a plurality of regions on the basis of two axis of the obtained CMEVs at step S3. It is determined at step S4 whether the partitioned regions belong to the last layer.
  • the operation returns to the above step S1 to obtain CMEVs from the partitioned regions and extract features of the partitioned regions on the basis of the obtained CMEVs.
  • the last layer is determined according to the number of layers which is arbitrarily set to obtain information minutely to a degree desired by the user.
  • the present invention is mainly described with respect to a binary mask as binary shape information, it may be applied to any other object expressible element such as a contour representing the boundary between a background and an object.
  • Figs. 4a and 4b are views illustrating a region partition procedure for obtaining multilayer CMEVs in accordance with the present invention.
  • Fig. 4a shows the first region partition.
  • CMEVs are obtained from the entire region of input shape information and two axes (major and minor axes) indicative of a distribution of the shape information are obtained from the CMEVs.
  • the entire region of the shape information is partitioned into four regions R1, R2, R3 and R4 due to the intersection of the two axes.
  • each of the partitioned regions R1, R2, R3 and R4 is repartitioned as in the above manner.
  • the region R1 is partitioned into regions R5, R6, R7 and R8 and the remaining regions R2, R3 and R4 are similarly partitioned into regions R9-R20.
  • a smaller number of partitioned layers leads to a smaller number of separated regions. Namely, as partitioned layers are increased in number, partitioned regions are similarly increased in number, thereby giving a more detailed expression of the shape information.
  • the number of layers may arbitrarily be set to express the shape information minutely to a degree desired by the user.
  • a shape information descriptor must appropriately be defined for the extraction of features of the shape information from the partitioned regions.
  • the shape information descriptor may be defined by a ratio of magnitudes (eigen values) of two eigen vectors of a covariance matrix, an eigen vector angle, a center point or a compactness, as will hereinafter be mentioned in detail.
  • a feature of shape information may be expressed by a ratio of magnitudes (eigen values) of eigen vectors of a covariance matrix forming two axes.
  • the ratio of the magnitudes of the two eigen vectors may be a ratio of the magnitude of the eigen vector of the minor axis to the magnitude of the eigen vector of the major axis or vice versa.
  • the reason why the eigen values of the major and minor axes themselves are not used but the ratio thereof is used as a shape information descriptor in the present invention is that the descriptor must have a consistency against scaling of the shape information. In other words, provided that eigen values are obtained from shape information with the same shape and different sizes, they will have a difference therebetween corresponding to the scaling of the shape information. However, the obtained eigen value ratio can be consistent regardless of the scaling of the shape information because the shape information is scaled in the same ratio with respect to both the major and minor axes.
  • Fig. 5a shows an example where elliptical shape information is partitioned into two layers in accordance with the present invention
  • Fig. 5b shows an example where rectangular shape information is partitioned into two layers in accordance with the present invention.
  • Eigen vector magnitude ratios and angles obtained in the first layers are nearly the same in both cases. Accordingly, both types of shape information can be equally featured when they are expressed by only an eigen vector magnitude ratio in one layer.
  • four eigen vector magnitude ratios obtained in the second layer of Fig. 5a are analogous to those of Fig. 5b, but eigen vector angles are different in both cases. Namely, it can be seen from Fig. 5a that the directions of eigen vectors in the second layer are slanted toward a contour. In this manner, the use of an eigen vector angle other than an eigen vector magnitude ratio as a shape information descriptor makes it possible to identify the different types of shape information as shown in Figs. 5a and 5b and accurately extract features of those shape information.
  • the eigen vector angle may be an angle of the eigen vector indicative of the major axis or an angle of the eigen vector indicative of the minor axis.
  • the associated eigen vector angle may be defined by an absolute value of a difference between an eigen vector angle obtained in the previous layer and an eigen vector angle obtained in the current layer. This allows the shape information descriptor to have a consistency against rotation of the shape information.
  • a center point of each region other than the above-mentioned eigen vector magnitude ratio and angle may be used as a shape information descriptor for the shape information feature extraction.
  • a center point of each partitioned region is used as a shape information descriptor to make a distinction between the features of the shape information in Figs. 6a and 6b.
  • a center point in each of the partitioned regions is far apart from a center of mass in the first layer.
  • a center of mass in the second layer is near the center of mass in the first layer.
  • the features of the shape information in Figs. 6a and 6b can be differently expressed by the center point.
  • the center point can be defined by any one of the following four methods.
  • a compactness may be obtained with respect to each region of a multilayer to express a feature of shape information.
  • the compactness may be defined by a ratio of the number of pixels in an object region to the number of pixels in the entire region.
  • Figs. 7a to 7f are views illustrating a shape information feature extraction procedure based on the compactness in accordance with the present invention.
  • Figs. 7a and 7b show an example where the compactness is obtained in the first layer.
  • the compactness may be defined by a ratio of the number of pixels in an object region of each layer to the number of pixels in a reference region of the associated layer instead of a ratio of the number of pixels in the original object region to the number of pixels in the entire region.
  • the compactness may have different values according to reference region determination methods.
  • a reference region in the first layer is defined by a minimum rectangular region which is parallel with two axes of eigen vectors obtained in the first layer and fully contains an object region in the first layer.
  • the reference numeral T0 denotes the reference region in the first layer.
  • the reference numerals T1, T2, T3 and T4 denote reference regions in the second layer, respectively.
  • the reference region T0 is partitioned into the reference regions T1, T2, T3 and T4 by the two eigen vectors.
  • reference regions in the third layer are defined as indicated by the reference numerals T5-T20 in Fig. 7f.
  • the compactness is used as a shape information descriptor in the present invention is that it can express, in distinction from each other, features of different types of shape information which have like eigen vector magnitude ratios but different shapes as shown in Figs. 8a and 8b.
  • the compactness in Fig. 8a is higher than that in Fig. 8b, thereby making a distinction between two images.
  • the step of comparing the feature of the query image with the features of the image data in the multimedia database to calculate similarities therebetween includes the step of calculating a difference eigen_ratio_diff between eigen vector magnitude ratios, a difference angle_diff between eigen vector angles, a difference center_diff between center points and a difference compact_diff between compactnesses with respect to each region of the query image and image data.
  • a similarity comparison operation for the first layer is performed in a different manner from that for the subsequent layers at the similarity comparison step. That is, an eigen vector angle and a center point are used for the second layer and over, not for the first layer.
  • N represents the total number of regions, or the sum of the region numbers in respective layers
  • L represents the number of layers.
  • f i and “s i” represent features of ith regions of two input images to be compared, respectively
  • w 1-4 signify weights applied respectively to descriptor differences (eigen_ratio_diff, angle_diff, center_diff and compact_diff) at the similarity comparison step.
  • the difference eigen_ratio_diff between the eigen vector magnitude ratios, the difference angle_diff between the eigen vector angles, the difference center_diff between the center points and the difference compact_diff between the compactnesses are calculated and summed.
  • the weights may be set differently according to features of input shape information and the similarity may be measured on the basis of the resultant weights. For example, in the case where an eigen vector angle is used for the similarity comparison, the probability that the eigen vector angle will have a different value irrespective of shape information of a binary image becomes higher as an eigen vector magnitude ratio is approximated to 1. As a result, in this case, it is necessary to apply a weight to an eigen vector angle difference.
  • the eigen vector magnitude ratio difference eigen_ratio_diff is defined by an absolute value of a difference between an eigen vector magnitude ratio of first shape information and an eigen vector magnitude ratio of second shape information.
  • the eigen vector angle difference angle_diff is defined by an absolute value of a difference between an absolute value of a difference between eigen vector angles in the previous and current layers of the first shape information and an absolute value of a difference between eigen vector angles in the previous and current layers of the second shape information.
  • angle_diff(n)
  • the eigen vector angle may be changed sensitively to a variation of shape information.
  • a smaller weight is applied to the eigen vector angle difference angle_diff if the eigen vector magnitude ratio eigen_ratio is approximated to 1
  • a larger weight is applied to the eigen vector angle difference angle_diff if the eigen vector magnitude ratio eigen_ratio is approximated to 0.
  • an eigen vector magnitude ratio eigen_ratio(f n ) in the above equation 7 is used as a weight control factor.
  • an eigen vector magnitude ratio eigen_ratio(s n ) may be used as the weight control factor or a combination of the eigen vector magnitude ratio eigen_ratio(f n ) and eigen vector magnitude ratio eigen_ratio(s n ) may be used as the weight control factor.
  • the center point difference center_diff is defined by an absolute of a difference between two center points
  • the compactness difference compact_diff is defined by an absolute value of a difference between two compactnesses.
  • the multimedia data retrieval apparatus comprises a shape information feature extractor 11 for receiving shape information of a query image and extracting a feature of the received shape information using a shape information descriptor based on eigen vectors of a multilayer covariance matrix.
  • An image feature extractor 22 is adapted to extract a feature of each image data in the same manner as the shape information feature extractor 11.
  • a multimedia database creator 33 is adapted to create a multimedia database on the basis of the features of the image data extracted by the image feature extractor 22.
  • the multimedia data retrieval apparatus further comprises a similarity comparator 44 for comparing the feature of the query image extracted by the shape information feature extractor 11 with the features of the image data in the created multimedia database to calculate similarities therebetween, and a compared result output unit 55 for outputting the results calculated by the similarity comparator 44.
  • the shape information feature extractor 11 receives shape information of a query image and extracts a feature of the received shape information using a shape information descriptor based on eigen vectors of a multilayer covariance matrix.
  • the image feature extractor 22 extracts a feature of each image data in the same manner as the shape information feature extractor 11, and the multimedia database creator 33 creates the multimedia database on the basis of the features of the image data extracted by the image feature extractor 22.
  • the similarity comparator 44 compares the feature of the query image extracted by the shape information feature extractor 11 with the features of the image data in the created multimedia database to calculate similarities therebetween. Then, the compared result output unit 55 outputs the results calculated by the similarity comparator 44.
  • the shape information feature extractor 11 and image feature extractor 22 are the same in construction, which will hereinafter be mentioned in detail with reference to Fig. 9.
  • each of the shape information feature extractor 11 and image feature extractor 22 includes a first switch S w1 10 for making a distinction between the initial input image and an output image from region partition means 60 and transferring the initial input image to covariance matrix calculation means 20 if a given layer is the initial layer and the output image from the region partition means 60 to the covariance matrix calculation means 20 if the given layer is not the initial layer.
  • the covariance matrix calculation means 20 is adapted to calculate a covariance matrix for shape information of the image transferred from the first switch S w1 10.
  • Each of the shape information feature extractor 11 and image feature extractor 22 further includes eigen vector calculation means 30 for calculating eigen vectors of the covariance matrix calculated by the covariance matrix calculation means 20, and feature extraction means 40 for extracting a feature of the shape information of the image transferred from the first switch S w1 10 on the basis of the eigen vectors calculated by the eigen vector calculation means 30.
  • the region partition means 60 is adapted to partition the shape information of the image transferred from the first switch S w1 10 into four regions according to two axes of the eigen vectors calculated by the eigen vector calculation means 30 if the given layer is not the last layer.
  • Each of the shape information feature extractor 11 and image feature extractor 22 further includes a second switch S w2 50 for transferring the feature extracted by the feature extraction means 40 to an output terminal if the given layer is the last layer and to the region partition means 60 if the given layer is not the last layer.
  • Fig. 10 is a detailed block diagram of the feature extraction means 40 in the shape information feature extractor 11 or image feature extractor 22 of Fig. 9.
  • the feature extraction means 40 may include any one or a combination of an eigen vector magnitude ratio calculator (referred to hereinafter as eigen_ratio calculator) 41, an eigen vector angle calculator (referred to hereinafter as angle calculator) 42, a center point calculator (referred to hereinafter as center_point calculator) 43 and a compactness calculator (referred to hereinafter as compact calculator) 44.
  • the feature extraction means 40 may include all of the eigen_ratio calculator 41, angle calculator 42, center_point calculator 43 and compact calculator 44.
  • the feature extraction means 40 extracts features such as an eigen vector magnitude ratio, an eigen vector angle, a center point and a compactness from the shape information of the image transferred from the first switch S w1 10 on the basis of the eigen vectors calculated by the eigen vector calculation means 30.
  • the feature extraction means 40 further includes a feature combiner 400 for combining the extracted features into a desired format and outputting the combined result to the second switch S w2 .
  • the above calculators in the feature extraction means 40 may be provided as means for realizing the respective embodiments at the shape information feature extraction step in the multimedia data retrieval method of the present invention as stated previously.
  • the first switch S w1 10 transfers the received image to the covariance matrix calculation means 20, which then calculates a covariance matrix for shape information of the transferred image.
  • the eigen vector calculation means 30 calculates eigen vectors of the covariance matrix calculated by the covariance matrix calculation means 20, and the feature extraction means 40 extracts a feature of the shape information of the image transferred from the first switch S w1 10 on the basis of the eigen vectors calculated by the eigen vector calculation means 30.
  • the second switch S w2 50 determines whether a given layer is the last layer. If the given layer is not the last layer, the second switch S w2 50 transfers the feature extracted by the feature extraction means 40 to the region partition means 60.
  • the region partition means 60 partitions the shape information of the image transferred from the switch S w1 10 into four regions according to two axes of the eigen vectors calculated by the eigen vector calculation means 30 and transfers an image of each of the partitioned regions to the covariance matrix calculation means 20 through the first switch S w1 10.
  • the covariance matrix calculation means 20 calculates a covariance matrix for shape information of the transferred image
  • the eigen vector calculation means 30 calculates eigen vectors of the covariance matrix calculated by the covariance matrix calculation means 20
  • the feature extraction means 40 extracts a feature of the shape information of the transferred image on the basis of the eigen vectors calculated by the eigen vector calculation means 30. This operation is repeated until the given layer becomes the last layer.
  • the second switch S w2 50 transfers the feature extracted by the feature extraction means 40 to the output terminal.
  • Fig. 11 is a block diagram of the similarity comparator 44 in the multimedia data retrieval apparatus of Fig. 2.
  • the similarity comparator 44 includes a first feature separator 101 for separating the feature of the query image extracted by the shape information feature extractor 11 into an eigen vector magnitude ratio, eigen vector angle, center point and compactness, and a second feature separator 102 for separating each of the features of the image data in the created multimedia database into an eigen vector magnitude ratio, eigen vector angle, center point and compactness.
  • the similarity comparator 44 further includes a calculator (referred to hereinafter as eigen_ratio_diff calculator) 201 for calculating a difference between the eigen vector magnitude ratios from the first and second feature separators 101 and 102, a calculator (referred to hereinafter as angle_diff calculator) 202 for calculating a difference between the eigen vector angles from the first and second feature separators 101 and 102, a calculator (referred to hereinafter as center_diff calculator) 203 for calculating a difference between the center points from the first and second feature separators 101 and 102, and a calculator (referred to hereinafter as compact_diff calculator) 204 for calculating a difference between the compactnesses from the first and second feature separators 101 and 102.
  • eigen_ratio_diff calculator a calculator for calculating a difference between the eigen vector magnitude ratios from the first and second feature separators 101 and 102
  • angle_diff calculator 202 for calculating a difference between the e
  • the similarity comparator 44 further includes first to fourth weight calculators 300-303 for calculating weights to the eigen vector magnitude ratio difference from the eigen_ratio_diff calculator 201, the eigen vector angle difference from the angle_diff calculator 202, the center point difference from the center_diff calculator 203 and the compactness difference from the compact_diff calculator 204, respectively.
  • the similarity comparator 44 further includes an adder 500 for adding the weighted eigen vector magnitude ratio difference, eigen vector angle difference, center point difference and compactness difference from the first to fourth weight calculators 300-303, and a similarity calculator 600 for calculating the similarity between the feature of the query image extracted by the shape information feature extractor 11 and each of the features of the image data in the created multimedia database on the basis of the result added by the adder 500.
  • an adder 500 for adding the weighted eigen vector magnitude ratio difference, eigen vector angle difference, center point difference and compactness difference from the first to fourth weight calculators 300-303
  • a similarity calculator 600 for calculating the similarity between the feature of the query image extracted by the shape information feature extractor 11 and each of the features of the image data in the created multimedia database on the basis of the result added by the adder 500.
  • the similarity comparator 44 may include any one or a combination of the eigen_ratio_diff calculator 201, angle_diff calculator 202, center_diff calculator 203 and compact_diff calculator 204 according to the presence of the eigen_ratio calculator 41, angle calculator 42, center_point calculator 43 and compact calculator 44 in each of the shape information feature extractor 11 and image feature extractor 22.
  • the similarity comparator 44 may include all of the eigen_ratio_diff calculator 201, angle_diff calculator 202, center_diff calculator 203 and compact_diff calculator 204 to express the similarity very precisely.
  • the first feature separator 101 separates the received feature into an eigen vector magnitude ratio, eigen vector angle, center point and compactness.
  • the second feature separator 102 receives each of the features of the image data in the created multimedia database and separates the received feature into an eigen vector magnitude ratio, eigen vector angle, center point and compactness.
  • the eigen_ratio_diff calculator 201 calculates a difference between the eigen vector magnitude ratios from the first and second feature separators 101 and 102
  • the angle_diff calculator 202 calculates a difference between the eigen vector angles from the first and second feature separators 101 and 102.
  • the center_diff calculator 203 calculates a difference between the center points from the first and second feature separators 101 and 102
  • the compact_diff calculator 204 calculates a difference between the compactnesses from the first and second feature separators 101 and 102.
  • the first to fourth weight calculators 300-303 apply proper weights to the eigen vector magnitude ratio difference from the eigen_ratio_diff calculator 201, the eigen vector angle difference from the angle_diff calculator 202, the center point difference from the center_diff calculator 203 and the compactness difference from the compact_diff calculator 204, respectively.
  • the adder 500 adds the weighted eigen vector magnitude ratio difference, eigen vector angle difference, center point difference and compactness difference from the first to fourth weight calculators 300-303, and the similarity calculator 600 calculates the similarity between the feature of the query image extracted by the shape information feature extractor 11 and each of the features of the image data in the created multimedia database on the basis of the result added by the adder 500. Then, the similarity calculator 600 provides the calculated result to the compared result output unit 55.
  • the present invention ensures a smaller amount of calculation as compared with conventional methods employing moments. Further, the present invention can regulate an expression degree of shape information by adjusting the number of layers and ensure a consistency against rotation, scaling and translation of the shape information.
  • the present invention has the effect of accurately expressing shape information with a small amount of calculation and a small amount of data to accurately and rapidly retrieve desired multimedia data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
EP00101895A 1999-02-01 2000-01-31 Méthode et appareil pour retrouver des données multimedia utilisant des informations concernant la forme des objets Withdrawn EP1026601A3 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1019990003189A KR100671098B1 (ko) 1999-02-01 1999-02-01 모양정보를 이용한 멀티미디어 데이터의 검색 방법 및 장치
KR9903189 1999-02-01

Publications (2)

Publication Number Publication Date
EP1026601A2 true EP1026601A2 (fr) 2000-08-09
EP1026601A3 EP1026601A3 (fr) 2001-03-14

Family

ID=19572928

Family Applications (1)

Application Number Title Priority Date Filing Date
EP00101895A Withdrawn EP1026601A3 (fr) 1999-02-01 2000-01-31 Méthode et appareil pour retrouver des données multimedia utilisant des informations concernant la forme des objets

Country Status (4)

Country Link
US (1) US6807303B1 (fr)
EP (1) EP1026601A3 (fr)
JP (1) JP2000222581A (fr)
KR (1) KR100671098B1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2844079A1 (fr) * 2002-08-30 2004-03-05 France Telecom Systeme associatif flou de description d'objets multimedia
EP1550976A2 (fr) 2003-12-02 2005-07-06 Sony Corporation Dipositif de traitement d'informations, procédé de traitement d'informations, programme pour mise en oeuvre du procédé de traitement d'informations, système de traitement d'informations, et procédé pour sytème de traitement d'informations
CN1909677B (zh) * 2004-09-23 2011-07-06 三菱电机株式会社 图像表示和分析方法
CN103886066A (zh) * 2014-03-20 2014-06-25 杭州禧颂科技有限公司 一种基于鲁棒非负矩阵分解的图像检索方法
CN111897864A (zh) * 2020-08-13 2020-11-06 创智和宇信息技术股份有限公司 一种基于互联网ai外呼的专家库数据抽取方法及系统
CN117085969A (zh) * 2023-10-11 2023-11-21 中国移动紫金(江苏)创新研究院有限公司 人工智能工业视觉检测方法、装置、设备及存储介质

Families Citing this family (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2352076B (en) * 1999-07-15 2003-12-17 Mitsubishi Electric Inf Tech Method and apparatus for representing and searching for an object in an image
KR100653026B1 (ko) * 1999-11-30 2006-11-30 주식회사 팬택앤큐리텔 임의의 모양의 텍스쳐 검색 방법 및 장치
KR100413679B1 (ko) * 2000-10-21 2003-12-31 삼성전자주식회사 형상 기술자 추출방법
US7253919B2 (en) 2000-11-30 2007-08-07 Ricoh Co., Ltd. Printer with embedded retrieval and publishing interface
JP4736201B2 (ja) * 2001-02-19 2011-07-27 ソニー株式会社 情報検索装置及び方法、並びに記憶媒体
KR100810002B1 (ko) * 2001-04-11 2008-03-07 김회율 모양 기술자 계산을 위한 정규화 방법 및 그를 이용한영상 검색 방법
US7314994B2 (en) 2001-11-19 2008-01-01 Ricoh Company, Ltd. Music processing printer
US7415670B2 (en) 2001-11-19 2008-08-19 Ricoh Co., Ltd. Printer with audio/video localization
US7861169B2 (en) 2001-11-19 2010-12-28 Ricoh Co. Ltd. Multimedia print driver dialog interfaces
US7747655B2 (en) * 2001-11-19 2010-06-29 Ricoh Co. Ltd. Printable representations for time-based media
KR20030067135A (ko) * 2002-02-07 2003-08-14 (주)지토 내용기반 동영상 자동분할 기술을 응용한 인터넷 방송기술
CN1445696A (zh) * 2002-03-18 2003-10-01 朗迅科技公司 自动检索图像数据库中相似图象的方法
US7739583B2 (en) 2003-03-31 2010-06-15 Ricoh Company, Ltd. Multimedia document sharing method and apparatus
US7703002B2 (en) * 2003-03-31 2010-04-20 Ricoh Company, Ltd. Method and apparatus for composing multimedia documents
US7757162B2 (en) 2003-03-31 2010-07-13 Ricoh Co. Ltd. Document collection manipulation
US7509569B2 (en) * 2003-03-31 2009-03-24 Ricoh Co., Ltd. Action stickers for nested collections
US20050036691A1 (en) * 2003-08-13 2005-02-17 Pascal Cathier Method and system for using structure tensors to detect lung nodules and colon polyps
US7864352B2 (en) 2003-09-25 2011-01-04 Ricoh Co. Ltd. Printer with multimedia server
US8077341B2 (en) 2003-09-25 2011-12-13 Ricoh Co., Ltd. Printer with audio or video receiver, recorder, and real-time content-based processing logic
JP2005108230A (ja) 2003-09-25 2005-04-21 Ricoh Co Ltd オーディオ／ビデオコンテンツ認識・処理機能内蔵印刷システム
JP2005277981A (ja) * 2004-03-26 2005-10-06 Seiko Epson Corp 画像処理のための対象画像の選択
US8274666B2 (en) 2004-03-30 2012-09-25 Ricoh Co., Ltd. Projector/printer for displaying or printing of documents
JP4738857B2 (ja) * 2005-03-23 2011-08-03 キヤノン株式会社 画像処理装置およびその方法
KR100623510B1 (ko) * 2005-03-28 2006-09-19 중앙대학교 산학협력단 지형/지물 이미지에 대한 특징추출방법
US8478074B2 (en) 2006-07-07 2013-07-02 Microsoft Corporation Providing multiple and native representations of an image
US20090021533A1 (en) * 2007-07-17 2009-01-22 Michael Guerzhoy Method For Extracting An Inexact Rectangular Region Into An Axis-Aligned Rectangle Image
KR100917755B1 (ko) * 2007-12-10 2009-09-15 한국전자통신연구원 가상 객체의 물리적 특성 변경 장치 및 방법
DE112008000017T5 (de) 2008-05-09 2009-11-05 Hankuk University of Foreign Studies Research and Industry-University Cooperation Foundation, Yongin Abbilden von Bildern mit Gestaltbeschreibern
KR100957191B1 (ko) * 2008-07-07 2010-05-11 연세대학교 산학협력단 숨은그림찾기 이미지 제작 방법 및 시스템
JP5528121B2 (ja) * 2010-01-05 2014-06-25 キヤノン株式会社 画像処理装置、画像処理方法、及びプログラム
KR101675785B1 (ko) 2010-11-15 2016-11-14 삼성전자주식회사 특징점을 이용한 영상 검색 방법 및 상기 방법을 수행하는 장치
US9256709B2 (en) 2014-02-13 2016-02-09 Taiwan Semiconductor Manufacturing Company, Ltd. Method for integrated circuit mask patterning

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5164992A (en) * 1990-11-01 1992-11-17 Massachusetts Institute Of Technology Face recognition system
US5612928A (en) * 1992-05-28 1997-03-18 Northrop Grumman Corporation Method and apparatus for classifying objects in sonar images
JP3026712B2 (ja) * 1993-12-09 2000-03-27 キヤノン株式会社 画像検索方法及びその装置
US5710833A (en) * 1995-04-20 1998-01-20 Massachusetts Institute Of Technology Detection, recognition and coding of complex objects using probabilistic eigenspace analysis
WO1997023844A1 (fr) * 1995-12-21 1997-07-03 Philips Electronics N.V. Attenuation directionnelle et adaptative de bruit
US6038337A (en) * 1996-03-29 2000-03-14 Nec Research Institute, Inc. Method and apparatus for object recognition
US5901244A (en) * 1996-06-18 1999-05-04 Matsushita Electric Industrial Co., Ltd. Feature extraction system and face image recognition system
US5852823A (en) 1996-10-16 1998-12-22 Microsoft Image classification and retrieval system using a query-by-example paradigm
US6134541A (en) * 1997-10-31 2000-10-17 International Business Machines Corporation Searching multidimensional indexes using associated clustering and dimension reduction information
US6332037B1 (en) * 1999-04-09 2001-12-18 Board Of Regents Of The University Of Nebraska Invariant, Eigenvalue based, non-degenerate data structure characterization, storage and retrieval indexing method

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "Method of Preparing a Set of Graphical Shapes for Quick Searching Based on Physical Proximity" IBM TECHNICAL DISCLOSURE BULLETIN, vol. 30, no. 5, 1 October 1987 (1987-10-01), pages 59-61, XP000046140 New York, US *
FALOUTSOS C ET AL: "Efficient and effective querying by image content" JOURNAL OF INTELLIGENT INFORMATION SYSTEMS: INTEGRATING ARTIFICIAL INTELLIGENCE AND DATABASE TECHNOLOGIES, JULY 1994, NETHERLANDS, vol. 3, no. 3-4, pages 231-262, XP000564354 ISSN: 0925-9902 *
MEHTRE B M ET AL: "Shape measures for content based image retrieval: a comparison" INFORMATION PROCESSING & MANAGEMENT, MAY 1997, ELSEVIER, UK, vol. 33, no. 3, pages 319-337, XP004091804 ISSN: 0306-4573 *
TAUBIN G ET AL: "Representing and comparing shapes using shape polynomials" PROCEEDINGS CVPR '89 IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CAT. NO.89CH2752-4), SAN DIEGO, CA, USA, 4-8 JUNE 1989, pages 510-516, XP002156989 1989, Washington, DC, USA, IEEE Comput. Soc. Press, USA ISBN: 0-8186-1952-x *

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7460715B2 (en) 2002-08-30 2008-12-02 France Telecom Fuzzy associative system for multimedia object description
WO2004021265A2 (fr) * 2002-08-30 2004-03-11 France Telecom SystEme associatif flou de description d'objets multimEdia
WO2004021265A3 (fr) * 2002-08-30 2004-04-08 France Telecom SystEme associatif flou de description d'objets multimEdia
FR2844079A1 (fr) * 2002-08-30 2004-03-05 France Telecom Systeme associatif flou de description d'objets multimedia
US7487151B2 (en) 2003-12-02 2009-02-03 Sony Corporation Information processing apparatus, information processing method, program for implementing information processing method, information processing system, and method for information processing system
EP1550976A3 (fr) * 2003-12-02 2006-12-27 Sony Corporation Dipositif de traitement d'informations, procédé de traitement d'informations, programme pour mise en oeuvre du procédé de traitement d'informations, système de traitement d'informations, et procédé pour sytème de traitement d'informations
EP1550976A2 (fr) 2003-12-02 2005-07-06 Sony Corporation Dipositif de traitement d'informations, procédé de traitement d'informations, programme pour mise en oeuvre du procédé de traitement d'informations, système de traitement d'informations, et procédé pour sytème de traitement d'informations
CN1909677B (zh) * 2004-09-23 2011-07-06 三菱电机株式会社 图像表示和分析方法
CN103886066A (zh) * 2014-03-20 2014-06-25 杭州禧颂科技有限公司 一种基于鲁棒非负矩阵分解的图像检索方法
CN103886066B (zh) * 2014-03-20 2017-03-29 杭州禧颂科技有限公司 一种基于鲁棒非负矩阵分解的图像检索方法
CN111897864A (zh) * 2020-08-13 2020-11-06 创智和宇信息技术股份有限公司 一种基于互联网ai外呼的专家库数据抽取方法及系统
CN117085969A (zh) * 2023-10-11 2023-11-21 中国移动紫金(江苏)创新研究院有限公司 人工智能工业视觉检测方法、装置、设备及存储介质
CN117085969B (zh) * 2023-10-11 2024-02-13 中国移动紫金(江苏)创新研究院有限公司 人工智能工业视觉检测方法、装置、设备及存储介质

Also Published As

Publication number Publication date
EP1026601A3 (fr) 2001-03-14
KR20000054864A (ko) 2000-09-05
KR100671098B1 (ko) 2007-01-17
JP2000222581A (ja) 2000-08-11
US6807303B1 (en) 2004-10-19

Similar Documents

Publication Publication Date Title
US6807303B1 (en) Method and apparatus for retrieving multimedia data using shape information
Wu et al. Deep matching and validation network: An end-to-end solution to constrained image splicing localization and detection
JP5121972B2 (ja) カラー画像を表現する方法、カラー画像を表現する装置、カラー画像を表現するシステム、コンピュータによる実行可能命令からなるプログラム、及びコンピュータ読取り可能媒体
EP1107136B1 (fr) Système et procédé pour retrouver des images en fonction de leur contenu
KR100737176B1 (ko) 신호 처리 방법 및 영상 음성 처리 장치
KR100413679B1 (ko) 형상 기술자 추출방법
US7620250B2 (en) Shape matching method for indexing and retrieving multimedia data
Song et al. Analyzing scenery images by monotonic tree
CN111709433A (zh) 一种多特征融合图像识别算法
Sahbi et al. Robust face recognition using dynamic space warping
Pavithra et al. An efficient seed points selection approach in dominant color descriptors (DCD)
JP2006060796A (ja) 映像検索装置，方法及びプログラム並びにプログラムを記録した記録媒体
KR100876280B1 (ko) 통계적 모양기술자 추출 장치 및 그 방법과 이를 이용한 동영상 색인 시스템
Ksibi et al. Deep salient-Gaussian Fisher vector encoding of the spatio-temporal trajectory structures for person re-identification
Daras et al. 3D model search and retrieval based on the spherical trace transform
Warbhe et al. Digital image forensics: An affine transform robust copy-paste tampering detection
Cheikh et al. Shape recognition based on wavelet-transform modulus maxima
Ni et al. Research on image segmentation algorithm based on fuzzy clustering and spatial pyramid
Paul et al. Issues in database management of multimedia information
KR100712341B1 (ko) 변형된 저니크 모멘트에 의한 3차원 영상 데이터의 특징추출 및 검색 방법 및 장치
JPH0991429A (ja) 顔領域抽出方法
Zhao et al. Visual keyword image retrieval based on synergetic neural network for web-based image search
Kim et al. A novel image retrieval scheme using DCT filter-bank of weighted color components
Memon et al. Localization in images matching through region-based similarity technique for content-based image retrieval
Aguilera et al. An information-theoretic approach to georegistration of digital elevation maps

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE

AX Request for extension of the european patent

Free format text: AL;LT;LV;MK;RO;SI

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

RIC1 Information provided on ipc code assigned before grant

Free format text: 7G 06F 17/30 A, 7G 06K 9/52 B

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE

AX Request for extension of the european patent

Free format text: AL;LT;LV;MK;RO;SI

17P Request for examination filed

Effective date: 20010717

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: HYNIX SEMICONDUCTOR INC.

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: HYUNDAI CURITEL, INC.

AKX Designation fees paid

Free format text: AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE

AXX Extension fees paid

Free format text: AL PAYMENT 20010717;LT PAYMENT 20010717;LV PAYMENT 20010717;MK PAYMENT 20010717;RO PAYMENT 20010717;SI PAYMENT 20010717

17Q First examination report despatched

Effective date: 20040601

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20041214