CA2290445A1 - Method and system for image retrieval - Google Patents

Method and system for image retrieval Download PDF

Info

Publication number
CA2290445A1
CA2290445A1 CA002290445A CA2290445A CA2290445A1 CA 2290445 A1 CA2290445 A1 CA 2290445A1 CA 002290445 A CA002290445 A CA 002290445A CA 2290445 A CA2290445 A CA 2290445A CA 2290445 A1 CA2290445 A1 CA 2290445A1
Authority
CA
Canada
Prior art keywords
similarity
database
image
regions
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
CA002290445A
Other languages
French (fr)
Inventor
John R. Smith
Shih-Fu Chang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Columbia University of New York
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of CA2290445A1 publication Critical patent/CA2290445A1/en
Abandoned legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5838Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G06F16/5854Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship

Abstract

In a system in which database images are represented by regions having specified feature attributes and spatial location attributes, image queries can be directed to region feature similarly and region spatial location similarity in combination. If desired, the relative spatial arrangement of regions can also be taken into account.

Description

METHOD AND SYSTEM FOR IMAGE RETRIEVAL
Technical Field This invention relates to computerized image retrieval and, more specifically, to retrieval based on image database querying.
Backaround of the Invention with advances in computer hardware technology, it has become possible to store, manipulate and transmit large numbers of images. When represented in computer-tractable form, the images can be included in an image database.
Systems have been developed, typically in the form of computer software, for image database management and image retrieval. For example, as disclosed in U.S. Patent 5,493,677, issued February 20, 1996 to Balogh et al., images can be retrieved by searching text associated with the images for a match with a query.
Systems have also been developed which use image content descriptors, for querying by image content.
Such a system is disclosed in U.S. Patent 5,579,471, issued November 26, 1996 to Barber et al. and in the paper by W. Niblack et al., "The QBIC Project: Querying Images by Content Using Color, Texture, and Shape", in Storage and Retrieval for Image and Video Databases, Wayne Niblack, Editor, Proc. SPIE 1908, pp. 173-187 ( 1993 ) .
Summary of the Invention We have recognized that, for greater accuracy in retrieving images from an image database, querying based on image content can be combined with querying based on spatial location. Thus, in a system in which each image is represented by a plurality of regions having feature attributes and spatial location attributes, queries can be directed to region feature similarity and region spatial location similarity in combination. If desired, the relative spatial arrangement of regions can also be taken into account.
Brief Description of the Drawincc Fig. 1 is an example of an image with regions for inclusion in a database.
Fig. 2 is a tabular display of a representation of the regions of Fig. 1.
Fig. 3 is an example of an image with regions for database querying.
Fig. 4 is a tabular display of a representation of the regions of Fig . 3 .
Fig. 5 is a flow diagram of database query processing for discriminating based on region feature and region absolute spatial location.
Fig. 6 is a flow diagram of database query processing for discriminating based on the relative location of regions.
Detailed Description of Preferred Embodiments The following description is primarily in terms of method steps for execution by a suitable processor under program control. The program may originate as software, or, for greater efficiency, it may be embodied at least in part in dedicated firmware or hardware.
A prototype system embodying features as described has been formulated in the JAVA language. The system can operate on suitable hardware such as a SUN
Workstation, a Silicon Graphics Workstation, or a PC with a Pentium processor, for example.
Conveniently, an image database to be queried has tabular form, with each record or table entry representing a region of an image. A record includes an image identifier, a region identifier, a region attribute and, for geometric characterization, the x- and y-coordinates of the centroid of the region, the width and height of the region, and the area of the region. The table may be generated by manual keyboard entry based on visual inspection of images. Alternatively, if a suitable pattern recognition system is available, table generation may be automated.
To illustrate database entries, the image 10 shown in Fig. 1 and having been given the identifier "T"
can be represented by the table entries shown in Fig. 2.
Included are:
region 100 (to, stretching across the bottom of the image, below a broken line);
region 101 (tl, bounded by a rectangle drawn with broken lines);
region 102 (t2, bounded by a rectangle drawn with chain-dotted lines), region 103 (t" bounded by a rectangle drawn with broken lines); and region 104 (t4, stretching across the top of the image, above a broken line).
The x,y-coordinates, the width w, and the height h of each region are given in percent of the respective maximal values. The values x, y, w and h define a "bounding rectangle" for each region, so that the area of a region is less than or equal to w times h.
As illustrated, regions may overlap, and their union need not cover the image.
The attribute f may simply represent color, for example, with color being represented by known means, e.g., by a color histogram or by color sets. Other simple attributes which may be used include texture and shape, and such attributes may be combined into more complex attributes.
A search query is expressed correspondingly.
For example, for the search pattern shown in Fig. 3, a query region table may be formed as shown in Fig. 4.
For a database and a query, e.g. with entries as illustrated by Figs. 2 and 4, respectively, Fig. 5 illustrates query processing for finding database entries based on the query. The general aim is to find images that contain arrangements of regions similar to those in the query.
According to Fig. 5, starting at "a", for each region in the query, the database regions are searched for a feature match (step 51) and a spatial match (step 52). For spatial matching, this involves using a suitable metric for comparing the spatial information such as x, y, h, w and area of the query region with the corresponding information for the database regions.
Suitable metrics include Euclidean distance and other Minkowski distances, and quadratic metrics whose definition involves a square matrix which expresses the relative similarity between the components of a vector.
A metric can also include weights which may be different for each of the geometric parameters.
Similar metrics can be used for feature matching (f;). For example, if color histogram information is included in terms of components "red", "green" and "blue", a 3-component Euclidean metric can be used. Analogously, this applies when such information is included in terms of components "hue", "saturation" and "intensity".
For efficiency, as shown in Fig. 5, thresholds are applied to the computed feature and spatial distances. Thus, if a distance exceeds the threshold, the database region is not included for further consideration. Instead of, or in addition to using separate thresholds for spatial and feature similarity as shown in Fig. 5, thresholding can be applied also to the combined region distance or score, i.e. before saving a region match in step 53. Distances may be combined by simple addition, or by suitable weighting followed by addition, for example.
If multiple processors are available, "k-loop"
feature similarity processing analogous to step 51 and spatial similarity processing analogous to step 52 may be carried out in parallel instead of pair-wise sequentially as illustrated in Fig. 5. Parallel processing then yields two sets of regions, namely (i) those which meet feature similarity regardless of spatial similarity, and (ii) those which meet spatial similarity regardless of feature similarity. Thus, to obtain the desired set of regions which meet both types of features, a "jain"
operation will be required. After joining, a final thresholding operation can be performed. Advantageously, multiple processors may also be used for parallel processing within steps 52 and 53.
Image matches are obtained as a result of the "join" operation in step 54, producing all those database images which meet each one of the region requirements of the query. A query may result in an image being saved in step 54 more than once, namely for different combinations of its regions which satisfy the query. Such multiplicity may be helpful to a user of the system;
otherwise, duplicates can be deleted by a simple one-pass search of the saved images.
If the relative spatial location or arrangement of regions is not important to a user, the computation may terminate at this point ((3), though preferably after the saved images are sorted by score.
For discriminating further based on relative spatial location of regions, a process can be used as illustrated by Fig. 6, using so-called 2-D strings.
Generation of 2-D strings at this point, i.e. after similarity processing, may be termed "query-time 2-D
string generation".
For a query image, a 2-D string includes the x-coordinates of the centroids of the regions, arranged as an increasing sequence, followed by the y-coordinates of the centroids, also arranged as an increasing sequence.
For a database image, correspondingly, the coordinates of those regions are used which were matched against the query image regions.
The 2-D string of the query image is formed in step 61, and, in step 62, this string is matched against the 2-D strings from each of the saved images. In step 63, only in case of a match, the database image is saved, so that only those images are ultimately sorted and produced in step 64 which have a 2-D string which matches the 2-D string of the query image.
Instead of or in addition to 2-D strings including x- and y-coordinates of centroids as described, 2-D strings can be produced after rotation of the coordinate system, e.g. by 45°. Such 2-D strings are defined analogously, using coordinates x' and y' of the centroids in the rotated coordinate system.

Claims (28)

CLAIMS:
1. A method for retrieving image representations from an image database, comprising:
for each of a plurality of regions of a query image, searching the database for image regions which match the query region with respect to at least one feature similarity and at least one spatial similarity;
and joining the matched image regions.
2. The method according to claim 1, wherein, in searching the database, feature similarity and spatial similarity of the database image regions are ascertained pair-wise sequentially.
3. The method according to claim 1, wherein, in searching the database, feature similarity and spatial similarity are ascertained using separate processes in parallel.
4. The method according to claim 1, further comprising saving those database images which include the joined image regions.
5. The method according to claim 4, further comprising sorting the saved images.
6. The method according to claim 4, further comprising discriminating among the saved images based on relative location of the joined regions.
7. The method according to claim 6, wherein discriminating among the saved images comprises comparing 2-D strings.
8. The method according to claim 7, wherein the 2-D strings comprise x- and y-coordinates of region centroids.
9. The method according to claim 7, wherein the 2-D strings comprise x'- and y'-coordinates of region centroids, in a rotated coordinate system.
10. The method according to claim 6, further comprising sorting the discriminated images.
11. The method according to claim 4, further comprising deleting duplicates from the saved images.
12. The method according to claim 1, wherein the feature similarity comprises color similarity.
13. The method according to claim 1, wherein the feature similarity comprises texture similarity.
14. The method according to claim 1, wherein the feature similarity comprises shape similarity.
15. A system for retrieving image representations from an image database, comprising:
means for searching the database, for each of a plurality of regions of a query image, for image regions which match the query region with respect to at least one feature similarity and at least one spatial similarity;
and means for joining the matched image regions.
16. The system according to claim 15, wherein the means for searching the database comprises means for ascertaining feature similarity and spatial similarity of the database image regions pair-wise sequentially.
17. The system according to claim 16, wherein the means for searching the database comprises means for ascertaining feature similarity and spatial similarity in parallel.
18. The system according to claim 15, further comprising means for saving those database images which include the joined image regions.
19. The system according to claim 18, further comprising means for sorting the saved images.
20. The system according to claim 18, further comprising means for discriminating among the saved images based on relative location of the joined regions.
21. The system according to claim 20, wherein the means for discriminating among the saved images comprises means for comparing 2-D strings.
22. The system according to claim 21, wherein the 2-D strings comprise x- and y-coordinates of region centroids.
23. The system according to claim 21, wherein the 2-D strings comprise x'- and y'-coordinates of region centroids, in a rotated coordinate system.
24. The system according to claim 20, further comprising means for sorting the discriminated images.
25. The system according to claim 18, further comprising means for deleting duplicates from the saved images.
26. The system according to claim 15, wherein the feature similarity comprises color similarity.
27. The system according to claim 15, wherein the feature similarity comprises texture similarity.
28. The system according to claim 15, wherein the feature similarity comprises shape similarity.
CA002290445A 1997-05-16 1997-05-16 Method and system for image retrieval Abandoned CA2290445A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US1997/009256 WO1998052119A1 (en) 1997-05-16 1997-05-16 Method and system for image retrieval

Publications (1)

Publication Number Publication Date
CA2290445A1 true CA2290445A1 (en) 1998-11-19

Family

ID=22260985

Family Applications (1)

Application Number Title Priority Date Filing Date
CA002290445A Abandoned CA2290445A1 (en) 1997-05-16 1997-05-16 Method and system for image retrieval

Country Status (3)

Country Link
JP (1) JP2001525959A (en)
CA (1) CA2290445A1 (en)
WO (1) WO1998052119A1 (en)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6285995B1 (en) * 1998-06-22 2001-09-04 U.S. Philips Corporation Image retrieval system using a query image
US6253201B1 (en) * 1998-06-23 2001-06-26 Philips Electronics North America Corporation Scalable solution for image retrieval
CN1193623C (en) * 1999-05-17 2005-03-16 三星电子株式会社 Color image processing method
US7212667B1 (en) 1999-05-17 2007-05-01 Samsung Electronics Co., Ltd. Color image processing method for indexing an image using a lattice structure
US6782395B2 (en) 1999-12-03 2004-08-24 Canon Kabushiki Kaisha Method and devices for indexing and seeking digital images taking into account the definition of regions of interest
FR2801992B1 (en) * 1999-12-03 2002-05-03 Canon Kk CONTENT-BASED IMAGE SEARCHING METHOD AND APPARATUS TAKING INTO ACCOUNT THE DEFINITION OF REGIONS OF INTEREST
EP1184796A1 (en) * 2000-08-29 2002-03-06 Sudimage Method of associative navigation in a multimedia database
FR2815741B1 (en) * 2000-10-24 2004-09-10 Canon Kk METHOD AND DEVICE FOR INDEXING AND SEARCHING IMAGES BASED ON THE CONTENT OF REGIONS OF INTEREST
US6801661B1 (en) 2001-02-15 2004-10-05 Eastman Kodak Company Method and system for archival and retrieval of images based on the shape properties of identified segments
US20030039410A1 (en) * 2001-08-23 2003-02-27 Beeman Edward S. System and method for facilitating image retrieval
GB2417801A (en) * 2004-09-07 2006-03-08 Pepperdog Ltd Image processing apparatus
JP4641414B2 (en) * 2004-12-07 2011-03-02 キヤノン株式会社 Document image search apparatus, document image search method, program, and storage medium
AT514355B1 (en) * 2013-05-17 2017-01-15 Ait Austrian Institute Of Technology Gmbh Used to select digital images from an image database
CN117370590A (en) * 2015-12-02 2024-01-09 杭州海康威视数字技术股份有限公司 Image retrieval method, device and system

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE69131251T2 (en) * 1991-08-15 1999-12-09 Ibm System and method for processing data representing stored images
US5546572A (en) * 1991-08-28 1996-08-13 Hitachi, Ltd. Method for retrieving database of image information
JPH0756652B2 (en) * 1992-03-24 1995-06-14 インターナショナル・ビジネス・マシーンズ・コーポレイション Search for video frame sequence
US5615112A (en) * 1993-01-29 1997-03-25 Arizona Board Of Regents Synthesized object-oriented entity-relationship (SOOER) model for coupled knowledge-base/database of image retrieval expert system (IRES)
US5493677A (en) * 1994-06-08 1996-02-20 Systems Research & Applications Corporation Generation, archiving, and retrieval of digital images with evoked suggestion-set captions and natural language interface

Also Published As

Publication number Publication date
WO1998052119A1 (en) 1998-11-19
JP2001525959A (en) 2001-12-11

Similar Documents

Publication Publication Date Title
Smith et al. Integrated spatial and feature image query
Fournier et al. Retin: A content-based image indexing and retrieval system
US6192150B1 (en) Invariant texture matching method for image retrieval
Niblack et al. QBIC project: querying images by content, using color, texture, and shape
US9007647B2 (en) Methods and apparatus to identify images in print advertisements
Faloutsos et al. Efficient and effective querying by image content
US5987456A (en) Image retrieval by syntactic characterization of appearance
CA2290445A1 (en) Method and system for image retrieval
JP2001521250A (en) Information search and search system
CA2397424A1 (en) Content-based image retrieval using positive and negative examples
Ahmed et al. A study on different image retrieval techniques in image processing
Yang Content-based image retrieval: a comparison between query by example and image browsing map approaches
KR100323364B1 (en) A method for extracting characterized vectors in color image and a retrieval method using the characterized vectors
James Face Image retrieval with HSV color space using clustering techniques
Fauqueur et al. New image retrieval paradigm: logical composition of region categories
JPH08263514A (en) Method for automatic classification of document, method for visualization of information space, and information retrieval system
Joint et al. PIRIA: a general tool for indexing, search, and retrieval of multimedia content
Kulkarni Natural language based fuzzy queries and fuzzy mapping of feature database for image retrieval
Chalechale et al. An abstract image representation based on edge pixel neighborhood information (EPNI)
Cho et al. A novel virus infection clustering for flower images identification
Buijs et al. Visual learning of simple semantics in imagescape
Golshani et al. Content-based image indexing and retrieval system in imageroadmap
Dai et al. Unbalanced region matching based on two-level description for image retrieval
Gregory et al. Using contextual information for image retrieval
Wu et al. Improving image retrieval with semantic classification using relevance feedback

Legal Events

Date Code Title Description
EEER Examination request
FZDE Discontinued