JP2004133812A - Program and device for retrieving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image retrieval program and image retrieving device capable of fast retrieving a practical image file. <P>SOLUTION: This image retrieval program registers and retrieves data consisting of image data and image related data associated with the image data with an image characteristic amount extracted from the image data and the image related data as a key. The image retrieval program has a characteristic amount extracting step 13 for extracting an image characteristic amount and a characteristic amount extraction preprocessing step 12 for performing prescribed image processing before the characteristic amount extracting step, and the characteristic amount extraction preprocessing step 12 can be replaced. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a technique for registering and retrieving an image and image-related data associated with the image.
[0002]
[Prior art]
Conventionally, when searching for a desired image file from a large number of image files, image features representing image contents are obtained for all the image files, and an image having a feature amount close to the feature amount of the desired image file is obtained. The file was searched and displayed as a candidate. The prior art relating to such a search includes the following.
(1) Japanese Patent Application Laid-Open No. 2000-187731 "Computer-readable recording medium storing a program for causing a computer to execute each step of image feature extraction and its method"
(2) Japanese Patent Application Laid-Open No. 2000-285141, "Image Searching Apparatus, Image Classifying Apparatus, and Computer-Readable Recording Medium Recording Program for Making Computer Function as These Apparatus"
(3) Japanese Patent Application Laid-Open No. 2000-31246 "Similar image display method and recording medium storing similar image display processing program"
[Patent Document 1] JP-A-2000-187731 [Patent Document 2] JP-A-2000-285141 [Patent Document 3] JP-A-2000-31246 [0003]
[Problems to be solved by the invention]
However, when searching for an image, it is extremely rare to search only for image features representing image contents, and in fact, various types of information such as text, date, geographical information, or information on the shooting environment of an image, and even voices are used. Generally, a search is performed by including image-related data in search conditions.
In such a case, if the application program integrates the function of only image search and the search function based on other information, it is necessary to further narrow down by other information conditions after the image search. Not only is large, but also when it is desired to limit the number of searches, an extremely difficult situation occurs. In other words, when 10 search results are ultimately desired, it is not known how many will be dropped due to the conditions of the application program. You have to do it.
The above-mentioned prior art discloses an image search technique for a large number of images, but does not mention a search method other than images.
In view of this problem, the present invention provides an image search program and an image search device capable of performing a high-speed and practical image file search by simultaneously performing a search using data associated with an image at the time of image search. With the goal.
It is another object of the present invention to provide a program structure that allows an appropriate program module to be easily defined when developing a search program.
[0004]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, a first aspect of the present invention is to provide an image feature amount and image-related data obtained by extracting data consisting of image data and image-related data associated with the image data from the image data. An image retrieval program for registering and retrieving with a key as a key, comprising: a feature value extraction step of extracting an image feature value; and a feature value extraction preprocessing step of performing predetermined image processing prior to the feature value extraction step. The feature amount extraction pre-processing step is characterized in that it is configured to be replaceable.
According to a second aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched using the image feature amount extracted from the image data and the image-related data as keys. In the image search program, a search step of searching the image data with an image feature amount and image-related data is provided.
According to a third aspect of the present invention, in the image search program according to the second aspect, the search step includes a search result obtained by executing a search for each image-related data based on a plurality of designated search conditions. The narrowing search order is changed according to a search result searched by the image feature amount.
According to a fourth aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched using the image feature amount extracted from the image data and the image-related data as keys. In an image search program, a registration step of arranging and registering image data, image-related data and image feature amount data in a tree structure in which the position of the data is specified by a path from the root, and referring to the data path at the time of search And performing a search.
According to a fifth aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched for using the image feature amount extracted from the image data and the image-related data as keys. The image search program is characterized in that the image search program has a search filter for removing unnecessary data from a search result performed by the search step, and the search filter is replaceable.
[0005]
According to a sixth aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched using the image feature amount extracted from the image data and the image-related data as keys. In the image search program, when registering the image data and the image-related data, a registration filter is provided for performing registration only when a pre-designated image feature amount or image-related data satisfies a registration condition.
According to a seventh aspect of the present invention, in the image search program according to the sixth aspect, the registration filter is configured to be replaceable.
According to an eighth aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched using the image feature amount extracted from the image data and the image-related data as keys. The image search device is characterized by including a registered data filtering means for registering only data satisfying a predetermined condition.
According to a ninth aspect of the present invention, data composed of image data and image-related data associated with the image data is registered and searched using the image feature amount extracted from the image data and the image-related data as keys. The image search device is characterized by including a search result filtering means for outputting only search results satisfying a predetermined condition.
[0006]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of an image search device / program of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration of the present invention. This configuration is configured to perform high-speed search by extracting an image feature amount used at the time of image search in advance. It should be noted that a method of extracting feature amounts one by one at the time of search may be used.
It is assumed that an image obtained by a digital camera or the like and data associated with the image are in the storage device 2. Of course, it is also conceivable that it can be obtained directly from an input device such as a digital camera. Although the data (image-related data) associated with the image is two in this figure, it increases and decreases according to the number of image-related data.
Further, not only the application program 4 and the image search program 5 are realized by one computer 1, but also the form in which the application program 4 and the image search program 5 exist on independent computers and are connected by a network. Good.
[0007]
Next, the processing of each module of the image search program 5 will be described.
When the application registration filter is specified in the application definition process 6, the registration filter process 11 calls the application registration filter of the application definition process 6, determines whether or not the image file needs to be registered, and registers the image file. The process of registering only the necessary image files is performed.
The preprocessing module 12 performs preprocessing of an image for extracting an image feature amount. For example, when extracting a feature value from only the central portion of the image, a process of extracting the central portion from the image is performed. When the pre-processing program is designated by the application definition processing 6, the pre-processing program of the application definition processing 6 is called.
When an image and data associated with the image are specified, the feature extraction module 13 extracts a feature from the image by using a feature extraction processing program.
The registration module 14 performs a process of registering an ID for specifying an image corresponding to the specified image, data associated with the image, and a feature amount extracted from the image in a storage device.
Then, the search module 15 performs a process of searching the data registered in the storage device 2 based on the image feature amount extracted in the feature extraction process and the data associated with the image, and returning the searched image ID.
The search filter processing 16 removes search results that do not satisfy the conditions specified for the search results. If a search filter is specified by the application definition processing 6, the search filter of the application definition processing 6 Call the process.
[0008]
By the way, when an application program is constructed using an image search function, a method of extracting feature amounts differs depending on the application program. For example, in the case of an image such as product data, the product is usually located at the center of the image, and therefore, it is possible to perform an accurate image search by extracting the feature amount only from the central region of the image. That is, it is necessary to perform preprocessing for extracting the central region of the image before extracting the feature amount. Therefore, it is necessary that the developer of the application program can freely set a preprocessing program for extracting the optimum feature amount for each application program.
The application program developer creates in advance a function (a function written in any programming language that can be called from this program) for executing the preprocessing of the feature extraction for each feature, and defines the function when creating the application program. Specify as a quantity preprocessing function (the image to be preprocessed is used as an argument). In this manner, the feature amount extraction process performs the following process.
1) Check whether a feature preprocessing function is specified.
a) If the feature amount preprocessing function is not specified, a default preprocessing is performed.
b) If a preprocessing function is specified, the preprocessing function is called with the specified image as an argument. In the preprocessing function, preprocessing is performed on the image of the argument and the image is returned.
2) Extract feature values from the preprocessed image.
When extracting two features, for example, two colors (color histogram) and edges (features obtained by dividing the edge image extracted from the image into a grid and extracting the number of pixels at the edges of each grid) It is possible to specify a preprocessing function for each feature extraction. In addition, if a process for activating other software is described in the preprocessing function, any described program can be executed as a preprocessing program. Since the pre-processing is an independent program, the pre-processing program can be easily replaced.
[0009]
Next, a data structure used for the image search of the present invention will be described. An image often has data associated with the image. For example, the title, creation date, and creator of an image, and more recently, positional information based on GPS (Global Positioning System) may be given. When searching for such an image, it is necessary to search not only the image feature amount but also various data associated with the image. For example, it is possible to prepare a DBMS (Data Base Management System) such as an RDB (Relational Data Base) on the application program side, but the development cost on the application program side increases and an optimal search process for speeding up the search is possible. I can't do that either. Therefore, information (image-related data) associated with an image to be searched is also managed and searched, whereby an application program is constructed to enable high-speed search. As shown in FIG. 1 described above, in the secondary storage of the storage device 2, a database of image-related data is prepared in addition to the database of the image ID and the image feature amount.
[0010]
Here, the processing at the time of registration will be described below with reference to the diagram showing the data structure of FIG. At the time of registration, an image and image-related data are designated (in FIG. 1, three files at the top in the storage device 2).
1) First, a feature amount extraction process is performed: the process described above is performed. 2) An image ID is registered in an image ID database (DB), and a feature amount ID is acquired. The image ID is an ID for the user to manage the image, and may be, for example, a character string such as a file name or a URL, or a numerical ID independently managed by the user. . The feature amount ID is an ID assigned to the feature amount extracted from the image, and the feature amount, the image ID, and the related data are all specified by this ID.
3) Register the extracted feature value at the position indicated by the feature value ID in the feature value DB 4) Register the related data 1 at the position indicated by the feature value ID 5) Position the related data 2 at the position indicated by the feature value ID Register with.
The number of related data is not particularly limited. The data structure processed and registered as described above is as shown in FIG.
Next, an example will be described in which the related data is the title and creation date of the image. The search condition shown in this example is an image whose title is “summer of Enoshima” or whose creation date falls after January 1, 2002, and whose image img121. This means searching for an image having a similarity with jpg within 0.2 or less. It should be noted that the search data in this case uses the registration data of FIG. 3 as an example.
img121. jpg (similarity <0.2) AND (title = “Summer in Enoshima” OR creation date> “01/01/2002”)
Actually, if a search condition is specified by a character string as described above, it is necessary to analyze the character string of the search condition. Instead, a simple method of specifying the search condition of each DB in each argument of the search function is required. Implementation is possible.
[0011]
Next, a search processing procedure will be described.
The search in this example is a process when not only an image but also a condition of related data is specified. In this example, a feature amount search is performed after searching for related data.
1) Feature extraction processing: As described above, an image feature (feature A) is extracted.
2) Retrieval of related data 1 based on search condition of related data 1 to obtain search result feature ID set 3) Feature ID set obtained by searching related data 2 based on search condition of related data 2 and search result A union set (when OR is specified in the search condition) or an intersection set (when AND is specified in the search condition) of the feature amount ID set is newly set as a search result feature amount ID set.
4) The union set (when OR is specified in the search condition) or the intersection set (the search condition) of the feature amount ID set obtained by searching the feature amount DB based on the image feature amount search condition and the search result feature amount ID set Is newly set as a search result feature amount ID set. Similar to a general similar image search, the feature amount search can be realized by comparing the feature amount A with the feature amounts stored in the feature amount DB one by one and calculating the similarity.
5) The search result feature amount ID set is sorted based on the similarity obtained at the time of searching the feature amount DB, and a search result list is created.
6) Acquire an image ID from the image ID DB and assign the image ID to the search result list.
[0012]
FIG. 4 shows the search result list obtained in this manner.
In the above example, the search was performed in the order of the related data 1, the related data 2, and the image feature amount. However, the search may be performed in any order. However, search time may be affected depending on search conditions. That is, when the search for the image feature amount and the search for the related data are OR searches, only the search result of the image feature amount and the search result of the related data are simply merged (union set). Does not make any difference in the processing time. However, when the AND search is included, the search speed can be improved as the feature ID set to be searched can be narrowed down at an early stage.
Therefore, the speed can be increased by checking the number of searches and changing the processing order as described below.
1) Feature amount extraction processing: The feature amount (feature amount A) is extracted as described above.
2) The search result feature quantity ID set is an empty set.
3) The feature quantity, related data 1 and related data 2 are searched based on the search condition, and only the respective search numbers are obtained.
4) The following processing is performed in ascending order of the number of searches.
A new search is performed on the union (when OR is specified in the search condition) or the intersection (when AND is specified in the search condition) of the search result of the related data or the image feature amount and the search result feature amount ID set. The result feature amount ID set is used.
5) The search result feature amount ID set is sorted based on the similarity obtained at the time of searching the feature amount DB, and a search result list is created.
6) Acquire an image ID from the image ID DB and assign the image ID to the search result list.
Examples of the relevant data include a path name of a directory in which the actual image data is stored, a keyword of the image, GPS position information, camera information (exposure, etc.) at the time of photographing the image, and audio information describing the contents of the image. Is also conceivable.
In particular, when data (image data) represented by a file system of a computer is managed in a tree structure, a path from the root of the tree structure is managed as related data, and is searched for. Image retrieval in consideration of a tree structure can be performed at high speed.
Taking a file system of a computer as an example, image data is often managed by the following paths. This path can also be a classification at the same time.
/ Image / Photo / Four seasons / Summer / Sea / Enoshima. jpg
If this path is managed as related data, it is possible to search for an arbitrary directory (that is, a classification) by prefixing the character string of the path. Further, a DBMS (database management system) such as a commercial relational database (RDB) may be used for managing and searching related data.
[0013]
According to the above embodiment, it is possible to manage and search various data together with the feature amount as the relevant data, but it is necessary that the program of the present invention has various related information search means. If the related data is text or numerical data, it can be easily realized. However, for voice or the like, it is necessary to use a search means unique to the application program. Therefore, after the search processing of this program, a filter processing (a filter function described by the application program) designated by the application program is called as a filter function of the search result.
The procedure is as follows. At the time of the call, the search result list is the argument.
1) Check whether the filter function of the application definition process 6 is specified.
a) If a filter function is specified in the application definition process 6, the filter function of the application definition process 6 is called using the search result list as an argument. The filter function of the application definition process 6 returns the search data indicated by the search result list excluding unnecessary (unsuitable for the conditions of the application program) search results.
b) If no filter function is specified in the application definition process 6, a default process is performed. The default is to do nothing normally.
It should be noted that, although the description will return to the previous description, for the filter processing, similarly to the search filter, a registration filter for determining whether or not the image specified at the time of registration should be registered can be specified. The registration filter of the application definition process 6 is called with an image, an image ID, and related data as arguments. The procedure is as follows.
1) Check whether a registration filter is specified in the application definition process 6.
a) If the registration filter function is specified in the application definition process 6, the registration filter function is called with the image, image ID, and related data as arguments. The registration filter function determines whether or not it should be registered, and returns whether or not the image should be registered.
b) If no registered filter function is specified in the application definition process 6, a default process is performed. The default is to do nothing normally.
2) When the registration filter function of the application definition process 6 determines data to be registered, the registration process is continued. If it is determined that the image should not be registered, the registration process is interrupted.
With such a registration filter, for example, a filter process at the time of registration, such as removing a small image that is not suitable for image search, can be easily defined by the developer of the application program.
[0014]
【The invention's effect】
As described above, according to the present invention, a developer can freely define preprocessing and filter processing appropriate for each application program.
Further, it is possible to perform a high-speed search based on various data related to an image other than the image.
Then, it is possible to efficiently search for images managed (classified) in a tree structure.
Also, it is possible to avoid registration of undesired data and output of search results that do not meet the conditions.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of the present invention.
FIG. 2 is a diagram showing a data structure according to an image search of the present invention.
FIG. 3 is a diagram showing an example of registration data.
FIG. 4 is a diagram showing an example of a search result list according to the present invention.
[Explanation of symbols]
REFERENCE SIGNS LIST 1 computer 2 storage device 3 display 4 application program 5 image search program 6 application definition processing 11 registration filter 12 preprocessing module 13 feature quantity extraction module 14 registration module 15 search module 16 search filter

Claims (9)

In an image search program for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as a key, the image feature amount A feature value extraction step of extracting, and a feature value extraction preprocessing step of performing predetermined image processing prior to the feature value extraction step, wherein the feature value extraction preprocessing step is configured to be replaceable. Image search program. In an image search program for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as a key, the image data is stored in an image. An image search program comprising a search step for searching for a feature amount and image-related data. The search step includes changing a narrowing search order according to a search result obtained by executing a search for each image-related data based on a plurality of designated search conditions and a search result searched by an image feature amount. 3. The image search program according to claim 2, wherein: An image retrieval program for registering and retrieving data composed of image data and image-related data associated with the image data with an image feature quantity extracted from the image data and the image-related data as a key includes image data, image-related A registration step of arranging and registering the data and the image feature amount data in a tree structure in which the position of the data is specified by a path from the root; and a search step of performing a search by referring to the data path during a search. An image search program characterized by the following. The search step is performed in an image search program for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as keys. An image search program having a search filter for removing unnecessary data from search results, wherein the search filter is replaceable. In an image search program for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as a key, the image data and the An image search program characterized by comprising a registration filter for registering image-related data only when a pre-designated image feature amount or image-related data satisfies registration conditions. 7. The image search program according to claim 6, wherein the registration filter is configured to be replaceable. In an image search apparatus for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as a key, a predetermined condition is satisfied. An image search device comprising a registered data filtering means for registering only data. In an image search apparatus for registering and searching data consisting of image data and image-related data associated with the image data with an image feature amount extracted from the image data and the image-related data as a key, a predetermined condition is satisfied. An image search device comprising a search result filtering means for outputting only search results.

Images