JP4426698B2 - Evaluation method for image retrieval system and recording medium on which evaluation program is recorded - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an evaluation method for an image search system and a recording medium on which an evaluation program is recorded.
[0002]
[Prior art]
In recent years, with the widespread use of digital cameras and the development of computer image processing technology, various image search systems for searching and displaying digital images have been developed. As such an image search system, for example, an image database classified by image feature vectors is constructed, and an image having a feature vector that matches or approximates a specific input image is searched from the image database. Known (Japanese Patent Laid-Open No. 9-44518).
[0003]
[Problems to be solved by the invention]
However, in such an image retrieval system, unlike document retrieval, etc., it is necessary to capture image-specific ambiguity as a feature quantity, so depending on the feature quantity extraction / analysis method, A large variation occurs. Also, due to the difference between the image characteristics of the search engine and the human visual characteristics, a search result that is satisfactory for the user is not always obtained even if the image analysis capability of the search engine itself is high.
[0004]
Therefore, a mechanism for objectively evaluating the performance of the image search system is required.
[0005]
The present invention has been made in consideration of the above-described circumstances, and the object of the present invention is to be able to objectively evaluate the performance of the image search system at the time of development, factory shipment, user purchase, or the like. An object of the present invention is to provide an image search system evaluation method and a recording medium in which an evaluation program is recorded.
[0006]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention is connected to an image retrieval system that automatically retrieves an image similar to a specified specific input image from a set of images stored in an image memory . a method for evaluating the image retrieval system evaluation apparatus executes, the evaluation device includes a selection step of selecting a particular category from the set of images belonging to different categories are stored in the evaluation image database, said the image retrieval system and a given el image providing step one image to the image retrieval system as the input image in the plurality of images belonging to the selected in the selection step category, the image provided by the image providing step as a key image an image retrieval step of retrieving an image similar to the key image from the set of the image, in the image retrieval step Was the ratio of the number of images belonging to the particular category of similar image candidates is calculated for all the images belonging to the specified category, calculates the total sum or the average value thereof of the ratio as the evaluation value of the performance of the image retrieval system Performing a performance evaluation step .
[0007]
In this evaluation method, among the similar image candidates searched by the image search system when one image out of a plurality of images belonging to a specific selected category is set as the input image, it belongs to the same category as the input image. A ratio of the number of images is determined. This process is performed while sequentially changing the input image, whereby the ratio of all the images belonging to a specific category as the input image is obtained. In this way, the performance of the image search system is automatically evaluated using the sum of the ratios obtained for each input image or the average value thereof.
[0008]
Therefore, it is possible to objectively evaluate the performance of the image search system simply by preparing an image set including a large number of images belonging to various categories in advance. In particular, when a large number of live-action images taken with a digital camera or the like are classified into various categories such as by scene according to judgment based on human visual characteristics, a digital camera is used. It is possible to objectively evaluate the performance of the retrieval system in consideration of the characteristics of captured images and the human visual characteristics when categorizing.
[0009]
The number of similar image candidates to be searched by the image search system is preferably set to be equal to or smaller than the number of images belonging to the specific category. If the number of similar image candidates that are output as search results is set to be large, evaluation of images that are not searched among images belonging to a specific category becomes a problem, but the number of similar image candidates that are output belongs to a specific category. By setting it equal to or smaller than the number of the above, such a problem can be eliminated and a valid evaluation can be performed.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 shows functional configurations of an evaluation system 11 and an image search system 21 to be evaluated according to an embodiment of the present invention. The evaluation system 11 is for evaluating the search performance of the image search system 21. For this purpose, an image set prepared in the image evaluation database 1 is used as an evaluation image.
[0011]
In the image evaluation database 1, an image file group 1a and category classification information 1b are recorded. The image file group 1a is an image set for evaluation, in which a large number of image data files belonging to various categories are recorded. Each image data is taken with a digital camera or the like. The category classification information 1b is information indicating which of the plurality of categories each image data in the image file group 1a belongs to, and defines the image file name and the number of images belonging to each category. .
[0012]
The evaluation system 11 is realized, for example, as software that can be executed on a computer, and includes a search activation unit 111, a hit rate calculation unit 112, and an overall hit rate calculation unit 113 as functional modules as illustrated. ing. Similarly, in many cases, the image retrieval system 21 is realized as software that can be executed on a computer including a microcomputer. The search processing engine 211 of the image search system 21 searches for similar image candidates similar to the input image by comparing the image feature amount of each search target image group with the image feature amount of the input image given as a key image. . In this case, the number of similar image candidates to be output as search results can be designated in advance.
[0013]
The search method used by the search processing engine 211 may be any method. Some search processing engines 211 extract an image feature from each search target image in advance to generate an index, but this index and the contents of the category classification information 1b are completely independent. That is, the search processing engine 211 uses the image file group 1a as a search target image, but does not use the category classification information 1b at the time of the search.
[0014]
The search activation unit 111 takes out one image belonging to a specific category in response to an inquiry to the evaluation image database 1 and gives it as a key image to the search processing engine 211 to execute search processing. The hit rate calculation unit 112 calculates the ratio (hit rate) of images belonging to the same category as the key image among the similar image candidates searched by the search processing engine 211, and records it in the evaluation database 2. When all images belonging to the same category as the key image are searched as similar image candidates, the hit rate is 100% (hit rate = 1).
[0015]
The key images given to the search processing engine 211 by the search activation unit 111 are sequentially changed within the same category. As a result, the hit rate described above is obtained for all images belonging to the same category.
[0016]
The overall hit rate calculation unit 113 reads the hit rate obtained for each image in the specific category from the evaluation database 2 and calculates the sum or average value of the hit rates. This value is output as a search performance evaluation value of the image search system 21.
[0017]
Next, the mechanism of the evaluation method used in the present embodiment will be specifically described with reference to FIGS.
[0018]
FIG. 2 shows an example of an image set stored in the evaluation image database 1 and its categorization. Here, categorization classifies images according to scenes such as “party”, “meeting”, “athletic meet”, “snow mountain”, “sea bathing”, “business center”, “flower”, “animal”, etc. Is done by. This classification operation is performed by human visual judgment. In FIG. 2, “category A”, “category B”, “category C”,... Are categorized. “Category A” includes images a1, a2,..., An. Similarly, “Category B” includes images b1, b2,. Images c2,..., cn are included. Here, the value of n is preferably about one hundred to several hundred. Further, it is not necessary for the values of n in all the categories to be completely matched, but it is desirable that they be comparable.
[0019]
These image sets a1, a2, ..., an, b1, b2, ..., bn, c1, c2, ..., cn, ... are stored in the image file group 1a, and these image sets are searched for evaluation. The target image. At the time of evaluation, all images belonging to the category are sequentially selected as key images for each category. FIG. 3 shows this state.
[0020]
That is, when evaluating “Category A”, images a1, a2,..., An belonging to “Category A” are sequentially selected as key images, and a hit rate is obtained for each key image. Similarly, when evaluating “Category B”, images b1, b2,..., Bn belonging to “Category B” are sequentially selected as key images, a hit rate is obtained for each key image, and “Category C” is obtained. , “C”, images c1, c2,..., Cn belonging to “Category C” are sequentially selected as key images, and a hit rate is obtained for each key image. The final evaluation value is obtained by calculating the sum or average value of hit rates for each category and averaging it for all categories. This procedure is shown below.
[0021]
(1) As a result of searching for hit rate images ak (k = 1 to n) of images belonging to “category A” as key images, Xk images among images a1, a2,. Assuming that the hit rate Hk (A) is
Hk (A) = Xk / n (1)
Given in.
[0022]
This is obtained for all of ak (k = 1 to n), and an average value Σ {k = 1, k = n} Hk (A) / n of the sum Σ {k = 1, k = n} Hk (A) ... (2)
Ask for.
[0023]
(2) Similarly, for “Category B”, “Category C”,..., The average value of the total hit rate Σ {k = 1, k = n} Hk (B) / n, Σ {k = 1, k = n} Hk (C) / n,.
[0024]
(3) The average value H of the above sum is obtained, and this is used as the final hit rate (evaluation value).
H = {Σ {k = 1, k = n} Hk (A) / n + Σ {k = 1, k = n} Hk (B) / n + Σ {k = 1, k = n} Hk (C ) / n} / M (3)
Where M is the number of categories.
If the number n of images belonging to each category is the same, it is not always necessary to obtain the average hit rate sum for each category, and the final hit rate ( An evaluation value corresponding to (evaluation value) can be obtained.
[0025]
FIG. 4 shows the relationship between the number of similar image candidates to be searched by the image search system 21 and the number n of images belonging to each category. FIG. 4A shows an example where the number of similar image candidates is larger than the number n of images in the category, and FIG. 4B sets the number of similar image candidates equal to the number n of images in the category. This is an example. If the number of similar image candidates output as search results is set large as shown in FIG. 4A, “the hit rate when all images belonging to the same category as the key image are searched is set to 1”. Since the evaluation index is incomplete, there is a problem in evaluating images that are not retrieved from images belonging to a specific category. Therefore, in the present embodiment, as shown in FIG. 4B, the number of similar image candidates to be output is set to be the same as the number of images belonging to the specific category. For example, prior to starting the search, for example, the search starting unit 111 reads the value of the number n of images in the category to which the image used as the key image belongs from the category classification information 1b, and similar image candidates to be searched and output. This can be realized by setting the number in the image search system 21 as a number.
[0026]
Next, the procedure of the evaluation process of this embodiment will be described with reference to the flowcharts of FIGS.
[0027]
FIG. 5 schematically shows a flow from “preparation of evaluation image” to “execution and evaluation of search”. First, the processing from step S101 to S104 is performed as the evaluation image preparation processing. Steps S101 to S104 are performed based on operations performed manually on the computer. First, a large number of images to be used as evaluation images are prepared on the computer (step S101), and scenes of these image groups are prepared. Classification and scene arrangement are performed based on judgment by one or more operators (steps S102 and S103). Thereby, the evaluation images are classified into a plurality of categories (scenes). Then, the number of images by category (by scene) is counted and registered in the category classification information 1b (step S104), whereby the evaluation image database 1 is created. Thereafter, the “search execution / evaluation” process (step S105) by the evaluation system 11 is started. The procedure of the “search execution / evaluation” process is shown in FIG.
[0028]
The flowchart of FIG. 6 has a nested structure. Loop A is a part for obtaining the hit rate of each category, and loop B is a part for obtaining the hit rate of all categories.
[0029]
The hit rate for each category is calculated by executing a search execution process for causing the image search system 21 to execute a search (step S111), a hit rate calculating process for determining the hit rate from the search result (step S112), and calculating the calculated hit rate in the evaluation database 2. The hit rate recording process (step S113) to be recorded is repeatedly executed for all the images in the same category (in the same scene) while changing the key image (step S114) (step S114).
[0030]
In the search execution process (step S111), as shown in the flowchart of FIG. 7, the search activation unit 111 makes an inquiry to the evaluation image database 1, and one image belonging to a specific category to be processed is extracted. Then, the retrieved image is given to the image retrieval system 21 as a key image, so that the retrieval process of the image retrieval system 21 is started (step S122).
[0031]
In the hit rate calculation process (step S112), as shown in the flowchart of FIG. 8, the hit rate calculation unit 112 first refers to the category classification information 1b, thereby totaling images in the category to which the key image belongs (total number of scene images). Then, the number of images (same scene image number) belonging to the category to which the key image belongs among the similar image candidates searched by the image search system 21 is checked (step S132). Then, a hit rate (= number of identical scene images / total number of scene images) is obtained from the number of identical scene images and the total number of scene images (step S133).
[0032]
When processing is completed for all images in a certain category (YES in step S114 in FIG. 6), a scene hit rate calculation process (step S115) is performed to calculate the average of the total hit rates obtained for each key image. The scene hit rate calculated in this way is recorded in the evaluation database 2 (step S116). In the scene hit rate calculation process (step S115), as shown in the flowchart of FIG. 9, the hit rate obtained for each key image is read from the evaluation database 2 (step S141), and the average of the total sum is obtained. Performed (step S142).
[0033]
When the processing up to step S116 is completed for a selected specific category, the same processing is performed for another category (steps S111 to S116). In this way, when the recording of the scene hit rate for all categories is completed (YES in step S117), an overall hit rate calculation process (step S118) for obtaining the average of the sum of the scene hit rates for all categories is executed. . In the overall hit rate calculation process (step S118), as shown in the flowchart of FIG. 10, the scene hit rate obtained for each category is read from the evaluation database 2 (step S143), and the average of the total sum is obtained. (Step S144).
[0034]
As described above, in this embodiment, an evaluation image group is prepared in advance by categorizing a large number of real images taken with a digital camera or the like, and for all images belonging to a specific category. By obtaining the hit rate when it is used as a key image, the performance evaluation of the search system 21 is objectively performed in consideration of the characteristics of images taken with a digital camera and human visual characteristics when categorizing. Can be done. Therefore, it is possible to correctly evaluate the search performance when the user is used.
[0035]
In this embodiment, the average of scene hit rates for all categories is used as the final evaluation value, but the scene hit rate for a specific selected category is used as the final evaluation value. A simple method may be used. Further, a computer program for search evaluation including the procedure of the evaluation system 11 of this embodiment is recorded on a computer-readable recording medium, so that the computer program for search evaluation is introduced into a normal computer through the recording medium. Only by doing this, it is possible to obtain the same effect as in the present embodiment. Moreover, since the present invention can be applied regardless of the type of image search system, it can be used for evaluation of various image search systems.
[0036]
【The invention's effect】
As described above in detail, according to the present invention, it is possible to objectively evaluate the performance of the image search system at the time of development, factory shipment, user purchase, or the like.
[Brief description of the drawings]
FIG. 1 is a block diagram showing functional configurations of an evaluation system and an image search system to be evaluated according to an embodiment of the present invention.
FIG. 2 is a view showing an example of categorization of an image group used as an evaluation image in the embodiment.
FIG. 3 is a view for explaining the principle of an evaluation method in the embodiment.
FIG. 4 is a view for explaining the relationship between the number of similar image candidates to be output as search results to the search system and the number of images belonging to each category in the embodiment.
FIG. 5 is a flowchart schematically showing the flow of evaluation processing in the embodiment.
FIG. 6 is an exemplary flowchart showing the procedure of search execution / evaluation processing in the embodiment;
7 is a flowchart showing the procedure of search execution processing performed in the search execution / evaluation processing of FIG. 6;
FIG. 8 is a flowchart showing the procedure of hit rate calculation processing performed in the search execution / evaluation processing of FIG. 6;
FIG. 9 is a flowchart showing a procedure of a scene hit rate calculation process performed in the search execution / evaluation process of FIG. 6;
FIG. 10 is a flowchart showing a procedure of an overall hit rate calculation process performed in the search execution / evaluation process of FIG. 6;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Evaluation image database 2 ... Evaluation database 11 ... Evaluation system 21 ... Image search system 111 ... Search starting part 112 ... Hit rate calculation part 113 ... Whole hit rate calculation part 211 ... Search processing engine

Claims (4)

An evaluation method for the image search system executed by an evaluation apparatus connected to an image search system for automatically searching an image similar to a specified specific input image from a set of images stored in an image memory Because
The evaluation device is
A selection step of selecting a specific category from a set of images belonging to various categories stored in the evaluation image database ;
A given el image providing step one image to the image retrieval system as the input image in the plurality of images belonging to the selected category by the selecting step,
An image retrieval step of retrieving an image similar to the key image from the set of the images in the image retrieval system image provided by the image providing step as a key image,
Calculated for all the images belonging to a ratio of the number of images belonging to the particular category of the image search similar image candidates obtained in step in the specific category, the sum or the average value thereof of the ratio of the image retrieval system And a performance evaluation step of calculating as an evaluation value of the performance .   2. The image search system evaluation method according to claim 1, wherein the number of similar image candidates to be searched by the image search system is set to be equal to or smaller than the number of images belonging to the specific category. A computer-readable recording medium on which an evaluation program for evaluating an image search system for automatically searching for an image similar to a specified specific input image from a set of images stored in an image memory is recorded Because
When a specific category is selected from a set of images belonging to various categories and one image among a plurality of images belonging to the selected category is given to the image search system as the input image, the image search is performed. A ratio of the number of images belonging to the specific category among similar image candidates searched from the set of images by the system is obtained for all images belonging to the specific category, and the sum of the ratios or an average value thereof is used. A computer-readable recording medium on which an evaluation program for evaluating the performance of the image search system is recorded.   4. The recording medium according to claim 3, wherein the number of similar image candidates to be searched by the image search system is set to be equal to or smaller than the number of images belonging to the specific category.

Images