JP2013131051A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a mechanism of image retrieval to enable a user to intuitively and visually confirm an influence of each feature quantity on a query image and the weighting ratio of each feature quantity.SOLUTION: In an information processing apparatus that uses a feature quantity of an image to retrieve a similar image, on an object that receives a change of feature quantity weighting in retrieving an image similar to a query image, an image to be retrieved in a case of using only one feature quantity is displayed.

Description

  The present invention relates to a search method and a display method in an image search device.

  Search systems (called full-text search systems) based on keywords (character information) for searching document information on the Internet are already widely used, and the basic technology has been established and matured. It can be said that. On the other hand, for image information that is not character string information, the search method is not established as much as the full-text search system, and various methods are being studied.

  A currently popular technique is to perform a search using the same technique as a full-text search system based on character string information (called keywords or tags) attached to an image. In recent years, a mechanism for presenting an image (referred to as a query image) that is a source of an image to be searched for and searching for an image close to the query image has been provided.

  In the image search system using the above-mentioned image as a query, a numerical or numerical vectorized image feature quantity extracted from the query image is used as a search index so that the full-text search system uses a keyword (character information) as a search index. . When a query image is received from a user, a feature amount is extracted from the query image, and the feature amount is compared with a feature amount group extracted from an image to be searched in advance. Is determined to be an image similar to the query image, and an operation of responding them as a search result is performed.

  Examples of image feature values here include, for example, statistics on colors (consisting of hue, saturation, brightness, etc.), patterns (textures), local area density change characteristics, and the shape of the object in the image. Various kinds of things such as shooting date and time, shooting place, etc. are taken into consideration.

  In order to solve such a problem, for example, in the example of image search in Patent Document 1, a plurality of types can be used in combination as feature amounts used for search. Also, when searching using a plurality of feature amounts, the user can specify the weights with a slider bar.

JP 2004-62804 A

  Generally, in a search system using keywords, whether or not a document simply includes a search character string (query keyword) itself, a synonym, or a synonym can be used as an index of validity of a search result. On the other hand, in the case of a similar image search system that searches for similar images based on images, it is possible to uniquely determine based on what index (image characteristics) it can be determined that the original image and the target image are similar. Difficult to determine.

  For example, when a user searches based on a photo A, the user may expect a photo showing a landscape similar to the landscape of A as a result. Another user may expect a photo that shows an object similar to an object (for example, a chair) in the photo based on the photo A. As described above, it is difficult to uniquely determine a result expected by the user from a certain query image, that is, to automatically determine which feature of the query image should be used to determine similarity.

  In the similar image search system of Patent Document 1, when a user designates the weight of each feature amount, the user must determine the meaning of the feature amount only with keywords such as the overall color, color distribution, and contour. Don't be. Also, it is difficult to intuitively determine the weight ratio of each feature amount from the value of the slider bar.

  Accordingly, an object of the present invention is to provide an image search mechanism that allows a user to intuitively and visually confirm the influence of each feature quantity on a query image and the weighting ratio of each feature quantity.

  The present invention is an information processing apparatus that displays an image similar to a search image input by a user, a storage unit that stores a plurality of search target images, a reception unit that receives input of a search image by a user, An image feature amount calculating unit that calculates a plurality of feature amounts of the search image received by the receiving unit and a plurality of feature amounts of the search target image stored in the storage unit for each feature amount; and the image feature amount calculating unit A feature-similar image search means for searching for a search target image similar to the search image, and a weight-specifying object generator for generating a weight-specifying object for adjusting the weight of each feature amount And the adjustment of the weight of each feature quantity of the weight designation object generated by the weight designation object generation means An image searched by the feature-similar image search unit is arranged in the area, and a display control unit that displays the weight designation object and an image searched by the feature-similar image search unit in the weight designation object are arranged Similar to the search image from the search target image in consideration of the weight adjustment reception unit that receives the adjustment of the weight of each feature amount by the user's operation on the region, and the weight of each feature amount received by the weight adjustment reception unit Comprehensive similar image search means for searching for an image to be performed, wherein the display control means further displays the similar image searched by the comprehensive similar image search means.

  Further, the present invention is an information processing method in an information processing apparatus that displays an image similar to a search image input by a user and stores a plurality of search target images. An accepting step for accepting an input of a search image by the image processing unit, and an image feature amount calculating unit of the information processing apparatus calculates a plurality of feature amounts of the search image accepted by the accepting step and a plurality of feature amounts of the stored search target image The image feature amount calculating step for each feature amount and the feature similar image search means of the information processing apparatus are similar to the search image for each feature amount calculated by the image feature amount calculating step. A feature-similar image search step for searching for a search target image and a weight designation object generation unit of the information processing apparatus adjust the weights of the respective feature amounts. A weight specifying object generating step for generating a weight specifying object, and the display control means of the information processing apparatus in a region that accepts adjustment of the weight of each feature quantity of the weight specifying object generated by the weight specifying object generating step; The display control step of arranging the image searched by the feature similar image search step and displaying the weight designation object, and the weight adjustment acceptance unit of the information processing device are the feature similar image search step of the weight designation object. A weight adjustment accepting step for accepting adjustment of the weight of each feature amount by a user's operation on the area where the searched image is arranged, and a comprehensive similar image search means of the information processing device accepted by the weight adjustment accepting step Considering the weight of each feature amount, the search image is classified into the search image With a comprehensive image retrieval step of retrieving an image that, the, the display control step further, and displaying the similar image retrieved by the overall similarity image retrieval process.

  The present invention also provides a program that can be executed by an information processing apparatus that displays an image similar to a search image input by a user and stores a plurality of search target images. An image feature quantity for calculating, for each feature quantity, an accepting means for accepting image input, a plurality of feature quantities of the search image accepted by the accepting means, and a plurality of feature quantities of the search target image stored in the storage means For each feature quantity calculated by the image feature quantity calculation means, a feature similarity image search means for searching for a search target image similar to the search image, and the weight of each feature quantity is adjusted. A weight specification object generating means for generating a weight specification object; and a weight specification object generated by the weight specification object generation means. Display control means for displaying the weight designation object by arranging the image searched by the feature similar image search means in an area for receiving adjustment of the weight of each feature amount of the object, and the feature similarity image in the weight designation object In consideration of the weight adjustment accepting unit that accepts the adjustment of the weight of each feature amount by the user's operation on the area where the image retrieved by the retrieval unit is arranged, and the weight of each feature amount accepted by the weight adjustment accepting unit The display control means further functions to search for similar images searched by the comprehensive similar image search means. The display control means further functions as a general similar image search means for searching for an image similar to the search image from the search target image. It is characterized by displaying.

  According to the present invention, when the user designates the weight of each feature amount, the influence of each feature amount on the query image and the ratio of the weight of each feature amount can be intuitively and visually confirmed. The convenience of the user in the image search is improved.

1 is a system configuration diagram showing an example of a configuration of an image search system of the present invention. It is a block diagram which shows the hardware constitutions of the information processing apparatus applicable to the user terminal 101 and the image search apparatus 102 shown in FIG. It is a figure showing the structure of the functional block of the user terminal 101 and the image search apparatus 102 which were shown in FIG. It is a figure showing the function block inside the feature-value extraction part 503 shown in FIG. It is a figure which shows an example of the feature-value table in the feature-value database 505 shown in FIG. It is a flowchart which shows an example of the registration procedure of the search object image in the image search system in this invention. It is a flowchart which shows the procedure of the search process in the image search system in this invention. It is a flowchart which shows the procedure of the update process of the search result in the image search system in this invention. It is a figure which shows an example of the weight designation | designated object of the similar image search in the image search system in this invention. It is a figure which shows an example of the display screen in the result display part 302 of the user terminal 101 of the search result in the image search system in this invention. It is a figure which shows an example of the display screen in the result display part 302 of the user terminal 101 of the search result in the image search system in this invention. It is a figure which shows an example of the type | formula which calculates | requires the total similarity of the similar image in the image search system in this invention. It is a figure which shows an example of the type | formula for calculating | requiring the weight of each feature-value after the weight change in the image search system in this invention. It is a figure which shows an example of the weight designation | designated object of the similar image search in the image search system in this invention. It is a figure which shows an example of the weight designation | designated object of the similar image search in the image search system in this invention. It is a figure which shows an example of the type | formula for calculating | requiring the weight of each feature-value after the weight change in the image search system in this invention. It is a figure which shows an example of the weight designation | designated object of the similar image search in the image search system in this invention. It is a figure which shows an example of the weight designation | designated object of the similar image search in the image search system in this invention. It is a figure which shows an example of the type | formula for calculating | requiring the weight of each feature-value after the weight change in the image search system in this invention. It is a flowchart which shows the detailed process of the weight designation | designated object generation in the image search system in this invention.

<First Embodiment>

  FIG. 1 is a diagram showing a configuration of a relay processing system in an embodiment of the present invention.

  FIG. 1 shows a configuration in which one or a plurality of user terminals 101 and an image search apparatus 102 are connected via a local area network (LAN) 103.

  The user terminal 101 is an information processing apparatus used by a user who searches for an image, and has a function of transmitting a search request and a function of receiving and displaying a search result.

  The image search apparatus 102 stores a plurality of images to be searched, and has a function of accepting a search request from the user terminal 101, performing an image search process, and returning a search result. In addition, a means for inputting an image to be searched from outside is provided.

  Hereinafter, the hardware configuration of the user terminal 101 and the image search apparatus 102 illustrated in FIG. 1 will be described with reference to FIG.

  FIG. 2 is a block diagram illustrating a hardware configuration of an information processing apparatus applicable to the user terminal 101 and the image search apparatus 102 illustrated in FIG.

  In FIG. 2, reference numeral 201 denotes a CPU that comprehensively controls each device and controller connected to the system bus 204. Further, the ROM 203 or the external memory 211 is necessary to realize a BIOS (Basic Input / Output System) or an operating system program (hereinafter referred to as an OS), which is a control program of the CPU 201, or a function executed by each server or each PC. Various programs are stored.

  A RAM 202 functions as a main memory, work area, and the like for the CPU 201. The CPU 201 implements various operations by loading a program necessary for execution of processing from the ROM 203 or the external memory 211 into the RAM 202 and executing the loaded program.

  An input controller 205 controls input from the input device 209 or the like. A video controller 206 controls display on a display device 210 such as a liquid crystal display. The display device is not limited to a liquid crystal display, and may be a CRT display or the like. These are used by clients as needed.

  A memory controller 207 is connected to the hard disk (HD), flexible disk (FD), or PCMCIA card slot for storing a boot program, various applications, font data, user files, editing files, various data, etc. via an adapter. The access to the external memory 211 such as a compact flash (registered trademark) memory is controlled.

  A communication I / F controller 208 connects and communicates with an external device via a network (for example, the LAN 400 shown in FIG. 1), and executes communication control processing in the network. For example, communication using TCP / IP is possible.

  Note that the CPU 201 enables display on the display device 210 by executing outline font rasterization processing on a display information area in the RAM 202, for example. Further, the CPU 201 enables a user instruction with a mouse cursor (not shown) on the display device 210.

  Various programs that operate on the hardware are recorded in the external memory 211 and are executed by the CPU 201 by being loaded into the RAM 202 as necessary.

  It should be noted that not all devices have these configurations, and it is only necessary to provide necessary devices.

  Next, the functional block configuration of the image search system of the present invention will be described with reference to FIG.

  The image search system includes a user terminal 101, an image search device 102, and an image source 600. The user terminal 101, the image search device 102, and the image source 600 are connected to each other via a network so that they can communicate with each other.

  The user terminal 101 is an information processing device for sending an image search request to the image search device 102 and receiving and displaying the search result. The user terminal 101 includes a search creation unit 301, a result display unit 302, and a re-search instruction unit 303.

  The search query unit 301 includes an input unit that receives a query image as a search request from a user. The search query unit 301 is a function processing unit that includes a unit that transmits the query image to the search processing unit 503 of the image search apparatus 102 via the network. is there.

  The result display unit 302 is a function processing unit having means for receiving a search result returned from the image search apparatus 102 and means for displaying the search result. Examples of display screens are shown in FIGS.

  The re-search instruction unit 303 collects the change information when the user changes the weight using a device such as a mouse on the weight designation object displayed on the result display unit 302 and uses it for the re-search. This is a function processing unit having means for calculating a feature amount weight vector and means for transmitting the feature amount weight vector to the total similarity calculation unit 506 of the image search apparatus 102.

  The image search apparatus 102 receives an image search request from the user terminal 101, executes the requested search process on the stored image, and transmits search result information to the user terminal 101. It is. The image search apparatus 102 includes an image registration unit 501, a feature amount extraction unit 502, a search processing unit 503, a feature amount approximation degree calculation unit 504, a weight designation object generation unit 505, an overall similarity calculation unit 506, The similar image search result generation unit 507, an image database 508, and a feature amount database 509 are configured.

  The image registration unit 501 is a function processing unit that registers an image to be searched into the system. Image data to be searched is received or acquired from an actor outside the system specified by the image source 600, the feature amount of the image data is stored in the feature amount database 509, and the image data itself is stored in the image database 508. Has the function of saving.

  The feature amount extraction unit 502 is a function processing unit that receives image data as input and extracts various feature amounts of the image data. FIG. 4 is a diagram illustrating an example of functional blocks inside the feature amount extraction unit 502. As shown in FIG. 4, the feature quantity extraction unit 502 includes a plurality of feature quantity extraction units. The feature amount extraction units indicated by 5021 to 5024 in FIG. 4 are function processing units that extract different types of image feature amounts. For example, scalable color feature quantities (feature quantities representing the color distribution state in the entire image), color layout feature quantities (feature quantities representing the color distribution state for each area of the image), edge histogram feature quantities (boundary lines in the image) For example, it is possible to apply an implementation for extracting feature amounts such as a feature amount (SIFT) and a scale inverse feature transform (SIFT) feature amount (a feature amount representing a density change feature of a local region). The number of individual feature amount extraction units is four in FIG. 4, but is not particularly limited, and may be implemented by the number of image feature amounts to be used.

  The search processing unit 503 accepts a query image as a search request, instructs the feature amount extraction unit 502 to perform feature amount extraction processing of the image data, and passes the feature amount information received as a result to the feature amount approximation degree calculation unit 504. Instructing the calculation of the approximation vector with the accumulated image, calculating the initial feature amount weight vector, passing the approximation vector data and the initial feature amount weight vector to the weight specifying object generation unit 505, This is a function processing unit that instructs generation, passes the initial feature weight vector to the total similarity calculation unit 506, and instructs calculation of the total similarity.

  The feature amount approximation degree calculation unit 504 receives a plurality of types of image feature amounts related to an image from the search processing unit 503, compares the image feature amounts with the feature amounts stored in the feature amount database 509, and compares them. Is a function processing unit that calculates the degree of approximation and generates an approximation vector.

  The weight designation object generation unit 505 receives the approximation vector data and the initial feature amount weight vector of the image group as a search result from the search processing unit 503, and the user intuitively and visually assigns the feature amount weighting when the search is performed again. This is a function processing unit that generates a weight designation object for adjustment and transmits it to the result display unit 302 of the user terminal 101.

  The overall similarity calculation unit 506 receives the feature weight vector from the re-search instruction unit 303 or the search processing unit 503 of the user terminal 101, and the overall similarity (approximation vector and feature) between the query image and the accumulated image. (Image similarity at the feature amount weight vector calculated from the amount weight vector) is calculated, and the total similarity is passed to the similar image search result generation unit 507 to generate a similar image search result at the feature amount weight vector. Is a function processing unit for instructing.

  The similar image search result generation unit 507 receives the total similarity of images from the total similarity calculation unit 506, generates similar image search results in descending order of similarity using the thumbnails of the corresponding images, and the user terminal 101 is a function processing unit that transmits the result to the result display unit 302.

  The image database 508 is a storage area for storing image data groups to be searched. The stored image data has means that can be directly accessed by an ID (identifier).

  The feature amount database 509 is a storage area for storing various feature amount data (FIG. 5) of image data to be searched. The feature amount data is managed in the table illustrated in FIG.

  In the table of FIG. 5, one row represents one piece of image data stored in the image database 508, and is indicated by a column name of feature amount 1 to feature amount 4 together with an image data ID (identifier) (also referred to as an image NO). In this column, each feature amount data of the image is stored.

  The image source 600 is an external actor that represents the source (input source) of the search target image in the image search system. For example, a site having a photo sharing function such as Flickr and Facebook on the Internet, a user who directly provides photo data, and video input devices such as various cameras and scanners are also conceivable.

  Next, a process for registering image data to be searched for in the image search system will be described with reference to FIG.

  The image source 600 (external device) described below can be of various types as described above. Here, an example in which an administrator terminal operated by a system administrator is used will be described.

  In step S601, the image registration unit 501 receives image data to be searched from an administrator terminal operated by a system administrator represented by the image source 600.

  In step S <b> 602, the image registration unit 501 inputs the received image data to the feature amount extraction unit 502. The feature quantity extraction unit 502 applies the image data to each individual feature quantity extraction unit exemplified by 5021 to 5024 in FIG. 4 and calculates 1 to 4 feature quantities of the image data. Usually, these feature quantities are expressed as a one-dimensional numerical vector. The feature quantity extraction unit 502 inputs the extracted feature quantity data (four from 1 to 4) to the image registration unit 501.

  In step S603, the image registration unit 501 registers the feature amount data acquired in step S602 in the feature amount database 509. The feature amount data is stored in a table structure as illustrated in FIG.

  A table structure for storing feature data will be described with reference to FIG.

  As already described, a plurality of types of feature amounts are extracted for a certain image. Various feature quantities are expressed as numerical (integer or floating point) multidimensional vectors. For example, a SIFT feature value, which is one of the local feature values, is expressed as a floating-point 128-dimensional vector. In the image search system described in this embodiment, four types of image feature amounts from feature amount 1 to feature amount 4 are acquired as shown in FIG. In the table of FIG. 5, one row stores feature amount data related to one search target image. Each feature amount data is stored in each column whose feature name is feature amount 1 to feature amount 4. Each feature quantity is basically stored as numeric multidimensional vector data, although the number of dimensions and the type of numeric value differ depending on the type of feature quantity.

In step S <b> 604, the image registration unit 501 stores the received image data in the image database 508.
This completes the image registration process.

  Next, a processing flow in image search will be described with reference to FIG.

  In step S <b> 701, the search processing unit 503 receives a search query image transmitted from the search query unit 301 of the user terminal 101.

  In step S702, the search processing unit 503 inputs the query image data to the feature amount extraction unit 502, and extracts feature amount data (four feature amount vectors in this embodiment) of the image.

  In step S <b> 703, the search processing unit 503 inputs the feature amount data acquired in step S <b> 702 to the feature amount approximation degree calculation unit 504. The feature amount approximation degree calculation unit 504 compares the received feature amount data with the feature amount data group stored in the feature amount database 509, and calculates the degree of approximation therewith. The degree of approximation related to one feature amount is expressed by, for example, a Eugrid distance between numerical vectors. The degree of approximation of the query image with respect to one search target image is expressed as an array of Eugrid distances for each feature quantity type (A to D in this embodiment), that is, a multidimensional numerical vector. The feature amount approximation degree calculation unit 504 generates a multidimensional numerical vector (hereinafter referred to as an approximation degree vector) representing the degree of approximation with the query image for all the search target images, and the search processing unit 503 uses the approximation degree vector group. To respond.

  In step S704, the search processing unit 503 calculates an initial feature amount weight vector. The initial feature quantity weight vector is a multi-dimensional numerical vector having the dimension of the number of feature quantities to be used, in which each weight value is 1 when added together, and represents the initial value of the weight of each feature quantity. In this embodiment, an initial feature amount weight vector (four-dimensional numerical vector) with an initial value of 0.25 is calculated so that the weights of the feature amounts 1 to 4 are all the same.

  In this embodiment, each value of the initial feature value weight vector is set to 0.25 from the feature value 1 to the feature value 4. However, this is not particularly fixed, and is matched to the usage form and the feature value to be used. To set the initial value.

  In step S705, the search processing unit 503 inputs the approximation vector group acquired in step S703 and the initial feature amount weight vector calculated in step S704 to the weight designation object generation unit 505. The weight designation object generation unit 505 generates an object (hereinafter referred to as a weight designation object) for the user to adjust the weight of each feature quantity from the approximation vector group and the initial feature quantity weight vector. Detailed processing in step S705 will be described later with reference to FIG.

  FIG. 9 shows an example of the weight designation object. The weight designation object is an object for the user to intuitively adjust the weight of each feature amount. An object indicated by 900 (image arranged at the center) represents a thumbnail of the query image.

  The objects indicated by reference numerals 902 to 906 represent thumbnails of similar images when focusing only on the feature quantity 1 of the approximation vector group. The thumbnail group is arranged in an area delimited by a line segment designated by 907 and a line segment designated by 908 in a circle designated by 901. The angle between the two line segments represents the weighting ratio of the feature amount 1.

  The objects indicated by reference numerals 909 to 913 represent thumbnails of similar images when attention is paid only to the feature quantity 2 of the approximation vector group. The thumbnail group is arranged in an area delimited by a line segment designated by 907 and a line segment designated by 914 in a circle designated by 901. The angle between the two line segments represents the weighting ratio of the feature amount 2.

  Objects indicated by reference numerals 915 to 919 represent thumbnails of similar images when focusing only on the feature amount 3 of the approximation vector group. The thumbnail group is arranged in an area delimited by a line segment designated by 914 and a line segment designated by 920 in a circle designated by 901. The angle between the two line segments represents the weighting ratio of the feature amount 3.

  Objects indicated by 921 to 925 represent thumbnails of similar images when focusing only on the feature quantity 4 of the approximation vector group. The thumbnail group is arranged in an area delimited by a line segment designated by 908 and a line segment designated by 920 in a circle designated by 901. The angle between the two line segments represents the weighting ratio of the feature amount 4.

  The angle of each line segment of the weight designation object is calculated from the initial feature amount weight vector. In this embodiment, since all of the initial feature weight vector values are 0.25, the angle between each line segment is generated as 90 degrees.

  By displaying such a weight designation object, the user can compare the thumbnail of the similar image for each feature amount with the query image. Thus, the weight of the feature amount 1 to the feature amount 4 can be adjusted by dragging each line segment while intuitively grasping the influence of each feature amount on the query image.

  Intuitively grasping the influence of each feature quantity on the query image makes it possible to easily determine which feature quantity should be searched to find the desired image, and the user can select the desired similarity to the query image. It is possible to easily find the image.

  Although five thumbnails of similar images for each feature amount are displayed in FIG. 9, the number of similar images is not particularly limited, and the number corresponding to the usage form may be displayed.

  It is also possible to set so that the number of thumbnails of similar images to be displayed changes according to the weight for each feature amount.

  In step S706, the weight designation object generation unit 505 transmits the weight designation object generated in step S705 to the result display unit 302 of the user terminal 101.

  In step S <b> 707, the search processing unit 503 inputs the initial feature weight vector calculated in step S <b> 704 and the approximation vector group acquired in step S <b> 703 to the total similarity calculation unit 506. The total similarity calculation unit 506 calculates the total similarity for each image using the initial feature amount weight vector and the approximation vector data.

  FIG. 12 shows a formula for calculating the total similarity. The number N of feature quantities to be used in the formula is 4 in this embodiment, and the total similarity is calculated by adding the approximate values of the feature quantities 1 to 4 and the weighted values. .

  In step S708, the total similarity calculation unit 506 inputs the total similarity calculated in step S707 to the similar image search result generation unit 507. The similar image search result generation unit 507 generates a thumbnail group of each corresponding image as a similar image search result based on the total similarity.

  In step S <b> 709, the similar image search result generation unit 507 transmits the generated similar image search result to the result display unit 302 of the user terminal 101.

  Next, detailed processing in step S705 will be described with reference to FIG.

  In step S2001, the weight designation object generation unit 505 substitutes 1 for n, and substitutes the number of feature values used for N (4 in this embodiment).

  In step S2002, the weight designation object generation unit 505 generates a similar image search result when focusing only on the feature quantity n in the approximation vector data group.

  In step S2003, if n is smaller than N, the process proceeds to step S2004. Otherwise, the process proceeds to step S2005.

  In step S2004, the weight designation object generation unit 505 increments the value of n by 1, and then proceeds to step S2002.

  In step S2005, the weight designation object generation unit 505 designates weights using the top five thumbnails of similar image search results for each feature amount generated in steps S2001 to S2004 and the initial feature amount weight vector. Create an object.

  The processing from when the user sends a search request to when the search result is responded has been described above.

  Next, a flow of processing for adjusting the weight of each feature amount used for the search by the user's operation and changing the search result will be described with reference to FIG.

  In step S801, the result display unit 302 of the user terminal 101 receives the weight designation object transmitted from the weight designation object generation unit 505 in step S705.

  In step S <b> 802, the weight designation object is displayed on the result display unit 302.

  In step S803, the result display unit 302 of the user terminal 101 receives the similar image search result transmitted from the similar image search result generation unit 507 in step S709.

  In step S804, the similar image search result is displayed on the result display unit 302.

  FIG. 10 is a screen example of the result display unit 302 displaying the weight designation object and the similar image search result. The object indicated by 1000 represents the weight designation object generated in step S705. The user adjusts the weights of the feature quantity 1 to the feature quantity 4 by dragging the line segment designated by 1001 to 1004.

  The method for adjusting the weight of each feature amount may be any method other than dragging the line segments 1001 to 1004.

  In the area designated by 1005, thumbnails of the similar images generated in step S708 are displayed in descending order of similarity. In FIG. 10, the number of similar image search results displayed is 25, but the number is not particularly limited, and the number corresponding to the usage form may be displayed.

  In step S805, it is determined whether a change in the weight of each feature amount has been accepted. Specifically, as described above, the determination is made based on whether or not there is an instruction for the weight designation object by the user.

  If a change in the weight of each feature amount has been received (step S805: YES), the process proceeds to step S806.

  When the change of the weight of each feature amount has not been received (step S805: NO), the processing shown in this flowchart is terminated.

  Here, although not shown in the present embodiment, a re-search button is displayed on the weight designation object, a change in the weight of each feature amount is received from the user, and the re-search button is pressed, You may make it change to S806 (it is judged as YES by step S805).

  That is, in step S805, a process of accepting a change in the weight of each feature value and determining whether or not the re-search button has been pressed may be performed.

  FIG. 11 is a screen example of the result display unit 302 displaying the weight designation object after the weight of the feature amount is changed by the user.

  In step S806, the re-search instruction unit 303 uses the angle of each line segment specified in 1101 to 1104 of the changed weight designation object specified in 1100 and the expression in FIG. Calculate feature weights. N in the equation represents the number of feature quantities to be used. In this embodiment, the weights after the weight change of the feature quantities 1 to 4 are calculated. The re-search instruction unit 301 calculates a four-dimensional numerical vector having these weights as an array as a feature amount weight vector.

  In step S807, the re-search instruction unit 303 transmits the feature amount weight vector calculated in step S806 to the total similarity calculation unit 506 of the image search apparatus 102.

  In step S808, the total similarity calculation unit 506 of the image search apparatus 102 receives the feature amount weight vector transmitted in step S807.

  In step S809, the total similarity calculation unit 506 uses the feature amount weight vector received in step S808, the approximation vector group generated in step S703, and the total similarity for each image using the equation of FIG. Is calculated.

  In step S810, the total similarity calculation unit 506 inputs the total similarity calculated in step S809 to the similar image search result generation unit 507. The similar image search result generation unit 507 generates a thumbnail group of each corresponding image as a new similar image search result based on the total similarity.

  In step S811, the similar image search result generation unit 507 transmits the generated similar image search result to the result display unit 302 of the user terminal 101.

  After step S811, the user terminal 101 continues to the process of step S803.

  As described above with reference to FIG. 8, by using the weight designation object shown in FIG. 9, the user can more intuitively and visually check the influence of each feature amount while checking the feature amount. The weight can be adjusted. This improves the convenience of the user when searching for an image by changing the weight of the feature amount.

<Second Embodiment>
Another example of the weight designation object in the present invention will be described with reference to FIG. 14, FIG. 15, and FIG.

  FIG. 14 shows an example of the weight designation object. An object indicated by 1400 represents a thumbnail of the query image. Similar to the example described with reference to FIG. 9, the objects indicated by reference numerals 1401 to 1404 represent thumbnail groups of similar images when each feature amount is used alone. In this example, the weight (ratio) of each feature amount is adjusted by the distance between the 1401 query image and each object indicated by 1401 to 1404. In addition, the weight ratio of each feature amount at that time is represented by a pie chart indicated by 1405.

  As described above, when adjusting the weight of each feature amount by the distance from the query image, if the distance between the object related to the other feature amount and the query image is not considered, the ratio of the weight of each feature amount is calculated. It is difficult to judge. Therefore, by displaying a pie chart showing a ratio such as 1405, the problem can be solved.

  FIG. 15 shows a weight designation object after weight adjustment. The object indicated by 1401 approaches the query image, and in the pie chart indicated by 1405, the ratio of the weight of the feature amount 1 is increased. The ratio of the weight of each feature amount is calculated using the distance from the query image and the formula in FIG. 16 so that the ratio increases as the distance from the query image decreases. However, for the feature quantity whose distance from the query image is 0, 0 is used as it is instead of the reciprocal of the distance as the value in FIG. 16 (the feature quantity is ignored and calculated). That is, for the feature quantity that should be ignored, the distance from the query image may be set to zero.

  Further, there may be a device for improving the convenience of the user such that the thumbnail of the similar image is displayed larger as the distance from the query image is closer.

  14, 15, and 16, an example of a weight designation object that enables the user to adjust the weight of the feature amount while confirming the influence of each feature amount more intuitively and visually. Explained.

<Third Embodiment>
Another example of the weight designation object in the present invention will be described with reference to FIGS. 17, 18, and 19.

  FIG. 17 shows an example of the weight designation object. An object indicated by 1700 represents a thumbnail of the query image. Similar to the example described with reference to FIG. 9, the objects designated by reference numerals 1701 to 1704 represent thumbnail groups of similar images when each feature amount is used alone. In this example, the weight (ratio) of each feature value is adjusted by the radius (area) of the circle of each object indicated by 1701 to 1704.

  When it is desired to change the weight of each feature amount, the user can make an adjustment by giving an instruction to the circle displaying each object (similar image thumbnail). For example, the adjustment can be performed by making the circle larger or smaller by dragging the circumferential portion. In addition, any method may be used for adjusting the size of the circle (adjusting the weight of the feature amount), such as an instruction by mouse operation or an instruction by keyboard operation on the circle.

  FIG. 18 shows the weight designation object after the weight change. This indicates that the area of the object indicated by 1703 is increased, and the weight ratio of the feature amount 3 is increased. The ratio of the weight of each feature amount is calculated using the radius of the object indicated by 1701 to 1704 and the equation of FIG.

  17 and 18 may also be displayed with a graph showing the weight ratio of each feature amount as indicated by 1405 in FIG.

  Furthermore, the number of similar images displayed in the object may be changed according to the area of the object.

  17, 18, and 19, an example of a weight designation object that enables the user to adjust the weight of the feature amount while confirming the influence of each feature amount more intuitively and visually. Explained.

  It should be noted that the configuration and contents of the various data described above are not limited to this, and it goes without saying that the various data and configurations are configured according to the application and purpose.

  Moreover, the program in this invention is a program which a computer can perform the processing method of FIGS. 6-8, FIG. Note that the program in the present invention may be a program for each processing method of each apparatus in FIGS.

  As described above, a recording medium that records a program that implements the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus stores the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium recording the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on an instruction of the program is actually It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the processing and the processing is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  The present invention may be applied to a system constituted by a plurality of devices or an apparatus constituted by a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or apparatus can enjoy the effects of the present invention. In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

101 User terminal 102 Image search device 103 LAN
301 Search Query Unit 302 Result Display Unit 303 Re-Search Instruction Unit 501 Image Registration Unit 502 Feature Quantity Extraction Unit 503 Search Processing Unit 504 Feature Quantity Approximation Calculation Unit 505 Weighted Object Generation Unit 506 Total Similarity Calculation Unit 507 Similar Image Search Result Generation unit 508 Image database 509 Feature quantity database 600 Image source

Claims (7)

An information processing apparatus that displays an image similar to a search image input by a user,
Storage means for storing a plurality of search target images;
Accepting means for accepting input of a search image by a user;
Image feature quantity calculating means for calculating a plurality of feature quantities of the search image received by the accepting means and a plurality of feature quantities of the search target image stored in the storage means for each feature quantity;
Feature similar image search means for searching for a search target image similar to the search image for each feature quantity calculated by the image feature quantity calculation means;
A weight specifying object generating means for generating a weight specifying object for adjusting the weight of each feature amount;
Display control for arranging the image searched by the feature-similar image search unit in an area for accepting adjustment of the weight of each feature amount of the weight specification object generated by the weight specification object generation unit and displaying the weight specification object Means,
Weight adjustment acceptance means for accepting adjustment of the weight of each feature amount by a user's operation on an area in which the image retrieved by the feature similar image retrieval means in the weight designation object is arranged;
Considering the weight of each feature amount received by the weight adjustment receiving means, the integrated similar image search means for searching for an image similar to the search image from the search target image,
The information processing apparatus, wherein the display control means further displays a similar image retrieved by the comprehensive similar image retrieval means. The comprehensive similar image search means further includes:
2. The information processing apparatus according to claim 1, wherein when an adjustment of the weight of each feature amount is received by the adjustment receiving unit, a similar image is searched again using the adjusted weight of the feature amount. The display control means further includes
The information processing apparatus according to claim 1, wherein a weight designation object including information indicating a weight ratio in each feature amount is displayed. The display control means further includes
The information processing apparatus according to claim 3, wherein the weight for each feature amount is displayed by an area of a region that accepts adjustment of the weight of each feature amount. The search image is further displayed on the weight designation object,
The display control means further includes
The information processing apparatus according to claim 3, wherein the weight for each feature amount is displayed by a distance from a search image displayed on the weight designation object. An information processing method in an information processing apparatus for displaying an image similar to a search image input by a user and storing a plurality of search target images,
A receiving step in which a receiving unit of the information processing apparatus receives an input of a search image by a user;
An image feature amount calculating unit of the information processing apparatus calculates, for each feature amount, a plurality of feature amounts of the search image received by the receiving step and a plurality of feature amounts of the stored search target image. A calculation process;
A feature-similar image search step in which a feature-similar image search unit of the information processing device searches for a search target image similar to the search image for each feature amount calculated by the image feature amount calculation step;
A weight designation object generation means for generating a weight designation object for adjusting the weight of each feature amount, wherein the weight designation object generation means of the information processing apparatus;
The display control means of the information processing apparatus arranges the image searched by the feature-similar image search step in an area that accepts adjustment of the weight of each feature amount of the weight-specified object generated by the weight-specified object generation step. Display control step for displaying the weight designation object,
A weight adjustment acceptance unit that accepts adjustment of the weight of each feature amount by a user operation on an area in which the image retrieved by the feature-similar image retrieval process in the weight designation object is placed in the weight adjustment acceptance unit of the information processing apparatus Process,
A comprehensive similar image search step for searching for an image similar to the search image from the search target image, in consideration of the weight of each feature quantity received by the weight adjustment reception step, by the comprehensive similar image search means of the information processing apparatus And comprising
The information processing method, wherein the display control step further displays a similar image searched in the comprehensive similar image search step. A program that can be executed in an information processing apparatus that displays an image similar to a search image input by a user and stores a plurality of search target images,
The information processing apparatus;
Accepting means for accepting input of a search image by a user;
Image feature quantity calculating means for calculating a plurality of feature quantities of the search image received by the accepting means and a plurality of feature quantities of the search target image stored in the storage means for each feature quantity;
Feature similar image search means for searching for a search target image similar to the search image for each feature quantity calculated by the image feature quantity calculation means;
A weight specifying object generating means for generating a weight specifying object for adjusting the weight of each feature amount;
Display control for arranging the image searched by the feature-similar image search unit in an area for accepting adjustment of the weight of each feature amount of the weight specification object generated by the weight specification object generation unit and displaying the weight specification object Means,
Weight adjustment acceptance means for accepting adjustment of the weight of each feature amount by a user's operation on an area in which the image retrieved by the feature similar image retrieval means in the weight designation object is arranged;
In consideration of the weight of each feature amount received by the weight adjustment reception unit, the function is functioned as a general similar image search unit that searches for an image similar to the search image from the search target image,
The display control means further displays a similar image retrieved by the comprehensive similar image retrieval means.

Images