JP4019093B2 - Similar image retrieval method and apparatus - Google Patents

Description

  The present invention relates to a similar image search system.

  In the conventional similar image search system, in many cases, the search results are displayed as a list arranged in order according to the degree of similarity with an image serving as a search key. In this case, although the quantitative value of the actual similarity is not expressed, the qualitative relationship is expressed as a rank in the list.

  On the other hand, in the list display, the relationship with the key image is shown, but the similarity between the search results is not expressed. The display position of each image has order only in relation to the key image, and is unrelated to the preceding and subsequent images. In particular, when a large number of images are displayed, or when a plurality of key images are designated, the resulting display gives a random impression to the user.

  On the other hand, as proposed by the inventors of the present application as Japanese Patent Application No. 9-336750 or Japanese Patent Application No. 9-336751 “an image search method and apparatus and a search service using the same”, a measure of similarity If the search result is expressed as a scatter diagram of an image on a two-dimensional or three-dimensional coordinate system with the feature quantity as an axis, the similarity between data is expressed as a spatial arrangement, It is possible to present orderly information. As a result, a user interface that can withstand the display of a large number of images can be realized.

  When the search result is expressed as scattering in the space, the similarity with the key image is not expressed. Also, in the case of a list display, the quantitativeness of the similarity itself is not expressed. It is important for the user himself / herself to appropriately carry out the search work for obtaining an image that meets his / her wishes to appropriately convey to the user the judgment regarding the similarity that the similar image search system makes in the system. I can say that. Therefore, it is necessary to realize an interface that quantitatively conveys the similarity determined by the system to the user.

  On the other hand, expressing search results as a spatial scatter diagram has certain effectiveness as a method of presenting a large amount of data to the user, but considering the increase in the current electronic data scale, more data Needs to be considered.

  Each image that is a search result is displayed according to its similarity, the image having a high similarity is large, and the image having a low similarity is displayed small. Thereby, the judgment regarding the similarity of the system is quantitatively visualized to the user. By combining this method with a method of expressing as a spatial scatter diagram, a user interface capable of expressing both the similarity to the key image and the similarity between search results can be realized.

  On the other hand, in the present invention, the concept of time is introduced into a spatial scatter diagram in order to display a larger amount of data to the user. The scatter diagram of the search result varies depending on which feature quantity corresponds to the two or three axes of the coordinate system used for display. By interpolating the coordinate values of each image on different coordinate systems, the transformation of the coordinate system can be expressed as data motion. Each expression in each coordinate system has order, and as a result, the overall movement also has order. In particular, when a three-dimensional space is virtually represented, the movement of the object in the space also has an advantage of enhancing the three-dimensional impression of the arrangement of the object.

  According to the present invention, in the similar image search system, the similarity determined by the system can be visualized to the user in a quantitative format. In addition, a large amount of search results can be presented in an orderly format for the user.

  An embodiment of the system configuration of the present invention will be described with reference to FIG. The system configuration of this embodiment includes a search server 110 that performs a search based on an image feature amount, a visualization server 120 that displays a virtual three-dimensional space, and a GUI (graphic user interface) client 130. Information transmission between the server and the client is performed by socket communication.

  The search server 110 calculates the similarity with the search key image sent from the GUI client 130 based on the image feature quantity obtained from the image feature quantity database 140 and expanded in the memory in the server. The visualization server 120 displays the state of the virtual space according to the calculation result of the similarity by the server 110 and the motion of the object in the virtual space according to the display request sent from the GUI client 130 and the walk-through operation in the user's virtual space. Update in real time. In addition, the visualization server 120 acquires a user event such as a mouse click that occurs in the screen area where the virtual space is displayed. As described above, the GUI client 130 acquires information on specifying search conditions and controlling walk-through operations in the visualization space from the user.

  The visualization server 120 and the GUI client 130 operate on a graphic workstation having a high-performance texture mapping function. The search server operates on the same computer or another computer capable of high-speed numerical calculation.

  The search server 110 fetches the feature amounts for all the images from the image feature amount database 140 into the memory when the system is activated. Similarly, when the system is activated, the visualization server 120 takes in all the image data reduced images from the image database 150, converts them into data for texture mapping, and takes them into the memory. The GUI client 130 captures necessary image data from the image database 150 at each time point. The operation panel 131, the image panel 133, and the fluffy exercise setting panel 132 in the GUI client 130 will be described later.

  A specific example of the user interface of this embodiment will be described with reference to FIG. The user interface of this embodiment includes a visualization space 210 in which display and event processing are managed by the visualization server 120, an operation panel 131 in which display and event processing are managed by the GUI client 130, and the GUI client 130. It consists of three types of image panels 133 for managing display and event processing. These three types of panels include a favorite image panel 230, a dislike image panel 240, and a collected image panel 250.

  In the visualization space 210, the search result image is displayed as a three-dimensional scatter diagram. The size of each image in three dimensions is associated with the similarity calculated from the feature amount data of each image and the feature amount data of a favorite image and a dislike image specified by the user. The user's designation regarding the favorite image and the dislike image is reflected on the favorite image panel and the dislike image panel managed by the GUI client 130, respectively. The similarity is expressed by a real value between 0 and 1, and if it is 1, the image is displayed with the maximum size. If it is less than that, it will be reduced and displayed in proportion to the similarity, and if it is less than 0.1, it will not be displayed. The similarity value is determined by the search condition.

  Any method can be used as the image search procedure. For example, on the initial screen, display is performed based on the similarity from a plurality of key images (a set of typical images) prepared in advance on the system side. Easy to do. This corresponds to display of a sample image in a normal similar image search system.

  The coordinate axis of the visualization space 210 is selected from image feature amounts. In this embodiment, the selection of this axis is automatically changed at regular time intervals. While moving from one coordinate system to the next coordinate system, linearly interpolated values on the two coordinate systems of each image are used as the trajectory of the image. For the user, this is observed as an automatic three-dimensional movement of each image. Hereinafter, this movement is called fluffy movement.

  A function for a user to search through the visualization space 210 while walking through is performed via the operation panel 131. This will be described later. The user can register the image in any one of the image panels 133 described later by picking the image in the visualization space 210 with the mouse.

  A walk-through function setting and a fluffy exercise setting by the operation panel 131 will be described with reference to FIG. Related to the walk-through function are a swing control button 350, a forward / backward / left / right movement control button 360, a height control button 370, a movement speed control slider 340, and a viewpoint initialization button 310. Further, as related to the fluffy motion, there are a speed control slider 320 for the fluffy motion and a display button 330 for controlling whether or not the display of the fluffy motion setting panel is necessary (ON, OFF).

  With reference to FIG. 4, setting of the orbit of fluffy movement by the fluffy movement setting panel will be described.

  On the panel, a button 430 is used to set which feature amount is selected as the feature amount of the coordinate axes x, y, and z in the visualization space at each time point (step). In the example of the figure, the filled feature quantity 430 is selected as the feature quantity of the visualization space on each coordinate axis. In the example of this figure, one of eight feature values can be selected for each step, and different feature value combinations can be made on each coordinate axis. On the same coordinate axis, the feature quantity at each step is the same. Further, by selecting a step with the button 420, it is possible to jump to an arbitrary coordinate system for display. When duplication (DUP) or deletion (DEL) of the selected step is performed with the button 410, the entire length of the step is changed. The time length of each step is constant. In the case of duplication, a step having the same contents is inserted after the selected step. In case of deletion, the selected step is removed.

  In the default state, the trajectory loops forward in steps, but the user has the choice of whether to loop with button 440 and whether to loop with button 450 (on, off) according to his / her preference. it can. Further, the trajectory created by the button 470 can be stored (SAVE) on the file, and the trajectory created in advance can be read from the file (LOAD). The exercise set by the restart button 460 can be checked. When the display of the setting panel is unnecessary, the display can be stopped by clicking the close button 400.

The fluffy movement will be described with reference to the feature value selection in FIG. Focusing on Steps 1 and 2, there is no change in the feature values in the y-axis and z-axis, but they are different in the x-axis. As a result, the search result changes, and the image appearing in the visualization space 210 or the position and size of the image change. The name fluffy motion used in the present embodiment is given because the motion of the image that appeared earlier gives a feeling of fluffy motion.
As shown in FIG. 2, there are three types of image panels 133, a favorite image panel 230, a dislike image panel 240, and a collected image panel 250. The collected image panel is a place where an image that the user wants to keep is placed and does not affect the search condition. Registration of an image in each panel is performed by clicking the image in the above-described visualization space with the mouse, copying from another image panel, or reading from an external file. The images registered in the favorite image panel and the disliked image panel are registered in the search server 110 as key images.

  FIG. 5 shows a configuration example of each image panel. On each image panel, there is a display panel 510 that functions as an image list. Up, down, left and right scroll bars are provided on the right and bottom of the display panel, and a slider 560 for enlarging and reducing the image display size is provided on the top. With the button 520, storage of an image list registered in the image panel to an external file (SAVE) and registration of an image list from the external file (LOAD) can be controlled. A button 530 can be used to delete all registered images (CLEAR).

  The favorite image panel 230 and the dislike image panel 240 are provided with sliders 540 and 550 for designating parameters for determining a search condition. The similarity calculation control slider 540 designates a parameter for configuring the final similarity of each search target image from the similarity from a plurality of key images. A color weighting (Color) and shape weighting (Shape) slider 550 designates a color-related feature amount and a shape-related feature amount, and how much importance is given to each. A specific procedure for calculating the similarity from the registration key image and the search conditions given by these various parameters will be described later.

  The movement of the image between the visualization space 210 and the image panel 133 and the deletion of the image from the image panel 133 are generally provided as an input / output operation means of a computer (not shown). It is done by the operation by the thing).

  In the visualization space 210, when an image is picked with the left button, the image is registered in the favorite image panel 230, when the middle button is selected, it is registered with the disliked image panel 240, and when the right button is selected, it is registered in the collected image panel 250. The In the favorite image panel 230, when an image is picked with the left button, registration of the image is deleted after confirmation. When an image is picked with the middle button, the image is registered in the disliked image panel 240, and when an image is picked with the right button, the image is registered in the collected image panel 250. Similarly, in the disliked image panel 240, when the image is picked with the left button, the image is registered in the favorite image panel, when the image is picked with the middle button, the image is deleted after confirmation, and the image is picked with the right button. In this case, an image is registered in the collected image panel 250. In the collected image panel 250, when an image is picked with the left button, the image is registered in the favorite image panel 230. When an image is picked with the middle button, an image is registered in the disliked image panel 240, and with the right button. If you pick an image, delete the registration after confirmation. Of course, if you have a two-button mouse, you cannot use the shift key or control key together because you cannot decide all of the buttons.

  Next, calculation of similarity will be described. The calculation of the similarity is performed in the search server 110.

  In the present embodiment, as a color-related image feature amount, a feature amount obtained by histogramizing the color distribution in the image, and as a shape-related image feature amount, a feature amount obtained by histogramating the distribution of the direction component of the luminance gradient vector of the image. Used. Both are expressed as real vectors, and the similarity between images is defined based on the square distance between these vectors. The following equation defines the degree of similarity between a given image (search key image) and a certain image registered in the favorite image panel 230 (represented by the subscript i).

  After weighting the square distance for the two types of feature quantities, an index is taken as the similarity. The two weight parameters in the equation can be changed by setting with a feature amount weighting slider 550 on a desired image panel.

  Similarly, the similarity with the image registered in the disliked image panel 240 is defined by the following equation.

  The distance between a given image and a plurality of registered key images is configured as follows based on the similarity to each key image.

(Equation 3) is the similarity with the favorite image, and (Equation 4) is the similarity with the disliked image. According to the similarity calculation parameter, the calculation method uses the similarity with the most similar key image, uses the average similarity with all the key images, uses the similarity with the least similar key image The three stages are changed continuously. The similarity parameter is controlled from the image panel as in the case of weighting the feature amount. Based on these two similarities, the following similarity is finally defined.

The coefficient added to the similarity with the disliked image is for reducing the effect of the similarity with the disliked image when the weighting of the feature amount is small as a whole. If all feature weights are set to 0 on a dislike image panel, the effect of registering the dislike image is lost.

  The visualization server 120 continuously generates a fluffy motion independently from the GUI client 130 and the search server 110. The change in the visualization space as the user walks through is caused by control information sent from the GUI client. The change in the visualization space accompanying the change in the search condition is caused by the data flow as shown in FIG.

  When the visualization server 120 obtains an image registration request from the user to the image panel on the visualization space 210, the visualization server 120 notifies the GUI client of the registration of the image 610. When the registration request to the image panel notified from the visualization server 120 is a registration request to the favorite image panel 230 or the disliked image panel 240, the GUI client 130 makes a search condition to the search server 110. 620 to notify the change. Similarly, the GUI client 130 registers / deletes a favorite image / dislike image and changes a parameter for specifying a search condition in accordance with a user operation on an image panel managed by the GUI client itself. Also, 620 notifies the search server 110 of a change in the search condition. When the search server 110 obtains a search condition change notification from the GUI client, the search server 110 recalculates the similarity. When the GUI client 130 confirms that the recalculation of the similarity in the search server 110 due to the change of the search condition is completed, the GUI client 130 acquires the recalculated similarity from the search server 110 to the visualization server 120. 630 to notify. The visualization server 120 acquires similarity data 640 from the search server 110 and reconfigures the display of the visualization space 210. When the reconfiguration of the visualization space 210 is completed, the visualization server 120 notifies the GUI client 130 of the completion of the reconfiguration. After confirming the reconfiguration of the visualization space 210, the GUI client 130 returns to a normal user input waiting state. From the notification of the search condition to the search server 110 to the confirmation of the reconfiguration of the visualization space 210, except for the processing that does not involve communication with the search server 110 and the visualization server 120 such as scrolling of the image panel, the GUI client 130 Ignore input events from the user.

  By realizing the above flow of data on a graphic workstation with a high-performance texture mapping function, an interface that allows the user to recognize changes in the search conditions and the changes in the display in the visualization space as continuous events. Realized.

  FIG. 7 is a diagram for explaining the system configuration and operation when the present invention is expanded to an online image / video content service.

  In this system, a large number of licensed users 720 and content service providers 710 exchange information on the Internet. Browsing software 721 is previously provided to each user from the content service provider itself or a software provider. On the service provider side, there are a large-scale image / video database 722, their feature quantity database 712, and a thumbnail image database 713 for browsing software, which are managed by the management server 714.

  Note that, for video, feature quantities related to cut still images are stored in a database.

  Similar to the visualization server 120 and the GUI client 130 in the previous embodiment, the browsing software 721 updates the state in the virtual three-dimensional space in real time. The configuration of the entire user interface is the same as in the previous embodiment.

  When the user starts up the browsing software 721, the user first selects an image or video category to be browsed. On the server side, a plurality of key images are prepared for each category in advance as a set of typical images, and the similarity to the typical image is registered in each image. Using this information, the user can easily specify the category of images or videos to be browsed. When the management server 714 receives the selection of the user, the thumbnail image of the image or video corresponding to the selected category (in the case of video, the thumbnail image of the representative image cut and cut), their feature amount, and the typical image Are sent to the user in descending order of similarity. The browsing software 721 acquires data within a range that falls within a state update rate (time for one screen reconstruction) in the visualization space, and the overflow data is suspended to be acquired at the next state update.

  When the browsing software 721 receives a search condition specification (key image registration, feature weighting specification, etc.) from the user, the browsing software 721 requests the server 714 to recalculate the similarity. When the recalculation is completed, the server sends the recalculated similarity data to the user in the order of the change in similarity. At this time, the thumbnail image and the feature amount are also sent for data that has not been previously acquired by the browsing software. The browsing software preferentially acquires the recalculated similarity data, and discards the data that cannot be acquired and has been suspended. Also in this case, data overflowing from the state update rate is suspended.

  With the above processing flow, the user can browse the server-side image / video database without losing real-time characteristics. During browsing, a user who meets the data he / she wants again requests the server to acquire the main image or video in the database 711. The server sends the data to the user after performing billing processing and the like.

The figure which shows one Example of the system configuration | structure of this invention. The figure which shows the specific example of the user interface provided to a GUI client. The figure explaining the operation panel in the specific example of the user interface of FIG. The figure explaining the fluffy exercise | movement setting panel in the specific example of the user interface of FIG. The figure explaining each structural example of the favorite image panel, disliked image panel, and collection image panel in the specific example of the user interface of FIG. The figure explaining the movement of the data corresponding to the change of the search condition of the user in the specific example of the user interface of FIG. FIG. 3 is a diagram for explaining a system configuration and operation when a specific example of the user interface of FIG. 2 is developed in a content service.

Explanation of symbols

  110: Search server, 120: Visualization server, 130: GUI client, 131: Operation panel, 132: Fluffy motion setting panel, 133: Image panel, 140: Image feature database, 150: Image database, 210: Visualization space, 220 : Operation panel 230: Favorite image panel 240: Disliked image panel 250: Collected image panel 310: Viewpoint initialization 320: Fluffy motion speed control slider 330: Fluffy motion setting panel display 340: Walkthrough Moving speed control slider, 350: Walk-through swing control button, 360: Walk-through front / rear left / right control button, 370: Walk-through height control button, 400: Fluffy motion setting panel close button, 410: Duplicate and delete selected step button 420: Step selection button, 430: Feature axis assignment button to each step visualization coordinate system, 440: Motion direction selection button, 450: Repetitive motion (loop) on / off button, 460: Remotion of motion Start button, 470: Read or write file of orbit file, 510: Image list display panel, 520: Image list input / output button, 530: Delete all images button, 540: Similarity calculation control Slider, 550: Color weighting (Color) and Shape weighting (Shape) slider, 560: Slider for enlargement / reduction of image display size, 610: Registration of image, 620: Update of search condition, 630: Request for similarity data acquisition 640: acquisition of similarity data, 710: content service provider, 711: image / video data 712: feature amount database, 713: thumbnail image database, 714: database management server, 720: user, 721: browsing software, 722: feature amount data, 723: thumbnail image.

Claims (7)

A search server including a feature amount database storing a plurality of feature amounts for a plurality of images to be searched, a visualization server including an image database storing a reduced image of the search target image, and a search key image A method for displaying an image set in a similar image retrieval system including a client having a user interface,
Associating feature amounts in advance so that the correspondence relationship between each coordinate axis and the feature amount changes over time with respect to a plurality of coordinate axes in the visualization space generated in the user interface to display the image search result; ,
The search server searches for an image similar to the search key image based on the feature amount read from the feature amount database, and calculates a similarity for each search image;
The visualization server obtaining a search result of a similar image including a similarity to the search key image from the search server;
The visualization server displays a plurality of reduced images corresponding to the similar image indicated by the search result in the visualization space generated in the user interface according to the correspondence between the coordinate axes and the feature amount and the similarity. And consist of
Each reduced image is displayed in the visualization space with a size corresponding to each similarity and automatically changing the display position in accordance with the change in the correspondence between each coordinate axis and the feature amount. A method for displaying a set of images. The client has a step of displaying on the user interface a feature quantity selection screen presenting a plurality of feature quantities that can be associated with each coordinate axis along a time axis for a predetermined period divided into a plurality of sections. And
The association between each coordinate axis of the visualization space and the feature amount is set by the selection of any one of the feature amounts in each of the sections by the user on the feature amount selection screen,
The visualization server automatically changes the display position of each reduced screen in the visualization space by repeatedly using the correspondence between each coordinate axis and the feature amount for a predetermined period set on the feature amount selection screen. The method for displaying an image set according to claim 1, wherein: The visualization server uses a correspondence relationship between each coordinate axis of the visualization space and a feature amount to change a display position of each reduced image so that each reduced screen moves gently and softly in the visualization space. 3. The method of displaying an image set according to claim 1, wherein the image set is displayed. When the correspondence between the coordinate axis and the feature quantity in the visualization space changes, the visualization server linearly interpolates between a reduced image displayed in the previous coordinate system and a reduced image displayed in the new coordinate system. 4. The image set display method according to claim 2, wherein the reduced screens are gradually moved in the visualization space. When the correspondence between the coordinate axis and the feature quantity in the visualization space changes, the visualization server linearly interpolates between a reduced image displayed in the previous coordinate system and a reduced image displayed in the new coordinate system. The display method of the image set according to claim 2, wherein a movement locus of each reduced screen is displayed in the visualization space. The image set according to claim 1, wherein the visualization server controls a moving speed of each reduced screen in the visualization space in accordance with a speed specified by the user interface. How to display. A search server including a feature amount database storing a plurality of feature amounts for a plurality of images to be searched;
A visualization server comprising an image database storing a reduced image of the search target image;
A similar image search system comprising a client having a user interface for designating a search key image,
The search server has a means for searching for an image similar to the search key image based on the feature amount read from the feature amount database, and calculating a similarity for each search image;
The visualization server is
Means for changing the correspondence relationship between each coordinate axis and a feature amount with respect to a plurality of coordinate axes forming a visualization space generated in the user interface for displaying an image search result;
Means for acquiring a search result of a similar image including a similarity to the search key image from the search server;
Means for displaying a plurality of reduced images corresponding to the similar image indicated by the search result in the visualization space generated in the user interface according to the correspondence between the coordinate axes and the feature amount and the similarity. ,
Each reduced image is displayed in the visualization space with a size corresponding to each similarity and automatically changing the display position according to the change in the correspondence between each coordinate axis and the feature amount. A similar image retrieval system characterized by being configured to do so.

Images