JP2010176235A - Image interface and image search program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily retrieve similar images satisfying some of a plurality of search conditions from an image database, and to visualize the degree of closeness between an image as search conditions and an image as a search result. <P>SOLUTION: Image data 403 recorded in a device are moved from a first region 401 of a screen to a second region 402 of the screen by a coordinate instruction input device. The image 405 corresponds to the moved image 403, attribute information applied to the image 405 is referred to by an image search program. Then, images whose attributes having high degree of closeness are displayed on the second region 402 of the screen, as the search result. A distance between an image group 406 displayed as the search result and the image 405 as the search conditions, accords with the degree of closeness of the attribute information applied to each of the images. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image interface and an image search program in which image thumbnails are arranged according to the tightness of attributes.

In recent years, with an increase in capacity of recording media for recording image data, a large amount of image data has been stored in a single recording medium.
Also, with the widespread use of devices that generate image data such as digital cameras, a large amount of image data is stored.
Therefore, the image database and its search method have the advantage that only the necessary images can be extracted from a large number of images by multiple search methods,
Widely used in practice.

FIG. 1 is a diagram for explaining a general practical image database interface screen. Reference numeral 101 denotes an interface window of the image database. An area 102 displays a list of image groups registered in the image database or thumbnails of images extracted from the image groups by image search processing. Reference numeral 103 denotes one of the image thumbnail groups displayed in the image display area 102. Reference numeral 104 denotes an operation panel for inputting a search condition to the image database, and displays a search window, a slider, or a check box corresponding to each of a plurality of attribute information added to the image. Reference numeral 105 denotes a list of image groups registered in the image database. Reference numeral 106 denotes an area for displaying details of the attribute information given to the image selected in the display area 102. A search condition is input to the search interface, and only images that satisfy all the conditions are extracted. Only the search results of the current image database satisfy all the search conditions are extracted, but it is practically sufficient for the purpose of extracting a specific image from many images.

As an example of a conventional image search processing system, JP 2006-18831 A (Publication 1) is cited. The system targets still image data or moving image data, extracts feature amounts and attribute information of the data, and assigns them to each data.
JP 2007-304694 A

However, when the current image database is used for the purpose of searching for an image that matches a part of a plurality of search conditions, there is a drawback that the search needs to be executed a plurality of times. Further, in order to input a plurality of search conditions, it is necessary to use search means corresponding to each attribute information, and there is a drawback that the input becomes complicated.

This disadvantage is a major obstacle to operability in the application of searching for a partially matching image among a plurality of search conditions from an image group.
Specifically, multiple searches require a lot of time,
In addition, there is a problem that it is difficult to grasp search conditions for obtaining necessary search results.

However, in the above conventional example, the search condition input becomes more complicated as the number of attribute information items added to the image increases.
Moreover, since the image group output as a search result is displayed as a list on the screen, it is difficult for the user to grasp the degree of partial match of the search in a short time.

The problems to be solved by the present invention are that an image that partially matches the search condition cannot be easily extracted from the image database, and that the input of a plurality of search conditions is complicated.

It is an object of the present invention to easily input a plurality of search conditions into an image database and extract an image that partially matches the search conditions. As a means for this, the present invention has two screen areas, a screen area for displaying image thumbnails and a screen area for displaying search results. The attribute information given to the image thumbnails is used as a search condition, and the search condition is closely related. The method is characterized in that the degree is expressed by the distance between the search condition image and the search result image.

The image interface of the present invention has an advantage that a plurality of search conditions can be easily input by using attribute information given to an image thumbnail as a search condition. Further, by outputting the attribute information given to the image thumbnail and the closeness of the search condition as a search result, there is an advantage that a similar image that does not completely match the search condition can be easily searched. Further, by replacing the tightness with the distance between the image thumbnail as the search condition and the image thumbnail as the search result, there is an advantage that it can be visually grasped on the screen.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

FIG. 2 is a diagram for explaining the structure of the apparatus according to the first embodiment of the present invention. Reference numerals 301 to 306 denote elements constituting the device.
301 is a central processing unit,
302 is a main memory,
303 is an auxiliary storage device,
304 is an output device,
Reference numeral 305 denotes an input device.
A bus 306 connects the devices 301 to 305 to exchange signals.
An image search program 307 is stored in the auxiliary storage device 303 and is read into the main storage device 302 at the time of execution.
It is executed by the central processing unit 301.
Reference numeral 308 denotes image data, which is stored in the auxiliary storage device 303 and appropriately read out by the executed image search program 307.
Reference numeral 309 denotes a display, which is an element constituting the output device 304.
The display 309 receives and outputs the processing result of the image search program 307.
Reference numeral 310 denotes a mouse and 311 denotes a keyboard, which are elements constituting the input device 305.
The image search program 307 receives input from the mouse 310 and the keyboard 311 and performs processing.

FIG. 3 is a diagram illustrating an example of a screen generated by the image search program 307 and displayed on the display 309. Reference numeral 401 denotes a first area of the screen on which thumbnails of the image data 308 are displayed.
Reference numeral 402 denotes a second area of a screen that displays an image as a search condition and a search result. Reference numeral 403 denotes a thumbnail of the image data 308. Reference numeral 404 denotes a button for scrolling the image data 403. Reference numeral 405 denotes an image thumbnail moved from the screen area 401 to the screen area 402 by an input from the mouse 310. Reference numeral 406 denotes a thumbnail of an image searched on condition of attribute information given to the image thumbnail 405.

FIG. 4 is a flowchart for explaining the operation method of the present invention. FIG. 5 shows the screen displayed on the display 309 at the start of the flowchart.
In step S01 of the flowchart, the user of the present invention selects one arbitrary image from the image area 401.
In step S02, the user of the present invention moves the image selected in step S01 to the image area 402 using the mouse 310.
FIG. 6 shows a screen displayed on the display 309 with the image moved in step S02.
In step S03, the image search program 307 receives the attribute information given to the image moved to the image area 402 in step S02, compares it with the attribute information given to each image in the image data 308, and measures the tightness. To do. An image group having a high tightness is extracted from the image data 308 and arranged around the image thumbnail 405.
The arranged image thumbnail group performs an animation that goes around the image thumbnail 405.
FIG. 7 shows a screen displayed on the display 309 in step S03.
In step S04, the image search program 307 selects an image having the next highest density after the image group extracted and displayed in step S03.
Extracted from the image data 308 and placed around the image thumbnail 405.
The arranged image thumbnail group performs an animation that goes around the image thumbnail 405.
The distance between the image thumbnail 405 and the image group extracted in step S04 follows the tightness.
Therefore, the image group extracted in step S04 is arranged on the outer periphery of the image extracted in step S03.
In step S 05, the user of the present invention selects an arbitrary image from the images arranged around the image thumbnail 405 with the mouse 310 and moves it onto the image thumbnail 405. When moved, the image search program 307 receives the attribute information given to the selected image, and the processing from step S03 is performed again.
If it is not moved, the series of operations ends.

The attribute information given to the image data 308 is composed of four pieces of information: shooting information, subject information, composition information, and color information.
Among these, shooting information is given by the shooting device or the user of the present invention, and the remaining subject information, composition information, and color information are extracted from the image by image analysis.

Table 1 shows the configuration of the shooting information. The shooting information includes four pieces of information: time information, position information, shooting settings, and photographer information. The information may be provided by an imaging device such as a camera at the time of image creation, or may be arbitrarily provided by the user of the present invention to an image.

The subject information is provided by the image search program 307 by extracting main subjects from the image data 308 and depending on the type of the subject. Subject type is human,
Classified into animals, plants, insects and landscapes. Each type is further classified. A person is classified into self-portrait, portrait, couple, family, group photo, unspecified majority, and sport. Animals are classified into terrestrial animals and fish, and terrestrial animals are classified into dogs, cats, cows, pigs, horses and the like. Landscapes are classified into natural objects and artifacts. Natural objects are classified as mountains, forests, seas, lakes, and sky, and artifacts are classified as urban areas, buildings, and fields. Since the subject of the present invention is the image search interface mounting method, description of the subject extraction method by image analysis is omitted.

FIG. 8 is a diagram for explaining a composition information extraction method.
Reference numeral 901 denotes image data from which composition information is extracted.
Reference numeral 902 denotes a point extracted from the image data 901 by the image search program 307.
A point 902 is extracted in the image data 901 based on a point where the contrast of color and brightness is high and the focus position of the camera.
Reference numeral 903 denotes a surface having a point 902 as a vertex.
The composition information is composed of the shape of the surface 903 and the center of gravity.

FIG. 9 is a flowchart and a diagram illustrating a method for extracting color information. In step S01 of FIG.
The image search program 307 divides the image data 308 into tiles.
In FIG. 9, 1001 is the image data 308 processed in step S01, and 1002 is one of the tiles into which the image 1001 is divided.
The number of divisions varies depending on the size of the image, but the maximum is about 32 divisions.
In step S02, the image search program 307 obtains one average color of each tile 1002 generated in step S01.
Reference numeral 1003 in FIG. 9 denotes the image data 308 processed in step S02. Each tile of the image 1003 has a single color.
In step S03, the image search program 307 assigns the average color group to the image as a representative color group of the image.
In step S04, the image search program 307 analyzes the color of each color of the representative color group.
Specifically, hue, saturation, and brightness are used as basic color information for each representative color.
In step S05, the image search program 307 compares the color information of each representative color and obtains the color tendency of the entire image.
In step S06, the image search program 307 compares the color trend obtained in step S05 with the color trend information table stored in the image search program 307, searches for a column that matches the condition, and searches for the color trend. Is replaced with the keyword assigned to the column.
Specifically, it is replaced with a keyword that is bright when the brightness of the entire representative color group is strong and dark when it is weak.
When the difference in brightness or saturation between the representative colors is large, the keyword is replaced with high contrast, and when the difference is small, the keyword is replaced with low contrast.
When the difference in hue between the representative colors is small, the keyword is replaced with a keyword such as monotone, vivid when the saturation of the entire representative color group is strong, or grayish when the saturation is weak.
In step S07, the image search program 307 adds the keyword obtained in step S06 to the image data 308.

The image search program 307 obtains a tightness between images by referring to the attribute information when executing the image search. The image search program 307
The tightness between images is obtained by the following formula 1.

In the above formula, as shown in Table 2, i is the tightness, n is the number of items of attribute information given to the image, E is the attribute information, and ix is the tightness correction value. Here, an image serving as a search condition is referred to as an image A, and an image as a comparison target for tightness is referred to as an image B. A value obtained by applying the mathematical formula to the attribute information given to the images is defined as the tightness between the images AB.
The distance between the two images in the screen area 402 follows the tightness.

The tightness correction ix in Equation 1 is obtained by Equation 2 below. In this embodiment, the attribute information of the image thumbnail for correcting the tightness of the attribute information between images is time information.

In Equation 2 above, as shown in Table 3, ix is the tightness correction value, and td is the absolute value of the difference in the photographing time information between the two screens.
FIG. 10 is a graph showing the calculation result of ix in the formula.
The tightness is corrected by the tightness correction value obtained by the mathematical formula.
By this correction, the tightness of the attribute information other than the time information is high, and the tightness is the highest between the two images with little or extremely large difference in time information.

In calculating the tightness correction value, the time information difference td exceeds about 2,
The search result image in which the tightness correction value ix starts to increase is
It is assumed that the rotation speed around the search condition image is faster than an image in which the time information difference td is closer to zero.
As a result, among the search result images with high density,
Difference between time information with little difference and extremely large one,
It enables a user of the present invention to grasp visually.

Since such an image interface mounting form is employed, a plurality of search conditions can be input to the image search program 307 using image thumbnails.
Therefore, there is an effect that an image search based on a plurality of conditions is facilitated. Also, using the attribute information given to the image as a search condition,
Images can be arranged according to the tightness between the search condition image of the attribute information and the search result image,
This has the effect that the similarity between images can be visually grasped. Furthermore, by correcting the arrangement of the search result image with the tightness correction value, it is possible to easily search and display an image that does not meet the search condition in the conventional search method and could not be easily extracted from the image database. There is an advantage.

Hereinafter, a first example of the embodiment of the present invention will be described. In the second embodiment of the present invention, the configuration of the apparatus, the structure of the image data, and the method for calculating the tightness are the same as those in the first embodiment, and thus the description thereof is omitted.

Of the attribute information given to the image, a method for giving color information will be described. The difference from the first embodiment is color trend information and keywords assigned to each column of the color trend table stored in the image search program 307. The color trend table is not a keyword that directly represents a color trend as in the first embodiment,
Stores keywords associated with image color trends. Specifically, keywords representing seasons such as spring, summer, autumn and winter, keywords representing emotions such as joy, anger, sorrow and comfort, keywords representing states such as spirit and loneliness are stored. Due to the difference, the color information added to the image in the second embodiment is different from that in the first embodiment, and therefore, the result output when the search is executed is also different between the two embodiments.

When the search result by the keyword of the color information in the second embodiment is obtained in the same manner using other attribute information given to the image as a search condition, the appropriate search condition is a limited range of very much attribute information. It is difficult to specify. According to the second embodiment, the user of the present invention can easily search for a desired search result reflecting the user's intention by using a keyword based on association from color information as a search condition. Has the effect of becoming possible.

As mentioned above, although this invention was demonstrated based on the preferable embodiment, this invention is not limited to these specific embodiment. Various embodiments that do not depart from the scope of the invention are also encompassed by the invention.

As an example, the apparatus in the embodiment of the present invention is not limited to the apparatus in the first embodiment, but is changed to a device having an image data browsing function and an input device such as a television, a digital camera, a mobile phone, a digital photo viewer, and a digital photo frame. It is possible to apply.

Further, the present invention includes a case in which a program and image data for realizing the functions of the embodiments of the present invention are supplied to a device from a storage medium or a wired or wireless connection network and executed. Examples of media for supplying the program and image data to the device include magnetic storage media such as a flexible disk, a hard disk, and a magnetic tape, MO, CD-ROM, and CD-R. It may be an optical storage medium such as CD-RW, DVD-ROM, DVD-R, DVD-RW, DVD-RAM, or BD, a magneto-optical storage medium, or a non-volatile semiconductor memory. Program and image data stored on a server on a computer network connected by wire or wireless
A method of downloading and supplying to a connected client computer is also included in the present invention. The present invention is not limited to a single server, and the present invention includes a case where a plurality of devices supply divided programs and images.

Furthermore, the processing device that executes the program that implements the embodiment of the present invention is not limited to the central processing unit built in the device, and the auxiliary processing device accepts and executes part or all of the processing of the program from the central processing unit. This case is also included in the present invention. Needless to say, the present invention includes a case where a function expansion board, a function expansion unit, or another computer arithmetic device connected to a computer executes a program for realizing the functions of the embodiments of the present invention.

It is explanatory drawing of the screen which displayed the interface of the image search software currently used. It is explanatory drawing which showed the structure of the apparatus of this invention. It is explanatory drawing which showed the display screen of this invention. It is the flowchart which showed the operation procedure of this invention. It is the figure which showed the state before performing a search among the display screens of this invention. It is the figure which showed the state which moved the image used as search conditions among the display screens of this invention, and performed the search. It is the figure which showed the state by which the image used as a search result was arrange | positioned among the display screens of this invention. It is explanatory drawing which showed the extraction method of composition information among the attribute information provided to an image. It is explanatory drawing which showed the extraction method of the representative color group of color information among the attribute information provided to an image. It is a graph which shows the calculation result of a tightness correction value.

101 Interface window in current image database software 102 Registered image or search result display area in current image database software 103 Registered image thumbnail in current image database software 104 Search panel in current image database software 105 Current image Image group list display area in database software 106 Detailed attribute information display area in current image database software 301 Central processing unit 302 Main storage unit 303 Auxiliary storage unit 304 Output unit 305 Input unit 306 Bus 307 Image search program 308 Image data 309 Display 310 Mouse 311 Keyboard 401 Screen for displaying image data 308 First area 402 Second area of screen for executing search and displaying search result 403 Thumbnail of image data 308 404 Scroll button 405 Image as search condition 406 Search result image 901 Image for extracting composition information 902 Focus of image Point with high position and contrast 903 Area with point 902 as vertex

Claims (4)

A first area displaying thumbnails of the images;
Means for moving the thumbnail of the image selected by the instruction means from the first area to the second image area;
The attribute information of the selected image thumbnail is referred to on the second area with the image thumbnail selected by the moving means as the center, and another image having a distance according to the tightness of the attribute around the image thumbnail. An image interface where thumbnails are placed. The method of referring to and comparing the attribute information of the selected image thumbnail and other image thumbnails according to claim 1;
An image search program comprising a method for calculating a tightness between images from a sum of differences in attribute information between images from the attribute information. The means for referring to the attribute information of the image thumbnail for correcting the tightness of the attribute information between the images according to claim 2,
An image search program comprising means for calculating a tightness correction value for correcting the tightness by division from the attribute information.   The image search program according to claim 3, wherein the tightness correction value according to claim 3 is calculated from a shooting time or a shooting position in the attribute information of the image thumbnail.