JP2010122931A - Similar area retrieval method, similar area retrieval device, and similar area retrieval program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve retrieval accuracy by properly reflecting intention of a user in retrieval. <P>SOLUTION: An input image is selected as a key image to be retrieved (S101), and a user selects partial areas and an attention degree in the key image (S102). After selecting all partial areas (S103), image characteristics are extracted from the partial areas (S104). After that, an area having a size equal to those of the partial areas in images stored in a data storing part 205 is processed in the same way as S103(S105). Next, S105 is applied to all the areas while sequentially shifting the area to be processed by S105 for each stored image (S106). After S104 to S106 are applied to all the areas (S107), image characteristics of the areas of both images are compared, and a similarity is calculated in consideration of the attention degree (S108). The stored images are rearranged and displayed in order of the similarity (S109). <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technique for searching an image similar to a target image to be searched from a database using an image processing technique.

  In recent years, an environment for easily acquiring and storing large-capacity digital images has been improved by the spread of digital cameras, the increase in storage capacity and price, and the spread of large-capacity and high-speed networks. There is a growing need for technology to search for images.

  For example, a technique for detecting an image in which a face of a specific person is photographed from a group of images stored in an image database, or a technique for searching a home page including an image in which a specific object or building is photographed Is required. As this technique, the partial image search method of Patent Document 1 is known.

The technique of Patent Document 1 divides a key image (target image) to be searched by reference and a stored image in a database into small regions, and searches the extracted features by vector quantization. At this time, a sample feature set relating to a typical pattern of image features is prepared in advance, a distance between each of the sample features is calculated in advance, and a sample feature corresponding to the stored image is selected, thereby obtaining a key image. Find sample features corresponding to, and save matching time.
JP 2006-011622 A

  However, since the code corresponding to the sample feature is assigned to the feature of the key image and the stored image, the prepared sample feature set may affect the search accuracy.

  In other words, if the key image is within the assumed range, it is sufficient to prepare a pattern that characterizes the image. However, since an appropriate pattern is not prepared for an unknown image that is not assumed, it can be correlated. Difficult cases arise.

  In addition, even if the pattern is similar to human eyes, if the physical feature quantity is not similar, the corresponding sample feature set may not be found and the search may not be executed.

  The present invention has been made in view of such circumstances, and realizes a search that reflects the user's search intention without associating with a sample feature set prepared in advance, and improves the search accuracy. It is a problem to be solved.

  Therefore, in order to solve the above-described problems, the invention according to claims 1 to 9 extracts the image features of the partial area to be noted in the key image, and then stores the accumulated image for each size equivalent to the partial area. Image features are extracted, and the image features of both images are compared to calculate the overall similarity. At this time, it is also possible to set the degree of similarity in consideration of the degree of attention specified for the partial area. The attention level may be specified in a stepwise manner, or a negative region that is not desired to be reflected in the search result may be specified as a negative value.

  Specifically, the invention described in claim 1 is a method for searching for an image including a similar partial area in a target image to be searched from an accumulated image in a database, wherein the area selection means is configured to input the input A first step for allowing a user to select a partial area to be noted from the target image; and a feature extracting unit extracts an image feature of the partial area from the target image, and for each accumulated image, a partial area of the target image; The second step of extracting image features for each region having the same size, and the similarity calculation means compare the image features of the partial region in the target image with the image features of each region in each accumulated image. And a fourth step in which the image sorting means presents the search results rearranged in the order of similarity calculated in the third step to the user.

  In the second aspect of the present invention, the second step extracts the image feature for each partial region when a plurality of partial regions are selected, and the third step extracts the image feature of each partial region. It is characterized by comparing with each image feature of the region.

  The invention according to claim 3 is characterized in that, when the degree of attention is specified in a stepwise manner in each selected partial region, the third step reflects the stage of the degree of attention in the similarity. Yes.

  According to a fourth aspect of the present invention, in the third step, when a positive value is specified for the attention level, the comparison result of the image features acts in a similar direction, while a negative value is specified for the attention level. In this case, the comparison result of the image features is caused to act in a dissimilar direction.

  The invention according to claim 5 is an apparatus for searching for an image including a similar partial region in a target image to be searched from stored images in a database, and a portion to be noted by the user from the input target image An area selection means for selecting an area, and an image feature of a partial area selected from the target image are extracted, and an image feature is extracted for each of the accumulated images for each area having the same size as the partial area of the target image. A feature extraction unit that compares the image feature of the partial region in the target image with the image feature of each region in each accumulated image, and calculates the similarity, and the calculated similarity Image sorting means for presenting the search results rearranged in order to the user.

  According to a sixth aspect of the present invention, the area selecting means is set so that a user can select a plurality of partial areas from the target image, and the feature extracting means is configured to select the image feature for each of the selected partial areas. And the similarity calculation means compares the image feature of each partial region with each image feature of each region.

  In the seventh aspect of the present invention, the region selecting means is set so that attention can be specified in a stepwise manner for each of the plurality of selected partial regions, and the similarity calculating means sets the attention degree step to the similarity. It is characterized by being reflected in.

  In the invention according to claim 8, the region selecting means is set so that either a positive value or a negative value can be specified for the stepwise attention degree, and the similarity calculating means has a positive value for the attention degree. The comparison result of the image feature acts in a similar direction when specified, and the comparison result of the image feature acts in a dissimilar direction when a negative value is specified for the degree of attention. It is a feature.

  A ninth aspect of the present invention relates to a similar region search program that causes a computer to function as the similar region search device according to any one of the fifth to eighth aspects.

  According to the first to ninth aspects of the invention, a similar image reflecting the user's intention is searched without being associated with a sample feature set prepared in advance, and the search accuracy is improved.

  FIG. 1 shows an example of the configuration of a similar area search apparatus 200 according to an embodiment of the present invention. The search apparatus 200 searches an image including a color area similar to a target image to be searched from a database. .

  Specifically, the similar region search apparatus 200 is configured by a computer, and includes an input unit 202, a region selection unit 204, a data storage unit 205, a feature extraction unit 206, a region movement unit 207, a feature amount calculation unit 208, a similarity degree. A calculation unit 209, a similarity comparison unit 210, an image sorting unit 211, a display unit 212, and a control unit 213 are included. These units 202 to 213 function by cooperation of software and hardware.

  That is, the input unit 202 is realized by a communication device or an input / output port for inputting image data from an external storage device or the like via a network. The input image 201 input here is set as the target image, that is, a key image to be searched. The region selection unit 204 causes the display unit 212 to select a partial region that should be noted by the user from the key image.

  The data storage unit 205 is constructed as a database for storing a stored image group to be searched. For the construction of this database, storage means such as a hard disk drive of a computer is used. The data storage unit 207 is not necessarily built in the search device 200, and may be a database connected via a network.

  The feature extraction unit 206 extracts image features from each partial area in the key image. In addition, image features are extracted from an area having the same size as each of the partial areas existing in the stored image of the data storage unit 205. At this time, the region moving unit 207 extracts the image features of the entire region of the accumulated image while shifting the range of the processing region.

  The feature amount calculation unit 208 calculates the feature amount of each image feature extracted by the feature extraction 206. The similarity calculation unit 209 compares the feature amounts calculated by the feature amount calculation unit 208 to obtain the similarity of each accumulated image.

  The similarity comparison unit 210 obtains an accumulated image including image features of a partial area of the key image using the similarity. The image sorting unit 211 sorts the stored images obtained by the similarity comparison unit 210 in order of similarity. The display unit 212 may be a monitor that displays the images sorted by the image sorting unit 211.

  The control unit 213 includes arithmetic means (for example, CPU: Central Processor Unit) that instructs the processing of each of the units 202 to 212.

  The similar area search apparatus 200 also includes an input unit such as a normal computer component, for example, a memory (RAM) for temporarily storing processing data of each of the units 202 to 212, a pointing device (such as a mouse), and a keyboard. ing. Hereinafter, the processing steps of the similar area search apparatus 200 will be described in detail with reference to the flowchart of FIG.

  S101: The image 201 input to the input unit 202 is selected as a key image to be searched and displayed on the display unit 212. Here, a key image may be selected from a plurality of input images 201.

  S102: When the key image is displayed in S101, the control unit 213 instructs the region selection unit 204 to start a partial region selection process. At this time, the area selection unit 204 prompts the user to select a partial area to be noted from the display image.

  Here, as shown in FIG. 3, the method of selecting a partial area is performed by designating a partial area to be noted with respect to the key image displayed on the display unit 212 by specifying a start point and an end point with a pointing device. . FIG. 3A shows an example of specifying a rectangle, FIG. 3B shows an example of specifying an ellipse, and FIG. 4 shows an example of specifying a circumscribed rectangle in the key image.

  At this time, a plurality of partial areas can be selected. In the realization system, a finite number of partial regions are selected, but the number of selection is theoretically unlimited. An example of selecting a plurality of areas is shown in FIG. Here, it is assumed that the partial area A indicates selection of an adjacent area of yellow and blue, the partial area B indicates selection of a yellow area, and the partial area C indicates selection of a blue area.

  The degree of attention can also be specified when selecting the partial area. Here, the degree of attention is assumed to be specified by a keyboard input as a numerical value (for example, 1 to 9) as a weighting coefficient. For example, in FIG. 5A, the partial area A is designated as “weight = 1”, the partial area B is designated as “weight = 2”, and the partial area C is designated as “weight = 1”. Yes.

  As the attention level, either a positive value or a negative value can be specified. As a method for specifying the positive value / negative value, a method of inputting plus / minus numbers or a method of inputting numbers of different colors can be used. FIG. 6A shows an example in which positive and negative are distinguished by the difference in the color of numbers. Here, a yellow numeral “2” is input to the partial area A of the key image X, a red numeral “1” is input to the partial area B, red indicates a positive value (plus), and yellow indicates a negative value (minus). ).

  A positive value designated as a partial area indicates that an image including an area similar to the partial area is searched from the data storage unit 205. On the other hand, a negative value indicates that an image including an area similar to the partial area is not searched from the data storage unit 205, that is, an area that does not include a negative partial area is searched.

  In other words, when specifying a normal partial area, it is common to specify an image including the desired area as a partial area to be specified for search. Here, a negative weight with a negative value is specified. Introducing positive weighting with positive values is important.

  S103: The control unit 213 asks the user whether or not all the partial areas to be noticed have been selected. If the answer “Yes” is confirmed, the process proceeds to S104. On the other hand, if the answer of “No” is confirmed, the process returns to S102 to allow the user to select a partial area again.

  S104: Based on an instruction from the control unit 213, the feature extraction unit 206 extracts image features such as color, texture, and moment from each partial area data in the key image.

  The feature amount of the image feature extracted here is calculated by the feature amount calculation unit 208. For example, each partial region is represented by an HSB (hue: Hue, saturation: Saturation, brightness: Brightness, or, Value) color model, and feature quantities relating to hue, saturation, and brightness can be calculated. Further, each partial region may be represented by an HSI color model.

  S105: Based on the instruction from the control unit 213, the feature extraction unit 206 refers to the data storage unit 205 and executes the process of S104 on an area of the same size as the partial area selected in S12 in each stored image. To extract image features.

  S106: The control unit 213 instructs the feature extraction unit 206 to extract image features of the entire range of each accumulated image. At this time, the region moving unit 207 performs the processing of S105 for each region while sequentially shifting the range of the partial region on each accumulated image by a predetermined distance n. This distance n may be an arbitrary value defined in a program or the like.

  S107: The control unit 213 confirms whether or not the processes of S104 to S106 have been performed on all the partial areas of the key image and all the areas of the accumulated images. If all the processes have been processed (Yes), the process proceeds to S108. On the other hand, if it is not processed (No), the process returns to S104 and the process is executed again. Note that all the calculated feature values are temporarily stored in a memory (RAM), and the processing after S108 is started.

  S108: Based on the instruction from the control unit 213, the similarity calculation unit 209 calculates the similarity by comparing the feature amount of each partial region in the key image with the image feature amount of each region in each accumulated image.

  Here, it is assumed that the similarity is a positive value, and the similarity is higher as the numerical value is larger. Further, the dissimilarity takes a positive value, and the smaller the numerical value, the more similar. Note that the same image can be treated as 100% completely similar, and the degree of similarity can be expressed as a percentage (%).

A processing example of similarity calculation will be described with reference to FIG. Here subregion selected from among key image X A and part regions B in the weighting factor W _A, W _B is designated. The weighting coefficient is positive for positive designation and negative for negative designation.

Further, as shown in FIG. 6 (b), the number of regions of the stored image Y to be compared with the partial areas A, as five in order to simplify the explanation, the region RA ₁ to RA _5. Similarly, the regions of the stored image Y that are the comparison targets of the partial region B are denoted by RB ₁ to RB ₅ .

Then, the region RA ₁ to RA ₅ for comparing the first partial region A (dis) similarity VA ₁ to VA _5, the region RB ₁ ~RB ₅ for comparing the partial region B (dis) similarity VB ₁ through Vb Calculate ₅ respectively.

Among these, the most similar (the one with the highest similarity or the smallest dissimilarity) is denoted by VA as the (non) similarity between the key image X and the stored image Y related to the partial area A. Similarly, the (non-) similarity VB for the partial region B is obtained. This similarity VA. Similarity S _Y combining VB is represented by image X. The similarity between Y is assumed. This total similarity S _Y is obtained by the following equation (1).

  Here, WA indicates the degree of attention of the partial area A, and WB indicates the degree of attention of the partial area B. If the degree of similarity is expressed as a percentage and WB is negatively specified, Expression (2) is obtained. Expressions (1) and (2) may be defined in advance in a program or the like.

The partial areas A. B similarity VA. VB, executes the processing of S109 and overall similarity S _Y temporarily stored in a memory (RAM).

  S109: Based on the control unit 213, the similarity comparison unit 210 compares the similarities. Based on the comparison result, the image sorting unit 211 arranges the stored images in the order of similarity (in descending order of similarity or in descending order of dissimilarity) and presents them to the user through the display unit 212.

  That is, the similarity comparison unit 209 compares the similarity between the key image and each stored image, and the image sorting unit 210 stores the stored image having the characteristics of the partial region closest to the key image (including the weights of the plurality of partial regions). The results are arranged on the display unit 212 in the order of similarity. At this time, the number of sheets to be displayed may be set in advance.

  As described above, according to the similar area search device 200, a search that emphasizes the area of interest of the user from the key image is executed, so that a search result that accurately reflects the user's search intention is obtained. Improves accuracy.

  In particular, the user's search intention can be reflected in the search result in detail by taking into consideration the attention level of the partial area and taking into consideration the attention level of the feature amount obtained from the partial area. Also, the user side can obtain an advantage that the degree of attention can be freely controlled by specifying the degree of attention. That is, when the weight of the partial area A is higher than the partial area B of the key image, the user always expects a search result including the elements of the partial area A and somewhat including the elements of the partial area B. Search results reflecting the user's intention can be presented.

  For example, according to the selection designation in FIG. 5A, the accumulated image including the partial areas A to C is searched from the data storage unit 205, but the weight of the yellow area of the partial area B is emphasized. The search results in 6 (b) reflect the weighting and are displayed in the order of images including many yellow areas.

  In addition, in the selection designation in FIG. 6A, the partial area A (weight = 2) indicates a blue area, and the partial area B (weight = −1) indicates a green area. In this case, it is important to include the blue region, and an image including the blue region and not including the green region is displayed in the search result of FIG. 6B.

  FIG. 7 shows a comparative example of negative designation and positive designation. Here, a comparison is made between partial area A when positive designation (weight = 1) and negative designation (weight = −1). FIG. 7A shows a case where the partial area A is positively designated. Here, the partial area A indicates a green area, the partial area B indicates a blue area, the partial area C indicates a white area, and the partial areas B.B. C is “weight = 1”. As a search result in this case, as shown in FIG. 7B, an image including blue, white, and green is displayed in the search result.

  On the other hand, in FIG. 7C, the partial area A is indicated by a broken line. Here, it is assumed that the partial area 1 is changed to negative designation. In the search result in this case, as shown in FIG. 7D, an image that does not include green but includes blue and white is displayed in the search result. In this way, by incorporating negative designation into the attention level of the partial area, it is possible to search for an image that does not include the negative designation area. Therefore, the user's intention can be narrowed down in detail by appropriately using negative designation and positive designation in combination.

  In addition, this invention is not limited to the said embodiment, For example, an apparatus structure etc. can be changed within the range described in each claim. The present invention can also be constructed as a program that causes a computer to function as the similar region search apparatus 200. This program may cause the computer to execute all steps of S101 to S109. Only some of the steps such as S109 may be executed.

  This program can be provided to a computer by downloading from a website or the like. The program is stored in a recording medium such as a CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, MO, HDD, Blu-ray Disk (registered trademark). It may be provided to a computer. Since the program code itself read from the recording medium realizes the processing of the above embodiment, the recording medium also constitutes the present invention.

The block diagram of the similar area | region search apparatus which concerns on embodiment of this invention. The flowchart of the processing step. (A) is a schematic diagram showing a rectangular designation example of a partial area, (b) is a schematic diagram showing an elliptic designation example of a partial area. Schematic which shows the example of designation | designated of the circumscribed rectangle in a key image. (A) is a figure which shows the example of multiple designation | designated of the weighted circumscribed rectangular area | region, (b) is a figure which shows the search result of (a). (A) is a figure which shows the example of designation | designated of the negative weighted circumscribed rectangular area on the image X, (b) is a figure which shows the comparison object partial area | region on the image Y. (A) is a comparison diagram showing a plurality of positive designation examples, (b) is a comparison diagram showing the search results of (a), (c) is a comparison diagram in which partial area A is changed to negative designation, and (d) is ( The comparison figure which shows the search result of c).

Explanation of symbols

DESCRIPTION OF SYMBOLS 200 ... Similar area search apparatus 201 ... Input image 202 ... Input part 203 ... Area selection part 204 ... Feature extraction part 205 ... Data storage part 206 ... Feature extraction part 207 ... Area movement part 208 ... Feature-value calculation part 209 ... Similarity calculation Unit 210 ... Similarity comparison unit 211 ... Image sorting unit 212 ... Display unit 213 ... Control unit

Claims (9)

A method for searching for an image including a similar partial area in a target image to be searched from stored images in a database,
A first step of causing the region selection means to select a partial region to be noticed by the user from the inputted target image;
A second step in which feature extraction means extracts image features of a partial area from the target image, and extracts an image feature for each area having the same size as the partial area of the target image for each stored image;
A third step of calculating a similarity by comparing the image feature of the partial region in the target image with the image feature of each region in each accumulated image;
A fourth step in which the image sorting means presents to the user the search results sorted in the order of similarity calculated in the third step;
A similar region search method comprising: The second step extracts the image features for each partial area when a plurality of partial areas are selected,
The similar region search method according to claim 1, wherein the third step compares an image feature of each partial region with each image feature of each region. In the third step, when the degree of attention is specified stepwise for each selected partial region,
The similar region search method according to claim 2, wherein the attention level is reflected in the similarity. The third step causes the comparison result of the image features to act in a similar direction when a positive value is designated for the degree of attention,
The similar region search method according to claim 3, wherein the comparison result of the image features is applied in a dissimilar direction when a negative value is designated for the degree of attention. An apparatus for searching for an image including a similar partial area in a target image to be searched from stored images in a database,
A region selecting means for allowing the user to select a partial region to be noted from the input target image;
Feature extraction means for extracting image features of a partial area selected from the target image, and extracting an image feature for each of the accumulated images for each area having the same size as the partial area of the target image;
Similarity calculation means for calculating the similarity by comparing the image feature of the partial region in the target image and the image feature of each region in each accumulated image;
Image sorting means for presenting to the user the search results sorted in the calculated order of similarity;
A similar region search device comprising: The region selection means is set so that a user can select a plurality of partial regions from the target image,
The feature extraction means extracts the image features for each of a plurality of selected partial areas,
6. The similar region search apparatus according to claim 5, wherein the similarity calculation unit compares the image feature of each partial region with each image feature of each region. The area selection means is set so that the degree of attention can be specified in stages for each of the selected partial areas.
3. The similar region search method according to claim 2, wherein the similarity calculation unit reflects the attention level in the similarity. The region selecting means is set so that either a positive value or a negative value can be specified for the stepwise attention degree,
The similarity calculation means causes the comparison result of the image features to act in a similar direction when a positive value is designated for the attention level,
The similar region search device according to claim 7, wherein the comparison result of the image features is applied in a dissimilar direction when a negative value is designated for the attention degree.   A similar region search program that causes a computer to function as the similar region search device according to claim 5.

Images