JPWO2017168601A1 - Similar image retrieval method and system - Google Patents

Abstract

In the prior art, when the area of the partial region of the query image is relatively small, a step of detecting a plurality of partial regions included in the query image in which a large number of images that are out of search intent are hit and detected A step of extracting a plurality of feature amounts of partial areas, a plurality of extracted feature amounts and feature quantities of image partial regions stored in advance in the DB are respectively associated with each other, and the similarity of the associated feature amounts is respectively determined. A step of calculating, a step of weighting each calculated similarity according to the area of each partial region included in the query image, and a sum of values obtained by weighting the similarity of each partial region And calculating a similarity between the query image and the search target image.

Description

  The present invention relates to an information search technique for an image.

  In recent years, the broadband network and the scale of various storage devices have made it possible to store, manage and distribute large-scale images and videos.

  Search technology is important in such a system that handles large-scale contents. A general search technique is a search technique for attached text information related to image / video content. In the document information search technology, one or more keywords are inputted as a query, and an image / video associated with text information including the keyword is generally returned as a search result. There is also a technique for extracting and retrieving information from the image itself. As described in Patent Literature 1, Patent Literature 2, and the like, in similar image retrieval, an image feature value obtained by quantifying the characteristics of an image is extracted in advance from an image to be retrieved, and a database is created. To achieve high-speed search.

  In Patent Literature 1, a region similarity indicating a region similarity is calculated for each combination of a plurality of regions for dividing a search target image and a plurality of regions for dividing a query image, and the region similarity is calculated for each region in a query image. The importance of the region is calculated based on the region similarity corresponding to the region, and for each search target image, based on the region similarity and importance corresponding to the combination of each region in the search target image and each region in the query image A configuration for calculating an image similarity indicating similarity to a query image is disclosed.

JP 2013-254367 A

  In the technique disclosed in Patent Document 1, when the area of the partial region of the query image is relatively small, a large number of images that are out of search intention are hit.

  In order to solve the above problems, the present invention includes a step of detecting a plurality of partial regions included in a query image, a step of extracting a plurality of feature amounts of the detected partial regions, a plurality of extracted feature amounts and a DB. A plurality of feature amounts of image partial regions stored in advance, a step of calculating the similarity of the associated feature amounts, and each portion included in the query image according to the calculated similarity A method comprising weighting according to the area of the region, and calculating a similarity between the query image and the search target image based on a total value of values obtained by weighting the similarity of each partial region. I will provide a.

  According to the present invention, when the area of the partial region of the query image is relatively small, the weight is reduced, and as a result, it is possible to reduce the possibility of hitting an image that is out of the search intention.

1 is a system configuration diagram of an embodiment of the present invention. It is a detailed view of the system configuration of the embodiment of the present invention. It is explanatory drawing of the attention area detection technique used in the Example of this invention. It is explanatory drawing of the symmetry axis in the attention area detection technique used in the Example of this invention. It is an example of a differential filter. It is explanatory drawing of a luminance gradient intensity distribution feature-value. It is a figure which shows the flow of the attention area detection process in the Example of this invention. It is a figure explaining the detail of an attention area detection process. It is explanatory drawing of the detailed process in an attention area detection process. It is a schematic diagram for showing an attention area detection result. It is a list of items that are made into a database regarding partial areas. It is a flow of similar search processing. It is a schematic diagram for showing the result of the similarity search regarding the partial area. It is a schematic diagram for showing the result of performing the association of partial areas by method 1. It is a schematic diagram for showing the result of performing the association of the partial area by the method 2. It is a system configuration | structure figure of the system which implement | achieves the service of Example 2. FIG. It is the list of the items database-ized regarding an image in the service of Example 2. FIG. FIG. 10 is a schematic diagram of a screen configuration of an application according to a second embodiment.

  FIG. 1 shows a system configuration when the method of the present invention is applied to an image search service. Reference numeral 100 in FIG. 1 denotes a computer system for providing a search service. Various functions provided by the search service are provided to a user who uses the terminal computer 120 via the network system 110.

  FIG. 2 shows an internal configuration of the computer system 100 for providing a search service. The processing flow of the image search system targeted by the present invention will be described with reference to FIG.

  First, a process for registering an image will be described. When registering an image, after the registered image 210 to be searched is received by the image registration unit 211, the focused region detection unit 212 detects a set of partial regions to be focused on from the registered image 210. Details of the attention area detection processing will be described later with reference to FIG. An image feature quantity is extracted by the search feature quantity extraction unit 213 for each detected partial area. The extracted image feature amount is stored in the database 220 in a format associated with the partial area.

  Next, processing when searching for an image will be described. When searching for an image, after the query image receiving unit 231 receives the given query image 230, the focused region detection unit 232 extracts a set of partial regions to be focused from the query image 230. Details of the attention area detection process will be described later with reference to FIG. Next, an image feature amount is extracted by the search feature amount extraction unit 213 for each detected partial area. In FIG. 2, the attention area detection unit 212 and the attention area detection unit 232 are described separately for convenience of explanation, but these may be the same processing unit in the computer. The same applies to the search feature quantity extraction unit 213 and the search feature quantity extraction unit 233.

  The similarity search unit 234 is applied with the method of the present invention. The similarity search unit 234 forms a result of the similar image search from the set of image feature amounts of the partial areas of the query image 230 and the set of image feature amounts of the partial areas of the registered image 210 stored in the database 220. Finally, the search result output unit 235 generates information to be returned to the search request source using the similar image search result and various types of information stored in the database 220, and transmits the information to the search request source as the search result 240. To do.

  In the present embodiment, in the attention area detection unit 212 of the registered image 210 and the attention area detection unit 222 of the query image 230, a detection method using local symmetry is used as a detection method of the attention region that does not limit the target. Yes. Details of the detection method will be described later with reference to FIG.

  FIG. 3 is an explanatory diagram relating to extraction of image feature amounts for evaluating local symmetry. An arbitrary rectangular partial area 302 in the image 301 is further divided into 3 × 3 blocks to extract an image feature vector group 303 of each block area. . These nine image feature quantity vectors are denoted by f00, f10, f20, f01, f11, f21, f02, f12, and f22. Here, the first subscript is the position in the x direction of each block, and the second subscript is the position in the y direction.

  FIG. 4 is a diagram showing an axis of symmetry in the attention area detection technique. Consider a mirror transformation centered on four axes for the image feature vector of each block. Here, T0, T1, T3, and T4 represent matrices for applying the mirror transformation about each axis to the image feature quantity vector. That is, T0 is a matrix for mirror conversion to the left and right, T1 is a matrix for mirror conversion about the axis at the upper right 45 degrees, and T3 is a matrix for mirror conversion up and down. T4 is a matrix for mirror conversion about the axis at the lower right 45 degrees.

  By applying the above transformation matrix to the image feature quantity vector extracted in each block area, the symmetry in the rectangular partial area can be evaluated. For example, in order to evaluate the left-right symmetry, f00 and f20 existing symmetrically with respect to the y axis are vectors obtained by mirror-transforming f20 left and right, that is, a vector obtained by multiplying f20 by T0 is f00. The closer it is, the higher the symmetry. Similarly, for f01 and f21, if f21 is multiplied by T0 and f02 and f22 are close to f22 multiplied by T0, the symmetry is considered high. In this way, the left-right symmetry can be expressed as a vector D0 composed of the three vector square distances as shown in [Equation 1].

  Similarly, the symmetry about the axis at the upper right 45 degrees is expressed as D1 in [Equation 2].

  Similarly, the vertical symmetry is expressed as D2 in [Equation 3].

  Similarly, the symmetry about the axis at the lower right 45 degrees is expressed as D3 in [Equation 4].

  On the other hand, in this detection method, the symmetry is evaluated to be high when the symmetry increases due to the transformation of the feature vector. For example, when f00 and f02 used in the calculation of D0 originally have a small variation with respect to the left and right reflection conversion, it is not considered that the left-right symmetry is large. As a correction term that quantitatively expresses such a property, it is composed of a square distance between feature quantity vectors between corresponding block areas when mirror conversion is not applied, as shown in [Formula 5]. Define the vector E0.

  Similarly, the correction term E1 for D1 is expressed as the following [Equation 6].

  Similarly, the correction term E2 for D2 is expressed as the following [Equation 7].

  Similarly, the correction term E3 for D3 is expressed as the following [Equation 8].

  D0, D1, D2, D3, and E0, E1, E2, E3 are used to evaluate the symmetry of the rectangular partial area. [Equation 9] is used as a specific evaluation function.

  The above evaluation function uses the maximum value after evaluating the symmetry in each of the four directions.

  Next, the image feature amount based on the intensity distribution of the luminance gradient vector used in this method will be described.

  FIG. 5 is an example of a filter for performing numerical differentiation. The luminance gradient vector is derived by applying a two-dimensional numerical differentiation to the black and white gray image. From the luminance gradient vector (gx, gy) on the pixel position (x, y) obtained by the differential filter, the vector direction θ and the square norm p of the vector can be calculated as in [Equation 10]. .

  The vector direction θ is distributed in the range of 0 to 360 degrees. By quantizing this to an appropriate level at equal intervals and summing the square norm p within the rectangular area, the intensity distribution in the direction of the luminance gradient vector can be expressed as histogram-like data.

  FIG. 6 is a schematic diagram expressing the concept of calculating the luminance gradient intensity distribution feature amount. First, the brightness gradient vector 601 is extracted from the image. Next, histogram-like data 602 is calculated by aggregation processing. In this embodiment, the number of quantization levels is 8. Also, the center of the first quantization range is made to coincide with the x-axis direction.

  If the number of quantization levels is 8 in this feature quantity, an 8-dimensional image feature quantity vector is extracted from each block area. The mirror transformation matrix T0 for evaluating the left-right symmetry at this time is as shown in [Equation 11].

  Similarly, the mirror transformation matrix T1 for evaluating the symmetry about the axis in the upper right 45 degree direction is as shown in [Equation 12].

  Similarly, the reflection transformation matrix T2 for evaluating the vertical symmetry is as shown in [Equation 13].

  Similarly, the reflection transformation matrix T3 for evaluating the symmetry about the axis in the lower right 45 degree direction is as shown in [Expression 14].

  Next, processing of the attention area detection unit 212 and the attention area detection unit 232 will be described with reference to FIG.

  FIG. 7 is a diagram illustrating a flow of processing of the region of interest detection unit.

  S701 is processing for generating a plurality of images converted to an appropriate scale and aspect from the input image.

  S702 is a process for converting the plurality of generated images into multiple resolutions.

  S703 is processing for generating a rectangular partial region that is a candidate for the target partial region by scanning processing on the multi-resolution image and calculating the symmetry of each partial region. Details of S701 to 703 will be described later with reference to FIG.

  In step S704, a large number of partial areas generated by the scanning process are sorted based on the evaluation value of symmetry, and the partial area candidates to be focused on are narrowed down to a predetermined upper number.

  S705 is processing for determining convergence.

  In S706, when it is determined in S705 that the convergence does not occur, the refinement process newly generates a partial area from the current focused area candidate, and calculates the symmetry of each partial area, whereby the focused area candidate Is a process of adding. Details of S706 will be described later with reference to FIG. Narrowing based on symmetry evaluation is performed again on the candidate region of interest configured in this way (S704). If there is no change in the region of interest in the convergence determination 705, it is determined that the target area has converged, and the process is terminated. It should be noted that the process may be terminated by determining that the process has converged when the number of repetitions of S704 to S706 exceeds a certain number.

  FIG. 8 illustrates S701 to S703 in FIG.

  In the detection of the region of interest, it is necessary to appropriately estimate the size of the partial region in the image according to the application field. In particular, if an area that is smaller than necessary is included as a partial area, unnecessary partial areas are detected. As a result, not only erroneous detection increases, but also processing time increases. For example, if the size of one block in the symmetry evaluation of this method is 8 × 8 pixels, the size of the partial area is 24 × 24 pixels. If the partial area to be focused on is sufficient up to about 10% of the size of the image, the size of the image is about 240 × 240 pixels.

  In addition, the shape of the target partial region is not necessarily limited to a square, and it is often necessary to extract a horizontally long and vertically long rectangular region. In this embodiment, when it is necessary to extract a horizontally long rectangle, the aspect of the original image is deformed to be vertically long, and the symmetry is evaluated by square lattice block division. If the rectangular area generated by such processing is returned to the coordinate system on the original image, a horizontally long rectangle is obtained. Similarly, when it is necessary to extract a vertically long rectangle, the aspect of the original image is transformed into a horizontally long process.

  Reference numeral 802 denotes processing performed from the above-described two viewpoints. An image obtained by scaling the input image 801 after reducing the image width to half, an image obtained by scaling conversion while maintaining the aspect ratio, an image obtained by scaling the image after reducing the image height by half, Three images are shown being generated.

  Reference numeral 803 denotes multi-resolution processing. In this process, an image is generated by reducing each image by half to two stages.

  Reference numeral 810 denotes scanning processing for the nine images generated in this way. In this process, a rectangular area is roughly generated by translating a rectangular window in each image by a certain number of pixels. FIG. 9 illustrates S706 of FIG.

  In S706, a rectangular region 910 that has been translated slightly vertically and horizontally, a rectangular region 920 that has been slightly enlarged / reduced, and a rectangular region that has been enlarged / reduced are further translated horizontally and vertically with respect to a candidate of a target partial region. A rectangular area is generated as a candidate for a new target partial area. The number of rectangular areas generated by the parallel movement is eight patterns in the up / down, left / right, and diagonal movements. There are two rectangular areas generated by the enlargement / reduction, and eight patterns of rectangular areas are generated for each of the enlargement / reduction by parallel movement of the rectangular areas. In addition, a maximum of 26 new rectangular areas are generated for a single rectangular area, and the symmetry is evaluated.

  As described above, the detailing process 706 is repeatedly executed. Each minute variation amount in each iteration is defined by [Equation 15].

  Here, q is the number of repetitions of the detailing process, sx and sy are the step widths in the horizontal and vertical directions when the translation is performed in the scanning process (S703), respectively, and dx and dy are the qth time, respectively. The amount of fluctuation in the horizontal and vertical directions in the detailed processing. On the other hand, dz is an enlargement ratio in the q-th refinement process, and the reduction ratio when reducing is 1 / dz. As is clear from the above equation, the magnitude of the fluctuation decreases with the number of repetitions of this process. Since the target image is a discrete digital image, if this process is sufficiently repeated, new region candidates will not be generated due to minute fluctuations. If at least a new area candidate is not generated, S704 to S706 are determined to be converged in S705 and are terminated.

  FIG. 10 schematically shows partial region detection results in the registered region of interest detection unit 212 and the query image of interest region detection unit 232. Various partial areas having different sizes are detected from the image 1010. Reference numeral 1020 denotes a set of the respective partial areas. The partial area set 1020 includes a partial area 1021 having a small area and a partial area 1022 having a large area. These will be described later in the description of Expression 18 in the description of FIG.

  Note that an area of the entire original image may be added as one of the partial areas. Thereby, it is possible to perform a search in consideration of the similarity of the entire image.

  Image feature amounts used in the search feature amount extraction unit 213 of the registered image 210 and the search feature amount extraction unit 233 of the query image 230 are calculated based on color distribution, luminance gradient vector distribution, and the like. Specific definition examples of image feature amounts are disclosed in Non-Patent Document 1, Non-Patent Document 3, and the like.

  Information regarding the partial area extracted from the registered image 210 in this way is stored in the database 220.

  FIG. 11 lists items related to the similarity search processing in the present embodiment, among the information stored for each partial area.

  An item 1101 is information for specifying which image each partial area is included in, and is expressed as an integer image ID. An item 1102 is a coordinate value of each rectangular partial area, and stores the coordinate values of the upper left end point and the lower right end point as four integer values. Item 1103 is the ratio of the area occupied by each partial region in the original image. For example, when the partial area matches the original image, the maximum value is 1.0. An item 1004 indicates an image feature amount expressing shape information, and an item 1005 indicates an image feature amount expressing color distribution information. In this embodiment, these two types of feature values are used, but one type may be used, or three or more types of feature values may be set.

  Note that information related to the query image is also extracted as shown in FIG. The information about the query image is temporarily stored in the memory, and the application of this embodiment can be operated. However, when it is assumed that a search request with different search conditions occurs a plurality of times using the same query image, it is inefficient to analyze the query image every time. Therefore, it is desirable that the information about the query image is cached beyond one search process. On the other hand, when considering a large-scale operation in which a large number of search requests are received at the same time, if the cache is performed on the memory, memory consumption increases, which is not appropriate. In such a case, it is preferable to adopt a format for temporarily storing information on the database for the query image.

  In this embodiment, an example in which the partial area is defined as a rectangular area will be described. However, in the method shown in FIG. 12, the shape of the partial area does not have to be rectangular. In this case, the requirements of the partial area are that the image feature amount can be extracted and that the area of the area is defined.

  FIG. 12 shows the flow of processing in the similarity search unit 234. S1201 is a process in which the similarity search unit 234 executes a similarity search for the partial region feature amount database. In this process, a similar search is performed by using the feature amount of each partial area detected from the query image as many as the number of partial areas of the query image.

  S1202 is a process of associating the partial area of the query image with the partial area of each image based on the set of search results acquired in S1201. Details will be described later with reference to FIG.

  S1203 is a process of configuring the distance between the query image and each image based on the result of association in process 1202.

  S1204 is a process of constructing a similar image search result by sorting the distances constructed in process 1203.

  S1205 is a process of delivering the result to the search result output unit 235.

  In S1201, a similarity search result is performed using the square distance between the feature amounts between the partial region of the query image and the partial region on the database as an index of dissimilarity. The square distance in the feature amount space between the i-th rectangular area of the query image and the j-th partial area stored in the database is as shown in [Equation 16].

  Here, q_ik is the k-th element of the image feature quantity of the i-th partial area, f_jk is the k-th element of the image feature quantity of the j-th partial area on the database, and M is the dimension of the image feature quantity. Is a number.

  The square distance is not calculated for all partial areas on the database. In this embodiment, an approximate neighborhood vector search algorithm based on clustering processing is used to speed up the similarity search. The approximate neighborhood vector search is a method for realizing a high-speed search by narrowing down vectors that are assumed to exist in the neighborhood range of a vector to be queried from vectors on a database. The algorithm used in the present embodiment returns the approximate solution of N vectors in the vicinity of the query as a search result in a form sorted in the order of decreasing square distance. Therefore, the value of the square distance to the partial area on the database not included in the N results is unknown at that time.

  In the method of the present invention, the similarity between the query image and the image stored on the database is constructed from the result of the similarity search for the partial region. This configuration is realized by associating the partial area of the query image with the partial area of each image on the database, and summing up the similarity for each partial area for each image. Hereinafter, the specific method will be described with reference to FIG.

  FIG. 13 is a schematic diagram showing the result of a similarity search for a partial region. If the number of partial areas detected from the query image is L and the number of similar search results for each partial area is N, L × N similar search results are acquired in 1201 of S described above. In order to make the figure easy to understand, L = 4 and N = 5 are set. However, in practice, a larger number of search results are often acquired for a partial region of a larger number of query images. Each result 1300 of the similarity search in this embodiment is composed of the calculated square distance, the ID of the partial area, and the ID of the image including the partial area. As for the image ID, after performing a similar search, the value is acquired by referring to the information on the database using the partial region ID. Each row of the table in FIG. 13 is a result of a similar search using each partial region of the query image as a query. The search results in each row are arranged in the order of small square distance, that is, the order of high similarity, regardless of which image contains the partial area on the database.

  In S1202, the query image partial area is associated with the partial area of each image on the database from the set of similar search results. In this embodiment, the following two methods are provided as association methods, and the system operator or the user to search selects either.

  In the first method, for each partial region of the query image, a search result having the smallest distance in the set of partial region search results included in each image on the database is employed. Search results for different partial areas of the query image may include partial areas on the same database. Therefore, in the first method, there is a possibility that different partial areas of the query image are associated with partial areas on the same database. Details of the first method will be described with reference to FIG.

  FIG. 14 shows the result of association performed by the first method from the search result shown in FIG. Since the search results include the image IDs 1, 2, 5, 6, and 7, the search results that are most similar to the partial regions of the query image are selected for the five images. .

  Unlike the first method, the second method does not associate multiple partial areas on the same database. In the second method, for each image on the database, a set of search results from all partial regions of the query image is configured, and the set is sorted in ascending order of the square distance, and then handled in order from the top. To do. At this time, by holding the ID of the partial area on the database adopted for the association, the partial area already adopted for the association is excluded and the process proceeds. In this manner, it is possible to associate the query images with a small square distance, that is, a combination of the query image partial area and the image partial area on the database without priority overlap. Details of the second method will be described with reference to FIG.

  FIG. 15 shows the result of association performed by the second method from the search results shown in FIG. For example, in the association of the partial areas included in the image whose image ID of the third partial area is 1, the partial area whose ID of the partial area whose similarity is higher is 2 is the first partial area Is associated with the partial area whose image ID is 3 in the lower partial area. In the fourth partial area, since there is no corresponding partial area, the image with the image ID of 1 is not associated.

  When attention is paid to an image on a certain database, it is not general that association is performed for all partial areas of the query image. As described above, the partial region search results do not include evaluation results for all partial regions. In addition, since the number of partial areas to be detected varies depending on the image, in the second method in which overlapping association is not performed, all associations are not performed in principle. When the similarity between the query image and each image is configured from the similarity for each partial area, if the similarity for each partial area has been determined, an image with a high sum of similarities for each partial area is displayed as an image. The natural assumption that the degree of similarity is high also holds. When the similarity for each partial area is partially indefinite and the number of indefinite partial areas differs between images, the handling of the indefinite partial area becomes a problem.

  It is reasonable to estimate that data that is not included in the search result when a similar search is performed is lower in similarity than data having the lowest similarity in the search result. Therefore, in this embodiment, the value of the square distance of the lowest search result in the search results of each partial area is set for the indefinite partial area. In the case of the similar search result shown in FIG. 13, when the first partial area is indefinite, the square distance 0.80, and when the second partial area is indefinite, the square distance 0.60 is set. When the area is indefinite, the square distance 0.50 is set, and when the fourth partial area is indefinite, the square distance 0.70 is set as the square distance when each area is indefinite.

  As another setting method, there is a method in which a statistical amount of square distance, for example, an average and a variance is obtained statistically on a sufficiently large database, and a value based on the statistical amount is set.

  Through the above series of processes, the square distance between each partial area included in the query image and the image on the database can be defined as the square distance of the associated partial area. The distance between the query image and the image on the database is defined as [Equation 17] using the square distance between the partial region of the query image defined in this way and the image on the database.

  Here, D_i is a combined square distance between the query image and the i-th image in the database, L is the number of partial regions included in the query image, and d_ji is the j-th partial region of the query image and i The square distance from the second image, S_j is the ratio of the area of the jth partial region to the area of the query image, and P is a parameter that controls the effect of S_j. On the other hand, Z is a normalization term defined as:

  This normalization term is intended to make it easier to handle the value of D_i on the application side, and does not affect the search process itself.

  The control parameter P in Equation 17 performs the following function.

  When P = 0.0, all d_jis are weighted equally to calculate D_i. If P is set to a large value, D_i is calculated with an emphasis on the value of d_ji of the partial region having a large area ratio S_j. In the example of the partial region detection result 1020 in FIG. 10, the similarity of the partial region 1021 having a small area is evaluated low, and the similarity of the 1022 partial region having a large area is emphasized.

  In this embodiment, the square distance, which is a dissimilarity, is used for comparing the similarity. However, a similarity that increases when the similarity is high may be used. As an example of the similarity, for example, a negative square distance such as [Equation 19] is exponentially converted.

  The similarity in Equation 19 reaches a maximum value of 1.0 when the square distance d is 0.0, and approaches 0.0 when d increases. Even when such a similarity is used, the method shown in Equation 16 can be applied. That is, the similarity between images is basically calculated as the sum of the similarities between partial areas. If there are many similarities between partial areas, the similarity between images increases. The effect of the weighted addition based on the area of the partial region of the query image is not different from that in the case of Expression 17.

  For all the images on the database including the partial area included in the similar search result for the partial area, after calculating the above D_i, the result of the similar image search is configured by sorting by the value of D_i.

  In the above description, the case where an image is given as a search query has been described, but an image already registered on the database may be searched as a query. In this case, since the information shown in FIG. 11 can be acquired from the database, the analysis of the query image becomes unnecessary. Therefore, the process can be started from the similarity search unit 234 of FIG.

  FIG. 16 is a system configuration diagram of the search service described in this embodiment. This service provides the user with the functions of the search system 1600 through the WebAPI 1602. An appropriate calculation resource such as the number of CPUs, the amount of memory, and the amount of hard disk is allocated to the hardware configuration of the search system 1600 according to the scale of the search target and the frequency of search requests. Various programs running on this system exchange information by inter-process communication.

  The user issues a search request using the Web browser 1601. The search request is received by the WebAPI 1602 of this service, and the result is returned to the Web browser 1601 as a response of the WebAPI 1602. Processing 231, 232, 233, 234, and 235 related to the search in FIG. 2 is performed on the Web server 1603. The Web server 1603 sends the received query image analysis result to the temporary DB server 1604, and the temporary DB server 1604 stores the contents on the file system. As for the temporary DB server 1604, a plurality of processes can be operated if redundancy is required for stable operation of the system. When a plurality of temporary DB server processes are operating, the Web server 1603 can select a temporary DB server with the smallest load as a registration destination.

  Of the processes performed by the similarity search processing unit 234 in FIG. 2, the process corresponding to the “similar search for a partial region feature DB” 1201 in FIG. 12 is performed not by the Web server 1603 but by the search server 1605. When large-scale data is targeted, parallel distributed processing is performed by operating a plurality of search servers 1605.

  Processing corresponding to the processing 211, 212, and 213 related to image registration in FIG. 2 is performed on the data registration program 1606. The data registration program 1606 sends the analysis result of the acquired registered image to the search server 1605, and the search server 1605 stores the contents on the file system.

  In the search server 1605 and the temporary DB server 1604, the information managed regarding the partial area is the same as that shown in FIG.

  FIG. 17 is a list of items managed on the database for each image. An item 1701 is an array of partial area IDs included in the image, and is managed as a variable-length integer array. An item 1702 is the size of an image, and the number of vertical and horizontal pixels is stored as two integer values. An item 1703 is the date when the image is registered, and is managed as an integer value. An item 1704 is a keyword for performing a narrowing search. Items 1703 and 1704 are not used in the temporary DB server 1604. An item 1705 is image data, and an item 1706 is a thumbnail image, both of which are used to display a search result screen.

  In the present embodiment, not only a simple similar image search but also a similar image search function combined with the search date 1703 and the search target narrowing by the keyword 1704 is provided. This function first narrows down the search target of the partial area by referring to the array 1701 of the partial area ID after narrowing down the target image by evaluating the search condition formula. As a result, an efficient search is possible.

  FIG. 18 is a schematic diagram of a search screen displayed on the Web browser 1601.

  1810 in FIG. 18 is a screen for setting search conditions. Reference numeral 1811 denotes an area for setting a query image. The user sets a query for similarity search by placing an arbitrary image file existing on his file system in the area 1811 on the screen by a drag-and-drop operation. To do. When the user clicks the search button 1812, a screen transition occurs and a search result screen 1820 is displayed. On the search result screen 1820, the query image specified in 1811 is converted into a thumbnail image and displayed in 1821, and a list of similar search results is displayed in 1822. A row of buttons 1823 is for page-turning, and is for browsing an image having a lower rank, that is, a lower similarity. If an image displayed in 1822 is clicked, a similarity search using the clicked image as a query is executed, and the content of the search result screen 1820 is updated. This makes it possible to perform a similar search using an image registered on the database as a query.

  A reference numeral 1813 on the search condition setting screen 1810 is a GUI component for narrowing down the search target image by the registered date. The user can input the registration date range by two numerical values representing the lower limit and the upper limit. If there is no input, narrowing down by registration date is not performed. For example, when only the lower limit is set, the search target is from the input date and after to the latest registered image. Reference numeral 1814 denotes a GUI component for setting a narrow-down condition based on a keyword. The user can input an arbitrary character string into the text field. If there is no input, filtering by keyword is not performed. When both the condition based on the registration date and the condition based on the keyword are set, the logical product (AND) is a narrowing condition.

  A GUI component 1815 composed of two toggle buttons is for selecting an image feature amount used for similarity search. The user checks the button labeled “SHAPE” if he / she wants to perform a search that emphasizes the similarity of shapes, and if the user wishes to perform a search which emphasizes the similarity of colors, Check the attached button.

  The slide bar 1816 is a GUI component for designating the value of the parameter P that controls the effect of the area of the partial region shown in Expression 16. In the present embodiment, the user can freely set the value in the range of 0.0 to 1.0. When the user wants to search for an image that partially includes the query image, the user sets a value close to 1.0. If you want to search for an image containing a part that is common to the part included in the query image, set a value close to 0.0.

  In this embodiment, a slide bar 1816 suitable for specifying a continuous amount is used as a GUI component for realizing this function. However, what can be actually specified is a value quantized to an appropriate level. It is. When the value of P changes, the result of the similarity search changes. The Web server 1603 caches search results under the same search condition in order to cope with screen transitions associated with the operation of the page switching button 1823. When the search condition changes, it is necessary to generate a new cache, and it is not efficient in terms of system operation to generate a cache following a subtle change in P that has no essential effect. In this embodiment, the similarity search process is performed after appropriately quantizing the value of P received by WebAPI. In addition, since it is premised that quantization processing is performed, there is no problem even if the GUI component is not a slide bar but a method such as selection of a value on a button row corresponding to a quantized value.

In the area 1824 of the search result screen 1820, GUI parts similar to those in the search condition setting screen 1810 1813, 1814, 1815, and 1816 are arranged. With these GUI parts, the user can perform a search by changing only other search conditions without changing the query image. This search is executed when the search button 1825 is clicked, and the contents of the search result screen 1820 are updated.

100: computer system 110: network 120: terminal computer 210: registered image 211: image registration unit 212: attention area detection unit 213: search feature extraction unit 220: database 230: query image 231: query image reception unit 232: attention Region detection unit 233: Search feature amount extraction unit 234: Similarity search unit 235: Search result output unit 240: Search result 301: Detection target image 302: Rectangular partial region 303: Image feature amount for each block 601: Brightness gradient vector 602 : Intensity distribution of luminance gradient vector 801: Input image 802: Scale and aspect conversion 803: Multi-resolution 810: Scan processing 910: Translation 920: Enlargement / reduction 1010: Image 1020: Partial area detection result 1021: Small area Example of area 1022: Example of partial area with large area 1101: Image ID 1102: Rectangle coordinate value 1103: Relative area of rectangle 1104: Image feature (shape) 1105: Image feature (color) 1600: Search system 1601: Web browser 1602: Web API 1 603: Web server 1604: Temporary DB server 1605: Search server 1606: Data registration program 1701: Partial area ID array 1702: Image size 1703: Registration date 1704: Keyword for narrowing down 1705: Image data 1706: Thumbnail image 1810: Search condition setting screen 1811: Query image display area 1812: Search button 1813: Registration date range specification 1814: Keyword specification 1815: Image feature selection 1816: Parameter setting to control the effect of the area of the partial area 1920: Search result Display screen 1921: Query image display area 1922: Search result display area 1923: Page transition button 1924: Search condition specification 1925: Search button

Claims (5)

A first step in which the region of interest detection unit detects a plurality of partial regions included in the query image;
A second step in which the feature amount extraction unit extracts a plurality of feature amounts of the detected partial areas;
A third step in which the similarity search unit associates the extracted feature quantities with the feature quantities of the image partial areas stored in advance in the DB, and calculates the similarity of the associated feature quantities, respectively. When,
A fourth step in which the similarity search unit weights each calculated similarity according to the area of each partial region included in the query image;
A fifth step in which the similarity search unit calculates the similarity between the query image and the search target image based on a total value of values obtained by weighting the similarity of each partial region;
A search result output unit that outputs a search result based on the similarity calculated in the fifth step to the display unit;
A similar image search method comprising: The similar image search method according to claim 1,
The first step includes a process of multi-resolutioning the query image, a process of generating a plurality of partial areas, and a process of narrowing down the generated partial areas based on symmetry evaluation. Similar image search method. The similar image search method according to claim 2,
The first step is a process of generating a detailed partial area and adding it to the object of the symmetry evaluation when the number of the partial areas narrowed down based on the symmetry evaluation is a predetermined value or more. Including
The similar image search method according to claim 2, wherein the second step is performed when the number of the plurality of partial areas narrowed down based on the symmetry evaluation is less than a predetermined value. The similar image search method according to claim 3,
The similar image search method according to claim 2, wherein the second step is performed when the number of narrowing trials based on the symmetry evaluation becomes a predetermined value or more. A region of interest detection unit that detects a plurality of partial regions included in the query image;
A feature quantity extraction unit that extracts a plurality of feature quantities of the detected partial area;
The extracted feature quantities are associated with the feature quantities of the image partial areas stored in advance in the DB, respectively, the similarity of the associated feature quantities is calculated, respectively, A similarity search unit that performs weighting according to the area of each partial region included in the query image, and calculates the similarity between the query image and the search target image based on a total value of the weighted values;
A search result output unit that outputs a search result based on the calculated similarity to the display unit;
A similar image search system comprising:

Images