JP5138790B2 - Image search apparatus and image search method - Google Patents

Description

  The present invention relates to an image search apparatus and an image search method that can quickly extract a desired image from a large-capacity image group using multidimensional feature data (face, color, etc.) extracted from an image. .

  In recent years, with the increase of crimes represented by picking, robbery, arson, etc., the spread of video surveillance systems for preventing crimes by installing cameras, sensors, storage devices, etc. is greatly advanced. In addition, with the increase in the capacity of surveillance camera IP and storage devices, systems that perform wide-area monitoring and long-time recording on the scale of several hundreds have increased. Under such circumstances, there is a demand for a technique for efficiently washing a specific person such as a shoplifter, a lost child, a lost person, etc. with the aim of reducing the work of a supervisor.

  As a conventional technique for identifying a specific person at high speed, a multi-dimensional feature data group (color, face, etc.) extracted from an image is clustered in advance in the order of the closest distance to form a tree structure. There is a technique for searching only near subtrees. Further, Patent Document 1 describes a method of performing a high-speed search by projecting a model space prepared by a statistical method and generating multidimensional feature data with high accuracy and a small number of dimensions.

JP 2002-183205 A

However, in the conventional tree structure method, when the number of dimensions of the multidimensional feature data increases, the adjacent space increases exponentially, and it takes a lot of time to cluster (register) the multidimensional feature data. Furthermore, since a neighborhood search including an adjacent space is performed at the time of search, a huge amount of time is required like registration.
Further, in the technique described in Patent Document 1, there is a limit to the dimensional reduction due to projection. In addition, as a countermeasure, a method of calculating similarity using only parts with high importance such as eyes / nose / mouth is also described, but where individual features appear is different, There is a limit to reducing the dimension while maintaining accuracy.

  The present invention has been made in view of the above-described conventional circumstances, and is an image search capable of efficiently and appropriately searching for an image desired by a user even when a large amount of multi-dimensional feature data such as face / color exists in high dimensions. An object is to realize an apparatus and an image search method.

  The image search apparatus according to the present invention includes an image storage unit that stores an image, an approximate data storage unit that stores the approximate data of the image in association with the image, and approximate data that is similar to a search target. An approximate space search means for searching from the approximate data stored in, and a means for calculating the similarity between the image stored in the image storage means corresponding to the search result of the approximate space search means and the search target, The approximate data stored in the approximate data storage means is generated so as to maintain the relationship between the images stored in the image storage means.

  In the image search device of the present invention, the amount of data stored in the approximate data storage means is smaller than the data amount of the image storage means that stores images corresponding to the approximate data, and the approximate data storage means is stored in the image storage means. It is characterized by high speed access.

  The data storage device of the present invention is a data storage device in an image search device, and associates image storage means for storing an image with approximate data approximating the image with an image stored in the image storage means. The approximate data storage means, and the amount of data stored in the approximate data storage means is less than the amount of data stored in the image storage means storing the image corresponding to the approximate data, and the approximate data storage means Is a storage medium that can be accessed at a higher speed than the image storage means, and the approximate data stored in the approximate data storage means is generated so as to maintain the relationship between the images stored in the image storage means. It is characterized by that.

  The image search method of the present invention is a method of searching for an image by a programmed computer using an image search device provided with the data storage device, wherein approximate data similar to a search target is stored in the approximate data storage. A step of searching from the approximate data stored in the means, a step of obtaining an image corresponding to the search result from the image storage means, and a step of calculating the similarity between the search target and the image of the search result obtained from the image storage means It is characterized by having.

  According to the present invention, the search results can be narrowed down to some extent in the approximate space with a reduced number of dimensions, and then finally narrowed down in the real space. Therefore, even when the number of dimensions becomes high, an image desired by the user can be efficiently obtained. You can search.

1 is a block diagram of an image search device according to a first embodiment of the present invention. The flowchart regarding the data registration operation | movement of the image search device in the 1st Embodiment of this invention The flowchart regarding the search operation | movement of the image search device in the 1st Embodiment of this invention The flowchart regarding the search operation | movement which eliminates the search omission of the image search device in the 1st Embodiment of this invention. Explanatory drawing regarding the difference between the two types of search processing of the image search device in the first embodiment of the present invention The flowchart regarding the re-search operation of the image search apparatus in the 2nd Embodiment of this invention Explanatory drawing about distortion rate calculation of the image search device in the second embodiment of the present invention Explanatory drawing regarding the data structure managed by the image search device in the second embodiment of the present invention Flow chart about re-search operation of image search apparatus in third embodiment of the present invention (No. 1) Flowchart (2) regarding re-search operation 1 of the image search device in the third embodiment of the present invention

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
FIG. 1 is a configuration diagram of an image search apparatus according to Embodiment 1 of the present invention. In FIG. 1, 11 is a camera that captures a person, 12 is a search server that searches for an image including a person corresponding to the specified search condition, and 13 is a search that specifies the search condition for the search server 12. It is a search terminal for executing. 101 is a multidimensional feature data generation unit that extracts multidimensional feature data for identifying a person such as a face / color / shape from an image photographed by the camera 11, and 102 is a multidimensional feature data generation unit 101. Dimensional reduction means 103 for generating approximate data by reducing the dimensions of the multi-dimensional feature data, 103 is associated with the approximate data generated by the dimensional reduction means 102 and the multi-dimensional feature data before dimensional reduction, respectively. Approximate data storage means 104 for storing as a group 103a and actual feature data group 103b, 104 is a search request receiving means for receiving at least an identifier for identifying multidimensional feature data of a person to be searched as a search key, and 105 is a search terminal 13 Approximate data and approximate data storage corresponding to the search key received by the search request receiving means 104 based on the search condition specified in Approximate space search means 106 that performs distance calculation with each of the plurality of approximate data accumulated in the stage 103 and arranges the calculation results in order of distance, that is, in order of similarity, is a result of high similarity obtained by the approximate space search means 105. This is real space final ranking means for performing distance calculation again on the group using multidimensional feature data before dimensional reduction and determining the final rank. The final rank determined by the real space final ranking means 106 is output as a search result.

  The human feature data extracted by the multi-dimensional feature data generation means 101 is image data of a moving object cut out from the image, or information for specifying the moving object by shape, color, size, movement, or the like, or This information identifies the shape and position of the face's eyes, nose and mouth. Methods for extracting and classifying these feature information are widely known. It is written. Human feature data such as face / clothing color generated using these existing technologies is composed of a plurality of elements (referred to as dimensions) to identify a person. For example, the face feature data is composed of a total of several hundred to several thousand elements: a group of elements for grasping the entire face and a group of elements for grasping the shape of a specific part such as eyes / nose / mouth.

  FIG. 2 shows a processing procedure of the dimension reduction means 102, and the operation will be described below.

  <Step 201> With respect to the input multidimensional feature data ([element number, value] series), all values are set to absolute values, and the values are sorted in descending order. The input multidimensional feature data includes face feature data having principal components of the entire face / part units, as shown in 2-a, or data representing the color distribution of a person's clothes as a color space histogram such as RGB / HSV. In addition, a set of [element number, value] of each element on the horizontal axis is given as input of the multidimensional feature data in step 201, such as data obtained by cutting out an area in which a person is captured and converting it into a frequency. Further, after sorting, the elements are arranged in descending order of absolute values such as 2-b, and each element on the horizontal axis is generated as [element number and value before sorting].

  <Step 202> The multidimensional feature data is separated (the dimension is cut) in the designated dimension (R). After separation, the elements within the specified dimension R (elements with a large absolute value) are output as R1 data as [element number, value before sorting], and the part larger than the specified dimension R is R2 data. It is generated as [sign bit string in representative values V, R2] (2-c). As the representative value V of R2 data, an average value of absolute values of R2 data or a maximum value of absolute values of R2 data is used. Also, the sign bit string is generated as a bit string in which the sign of the number of elements of R2 is known, with the bit value = 1 if the value of the Nth element of R2 is positive, and the bit value = 0 if it is negative.

  <Step 203> The R1 and R2 data generated in Step 202 is stored in the approximate feature data group 103a of the approximate data storage means 103. In addition, the pre-sort vector data input in step 201 is stored in the actual feature data group 103b. In this way, an index is generated. Note that the approximate feature data group 103a is a data group with reduced dimensions, and may be arranged on a memory that can be accessed at high speed.

  FIG. 3 shows the processing procedure of the approximate space search means 105 and the real space final ranking means 106, and the operation will be described below.

<Step 301> Approximate distance calculation between approximate data (3-a) corresponding to the search key received by the search request receiving means 104 and a plurality of approximate data (3-b) stored in the approximate data storage means 103 is performed. The plurality of approximate data stored in the approximate data storage means 103 are arranged in ascending order of the approximate distance. As shown in (3-c), the approximate distance calculation is performed for all dimensions before sorting.
1) If each dimension is included in R1 of (3-a) and (3-b), distance calculation is performed using a value in R1.
2) If each dimension is included in R2 of (3-a) and (3-b), an approximate value is calculated from the representative value (V) of R2 and the sign bit for those included in R2. Then, the distance is calculated using it.
The process is performed.

  <Step 302> The real distance calculation with the search key received by the search request receiving means 104 is performed on the top M items arranged in ascending order of the distance in Step 301, and the actual distance is small as shown in (3-d). The top K items are extracted in order and returned as a result. The actual distance is calculated from the pre-sort vector data stored in the actual feature data group 103b.

  The process of FIG. 3 performs final ranking in the real space for the top M items arranged in ascending order of the approximate distance, but there is a possibility that a search omission may occur when the top M items are narrowed down. FIG. 4 shows a processing procedure of the real space final ranking means 108 for setting search omission due to narrowing down in the approximate space to 0, and the operation will be described below.

  <Step 401> A list of approximate distances with the search key generated by the approximate space search means 105 is acquired. It is assumed that the list is stored in order from the smallest approximate distance.

  <Step 402> Data with a small approximate distance is acquired from the approximate distance list. At the time of acquisition, the corresponding data is deleted from the approximate distance list.

  <Step 403> The actual distance between the data acquired in step 402 and the data corresponding to the search key is calculated.

  <Step 404> The data acquired in step 402 is added to the actual distance list. The list is stored in order from the smallest actual distance.

  <Step 405> It is determined whether all the top K distances in the real distance list are smaller than the minimum distance in the approximate distance list. If yes, go to step 406; if no, go to step 402.

  <Step 406> The top K items in the real distance list are output as the search results of the real space final ranking means 106.

  Differences in search results between the processing procedures of FIGS. 3 and 4 will be described with reference to FIG. (5-a) shows an example of “approximate distance and actual distance” between the search key and the data A to H. In contrast to (5-a), when a search is performed with the flow of FIG. 3, the data G causes a search omission as shown in (5-b). On the other hand, when the search is performed according to the flow of FIG. 4, the search for the data G does not occur as shown in (5-c). Which of the search processes in FIG. 3 and FIG. 4 is used depends on the relationship between the approximate distance and the actual distance. For example, when “approximate distance≈actual distance” for all data, there is no possibility that the search order will be greatly changed by approximation, so FIG. 4 is suitable because high-speed processing is possible while preventing search omissions. . On the other hand, when “approximate distance << actual distance” is satisfied, the search of all data eventually occurs in the process of FIG. 4 and the process is slowed down. Therefore, the process of FIG. Is suitable.

  As described above, after narrowing down the search results to some extent in the approximate space with a reduced number of dimensions, the final refinement is performed in the real space. The desired image can be searched efficiently. In addition, for each image, only components that are assumed to be highly important such as eyes / nose / mouth are used in advance by separating the component into a component that strongly expresses human characteristics and an average component and reducing the dimensions. Compared with the case where the dimension is reduced, even when a person image whose features appear to be large in a component that is assumed not to be high in importance is registered, the person image can be searched flexibly and appropriately. . Furthermore, by introducing a search process as shown in FIG. 4, search omissions in the process of narrowing down search results in the approximate space can be reduced to zero.

  In addition, as a dimension reduction method, the method of separating the component characteristic into a component that strongly expresses the characteristics of the person and the average component has been described, but it corresponds to a low frequency by using wavelet transform represented by Haar et al. The R2 data may be generated from the “average component” and the R1 data may be generated from the “difference component from the average” corresponding to the high frequency. In this case, since the element numbers constituting the R1 / R2 data are fixed without depending on the input multidimensional feature data, an effect of reducing the amount of calculation in the approximate space search unit 105 can be obtained. In addition, when the difference between the multidimensional feature data strongly appears in the average component, the “average component” corresponding to the low frequency after the wavelet transform described above is referred to as the R1 data and the “average” corresponding to the high frequency. This can be dealt with by changing the “difference component” to R2 data.

(Embodiment 2)
In the second embodiment, an image search apparatus will be described in which a user can easily perform a narrow-down operation by re-search even if many search omissions occur in the process of narrowing down search results in the approximate space.

  Since the configuration described in the second embodiment of the present invention is almost the same as that of the first embodiment, only the processing procedure of the refinement operation at the time of additional re-search is described here, and the others are omitted.

  FIG. 6 shows a result of “distortion rate due to dimension reduction” indicating how much the search result obtained by the approximate space search unit 105 has changed in the final ranking of the real space final ranking unit 106. An example in which “approximate number of dimensions” and “number of cases to narrow down” are set as re-search conditions by the approximate space search means 105 while referring to “distortion rate due to dimension reduction”. (6-a) is an initial search condition designation screen for first designating a search target person's search key and time / place search range, and (6-b) is a search result including “distortion rate due to dimension reduction”. (6-c) shows three types of re-search methods: (1) Adjustment of the number of used dimensions (number of dimensions used), 2) Adjustment of the approximate range (number of items to be narrowed down), 3 ) Is a re-search condition designation screen in which the user designates the next search condition from [Output next K items without adjusting in 1) and 2) above; and (6-b) and (6-c) The operation is repeated.

  A calculation example of “distortion rate due to dimension reduction” will be described with reference to FIG. (7-a) represents an example of a search omission (data L, G) that occurs when a search is performed with the configuration shown in FIG. (7-b) represents the difference between the search results obtained by the approximate space search means 105 and the real space final ranking means 106 when a search omission occurs. In (7-b), if the M value of the top M items, which is the threshold for narrowing down in the approximate space, is increased, the probability of search omission is reduced, but it is possible to determine how much the user should increase the M value. Absent. Therefore, as the “dimensional reduction factor,” a) the ratio of the top K items obtained in the approximate space included in the final ranking top K items, or b) the data of the top ranking top K items, “ Tell the user how much distortion due to dimensional degeneracy has occurred by using the rank ratio of the sum of the final ranks (ie, K * (K + 1) / 2) / the sum of the ranks in the recent space. Can do. In addition, distortion becomes large, so that the value of said a) b) is small.

  Next, an operation for specifying a re-search condition by the user with reference to “dimensional distortion factor” will be described in detail. As shown in (6-c), there are three patterns for specifying the re-search condition: 1) adjustment of the number of used dimensions, 2) adjustment of the approximate range, and 3) output of the next K items.

  The number of used dimensions of 1) is the number of elements of R1 data used in the approximate space search means 105. By increasing the number of elements of R1 data, the “dimensional reduction distortion rate” can be reduced. When the number of use dimensions is adjusted, the approximate space search unit 105 can change the number of elements in the R1 data and perform a re-search process so that a plurality of cut dimensions (R # a as shown in FIG. , R # b, R # c) is generated in advance. The data structure does not need to be generated for each cut dimension, but can be prepared by preparing for the cut dimension (R # c) where the number of elements of R1 data is large as shown in (8-b). It is.

  The adjustment of the approximate range in 2) is for adjusting the range to be narrowed down in the approximate space (M values of the top M cases). Even when the “dimensional distortion factor” is large, the M value is increased. Search omissions can be prevented.

  The next K cases of 3) are used when the user determines that “dimensional reduction distortion rate is small” or “dimensional reduction distortion rate is large, but 1) adjustment in 2) is difficult”.

  As described above, by referring to the “distortion rate due to dimension reduction” and adjusting the “number of dimensions to use” and “number of cases to narrow down” with the approximate space search means, the search results in the approximate space It is possible to realize a re-search operation that suppresses a search omission in the process of narrowing down. The re-search condition specifying unit may automatically re-specify the search condition with reference to the “distortion rate due to dimension reduction” output by the real space final ranking unit. With this configuration, an image can be searched more efficiently.

(Embodiment 3)
In the third embodiment, even if many search omissions occur in the process of narrowing down search results in the approximate space, the user can easily perform a narrowing operation by re-searching. A search device will be described.

  Since the configuration described in the third embodiment of the present invention is almost the same as that of the first embodiment, only the processing procedure of the refinement operation at the time of additional re-search is described here, and the others are omitted.

  FIG. 9 shows that when the user designates a correct answer or an incorrect answer for the search result, the approximate space search means 105 strongly weights the element number used in the approximate data designated as the correct answer, and designates an error. In this example, the weighting of the element numbers used in the approximate data is weakened, and the distance calculation in the approximate space is performed again. As shown in (9-b), since the element number that strongly represents the characteristics of the person is different for each search result, the element number for deriving the correct answer and the element for removing the incorrect answer from the result designated by the user as the correct answer / incorrect answer By extracting each number and performing weighting, it is possible to realize a re-search process that suppresses search omissions. In addition, weighting means assigning a weighting factor to the distance calculation result in the distance calculation of each dimension in (3-c) of FIG.

  Further, FIG. 10 shows that when the user designates an incorrect answer for the search result, the approximate space search means 105 “number of dimensions to be used” prevents the data designated as the incorrect answer from being output by the real space final ranking means 106. ”And“ number of items to narrow down ”are shown as examples of automatic adjustment. In (10-b), when the user designates “data H / E = incorrect answer”, the threshold for narrowing down in the approximate space until an element having a smaller distance than data H / E appears in the final ranking. The process is performed by increasing the M value of the top M items or increasing the number of dimensions to be used.

  As described above, the user specifies correct / incorrect answers for each image, and automatically adjusts approximate distance calculation parameters (element number weight, number of dimensions used, approximate range) at the time of re-search. By doing so, it is possible to realize a re-search operation that suppresses search omissions in the process of narrowing down search results in the approximate space.

  As described above, the image search apparatus and the image search method according to each embodiment of the present invention narrow down the search results to some extent in the approximate space with a reduced number of dimensions, and then perform final narrowing in the real space, Even when the number of dimensions becomes high, it has the effect of reducing the amount of calculation and efficiently searching for the image that the user wants, and it can be used for shoplifters, lost children, and lost persons targeting multiple cameras. In addition to the monitoring purpose of grasping all the actions, it can be applied to the purpose of browsing / searching / editing contents (still images / moving pictures) taken by individuals such as travel and athletic meet.

  The present invention can narrow down the search results to some extent in the approximate space with a reduced number of dimensions, and then can finally narrow down in the real space. Therefore, even when the number of dimensions becomes high, the user can search efficiently. Image retrieval device and image retrieval that can quickly retrieve a desired image from a large-capacity image group using multidimensional feature data (face, color, etc.) extracted from the image. Useful for methods and the like.

DESCRIPTION OF SYMBOLS 11 Camera 12 Search server 13 Search terminal 101 Multidimensional feature data generation means 102 Dimension reduction means 103 Approximate data storage means 103a Approximate feature data group 103b Real feature data group 104 Search request reception means 105 Approximate space search means 106 Real space final ranking means

Claims (5)

Image storage means for storing multidimensional images;
And approximate data storing means for storing in association with the previous image that approximates the approximate data of the image,
Approximate space search means for searching for approximate data similar to the search object from the approximate data stored in the approximate data storage means;
Means for calculating a distance between an image stored in the image storage unit corresponding to a search result of the approximate space search unit and a search target;
The distance search is performed on M search results (K, M: natural numbers) more than K output as a result . 2. The image according to claim 1, wherein the approximate data stored in the approximate data storage unit is generated by averaging a part of dimensions of the multidimensional image stored in the image storage unit. Search device. The approximate data stored in the approximate data storage means is composed of individual dimensional components with high frequency components of the multi-dimensional image stored in the image storage means and components obtained by averaging low-dimensional dimensions of frequency components. The image search apparatus according to claim 1 , wherein the image search apparatus is an image search apparatus. The approximate data stored in the approximate data storage means is composed of individual dimension components having low frequency components of the multidimensional image stored in the image storage means and components obtained by averaging high dimensions of frequency components. The image search apparatus according to claim 1 , wherein the image search apparatus is an image search apparatus. A method for searching for images by a programmed computer,
An image accumulation step for accumulating multidimensional images;
An approximate data storage step for storing the approximate data of the image in association with the image;
An approximate space search step for searching for approximate data similar to the search object from the approximate data stored in the approximate data storage step;
Calculating the distance between the image stored in the image storage step corresponding to the search result of the approximate space search step and the search target,
The distance calculation is performed on M search results (K, M: natural numbers) more than K output as a result .

Images