JP2019082959A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To successively output newly retrieved results while maintaining an order of retrieved results.SOLUTION: According to an information processing method of the present invention, a multi-dimensional vector is calculated from query data from storage means registering a first vector which is a retrieval object at a plurality of groups in a multi-dimensional space, thereby a short distance to the multi-dimensional vector registered at the plurality of groups of an index is determined. Based on the determined shortest distance, an order of groups to be compared with multi-dimensional vectors of query data are determined, and based on the determined order, comparison between the registered multi-dimensional vectors of each group and the multi-dimensional vector of the query data is carried out. A registered multi-dimensional vector having distance between vectors set shorter than the shortest distance is output as the retrieval result.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a technique for searching for an object of interest.

  Conventionally, there is known an apparatus which detects the face of a person from each frame of a video to be monitored, calculates an image feature amount from the face, and stores it in association with the frame of the video (see Patent Document 1). ). In the device, it is possible to compare the accumulated image feature amount with the face of a person to be retrieved as a query, and to display a video in which the person is shown.

  Further, Non-Patent Document 1 discloses a method of performing a search while sequentially expanding a search range in a search using an image feature amount of a human face. In this method, when the number of search results is sufficient, or when sufficient reliability of the search results is obtained (when the ratio of the number of votes obtained for the first place to the number of votes obtained for the second place exceeds a predetermined value) It is disclosed to abort the search process and display the results.

JP, 2009-199322, A JP, 2002-373332, A JP, 2010-165156, A

Keisuke Maekawa, Yuzuko Uchiumi, Masakazu Iwamura, Kouichi Kose (Osaka Prefecture University): "Realization of verification in 1 ms face image database in 34 ms", Technical Report of IEICE. PRMU, Pattern recognition and Media understanding 111 (353), 95-100, 2011-12-08 Erik Muphy-Chutorian, “Head pose estimation for driver assistance systems: A robust algorithm and experimental evaluation,” in Proc. IEEE Conf. Intelligent Transportation Systems, 2007, pp. 709-714. C. Harris and M. J. Stephens, "A combined corner and edge detector," In Alvey Vision Conference, pages 147-152, 1988. David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110.

  Both the method described in Patent Document 1 and the method described in Non-Patent Document 1 output and display the results after the search for all search targets is completed. However, when video of a large number of surveillance cameras or video for a long time is to be searched, it takes time to obtain a search result. Therefore, in the above-mentioned prior art, the supervisor (user) instructs the search It takes a long time before you can check the search results. This is not limited to the search for a person in the video of the surveillance camera, and the same applies to a search for other objects.

  By outputting and displaying the search processing results so far in the middle of the search processing, it is possible to shorten the time until the observer (user) confirms. However, when the search results are updated and new search results are added, it is desirable to maintain the order of the search results so far. Therefore, an object of the present invention is to sequentially output new search results while maintaining the order of the search results.

  According to the present invention, the first vector similar to the second vector representing the feature value calculated from the query data is stored from the storage means in which the first vectors to be searched are registered in a plurality of groups in the multidimensional space. An information processing apparatus for outputting, the input means for inputting the query data, the calculation means for calculating the second vector from the input query data, and the first vector registered in the group A first determining unit that determines the shortest distance that the second vector can take as the shortest distance, and an order of the plurality of groups to be compared with the second vector based on the determined shortest distance And comparing the first vector and the second vector for each group based on the determined second determining means and the determined order; Wherein the distance between said first vector and said second vector each over-flops and an output means for outputting the short first vector than the shortest distance search result.

  According to the present invention, it is possible to sequentially output new search results while maintaining the order of the search results.

FIG. 2 is a block diagram showing the hardware configuration of the information processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing a functional configuration of the information processing apparatus according to the first embodiment. FIG. 6 is a conceptual diagram showing a feature amount space of face image features stored by the feature storage unit according to the first embodiment. FIG. 3 is a conceptual diagram showing a feature amount space of face image features searched by a search unit according to the first embodiment. FIG. 6 is a view showing a display example of search results by a search result display unit according to the first embodiment. 6 is a flowchart showing a processing procedure of accumulating face image features according to the first embodiment. 6 is a flowchart showing a processing procedure for searching for a face image according to the first embodiment. FIG. 10 is a conceptual diagram showing a feature amount space of face image features stored by a feature storage unit according to the second embodiment. The conceptual diagram showing the feature-value space of the face image feature which the search part which concerns on 2nd Embodiment searches. The flowchart which shows the process sequence which accumulate | stores the face image characteristic which concerns on 2nd Embodiment. The flowchart which shows the process sequence which searches the face image which concerns on 2nd Embodiment. The conceptual diagram which shows an example of the search result from each index in 3rd Embodiment.

First Embodiment
Hereinafter, the details of the first embodiment of the present invention will be described with reference to the drawings. In the present embodiment, a facial image feature is calculated from an image of a person in a video captured by a surveillance camera, and the facial image feature amount is stored in association with camera information, shooting time, and the like. Then, face image search is performed based on the face image given as a query (search source). At that time, the results are displayed one after another while performing a search.

  FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing apparatus 100 constituting a server apparatus and a client apparatus in the present embodiment. The server apparatus and the client apparatus may be realized by a single information processing apparatus, or the functions may be distributed to a plurality of apparatuses as needed to realize the information processing apparatus. When configured by a plurality of devices, they are connected by a LAN (Local Area Network) or the like so that they can communicate with each other. Further, the information processing apparatus can be realized by an apparatus such as a personal computer (PC) or a workstation (WS).

  In FIG. 1, a CPU (Central Processing Unit) 101 controls the entire information processing apparatus 100. A ROM (Read Only Memory) 102 is a memory that stores programs and parameters that do not need to be changed. A random access memory (RAM) 103 is a memory for temporarily storing programs and data supplied from an external device or the like. The external storage device 104 is a storage device such as a hard disk or a memory card fixedly installed in the information processing apparatus 100. Note that the external storage device 104 may be a flexible disk (FD) or an optical disk such as a CD or the like removable from the information processing apparatus 100, a magnetic or optical card, an IC card, a memory card, or the like. Each operation described later is executed by the CPU 101 executing a program stored in the ROM 102 or the external storage device 104.

  The input device interface 105 is an interface with an input device 109 such as a pointing device or a keyboard which receives data from a user operation. The output device interface 106 is an interface with the monitor 110 for displaying data held by the information processing apparatus 100 and supplied data. The communication interface 107 is a communication interface for connecting to a network line 111 such as the Internet. The network camera 112 is an imaging device of a video such as a monitoring camera, and is connected to the information processing apparatus 100 via the network line 111. A system bus 108 is a transmission line communicably connecting the units described above.

  FIG. 2 is a block diagram showing an example of the functional configuration of the information processing apparatus 100 according to the present embodiment. The video input unit 201 inputs video data (continuous image) from the network camera 112 via the communication interface 107. The video storage unit 202 stores the video data input to the video input unit 201 in the external storage device 104. At this time, information such as a photographing time and a photographing camera is associated with the video data and stored as metadata of the video.

  The tracking processing unit 203 tracks the person in the video input from the video input unit 201. For the person tracking process, for example, a known technique shown in Patent Document 2 may be used. In the method described in Patent Document 2, an object is detected from a motion vector, a search position in the next frame is estimated, and person tracking is performed by template matching. The tracking processing unit 203 issues the same tracking track ID to tracking tracks tracking the same person and secures uniqueness by issuing different tracking track IDs to tracking tracks of different persons. It is possible to identify the same person from the tracking track ID. Also, even if the person is the same, when the tracking is interrupted, different tracking track IDs are issued.

  The face detection unit 204 performs face detection from each of the frame images of the person tracked by the tracking processing unit 203. Also, the video input unit 201 performs face detection from the video including the face image input by the query video input unit 208 described later. As a method of detecting the face of a person from an image, for example, a known technique shown in Patent Document 3 may be used. That is, a one-eye candidate area is detected for an image to be processed, pairing is performed from a plurality of one-eye candidate areas, and a face area is determined based on the paired eye positions.

  The representative face image determination unit 205 selects a representative face image from a group of frame images of the tracked person. For the process of selecting a representative face image, for example, an image with a large face size detected by the face detection unit 204 is selected. The reason for using an image with a large face size is that the larger the face image, the more accurate image features can be obtained. That is, when calculating an image feature from a face image, it is necessary to perform face size normalization processing for scaling the size of the face image to a fixed size. At that time, if the face image is larger than the above-mentioned fixed size, reduction processing is performed and the loss of information is relatively small, but if smaller than the above-mentioned fixed size, it is necessary to perform pixel complementation like super resolution processing. There is severe deterioration of information.

  Further, a plurality of images may be selected from the frame image group as the representative face image. For example, there is a method of selecting a plurality of face orientation images. This is because even if the image of the same person is different in the face orientation, the image features obtained from the image are different. As a method of detecting the face direction of a person from an image, for example, a known technique shown in Non-Patent Document 2 may be used. In the technology disclosed in Non-Patent Document 2, a gradient direction histogram (Histogram of Oriented Gradient, HOG) is extracted as a feature amount, and the face direction is estimated by SVR. The gradient direction histogram is a feature quantity obtained by histogramming the brightness gradient information of an image for each of the images, and is known as a robust feature quantity for local noise or light and dark of the image. By selecting a feature that is robust to variations unrelated to the face orientation, such as noise and illumination variations, stable face orientation estimation is realized even in a real environment.

  Furthermore, an image with less blurring may be selected as a representative face image. Even with a camera that shoots a moving image, the shutter speed may change according to the brightness of the location, as with a still image camera. Therefore, blurring of the face image may occur due to the dark place or the movement speed of the subject, which directly causes deterioration of the image feature amount and attribute information. In order to estimate blurring, the frequency component of the face image area is determined, the ratio of the low frequency component to the high frequency component is determined, and it is determined that blurring occurs when the ratio of the low frequency component exceeds a predetermined value. Is possible. In addition, the representative face image may be selected from the viewpoint of whether or not there are eyelids and mouths. If there is eyelid or mouth, there is a possibility that the image feature of the organ may be altered, and these images should not be selected as representative face images.

  In this embodiment, the tracking processing unit 203 tracks a person, the face detection unit 204 detects the face of the person tracked, and the representative face image determination unit 205 represents a group of frame images of the person tracked. The face image is selected. However, the face detection unit 204 may detect the face of a person from the video input to the video input unit 201 and pass all the face images to the feature calculation unit 206 described later.

  The feature calculation unit 206 calculates a face image feature. In this embodiment, the LBP (Local Binary Pattern) feature calculated by dividing the whole face into blocks is used. Note that this feature is an example, and the present embodiment is not limited to this. Organ points such as eyes and mouth in the face image of a person may be determined, and SIFT (Scale Invariant Feature Transform) features of each organ point may be calculated and used, or Deep Learning may be performed to calculate the features. . These face image features are represented by multidimensional vectors (multidimensional vectors).

  The feature storage unit 207 calculates a face image feature (search target vector) calculated through the tracking processing unit 203, the face detection unit 204, the representative face image determination unit 205, and the feature calculation unit 206 from the video input to the video input unit 201. Are stored in the external storage device 104. In addition, information such as the ID of a person, a tracking track ID when tracking a person, shooting time, and a shooting camera are stored as metadata of a face image feature in association with the face image feature. At this time, an index is created in order to be able to retrieve the similarity of face image features at high speed. Specifically, face image features are grouped and registered, and a representative of the group is stored. At the time of search, first, the group to be compared is narrowed down in comparison with the representative of the group, and comparison is made with the face image features registered in the narrowed group. As a result, comparison with all registered face image features is not necessary, and high-speed search can be realized. Further, in the present embodiment, the distance between the face image feature registered in the group and the representative of the group is the longest. This distance is used at the time of sequential search described later.

  FIG. 3 is a conceptual diagram showing a feature amount space of face image features stored by the feature storage unit 207. As shown in FIG. The face image feature used in the present embodiment is a multidimensional vector such as 256 dimensions, but here, in order to simplify the description, it will be described with a two dimensional diagram.

  A point 301 is a feature on the feature space. The feature amount space is divided into a plurality of spaces by the k-means method or the like, and the feature amount is registered in the cluster generated by the division. An area 302 obtained by dividing the feature space is a cluster, and a line 303 obtained by dividing the feature space is a boundary of each cluster. In a multi-dimensional feature space, it is divided into clusters in a hyperplane.

  A triangle mark 304 is a feature (representative vector) representing each cluster. The representative feature may be an average vector of multidimensional vectors of face image features included in a cluster, a centroid of a cluster, or a feature closest to the average of multidimensional vectors or the centroid of a cluster.

  In addition, the feature storage unit 207 stores, for each cluster, a distance r (for example, 305) having the longest distance between the representative feature of the cluster and the feature registered in the cluster. From the representative features of the cluster, all feature quantities of the cluster are registered within a circle 306 of radius r (super-circle in a multi-dimensional feature quantity space).

  The query video input unit 208 inputs a face image of a person used for a search designated by the monitor (user). Specifically, first, the face image of a person stored in the external storage device 104 is displayed on the monitor 110, and designated by the user via the input device 109. The query video input unit 208 inputs the specified face image. In the present embodiment, the method of specifying the face image of the person used for the search is not limited to a specific method. In addition, the face image of the person used for the search may be one or more.

  The search unit 209 uses the face image feature (query vector having the same dimension as the search target vector) calculated from the query image (search source image) via the face detection unit 204 and the feature calculation unit 206 as a query. A search is performed from a large number of stored face image features. Then, a face image having a face image feature similarity equal to or greater than a predetermined threshold TH1 is specified as a search result. In the present embodiment, the reciprocal of the distance of the LBP feature is normalized and obtained as the similarity. The search unit 209 sequentially outputs the search results to the search result display unit 210 while performing the search process.

  FIG. 4 is a conceptual diagram showing a feature amount space of face image features when the search unit 209 searches. The feature actually used in the search is a multi-dimensional vector such as 256 dimensions, but here, in order to simplify the explanation, it will be described with a two-dimensional diagram.

  A point 401 is a feature amount on the feature amount space, and an area 402 is a cluster. A line 403 is a boundary of each cluster, a triangle mark 404 is a feature (representative vector) representing each cluster, and 405 is a longest distance r between the cluster representative feature and the in-cluster feature. Here, cluster names are given to some of the clusters of the cluster 402, and these are designated as G-1, G-2, and G-3.

  Asterisk 406 is a query face image feature. Here, it is assumed that the query face image feature is present in cluster G-1. First, the search unit 209 compares the query face image feature with the representative feature of each cluster to calculate a distance 407 (dk (k = 1, 2,...)). Next, the shortest distance between the query face image feature and the image feature registered in each cluster is the longest distance 405 between the representative feature of the cluster and the intra-cluster feature (rk (k = 1, 2,...)) Use to guess.

  Here, when focusing on the cluster G-2, the distance between the query face image feature and the representative feature of the cluster G-2 is d2. Also, the longest distance between the cluster representative feature and the intra-cluster feature is r2. At this time, (d2-r2) is the shortest distance between the query face image feature and the circle of radius r2 centered on the representative feature of the cluster G-2. Since the image feature of cluster G-2 does not exist in the range shorter than this distance (d2-r2) from the query face image feature, the distance between the query face image feature and the image feature registered in cluster G-2 Is (d2-r2) at the shortest. That is, the shortest distance corresponds to the shortest distance that can be taken between the query face image feature and the image feature belonging to the cluster of interest.

  The search unit 209 estimates the shortest distance between the query face image feature and the image feature registered in each cluster in the same manner for other clusters. Then, first, comparison is performed with the image features registered in the cluster in which the query image features exist. After that, the comparison with the image features registered in the clusters is performed sequentially from the cluster having the shortest shortest distance to each cluster.

  At this time, after the search unit 209 compares the query face image feature with the image feature registered in each cluster, the shortest among the comparison results is estimated between the cluster to be compared next. Output a result shorter than the distance. For example, in FIG. 4, after first comparing with the cluster G-1 in which the query face image feature is present, the shortest distance (d2-r2) estimated with the cluster G-2 to be compared next is The result inside the circle 408 is output as the radius. Then, after comparison with cluster G-2, the result inside circle 409 with the shortest distance (d3-r3) estimated between cluster G-3 to be compared next being a radius is output .

  In this way, a result shorter (higher in similarity) than the shortest distance estimated with the cluster to be compared next is output. As a result, it becomes possible to sequentially output search results in which the distance order is short, that is, the order in which the degree of similarity is high is guaranteed.

  The search result display unit 210 sequentially displays the search results sequentially output from the search unit 209 on the monitor 110. FIG. 5 shows a display example of a search result by the search result display unit 210. For example, as shown in FIG. 5A, the search result is displayed on the monitor 110 in order of similarity, assuming that the image is more similar to the image located closer to the left side of the upper row. Alternatively, as shown in FIG. 5B, the search results may be displayed on the monitor 110 in the order of the degree of similarity after being classified for each camera. Also in the cases of FIGS. 5A and 5B, the search result display unit 210 sequentially displays the search results of the search unit 209. However, in the present embodiment, since the search unit 209 sequentially outputs the search results in the descending order of the degree of similarity, the results to be added are added as the search results having a degree of similarity lower than the current state. Therefore, for example, even if the user aborts the confirmation of the search result displayed on the monitor 110 halfway, the situation in which the search result having a higher degree of similarity than the current state is displayed later is suppressed. In the present embodiment, as long as the display method can sequentially display the search results sequentially output from the search unit 209, the display method is not limited thereto.

  Next, details of processing for storing video data input from the video input unit 201 in a searchable manner will be described with reference to FIG. FIG. 6 is a flowchart showing a processing procedure for accumulating face image features according to the present embodiment. This processing corresponds to the processing from the video input unit 201 to the feature storage unit 207 described above.

  In step S601, the video input unit 201 inputs video data from the network camera 112 via the communication interface 107.

  In step S602, the video storage unit 202 stores the video data input in step S601 in the external storage device 104. In addition, information such as shooting time and a captured camera are associated and stored as video metadata.

  Steps S603 to S604 are processes performed by the tracking processing unit 203. First, in step S603, the tracking processing unit 203 detects a person from each frame image and performs tracking. Here, a separate person ID is assigned to the detected person for each frame image, and is temporarily stored together with the coordinates of the person in the frame image. The same tracking track ID is assigned to a person who is tracking, and is temporarily stored together with the ID of the frame image being tracked.

  In step S604, the tracking processing unit 203 determines whether there is a person whose tracking has been interrupted. As a result of this determination, when there is a person whose tracking has been interrupted, the tracking image group of the person is determined, and the process proceeds to the next step S605. On the other hand, when there is no person whose tracking has been interrupted, the processing returns to step S601 to continue the tracking.

  In step S605, the face detection unit 204 performs face detection from each of the frame images including the person tracked by the tracking processing unit 203.

  In step S606, the face detection unit 204 determines whether a face is detected in the process of step S605. As a result of the determination, if a face is detected, the process proceeds to step S607, and if a face is not detected, the process ends.

  In step S 607, the representative face image determination unit 205 selects one or more face images representing the detected face from the frame image group of the tracked person.

  In step S608, the feature calculation unit 206 calculates a face image feature from the one or more representative face images selected in step S607.

  In step S609, the feature storage unit 207 performs clustering on the face image features calculated in step S608 and the face image features stored so far. The clustering method is as described above (FIG. 3).

  In step S610, the feature storage unit 207 calculates the center of gravity of the cluster, and in step S611 calculates the longest distance from the center of gravity of the cluster among the face image features included in the cluster.

  In step S612, the feature storage unit 207 stores the face image feature in the external storage device 104. In addition, information such as a person ID, a tracking track ID when tracking a person, shooting time, and a shooting camera is stored as metadata of a face image feature in association with the face image feature.

  In step S613, when the video continues, the process returns to step S601, and when the video ends, the process ends.

  By the above processing, the feature storage unit 207 stores the face image feature of the face image of the person appearing in the image input from the network camera 112 in the external storage device 104, and the search becomes possible.

  In the present embodiment, clustering is performed each time an image is input and a face is detected. However, when the amount of data to be registered is small, the effect of clustering is small, so data may be registered as it is without clustering until the number of detected faces reaches a predetermined number. Also, clustering is a relatively heavy process of looking at the distribution of data and grouping. Therefore, instead of performing clustering each time (updating clusters), it is also possible to perform classification of face image features to be registered into each cluster only for a while after clustering. In that case, when the distance from the cluster centroid is the longest, the longest distance from the cluster centroid is updated.

  Next, details of processing for searching for a face image of a person from the query image input from the query video input unit 208 will be described using FIG. 7. FIG. 7 is a flowchart showing a processing procedure for searching for a face image of a person in the present embodiment. This processing corresponds to the processing from the face detection unit 204, the feature calculation unit 206, and the query video input unit 208 to the search result display unit 210 in FIG. 2 described above.

  In step S701, the query video input unit 208 inputs a face image of a person used for the search.

  In step S702, the face detection unit 204 performs face detection from the video including the face image input by the query video input unit 208.

  In step S703, the feature calculation unit 206 calculates a face image feature from the face image detected in step S702.

  In step S704, using the face image feature calculated in step S703 as a query, the search unit 209 compares each of the cluster centroids stored in the external storage device 104 with each other to obtain a distance 407 (dk (k = 1, 2,. ...)) is calculated.

  In step S 705, the search unit 209 sets the shortest distance between the representative feature of the cluster and the intra-cluster feature to the shortest distance 405 (rk (k = 1, 2,...)) Determine the comparison order by guessing and sorting using. The shortest distances are compared in order from the shortest cluster.

  In step S706, the search unit 209 sets a cluster having a query face feature as a comparison target cluster.

  In step S 707, the query face feature is compared with each image feature in the comparison target cluster, and the distance is calculated.

  In step S708, if the comparison target cluster still exists, the process proceeds to step S709, and if the comparison target cluster does not exist, the process proceeds to step S712.

  In step S709, among the comparison results in step S707, the comparison result whose distance is shorter than the shortest distance to the next comparison target cluster is output.

  In step S710, the search result display unit 210 adds the comparison result output in step S709 to the previous result and displays the result on the monitor 110. In addition, when a plurality of results are obtained from the same camera, it is also possible to display a predetermined number of results from the one with the highest or the highest similarity of the face image, instead of displaying all the results. Also, if there are a large number of display results, they may be divided and displayed, or may be updated and displayed after a user's instruction.

  In step S711, the cluster with the shortest shortest distance is set as the comparison target cluster, and the process proceeds to step S707.

  In step S712, since comparison with all comparison target clusters is completed, all comparison results are output.

  In step S713, the search result display unit 210 adds the comparison result output in step S712 to the previous result, displays the result on the monitor 110, and ends the processing.

  If the amount of registered data is small and clustering is not performed at the time of registration, a normal process of comparing with all registered features regardless of this processing flow may be performed, and the result may be output. .

  As described above, according to the present embodiment, facial image features are calculated from images of the same person obtained by tracking the person in the video of the surveillance camera, and they are clustered and stored. Then, when performing face image search based on the face image given as a query, first, the shortest distance between the query face feature and the feature in each cluster is estimated. Then, the query face feature and the stored image feature are compared in order from the cluster with the shortest shortest distance, and the search result is sequentially output. Therefore, in the present embodiment, it is possible to shorten the time until the user can confirm the search result.

  Further, in the present embodiment, after the comparison with each image feature of the cluster in which the query face feature is present is completed, the comparison with each intra-cluster feature is performed in ascending order of shortest distance, and estimation with the cluster to be compared next Output a result shorter than the shortest distance. According to this configuration, in the present embodiment, it is possible to sequentially output the results of guaranteeing the order of short distance, that is, the order of high similarity.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, the feature amount space is divided into a plurality of spaces by the k-means method or the like, and the feature amounts are registered in the divided clusters. In the present embodiment, the feature amount space is divided according to a predetermined rule, and the feature amounts are registered in the divided clusters. Here, the boundaries of each cluster are divided so that they can be expressed by multidimensional expressions. The components already described in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 8 is a conceptual diagram showing a feature amount space of face image features stored by the feature storage unit 207 according to the present embodiment. The features used in the present embodiment are multi-dimensional vectors such as 256 dimensions, but will be described here in a two-dimensional diagram for the simplification of the description. In the first embodiment, the feature storage unit 207 stores the longest distance among the distances between each image feature registered in the cluster and the representative feature of the cluster, but this feature is not necessary in this embodiment.

  A point 801 is a feature on the feature space. The feature space is divided into a plurality of lattice-like spaces, and the feature is registered in the divided clusters. In this embodiment, an example of a square lattice is shown, but a triangular lattice or a hexagonal lattice may be used. Moreover, in a multidimensional space, it may divide | segment by a simple super cubic lattice, and may divide | segment by a face-centered super cubic lattice. An area 802 obtained by dividing the feature space is a cluster, and a line 803 obtained by dividing the feature space is a boundary of each cluster. In a multi-dimensional feature space, it is divided into clusters in a hyperplane.

  FIG. 9 is a conceptual diagram showing a feature amount space of face image features searched by the search unit 209 according to the present embodiment. The features used in the present embodiment are multi-dimensional vectors such as 256 dimensions, but will be described here in a two-dimensional diagram for the simplification of the description.

  A point 901 is a feature amount on the feature amount space, and an area 902 is a cluster. Line 903 is the boundary of each cluster. Here, cluster names are given to some of the clusters of the cluster 902, and these are referred to as G-1, G-2, G-3, and G-4.

  Asterisk 904 is a query face image feature. Here, the query face image feature is present in cluster G-1. The search unit 209 first calculates the shortest distance between the query face image feature and each cluster. For example, the shortest distance between the query face image feature and the cluster G-2 is the distance of the normal from the query image feature to the boundary between the cluster G-1 and the cluster G-2. The shortest distance between the query face image feature and the cluster G-3 is the distance of the normal from the query image feature to the boundary between the cluster G-1 and the cluster G-3. The shortest distance between the query face image feature and cluster G-4 is the distance from the query image feature to the intersection of cluster G-1, cluster G-2, cluster G-3, and cluster G-4.

The distance of the normal from a certain point (q ₁ , q ₂ ,..., q _n ) in the n-dimensional space to the hyperplane represented by equation 1 can be calculated by equation 2.

Further, the distance between two points (p ₁ , p ₂ ,..., P _n ) and (q ₁ , q ₂ ,..., Q _n ) in the n-dimensional space can be calculated by Equation 3.

  The search unit 209 first compares each image feature registered in the cluster in which the query face feature is present. After that, comparison is performed with each image feature registered in the clusters in order from the cluster having the shortest shortest distance to each cluster.

  Also in the present embodiment, after comparison with each feature registered in each cluster, when the comparison result is output, the comparison result is shorter than the estimated shortest distance to the cluster to be compared next. Output the result. For example, in FIG. 9, after the comparison with the cluster G-1, the result inside the circle 905 is output with the radius being the shortest distance to the cluster G-2 to be compared next. After the comparison with the cluster G-2, the result inside the circle 906 is output with the radius being the shortest distance to the cluster G-3 to be compared next. In this way, by outputting a result (high in similarity) shorter than the shortest distance to the cluster to be compared next, the results in which the order of short distance, that is, the order in which high similarity is guaranteed, are sequentially It is possible to output to

  Next, details of processing for storing video data input from the video input unit 201 so as to be searchable will be described with reference to FIG. FIG. 10 is a flowchart showing a processing procedure for accumulating face image features in the present embodiment. This processing corresponds to the processing from the video input unit 201 to the feature storage unit 207 described above.

  Steps S1001 to S1008 are the same as steps S601 to S608 in FIG.

  In step S1009, the feature storage unit 207 registers the face image feature calculated in step S608 in a cluster determined according to a predetermined rule. In addition, information such as a person ID, a tracking track ID when tracking a person, shooting time, and a shooting camera is stored as metadata of a face image feature in association with the face image feature. As described above, in the present embodiment, the predetermined rule for determining each cluster is to divide the feature amount space into a plurality of lattice-like spaces.

  Step S1010 is the same as step S613 in FIG.

  By the above-described processing, the feature storage unit 207 stores face image features of the face image of the person in the video input from the network camera 112 in a searchable state in the external storage device 104.

  Next, details of processing for searching for a face image of a person from the query image input from the query video input unit 208 will be described using FIG. FIG. 11 is a flowchart showing a processing procedure for searching for a face image of a person in the present embodiment. This processing corresponds to the processing from the face detection unit 204, the feature calculation unit 206, and the query video input unit 208 to the search result display unit 210 in FIG. 2 described above.

  Steps S1101 to S1103 are the same as steps S701 to S703 in FIG.

  In step S1104, the search unit 209 calculates the shortest distance from each query face feature to each cluster, and determines the comparison order by sorting. After the comparison with each image feature of the cluster in which the query face feature is present, the search unit 209 performs comparison in order from the cluster having the shortest shortest distance.

  Steps S1105 to S1112 are the same as steps S706 to S713 in FIG.

  As described above, according to the present embodiment, face image features are calculated from images of the same person obtained by tracking the person in the video of the monitoring camera, and these are stored in clusters divided according to a predetermined rule. I did it. Then, when performing face image search based on the face image given as a query, first, the shortest distance between the query face feature and the feature in each cluster is estimated. Then, the query face feature and the stored image feature are compared in order from the cluster with the shortest shortest distance, and the search result is sequentially output. Therefore, in the present embodiment, it is possible to shorten the time until the user can confirm the search result.

  Further, in the present embodiment, after the comparison with each image feature of the cluster in which the query face feature is present is completed, the comparison with each intra-cluster feature is performed in ascending order of shortest distance, and estimation with the cluster to be compared next Output a result shorter than the shortest distance. According to this configuration, in the present embodiment, it is possible to sequentially output the results of guaranteeing the order of short distance, that is, the order of high similarity.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the first and second embodiments, face image features are registered in one index, and a search is performed on the index. In the third embodiment, the face image feature is divided into a plurality of indexes and registered, and the search is performed on the plurality of indexes. The components already described in the first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

  First, an advantage of dividing and registering a face image feature into a plurality of indexes will be described. For example, by dividing an index for each camera, it is possible to easily carry out a search with limited cameras to be searched. Further, by dividing the index for each fixed time or for each fixed number of face image features, it is possible to easily perform a search specifying a time zone. When the index is divided into a plurality of indexes, the index is created for each index according to the same procedure as in the first embodiment or the second embodiment.

  When the search is performed, the search unit 209 searches each index and acquires the search result for each index. FIG. 12 is a conceptual diagram showing an example of search results of each index. In the same figure, the search results of each index are shown in order of decreasing distance (in order of high similarity). Here, from index 1 1201 to 1203, from index 2 1204 to 1207, and from index 3 1208 to 1210 are acquired as search results. The numbers at the bottom of each search result are the distances from the query features.

  In this state, the search unit 209 first acquires the longest distance for each index. In the example of FIG. 12, the longest distance of index 1 is 110, the longest distance of index 2 is 140, and the longest distance of index 3 is 120. Then, the search unit 209 outputs a search result in which the distance is shorter than the shortest longest distance among these distances. In the example of FIG. 12, the shortest longest distance among the longest distances of each index is 110 of index 1. When acquiring the next search result, no search result whose distance is shorter than 110 is obtained from any index. Therefore, the search result whose distance is shorter than 110 is output. After that, the same process is repeated as acquiring search results for each index and outputting search results whose distance is shorter than the shortest longest distance among the longest distances of each index. This makes it possible to sequentially output search results guaranteed for all indexes in order of ascending distance, that is, in order of high similarity.

  In addition, when acquiring the search result for each index, it is also possible to acquire the result whose index is shorter than which distance, that is, the shortest distance to the cluster to be compared next. For example, in index 1 of FIG. 12, it is assumed that the shortest distance to the cluster to be compared next is 118 and there is no result longer than 110 and shorter than 118 in the comparison results so far. At this time, the comparison result that can be acquired from index 1 does not change. If it is possible to obtain in advance information that the shortest distance to the cluster to be compared next is 118, when obtaining the next search result, the result is that the distance is shorter than 118 from any index It turns out that it is not acquired. Therefore, by using this value, by outputting only results with a distance shorter than 118, it is possible to output more ranked results, that is, faster.

  As described above, according to the present embodiment, even when the face image feature is divided into a plurality of indexes and registered, it is possible to shorten the time until the user can confirm the search result. In addition, it is possible to sequentially output the results of guaranteeing the order in which the distances are short, that is, the order in which the degree of similarity is high.

Other Embodiments
In each embodiment described above, face image features are extracted from the image of a person in the video of the surveillance camera, and they are stored and can be searched. However, the present invention does not limit the object as the search target to a person. For example, the present invention may be applied to animals such as dogs and cats and vehicles such as cars. In addition, the features calculated from the target object are not limited to the face features calculated from the face image. For example, in the case of a person search, other than face images, feature amounts calculated from images of the whole body, clothes, and belongings may be used. When the present invention is applied to animals, facial images, body patterns, clothes, overall shapes and colors may be used as features. When the present invention is applied to a car, an image around the identification number, an overall shape or color may be used as a feature.

  The present invention is also applicable to searching for similar images using local feature quantities (local feature quantities) of images. In this method, first, characteristic points (local feature points) are extracted from the image (Non-Patent Document 3). Next, a feature amount (local feature amount) corresponding to the local feature point is calculated based on the local feature point and image information on the periphery thereof (Non-Patent Document 4). The feature quantities (multidimensional vectors) obtained in this manner may be clustered and accumulated, and may be sequentially compared with the in-cluster features at the time of retrieval and sequentially output.

  In the above embodiments, the data to be searched and the type of query data are described as images. However, the type of data is not limited to an image, and may be another type of data such as voice.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Processing is also feasible. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100 information processing apparatus 201 video input unit 202 video storage unit 203 tracking processing unit 204 face detection unit 205 representative face image determination unit 206 feature calculation unit 207 feature storage unit 208 query video input unit 209 search unit 210 search result display unit

Claims (14)

Information processing for outputting the first vector similar to the second vector representing the feature value calculated from the query data from the storage means which registered the first vector to be searched in a plurality of groups in the multi-dimensional space A device,
Input means for inputting the query data;
Calculation means for calculating the second vector from the input query data;
First determining means for determining the shortest distance that the first vector registered in the group and the second vector can take as the shortest distance;
Second determining means for determining the order of the plurality of groups to be compared with the second vector based on the determined shortest distance;
The first vector and the second vector are compared for each of the groups based on the determined order, and the distance between the first vector and the second vector in each of the plurality of groups is the Outputting means for outputting the first vector shorter than the shortest distance as a search result;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the plurality of groups are formed by dividing the multidimensional space.   The information processing apparatus according to claim 2, wherein the plurality of groups are generated by clustering the multidimensional space.   The first determination means is a longest distance which is the longest distance among the distance between the representative vector of the group and the first vector registered in the group, the representative vector, and the second vector. The information processing apparatus according to claim 2 or 3, wherein the shortest distance is determined based on the distance.   The representative vector is an average vector of a first vector registered in the group, a barycenter of the group, and a first vector closest to the average vector among the first vectors registered in the group, 5. The information processing apparatus according to claim 4, wherein one of the first vectors registered in the group is one of the first vectors closest to the center of gravity of the group.   The information processing apparatus according to claim 2, wherein the first determination unit determines a distance between the second vector and the boundary of the group as the shortest distance.   The second determination means narrows a group out of the plurality of groups based on the shortest distance of the plurality of groups, and determines an order of comparing the narrowed group with the second vector. The information processing apparatus according to any one of claims 1 to 6, wherein In the storage means, the first vector is registered for each of a plurality of indexes;
The output means compares the first vector with the second vector in each of the plurality of indexes, and the distance between the first vector and the second vector is longest in each of the plurality of indexes. 8. The method according to claim 7, further comprising: acquiring a first distance of the first vector, which is shorter than a shortest distance among the longest distances of the plurality of indexes acquired, as a search result. Information processing device. In the storage means, a multidimensional vector calculated from an image to be searched is registered as the first vector,
The input unit inputs a query image as the query data,
The information processing apparatus according to any one of claims 1 to 8, wherein the calculation means calculates a multidimensional vector from the input query image as the second vector. It further comprises detection means for detecting a person from the input query image,
10. The information processing apparatus according to claim 9, wherein the second vector is a multidimensional vector representing a feature amount extracted from the area of the detected person.   The information processing apparatus according to any one of claims 1 to 10, further comprising display means for displaying the output search result on a display unit. The input unit inputs data to be searched.
12. The apparatus according to claim 1, wherein the calculation means calculates the first vector from the data to be searched, and registers the calculated first vector in the storage means. The information processing apparatus according to claim 1. Information processing for outputting the first vector similar to the second vector representing the feature value calculated from the query data from the storage means which registered the first vector to be searched in a plurality of groups in the multi-dimensional space Method,
Inputting the query data;
Calculating the second vector from the input query data;
Determining the shortest distance that the first vector registered in the group and the second vector can take as the shortest distance;
Determining an order of the plurality of groups to be compared with the second vector based on the determined shortest distance;
The first vector and the second vector are compared for each of the groups based on the determined order, and the distance between the first vector and the second vector in each of the plurality of groups is the Outputting the first vector shorter than the shortest distance as a search result;
An information processing method characterized by comprising:   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 12.

Images