JP6598480B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to a technique for comparing local feature amounts of images.

  There has been proposed a method of searching for a similar image using a local feature amount (local feature amount) of an image. In this method, first, characteristic points (local feature points) are extracted from an image. Next, a feature amount (local feature amount) corresponding to the local feature point is calculated based on the local feature point and surrounding image information.

  In the method using the local feature amount, the local feature amount is defined as information including a plurality of elements that are rotation invariant and enlargement / reduction invariant. Thereby, even when the image is rotated, enlarged or reduced, the search can be performed. The local feature amount is generally expressed as a vector. However, it is a theoretical story that the local feature is invariant to rotation and enlargement / reduction. In an actual digital image, the local feature before image rotation and enlargement / reduction processing corresponds to that after processing. Some variation occurs between the local feature amount.

  In order to extract rotation-invariant local feature values, the main direction is calculated from the pixel pattern of the local region around the local feature point, and the local region is rotated based on the main direction when the local feature value is calculated to normalize the direction. . Further, in order to calculate the local feature quantity that does not change in size, the image of different scales is generated internally, and local feature points are extracted from the images of the respective scales and the local feature quantities are calculated. Here, a series of image sets of different scales generated internally is generally called a scale space.

  With the above-described method, a plurality of local feature points are extracted from one image. In image retrieval using local feature amounts, matching is performed by comparing local feature amounts calculated from respective local feature points. In a widely used voting method (Patent Document 1), if a feature point having a feature amount similar to a local feature amount of each feature point extracted from an input image exists in the registered image, the comparison destination is registered. Vote for an image. It is assumed that a registered image having a larger number of votes is more similar to the comparison source image. The input image is also referred to as a comparison source image, a query image, or a search source image. The registered image is also referred to as a comparison destination image or a search destination image.

  In addition, there is a method of cutting out a target object to be searched from an image in advance in order to perform image collation processing or image search processing with high accuracy (Patent Document 2). In this method, the user inputs the contour information of the target object on the image via the pen input unit, and extracts the object image of the part from which the background is removed.

JP 2009-284084 A Japanese Patent No. 10-254901

  However, in the voting method of Patent Document 1, for an image in which the foreground and background feature quantities having different distances from the imaging position are mixed, the feature quantity extracted from the foreground and the feature quantity extracted from the background are not distinguished. Similarity with other images was determined. For example, when obtaining the similarity with an image having more background feature points than the foreground as shown in FIG. 1, the local feature amount of the feature points in the background may have a similarity greater than or equal to a predetermined value. is there. In this case, the feature quantity in the background may affect the vote value, and the obtained vote value may become larger. Therefore, an appropriate similarity between images may not be obtained, and collation accuracy is reduced. In addition, when applying to image search and sorting a plurality of similar images by similarity to create a search result, an appropriate similarity cannot be obtained, thus reducing the search accuracy.

  In the method of Patent Document 2, the user has to perform special processing such as extraction processing by pen input on all target objects in the image. It was.

  The present invention has been made in view of the above problems, and an object of the present invention is to perform image matching or image search by focusing on feature points to be compared without requiring a user to perform special processing.

In order to solve the above problems, an image processing apparatus according to the present invention includes a first local feature amount for each of a plurality of feature points of a first image and a plurality of features of a second image obtained by imaging an imaging target. Based on the depth information, extraction means for extracting a second local feature amount for each point, holding means for holding depth information of the imaging target corresponding to each of the feature points of the second image, If the similarity between the specifying means for specifying at least one distance range and the second local feature value and the first local feature value obtained for the distance range is equal to or greater than a predetermined value, the second image Determining means that is similar to the first image, and the determining means obtains the similarity between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges. time of at least two or more, if the predetermined value or more Based on the sum of the similarity of the above predetermined value, the second image and judging whether similar to the first image.

  According to the present invention, by using the depth information of the comparison destination image, the feature amount of the comparison destination image is specified for each distance range, so that the user can focus on the feature points to be compared without performing any special processing. Thus, image collation processing or image retrieval processing can be performed.

It is an example of the image which an image processing apparatus processes. It is a figure of the structural example of the computer apparatus in 1st Embodiment. It is a figure of the example of composition of the image collation device in a 1st embodiment. It is explanatory drawing of the example of the data structure of the vote in 1st Embodiment. It is a flowchart for demonstrating the flow of the image collation process in 1st Embodiment. It is a flowchart for demonstrating the flow of the local feature-value extraction process in 1st Embodiment. It is explanatory drawing of a reduction image. It is a figure of the structural example of the image search device in 2nd Embodiment. (A) It is explanatory drawing of the example of the data structure of the image index and vote in 2nd Embodiment. (B) It is explanatory drawing of the example of the voting value produced | generated for every cluster. (C) It is explanatory drawing of the example which calculates | requires similarity from vote value. It is a flowchart for demonstrating the flow of the registration process in 2nd Embodiment. It is a flowchart for demonstrating the flow of the image search process in 2nd Embodiment. It is explanatory drawing of the quantization space in 2nd Embodiment. 14 is a flowchart for explaining a flow of image search processing in the third embodiment. It is explanatory drawing of the example which calculates | requires a similarity from a vote value.

[First Embodiment]
The configuration of the computer device that constitutes the server device and the client device of this embodiment will be described with reference to the block diagram of FIG. Each of the server device and the client device may be realized by a single computer device, or may be realized by distributing each function to a plurality of computer devices as necessary. When configured by a plurality of computer devices, they are connected by a local area network (LAN) or the like so that they can communicate with each other. The computer device can be realized by an information processing device such as a personal computer (PC) or a workstation (WS).

  In FIG. 2, a CPU 201 is a central processing unit that controls the entire computer apparatus 200. The ROM 202 is a Read Only Memory that stores programs and parameters that do not need to be changed. A RAM 203 is a Random Access Memory that temporarily stores programs and data supplied from an external device. The storage device 204 is a storage device such as a hard disk or a memory card that is fixedly installed in the computer device 200. Note that the storage device 204 may be a hard disk inside the computer device 200, a flexible disk (FD) that can be attached to and removed from the computer device 200, an optical disk such as a CD, a magnetic or optical card, an IC card, and a memory card. . An input device interface 205 is an interface with an input device 209 such as a pointing device or a keyboard that receives data from a user and inputs data. The output device interface 206 is an interface with the monitor 210 for displaying data held by the computer apparatus 200 and supplied data. A communication interface 207 is a communication interface for connecting to a network line 211 such as the Internet or an imaging device such as a digital camera 212, a digital video camera 213, and a smartphone 214. A system bus 208 is a transmission path that connects the units 201 to 207 so that they can communicate with each other.

  The imaging device used in the present embodiment includes a generation unit (not shown) that generates depth information of an imaging target. The depth information of the imaging target is a depth value or distance information indicating a distance from the generation unit to the imaging target.

  Each operation described below is executed by the CPU 201 executing a program stored in a computer-readable storage medium such as the ROM 202.

[Configuration of image verification device]
An example in which the image processing apparatus of this embodiment is used as an image collation apparatus will be described. In the image collation device, it is determined whether or not the two images are similar. In the present embodiment, two images processed by the image collating apparatus will be described as a comparison source image and a comparison destination image. Hereinafter, the configuration of the image collating apparatus of the present embodiment will be described with reference to FIG. The image collation apparatus includes an image input unit 301, a feature amount extraction unit 302, a data holding unit 303, a feature amount identification unit 305, a feature amount comparison unit 306, a similarity determination unit 307, and a collation result output unit 308.

  The image input unit 301 inputs a comparison source image (query image) from the outside of the image matching apparatus via the input device 209 or the like. Since the comparison destination image (registered image) is stored in the storage device 204, it is not necessary to input it.

  The feature amount extraction unit 302 extracts a characteristic point (local feature point) from the comparison source image and the comparison destination image, and corresponds to the local feature point based on the extracted local feature point and surrounding image information. The feature amount (local feature amount) to be calculated is calculated. Details of the local feature amount extraction processing will be described later with reference to FIGS. 6 and 7.

  Since the data holding unit 303 holds the depth information of the imaging target, the depth information of the feature amount calculation area is acquired from the data holding unit 303. Further, when the depth value is held, the depth value of the imaging target corresponding to the position of the feature point in the image is acquired. Alternatively, when the captured image and the distance image are separate data, mapping between the pixel of the captured image and the distance image is held in advance, and the depth value of the imaging target corresponding to the position of the feature point is acquired according to the mapping. Further, when there is a lot of noise in the distance image, the depth value may be acquired after smoothing the distance image in advance to reduce the noise. In addition, the acquisition method of depth information in this embodiment is not limited to these methods.

  The feature amount specifying unit 305 first specifies a plurality of distance ranges based on the depth value of the imaging target corresponding to the position of the feature point. For example, when the depth value of the imaging target is distributed in a distance range of 10 to 100 emails, the distance range of 10 to 100 emails is a plurality of 10 meter intervals such as 10 to 20 emails, 20 to 30 meters, and 30 to 40 emails. Divide into distance ranges. In the present embodiment, this process is referred to as depth value gradation. After gradation of the depth value, the feature values are clustered based on the depth value. Specifically, a depth value is obtained by the depth information holding unit 303 for each feature amount obtained by the feature amount extraction unit 302. Then, the feature values are clustered by gradation of the depth value. For example, the gradation width is determined in advance, and the quotient obtained by dividing the depth value of the feature amount by this gradation width is obtained. The gradation width is a distance indicating a constant interval, and in the above example, is a distance indicating an interval of 10 meters. Then, clustering is performed by assigning feature quantities having the same quotient to the same cluster. As a result, the feature amount can be specified and separated for each width (distance range) of the predetermined depth value, so that the feature amount of the image corresponding to the imaging target in the foreground and the imaging target in the background is separated. be able to.

  Further, as an example of clustering, a cluster may be determined using a discriminant analysis method used for image binarization processing or the like. Thereby, since the feature amount can be divided into two clusters, the feature amount can be divided into the foreground and the background.

  Further, clustering may be performed using hierarchical clustering or the like. The hierarchical clustering of the present embodiment is performed when a plurality of clusters are obtained by clustering feature amounts corresponding to a plurality of continuous distance ranges. In this case, if the number of feature amounts assigned to two clusters corresponding to two adjacent distance ranges is equal to or greater than a predetermined value, the two clusters are integrated into one cluster. When this process is repeated, a cluster of feature amounts corresponding to each of a plurality of imaging targets having different depth values is obtained. Accordingly, when a plurality of objects having different depth values exist as the foreground, a cluster of feature amounts corresponding to each of the plurality of objects can be obtained.

  In the present embodiment, the imaging target (object) with a small depth value is a foreground portion viewed from the viewpoint at the time of imaging, and the imaging target (object) with a large depth value is the background of the imaging target viewed from the viewpoint at the time of imaging. Part.

  In addition, the clustering method of the feature amount using the depth value in the present embodiment is not limited to these methods.

  The feature amount comparison unit 306 compares the feature amount of the comparison source image with the feature amount of the comparison destination image, and obtains the feature amount of the comparison destination image similar to the feature amount of the comparison source image as a pair. Specifically, the Euclidean distance between vectors of the feature amount of the comparison source image extracted by the feature amount extraction unit 302 and the feature amount of the comparison destination image is calculated. If the calculated Euclidean distance between the two feature vectors is less than or equal to a predetermined value, it is determined that the two feature values are similar, and the feature value of the comparison source image and the feature value of the comparison destination image are paired. Asking.

  The similarity determination unit 307 counts the number of pairs for each cluster of the comparison target image. Specifically, one vote is voted for the “cluster” corresponding to the paired feature amount. As a result, the number of pairs is counted for each cluster.

  Conceptually, as shown in FIG. 4, a vote value is generated for each cluster ID. The number of pairs 401 represents the vote value of the cluster ID: 002. The number of pairs 402 represents the vote value of the cluster ID: 005. In this figure, the smaller cluster ID represents the cluster with the depth closer to the front. Therefore, it can be seen that the number of pairs 401 of clusters corresponding to the foreground and the number of pairs 402 of clusters corresponding to the background are obtained separately. In this way, the number of pairs is counted for each predetermined distance range.

  The similarity determination unit 307 identifies a cluster having a predetermined number of pairs based on the cluster of the comparison destination image, and generates the similarity between the comparison source image and the comparison destination image from the number of pairs of the cluster. Specifically, the cluster having the maximum number of pairs is specified, and the number of pairs of the clusters is set as the similarity. Alternatively, it is possible to specify a cluster having the number of pairs equal to or greater than 80% of the maximum value and add the number of pairs of the specified clusters as the similarity. Alternatively, a cluster having the number of pairs equal to or greater than a predetermined threshold value may be specified, and the sum of the specified number of pairs of clusters may be used. If the similarity is greater than or equal to a predetermined value, the similarity determination unit 307 determines that the comparison target image and the comparison source image are similar.

  In addition, when it is known in advance which distance range the depth value of the target object to be confirmed is, the similarity determination unit 307 does not need to generate a vote value for each cluster ID, and corresponds to a specific distance range It is also possible to generate voting values only for the cluster IDs to be performed. In the example of FIG. 4, if the distance range is accurately known, only the vote value of the cluster ID: 002 needs to be generated. If the distance range is not accurately known, but it is known that the target object is in the foreground, only the vote values of the cluster IDs: 001 to 003 need be generated.

  In addition, although the similarity between the images calculated | required by this embodiment is the sum of the number of pairs or the number of pairs of the identified cluster, it is not restricted to this. For example, instead of counting the number of pairs, the similarity between images may be obtained using the Euclidean distance between two feature quantity vectors equal to or less than a predetermined value as a parameter.

  The collation result output unit 308 outputs the result determined by the similarity determination unit 307 to the monitor 210 or the like.

  The flow of the image matching process of the present embodiment will be described using the flowchart shown in FIG.

  In step S501, local feature amounts are extracted from the comparison source image and the comparison destination image. Detailed processing contents will be separately described with reference to FIGS. 6 and 7 as local feature amount extraction processing.

  In step S502, the feature quantities of the comparison target image (registered image) are clustered. Specifically, the depth value of the imaging target corresponding to the position of each feature point in the image is obtained from the data holding unit 303. Based on the depth value, the feature amount specifying unit 305 divides the feature amount corresponding to each feature point into clusters.

  In steps S503 to S506, a feature amount of the comparison target image similar to the local feature amount extracted from the comparison source image is found, and a process of voting one vote for the “cluster” corresponding to the found feature amount is performed. . Conceptually, as shown in FIG. 4, a vote value is generated for each cluster ID. The following will be described step by step.

  Step S503 is a loop for sequentially processing the local feature values of the comparison source image obtained in Step S502, and numbers are assigned to the feature values in order from 1. In order to refer to this using variable i, first, i is initialized to 1. Further, when i is equal to or smaller than the number of local feature values, the process proceeds to step S504. When i is not satisfied, the process exits the loop and proceeds to step S507.

  In step S504, the feature amount comparison unit 306 finds a feature amount of the comparison destination image similar to the feature amount i of the comparison source image. Specifically, the Euclidean distance is obtained between the feature amount i of the comparison source image and all the feature amounts of the comparison destination image, and the feature amount of the comparison destination image at the Euclidean distance equal to or less than a predetermined threshold is specified.

  In step S <b> 505, the similarity determination unit 307 adds the number of pairs to “cluster”. Specifically, in order to hold the information as shown in FIG. 4, a list for holding the cluster ID and the vote value is prepared in advance. Then, 1 is added to the vote value of the cluster ID corresponding to the similar feature amount obtained in step S504. This is performed for all the feature values found in step S504.

  Step S506 is the end of the loop, 1 is added to i, and the flow returns to step S503.

  In step S507, the similarity determination unit 307 generates a similarity between images. Specifically, the cluster having the maximum vote value is specified, and the vote value of this cluster is set as the similarity. Thereby, the similarity between the comparison source image and the comparison destination image is generated. If the similarity is greater than or equal to a predetermined value, the similarity determination unit 307 determines that the comparison source image and the comparison destination image are similar.

  Hereinafter, an example of “local feature extraction processing” used in the present embodiment will be described.

[Local feature extraction processing]
An example of a method for extracting a local feature amount from an image will be described with reference to FIG.

  In step S601, a luminance component is extracted from the input image, and a luminance component image is generated based on the extracted luminance component.

  In step S602, the luminance component image is sequentially reduced in accordance with the magnification (reduction rate) p, and n reduced images including the original image, which are reduced stepwise from the original size image, are generated. . Here, it is assumed that the magnification p and the number n of reduced images are determined in advance.

  FIG. 7 is a diagram illustrating an example of a reduced image obtained by the reduced image generation process. The example shown in FIG. 7 is a case where the magnification p is “2 to the power of − (1/4)” and the number n of reduced images is “9”. Of course, the magnification p is not necessarily "2 to the power of-(1/4)". In FIG. 7, an image 701 is the luminance component image generated in step S601. The reduced image 702 is a reduced image obtained by recursively reducing the luminance component image 701 four times in accordance with the magnification p. The reduced image 703 is a reduced image that is reduced from the luminance component image 701 eight times according to the magnification p.

  In this example, the reduced image 702 is an image obtained by reducing the luminance component image 701 to ½, and the reduced image 703 is an image obtained by reducing the luminance component image 701 to ¼. Any method may be used to reduce the image. In this embodiment, the reduced image is generated by a reduction method using linear interpolation.

  Next, in step S603, local feature points (local feature points) that are robustly extracted even if there is image rotation in each of the n reduced images are extracted. As a method for extracting local feature points, a Harris operator is used in the present embodiment.

  Specifically, with respect to the pixel on the output image H obtained by applying the Harris operator, the pixel values of the pixel and pixels in the vicinity of the pixel (eight pixels in total) (total nine pixels) are examined. Then, a point at which the pixel becomes a local maximum (a pixel value of the pixel becomes the maximum among the nine pixels) is extracted as a local feature point. Here, even when the pixel reaches a local maximum, it is not extracted as a local feature point if the value of the pixel is less than or equal to the threshold value.

  Note that any feature point extraction method may be used as long as it is a method capable of extracting local feature points, without being limited to the feature point extraction method using the above Harris operator.

  Next, in step S604, for each of the local feature points extracted in step S603, a feature quantity (local feature quantity) defined so as to be unchanged even when the image is rotated is calculated. As a method for calculating the local feature amount, in this embodiment, a local jet and a combination of derivatives thereof are used.

  Specifically, the local feature amount V is calculated by the following equation (1).

  However, the symbols used on the right side of the equation (1) are defined by the following equations (2) to (7). Here, G (x, y) on the right side of Expression (2) is a Gaussian function, I (x, y) is a pixel value at image coordinates (x, y), and “*” is a symbol representing a convolution operation. is there. Equation (3) is a partial derivative of variable L defined by equation (2) with respect to x, and equation (4) is a partial derivative of variable L with respect to y. Equation (5) is the partial derivative of variable Lx defined in equation (3) with respect to y, equation (6) is the partial derivative of variable Lx defined in equation (3) with respect to x, and equation (7) is the equation. It is a partial derivative with respect to y of Ly defined in (4).

  In addition, as long as a local feature value can be calculated, the feature value calculation method is not limited to the above-described feature value calculation method, and any feature value calculation method may be used.

  As described above, the local feature amount can be extracted from the target image.

  The image processing apparatus according to the present embodiment is configured as an image collation apparatus that obtains a similarity between two images and outputs a result of similarity determination between the two images. With the configuration of the present embodiment, it is possible to generate a cluster of feature quantities corresponding to the imaging target based on the depth value of the imaging target in the comparison target image. Can be improved. The collation accuracy is a determination accuracy when it is determined whether or not the two images of the comparison source image and the comparison destination image are similar.

[Second Embodiment]
An example in which the image processing apparatus of this embodiment is used as an image search apparatus will be described. An image processing apparatus registers an image having depth information as a registered image or a search destination image, and searches for an image similar to a query image or a search source image from a plurality of registered images or search destination images. It is. The image collation device of the first embodiment has one comparison destination image, but there are a plurality of registered images or search destination images of the image retrieval device of the present embodiment. Here, description of the common parts of the image search apparatus of the present embodiment and the image collation apparatus of the first embodiment is omitted, but different parts will be described below.

  Specifically, when registering a registered image or a search destination image, the feature amount of the registered image is clustered using depth information (depth value), and the feature amount is stored for each cluster. At the time of search, the number of feature values of registered images that are paired with feature values of query images is counted for each cluster. Next, a cluster having a predetermined number of pairs is obtained, and similarity between images is generated using the obtained number of pairs of clusters. By outputting a list of registered images in descending order of similarity between images, an image search for a query image is realized.

  In the present embodiment, the registered image uses an image having depth information, but the query image may not have depth information.

[Configuration of image search device]
Hereinafter, the configuration of the image search apparatus of the present embodiment will be described with reference to FIG.

  The processing of the image input unit 801, feature amount extraction unit 802, and data holding unit 803 of the image search device of FIG. 8 is the same as the processing of the image input unit 301, feature amount extraction unit 302, and data holding unit 303 of the image matching device of FIG. Is the same. Further, the processing of the feature amount specifying unit 805, the feature amount comparing unit 806, the similarity determining unit 807, and the processing result output unit 808 of FIG. The processing is the same as the processing result output unit 308. In the present embodiment, description of these components is omitted. The image index unit 804 will be described.

  The image index unit 804 manages the registered image feature amount, image ID, and cluster ID in association with each other.

  An example of the data structure of the data managed by the image index unit 804 of this embodiment is shown in FIG. The image ID and the cluster ID are managed in association with the quantized feature amount. The quantized feature value is generated by a “local feature value quantization process” to be described later. When the quantized values are the same, it can be determined that the local feature amount has a predetermined degree of similarity or more. In the following description, one line shown in FIG. 9A is called a record.

  In the data structure of the data managed by the image index unit 804 of the present embodiment, a plurality of records having the same quantization value are created. To avoid this, “quantity ID and cluster” are used for the quantization value. A list of “ID pairs” may be associated. Furthermore, when there are a plurality of feature quantities having the same quantization value in the same cluster of the same image, the same set appears multiple times. To avoid this, a list of “a set of image ID, cluster ID, and number” may have a data structure that is associated with a quantized value. The data structure of data managed by the image index unit 804 is not limited to these. For example, the cluster ID of the quantized feature value can be managed in association with each of a plurality of distance ranges to which the depth value belongs.

  Note that the data managed by the image index unit 804 is held in the data holding unit 803, so that the image index unit 804 can be configured as a part of the data holding unit 803.

  The feature amount comparison unit 806 obtains a feature amount of the registered image similar to the feature amount of the query image as a pair. Specifically, the feature amount of the query image is quantized, and a record having the same quantized value is specified from the image index unit 804. As a result, a cluster of registered images including a feature amount similar to the feature amount of the query image is specified.

  The similarity determination unit 807 counts the number of pairs obtained by the feature amount comparison unit 806 for each set of image ID and cluster ID. Specifically, using the feature amount comparison unit 806, a feature amount similar to the local feature amount extracted from the query image is found from the image index unit 804. Next, one vote is voted for “a set of image and cluster” corresponding to the found feature amount. As a result, the number of pairs is counted for each set of image ID and cluster ID.

  Conceptually, as shown in FIG. 9B, a vote value is generated for each image ID and cluster ID. The number of pairs 901 represents the vote value of the cluster ID: 002 of the image ID: 001. The number of pairs 902 represents the vote value of the cluster ID: 005 of the image ID: 001. In this figure, the smaller cluster ID represents a cluster with a smaller depth value (the imaging target is in front). Therefore, in the example of FIG. 9, it can be seen that the number 901 of pairs of clusters corresponding to the foreground and the number 902 of pairs of clusters corresponding to the background are obtained separately. In this way, the number of pairs is counted for each predetermined distance range.

  The similarity determination unit 807 identifies clusters having a predetermined number of pairs for each image ID, and generates the number of pairs of the identified clusters as the similarity between the query image and the registered image. Specifically, the cluster having the maximum number of pairs is specified, and the number of pairs of the specified clusters is set as the similarity. For example, when the similarity determination unit 807 obtains the number of pairs for each cluster as shown in FIG. 9B, the maximum number of pairs is generated as the similarity for each image ID as shown in FIG. 9C. Is done.

  Or it is good also considering the cluster which has the number of pairs more than the value equivalent to 80% of the maximum value of the number of pairs, and adding the number of pairs of the identified cluster as a similarity. Alternatively, a cluster having the number of pairs equal to or greater than a predetermined threshold value may be specified, and the sum of the specified number of pairs of clusters may be used.

  The processing result output unit 808 can output the similarity obtained for each image ID when the similarity determination unit 807 applies the above processing to all of the image IDs. Alternatively, the processing result output unit 808 may output images or image IDs in descending order of similarity.

[Image registration process]
Next, the image registration process will be described with reference to the flowchart of FIG. When this process is executed, a registered image and an image ID of the registered image are given. Then, the local feature amount extracted from the registered image is divided into clusters and stored in the image index unit 804. Specific control contents will be described with reference to FIG.

  In step S1001, a local feature amount is extracted from the registered image. Characteristic points (local feature points) are extracted from the image. Next, a feature amount (local feature amount) corresponding to the local feature point is calculated based on the local feature point and surrounding image information. The local feature amount extraction process is the same as that in the first embodiment.

  In step S1002, the local feature is quantized. Quantization is performed so that similar feature quantities have the same quantization value. Specific processing contents will be described later with reference to FIG. 12 as local feature quantization processing.

  In step S1003, feature quantities are clustered. Specifically, the depth information of the position of each feature point is obtained by the distance information acquisition unit 702. Then, the feature value is divided into clusters by the feature value specifying unit 805 using the depth value.

  In step S1004, the feature amount is registered in the image index unit 804. Specifically, an ID is assigned to the cluster obtained in step S1003. For example, consecutive integers are assigned as cluster IDs by adding 1 to 1 in order from a cluster of feature values corresponding to a distance range having a small depth value (an imaging target is in front). Then, the image index unit 804 stores the quantization value of each feature value, the image ID, and the cluster ID in association with each other.

[Image search processing]
Next, image search processing will be described with reference to the flowchart of FIG. A query image is given when this process is executed. Then, local feature amounts are extracted from the query image, and the number of registered image feature amounts paired with the query image feature amount is counted for each cluster. Next, a cluster having a predetermined number of pairs is obtained, and similarity between images is generated using the number of pairs of the clusters. A list of combinations of similarity and image ID of each image is generated as a search result. Specific control contents will be described with reference to FIG.

  In step S1101, a local feature amount is extracted from the query image. The local feature amount extraction process is the same as that in the first embodiment.

  In step S1102, the local feature is quantized. Specific processing contents will be described later with reference to FIG. 12 as local feature quantization processing.

  In step S1103 to step S1106, a feature amount similar to the local feature amount extracted from the query image is found from the image index unit 804, and one vote is voted for “a set of image and cluster” corresponding to the found feature amount. Perform the process. Conceptually, as shown in FIG. 9B, a vote value is generated for each image ID and cluster ID. Each step will be described below.

  Step S1103 is a loop for sequentially processing the quantized local feature quantities of the query obtained in step S1102, and numbers are assigned to the feature quantities in order from 1. In order to refer to this using the variable i, first, i is initialized to 1. Further, when i is equal to or smaller than the number of local feature values, the process proceeds to step S504. When i is not satisfied, the process exits the loop and proceeds to step S1107.

  In step S1104, the feature amount comparison unit 806 finds the feature amount of the registered image similar to the feature amount i of the query image. Specifically, the quantization feature amount of the image index unit 804 shown in FIG. 9A is scanned to identify a record that matches the quantization feature amount i of the query. When there are a plurality of records having the same quantized value, a plurality of records are obtained.

  In step S <b> 1105, the similarity determination unit 807 adds the number of pairs to “a set of images and clusters”. Specifically, in order to hold information as shown in FIG. 9B, a list for holding an image ID, a cluster ID, and a vote value is prepared in advance. Then, 1 is added to the vote value of the set of the image ID and cluster ID of the record obtained in step S1104. This is performed for all the records found in step S1104.

  Step S1106 is the end of the loop, 1 is added to i, and the flow returns to step S503.

  The processes in steps S1107 to S1109 are performed by the similarity determination unit 706 and generate a similarity with the query image. Conceptually, as shown in FIG. 9C, a cluster having a predetermined vote value is specified, and the similarity is generated using the vote value of the cluster. Each step will be described below.

  Step S1107 is a loop for sequentially processing the image IDs obtained in steps S1103 to S1106. Assume that numbers are assigned in order from 1 to the image IDs obtained in steps S1103 to S1106. In order to refer to this using the variable i, first, i is initialized to 1. Furthermore, when i is equal to or less than the number of image IDs obtained in steps S1103 to S1106, the process proceeds to step S1108, and when this is not satisfied, the process exits the loop and proceeds to step S1110.

  In step S1108, the similarity determination unit 706 generates the similarity of the i-th image ID. Specifically, a cluster having the largest vote value in the i-th image ID is specified, and the vote value of the cluster is set as the similarity. This similarity is stored in association with the image ID.

  Step S1109 is the end of the loop, 1 is added to i, and the flow returns to step S1107.

  In step S1110, the image IDs stored in step S1108 are sorted in descending order based on the similarity, and the processing result output unit 808 lists the sorted images or information for identifying the images (image IDs or file names). Is output to the monitor 210 or the like as a processing result. Thereby, the image IDs of registered images similar to the query image can be obtained in the order of similarity. Note that the processing result output unit 808 may output not only all images specified by the image ID stored in step S1108 but only registered images having a similarity equal to or higher than a predetermined value as processing results.

  In step S1108, the cluster having the maximum vote value is specified, and the vote value of the cluster is set as the similarity. However, the object shown in the query may exist in a distance range (for example, foreground and background) in which the depth value of the imaging target is different on the registered image side. At this time, if only the maximum voting value is taken, the similarity is determined by matching with either the foreground object or the background object. Therefore, a cluster having a predetermined number or more of voting values may be specified, and the sum of the voting values of the clusters may be used as the similarity. For example, a cluster having a vote value equal to or greater than a value corresponding to 80% of the maximum value can be specified, and the sum of the vote values of the specified cluster can be used as the similarity.

  As a result, an appropriate similarity can be obtained even when it is desired to search for an image in which a plurality of similar objects appear in each of the foreground and the background.

  FIG. 9B also shows a set of an image ID and a cluster ID with a vote value of 0. However, the list that holds the image ID, the cluster ID, and the vote value used in step S1105 may not be held on the list when the vote value is 0. In this way, the memory usage can be reduced. Conversely, an associative array that enables referencing of vote values for all combinations of image IDs and cluster IDs may be configured in advance. At this time, the voting value can be specified at high speed, and speeding up of the voting process can be expected. In this case, since the number of clusters needs to be known for each image, the image index unit 804 may manage the number of clusters simultaneously. Alternatively, the number of clusters may be managed by another means.

  In the following, an example of “local feature quantization processing” used in the present embodiment will be described.

[Local feature quantization]
In order to facilitate matching between local feature quantities, the local feature quantities are quantized.

For example, the local feature amount is represented as an N-dimensional vector V, and each dimension is represented as Vn. At this time, the quantization of the Kn gradation can be performed by the following formula (8) for the feature quantity of the nth dimension. N and Kn are predetermined values.
Qn = (Vn * Kn) / (Vn_max−Vn_min + 1) (8)
Here, Qn is a value obtained by quantizing the feature quantity Vn of the nth dimension among the N dimensions. Vn_max and Vn_min are a maximum value and a minimum value that can be taken by the feature quantity of the nth dimension, respectively.

  In the above quantization, the number of quantization gradations is determined for each dimension, but a common number of gradations may be used in all dimensions. As shown in FIG. 12, this quantization method is a method of dividing the feature amount space into a lattice pattern. In this figure, a lattice 1201 represents a quantization region in the feature amount space, and a point 1202 represents each feature. FIG. 12 shows an example in which a two-dimensional feature amount space is quantized and divided, and the division is performed by expanding this to multi-dimensions corresponding to the number of dimensions of the local feature amount.

  Further, as long as it is a method that can divide the feature amount space, any division method can be applied without being limited to the method of quantizing based on the rules as described above. For example, a clustering rule may be created by machine learning of a plurality of images, and the feature amount space may be divided and quantized according to the rule.

In addition, after performing quantization for each dimension, the quantization value group is labeled according to the following equation (9), so that it can be handled substantially equivalent to a one-dimensional feature amount. .
IDX = Q ₁ + Q ₂ * K ₁ + Q ₃ * K ₁ * K ₂ + ... + Q _n * K ₁ * K ₂ * ... * K _n-1 (9)
Further, when the number of gradations of all dimensions is common, the quantization value group can be labeled by the following equation (10). Here, K is the number of gradations.

  Note that any labeling method may be used as long as the calculation method allows labeling, without being limited to the above-described calculation method.

  By configuring a database or the like that can search for an image ID or the like using a quantized value as a key, matching between local feature quantities can be performed at high speed. This is called an image index.

  With the image search device of the present embodiment, as shown in FIG. 9C, the vote value of the number of pairs is obtained separately for each of a plurality of clusters corresponding to a plurality of distance ranges in the depth direction. Therefore, when the image of the object in the foreground of the registered image is a query image, the “number of matches with the foreground feature quantity (number of pairs)” and “number of coincidence with the background feature quantity” are separated. can get. Since the chance coincidence is smaller than the correct coincidence, it is possible to obtain the vote value obtained by the correct coincidence by specifying the cluster having the maximum value. Accordingly, it is possible to improve the search accuracy for searching for an image similar to the query image or the search source image from the registered image or the search destination image.

  For example, in the example shown in FIG. 9C, conventionally, the similarity between the image ID 001 and the image ID 002 is 260, which is the same. However, in the image search apparatus of the present embodiment, the similarity between the image ID 001 and the image ID 002 is different from 200 and 180, and it can be seen that the image ID 001 is more similar.

[Third Embodiment]
In the first embodiment or the second embodiment, if the depth values are gradationized and the feature amounts are clustered, the feature amounts of the object may not be contained in one cluster. For example, when an object having a depth such as a car is shown in the query image, there is a possibility that the feature amount of the car is assigned to a plurality of clusters corresponding to two or more distance ranges. Therefore, an image search apparatus that identifies clusters whose vote values obtained by adding the vote values of neighboring clusters to a predetermined value or more and generates the similarity between the query image and the registered image will be described. Note that, as in the first embodiment or the second embodiment, each of the plurality of clusters of feature amounts corresponds to a distance range.

  The configuration of the image search apparatus of the present embodiment is the same as the configuration of FIG. 8 shown in the second embodiment. However, the operation of the similarity determination unit 807 is different. The similarity determination unit 807 of this embodiment adds the voting values of neighboring clusters for each image ID. Next, a cluster having a vote value greater than or equal to a predetermined value is specified, and the similarity between the query image and the registered image is generated from the cluster.

  For example, when the similarity determination unit 807 obtains voting values for the set of images and clusters shown on the upper side of FIG. 14, the voting values of neighboring clusters are added as shown on the lower side of FIG. In this example, the vote values of three clusters are added together on the left and right. However, when there is no cluster on the left like cluster ID: 001, the vote value in that direction is ignored and only the vote values of the existing clusters are added. In this way, the vote values of neighboring clusters are added together. Thereafter, as in the first embodiment, a cluster having a vote value greater than or equal to a predetermined value is specified, and the similarity between the query image and the registered image is generated from the cluster. Specifically, as in the first embodiment, the maximum value of the identified cluster is used as the similarity. Alternatively, it is also possible to specify a cluster having the number of pairs equal to or greater than the value corresponding to 80% of the maximum value and add the number of pairs of the clusters as the similarity. Alternatively, a cluster having the number of pairs equal to or larger than a predetermined threshold value may be specified and the number of pairs of the clusters may be added.

  Next, image search processing according to the present embodiment will be described with reference to a flowchart of FIG. The flowchart of FIG. 13 is different in that processing is added between step S1106 and step S1107 of the flowchart of FIG. 11 of the first embodiment. Therefore, step S1301 to step S1306 are the same as step S1101 to step S1106. Similarly, steps S1312-S1315 are the same as steps S1107-S1110. Hereinafter, newly added processing steps S1307 to S1311 will be described step by step.

  Steps S1307 to S1311 add the voting values of neighboring clusters for each image ID. It is assumed that the number of neighbors is given in advance. In this example, three clusters including the front and rear are assumed to be neighbors. Each step will be described below.

  Step S1307 is a loop for sequentially processing the image IDs obtained in steps S1303 to S1306. Assume that the image IDs obtained in steps S1303 to S1306 are assigned numbers in order from 1. In order to refer to this using the variable i, first, i is initialized to 1. Further, when i is equal to or less than the number of image IDs obtained in steps S1303 to S1306, the process proceeds to step S1308, and when this is not satisfied, the process exits the loop and proceeds to step S1312.

  Step S1308 is a loop for sequentially processing the cluster of the image i. Assume that the cluster IDs are assigned numbers in order from 1. In order to refer to this using the variable j, j is first initialized to 1. Further, when j is equal to or less than the number of clusters, the process proceeds to step S1309, and when this is not satisfied, the process exits the loop and proceeds to step S1311.

  In step S1309, the number of pairs in the vicinity of cluster j of image i is added. For example, as shown in the example of FIG. 14 described above, when the cluster j is the cluster ID: 002, the number of pairs of the cluster IDs: 001 to 003 is added in order to add up the number of pairs of neighboring three clusters. This is stored in a list that stores separately prepared image IDs, cluster IDs, and the number of pairs. This list is used in step S1312 and subsequent steps.

  Step S1310 is the end of the loop of the cluster, and 1 is added to j, and the process returns to step S1308.

  Step S1311 is the end of the image loop, and 1 is added to i, and the flow returns to step S1307.

  In the present embodiment, it is assumed that the number of neighbors is given in advance. However, this may be obtained from the query image. For example, when the query image also has a depth value, the depth value is clustered to obtain the thickness of the object from the distance range corresponding to the cluster of the feature amount group of the object shown in the query image. Alternatively, the thickness of the cluster having the maximum number of features is obtained. The number of neighbors can be obtained by dividing the thickness of the object in the obtained query image by the gradation width. For example, when the car is shown in the query image, the thickness of the object obtained by clustering is 3 m, and the gradation width is 1 m, the number of neighbors can be calculated as 3.

  Alternatively, the object type of the query image may be obtained by a technique such as image recognition, and the number of neighbors may be obtained using the thickness suitable for the object type. For example, when a person is detected from a query image as a result of applying human body detection or the like, it is conceivable to use a general person thickness prepared in advance. Or you may make it obtain simply by a user input with a query image. The method for specifying the thickness of the object in the present embodiment is not limited to these.

  This corresponds to the case where the image search apparatus of the present embodiment has a gradation width when gradation of depth values is smaller than the thickness of the object, and the feature amount of the same object is divided and registered in different clusters. At this time, by adding the number of pairs of neighboring clusters, the similarity can be generated based on the number of pairs obtained appropriately. For example, it is assumed that the distance (gradation width) of the distance range corresponding to the cluster registered in the image index unit 804 is 1 m. At this time, an object such as a car has a width of about 3 m in the depth direction. As a result, car features are registered separately in several clusters. In the first embodiment or the second embodiment, the similarity is determined from one of the divided clusters. However, in this embodiment, the number of pairs of neighboring clusters is added. Therefore, if the number of neighbors is 3, the number of pairs can be obtained with a depth width of about 3 m. Therefore, the number of pairs that correctly matches the feature amount of the entire vehicle is obtained.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 301 Image input part 302 Feature-value extraction part 303 Data holding part 305 Feature-value specific | specification part 306 Feature-value comparison part 307 Similarity determination part 308 Processing result output part

Claims (13)

Extraction means for extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding means for holding depth information of the imaging target corresponding to each feature point of the second image;
A specifying means for specifying at least one distance range based on the depth information;
A determination unit that determines that the second image is similar to the first image if the similarity between the second local feature and the first local feature obtained for the distance range is equal to or greater than a predetermined value. When,
If at least two or more of the similarities between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges are equal to or greater than the predetermined value, the determination unit Determines whether the second image is similar to the first image based on a total of the similarities equal to or greater than the predetermined value . Extraction means for extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding means for holding depth information of the imaging target corresponding to each feature point of the second image;
Based on the depth information, the specifying means for specifying at least one distance range, and the similarity between the second local feature and the first local feature obtained for the distance range is a predetermined value or more. If there is a determination unit that determines that the second image is similar to the first image,
With
If at least two or more of the similarities between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges are equal to or greater than the predetermined value, the determination means obtained for each two or more of the distance range to fit, on the basis of the sum of the similarity between the first local feature quantity and the second local features, the second image is similar to the first image An image processing apparatus characterized by determining whether or not.   The one distance range specified by the specifying means corresponds to a foreground part or a background part in the second image, and a similarity between the foreground part or the background part and the first image is equal to or greater than the predetermined value. The image processing apparatus according to claim 1, wherein the determination unit determines that the second image is similar to the first image.   The plurality of distance ranges specified by the specifying means correspond to the foreground part and the background part in the second image, the similarity between the foreground part and the first image, the background part and the first image 2. The image processing according to claim 1, wherein the determination unit determines that the second image is similar to the first image if any of the similarity to the first image is equal to or greater than the predetermined value. apparatus.   The plurality of distance ranges specified by the specifying means correspond to the foreground part and the background part in the second image, the similarity between the foreground part and the first image, the background part and the first image 2. The image processing apparatus according to claim 1, wherein the determination unit determines that the second image is similar to the first image if the similarity to each other is equal to or greater than the predetermined value.   The holding means includes information for specifying the second local feature amount and the second image for each of the plurality of feature points of the second image extracted by the extraction means, and the each of the second images. The image processing apparatus according to claim 1, wherein the depth information of the imaging target corresponding to a feature point is held in association with each other.   The specifying means specifies the second local feature quantity corresponding to the distance range, and the holding means holds the second local feature quantity corresponding to the distance range and the distance range in association with each other. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   8. The similarity between the second local feature and the first local feature is the number of pairs of the first local feature that is similar to the second local feature. The image processing apparatus according to any one of the above.   9. The depth information is generated by a generation unit outside the image processing apparatus, and is a depth value indicating a distance from the generation unit to the imaging target. An image processing apparatus according to 1. An extraction step of extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding the depth information of the imaging target corresponding to each feature point of the second image in a holding unit;
A specific step of identifying at least one distance range based on the depth information;
A determination step of determining that the second image is similar to the first image if the similarity between the second local feature value and the first local feature value obtained for the distance range is equal to or greater than a predetermined value. When,
With
If at least two or more of the similarities between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges are greater than or equal to the predetermined value, in the determination step, An image processing method comprising: determining whether the second image is similar to the first image based on a sum of the similarities equal to or greater than a predetermined value . An extraction step of extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding the depth information of the imaging target corresponding to each feature point of the second image in a holding unit;
Based on the depth information, the specific step of specifying at least one distance range, and the similarity between the second local feature amount and the first local feature amount obtained for the distance range is a predetermined value or more. If there is a determination step for determining that the second image is similar to the first image;
With
If at least two or more of the similarities between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges are equal to or greater than the predetermined value, in the determination step, obtained for each two or more of the distance range to fit, on the basis of the sum of the similarity between the first local feature quantity and the second local features, the second image is similar to the first image An image processing method characterized by determining whether or not. An extraction step of extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding the depth information of the imaging target corresponding to each feature point of the second image in a holding unit;
A specific step of identifying at least one distance range based on the depth information;
Determination step of determining that the second image is similar to the first image if the similarity between the second local feature value and the first local feature value obtained for the distance range is equal to or greater than a predetermined value. When,
To the computer,
Wherein in the determination step, was obtained for each of a plurality of said distance range, said second local characteristic amount and the first local feature quantity and the similarity of at least two of, if the predetermined value or more, the A program for determining whether the second image is similar to the first image based on a sum of the similarities equal to or greater than a predetermined value . An extraction step of extracting a first local feature amount for each of a plurality of feature points of the first image and a second local feature amount of each of a plurality of feature points of the second image obtained by imaging the imaging target;
Holding the depth information of the imaging target corresponding to each feature point of the second image in a holding unit;
Based on the depth information, the specifying step for specifying at least one distance range, and the similarity between the second local feature and the first local feature obtained for the distance range is a predetermined value or more. If there is a determination step for determining that the second image is similar to the first image;
To the computer,
In the determination step , if at least two or more of the similarities between the second local feature quantity and the first local feature quantity obtained for each of the plurality of distance ranges are equal to or greater than the predetermined value, obtained for each two or more of the distance range to fit, on the basis of the sum of the similarity between the first local feature quantity and the second local features, the second image is similar to the first image A program characterized by determining whether or not.

Images