JP2016103084A - Image search apparatus and image search system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image search apparatus configured to prevent reduction in search accuracy even when similarity between categories is high.SOLUTION: An image search apparatus includes: a search object storage section which stores feature vectors of images to be searched; a dictionary information storage section for storing a first feature vector corresponding to each of search keys; a first search section 305 which calculates first similarity between a feature vector of an image and the first feature vector corresponding to an input search key; a second search section 306 which calculates second similarity between the feature vector of the image and a second feature vector, which is a first feature vector corresponding to a search key other than the input search key, to calculate general similarity of feature vector images, on the basis of the first and second similarities; and an output section 307 which outputs the feature vector images in the order of general similarity. The second search section 306 calculates general similarity so that a feature vector having higher second similarity may have lower general similarity of feature vectors.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image search apparatus that searches for an image corresponding to a search key.

  In recent years, with the increasing awareness of safety, the digitalization of cameras, and the increase in the scale of recording devices, video surveillance systems have become popular. Cameras installed in various places shoot various objects such as people, cars, motorcycles, signs, or products in stores.

  There is an increasing need for an image search system in order to efficiently manage enormous amounts of image data acquired from a large number of cameras. The image search system detects a specific object such as a face from an image, calculates a feature amount from the detected specific object, and creates a database. When data that the user wants to search is input to the image search system, the image search system calculates the similarity between the feature amount corresponding to the input data and the feature amount in the database. Search for images similar to the data.

  In an image search system, there is a need to search only images of objects belonging to a specific category from a large number of recorded images. For example, when the object in the image is a vehicle, there is a need to search only a vehicle in a specific category from a category such as a sedan, a wagon, or a compact car. In addition, when an object in an image is a person, there is a need for a search using an attribute such as a teenager or a twenties as an attribute, and a search using a physique such as large or small.

  Two methods are known as methods for realizing this need.

  In the first method, when an image is registered, a discriminator such as Support Vector Machine is used to determine which category the image belongs to, and the determination result is associated with a feature amount as tag information. sign up. By searching for tag information that matches the input data, an image belonging to a desired category can be acquired. However, in this method, it is necessary to execute determination processing at the time of data registration, and there is a problem that it takes too much time to register data when a large number of images are input from a plurality of cameras.

  In the second method, the category determination process is not executed at the time of data registration, but only the image feature amount is registered in the database, and the category determination process is executed at the time of retrieval. This method requires time for search processing, but does not execute category determination processing for data registration, and is therefore suitable for a system in which enormous images are input from a plurality of cameras in real time.

  As a technique for realizing such an image search with high accuracy, JP 2013-246739 A (Patent Document 1) and JP 2011-107795 A (Patent Document 2) are known.

  The gazette of Japanese Patent Application Laid-Open No. 2005-228561 uses a similar vector search means for searching for a set D of reference content feature vectors similar to each feature vector q i of query content and each feature vector of query content using a mixture parameter λ. qi is based on a probability ratio between a probability λ · p (qi | Rj) generated from each reference content and a probability (1-λ) · p (qi) generated from a background model unrelated to the reference content. And adding voting means for each reference content for all the feature vectors qi of the query content, and having a voting unit that uses the reference content with the higher score as a search result. (See summary).

  Patent Document 2 states that “a system that realizes the present invention is a means for generating a large number of image feature quantity vectors from a single image and a small number of generated image feature quantity vectors by clustering processing. A means for aggregating each representative vector, and a means for performing a similar vector search using each aggregated representative vector as a search query and obtaining closest data of each image feature vector generated in advance ”. (See summary).

JP 2013-246739 A JP 2011-107995 A

  Patent Document 1 is a technique for modeling the features of a background portion unrelated to a search target. Patent Document 2 registers a small number of feature vectors as representative vectors for each category in a dictionary database, and uses a small number of representative vectors. This is a technique for speeding up the search process by searching.

  For example, if the similarity between categories is high, such as between the category “sedan” and the category “compact car” in the type of vehicle, the image of the category “sedan” is searched, but the “compact car” There is a problem that the search accuracy is lowered, such that an image appears at the top of the search result.

  It is an object of the present invention to provide an image search apparatus that suppresses a decrease in image search accuracy even when the similarity between categories is high.

  If a typical example of the present invention is shown, a search target storage unit that stores a feature vector of an image to be searched, search keys for searching the search target storage unit, and first search keys corresponding to the search keys. A dictionary information storage unit in which one feature vector is stored in association with each other, a search key input receiving unit that receives an input of a search key, and a first feature vector corresponding to the input search key from the dictionary information storage unit A first search unit that calculates the first similarity between the acquired first feature vector and the feature vector of the image stored in the search target storage unit, and a search key other than the input search key Calculate a second similarity between the second feature vector corresponding to the first feature vector and the feature vector of the image stored in the search target storage unit, and based on the first similarity and the second similarity The above A second search unit for calculating a total similarity indicating a similarity between the image of the collection vector and the input search key, and an image of feature vectors stored in the search target storage unit arranged in the order of the total similarity and output And the second search unit calculates the overall similarity so that the feature vector having the higher second similarity has a lower overall similarity of the feature vector. .

  The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows. That is, it is possible to provide an image search apparatus that suppresses a decrease in image search accuracy even when the similarity between categories is high.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is explanatory drawing of the whole structure of the search system of Example 1. FIG. 1 is a hardware configuration diagram of an image search apparatus according to Embodiment 1. FIG. FIG. 3 is a functional block diagram of a category search unit according to the first embodiment. 6 is an explanatory diagram of processing of an image registration unit of Embodiment 1. FIG. It is explanatory drawing of the dictionary database (DB) of Example 1. FIG. It is explanatory drawing of distribution on the feature space of the feature vector calculated from the image of each category of Example 1. FIG. It is explanatory drawing of the 2nd search vector determined using the 1st method of Example 1. FIG. It is explanatory drawing of the 2nd search vector determined using the 2nd method of Example 1. FIG. It is explanatory drawing of the search result table of Example 1. FIG. It is a flowchart of the 2nd search process performed by the 2nd search part of Example 1. FIG. It is explanatory drawing of the search process by the image search device of Example 1. It is explanatory drawing of the search result confirmation screen of Example 1. FIG. 10 is a flowchart of a second search process by a second search unit according to the second embodiment. 12 is a flowchart of a second search process according to a modification of the second embodiment. FIG. 10 is an explanatory diagram of a reduced screen display according to the third embodiment. It is explanatory drawing of the search result confirmation screen of Example 4. FIG.

  In the present embodiment, an example of an image search apparatus that searches for an image in consideration of the similarity with a feature vector corresponding to a search key other than the search key will be described.

  FIG. 1 is an explanatory diagram of the overall configuration of the search system according to the first embodiment.

  The search system includes a terminal device 101, a camera 102, and an image search device 200. The terminal device 101, the camera 102, and the image search device 200 are connected via the network 103, respectively. The terminal device 101 manages the entire search system. Specifically, the terminal device 101 instructs setting of the camera 102, transmits a search instruction to the image search device 200, and displays a search result from the image search device 200. An image taken by the camera 102 is transmitted to the image search device 200 via the network 103, and the image search device 200 stores the received image. Note that an image captured by the camera 102 may be input to the terminal device 101 offline, and the terminal device 101 may transmit the input image to the image search device 200 via the network 103.

  Based on the search instruction from the terminal apparatus 101, the image search apparatus 200 searches the stored image for an image corresponding to the search key included in the search instruction, and transmits the search result to the terminal apparatus 101. In this embodiment, a vehicle is described as an example of an object to be searched, but the present invention is not limited to this. The object to be searched may be a person, an animal, a plant, a mark, or the like. When the object to be searched is a vehicle, the search key included in the search instruction is, for example, a vehicle category such as a sedan or a wagon, but is not limited thereto.

  The number of cameras 102 is not limited to the number shown in FIG. 1, and the search system only needs to have at least one camera 102. In addition, the terminal device 101 and the image search device 200 may be mounted on one housing.

  FIG. 2 is a hardware configuration diagram of the image search apparatus 200 according to the first embodiment.

  The image search apparatus 200 includes an interface (I / F) 210, a CPU (processor) 220, an auxiliary storage device 230, a network interface (I / F) 240, and a memory 250.

  The I / F 210 is an interface connected to an input device such as a keyboard and a mouse (not shown) and an output device such as a display (not shown). The CPU 220 executes various programs stored in the memory 250 and reads / writes data stored in the memory 250. The auxiliary storage device 230 is a non-volatile storage area, and the auxiliary storage device 230 stores a search target database (DB) 302 and a dictionary database (DB) 304. In the search target DB 302, an image received by the image search device 200 and a feature vector calculated from the image are registered. Note that the feature vector of the image registered in the search target DB 302 is a search target for the search key. The feature vector indicates at least one entire feature amount calculated from the image. The dictionary DB 304 in which a representative vector (first feature vector) and a second search vector (second feature vector) corresponding to each search key category are registered in the dictionary DB 304 will be described in detail with reference to FIG. The search target DB 302 and the dictionary DB 304 may register information in a table format or the like, and are not limited to the database format. Therefore, in the claims, the search target DB 302 is described as a search target storage unit, and the dictionary DB 304 is described as a dictionary information storage unit.

  The network I / F 240 is an interface that connects the image search apparatus 200 to the network 103. The memory 250 is a volatile storage area, and the category search unit 300 is stored in the memory 250. The category search unit 300 is stored in the auxiliary storage device 230, and the CPU 220 loads the category search unit 300 from the auxiliary storage device 230 to the memory 250 and executes the category search unit 300. When the category search unit 300 is executed by the CPU 220, a search process for searching for an image corresponding to the search key included in the search instruction is executed.

  The category search unit 300 includes an image registration unit 301, a search key selection unit 303, a first search unit 305, a second search unit 306, and an output unit 307. Details of these will be described with reference to FIG.

  FIG. 3 is a functional block diagram of the category search unit 300 according to the first embodiment.

  The image registration unit 301 receives an image photographed by the camera 102, calculates at least one feature amount of the received image, and registers it in the search target DB 302 as a feature vector of the calculated feature amount. The processing of the image registration unit 301 will be described in detail with reference to FIG.

  The search key selection unit 303 acquires from the dictionary DB 304 a category representative vector to be a search key included in the search instruction from the terminal device 101. In the dictionary DB 304, feature vectors calculated from a plurality of images of each category are registered as representative vectors. For example, when the search key is “sedan”, the search key selection unit 303 acquires a representative vector whose category is “sedan”.

  The first search unit 305 calculates a distance value (first distance value) between the representative vector selected by the search key selection unit 303 and each feature vector registered in the search target DB 302 as a first similarity. A smaller first distance value indicates that the image of the feature vector is similar to the category of the search key. The second search unit 306 acquires the second search vector of the category of the search key from the dictionary DB 304, and sets the second distance value between the acquired second search vector and each feature vector registered in the search target DB 302 to the second similarity. Calculate as degrees. A smaller second distance value indicates that the image of the feature vector is similar to the category of the second vector. Then, the second search unit 306 integrates each feature vector registered in the search target DB 302 based on the first similarity calculated by the first search unit 305 and the second similarity calculated by the second search unit 306. A distance value (total similarity) search result is output to the output unit 307. For example, the second search vector whose category is “sedan” is a representative vector of another category (such as “compact car” and “wagon”). The processing of the second search unit 306 will be described in detail with reference to FIGS. The total distance value indicates the similarity of the feature vector image with respect to the search key category in consideration of the first distance value and the second distance value. In the present embodiment, the first similarity, the second similarity, and the total similarity are expressed using distance values, but may be expressed using other parameters. In the present embodiment, since the distance value is used for the first similarity, the second similarity, and the overall similarity, the smaller the distance value is, the higher the similarity with respect to the category of the image of the feature vector is. Depending on the parameters used for the first similarity, the second similarity, and the overall similarity, the larger the value of the parameter, the higher the similarity of the feature vector to the image category may be.

  Based on the search result input from the second search unit 306, the output unit 307 displays search result confirmation screen data in which images corresponding to each feature vector registered in the search target DB 302 are rearranged in ascending order of the total distance value. The generated search result confirmation screen data is transmitted to the terminal device 101. When receiving the search result confirmation screen data, the terminal device 101 displays the search result confirmation screen 1000. The search result confirmation screen 1000 will be described in detail with reference to FIG.

  The first search unit 305 calculates the similarity of each feature vector from the viewpoint of being similar to the search key category. The second search unit 306 calculates the similarity of each feature vector from the viewpoint that it is similar to a category different from the search key. For this reason, the image search apparatus 200 can evaluate each feature vector from two viewpoints and is similar to the category of the search key, but the search order of images corresponding to feature vectors belonging to other categories can be lowered. it can. Also, since the similarity between the representative vector registered in the dictionary DB 304 and each feature vector registered in the search target DB 302 is calculated, the more images for registering the representative vector in the dictionary DB 304, the higher the accuracy. Search can be realized. Even if representative vectors are registered in the dictionary DB 304 based on a large number of images, an increase in the search processing load can be suppressed if the number of registered representative vectors in each category is unchanged.

  FIG. 4 is an explanatory diagram of processing of the image registration unit 301 according to the first embodiment.

  The image registration unit 301 detects a vehicle region from an image captured by the camera 102 using, for example, a Haar feature amount and an AdaBoost classifier, and extracts the detected vehicle region. Next, the image registration unit 301 calculates a feature amount from the extracted vehicle region. An appropriate feature amount may be selected according to the property of the search target, and the selected feature amount may be calculated. For example, when searching for a vehicle type, the shape of the vehicle is an important factor. In this case, the image registration unit 301 may calculate a feature amount based on HOG (Histogram Oriented of Gradients) or intensity distribution in the luminance gradient direction. When searching for the color of the vehicle body, the image registration unit 301 may calculate a feature amount based on the color histogram. It should be noted that the image registration unit 301 can search for more detailed categories such as “red sedan” by calculating the feature amounts of both. A set of feature amounts calculated from an image with the image registration unit 301 is referred to as a feature vector. Further, the image registration unit 301 registers the feature vector of the calculated feature quantity in the search target DB 302.

  FIG. 5 is an explanatory diagram of the dictionary DB 304 according to the first embodiment.

  The dictionary DB 304 includes a category 501, a representative vector 502, a second search category 503, and a second search vector 504.

  In the category 501, identification information of each category that can be a search key is registered. In the representative vector 502, at least one feature vector calculated from images of each category is registered as a representative vector.

  The representative vector extraction process will be described. Similar to the processing of the image registration unit 301 described with reference to FIG. 4, a vehicle region is detected from at least one image of a certain category, and the vehicle region is extracted. Then, the feature amount of the image in the region is calculated. A set of calculated feature values is defined as a feature vector. The feature vectors calculated from the K ′ images are clustered using a technique such as k-means clustering, so that K (K ≦ K ′) feature vectors are extracted as representative vectors. It should be noted that the more representative images are extracted, the more appropriate representative vectors can be extracted, and the number of representative vectors is a predetermined number K. Therefore, it is only necessary to execute K search processes. An increase in processing load can be suppressed.

  In the second search category 503, the identification information of the category in which the representative vector is registered in the second search vector 504 is registered. In the second search vector 504, a representative vector of a category different from the category identified by the identification information registered in the category 501 is registered. By registering the representative vector to be the second search vector 504 in the dictionary DB 304, the second search unit 306 is similar to the representative vector registered in the second search vector 504 and the feature vector registered in the search target DB 302. It is only necessary to calculate the degree, and it is not necessary to calculate the degree of similarity between all the representative vectors different from the category corresponding to the search key and the feature vector registered in the search target DB 302. For this reason, the number of searches of the second search unit 306 can be reduced, and the search time can be shortened.

  Next, a method for extracting representative vectors registered in the second search vector 504 will be described.

  In the first method, a category to be searched by the second search unit 306 is selected by an administrator of the image search device 200 and the representative vector of the selected category is registered in the second search vector 504. For example, the administrator or the like selects a category (for example, a compact car) that is considered similar to a certain category (for example, a sedan), and the image search apparatus 200 registers the representative vector of the selected category in the second search vector 504. Further, the administrator or the like identifies the category of the image by looking at the image that is erroneously searched for the category of a certain search key in the past search results, and registers the specified category in the second search vector 504. It may be selected as a category of representative vectors to be displayed.

  In the first method, it takes time to make a presetting, but it is possible to perform a search that reflects the actual search results and the subjective judgment of the administrator or the like.

  In the second method, the image search apparatus 200 calculates a distance value between representative vectors of each category, and is registered in the second search vector 504 of a certain category based on the calculated distance value and a preset threshold value. The representative vector is automatically determined. A distance value between a representative vector of a category and a representative vector of another category is calculated in advance. Among representative vectors of other categories, a representative vector whose distance value is less than or equal to a preset threshold value from a representative vector of a certain category It is registered in the second search vector 504.

  In the second method, the second search vector is determined depending on the image used to calculate the representative vector, but the second search vector can be automatically registered. In the second method, a representative vector of a certain category and a representative vector whose distance value is equal to or less than the threshold value are registered as the second search vector, so that all the representative vectors of the selected category are registered as the second search vector. From the first method, the number of representative vectors registered as the second search vector is reduced. Thereby, the number of searches of the second search unit 306 can be reduced, and the search time can be shortened.

  The representative vector and the second search vector will be described with reference to FIGS. 6A to 6C. FIG. 6A is an explanatory diagram of the distribution on the feature space of the feature vectors calculated from the images of each category according to the first embodiment.

  In FIG. 6A, a circle, a triangle, and a square indicate each category, and a blacked-out point indicates a representative vector. By extracting representative vectors, ranges belonging to each category can be determined with a small number of feature vectors.

  FIG. 6B is an explanatory diagram of a second search vector determined using the first method of the first embodiment.

  In the first method, for example, the administrator determines that there is a high possibility that the category “circle” and the category “square” are similar, and the category of the representative vector registered in the second search vector of the category “circle”. Select the category “Square”.

  FIG. 6C is an explanatory diagram of a second search vector determined using the second method of the first embodiment.

  The distance value between the representative vector 601 of the category “circle” and the representative vector 611 of the category “square” is equal to or smaller than the threshold value, and the distance value between the representative vector 602 of the category “circle” and the representative vector 612 of the category “square” is the threshold value. Assume that: Therefore, the representative vectors 611 and 612 of the category “square” are registered as the second search vector of the category “circle”.

As for the category “sedan” shown in FIG. 5, a plurality of categories (“compact car” and “light car”) are selected as the representative vector categories registered in the second search vector using the first method. An example is shown. The category “wagon” shown in FIG. 5 shows an example in which one category (“bus”) is selected as the category of the representative vector registered in the second search vector using the first method. Further, the category “compact car” shown in FIG. 5 uses the second method, and the representative vector (V ₀₀ , V ₀₂ ) of the category “sedan” and the representative vector (V ₃₁ ) of the category “light car” are the second. The example registered into the search vector is shown.

  FIG. 7 is an explanatory diagram of the search result table 700 according to the first embodiment.

The search result table 700 is a table in which search results of the first search unit 305 are registered. The first search unit 305 includes each representative vector (V _ck ) registered in the representative vector 502 of the record in which the identification information of the search key category is registered in the category 501 of the dictionary DB 304, and each registered in the search target DB 302. The first distance value (d) with the feature vector (V _n ) is calculated using (Equation 1). Then, a search result table is generated in which the feature amount vectors registered in the search target DB 302 are arranged in ascending order of the first distance value.

  The search result table 700 includes a ranking 701, a management ID 702, a feature vector 703, and a distance value 704.

  In the rank 701, the rank of each feature vector arranged in the order of the first distance value is registered. In the management ID 702, image identification information corresponding to each feature vector is registered. In the feature vector 703, identification information of each feature vector is registered. In the distance value 704, the minimum first distance value among the first distance values between each feature vector and each representative vector is registered. A smaller first distance value indicates that the image corresponding to the feature vector is similar to the category of the representative vector.

  FIG. 8 is a flowchart of the second search process executed by the second search unit 306 according to the first embodiment.

  The second search unit 306 acquires J representative vectors registered in the second search vector 504 of the record in which the identification information of the category of the search key is registered in the category 501 of the dictionary DB 304 (801). Note that the representative vector registered in the second search vector 504 acquired in the process of step 801 is referred to as a second search vector.

  The second search unit 306 executes the processes in steps 803 and 804 for the J second search vectors acquired in the process in step 801. That is, the second search unit 306 executes the processes of steps 803 and 804 J times. First, the second search unit 306 selects a second search vector j to be processed (802).

  Then, the second search unit 306 searches each feature vector registered in the search target DB 302 with the second search vector j of the processing target selected in the processing of Step 802, and the second search vector j of the processing target and each A second distance value from the feature vector is calculated (803). For example, Formula 1 can be used for the calculation of the second distance value in the processing of Step 803. The second search unit 306 acquires a second distance value between the second search vector j to be processed and each feature vector as a search result (804).

  Next, if the processes in steps 803 and 804 are not executed for the J second search vectors, the second search unit 306 selects the next second search vector as the second search vector to be processed. , Steps 803 and 804 are executed (805).

  On the other hand, when the processes of steps 804 and 804 are performed on the J second search vectors, the second search unit 306 includes the feature vectors registered in the search target DB 302 calculated by the first search unit 305 and Based on the first distance value with the representative vector and the second distance value between each feature vector and the second search vector calculated by the second search unit 306, the total distance value of each feature vector is calculated (806). The total distance value is calculated so that the total distance value increases as the second distance value decreases. In other words, the total distance value is calculated so that a penalty is imposed by the second distance value. For example, the second search unit 306 calculates the total distance value of each feature vector using Expression 2.

  The total distance value calculated using Expression 2 indicates the similarity between the search key category c and the feature vector n registered in the search target DB 302. In Equation 2, the total distance value is calculated by weighting and adding the maximum value of the reciprocal of the second distance value to the minimum value of the first distance value.

  Thus, the image search apparatus 200 can preferentially present an image that is similar to the category of the search key and not similar to any category other than the category of the search key.

  Next, the second search unit 306 rearranges the feature vectors registered in the search target DB 302 in ascending order of the total distance value calculated in the processing of Step 806, and displays the images of the feature vectors rearranged in the order of the total distance value. The search result is input to the output unit 307 (807), and the second search process is terminated.

  The output unit 307 transmits search result confirmation screen data in which the images of the feature vectors registered in the search target DB 302 are rearranged to the terminal device 101 based on the input search result. A search result confirmation screen 1000 is displayed based on the search result confirmation screen data.

  FIG. 9 is an explanatory diagram of search processing performed by the image search apparatus 200 according to the first embodiment.

  FIG. 9 illustrates an example of searching for an image corresponding to a feature vector similar to the category “square” from the search target DB 302. Each figure indicated by a solid line is a feature vector calculated from an image of each category, and is the same as that shown in FIGS. The figure painted in black is a representative vector of each category. Broken line graphics 901 to 904 are feature vectors registered in the search target DB 302. The images corresponding to the figures 901 and 902 are images obtained by shooting the category “square”, and the images corresponding to the figures 903 and 904 are images obtained by shooting the category “circle”.

  FIG. 9A shows the ranks assigned to the graphics 901 to 904 based on the first distance value calculated by the first search unit 305. In this case, since the first distance value between the feature vector of the figure 903 and the representative vector of the category “square” is the smallest, the figure 903 is ranked first. Also, FIG. 902 is second, figure 901 is third, and figure 904 is fourth.

  In this case, the figure 903 is an image obtained by shooting the category “circle”, and the search result in which the figure 903 is ranked first is not appropriate even though the category “square” is searched.

  (B) shown in FIG. 9 shows the ranks assigned to the figures 901 to 904 based on the total distance value calculated by the second search unit 306. Assume that the representative vector of category “circle” and the representative vector of category “triangle” are associated as the second search vector of category “square”.

  Since the second distance value between the feature vector of the figure 903 in which the category “circle” is photographed and the representative vector of the category “circle” that is the second search vector of the category “square” is small, the total distance value of the figure 903 is reduced. Becomes larger than the first distance value, and the figure 903 is third. Further, the distance value between the feature vector of the figure 902 whose first distance value is smaller than the figure 901 and the representative vector of the category “triangle” that is the second search vector of the category “square” is the feature vector of the figure 901 and the category “square”. ”Is smaller than the distance value from the representative vector of the category“ triangle ”, which is the second search vector of“ ”, the graphic 901 is ranked first and the graphic 902 is ranked second.

  As a result, the figures 901 and 902 that are images obtained by capturing the category “square” to be searched are higher in the search result, and the figures 903 and 904 that are images obtained by capturing the category “circle” that is not the search target are the lower results. Become. That is, an image that is similar to the category of the search key and not similar to the category different from the search key can be ranked higher in the search result.

  FIG. 10 is an explanatory diagram of a search result confirmation screen 1000 according to the first embodiment.

  The search result confirmation screen 1000 includes a search key selection area 1010, a condition selection area 1020, a search result display area 1030, an other category removal check box 1040, a PREV button 1050, and a NEXT button 1060.

  The search key selection area 1010 is an area for receiving a search key selection input from the user of the terminal device 101, and is a search key selection input receiving unit. The search key selection area 1010 includes a vehicle type 1011 and a color 1012. The vehicle type 1011 displays the type of vehicle (sedan, wagon, etc.), and the type of vehicle serving as a search key is selected from the displayed types. Note that the color 1012 accepts an input of selection of the color of the vehicle serving as a search key.

  The condition selection area 1020 includes a period 1021 and a search button 1022. In a period 1021, an input for selecting a period of an image to be searched among images corresponding to the feature vector registered in the search target DB 302 is received. When the search button 1022 is operated, the terminal apparatus 101 transmits a search instruction including the search key selected in the search key selection area 1010 and the period selected in the period 1021 to the image search apparatus 200.

  In the search result display area 1030, when the other category removal check box 1040 is checked, images as search results are displayed in a list in order of the total distance value. In addition, in the search result display area 1030, when the other category removal check box 1040 is not checked, images as search results are displayed in a list in the order of the first distance value. In the search result display area 1030, the total distance value is displayed below each image when the other category removal check box 1040 is checked, and the other category removal check box 1040 is not checked. The first distance value is displayed in.

  Thereby, the user of the terminal device 101 can confirm in order from images similar to the category serving as the search key. Further, since the distance value at which the images displayed in the search result display area 1030 are arranged can be changed depending on whether or not the other category removal check box 1040 is checked, the search result that the user desires to search can be displayed.

  When the PREV button 1050 is operated, the previous search result is displayed in the search result display area 1030. When the NEXT button 1060 is operated, the next search result is displayed in the search result display area 1030.

  As described above, the second search unit 306 calculates the total similarity so that the feature vector having the higher second similarity has a lower total similarity of the feature vector. This prevents images that are similar to the search key category and similar to categories other than the search key from being ranked higher in the search results, and even if the similarity between categories is high, the image search accuracy Can be suppressed.

  In addition, a representative vector of a certain category and a second search vector of the category are associated in advance in the dictionary DB 304. As a result, it is not necessary to calculate the second similarity between the second search vector and the feature vector by using the representative vectors of all categories different from the category of the search key as the second search vector. Can be shortened.

  In addition, the distance value between the representative vector of a certain category registered in the dictionary DB 304 and the representative vector of another category is calculated as the similarity, and the representative vector of the other category whose similarity is equal to or less than the threshold value is calculated for the certain category. Let it be the second search vector. As a result, the second search vector can be automatically set, and the labor of the administrator or the like can be omitted.

  Further, an input for selecting another search key is accepted as the second feature vector of a certain search key, and the representative vector of the other search key that has received the input is set as the second search vector of the certain search key. As a result, the determination result of the administrator or the like can be reflected in the second search vector of the category.

  In the present embodiment, an example will be described in which a feature vector for executing the second search process is selected and the second search process is executed only for the selected feature vector. Thereby, the search processing time can be shortened.

  FIG. 11 is a flowchart of the second search process performed by the second search unit 306 according to the second embodiment. In FIG. 11, the same processes as those shown in FIG.

  The second search unit 306 selects a feature vector for executing the second search process (1101), and executes the processes of steps 801 to 807 for the selected feature vector.

  There are a first method and a second method for selecting feature vectors in the processing of step 1101.

  First, the first method will be described.

  In the first method, the second search unit 306 executes a second search process on a predetermined number (N) of feature vectors in ascending order of the first distance value from the feature vector having the smallest first distance value. Select as feature vector. Then, the second search unit 306 performs a second search process on the selected N feature vectors.

  In this embodiment, there are a plurality of clusters in the search target DB 302 in order to realize high-speed search, and the image registration unit 301 corresponds to the cluster corresponding to the image whose feature vector is calculated based on the calculated feature vector of the image. Register with. For this reason, images of similar feature vectors are registered in one cluster.

  An image is not registered in a cluster based on text or captured position information, but is registered in a cluster based on a feature vector. For this reason, the number of clusters in which the images of the N feature vectors selected by the second search unit 306 are registered is reduced. As a result, it is possible to prevent all N clusters selected by the second search unit 306 from being registered in different clusters and to prevent the search speed from being lowered. Therefore, in the first method, registration and search of images to clusters are also performed based on feature vectors, so the number of clusters in which images of N feature vectors selected by the second search unit 306 are registered. The search speed can be prevented from being lowered.

  Next, the second method will be described.

  In the second method, the second search unit 306 selects a feature vector whose first distance value is equal to or smaller than a threshold value as a second vector for executing the second search process. For example, when the search key is “sedan”, 80% of the images registered in the search target DB 302 are “sedans”, and 20% are images other than “sedans”, according to the first method, The feature vectors of the top 50% images are selected as the feature vectors for executing the second search process. Therefore, the second search process is not executed for the feature vectors of all “sedans” images registered in the search target DB 302. In addition, according to the first method, there is a possibility that the second search process is performed on the feature vector of the image in a category other than the category of the search key, and a useless process is performed.

  On the other hand, the second method has a demerit that the search time also changes because the number of feature vectors for executing the second search process changes. However, the second method has a bias in the category of images registered in the search target DB 302. Regardless, high-precision search can be realized.

  An example of the threshold value setting method of the second method will be described. When constructing the dictionary DB 304, the image search device 200 calculates the maximum value of the distance values between all the representative vectors of each category of the dictionary DB 304, and sets the maximum value of each category as the threshold value of each category. Thus, an appropriate threshold value can be automatically set for each category, and the second search process can be executed with high accuracy and high speed.

  Next, a modification of the second embodiment will be described. FIG. 12 is a flowchart of a second search process according to a modification of the second embodiment. In FIG. 12, the same processes as those shown in FIG. 8 and FIG.

  In the second embodiment, the second search vector to be processed is selected from the J second search vectors, the selected second search vector is used as a query, and the second search process selected in step 1101 is executed. N feature vectors are searched. In the present modification, a feature vector to be processed is selected from the N feature vectors for executing the second search processing selected in the processing of step 1101, and the selected second feature search is performed using the selected feature vector as a query. Search for a vector.

After executing the processing in step 801, the second search unit 306 performs the processing in steps 1202 and 1203 on the N feature vectors selected in the processing in step 1101. That is, the second search unit 306 executes the processing of steps 1202 and 1203 N times. First, the second searching unit 306 selects a feature vector _{V n} of the processing target (1201).

Then, the second searching unit 306 searches the J-number of the second search vector acquired in the process of step 801 in the feature vector V _n of the selected processing target in the process of step 1201, the processed feature vectors V _n And a distance value between each of the second search vectors (1202). For example, Formula 1 can be used for the calculation of the distance value in the processing of Step 1202. The second searching unit 306 obtains a distance value between the feature vector V _n and the second search vector of the processing target as the search result (1203).

  Next, if the processing of steps 1202 and 1203 is not executed for the N feature vectors selected in the processing of step 1101, the second search unit 306 uses the next feature vector as the processing target feature vector. And execute the processing of steps 1202 and 1203 (1204).

  On the other hand, when the processes of steps 1202 and 1203 are executed for the N feature vectors selected in the process of step 1101, the process proceeds to step 806.

  Since the number of second search vectors to be searched does not change, in this modification, the second search vector is searched using the feature vector as a query. Thus, the dictionary DB 304 can be set in advance so that the second search vector can be optimally accessed, and the search process can be speeded up. On the other hand, in the second search process of the second embodiment shown in FIG. 11, the feature vector is searched using the second search vector as a query. Since the number of feature vectors selected in the processing of step 1101 varies depending on the search target DB 302, it cannot be set in advance as in this modification.

  Therefore, the process of searching for the J second search vectors using the N feature vectors selected in the process of step 1101 as a query is executed N × J times, and the J second search vectors are used as a query. The search process can be performed faster than the process of searching for N feature vectors selected in the process 1101 is performed J × N times.

  As described above, the feature vector on which the second search process is executed is selected based on the first distance value, and the second search process is executed only on the selected feature vector. Thereby, the search processing time can be shortened. As an example of a feature vector selection method in which the second search process is performed, there is an example in which feature vectors for which a predetermined number of second search processes are performed are selected in ascending order of the first distance value. According to this example, it is possible to prevent the second search process from being performed on the feature vector that is distributed and registered in the plurality of clusters of the first search target DB 302.

  As another example, there is an example in which a feature vector whose first distance value is equal to or less than a threshold is selected as a feature vector for which the second search process is executed. According to this example, a highly accurate search can be realized regardless of the bias of the category of the image registered in the search target DB 302.

  In addition, a distance value between representative vectors of each category is calculated, and a threshold is set based on the calculated distance value. Thereby, an appropriate threshold value can be automatically set.

  In addition, a feature vector to be processed is selected from the feature vectors for executing the second search process, and a second distance value between the selected feature vector of the process target and the second feature vector is calculated. The process is repeated for the number of feature vectors to be processed. Accordingly, since the number of second feature vectors is determined in advance, the second feature vectors can be set in advance so that optimum data access is possible, and the search processing time can be shortened.

  In this embodiment, when a representative vector having the smallest first distance value among the plurality of feature vectors is the same, only the image corresponding to one feature vector having the smallest total distance value of the representative vector is reduced. This is an example of the image search apparatus 200 displayed as 1310. FIG. 13 is an explanatory diagram of display of the contracted screen 1310 according to the third embodiment.

(A) shows the second search result table 1300 The second search result table 1300 corresponds to the identification information of the image corresponding to the feature vector rearranged in the order of the total distance value by the second search unit 306 and the search key. A representative vector (nearest neighbor representative vector) having a minimum distance from each feature vector among the representative vectors of the category to be registered is registered.

  The second search result table 1300 includes a rank 1301, a nearest representative vector 1302, and a management ID 1303.

  In the rank 1301, the rank of each feature vector arranged in the order of the total distance value is registered. In the nearest neighbor vector 1302, identification information of the nearest representative vector of each feature vector is registered. In the management ID 1303, image identification information corresponding to each feature vector is registered.

  In the present embodiment, in the search result display area 1030 of the search result confirmation screen 1000 shown in FIG. 10, when the nearest representative vectors of a plurality of feature vectors are the same representative vector, the feature vector having the minimum total distance value is displayed. A contracted screen 1310 (FIG. 13B) is displayed only for images corresponding to.

  In the second search result table 1300 shown in (A), the representative vector “03” has a plurality of feature vectors (management IDs “00911”, “00831”, “12211”, and “00443” in ascending order of the total distance value). Is the nearest representative vector. Therefore, on the contraction screen 1310, for the representative vector “03”, only the image of the management ID “00911” with the smallest total distance value (see FIG. 13B 1311) is displayed. For the representative vector “04”, only the image with the management ID “00913” having the smallest total distance value is displayed. In other words, as many images as the number of representative vectors are displayed on the contracted screen 1310. Accordingly, the user of the terminal device 101 can check a plurality of variations of images by browsing the contracted screen 1310. In addition, an image of one feature vector is displayed for one representative vector of the search key category. Since the feature vector is a feature vector having the lowest overall similarity, a category different from the category of the search key. Can be prevented from being displayed on the contracted screen 1310.

  A check box 1312 is displayed corresponding to each image 1311 displayed on the contracted screen 1310. When the user checks a check box 1312 corresponding to an image that is determined to be similar to an object desired to be searched, a selected representative vector screen 1320 shown in FIG. 13C is displayed.

  On the selected representative vector screen 1320 shown in (C), only images corresponding to feature vectors having representative vectors “03”, “04”, and “01” as the nearest representative vectors are displayed in the order of overall similarity.

  Thereby, the user can check the search result step by step, and the user's image search efficiency can be improved.

  The present embodiment is an example of an image search apparatus 200 that realizes a search for images similar to a plurality of categories (“fuzzy category search”) by specifying a plurality of categories in a search key. The definition of the vehicle type and the color of the vehicle is ambiguous, and there is a vehicle that clearly has a “sedan” shape, but there is also a vehicle that has an ambiguous shape whether it is a “sedan” or a “compact car”. Such category ambiguity is not limited to the vehicle object. For example, in the case of searching for a 29-year-old person, the search in the “30s” category is found rather than the search in the “20s” category. It may be easy to do. According to the present embodiment, it is also possible to search for objects that exist in the vicinity of the boundaries of such a plurality of categories.

  FIG. 14 is an explanatory diagram of a search result confirmation screen 1000 according to the fourth embodiment. In FIG. 14, the same components as those in FIG.

  In the search result confirmation screen 1000 shown in FIG. 14, the search key selection area 1400 is different from the search key selection area 1010 shown in FIG. The search key selection area 1400 includes a first vehicle type 1401 and a second vehicle type 1402. In the first vehicle type 1401 and the second vehicle type 1402, the type of vehicle that serves as a search key is selected. Therefore, in this embodiment, the search key has two categories. Note that the search key category is not limited to two, and the search key may be three or more categories.

  Here, the processing of the image search apparatus 200 when “sedan” and “compact car” are selected as the search key categories in the first vehicle type 1401 and the second vehicle type 1402 will be described.

  First, the search key selection unit 303 acquires representative vectors corresponding to “sedans” and “compact cars” from the dictionary DB 304. Next, the first search unit 305 is a distance value (first distance value) between the representative vectors of “sedans” and “compact cars” selected by the search key selection unit 303 and each feature vector registered in the search target DB 302. Is calculated.

  Next, the second search unit 306 acquires a second search vector corresponding to “sedan” and “compact car” from the dictionary DB 304. In this case, if the second search vector is a “sedan” representative vector or a “compact car” representative vector, the second search vector is not used in the second search process. In the dictionary DB 304 shown in FIG. 5, the second search category 503 for “sedans” is “compact cars” and “light cars”, and the second search category 503 for “compact cars” is “sedans” and “light cars”. is there. The second search unit 306 does not use the representative vectors of “sedan” and “compact car” for the second search process, but uses only the representative vector of “light car” for the second search process.

  Then, the second search unit 306 calculates the distance value between the representative vector of “light car” and each feature vector, and calculates the total distance value based on the first distance value and the second distance value. The output unit 307 transmits search confirmation screen data including the feature vector images rearranged in the order of the total distance value to the terminal device 101.

  As described above, the total distance value of feature vectors of images similar to categories other than “sedans” and “compact cars” is increased. In addition, since the total distance value of the feature vector of an image similar to both “sedans” and “compact cars” is small, it can be searched appropriately.

  In the search result confirmation screen 1000, when images are arranged in the order of the first distance value, a large number of images of categories other than “sedans” and “compact cars” as search keys appear. In this case, an image similar only to “sedan” and an image similar only to “compact car” may be higher in the search result and may result in a search result that does not match the user's intention. Since the second search unit 306 calculates the total distance value based on the first distance value and the second distance value, the appearance of images of other categories can be suppressed. Also, since the feature vector of the image similar to both “sedan” and “compact car” has a smaller distance from the second search vector, the total distance value similar to both “sedan” and “compact car” becomes smaller. It is also possible for the image to be higher in the search results.

  Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment. Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit.

  Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor.

  Information such as programs, tables, and files for realizing each function is stored in memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC (Integrated Circuit) card, an SD card, a DVD (Digital Versatile Disc), etc. Can be placed on any recording medium.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

101 Terminal Device 102 Camera 103 Network 200 Image Search Device 210 I / F
220 CPU
230 Auxiliary storage device 240 Network I / F
250 Memory 300 Category Search Unit 301 Image Registration Unit 302 Search Target DB
303 Search key selection unit 304 Dictionary DB
305 First search unit 306 Second search unit 307 Output unit 700 Search result table 1000 Search result confirmation screen 1300 Second search result table 1310 Reduction screen

Claims (12)

A search target storage unit for storing a feature vector of an image to be searched;
A dictionary information storage unit in which each search key for searching the search target storage unit and a first feature vector corresponding to each search key are stored in association with each other;
A search key input accepting unit that accepts input of a search key;
A first feature vector corresponding to the input search key is acquired from the dictionary information storage unit, and a first similarity between the acquired first feature vector and an image feature vector stored in the search target storage unit A first search unit for calculating
Calculating a second similarity between a second feature vector that is a first feature vector corresponding to a search key other than the input search key and a feature vector of an image stored in the search target storage unit; A second search unit that calculates an overall similarity indicating a similarity between the image of the feature vector and the input search key based on the similarity and the second similarity;
An output unit that arranges and outputs the images of feature vectors stored in the search target storage unit in the order of the overall similarity, and
The image search apparatus according to claim 2, wherein the second search unit calculates the overall similarity so that a feature vector having a higher second similarity has a lower overall similarity. The image search device according to claim 1,
The second search unit selects, based on the first similarity, a feature vector for calculating the second similarity from the feature vector of the image stored in the search target storage unit, and only the selected feature vector An image search apparatus for calculating the second similarity. The image search device according to claim 2,
The second search unit selects a predetermined number of feature vectors for calculating the second similarity in descending order of the first similarity from the image feature vectors stored in the search target storage unit. Image search device. The image search device according to claim 2,
The second search unit selects, from the feature vectors of the image stored in the search target storage unit, a feature vector having the first similarity higher than a threshold as a feature vector for calculating the second similarity. A featured image retrieval device. The image search device according to claim 4,
The dictionary information storage unit stores a plurality of first feature vectors for one search key,
For each search key, calculate the similarity between the plurality of first feature vectors corresponding to each search key;
An image search apparatus characterized in that the calculated similarity is set to the threshold value of each search key. The image search device according to claim 2,
The second search unit
Selecting a feature vector to be processed from a feature vector selected as a feature vector for calculating the second similarity, and calculating a second similarity between the selected feature vector of the processing target and the second feature vector;
By repeatedly executing the process of calculating the second similarity by the number of feature vectors for calculating the second similarity, the first of all feature vectors selected as the feature vector for calculating the second similarity is calculated. An image search apparatus characterized by calculating a two-similarity. The image search device according to claim 1,
In the dictionary information storage unit, a first feature vector of a search key other than each search key is stored in association with the first feature vector of each search key in advance as the second feature vector. An image search device characterized by the above. The image search device according to claim 7,
Calculating a similarity between a first feature vector of a search key stored in the dictionary information storage unit and a first feature vector of another search key;
An image search device, wherein a first feature vector of another search key having a similarity higher than a predetermined threshold is stored in the dictionary information storage unit as the second feature vector of the certain search key. The image search device according to claim 7,
Accept input of selection of other search key that becomes the second feature vector of a certain search key,
An image search apparatus, wherein a first feature vector of another search key that has received the input is stored in the dictionary information storage unit as the second feature vector of the search key. The image search device according to claim 1,
The dictionary information storage unit stores a plurality of first feature vectors for one search key,
The output unit outputs only an image of a feature vector having the highest overall similarity among the feature vectors when there are a plurality of feature vectors having the same first feature vector having the highest first similarity. An image search device. The image search device according to claim 1,
When a plurality of search keys are input, the first search unit includes a first feature vector corresponding to each of the input search keys and an image feature vector stored in the search target storage unit. Calculating the first similarity,
The second search unit
Calculating the second similarity between a second feature vector corresponding to a search key other than the plurality of input search keys and a feature vector of an image stored in the search target storage unit;
The image search device characterized in that the total similarity is calculated based on the first similarity and the second similarity. An image search system for searching for an image similar to an input search key,
A camera for taking images,
An input receiving unit for receiving an input of the search key;
A search target storage unit for storing feature vectors of images captured by the camera;
A dictionary information storage unit in which each search key for searching the search target storage unit and a first feature vector corresponding to each search key are stored in association with each other;
A first search unit that calculates a first similarity between a first feature vector corresponding to a search key received by the input receiving unit and a feature vector of an image stored in the search target storage unit;
Calculating a second similarity between a second feature vector that is a first feature vector corresponding to a search key other than the input search key and a feature vector of an image stored in the search target storage unit; A second search unit that calculates an overall similarity indicating a similarity between the image of the feature vector and the search key based on the similarity and the second similarity;
An output unit that arranges and outputs the images of feature vectors stored in the search target storage unit in the order of the overall similarity, and
The image search system, wherein the second search unit calculates the overall similarity so that a feature vector having a higher second similarity has a lower overall similarity.

Images