JP2017045291A - Similar image searching system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a similar image searching system that searches for a similar image from among images registered in advance.SOLUTION: A similar image searching system includes: a registered image feature vector storage unit which stores a feature vector formed by vectorizing a feature quantity of an image registered in advance, on a data table; a query image input receiving processing unit which receives an input query image; a query image feature quantity extraction processing unit which extracts a feature quantity on the basis of the received query image; a query image vectorization processing unit which calculates a feature vector by vectorizing the extracted feature quantity; a query image feature vector search processing unit which searches for a registered image similar to the query image, in reference to the feature vector in the registered image feature vector storage unit, on the basis of the calculated feature vector; and a search result output processing unit which outputs the registered image similar to the query image as a result of searching.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a similar image search system for searching for similar images from pre-registered images.

  In an image search, an image similar to a query image is registered as a plurality of images (registration) using a certain image (query image) (an image means image information that can be displayed on a computer and converted into data) as a search condition. In some cases, it is necessary to search for whether or not the image (registered image database) exists in the registered database. In this case, it is necessary to determine whether the images are similar by comparing the query image and the registered image registered in the registered image database. However, if the number of registered images increases, A large amount of processing time is required for the determination process. Therefore, there is a need for a similar image retrieval system that can determine at high speed.

  Therefore, in order to perform high-speed image similarity search processing for a large number of images, conventionally, such as local sensitive hashing (LSH) or data management by distance from the centroid, A method of narrowing down search candidates by using a data structure for grouping images with high similarity is generally used. The following nonpatent literature 1 exists as an example of this. Further, there is a system disclosed in Patent Document 1 as a system that searches for similar registered images based on similarity to a query image.

  In addition, when searching for similar images, it may be desired to exclude images other than those having attributes desired by the user who searches for images from the search results. In such a case, there is a method in which a sufficiently large number of similar image candidates are obtained by a method such as Non-Patent Document 1 and then narrowed down by attribute data. As this attribute data, there is a method of narrowing down using text data related to the image as a search keyword (Patent Document 2). In this method, a keyword related to text data is further specified, and a similar image is searched by searching for a search keyword and an image associated with the related keyword.

Japanese Patent Laid-Open No. 2001-160062 JP 2008-217117 A

Takuya Shigemura, Daiki Shimizu, Kengo Terasawa, “High-speed similarity search from large-scale image data by LSH”, March 6, 2013, Information Processing Society of Japan, Annual Conference Proceedings, 2013 (1), p. 393-395

  The process using LSH in Non-Patent Document 1 is a method of determining a hash value according to a feature amount in an image and executing a similar image search at a high speed based on the determined hash value. This is based on the fact that if the hash value of the query image and the hash value of the registered image are the same, the feature amount is a close value, that is, it can be determined that the two images are similar.

  In this case, when the number of registered images increases tenfold, the number of images obtained as a result of the search also increases tenfold, causing inconveniences such as a rapid increase in processing time. Therefore, in general, processing such as re-hashing all registered images with new hash values is required.

  Furthermore, when narrowing down by attribute data of images, in order to secure the required number of search results, the size of one group is kept large when searching for similar images, or over several groups. It is necessary to deal with such as searching. As a result, problems such as deterioration in operating efficiency and a significant increase in processing time occur. Therefore, even if the search for similar images is speeded up, the effect is lost.

  In view of the above problems, the present inventor has made the following invention.

  A first invention is a similar image retrieval system for retrieving an image similar to a query image from images registered in advance, wherein the similar image retrieval system is a registered image that is an image registered in advance. A registered image feature vector storage unit that stores a feature vector obtained by vectorizing the feature amount in a data table, a query image input reception processing unit that receives an input of a query image, and a feature amount based on the query image that has received the input A query image feature amount extraction processing unit to extract, a query image vectorization processing unit to calculate a feature vector obtained by vectorizing the extracted feature amount, and a registered image feature vector storage unit based on the calculated feature vector A query image feature vector search process for searching a registered image similar to the query image with reference to the feature vector. Parts and the searched result, a similar image retrieval system with a search result output unit for outputting an image similar to the query image.

  As in the present invention, unlike the prior art, the feature vector of the registered image is stored in the data table, and the comparison with the feature vector of the query image is executed, so that a similar image is specified only by the processing of the data table. Can do. For this reason, since only simple calculations are performed, processing can be realized at high speed.

  Also, since feature vectors are managed in the data table, even if registered images are added or deleted, there is no need for re-hashing due to changes in the hash value of local features, as in the past. It is also possible to facilitate management.

  In the above-described invention, the query image feature vector search processing unit uses the calculated feature vector and the feature vector stored in the registered image feature vector storage unit to determine the similarity between the query image and each registered image. The search result output processing unit can be configured as a similar image search system that outputs a registered image having a specified similarity in a predetermined range as a registered image similar to the query image.

  As in the present invention, when searching for similar images, it is preferable to specify a similarity using a feature vector and specify a similar image based on the similarity.

  In the above-described invention, the similar image search system further encodes the attribute data of the registered image, stores the image in association with the registered image, and inputs the attribute data of the query image A query image attribute data input reception processing unit that receives the reference, a query image attribute data encoding processing unit that encodes the attribute data that has received the input, and the registered image attribute data storage unit based on the encoded value, It can be configured as a similar image search system having a query image attribute data search processing unit that executes a narrowing process using attribute data.

  In the above-described invention, the search result output processing unit further changes the similarity of registered images corresponding to attribute data that does not satisfy the filtering condition as a result of the narrowing processing performed by the query image attribute data search processing unit. Then, it can be configured as a similar image search system that outputs a registered image having a similarity in a predetermined range based on the changed similarity as a registered image similar to the query image.

  As in these inventions, similar image retrieval processing can be realized by executing narrowing processing using image attribute data. The attribute data in the present invention refers to text-type information associated with an image and indicating the attribute of the image.

  In the above invention, the registered image attribute data storage unit stores a bit string that is binary data of registered image attribute data in a data table, and the query image attribute data search processing unit stores the query image attribute data. By performing a logical operation on the bit string that is binary data of the attribute data of the query image encoded by the encoding processing unit and the bit string that is binary data of the attribute data of the registered image stored in the registered image attribute data storage unit, , A similar image retrieval system that executes a narrowing-down process using attribute data can be configured.

  Instead of processing the attribute data as text data as in the present invention, it is managed in the data table as a bit string of binary data, and a narrowing process is executed by a logical operation between the binary data, so that higher speed processing can be performed. Can be realized.

  In the above-described invention, the similar image search system further includes boundary information indicating a boundary of the attribute data, and the query image attribute data search processing unit stores binary data obtained by encoding the attribute data of the query image. When performing a logical operation on a bit string and a bit string of binary data obtained by encoding the registered image attribute data stored in the registered image attribute data storage unit, the boundary of the attribute data is determined based on the boundary information, and the attribute data It can be configured like a similar image retrieval system that executes a comparison process of binary data for each boundary.

  When using multiple attribute data as filtering conditions, if attribute data is managed as a bit string in the data table, the boundary of attribute data cannot be specified, and logical operations between bit strings cannot be executed. . Therefore, a plurality of attribute data can be used as narrowing conditions by using the boundary information indicating the boundary of the attribute data as in the present invention.

  In the above-described invention, the search result output processing unit generates a histogram in which the similarity is classified for each predetermined range, specifies in which range of the histogram the number of search results to be output is included, and A similar image retrieval system that outputs a registered image similar to the query image by sorting registered images having similarities exceeding the specified range as a sort target and sorting based on the similarities is configured. be able to.

  When sorting the registered images according to the degree of similarity and outputting the search result, an execution time of O (n log (n)) is required. For this reason, the smaller the number of registered images to be sorted, the shorter the sort execution time can be. Therefore, by configuring as in the present invention, the number of registered images to be sorted based on the similarity can be reduced, so that higher-speed processing can be realized.

  The above-described similar image retrieval system can be realized by causing a computer to read the program of the present invention. In other words, the computer has a query image input reception processing unit that receives an input of a query image, a query image feature amount extraction processing unit that extracts a feature amount based on the query image that has received the input, and the extracted feature amount is vectorized A query image vectorization processing unit that calculates a feature vector, a registered image feature vector storage unit that stores a feature vector obtained by vectorizing a feature amount of a registered image based on the calculated feature vector in a data table, and A similar image search program that functions as a query image feature vector search processing unit that searches for a registered image similar to a query image, and a search result output processing unit that outputs a registered image similar to the query image as a result of the search.

  By using the similar image retrieval system of the present invention, it is possible to realize a high-speed retrieval of similar images. This is because simple processing such as grouping of images with high similarity is not performed as in the past, and processing using simplification of the data structure of the image and simplification of the iterative processing is realized. By utilizing the CPU's high-speed execution capability, it is possible to increase the speed of similar images.

  In addition, for attribute data used as a narrowing-down condition, similar to the search for similar images, by implementing processing using simplified data structure and simplified iterative processing, the CPU can execute high-speed execution for simple repeated operations. Can be used to speed up similar images.

It is a conceptual diagram which shows typically an example of the whole concept of the similar image search system of this invention. It is a figure which shows an example of the hardware constitutions of a computer typically. It is a flowchart which shows an example of the whole processing process of the similar image search system of this invention. It is a flowchart which shows an example of the process of a search process of a similar image. It is an example of the flowchart which shows an example of the process of the narrowing-down process by attribute data. It is a figure which shows typically an example of the registration image feature vector memory | storage part. It is a figure which shows an example of a registration image attribute data memory | storage part typically. It is a schematic diagram which shows typically the concept of the data structure of the similar image search system of this invention. It is the schematic diagram which simplified FIG. 8 for description of the similar image search system of this invention. It is a figure which shows typically an example of the attribute data used as narrowing-down conditions. It is a figure which shows typically an example of feature vector Q []. It is a figure which shows typically an example of initialized similarity R []. It is a figure which shows typically an example of feature vector V [] []. It is a figure which shows typically the process of S260 when j = 1. It is a figure which shows typically the process of S260 when j = 3. It is a figure which shows typically the process of S260 when j = 4. It is a figure which shows typically an example of the concept at the time of performing a narrowing-down process. It is a figure which shows typically an example of the concept regarding narrowing-down conditions. It is a figure which shows typically the attribute data in case there exists dependence.

  An example of an overall conceptual diagram of a similar image search system 1 of the present invention is shown in FIG. The similar image search system 1 of the present invention includes a registered image management system 2 that manages registered images such as registration and deletion of registered images, and a search processing system 3 that searches for similar images.

  The registered image management system 2 includes a registered image input reception processing unit 20, a registered image feature amount extraction processing unit 21, a registered image vectorization processing unit 22, a registered image feature vector registration processing unit 23, and a registered image feature vector storage. A registration image attribute data input reception processing unit 25, a registered image attribute data encoding processing unit 26, and a registered image attribute data storage unit 27.

  The search processing system 3 includes a query image input reception processing unit 30, a query image feature amount extraction processing unit 31, a query image vectorization processing unit 32, a query image feature vector search processing unit 33, and a query image attribute data input reception. A processing unit 34, a query image attribute data encoding processing unit 35, a query image attribute data search processing unit 36, and a search result output processing unit 37 are included.

  The similar image retrieval system 1, registered image management system 2, and retrieval processing system 3 are realized by various computers such as a server, a personal computer, and a portable communication terminal. FIG. 2 shows an example of the hardware configuration of the computer. The computer includes an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM and a hard disk for storing information, a display device 72 such as a display, a processing result of the arithmetic device 70 and a storage device. A communication device 73 that transmits and receives information stored in the network 71 via a network such as the Internet or a LAN, and a part or all of an input device 74 such as a keyboard or a pointing device (such as a mouse or a numeric keypad) are provided.

  Although FIG. 1 shows the case where the similar image retrieval system 1, the registered image management system 2, and the retrieval processing system 3 are each realized by one computer, the functions are distributed and implemented in a plurality of computers. Also good.

  Each means in the present invention is only logically distinguished in function, and may be physically or virtually the same area.

  In this specification, for convenience of explanation, the registration image management system 2 and the search processing system 3 will be described separately. However, actually, these may be configured as one system, or a plurality of different systems. It may be configured as a system. Further, the processing functions of each system may be shared.

  A registered image input reception processing unit 20 in the registered image management system 2 receives an input of an image (registered image) to be searched in the similar image search system 1. The registered image input acceptance processing unit 20 accepts input of an image in an image format that is processed in the similar image search system 1 in advance. At this time, an ID for identifying the image is given.

  The registered image feature amount extraction processing unit 21 executes a process of extracting the feature amount of the image received by the registered image input reception processing unit 20 by a predetermined method. The feature amount may be any index for determining similarity. As the feature amount, for example, an upper limit is set from 0 to 255 or 65535, and the size is standardized.

  In this specification and the following description, a case where Histogram intersection is used as a feature amount will be described. In the similar image search system 1 of the present invention, it is effective to use a local feature, Bag-of Visual-Words, or Histogram intersection as a feature. ), A color histogram, a color correlogram, or other image feature amount may be used. It may be a combination of them and Bag-of-Visual Words.

  The registered image vectorization processing unit 22 performs a process of vectorizing (fixing length vectorization) the feature amount extracted by the registered image feature amount extraction processing unit 21.

  In calculating a feature vector, if a feature used for processing and a bin (granularity of a selected feature) are selected, the number of features that match each bin is counted and a histogram is generated. Further, normalization is performed on the histogram. By performing normalization, it is possible to obtain similar values between images having high similarity. Thus, the similarity between images can be obtained by histogram intersection or the like.

  Furthermore, it is preferable to correct the bias by distorting the characteristics according to the appearance frequency of each feature vector. For example, when an image occupies most of an image as a population (a population of registered images), the middle part is pushed down by multiplying by an exponential function, and the appearance frequency is spread over the entire region. Average. As a result, when a feature vector is constructed by combining highly independent feature quantities, one of the feature quantities becomes stronger and the other feature quantities are buried, which contributes to similarity determination. It is possible to avoid disappearance and increase the accuracy of similarity determination.

  In this way, by aligning the dynamic range of each element of the feature vector and forcing the bias, it is possible to define an upper limit and a lower limit of the value that can be taken by the feature vector to perform integerization. As a result, it is possible to represent the elements of the feature vector with 2 bytes (values from 0 to 65535), and it is also possible to represent them with 1 byte (values from 0 to 255).

  The registered image feature vector registration processing unit 23 stores a value (feature vector) vectorized by the registered image vectorization processing unit 22 and an ID assigned by the registered image input reception processing unit 20 in a registered image feature vector storage which will be described later. The unit 24 stores the information in association with each other.

  The registered image feature vector storage unit 24 stores the ID and feature vector of each registered image in association with each other. FIG. 6 schematically shows an example of the registered image feature vector storage unit 24.

  The registered image attribute data input acceptance processing unit 25 accepts input of attribute data when there is attribute data corresponding to the registered image accepted by the registered image input acceptance processing unit 20.

  The attribute data is information used as a narrowing condition when searching for similar images. The attribute data is a combination of counting types. For example, the attribute data is “prefecture”, and the possible values of the attribute data are 47 types of combinations such as “Hokkaido” and “Aomori”. Need to be.

  The relationship between the attribute data does not have to be independent of each other, and may have a dependency relationship. As this dependency relationship, there is a case where the value of the attribute data of the upper level is an alternative, and the value that the lower level can take varies depending on the upper level. This is schematically shown in FIG.

  For example, when the value of attribute data of attribute A is “table, mat”, the value of attribute A is “table”, attribute B (height) is “45 cm, 70 cm”, and the value of attribute A is “mat”. ", The case where the attribute C (area) is" 2 tatami mats, 4 tatami mats "or the like is applicable. In this case, the encoding method of attribute A, attribute B, and attribute C is determined independently, and the maximum length of the number of bits of attribute B and attribute C is assigned as a common area. At the time of encoding, whether to use the encoding of the attribute B or the encoding of the attribute C is determined depending on the value of the attribute A, so that processing can be performed even when there is a dependency.

  In addition, when using continuous quantities (length, area, etc.) or extremely large variations (time data, etc.) as narrowing conditions, they are separated by appropriate sections, bits are assigned to the sections, At the time of narrowing down processing, the search target is narrowed down in units of sections.

  The registered image attribute data encoding processing unit 26 encodes the attribute data received by the registered image attribute data input reception processing unit 25 (bit data is converted into binary data) and stores it. The encoded attribute data and the ID to which the registered image corresponding to the attribute data is assigned by the registered image input acceptance processing unit 20 are stored in association with each other in the registered image attribute data storage unit 27 described later.

  The registered image attribute data storage unit 27 stores the ID of each registered image and the encoded attribute data in association with each other. FIG. 7 schematically shows an example of the registered image attribute data storage unit 27.

  The query image input reception processing unit 30 in the search processing system 3 receives an input of a query image. The query image input acceptance processing unit 30 accepts input of an image in an image format that is processed in the similar image search system 1 in advance.

  The query image feature amount extraction processing unit 31 executes a process of extracting the feature amount of the image by a predetermined method with respect to the query image received by the query image input reception processing unit 30. Note that the process of extracting the feature amount of the image is the same as the process of extracting the feature amount from the registered image by the registered image feature amount extraction processing unit 21. Therefore, the feature quantity is the same index as the index extracted as the feature quantity of the registered image. The feature amount at this time is also standardized in size by setting an upper limit, for example, from 0 to 255, or 65535.

  The query image vectorization processing unit 32 executes a process of vectorizing (preferably fixed length vectorizing) the feature amount extracted by the query image feature amount extraction processing unit 31.

  The query image feature vector search processing unit 33 refers to the registered image feature vector storage unit 24 in the registered image management system 2 on the basis of the value (feature vector) vectorized by the query image vectorization processing unit 32, and registers each registered image. The similarity is specified. The “similarity” is the degree that the feature vector of the registered image matches the feature vector of the query image. If the similarity is high, it can be determined that the similarity between the two images is high.

  The query image attribute data input acceptance processing unit 34 accepts input of attribute data when there is attribute data used as a narrowing condition for searching for a registered image similar to the query image.

  The query image attribute data encoding processing unit 35 encodes the attribute data received by the query image attribute data input reception processing unit 34 (converted into a bit string as binary data).

  The query image attribute data search processing unit 36 refers to the registered image attribute data storage unit 27 in the registered image management system 2 based on the attribute data encoded by the query image attribute data encoding processing unit 35, and searches for the matching registered image. Specify the ID.

  The search result output processing unit 37 is based on the registered image searched and extracted by the query image attribute data search processing unit 36 and the similarity of each registered image searched and specified by the query image feature vector search processing unit 33. , The registered image to be output as the search result is specified. For example, as a result of the processing in the query image feature vector search processing unit 33, the similarity between the query image and each registered image is specified. Therefore, the attribute data specified by the query image attribute data search processing unit 36 is satisfied. The similarity of registered images that are not registered is changed to “0”, and the similarities are sorted. Then, the IDs of registered images having a high degree of similarity are sequentially output as search results.

  The administrator terminal 4 is a computer used by an administrator who manages registered images stored in the registered image management system 2 according to the present invention.

  The user terminal 5 is a computer used by a user who desires to search for similar images in the similar image search system 1 of the present invention. In addition to searching, registration processing of registered images may be performed in the same manner as the administrator.

  A schematic diagram schematically showing the concept of the data structure in the similar image search system 1 of the present invention is shown in FIG. An array I [] in FIG. 8 stores an ID for identifying each registered image. The array V [] [] stores a feature vector that is a value obtained by converting the feature amount of each registered image into a fixed length vector. The array A [] [] stores the value (binary number) of encoded attribute data associated with each registered image. The array Q [] is a feature vector of the query image. The array C [] is encoded attribute data associated with the query image.

  The array B [] is boundary information and indicates the boundary of attribute data as each narrowing-down condition. For example, in the case of FIG. 8, B [] contains “1”, “1”, “2”, “2”, “3” as values. In this case, there are three types of attributes: the value of the first attribute is the first and second bytes, the value of the second attribute is the third and fourth bytes, and the value of the third attribute Is stored in the fifth byte. Therefore, B [] in FIG. 8 indicates that three types of attribute data are used.

  The attribute data 1 has 16 bits (2 bytes), the attribute data 2 has 16 bits (2 bytes), the attribute data 3 has 8 bits (1 byte). It is used as a bit string.

  FIG. 18 schematically shows an example of a conceptual diagram of a narrowing process using attribute data as a narrowing condition. FIG. 18 shows a case where three types of attribute data are used as the narrowing-down conditions, where attribute data 1 is “local name”, attribute data 2 is “plant name”, and attribute data 3 is “growth state”. . Assume that the number of local governments is 700, the plant names are 20 types, and the growth state is 5 stages. Then, if 1 bit is assigned to one local government, 700 bytes are required for the attribute data 1 “local name”, so 88 bytes (704 bits) are assigned. If 1 bit is assigned to one plant name, 24 bits are required for the attribute data 2 “plant name”, so 3 bytes (24 bits) are assigned. Similarly, if 1 bit is assigned to each growth state, 5 bits are required for the attribute data 3 “growth state”, so 1 byte (8 bits) is assigned.

  Therefore, in the bit string of attribute data, B [] indicating the boundary information is “1” indicating attribute data 1 from the first byte to the 88th byte, and attribute data 2 from the 89th byte to the 91st byte. “2” indicating “3”, and “3” indicating attribute data 3 is stored in the 92nd byte.

  Furthermore, A [] [i] indicating the attribute data of the registered image includes a bit string indicating the local government name in the bit string from the first byte to the 88th byte, and a plant in the bit string from the 89th byte to the 91st byte. A bit string indicating a name is stored, and a bit string indicating a growing state is stored in the bit string of the 92nd byte. Thus, a 92-byte bit string is stored in A [] [i].

  Since A [] [i] is simply assigned a bit string of “0” or “1”, the bit from the first bit to where is attribute data 1 and the bit from where is attribute data 2 or attribute. Data 3 cannot be specified. Therefore, by referring to B [], the first byte (first bit) to the 88th byte (704th bit from the first) is attribute data from the first 89th byte (705th bit from the first) to 91 bytes. It can be specified that the attribute data from the first bit (728th bit to the 728th bit) indicates the attribute data 2 and the 92nd byte (from the 729th bit to the 736th bit) indicates the attribute data 3.

  At the time of narrowing-down processing using attribute data C [] as a narrowing condition and attribute data A [] [i] corresponding to a registered image in a query image attribute data search processing unit 36 to be described later, attribute data As a narrowing-down condition for 1, the bit string from the first bit to the 704th bit of C [] and the logical operation from the first bit to the 704th bit of A [] [i] are performed, and the attribute data As a filtering condition of 2, a logical operation is performed from the bit string from the 705th bit to the 728th bit from the beginning of C [] and the bit string from the 705th bit to the 728th bit from the beginning of A [] [i] As a filtering condition for attribute data 3, the bit string from the 729th bit to the 736th bit from the beginning of C [] and the bit string from the 729th bit to the 736th bit from the beginning of A [] [i] By performing the management operations can execute narrow-down process, respectively. Here, the bit arrangement of the attribute information A [] [i] and the narrowing condition C [] is aligned, and each attribute is assigned in byte units, so that the narrowing operation is performed by a logical operation instruction (or byte unit) (or High-speed computation is realized by using the fact that computation can be performed by a large unit exceeding bytes).

  In this way, the attribute data as the filtering condition is only a bit string stored as binary data, so if you do not know the boundary, specify how far to process as one unit of attribute data I can't. However, by managing the boundary of each attribute data with B [], the data area of the attribute data can be managed dynamically. Therefore, the type of attribute data used as a narrowing condition can be changed according to the content and type of the image that the administrator wants to manage.

The array R [] stores the similarity between the feature vector (Q [j]) of the query image and the feature vector (V [j] [i]) of the registered image. In this specification, Histogram intersection is used as the feature quantity, so the similarity is as in S260 described later.
R [i] = R [i] + min (Q [j], V [j] [i])
In addition, L1 distance, L2 distance, etc. can also be used.

When using the L1 distance as the similarity,
R [i] = R [i] + | Q [j] -V [j] [i] |
It can be calculated as follows.
When the L2 distance is used as the similarity,
R [i] = R [i] + (Q [j] −V [j] [i]) ²
It can be calculated as follows.

  The array P [] stores the intermediate state and result of narrowing down attribute units. In other words, P [] is a register that operates on the logical operator OR based on the value of attribute data stored in C [] and the value of attribute data stored in A [] []. For example, when “local government name”, “organism name”, and “growth state” are provided as attribute data, each attribute data is connected by a logical operator “AND”. On the other hand, with respect to the attribute data “local government”, the user can set a plurality of local government names, for example, “A city”, “B city”, and “C town” as narrowing conditions. An example of attribute data as the narrowing condition is shown in FIG. Therefore, when the attribute data used as the narrowing-down condition is as shown in FIG. 10, “A city”, “B city”, and “C town” are connected by the logical operator OR to the “local name” which is the unit of the attribute data. Therefore, a search is made as to whether any of “A city”, “B city”, and “C town” is included in the attribute data “municipal name” of A [] []. “1” is stored when the ID is included, and “0” is stored when the ID is not included.

  The array N [] stores the status and result of narrowing down the narrowing processing based on attribute data. In other words, this is a register that operates on the logical operator AND based on the value of the attribute data (binary number) stored in C [] and the value of the attribute data (binary number) stored in A [] []. . For example, when the attribute data used as the narrowing-down condition is FIG. 10, the attribute data “city name” includes any one of “A city”, “B city”, and “C town”. "Rice" is included as the search result, and "D" or "E" is included in the attribute data "growth state", the search result includes "1" "Is not stored, it is stored as" 0 ".

  Next, an example of a typical processing process of the similar image retrieval system 1 of the present invention is shown in FIGS. FIG. 9 is a conceptual diagram obtained by simplifying the processing concept of the similar image retrieval system 1 of FIG. In the following description, FIG. 9 is used. For convenience of explanation, in each function of the registered image management system 2, the registered image is stored in the registered image feature vector storage unit 24 as a value obtained by converting the feature amount of the image into a fixed length vector. To do. The attribute data corresponding to the registered image is encoded and stored in the registered image attribute data storage unit 27 as a bit string that is binary data. These operations are performed from the administrator terminal 4 or the user terminal 5.

  In FIG. 9, the number of registered images is 5, and the feature amount is expressed by four fixed length vectors. In addition, there are two attribute data as a narrowing condition, and the two attribute data are each 1 byte (8 bits).

  Specifically, the administrator operates the administrator terminal 4 to cause the registered image management system 2 to read the registered image. The registered image input acceptance processing unit 20 accepts input of the registered image. At this time, an ID (I [i] in FIG. 9) corresponding to the registered image is attached. When the registered image input acceptance processing unit 20 accepts an input, the registered image feature amount extraction processing unit 21 extracts the feature amount of the registered image that has received the input, and the registered image vectorization processing unit 22 calculates a vectorized value. Then, it is registered in the registered image registration processing unit. This value is stored in V [j] [i] in FIG.

  If the registered image has attribute data as a narrowing condition, the attribute data corresponding to the registered image is input to the registered image management system 2 by operating the administrator terminal 4, and the input is registered. The image attribute data input acceptance processing unit 25 accepts input. Then, a value (bit string of binary data) encoded by the registered image attribute data encoding processing unit 26 is calculated from the attribute data. The calculated value is stored in the registered image attribute data storage unit 27. This value is stored in A [k] [i] in FIG. In FIG. 9, for ease of viewing, decimal numbers are used instead of bit strings, but actually, bit strings of binary data are stored.

  I [i], V [j] [i], and A [k] [i] are stored correspondingly.

  Note that the registration image and attribute data input operations are preferably performed simultaneously, but may be performed at different timings. In that case, either may be input first.

  8 and 9, the row direction and the column direction are shown opposite to the normal direction (that is, the normal row direction is the column direction and the normal column direction is the row direction). Therefore, for example, in V [j] [i], j is expressed in the horizontal direction and i is expressed in the vertical direction. This is because when the last subscript is incremented, the memory is usually arranged immediately after that and access is fast, so that in the inner loop such as S260 described later, the last subscript changes from 1 to the number of registered images. This is because it can be executed at the highest speed. On the other hand, in FIG. 8 and FIG. 9, since it is easier to understand that the feature vector is represented horizontally and the feature vectors of a large number of registered images are arranged in the vertical direction, x in V [x] [y] is the dimension number of the feature vector. , Y correspond to the number of registered images. The control variable of the inner loop is because it is desired to process using i as will be described later. These are general cases, and when implemented on various computer systems, it is necessary to change the order of the subscripts according to how the compiler to be used arranges the arrays in the memory. There is also.

  As shown in FIG. 9, when values are stored in the arrays I [], V [], A [], B [], a user who desires to search for a similar image from registered images The similar image search system 1 is accessed by performing a predetermined operation from the user terminal 5, and the query image is input to the search processing system 3. The query image input from the user terminal 5 is input by the query image input reception processing unit 30 (S100).

  Also, attribute data corresponding to the query image is input to the search processing system 3. The attribute data input from the user terminal 5 is input by the query image attribute data input reception processing unit 34 (S100). If there is no attribute data, the input process and the narrowing process described later are not necessary.

  When the query image input acceptance processing unit 30 accepts an input of a query image, a similar image search process is executed (S200).

  First, when the query image input reception processing unit 30 receives an input of a query image, the query image feature amount extraction processing unit 31 extracts a feature amount from the query image that has received the input. Then, the query image vectorization processing unit 32 calculates the feature vector Q [j] (FIG. 9) by vectorizing the feature amount extracted by the query image feature amount extraction processing unit 31 (preferably, making it a fixed length vector). (S210). As a result, it is assumed that the value of the feature vector Q [] is “2”, “0”, “3”, “5” as shown in FIG.

  Next, the query image feature vector search processing unit 33 initializes the value of R [i] (FIG. 9) indicating the similarity between the query image and each registered image (for example, substitutes “0”) (S220). . Note that i indicates identification information of a registered image. FIG. 9 shows a case where identification information is given in the order of registration so that the first image is 1, the second registered image is 2, and the last registered image is 5. FIG. 12 schematically shows the initialized similarity R [] value.

  The query image feature vector search processing unit 33 then calculates the feature vector Q [j] calculated by the query image vectorization processing unit 32 and the feature vector V [j] [ i] (an example of the feature vector V [j] [i] in FIG. 9 is shown in FIG. 13), and the similarity is stored in R [i].

Specifically, the processing of S240 to S260 is repeatedly executed for all feature vectors Q [] (S230). Note that when the value of Q [] is "0", it indicates that there is no feature amount, so that the processing moves to the processing of the next element of Q [] (S250). If the value of Q [] is not "0", the element (subscript i) of V [j] []
R [i] = R [i] + min (Q [j], V [j] [i])
Is executed (S260).

  By executing the above process for all Q [], the similarity between the query image and the registered image can be specified.

  This processing will be described with reference to FIG. 9. “2”, “0”, “3”, and “5” are assigned to Q [] as elements, and the maximum value of i is 5 and the maximum value of j Is 4 (the initial value of the array index is 1).

When j = 1, since Q [1] = “2” (S250), the process of S260 is executed for each of i = 1 to 5. That is,
R [1] = R [1] + min (Q [1], V [1] [1])
R [2] = R [2] + min (Q [1], V [1] [2])
R [3] = R [3] + min (Q [1], V [1] [3])
R [4] = R [4] + min (Q [1], V [1] [4])
R [5] = R [5] + min (Q [1], V [1] [5])
Execute.

  Then, R [1] to R [5] are all initial values “0”, and V [1] [1] to V [1] [5] are “2”, “0”, “7”, “1”, respectively. ”And“ 2 ”, the values of R [1] to R [5] are“ 2 ”,“ 0 ”,“ 2 ”,“ 1 ”, and“ 2 ”as a result of the process of S260. FIG. 14 schematically shows this processing.

  Since the process of S260 has been executed for all i, the process is terminated for j = 1, the value of j is incremented, and j = 2 is set (S230).

  When j = 2, since Q [2] = “0” (S250), the process of S260 is not executed, and the value of j is incremented to j = 3 (S230).

Since Q [3] = “3” when j = 3 (S250), the process of S260 is executed for each of i = 1 to 5. That is,
R [1] = R [1] + min (Q [3], V [3] [1])
R [2] = R [2] + min (Q [3], V [3] [2])
R [3] = R [3] + min (Q [3], V [3] [3])
R [4] = R [4] + min (Q [3], V [3] [4])
R [5] = R [5] + min (Q [3], V [3] [5])
Execute.

  R [1] to R [5] before the processing of S260 are “2”, “0”, “2”, “1”, “2”, V [3] [1] to V [3] [5]. Are “0”, “6”, “2”, “0”, and “2”, respectively, so that the values of R [1] to R [5] are “2” and “3” as a result of the processing of S260. , “4”, “1”, “4”. FIG. 15 schematically shows this processing.

  Since the process of S260 has been executed for all i, the process is terminated for j = 3, the value of j is incremented, and j = 4 is set (S230).

Since Q [4] = “5” when j = 4 (S250), the process of S260 is executed for each of i = 1 to 5. That is,
R [1] = R [1] + min (Q [4], V [4] [1])
R [2] = R [2] + min (Q [4], V [4] [2])
R [3] = R [3] + min (Q [4], V [4] [3])
R [4] = R [4] + min (Q [4], V [4] [4])
R [5] = R [5] + min (Q [4], V [4] [5])
Execute.

  R [1] to R [5] before the processing of S260 are “2”, “3”, “4”, “1”, “4” V [4] [1] to V [4] [5] Since they are “4”, “1”, “0”, “2”, and “1”, respectively, the values of R [1] to R [5] are “6”, “4”, “4”, “3”, “5”. This process is schematically shown in FIG.

  By executing the above processing, the processing is executed even when j is the maximum value 4, so the processing of S230 to S260 is terminated (S240), and the processing after S270 is executed. Since the query image attribute data input acceptance processing unit 34 accepts input of attribute data as a refinement condition (S270), a refinement process with the refinement condition in the query image attribute data search process is executed (S280). ).

  The concept of the narrowing process will be described with reference to FIG. First, the query image attribute data encoding processing unit 35 encodes (binarizes) the attribute data as the narrowing condition received by the query image attribute data input reception processing unit 34, and the narrowed down condition code C [] The value (binary number) is stored (S400). Since FIG. 9 shows a case where “1” and “3” are stored as values (binary numbers) in C [], respectively, C [1] = “00000001”, C [2] = “00000011” It becomes. Note that A [] [] also stores a bit string (binary number) of binary data corresponding to FIG. Here, the narrowing condition 1 indicates “bit 1 of attribute 1 = YES”, and the narrowing condition 2 indicates “bit 1 or bit 2 of attribute 2 = YES”.

  Next, the query image attribute data search processing unit 36 initializes the arrays N [] and P [], and substitutes “1” for the array N [] and “0” for P [] (S 410).

  Then, the query image attribute data search processing unit 36 repeats and executes the processes after S430 for the elements of the array C [], thereby searching for attribute data of each registered image corresponding to the narrowing-down conditions ( S420).

  Specifically, in the case of FIG. 17, since the type of attribute data is 2 for the array C [], the maximum value of the subscript k is 2. First, when k = 1, since the process is not completed for all k (S430), the determination process of S440 is executed. Then, referring to B [1], which is an array of boundary information, is not a process for new attribute data (since it is an attribute 1 process), the process of S450 is executed as it is.

Specifically, it is determined whether all the bit strings of C [1] are 1, that is, “11111111” (S450). In the case of FIG. 17, since C [1] = “00000001”, it is determined whether all the bit strings of C [1] are 0, that is, “00000000” (S460). In the case of FIG. 17, since C [1] = “00000001”, for subscript i, which is an element of A [1] [], for i = 1 to 5,
P [i] = P [i] OR (C [1] AND A [1] [i])
The process is executed (S470).

  Then, for i = 1, 3 to 5, the value of A [1] [i] is “1 (bit 1 is“ 1 (YES) ”)” (00000001), so P [1], P [3] ˜ “1” is stored for P [5]. On the other hand, for i = 2, since the value of A [1] [2] is “4 (bit 1 is“ 0 (NO) ”)” (00000100), P [2] = 0 is stored. FIG. 17 shows P [] at this time.

  For k = 1, since the process of S470 has been executed for all cases of i = 1 to 5, k is incremented and k = 2 is obtained (S420).

If k = 2, the process is not finished yet (S430), and it is determined whether B [2] is a new attribute. That is, referring to B [2], which is an array of boundary information, B [2] is processing for new attribute data (attribute 2) (S440).
N [] = N [] AND P []
Perform the operation.

  That is, “1” is stored in each of i = 1 to 5 as an initial value for N [], and P [1], P [3 is the result of calculation when k = 1 for P []. ] To P [5] are stored as “1”, and P [2] are stored as “0”. For i = 1 to 5, N [] and P [] are ANDed to calculate N []. Therefore, “1” is stored for N [1] and N [3] to N [5], and “0” is stored for N [2] (S490).

  Further, in order to initialize P [], P [] = 0 is substituted for i = 1 to 5 (S490).

Next, it is determined whether all the bit strings of C [2] are 1, that is, “11111111” (S450). In the case of FIG. 17, since C [2] = “00000011”, it is determined whether all the bit strings of C [1] are 0, that is, “00000000” (S460). In the case of FIG. 17, since C [2] = “00000011”, for the subscript i that is an element of A [2] [], for i = 1 to 5,
P [i] = P [i] OR (C [2] AND A [2] [i])
The process is executed (S470).

Then
When i = 1, P [1] = P [1] OR (C [2] AND A [2] [1])
Since the value of A [2] [1] is “1 (bit 1 is“ 1 (YES) ”)” (00000001), P [1] = 1 is stored. The

When i = 2, P [2] = P [2] OR (C [2] AND A [2] [2])
Since the value of A [2] [2] is “2 (bit 2 is“ 1 (YES) ”)” (00000010), P [2] = 1 is stored. The

When i = 3, P [3] = P [3] OR (C [2] AND A [2] [3])
Since the value of A [2] [3] is “4 (bit 2 is“ 0 (NO) ”)” (00000100), P [3] = 0 is stored. The

  When i = 4, 5, the same calculation as i = 2 is performed, and P [4] = 1, P [5] = 1 are stored.

  As described above, when the processing of S470 is performed for i = 1 to 5, “1” for P [1], P [2], P [4], and P [5] and “ “0” is stored.

  For k = 1, since the process of S470 has been executed for all cases of i = 1 to 5, k is incremented to k = 3 (S420).

Since all processing is completed at k = 3 (S430),
N [] = N [] AND P []
(S500).

  Then, “1” is stored for N [1], N [3] to N [5], and “0” is stored for N [2]. P [1], P [2], P [4 ] And P [5] are stored as "1" and P [3] is stored as "0". Therefore, when these AND operations are performed, N [1], N [4], and N [5] "1" is substituted for N [2] and N [3], and "0" is substituted as the operation result, which is the final output result of N [] (S500).

  By executing the processing as described above, it is possible to execute the narrowing processing under the narrowing conditions.

In S450, when all the bits of C [k] are “1”, it is clear that P [] = 1 even if the OR operation is performed in S470. Then, in S470,
P [i] = P [i] OR (C [k] AND A [k] [i])
In the operation of
C [k] AND A [k] [i]
Even if the operation is performed, the result is the same. for that reason,
C [k] AND A [k] [i]
If one of the operations is omitted and the processing is performed as the operation of S480, one operation can be selected, so that the processing speed can be increased. In this respect, although there is a technical significance of the process of S450, the branch process of S450 may not be provided.

When the narrowing-down process is completed as described above, the search result output processing unit 37 determines i = 1 to 5 that are elements of R [].
If N [i] = 0, then R [i] = 0
(S290). In this process, the similarity R [i] of a registered image that does not satisfy the narrowing condition is set to “0” and the similarity is lowered. It should be noted that lowering the similarity is only required to be lower than the similarity specified in the similar image search process, and the similarity R [i] does not necessarily have to be set to “0”.

  In the case of FIG. 17, “6”, “4”, “4”, “3”, “5” are stored in R [i] for i = 1 to 5, but N [1], N [ 4] and N [5] contain “1”, and N [2] and N [3] contain “0”. Therefore, for i = 2 and 3, R [i] is set to “0”. substitute. As a result, for i = 1 to 5, “6”, “0”, “0”, “3”, and “5” are substituted for R [i].

  When the similarity is corrected in consideration of the narrowing-down conditions as described above, sorting is performed in descending order of similarity based on R [], and the result is output (S300). In this case, the output results are rearranged in the order of R [1] = “6”, R [5] = “5”, R [4] = “3”, and registered images corresponding to i = 1, 5, 4 And is output to the user terminal 5 as a similar image search result. In this case, the registered image corresponding to the output result may or may not be displayed as the output result.

  In the first embodiment, the similar image search is performed based on the registered images. Therefore, in the present embodiment, even when a registered image is deleted, a case where a similar image search process is executed by simply deleting data from the data table or excluding it from the processing target is shown. By using the processing of this embodiment, the search processing process itself does not require correction, the search speed is not reduced, and the search result can be easily obtained as a correct result.

  That is, when a request for deleting a registered image already registered in the registered image management system 2 is received from the administrator terminal 4 or the user terminal 5, the registered image management system 2 displays the registered image feature vector. Processing for setting the flag of the feature vector V [] [i] and attribute data A [] [i] of the registered image from the storage unit 24 and the registered image attribute data storage unit 27 to zero and setting the flag to be excluded from the search target. Run.

  By setting the value of the attribute information A [] [i] to zero, the deleted image becomes the image that is most similar to any query image. Therefore, even if no special processing is added to the search processing, Excluded from the search results by the similar search and narrowing-down process of the first embodiment. Therefore, processing such as recalculation of the hash value of the registered image is not required as in the prior art. When adding an image, it is possible to increase the memory utilization efficiency by finding an erased image from the flag to be excluded and re-registering the memory area.

  In the case where the registered image deletion process is performed as in the present embodiment and is excluded from the processing target using the flag, when the search processing system 3 executes the similar image search process, The flag is checked, and if there is a flag indicating that the flag is excluded from processing, it is not included in the search processing target.

  In the output processing in the search result output processing unit 37 in S300, instead of simply sorting based on the similarity, the search is performed based on the similarity histogram and the list length (number) of registered images required as the search result. May be.

  In this case, when the similarity R [] is simply sorted, the execution time required for the sorting is O (n log (n)). Therefore, if the number (n) of sort objects is reduced in advance before sorting, the execution time required for sorting can be greatly reduced.

  Since the similarity R [] for all the registered images has already been obtained in the processing in S300, the search result output processing unit 37 first sufficiently sets an interval including the minimum value and the maximum value of the similarity. Create a histogram with small granularity. For example, when the minimum value of similarity is 0 and the maximum value is 1000, a histogram with 10 similarities such as 0 to 9, 10 to 19, ..., 990 to 999, 1000 is created. The total similarity is counted up. The particle size at this time can be set as appropriate.

  The number of search results k to be output by the search result output processing unit 37 is set in advance so as to obtain similar images up to the k-th position, and therefore the range of the k-th position in the similarity histogram created above. To determine whether it is in, and limit the sort target.

  For example, when it is determined from the histogram that the k-th place is in the range of similarity 800 to 809, the sort target is set to similarity 800 or more, and I [], which is the ID of the registered image, and its similarity again. A list consisting of R [] (length is k + Δ) is generated, and the list is sorted using similarity R [] as a key.

  By executing the processing as described above, the number of objects to be sorted can be reduced in advance, and the execution time required for sorting can be greatly reduced.

  By using the similar image search system 1 of the present invention, the pre-processing such as grouping of images having high similarity is not performed as in the prior art, and the processing using the simplification of the data structure of the image and the simplification of the iterative processing is performed. By implementing the above, it is possible to realize the speeding up of similar images by utilizing the high-speed execution capability of the CPU for simple repetitive calculations.

  In addition, for attribute data used as a narrowing-down condition, similar to the search for similar images, by implementing processing using simplified data structure and simplified iterative processing, the CPU can execute high-speed execution for simple repeated operations. Can be used to speed up similar images.

1: Similar image search system 2: Registered image management system 3: Search processing system 4: Administrator terminal 5: User terminal 20: Registered image input reception processing unit 21: Registered image feature amount extraction processing unit 22: Registered image Vectorization processing unit 23: registered image feature vector registration processing unit 24: registered image feature vector storage unit 25: registered image attribute data input reception processing unit 26: registered image attribute data encoding processing unit 27: registered image attribute data storage unit 30: Query image input reception processing unit 31: Query image feature quantity extraction processing unit 32: Query image vectorization processing unit 33: Query image feature vector search processing unit 34: Query image attribute data input reception processing unit 35: Query image attribute data encoding process Unit 36: Query image attribute data search processing unit 37: Search result output processing unit 70: Computing device 71: Storage device Device 72: Display device 73: Communication device 74: Input device

Claims (8)

A similar image retrieval system for retrieving an image similar to a query image from previously registered images,
The similar image retrieval system includes:
A registered image feature vector storage unit for storing a feature vector obtained by vectorizing a feature amount of a registered image, which is an image registered in advance, in a data table;
A query image input acceptance processing unit for accepting input of a query image;
A query image feature extraction processing unit that extracts a feature based on the query image that has received the input;
A query image vectorization processing unit for calculating a feature vector obtained by vectorizing the extracted feature amount;
A query image feature vector search processing unit that searches for a registered image similar to the query image with reference to a feature vector of the registered image feature vector storage unit based on the calculated feature vector;
As a result of the search, a search result output processing unit that outputs a registered image similar to the query image;
A similar image retrieval system comprising: The query image feature vector search processing unit includes:
Using the calculated feature vector and the feature vector stored in the registered image feature vector storage unit, the similarity between the query image and each registered image is specified,
The search result output processing unit
Outputting a registered image having the specified similarity in a predetermined range as a registered image similar to the query image;
The similar image search system according to claim 1. The similar image retrieval system further includes:
A registered image attribute data storage unit that encodes attribute data of the registered image and stores the encoded image data in association with the registered image;
A query image attribute data input acceptance processing unit for accepting input of attribute data of the query image;
A query image attribute data encoding processing unit that encodes the attribute data that has received the input;
A query image attribute data search processing unit for referring to the registered image attribute data storage unit based on the encoded value and executing a narrowing process using the attribute data;
The similar image search system according to claim 1, further comprising: The search result output processing unit further includes:
As a result of executing the narrowing process in the query image attribute data search processing unit, the similarity of registered images corresponding to attribute data that does not satisfy the narrowing condition is changed,
Based on the changed similarity, a registered image having a predetermined similarity is output as a registered image similar to the query image.
The similar image search system according to claim 3, wherein: The registered image attribute data storage unit
A bit string that is binary data of the attribute data of the registered image is stored in a data table.
The query image attribute data search processing unit
A bit string that is binary data of the attribute data of the query image encoded by the query image attribute data encoding processing unit, and a bit string that is binary data of the attribute data of the registered image stored in the registered image attribute data storage unit, Performs a narrowing process using attribute data by performing a logical operation.
The similar image search system according to claim 3 or 4, wherein The similar image retrieval system further includes:
Boundary information indicating the boundary of the attribute data is provided,
The query image attribute data search processing unit
Based on the boundary information, when performing a logical operation on the bit string of binary data that encodes the attribute data of the query image and the bit string of binary data that encodes the attribute data of the registered image stored in the registered image attribute data storage unit , Determine the boundary of the attribute data, and execute the comparison processing of each binary data for each boundary of the attribute data,
The similar image search system according to claim 5, wherein: The search result output processing unit
Generating a histogram that classifies the similarity into predetermined ranges;
Specify in which range of the histogram the number of search results to be output is included,
The registered image having a similarity degree equal to or greater than the specified range is subjected to sorting, and the registration image similar to the query image is output by sorting based on the similarity degree.
The similar image search system according to claim 2, wherein: Computer
A query image input reception processing unit for receiving an input of a query image,
A query image feature amount extraction processing unit that extracts a feature amount based on the query image that has received the input;
A query image vectorization processing unit that calculates a feature vector obtained by vectorizing the extracted feature amount. A registered image feature vector that stores a feature vector obtained by vectorizing a feature amount of a registered image based on the calculated feature vector in a data table. A query image feature vector search processing unit for searching a registered image similar to the query image with reference to a storage unit;
As a result of the search, a search result output processing unit that outputs a registered image similar to the query image;
A similar image search program characterized by functioning as

Images