Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
  • G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
  • G06V30/10—Character recognition
  • G06V30/19—Recognition using electronic means
  • G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
  • G06V30/19127—Extracting features by transforming the feature space, e.g. multidimensional scaling; Mappings, e.g. subspace methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
  • G06V30/10—Character recognition
  • G06V30/24—Character recognition characterised by the processing or recognition method
  • G06V30/248—Character recognition characterised by the processing or recognition method involving plural approaches, e.g. verification by template match; Resolving confusion among similar patterns, e.g. "O" versus "Q"

Definitions

• the present invention relates to an image search device and an image search that can quickly extract a desired image from a large-capacity image group using multidimensional feature data (face, color, etc.) extracted from an image cover. It is about the method.
• Patent Document 1 describes a high-speed search method by projecting onto a model space prepared by a statistical method and generating multi-dimensional feature data with high accuracy and a small number of dimensions.
• Patent Document 1 JP 2002-183205 A
• the present invention has been made in view of the above-described conventional circumstances, and even when a large amount of multidimensional feature data such as face / color exists in a high dimension, an image desired by a user can be efficiently and appropriately searched.
• An object of the present invention is to realize a possible image search device and image search method.
• An image search apparatus includes a dimension reduction means for generating approximate data by reducing the dimensions of multidimensional feature data extracted from an image, and the approximate data generated by the dimension reduction means by using the multidimensional data before the dimension reduction.
• Approximate data storage means for storing in association with dimensional feature data, at least a search request receiving means for receiving, as a search key, an identifier for identifying multidimensional feature data of a person to be searched, and the search request based on a search condition
• An approximate space search means for calculating a distance between the approximate data corresponding to the search key received by the receiving means and each of the plurality of approximate data stored by the approximate data storage means and arranging them in order of similarity; and the approximate space search means Perform the distance calculation again using the multidimensional feature data before dimensional reduction for the result group with high similarity obtained in step 1).
• a real space final ranking means for outputting Te.
• the dimension reduction means rearranges the elements constituting the multi-dimensional feature data in descending order of absolute value, and the top N (N: natural number) (element number, (Value) is generated as the approximate data.
• the dimension reduction means generates the top N values of the high frequency component or the low frequency component as the approximate data from the wavelet transform result of the input multidimensional feature data.
• the image search apparatus of the present invention is characterized in that multidimensional feature data other than the top N obtained by the dimension reduction means is generated as (representative value, code bit) and managed as the approximate data. .
• the real space final ranking means assigns a final ranking to the top M items (M: natural number) having a high degree of similarity obtained by the approximate space search means.
• the top K items (K ⁇ M) are output as search results.
• the real space final ranking means is based on the high similarity and the element (that is, the approximate distance is small! And the element) obtained by the approximate space search means.
• re-distance calculation is performed using multidimensional feature data before dimensional reduction, and the similarity obtained by re-distance calculation is high!
• the top K actual distances are re-distance calculated! /
• the processing is completed when it becomes smaller than the approximate distance of all data, and the top K items are output as search results.
• the real space final ranking means indicates how much the result obtained by the approximate space search means has been changed by the final ranking.
• the distortion rate is output as a result.
• the image search device of the present invention includes a re-search condition specifying means for specifying "number of dimensions to be used" and “number of items to be narrowed down" as search conditions used by the approximate space search means. To do.
• the image search apparatus of the present invention is characterized by comprising correct / incorrect designation means for designating a correct answer or an incorrect answer for the search result output by the real space final ranking means.
• the approximate space search means sets the element numbers of the approximate data designated as correct by the correct / incorrect designation means as the “correct answer element number group” and the approximate data designated as incorrect answers.
• the image search method of the present invention uses the approximate data generated by degenerating the dimension of the multidimensional feature data extracted from the image, and searches the image group accumulated in the image storage means.
• Approximation space search step that arranges in order of similarity to image, and re-similarity using multi-dimensional feature data before dimensional reduction for the high similarity results obtained in the approximate space search step
• a real space final ranking step for determining the final rank.
• the search result can be narrowed down to some extent in the approximate space with a reduced number of dimensions, and then finally narrowed down in the real space. Therefore, even when the number of dimensions becomes high, the image desired by the user can be obtained. Can be searched efficiently.
• FIG. 1 is a block diagram of an image search device according to a first embodiment of the present invention.
• FIG. 2 is a flowchart relating to data registration operation of the image search device according to the first embodiment of the present invention.
• FIG. 3 is a flow related to a search operation of the image search device in the first embodiment of the present invention.
• [4] Flow diagram related to the search operation for eliminating the search omission of the image search device according to the first embodiment of the present invention.
• FIG. 1 is a configuration diagram of an image search device according to Embodiment 1 of the present invention.
• 11 is a camera that shoots a person
• 12 is a search server that searches for an image that includes a person corresponding to the specified search condition
• 13 specifies a search condition for the search server 12.
• It is a search terminal for executing search.
• 101 is a multidimensional feature data generation means for extracting multidimensional feature data for identifying a person such as a face / color / shape from an image taken by the camera 11
• 102 is a multidimensional feature data generation means.
• the dimension reduction means 103 that generates the approximate data by reducing the dimensions of the multidimensional feature data extracted in step 103 is associated with the approximate data generated by the dimension reduction means 102 and the multidimensional feature data before the dimension reduction, Approximate data storage means for storing them as approximate feature data group 103a and actual feature data group 103b, respectively, 104 is a search request receiving means for receiving at least an identifier for identifying multidimensional feature data of a person to be searched as a search key, Based on the search condition specified by the search terminal 13, 105 represents the approximate data corresponding to the search key received by the search request receiving unit 104 and a plurality of approximate data stored by the approximate data storage unit 103.
• Approximation space search means that performs distance calculation with each of them and arranges them in the order of distance of the calculation results, that is, in order of similarity. It is a real space final ranking means that performs distance calculation again using multidimensional feature data before degeneration! The final rank determined by the real space final ranking means 106 is output as a search result.
• the human feature data extracted by the multi-dimensional feature data generation means 101 is image data of a moving object whose image power is also extracted, or information for specifying the moving object based on the shape 'color' size, movement, etc. Or, it is information for specifying the shape and position of the face's eyes 'nose' mouth.
• These feature information extraction and classification methods are widely known. For example, in Japanese Patent Application Laid-Open No. 2001-268657 and “Processing and Recognition of Images” (authored by Takeshi Yasui, Tomoharu Nagao, Shoshodo Publishing) It is written strictly.
• Human feature data such as face / clothing color generated using these existing technologies is also composed of multiple elements (referred to as dimensions) to identify a person.
• the facial feature data is a group of elements for grasping the overall facial appearance and specific parts such as eyes / nose / mouth. Total of element groups for grasping the shape: composed of several hundred to several thousand dimensions.
• FIG. 2 shows a processing procedure of the dimension reduction means 102, and the operation will be described below.
• the input multidimensional feature data ([element number, value] series) are all made absolute values and sorted in descending order of the value.
• Input multi-dimensional feature data includes face feature data with principal components of the entire face / parts as shown in 2-a, and data representing the color distribution of a person's clothes as a color space histogram such as RGB / HSV.
• data obtained by cutting out an area in which a person is captured and converting it into a frequency is given as input of multidimensional feature data in step 201 of collective power of [element number, value] of each element on the horizontal axis.
• the elements are arranged in descending order of absolute values, such as 2-b, and each element on the horizontal axis is generated as [element number and value before sorting].
• Step 203 The R1 and R2 data generated in step 202 are stored in the approximate feature data group 103a of the approximate data storage means 103.
• the pre-sort vector data input in step 201 is stored in the actual feature data group 103b. In this way, an index is generated.
• the approximate feature data group 103a is a data group with reduced dimensions, so it may be placed on a memory that can be accessed at high speed.
• FIG. 3 shows a processing procedure of the approximate space search means 105 and the real space final ranking means 106, and the operation will be described below.
• Step 301> The approximate data (3-a) corresponding to the search key received by the search request receiving means 104 and the approximate data (3-b) stored by the approximate data storage means 103 are approximated.
• the similar distance calculation is performed! / ⁇ , and the plurality of approximate data stored in the approximate data storage means 103 are arranged in ascending order of the close distance. Approximate distance calculation is performed for all dimensions before sorting as shown in 3-c.
• FIG. 3 shows the processing procedure of the real space final rank attaching means 108 for setting the search omission due to narrowing down in the approximate space to 0, and the operation will be described below.
• Step 401> Obtain a list of approximate distances with the search key generated by the approximate space search means 105. It is assumed that the list is stored in order from the smallest approximate distance.
• Step 402> Obtain small /! Data of approximate distance from the approximate distance list. At the time of acquisition, the corresponding data is deleted from the approximate distance list.
• Step 403 The actual distance between the data acquired in step 402 and the data corresponding to the search key is calculated.
• Step 404> Add the data acquired in step 402 to the real distance list.
• the list is
• the actual distance is stored in ascending order.
• Step 405 All K distance powers in the real distance list It is determined whether the distance is smaller than the minimum distance in the approximate distance list. If yes, go to step 406, if no, go to step 40
• Step 406> Search top 106 real space final ranking means 106 in the real distance list Output as a result.
• FIG. 5-a shows an example of "approximate distance and actual distance" between the search key and data A to H.
• data G causes a search omission as shown in 5–b.
• searching in the flow of Fig. 4 there is no omission of data G search as shown in 5-c.
• the search process in Fig. 3 or 4 depends on the relationship between the approximate distance and the actual distance. For example, when the “approximate distance is the actual distance” for all data, there is no possibility that the search order will be greatly changed by approximation, so FIG.
• a wavelet transform represented by the force Haar et al. which describes a method for separating a component into a component that strongly represents a person's characteristics by rearrangement of component values and an average component
• R2 data may be generated from the “average component” corresponding to the low frequency
• R1 data may be generated from the “difference component from the average” corresponding to the high frequency.
• the element numbers constituting the R1 / R2 data are fixed without depending on the input multidimensional feature data, there is an effect that the amount of calculation in the approximate space search means 105 can be reduced.
• the “average component” corresponding to the low frequency after the wavelet transform described above is applied to the R1 data and high frequency. This can be handled by changing the corresponding “difference component from the average” to R2 data.
• an image search apparatus in which a user can easily perform a narrow-down operation by re-search even if many search omissions occur in the process of narrowing down search results in the approximate space.
• FIG. 6 shows a result of “distortion rate due to dimensional reduction” indicating how much the search result obtained by the approximate space search means 105 has changed in the final ranking of the real space final ranking means 106.
• the user sets “number of dimensions to be used” and “number of cases to be narrowed down” as re-search conditions by the approximate space search means 105 while referring to “distortion rate due to dimension reduction”.
• 6—a is the initial search condition specification screen that first specifies the search key of the search target person and the search range of the time / location, and 6—b displays the search result together with the “distortion rate due to dimension reduction” Search result display screen, 6—c, 3 ways of re-search [1) Adjustment of the number of dimensions used (number of dimensions used), 2) Adjustment of approximate range (number of items to be narrowed down), 3) Above 1) 2 This is a re-search condition specification screen in which the user specifies the next search condition from [Output next K items without adjusting)), and operations 6-b and 6-c are repeated.
• a represents an example of a search omission (data L, G) that occurs when a search is performed with the configuration shown in FIG. 7-b shows the difference between the search results obtained by the approximate space search means 105 and the real space final ranking means 106 when a search omission occurs.
• the M value of the top M items which is the threshold for narrowing down the approximate space
• the probability of search omission decreases, but the user can determine how much the M value should be increased. Absent.
• a) the ratio of the top K obtained in the approximate space is included in the top K ranking, or b) the top ranking top K data,
• the rank ratio of the sum of the final ranks i.e., ⁇ * () + 1) / 2) ⁇ the sum of the ranks in the approximate space
• dimensional reduction distortion rate As shown in 6-c, there are three patterns for specifying the re-search conditions: 1) adjusting the number of dimensions used, 2) adjusting the approximate range, and 3) outputting the next K items.
• the number of used dimensions in 1) is the number of elements in the R1 data used in the approximate space retrieval means 105.
• the "dimensional reduction factor" can be reduced. it can .
• the approximate space search means 105 can change the number of elements in the R1 data and perform a re-search process, so that multiple cut dimensions (R_a , R_b, R_c) are generated in advance.
• the data structure does not need to be generated for multiple force dimensions, as shown in 8-b. This can be handled by preparing for the cut dimension (R_c) where the number of elements in the R1 data is large. is there.
• the adjustment of the approximate range in 2) is to adjust the range to be narrowed down in the approximate space (the M values of the top M items). Even when the “dimensional reduction factor” is large, the M value is adjusted. Increasing the size can prevent search omissions.
• the re-search condition specifying means may automatically re-specify the search conditions with reference to the “distortion rate due to dimension reduction” output by the real space final ranking means.
• the search device even if many search omissions occur in the process of narrowing down the search results in the approximate space, the user can easily perform a narrowing operation by re-searching.
• the search device will be described below.
• FIG. 9 shows an approximate space detection when a user specifies a correct answer or an incorrect answer for a search result.
• search means 105 increase the weight of the element number used in the approximate data specified as correct answer, and decrease the weight of the element number used in the approximate data specified as error in the approximate space. This is an example of performing the distance calculation again.
• element numbers that strongly represent the characteristics of a person differ for each search result, as shown in b the element numbers for deriving the correct answer and the element numbers for removing the incorrect answer are determined from the results specified by the user as correct / incorrect.
• weighting means assigning a weighting factor to the distance calculation result in the distance calculation in each dimension of 3c in Fig. 3.
• the user specifies the correct / incorrect answer for each image in the search result, and the approximate distance calculation parameters at the time of the re-search (element number weight, number of dimensions used, approximate range) By automatically adjusting! /, It is possible to realize a re-search operation that suppresses search omissions in the process of narrowing search results in the approximate space.
• the image search apparatus and the image search method that are useful in each embodiment of the present invention narrow down the search results to some extent in the approximate space with a reduced number of dimensions, and then narrow down the final search in the real space. By doing so, it has the effect of reducing the amount of computation even when the number of dimensions becomes high and efficiently searching for the image desired by the user, and the shoplifting crime targeting multiple cameras.
• it is possible to apply to the purpose of browsing 'contents (still images' videos) taken by individuals such as travel athletic meet 'search' editing etc. .
• An image search apparatus capable of quickly extracting a desired image from a large-capacity image group using the multidimensional feature data (face, color, etc.) extracted from the image power. This is useful for image retrieval methods.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Library & Information Science (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• General Engineering & Computer Science (AREA)
• Computing Systems (AREA)
• Evolutionary Computation (AREA)
• General Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Health & Medical Sciences (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Image Analysis (AREA)
• Processing Or Creating Images (AREA)
• Image Processing (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Related Child Applications (2)

Publications (1)

Family

ID=38188721

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (19)

Citations (5)

Family Cites Families (15)

• 2005
  • 2005-12-22 JP JP2005370613A patent/JP4777059B2/ja not_active Expired - Fee Related
• 2006
  • 2006-12-22 EP EP06835124.6A patent/EP1965349A4/en not_active Withdrawn
  • 2006-12-22 CN CN200680031384.7A patent/CN101253535B/zh not_active Expired - Fee Related
  • 2006-12-22 US US12/064,928 patent/US8150854B2/en not_active Expired - Fee Related
  • 2006-12-22 WO PCT/JP2006/325646 patent/WO2007072947A1/ja not_active Ceased
• 2012
  • 2012-02-20 US US13/400,380 patent/US8560551B2/en not_active Expired - Fee Related

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events