JP6364387B2 - Feature generation apparatus, method, and program - Google Patents

Description

  The present invention relates to a feature value generation device, method, and program, and more particularly, to a feature value generation device, method, and program for converting a feature vector of content into a binary vector.

  Due to the advancement and quality of communication environments, computers, and distributed processing infrastructure technologies, the number of media contents distributed on networks has become enormous. For example, a search engine is said to have trillions of web pages indexed. Some sites report that 350 million images are uploaded every day, and some sites report that 64 hours of video per minute are newly released. .

  Such an enormous amount of content is a rich source of information for users, but also brings about the problem that it becomes increasingly difficult to quickly access the content to be viewed. In such a trend, there is an increasing demand for media analysis technology for efficiently searching for contents to be browsed and viewed.

  In the applications as described above, the procedure for finding content that is the same or similar as the semantic content plays an important role. The meaning content mentioned here is an instance representing the content content, and refers to something that can be named with actuality. For example, an object shown in an image or video is a typical example.

  For example, considering the case of classifying content, it is often desirable to classify content having the same semantic content into the same category. Alternatively, in the case of a search, when a certain content is given as a query, it is a basic requirement to search for a content having the same or similar meaning content as this content. In addition, in content recommendation, a user finds and recommends content that is the same as or similar to the content that the user has browsed / viewed so far. Also in the case of content summarization, it is necessary to find a portion where there is no overlap in semantic content and to collect this.

  Here, a typical procedure for finding content having the same or similar meaning will be described. First, each content is expressed by a set of one or more feature vectors. Next, the proximity of feature vectors included in two different contents is measured, and the closer the content (matching pairs) are, the closer the semantic content is, or the same content I look forward to it.

  Here, the reason why one content is expressed as a set of one or more feature vectors will be described by taking an example where the content is an image and the semantic content is an object. The same applies to the case where the semantic content represents something other than an object, for example, a character or a figure, a scene, or a place. Also, when the content is a video, the video is considered a series of images. The same applies to the case.

  Normally, even if the object is the same object, it does not always appear in the same position, posture (angle, etc.) and size in any image, and varies depending on the image. However, it is desirable that the feature amount extracted from the image has invariance with respect to the position, posture, and size. It is difficult to obtain such invariance with a simple feature vector that represents one image itself. For example, a pixel value arranged in a vector is not invariant with respect to position, posture, and size. On the other hand, an abstraction of some information, such as a color histogram, can have invariance with respect to position and orientation, but is not invariant with respect to size. Further, it is vulnerable to a case where a part of an object is missing, and the discrimination accuracy is easily deteriorated as information is abstracted. On the other hand, if a small area of an image is extracted in large quantities to obtain a fixed-dimension feature vector representing each micro-area, and the image is represented by a set of these multiple feature vectors, the position and orientation -It can be a feature quantity that is invariant to size. More specifically, when there are two images expressed in this way, each sample is taken from a set of feature vectors expressing each image and compared to determine the identity. If the number of identical images is large, the two images are likely to contain the same object, and if the number is small, the images are low. Comparison between feature quantity vectors (in small areas) of each image is performed regardless of the position and size of the small area. Further, if the feature vector itself has invariance with respect to the posture, such expression and comparison thereof are invariant with respect to any of position, posture, and size. Actually, many methods are known for obtaining a feature vector having invariance with respect to the posture. For example, Scale Invariant Feature Transform (SIFT) described in Non-Patent Document 1 and Non-Patent Document 2 describe. Speed-up Robust Features (SURF) and the like exist.

  As described above, according to the above typical procedure, by expressing the content as a set of one or more feature vectors, the semantic content is robustly the same regardless of the position / posture / size, or , Can find close content.

  On the other hand, there is a problem that such a method is very inefficient because a large number of feature vectors must be compared in order to compare contents.

  Suppose that the content is represented by 10,000 128-dimensional feature vectors. At this time, in order to compare two arbitrary contents, a total of 12.8 billion operations of 10,000 × 10000 × 128 are required, and when executed on a general computer, it takes about 10 to 15 seconds. If the number of contents is 10,000, and if contents having the same or similar meaning are found from these, it will take 10,000 times more time.

  In addition, when such a feature vector stores all real values (4-byte precision) in a memory, a very large memory of about 50 gigabytes (GB) is required.

  In view of the above problems, there is a demand for a feature value generation technique that makes it possible to discover content having the same or similar meaning content at high speed and high efficiency.

  Several inventions have been made and disclosed for the above problems.

  Non-Patent Document 3 discloses a method of making it possible to find an image including the same object with a document search capacity by quantizing a feature vector. A large number of feature points that are minute regions are detected from the image, and each feature point is described by a SIFT feature value. A vector quantizer (sign extension) is learned in advance by clustering a large number of SIFT feature quantities extracted in this way by the K-means method. By using this quantizer to vector-quantize the feature vector of the original image into quanta (code), the image is represented by a plurality of quanta. As a result, the relationship between the image and the quantum is equivalent to the relationship between the document and the words included therein, so that the search can be executed with the same capacity as the document search.

  In Patent Document 1, a procedure similar to the procedure disclosed in Non-Patent Document 3 is taken, and furthermore, by obtaining a plurality of types of appearance frequencies of each quantum, an image including the same object is found with higher accuracy. A method that can be used is disclosed.

  The technique disclosed in Non-Patent Document 4 is as follows. First, the quantizer is learned similarly to the procedure disclosed in Non-Patent Document 3. Also, a random transformation matrix having a size equal to the dimension of the feature vector is generated, and further obtained when this transformation matrix is applied to the feature vector assigned to each quantum for each quantum. The median vector of the obtained vectors is obtained. When expressing an image, first, a feature vector included in the image is quantized using a quantizer, and then the previous transformation matrix is applied to the feature vector. By determining whether or not it is larger than the calculated median vector, a binary (eg, 1, 0) binary vector is generated. As a result, each feature vector is represented by two types of information of a binary vector of quantum and binary. When performing comparison between images (or between feature vectors included in different images), the quantum to which the feature vector is assigned is the same, and the distance between corresponding binary vectors is less than a certain value. Only in some cases, it is determined that both match.

  The technique disclosed in Non-Patent Document 5 is as follows. First, the quantizer is learned similarly to the procedure disclosed in Non-Patent Document 3. When expressing an image, first, a feature vector included in the image is quantized using a quantizer. Subsequently, for each quantum, a residual vector between the feature vector assigned to the quantum and the representative vector of the quantum is obtained, and the sum is obtained. Finally, a vector in which the residual vectors calculated for each quantum are arranged in a line (that is, having a dimension of “quantum number × feature vector dimension number”) is obtained and used as the representation of the image.

  Non-Patent Document 6 discloses a technique similar to Non-Patent Document 5. The difference from the technique of Non-Patent Document 5 is that, as in Non-Patent Document 4, a random transformation matrix having a size equal to the dimension of the feature vector is applied to the sum of residual vectors for each quantum. It is converted into a binary vector (for example, 1, 0) by determining whether or not it is larger than the value vector. Finally, a vector in which binary vectors calculated for each quantum are arranged in a line (that is, this also has a dimension of “quantum number × feature amount vector dimension number”) is obtained and used as a representation of the image.

JP2014-229063

D.G.Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp.91-110, 2004 H. Bay, T. Tuytelaars, and L.V.Gool, “SURF: Speeded Up Robust Features”, Lecture Notes in Computer Science, vol. 3951, pp.404-417, 2006 Josef Sivic, Andrew Zisserman. Video Google: A Text Retrieval Approach to Object Matching in Videos.In Proc.IEEE International Conference on Computer Vision (ICCV). Pages. 1470-1477, 2003. Herve Jegou, Matthijs Douze, Cordelia Schmid. Hamming Embedding and Weak Geometric Consistency. In Proc. European Conference on Computer Vision (ECCV). Pages. 304-317, 2008. Herve Jegou, Matthijs Douze, Cordelia Schmid, Patrick Perez. Aggregating Local Descriptors into a Compact Image Representation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Pages. 3304-3311, 2010. Giorgos Tolias, Yannis S. Avrithis, Herve Jegou. To Aggregate or Not To Aggregate: Selective Match Kernels for Image Search.In Proc.IEEE International Conference on Computer Vision (ICCV). Pages. 1401-1408, 2013.

  From a broad perspective, the problem is that the existing technology does not achieve both accuracy and processing efficiency.

  The technologies disclosed in Non-Patent Document 3 and Patent Document 1 quantize feature quantity vectors and express content as a set of quanta, thereby eliminating inefficient feature quantity vector comparisons and improving processing efficiency. It was possible to become. However, if a feature value vector that is originally a real value is simply quantized, the influence of the quantization error becomes very large, resulting in a problem that the accuracy is greatly degraded.

  On the other hand, the technique of Non-Patent Document 5 reduces the influence of this quantization error by obtaining the sum of residual vectors of feature vectors and representative vectors assigned to each quantum. However, the residual vector obtained in this way is still a real value, and there is a problem that when it is quantized with a realistic quantum number, it becomes an inefficient process. That is, for example, when the quantum number K is 16 bits, that is, 2 ＾ 16 = 65536 quanta, the length of the vector that finally represents the content is 65536 × 128 = 8 per content. It becomes an enormous number of dimensions of 388,608. In the worst case, if the amount of information is 32 megabytes (MB) per content and the number of content is 10,000, the required memory capacity will increase to 312 GB. Furthermore, when comparing contents, it takes a great deal of time because the distances of these very high-dimensional real-value vectors are compared.

  The technique of Non-Patent Document 6 can be regarded as a technique for improving the efficiency of the technique of Non-Patent Document 5. The sum of the residual vectors of the feature vector and the representative vector assigned to each quantum is converted into a binary vector using a random transformation matrix and a median vector. That is, for example, as in the previous example, in the case of 16-bit quantization and the number of contents 10000, the necessary worst memory capacity can be suppressed to about 10 GB. In addition, binary vector comparisons can be performed using only logical operations (exclusive OR and counting the number of bits that are 1), which is much faster than comparing residual vectors that are real values. It can be processed. However, this technique converts a feature quantity vector that is originally a real number into a binary vector using a random transformation matrix, and therefore does not necessarily preserve the information that the original feature quantity vector has. In other words, the influence of quantization error due to binarization becomes so large that it cannot be ignored. Further, as in Non-Patent Document 3 and Patent Document 1, since it is affected by quantization error due to quantization, there is a problem that the influence of the quantization error as a whole is very large, resulting in tremendous accuracy degradation. It remained.

  As described above, a technology capable of generating a feature amount of a content that can be efficiently found from the viewpoint of processing time and memory capacity and can accurately find a content having the same or similar meaning content until the present is invented. Was not.

  The present invention has been made to solve the above-described problems, and provides a feature amount generation apparatus, method, and program capable of accurately and efficiently finding content having the same or similar meaning content. The purpose is to provide.

  In order to achieve the above object, a feature value generating apparatus according to a first aspect of the present invention relates to content expressed by a set of one or more feature value vectors more than the information amount of the one or more vector sets. A feature amount generating apparatus for converting into a binary vector that results in a small amount of information, and for each of the feature amount vectors that are elements of the set of one or more feature amount vectors that express the content of the feature amount generation target And applying a predetermined quantizer to each of the feature vector to quantize at least one of a predetermined number of quanta, and each of the feature vectors to which the quantum is assigned A binarization unit that obtains the binary vector by applying a predetermined transformation matrix obtained in advance, and the quantizer, and The transformation matrix minimizes an error between the feature vector and the binary vector obtained from the feature vector using the quantizer and the transformation matrix based on the plurality of input feature vectors. It is characterized by being determined.

  In the feature value generation apparatus according to the first aspect of the invention, the quantization unit applies the quantizer to each feature value vector that is an element of the set of one or more feature value vectors. Thus, at least one of the predetermined number of quanta is assigned to each of the feature quantity vectors, and an integrated feature quantity vector is obtained based on one or more feature quantity vectors to which the same quantum is assigned, The binarization unit may obtain the binary vector by applying the transformation matrix to the integrated feature vector.

  According to a second aspect of the present invention, there is provided a feature quantity generation method in which a content expressed by a set of one or more feature quantity vectors has a binary information amount that is smaller than an information quantity of the one or more vector sets. A feature value generation method in a feature value generation device that converts to a vector, wherein each of the feature value vectors is an element of a set of the one or more feature value vectors that represents a content to be generated by the quantization unit On the other hand, by applying a predetermined quantizer, a quantization step for assigning at least one of a predetermined number of quanta to each of the feature vector, and a binarizing unit, the quantum is assigned. A binarization step for obtaining the binary vector by applying a predetermined transformation matrix obtained in advance to each of the feature quantity vectors. Then, based on the quantizer and the plurality of input feature quantity vectors, the feature quantity vector and the binary vector obtained from the feature quantity vector using the quantizer and the transformation matrix It is characterized in that it is determined so as to minimize the error.

  In the feature value generation method according to the second aspect of the invention, the step of assigning by the quantizing unit includes the quantizer for each feature value vector that is an element of the set of one or more feature value vectors. By applying at least one of the predetermined number of quanta to each of the feature quantity vectors, and based on one or more feature quantity vectors to which the same quantum is assigned, an integrated feature quantity vector And the step of obtaining by the binarizing unit may obtain the binary vector by applying the transformation matrix to the integrated feature vector.

  A program according to a third invention is a program for causing a computer to execute each step of the feature value generation method according to the second invention.

  According to the feature amount generation apparatus, method, and program of the present invention, the quantizer and the transformation matrix are converted into an error between the feature amount vector and a binary vector obtained from the feature amount vector using the quantizer and the transformation matrix. Assigning at least one of a predetermined number of quanta to each feature vector and applying a transformation matrix by defining a minimum and applying a quantizer to each feature vector Thus, by obtaining the binary vector, it is possible to obtain an effect that contents having the same or similar meaning contents can be found accurately and efficiently.

It is a block diagram which shows the structure of the feature-value production | generation apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the output / storage of a binary vector. It is a figure which shows an example of the output / storage of a binary vector. It is a figure which shows an example of the output / storage of a binary vector. It is a flowchart which shows the conversion learning process routine in the feature-value production | generation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the binary vector production | generation processing routine in the feature-value production | generation apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the difference | error of a feature-value vector and a binary vector.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<< Overall structure >>

  FIG. 1 is a functional block diagram illustrating an example of a configuration of a feature quantity generation device 1 according to an embodiment of the present invention.

  The feature quantity generation device 1 shown in FIG. 1 includes a feature extraction unit 11, a conversion learning unit 12, a quantization unit 13, and a binarization unit 14, and also includes a storage unit 3 as a storage device.

  In addition, the feature quantity generation device 1 is connected to the content database 2 via a communication unit and communicates information with each other. Based on the content 21 registered in the content database 2, the quantizer 31 and the transformation matrix 32. Using the learning process for storing the quantizer 31 and the transformation matrix 32 in the storage unit 3, and the quantizer 31 and the transformation matrix 32 that have been learned and generated. Binary vector generation processing for generating binary vector information 5 is executed.

  The content database 2 may be inside or outside the feature value generation apparatus 1, and any known communication means can be used. However, in the present embodiment, the content database 2 is assumed to be outside. It is assumed that the communication means is connected to communicate via the Internet or TCP / IP. Further, the content database 2 may be configured by a so-called RDBMS (Relational Database Management System) or the like.

  It is assumed that content 21 is stored in the content database 2. The content 21 may be any type of media as long as it satisfies the requirements of the embodiment of the present invention. For example, the content 21 is an image file for an image, a sound file for sound, a video file for video, If it is a document, it may consist of a collection of document files. Further, the content database 2 is associated with each content file and each document file by an identifier (for example, an ID based on a file-specific serial number) that can uniquely identify each content file, and refers to any file. Suppose you can. In addition, the metadata may include, for example, content expressing the content (content title, summary sentence, keyword, etc.), content format (content data amount, thumbnail size, etc.), and the like. Although not used in an example embodiment of the present invention.

  Moreover, each part with which the feature-value production | generation apparatus 1 is provided, and the content database 2 are comprised by the computer or server provided with the arithmetic processing apparatus, the memory | storage device, etc., and the process of each part is good also as what is performed by a program. This program is stored in a storage device included in the feature value generation apparatus 1 and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Of course, any other components need not be realized by a single computer or server, but may be realized by being distributed to a plurality of computers connected by a network.

  Although not an essential component in the embodiment of the present invention, a search device is externally used when a content search is executed using the binary vector generated by the feature value generation device 1 according to the embodiment of the present invention. 6 is provided. It is assumed that the search device 6 is connected to the feature value generation device 1 and the content database 2 so as to communicate with each other.

  << Processor >>

  Each processing unit of the feature quantity generation device 1 in the present embodiment will be described.

  The feature extraction unit 11 receives the content 21 stored in the content database 2 or the content 4 input from the outside, analyzes this, and extracts a set of one or more feature vectors to extract the content. The data is output to the database 2 or the quantization unit 13.

  The transformation learning unit 12 calculates an error between the feature vector and a binary vector obtained from the feature vector using the quantizer 31 and the transformation matrix 32 based on the plurality of feature vectors stored in the content database 2. The quantizer 31 and the transformation matrix 32 are learned and generated so as to be minimized, and the learned and generated quantizer 31 and the transformation matrix 32 are stored in the storage unit 3.

  The quantizing unit 13 applies the quantizer 31 stored in the storage unit 3 to each of the feature quantity vectors that are elements of a set of one or more feature quantity vectors representing the feature quantity generation target content. Then, at least one of a predetermined number of quanta is assigned to each feature vector and stored in the content database 2 or output to the binarization unit 14 together with the feature vector. The quantum number K is determined in advance and may be any positive integer. For example, K = 512, 2048, K = 65536 may be set. The process of assigning quanta to a feature vector is a process of associating one or more quanta from among K quanta with each feature vector. In addition, an integrated feature vector is obtained based on one or more feature vectors assigned the same quantum.

  The binarizing unit 14 obtains a binary vector by applying the transformation matrix 32 stored in the storage unit 3 to each feature quantity vector to which the quantum is assigned, and stores the obtained binary vector in the content database 2. Or output to the outside. In the present embodiment, a binary vector is obtained by applying a transformation matrix to the integrated feature vector.

  There are several formats for storing / outputting quantum and binary vectors. An example is shown in FIGS. When the content is input, the feature quantity generation device 1 in the embodiment of the present invention generates a short binary vector, one for each content and quantum. Here, it should be noted that there is content that does not have a specific quantum as a result of assigning quanta to one or more feature quantity vectors expressing a certain content. For example, in the example of FIG. 2, the content with content ID = 2 does not have a feature quantity vector to which the quantum with quantum ID = 2 is assigned, and this is expressed as N / A for convenience.

  In the simplest case, for example, as shown in FIG. 2, there can be a method of outputting and storing a binary vector so that it can be referred to using a content ID and a quantum ID as keys. Alternatively, if there is an interest in the entire binary vector for each content, as shown in FIG. 3, the short binary vectors generated for all quanta for each content are arranged in a line, and this is associated with the content ID. It may be output / stored. In the case of this format, a null character (tentatively “*” in the figure) is virtually assigned to a quantum (N / A) that does not exist, thereby expressing that a binary vector does not exist at that position. In fact, when comparing binary vectors, it is only necessary not to execute (skip) any operation where there is this character.

  Further, the format shown in FIG. 4 is practical for the purpose of content search. This enumerates and stores content IDs having the quantum ID using the quantum ID as a key. Further, similarly, with the quantum ID as a key, binary vectors in the content ID / quantum ID are listed and stored in the same order as the content ID having the quantum ID. The advantage of this format is that when executing a search process in content search, content having a certain quantum (quantum ID) is referred to, and comparison (binary vector) is often performed within that range. Necessary information (content ID and binary vector) can be referred to immediately without excess or deficiency.

  When the configuration including the search device 6 is adopted, a binary vector is generated by the feature quantity generation device 1 for the content 21 stored in advance in the content database 2 and is also stored in the content database 2 in the previous format. And At this time, when the content 4 is input by the user and the quantum / binary vector information 5 generated / output by the feature quantity generation device 1 is received as an input, the search device 6 searches the content database 2. The search result 7 is output. The quantum / binary vector information 5 is information that can identify the quantum of the content 4 and the corresponding binary vector. As long as the embodiment of the present invention can be applied, the information possessed by the quantum / binary vector information and the binary vector is equivalent, and hereinafter, the corresponding information will be referred to as a binary vector unless confused.

<< Process overview >>

  Next, processing of the feature quantity generation device 1 in the present embodiment will be described. The feature quantity generation device 1 in the present embodiment executes a conversion learning process for learning and generating a feature quantity conversion function and a binary vector generation process for converting an initial feature quantity vector into a binary vector. Hereinafter, these two processes will be described.

  First, the conversion learning process will be described. FIG. 5 is a flowchart showing the flow of the conversion learning process. The conversion learning process is a process of learning and generating the quantizer 31 and the conversion matrix 32, and is a process that is performed at least once before a binary vector of content is actually generated.

  First, in step S201, the feature extraction unit 11 acquires the content 21 from the content database 2, performs a feature extraction process on each of the content files included in the content 21, and extracts a plurality of feature quantity vectors. Store in database 2.

  Subsequently, in step S202, the conversion learning unit 12 reads a plurality of feature quantity vectors from the content database 2, and based on the plurality of feature quantity vectors, the feature quantity vector, the quantizer, and the transformation matrix are used. The quantizer and the transformation matrix are learned and generated so as to minimize the error between the binary vector obtained from the vector and stored in the storage unit 3.

  The quantizer 31 and the transformation matrix 32 can be generated from the content 21 stored in the content database 2 by the processes in steps S201 to S202 described above. Details of various processes will be described later.

  Next, the binary vector generation process will be described. FIG. 6 is a flowchart showing the flow of binary vector generation processing. The binary vector generation process is a process of generating a binary vector from a set of one or more feature quantity vectors representing the input content using the quantizer 31 and the transformation matrix 32 stored in the storage unit 3. .

  The content input in this processing may be read from the content database 2 or input from the outside, and essentially the same processing is applied. Hereinafter, for the sake of simplicity, a description will be given of a case where input is performed from the outside (when content 4 in FIG. 1 is input).

  First, in step S <b> 301, the feature extraction unit 11 obtains the content 4 from the outside, extracts each feature vector that is an element of a set of one or more feature quantity vectors, and transmits it to the quantization unit 13. The transmission destination may be the content database 2 instead of the quantization unit 13.

  Subsequently, in step S302, the quantization unit 13 applies the quantizer 31 acquired from the storage unit 3 to each of the one or more feature amount vectors extracted in step S301, thereby obtaining the feature vector. At least one of a predetermined number of quanta is assigned to each and output to the binarization unit 14.

  Finally, in step S303, the binarizing unit 14 applies the transformation matrix 32 acquired from the storage unit 3 based on the assigned quantum to each of the feature vectors to which the quantum is assigned in step S302. A binary vector is generated and output to the search device 6.

  Through the above processing, a binary vector (quantum / binary vector information 5) can be obtained for the input content 4.

<< Details of each process >>

  Hereinafter, an example of the detailed processing of each processing will be described in the present embodiment.

[Feature extraction]

  A method for extracting a set of one or more feature quantity vectors representing each content will be described. The initial feature amount extraction process depends on the type of content, and the feature amount vector that can be extracted / changed by this changes, but the extracted feature amount vector is a vector having a fixed dimension that is uniquely determined for the content. Any known one can be used as long as it can be expressed as: Here, the case where the content is an image will be described.

  As a most preferable example, a local feature amount is extracted. In general, the local feature amount is composed of a feature point detection method for detecting a region called a feature point from an image and a descriptor that describes an image feature of the region as a feature amount vector. For example, the SIFT described in Non-Patent Document 1 detects feature points using a feature point detection method called “Difference of Gaussian (DoG)”, and each feature point is described by a descriptor expressing the gradient of the luminance value. Various other feature point detection methods and descriptors are generally known, and any method may be used. Further, it is not always necessary to detect the feature points based on the feature point detection method, and any procedure may be taken as long as the region in the image is determined. For example, the entire image may be set as a region, or the region may be taken at a fixed ratio (in a lattice pattern) with respect to the height and width of the image.

  It is preferable to employ a method that can obtain a feature vector that is invariant to the position / posture (angle) of the region and its size, such as SIFT. Hereinafter, an example of this embodiment will be described assuming that SIFT is used. When SIFT is used, the extracted local feature amount is usually a 128-dimensional real value vector. Therefore, one image is expressed as a set of 128-dimensional real value vectors corresponding to the number of feature points.

  Further, it is not always necessary to use local feature amounts. For example, brightness features, color features, or shape features may be extracted for each region.

  The brightness feature can be extracted as a histogram by counting the V values in the HSV color space. In this case, each image is represented as a vector having the same number of dimensions as the number of quantizations of the V value (for example, 256 gradations for 16-bit quantization).

  The color feature can be extracted as a histogram by counting the values of the respective axes (L *, a *, b *) in the L * a * b * color space. The number of histogram bins on each axis may be, for example, 4 for L *, 14 for a *, 14 for b *, etc. In this case, the total number of bins for 3 axes is 4 × 14 × 14 = 784, that is, a 784-dimensional vector.

  For example, a feature vector known as Histogram of Oriented Gradients (HOG) described in Reference 1 may be extracted as the shape feature.

[Reference 1] Navneet Dalal, Bill Triggs. Histograms of Oriented Gradients for Human Detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 886-893, 2005.

  Further, the feature amount by the neural network described in Reference 2 may be extracted.

[Reference 2] Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Proc. Advances in Neural Information Processing Systems (NIPS). Pages. 1097-1105, 2012.

  As described above, one or more feature quantity vectors expressing content can be obtained.

[Transform learning: Quantizer, transformation matrix generation]

  Next, a quantizer and a method for generating a transformation matrix will be described.

Here, a quantizer and a transformation matrix are generated using feature quantity vectors extracted from M contents. As shown below (1), a set of feature vectors extracted from the j-th content expressed as S ^j, represents the l-th element of S ^j and s ^j _l. s ^j _l is a feature vector, and its dimension is D. For convenience, let S be a set of feature quantity vectors extracted from all contents.

Here, the number of elements of S (that is, the number of feature quantity vectors extracted from all contents) is represented as N. Further, X represents a matrix representation of the elements of S. X is a matrix having a size of D × N. Each column of X is the original feature vector itself, which is expressed as x _i (i = 1, 2,..., N).

  Next, an example of a procedure for converting an arbitrary feature vector x into a binary vector will be described.

  In an example of an embodiment of the present invention, the image is converted into a binary vector roughly by two steps of quantization and binarization. Quantization is a process of assigning one or more feature vector x to the nearest quantum using a quantizer, and binarization obtains a binary vector based on the feature vector and the assigned quantum. It is processing.

First, the quantization in the quantization unit 13 will be described. The quantizer 31 according to an example of the embodiment of the present invention is defined by a representative vector representing a predetermined number of K quanta (sometimes referred to as code extension or the like). However, obtaining a quantizer that quantizes to K quanta is equivalent to obtaining K representative vectors. A representative vector corresponding to the k-th quantum is denoted by μ _k . The number of dimensions of this representative vector is the same as that of the feature vector. If such a representative vector group is used, a quantum for an arbitrary feature vector x can be assigned. That is, assuming that P quanta are assigned, P representative vectors that are closest to the feature vector x are obtained, and quanta corresponding to the representative vector may be assigned. P may be any natural number, for example, P = 5. With this process, a quantum (and a representative vector corresponding to the quantum) can be assigned to an arbitrary feature vector.

Next, binarization in the binarization unit 14 will be described. In an example embodiment of the present invention, binarization is performed using a transformation matrix R of size c × D. Here, c is the number of bits of the generated binary vector, and an arbitrary natural number determined in advance may be designated. In an example of an embodiment of the present invention, c ≦ D. For example, if D = 128, c = 128, c = 64, c = 32, and the like. Also assume that each column of R is orthogonal to each other. Through the previous quantization, one or more quanta and a representative vector corresponding thereto are assigned to the feature vector x. If the k-th quantum is assigned to the feature vector x, the binary vector b _k of the feature vector x corresponding to the k-th quantum is

Is required. The value of the binary vector b _k obtained in this way is determined by the direction (angle) of the feature vector x when the representative vector μ _k is regarded as the starting point. For example, suppose that there are two feature quantity vectors x _i and x _j . At this time, when the directions of these two feature vectors with respect to the representative vector μ _k completely match, all the bits of the binary vector obtained by the expression (2) match (the distance between the two binary vectors is 0). Conversely, if the direction for μ _k is true, all bits are reversed (the distance between the two binary vectors is c). In addition, the calculation using the equation (2) is similarly executed for other quantum assigned to the feature vector x.

  The above is the procedure for obtaining a binary vector for the feature vector x.

Now, the point in the embodiment of the present invention is that the quantum vector and the feature vector represented by the binary vector b _k generated as a result of using the quantizer and the transformation matrix are minimized. It is to obtain a binary vector that can obtain good search accuracy by obtaining a generator and a transformation matrix.

This error will be described with reference to FIG. When the quantization and binarization procedures described above are followed, the positional relationship between the original feature vector x, the quantum representative vector μ _k assigned to the feature vector x, and the generated binary vector b _k Can be shown as shown in FIG. That is, the above quantization and binarization procedures are equivalent to trying to approximate the feature vector x with a vector of μ _k + Rb _k , and as a result, the error is

Can be expressed as

In the conversion learning in the embodiment of the present invention, the error represented by the equation (3) is minimized with respect to N feature vector vectors x _i (i = 1, 2,..., N). This is to obtain a representative vector μ _k (k = 1, 2,..., K) and a transformation matrix R. Hereinafter, for simplicity, the number of quantum assigned to the feature vector x is assumed to be one. At this time, a unique binary vector b _i is obtained for x _i . Therefore, consider the sum total of N feature vectors for equation (3). If this is expressed as an index set N ^k of feature vectors assigned to the k th quantum, this sum is

It can be expressed as. As shown in the equation (2), the binary vector is determined by the sign taken by R ^T (x−μ _k ) and is invariant to the scale. Therefore, the size c is allowed to allow the degree of freedom. Of the diagonal matrix Λ,

Write. In this way, by using the transformation matrix R and the diagonal matrix Λ, an error is obtained by restoring the feature vector represented by the binary vector for the feature vector x _i . As a result, in order to obtain the quantizer 31 and the transformation matrix 32 so as to minimize the error between the feature vector x _i and the binary vector, all unknown variables are set so as to minimize the equation (5). You just have to ask for it. Specifically, the optimization problem of the following equation (6) is solved.

  However, Δ is a set of diagonal matrices.

A procedure for solving the problem of the above equation (6), that is, K representative vectors {μ _k }, transformation matrix R, and other unknown variables diagonal matrix Λ, {N ^k }, and {b _i } The required procedure is explained below.

First, as initial values, {μ _k } and a transformation matrix R are generated by an appropriate method. For example, {μ _k } may be generated by the K-means method, and the transformation matrix R may be generated randomly. Subsequently, the following procedures <1> to <5> are repeated until a predetermined condition (for example, a predetermined number of times is repeated or the value of the expression (5) becomes smaller than a predetermined value) is satisfied.

<1> A binary vector {b _i } is obtained. {B _i } may be obtained according to equation (2) based on the feature vector x, the representative vector μ _k, and the transformation matrix R.

<2> Find a diagonal matrix Λ. Λ is obtained according to the following equation (7) based on the matrix X of the feature vector x and the transformation matrix R.

  However,

Is a column vector having 1 for all elements, mean is a function that takes a column average, and diag is a function that converts the vector into a diagonal matrix.

<3> A transformation matrix R is obtained. First, a matrix in which binary vectors {b _i } are arranged in the column direction is represented as B. R is determined by the following procedure. Next, based on the set X of the feature vector x, the matrix B in which the binary vectors {b _i } are arranged in the column direction, and the diagonal matrix Λ, a matrix Θ shown in the following equation (8) is obtained.

  Subsequently, the singular value decomposition of Θ in the above equation (8) is performed to obtain unitary matrices U and V and a singular value matrix Ω as in the following equation (9).

  Finally, the transformation matrix R is obtained by the following equation (10).

<4> Find all representative vectors {μ _k }. Of all of the feature matrix X vector storing represent k-th matrix obtained by extracting only the columns corresponding to feature vector assigned to the quantization and X _k. Also, similarly expressed as B _k binary vector matrix corresponding thereto. At this time, the representative vector μ _k corresponding to the k-th quantum is _obtained by the following equation (11) (the same applies to all k).

<5> Find {N _k }. N _k is an index set of feature vectors to which the k th quantum is assigned. This operation is nothing but the process of reassigning the quantum to each feature vector. For each feature vector x _i , a vector x _i -RΛb _i obtained by shifting x _i is obtained. Among all the representative vectors {μ _k }, the one closest to the previous shifted vector is selected, and the selected representative vector is newly set as the quantum of the feature vector x _i .

  By the above procedure, a quantizer (representing vector that defines the above equation (5)) and a transformation matrix can be obtained.

[Quantization]

As described above, the quantizer 31 according to an example of the embodiment of the present invention is defined by the representative vector μ _k representing the K quanta generated as described above. What is necessary is just to obtain P representative vectors that are closest to the feature vector x and assign a quantum represented by the representative vector.

  In the quantization unit 13, when quantization is performed on each of all feature amount vectors extracted for a certain content, one or more feature amount vectors assigned to the same quantum exist. It happens. In such a case, it is preferable to obtain an integrated feature vector obtained by integrating one or more feature vectors assigned to the same quantum into a single feature vector. The integrated feature vector may be, for example, an average vector or median vector of feature vectors assigned to the same quantum, or a feature vector closest to the representative vector of the quantum. By obtaining the integrated feature quantity vector in this way, it becomes possible to effectively compress information of a plurality of feature quantity vectors. The integrated feature vector also has no difference from the feature vector in that it is a vector having the same dimensionality as the feature vector, and these are used without particular distinction hereinafter.

[Binaryization]

  In the binarizing unit 14, each feature vector is assigned to the quantum through the processing of the quantizing unit 13. Therefore, as described above, according to the equation (2), A binary vector may be obtained by applying the transformation matrix 32 to binarize.

<< Application to content search >>

  An example of an embodiment in the case where the feature quantity generation device according to the embodiment of the present invention described above is used for content search will be described. Here, a case where the content is an image will be described.

For example, it is assumed that M database images are stored in the content database 2. It is assumed that the representative vector group {μ _k } that defines the quantizer and the transformation matrix R are obtained through the transformation learning process described above and stored in the storage unit 3. Further, the M database images are described above. Assume that a binary vector has already been generated by the processing described above and stored in the format of FIG. At this time, when a new query image is given by the user, the purpose is to search for a database image having the same meaning content as the query image.

First, it is assumed that a feature extraction process is performed on the query image Q to extract a set of one or more feature quantity vectors {x ^q ₁ ,..., X ^q _n }. Quantities are assigned to each feature vector by applying the previous quantizer and quantizing. As a result, it is assumed that L (≦ K) quanta are allocated. For the feature quantity vectors to which the same quantum is assigned, the average vector is obtained and set as an integrated feature quantity vector. By this processing, the query image Q is expressed by L (integrated) feature quantity vectors.

  Subsequently, based on the equation (2), the feature vector is converted into a binary vector for each quantum using the transformation matrix R. Through the above processing, the query image Q is expressed by a binary vector for the quantum number.

  Next, search processing is performed. Assume that a binary vector of a database image is stored in the format shown in FIG. First, in the query image, quantum IDs that are not empty (that is, L number of quantum assigned to the feature vector) are listed, and the binary vector of the database image is referenced using the quantum ID as a key. For example, if the quantum ID for the kth quantum is k, all binary vectors registered in the kth quantum ID field may be referred to.

Next, a binary vector comparison is performed for each binary vector of the referenced database image. For example, if the binary vector of the query image for the k-th quantum is expressed as z ^q _k and one of the binary vectors of the referenced database image is expressed as z ^d _k , these are binary vectors. Can be compared by taking the exclusive OR of the two and then measuring the distance by counting the number of 1 bits. Alternatively, the inner product of z ^q _k and z ^d _k may be obtained, and in this case, the similarity between the two can be obtained.

  After the above comparison is made for all quantum IDs and all binary vectors, the sum of distances (or similarities) obtained for all quanta is taken for each database image, and this sum is the smallest (or largest). ) Output as database images with the same or similar meaning in order.

  Since the binary vector has a small amount of information and a low capacity, and can calculate the distance (or similarity) at high speed with a small number of operations, it can be processed efficiently. Further, the binary vector obtained by the present invention is obtained so that the error between the original feature vector and the binary vector is minimized, and the loss is very small. As a result, it is possible to find a database image with high accuracy as in the original feature vector.

  The above is an example of an embodiment of the present invention.

  As described above, according to the feature value generating apparatus according to the embodiment of the present invention, the quantizer 31 and the transformation matrix 32 are converted into the feature value vector using the feature value vector, the quantizer, and the transformation matrix. And the quantizer 31 is applied to each of the feature amount vectors so that at least one of a predetermined number of quanta is applied to each of the feature amount vectors. By assigning one and applying the transformation matrix 32, it is possible to accurately and efficiently find content having the same or similar meaning content by obtaining a binary vector.

  In addition, according to the embodiment of the present invention, a result of converting a set of one or more feature vectors into a binary vector having a smaller information amount that minimizes an error that is an information loss is obtained. As described above, it is possible to provide a feature amount generation method, a feature amount generation apparatus, and a feature amount generation program that can generate a binary vector with a small amount of information representing content.

  In addition, since the binary vector of the content generated by the feature generation device according to the embodiment of the present invention has a smaller amount of information compared to the original set of one or more feature vector, and is a binary vector, High-speed comparison operation is possible.

  Further, since the binary vector of the content generated by the feature generation device according to the embodiment of the present invention is generated so that an error from the original one or more feature vector is minimized, there is a problem in the conventional technique. Therefore, it is possible to suppress the deterioration of accuracy due to the quantization error, and it is possible to search the content with high accuracy.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 Feature-value production | generation apparatus 2 Content database 3 Memory | storage part 4, 21 Content 5 Quantum / binary vector information 6 Search apparatus 7 Search result 11 Feature extraction part 12 Conversion learning part 13 Quantization part 14 Binaryization part 31 Quantizer 32 Conversion matrix

Claims (5)

A feature quantity generation device that converts content expressed by a set of one or more feature quantity vectors into a binary vector that has an information quantity smaller than the information quantity of the one or more vector sets,
By applying a predetermined quantizer to each feature quantity vector that is an element of the set of one or more feature quantity vectors representing the feature quantity generation target content, each of the feature quantity vectors A quantization unit that assigns at least one of a predetermined number of quanta to
A binarization unit that obtains the binary vector by applying a predetermined transformation matrix obtained in advance for each of the feature quantity vectors to which the quantum is assigned;
Including
The quantizer and the transformation matrix are the binary obtained from the feature quantity vector and the feature quantity vector using the quantizer and the transformation matrix, based on the plurality of inputted feature quantity vectors. A feature quantity generation device characterized by being determined so as to minimize an error from a vector. The quantization unit applies the quantizer to each feature vector that is an element of the set of one or more feature vectors, so that the predetermined number of feature vectors are applied to each feature vector. Assigning at least one of the quanta and obtaining an integrated feature vector based on one or more feature vectors assigned the same quantum;
The feature quantity generation device according to claim 1, wherein the binarization unit obtains the binary vector by applying the transformation matrix to the integrated feature quantity vector. Feature quantity generation in a feature quantity generation device that converts content expressed by a set of one or more feature quantity vectors into a binary vector that has a smaller information quantity than the information quantity of the one or more vector sets. A method,
The quantizing unit applies a predetermined quantizer to each of the feature quantity vectors that are elements of the set of one or more feature quantity vectors that express the content of the feature quantity generation target, A quantization step of assigning at least one of a predetermined number of quanta to each feature vector;
A binarization unit that obtains the binary vector by applying a predetermined transformation matrix obtained in advance for each of the feature vector to which the quantum is assigned;
Including
The quantizer and the transformation matrix are the binary obtained from the feature quantity vector and the feature quantity vector using the quantizer and the transformation matrix, based on the plurality of inputted feature quantity vectors. A feature value generation method characterized by being determined so as to minimize an error from a vector. The step of assigning by the quantization unit includes applying the quantizer to each feature vector that is an element of the set of one or more feature vectors, so that the feature vector is applied to each feature vector. Assigning at least one of a predetermined number of quanta and obtaining an integrated feature vector based on one or more feature vectors assigned to the same quantum;
4. The feature quantity generation method according to claim 3, wherein the step of obtaining the binarization section obtains the binary vector by applying the transformation matrix to the integrated feature quantity vector.   The program for making a computer perform each step of the feature-value production | generation method of Claim 3 or Claim 4.