JP5382786B2 - Feature quantity generation device, feature quantity generation method and feature quantity generation program, class discrimination device, class discrimination method, and class discrimination program - Google Patents

Description

  The present invention relates to a feature amount generation apparatus, a feature amount generation method, and a feature amount generation method, which generate a feature vector indicating the entire feature of one piece of data using a plurality of higher-order local feature vectors extracted from one piece of data indicating real world information. The present invention relates to a feature amount generation program, a class determination device, a class determination method, and a class determination program for determining to which of a plurality of classes new data indicating real world information belongs.

  Image data, audio data, and the like indicate real world information such as visual information and auditory information. In order to be able to search for data indicating this type of real world information and to determine the contents of novel data, It is necessary to appropriately grasp the feature amount of the entire data indicating the world information. Conventionally, the Bag-of-Keypoints method is known as a method used to represent a global feature of one piece of image data (see, for example, Non-Patent Document 1). The Bag-of-Keypoints method uses a predetermined local feature descriptor to cluster local feature vectors extracted from target image data, obtains representative vectors (visual words) of the clusters, and extracts them from the image data. By assigning local features to the closest “visual words”, the features of the entire image data are expressed as a set of local features. In addition, as a feature point detection (selection) method necessary for local feature extraction, “Difference of Gaussian”, a random feature point detection method (for example, refer to Non-Patent Document 2), or a grid called “Dense Sampling” is used. A feature point detection method (see, for example, Non-Patent Document 3) is known. As local feature descriptors, edge histograms, HSV color histograms, and the like are known, but in recent years, “SIFT descriptors” (for example, see Non-Patent Document 4) are also used.

G. Csurka, C. R. Dance, L. Fan, J. Willamowski and C. Bray. Visual Categorization with bags of keypoints. In Proc. ECCV Workshop on Statistical Learning in Computer Vision, 2004. E. Nowak, F. Jurie, and B. Trigges. Sampling strategies for bag-of-features image classification.In Proc.European Conference on Computer Vision, pages 490 ・ 503, 2006. L. Fei-Fei and P. Perona.A bayesian hierarchical model for learning natural scene categories.In Proc.IEEE Conf.Computer Vision and Pattern Recognition, pages 524 ・ 531, 2005. D. G. Lowe.Object recognition from local scale-invariant features.In Proc.IEEE International Conference on Computer Vision, pages 1150/1157, 1999.

Here, the following processes 1) to 3) are basically required in order to acquire the characteristics of the entire data indicating the real world information.
1) Detection of characteristic points (feature points) of images and normalization of scale orientation of the feature points 2) Description of partial image features (local features) around the feature points 3) All local features Here, the final image feature is calculated by using the above-described processes 1) and 2). For example, it is possible to extract a local feature with high accuracy such as the above-mentioned “SIFT descriptor” with less calculation cost. What to do has been proposed. However, with regard to the calculation of the final feature value from the local feature in 3) above, the problem in terms of calculation cost has not yet been solved, and the accuracy of the finally obtained feature value (feature expression level) ) Is still a problem. For example, the Bag-of-Keypoints method described above requires an extremely large amount of time for clustering, and even if a feature vector obtained over a long time is used, a dramatic improvement in image recognition accuracy has not been recognized. It is inferior in so-called scalability. In addition, if the accuracy of the feature quantity indicating the characteristics of one piece of data is low, the accuracy of content determination (class determination) of the novel data at the time of appearance of the novel data will naturally decrease.

  In view of the above, a feature amount generation apparatus, feature amount generation method, and feature amount generation program according to the present invention are mainly intended to enable generation of an accurate feature amount from data indicating real world information with less calculation cost. The main object of the class discriminating apparatus, class discriminating method and class discriminating program according to the present invention is to make it possible to discriminate with high accuracy which of a plurality of classes new data indicating real world information belongs.

  The feature quantity generation device, feature quantity generation method and feature quantity generation program, class discrimination device, class discrimination method and class discrimination program of the present invention employ the following means in order to achieve the main object described above.

The feature value generation apparatus of the present invention is:
A feature quantity generation device that generates a feature vector that represents a feature of the entire data using a plurality of higher-order local feature vectors extracted from one data that represents real world information,
Average acquisition means for acquiring an average vector of the plurality of higher-order local feature vectors;
First-order to M-th order m-order correlation vectors between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to value 1, and “m” is an integer between value 1 and value M) Correlation acquisition means for acquiring
Feature vector obtaining means for obtaining the feature vector based on elements constituting the average vector obtained by the average obtaining means and elements constituting the m-th order correlation vector obtained by the correlation obtaining means;
Is provided.

  From the viewpoint of reducing the calculation cost when expressing the global feature of one piece of data indicating real world information, the present inventors simply use a plurality of higher-order local feature vectors extracted from the data. We returned to expressing the global feature of the whole data based on the average vector. However, if only the average of the higher-order local feature vectors is used, all the distribution information of the local features that are important in properly expressing the features of the entire data will be lost. Therefore, in order to obtain a feature vector having a high feature expression level, it is important to express the distribution information of local features more appropriately. However, the distribution information requires a great amount of calculation time to generate a final feature vector. However, it is not sufficiently expressed even in a relatively complicated conventional method that requires a large amount of data. This is because the number of local feature vectors extracted from one data is generally large from the viewpoint of calculation processing, but is not so large (sparse) from a global viewpoint. It is thought to be caused. Based on this, the present inventors focused on the m-th order correlation vector between a plurality of higher-order local feature vectors, and decided to express the distribution information of local features using the m-th order correlation vector. That is, in the feature quantity generation device according to the present invention, the elements constituting the average vector of a plurality of higher-order local feature vectors extracted from one data indicating real world information and the first between the plurality of higher-order local feature vectors. Based on the elements constituting the m-th order correlation vectors from the next to the M-th order, a feature vector indicating the characteristics of the entire data is acquired. Here, the m-th order correlation vector can be obtained by calculation processing that is significantly lighter than that of, for example, clustering and the like, and represents a correlation between important feature elements, that is, distribution information of local features. Is. As a result, according to the feature quantity generation device, it is possible to quickly obtain a feature quantity with high accuracy (high feature expression) from data indicating real world information while greatly reducing the calculation cost. The maximum order (value M) of the m-th order correlation vector at the time of generating the feature vector is arbitrarily determined according to the number and dimensions of higher-order local feature vectors, and may be the first order. It may be second order, third order or higher order. Furthermore, since the average vector of a plurality of higher-order local feature vectors can also be expressed as a zero-order correlation vector between the plurality of higher-order local feature vectors, the average acquisition means and the correlation acquisition means are a single calculation processing module. It may be constituted by.

Further, when p d-order local feature vectors extracted from one data I indicating real world information are V _k = (v ₁ ,..., V _d ) (where “p” and “d”) Each is an integer greater than or equal to value 2 and “k” is an integer from value 1 to value p), and the average acquisition means calculates the average vector μ of the p d-order local feature vectors V _k as The correlation acquisition means acquires the autocorrelation matrix R of the p d-order local feature vectors V _k according to the following equation (2) and the elements of the upper triangular matrix of the autocorrelation matrix R: To obtain the primary correlation vector upper (R), and the feature vector acquisition means, when the feature vector is X, is an element of the average vector μ according to the following equation (3) And elements of the primary correlation vector upper (R) It may be configured to obtain the feature vector X by. This makes it possible to quickly generate a feature vector that more appropriately represents the features of the entire data from a large number of relatively high-order local feature vectors.

  Furthermore, the correlation acquisition means may acquire the m-th order correlation vector with dimensional compression of the higher-order local feature vector by principal component analysis. As a result, when the dimension of the higher-order local feature vector is higher, it is possible to reduce the calculation cost associated with the acquisition of the m-th order correlation vector. Also, it is possible to remove local features that are unnecessary for expressing the features of the entire data by dimensional compression.

Also, assume that there are N pieces of data I ^(j) indicating real world information (where “N” is an integer greater than or equal to value 2 and “j” is an integer between value 1 and value N); The p ^(j) d-th order local feature vectors extracted from one data I ^(j) are set as V _k ^(j) = (v ₁ ,..., V _d ) (where “p ^(j) ” and “ d ”is an integer of value 2 or more, and“ k ”is an integer from value 1 to value p), and the p ^(j) d-th order local feature vectors V _k acquired by the average acquisition means. Is an average vector of μ ^(j) shown in the following equation (4), and an autocorrelation matrix of the p ^(j) d-order local feature vectors V _k ^(j) is R ^(j) shown in the following equation (5 ^). When the autocorrelation matrix of the entire d-th order local feature vector extracted from the N pieces of data is R _all shown in the following equation (6) and the novel data is I ^{(j + 1)} , the correlation acquisition is performed. Means A projection matrix U _dl to dl next principal component space than d following obtained by solving the eigenvalue problem is a low following equation (7), extracted from the novel data ^{I (j + 1) p (} j ^A diagonal matrix U _dl ^T R ^{(j + 1)} U _dl based on the autocorrelation matrix R ^{(j + 1) of +1)} d-th order local feature vectors V _k ^{(j + 1)} , and Even if the diagonal matrix U _dl ^T R ^{(j + 1)} U _dl is enumerated and the primary correlation vector upper (U _dl ^T R ^{(j + 1)} U _dl ) is obtained by enumerating the elements of the upper triangular matrix The feature vector acquisition means may include an element constituting an average vector μ ^{(j + 1)} of the p ^{(j + 1)} d-order local feature vectors V _k ^{(j + 1)} according to the following equation (8): intended to obtain the feature vector X of the novel data ^{I (j + 1) (j} + 1) by enumerating the elements constituting the primary correlation vector _{^{upper (U dl T R (j}} + 1) U dl) There may be. In this case, by calculating the projection matrix U _dl using N pieces of data I ^(j) in advance, when the new data I ^{(j + 1)} appears, the new data I ^{(j + 1)} The feature vector X ^{(j + 1)} can be acquired quickly.

A feature value generation method according to the present invention includes:
A feature amount generation method for generating a feature vector indicating a feature of the entire data using a plurality of higher-order local feature vectors extracted from one data indicating real world information,
The average vector of the plurality of higher-order local feature vectors and the m-th order correlation vector from the first order to the M-th order between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to 1) , “M” is an integer from value 1 to value M),
The feature vector is acquired based on elements constituting the acquired average vector and elements constituting the acquired m-th order correlation vector.

  According to this method, it is possible to quickly generate a feature quantity with high accuracy (high feature expression) from data indicating real world information while greatly reducing the calculation cost.

The feature value generation program according to the present invention is:
A feature amount generation program that causes a computer to function as a device that generates a feature vector indicating the characteristics of the entire one data using a plurality of higher-order local feature vectors extracted from one data indicating real world information,
An average acquisition module for acquiring an average vector of the plurality of higher-order local feature vectors;
First-order to M-th order m-order correlation vectors between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to value 1, and “m” is an integer between value 1 and value M) A correlation acquisition module for acquiring
A feature vector acquisition module for acquiring the feature vector based on an element constituting the average vector acquired by the average acquisition module and an element constituting the m-th order correlation vector acquired by the correlation acquisition module;
Is provided.

  By using a computer in which this feature quantity generation program is installed, it is possible to quickly generate feature quantities with high accuracy (high feature expression) from data indicating real world information while greatly reducing calculation costs. .

The class identification device according to the present invention comprises:
A class discriminating apparatus for discriminating to which one of a plurality of classes each corresponding to at least one known data each of novel data indicating real world information belongs,
Each of the novel data and the known data in the h-th layer is h × h (where “h” is an integer from value 1 to value H, and “H” is an integer greater than or equal to value 2). The feature vector derived for each region based on the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer is set as a latent space. Conversion storage means for storing conversion for projecting;
Assuming that the novel data is divided into h × h regions in the h-th layer, a plurality of higher-order local features from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer Local feature extraction means for extracting vectors;
An average vector of a plurality of higher-order local feature vectors extracted by the local feature extraction means from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and the plurality of higher-order M-th order correlation vectors from the first order to the M-th order between local feature vectors (where “M” is an integer greater than or equal to value 1 and “m” is an integer between value 1 and value M). And a feature vector acquisition means for acquiring a feature vector of each of the regions based on an element constituting the average vector and an element constituting the m-th order correlation vector,
For each class, it is obtained by projecting the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer into the latent space by the transformation corresponding to the region. A projection point, and a projection point obtained by projecting the feature vector of each region obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the region; Based on the feature vector of the i-th area (where “i” is an integer from 1 to h ² ) in the h-th layer of the known data, the i-th in the h-th layer of the novel data Probability derivation for deriving the sum of the probability from the first layer to the H-th layer from i = 1 to i = h ² and the probability that the feature vector of the new data appears from the class Means,
Class setting means for setting a class having a maximum probability derived by the probability deriving means as a class to which the novel data belongs;
Is provided.

This class discriminating device is an area obtained by dividing a plurality of known data and novel data by h × h in each layer from the first layer to the H layer when discriminating which of the plurality of classes the new data belongs to. Elements constituting an average vector of a plurality of higher-order local feature vectors extracted from each of the elements, and elements constituting m-th order correlation vectors from the first order to the M-th order among the plurality of higher-order local feature vectors, The feature vector of each area acquired based on the above is used. Such a feature vector can be obtained at a low calculation cost and can well express the feature of the target region. Then, this class discriminating apparatus projects the feature vector of each area obtained by dividing each of the known data in each layer from the first layer to the Hth layer into the latent space for each class by conversion corresponding to the area. And the projection points obtained by projecting the feature vectors of the regions obtained by dividing the novel data in each layer from the first layer to the H layer into the latent space by conversion corresponding to the regions. Based on the above, i = 1 to i = h ^{2 of} the probability that the feature vector of the i-th region in the h-th layer of novel data appears from the feature vector of the i-th region in the h-th layer of known data and The sum from the first layer to the Hth layer is derived as the probability that a feature vector of novel data appears from the class. In this way, a feature vector that can be acquired at a low calculation cost and has a high degree of feature expression, and probabilistic linear discriminant analysis (S. Ioffe. Probabilistic linear discriminant analysis. In Proc. European Conference on Computer) Vision, pages 531-542, 2006.) and a method that introduces an extension that multiplexes the latent space, and the probability that the feature vector of novel data will appear from a certain class is more accurate and prompt. Can be derived. Therefore, according to this class discriminating apparatus, it is possible to discriminate with high accuracy which of the plurality of classes the novel data indicating the real world information belongs from the probability derived for each class. The known data and novel data divided into 1 × 1 = 1 pieces in the first layer become the known data and the novel data themselves, and higher-order local feature vectors and feature vectors extracted and generated in the first layer. Is extracted and generated directly from known data and novel data itself.

In the transformation for the i-th region in the h-th layer, the number of the classes is G (where “G” is an integer equal to or greater than 2), and the class is C _g (where “g “Is an integer from value 1 to value G), and n is the number of sample data that is known data extracted as a sample from class C _g (where“ n ”is an integer greater than or equal to value 1). ), The feature vector of the i-th region in the h-th layer of the j-th sample data belonging to the class C _g (where “j” is an integer from the value 1 to the value n) is represented by X _j ^{g (h , i)} , the average vector of the feature vector X _j ^{g (h, i)} of the i-th region in the h-th layer of sample data belonging to class C _g is X ^{−g (h, i),} and class C _g The average vector of feature vectors of the i-th region in the h-th layer of all sample data to which it belongs is μ _x ^{(h, i)} , the intra-class covariance matrix for the i-th region in the h-th layer is Σ _w ^{(h, i)} shown in the following equation (9), and the i-th region in the h-th layer is When the out-of-class covariance matrix is Σ _b ^{(h, i)} shown in the following equation (10), even if it is a projection matrix W ^{(h, i)} obtained by solving the eigenvalue problem in the following equation (11) Well (however, “Λ ^{(h, i)} ” in equation (11) is a diagonal matrix obtained by sequentially arranging eigenvalues as discrimination criteria diagonally), and the feature vector is X, and the projection matrix is When W is the projection point of the feature vector X and u is the projection point of the feature vector of each region obtained by dividing each of the sample data in each layer from the first layer to the H layer, Projection points of feature vectors of regions obtained by dividing the novel data in each layer from the 1st layer to the Hth layer are as follows: 12) may be derived in accordance with the feature vector of the novel data when the X _s, the probability the feature vector X _s appears from class _{C g p (X s | C} g) , the following equation (13 ). However, the subscript (h, i) in equation (13) indicates that it is derived from the i-th region in the h-th layer, the subscript s indicates that it is derived from novel data, and the subscript C _g is the class. It indicates that it belongs to C _g, subscript 1 ... n indicates that from the first 1~n th sample data belonging to the class C _g, "alpha ^h" is the weight to be given to the h layer “Z ^{(h, i) Cg} ” and “Θ ^{(h, i)} ” in the equation (13) are as shown in the following equations (14) and (15), and u ^{− ( h} , ^{j) Cg} is the average of the projection points u ^(h , ^{j) Cg} of the feature vector X ^(h , ^{j) Cg} belonging to the class C _g , and “Ψ ^{(h, i} ) in the equations (14) and (15) ⁾ "is the variance of the latent variables in the following equation (16), lambda in the equation (16) ^{(h, i)} is the solution of the eigenvalue problem in the i-th region in the h layer It is a diagonal matrix obtained by arranging eigenvalues diagonally in order. This makes it possible to derive the probability that a feature vector of novel data appears from a certain class with higher accuracy.

The class discrimination method according to the present invention includes:
A class discrimination method for discriminating to which one of a plurality of classes corresponding to at least one known data each new data indicating real world information belongs,
Each of the novel data and the known data in the h-th layer is h × h (where “h” is an integer from value 1 to value H, and “H” is an integer greater than or equal to value 2). Is converted into a potential space based on the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer. Derived for each region,
Assuming that the novel data is divided into h × h regions in the h-th layer, a plurality of higher-order local features from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer Extract the vector,
An average vector of a plurality of higher-order local feature vectors extracted from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and a first vector between the plurality of higher-order local feature vectors. M-order correlation vectors from the first order to the M-th order (where “M” is an integer greater than or equal to value 1, and “m” is an integer from value 1 to value M) and the average Obtaining a feature vector of each of the regions based on an element constituting the vector and an element constituting the m-th order correlation vector;
For each class, it is obtained by projecting the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer into the latent space by the transformation corresponding to the region. A projection point, and a projection point obtained by projecting the feature vector of each region obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the region; Based on the feature vector of the i-th area (where “i” is an integer from 1 to h ² ) in the h-th layer of the known data, the i-th in the h-th layer of the novel data th derives a sum of i = 1 of the probability that the feature vector appears in the area from the i = h ² and the first layer to the second H layer as the probability that the feature vector of the novel data from the class appears,
The class having the maximum derived probability is set as the class to which the novel data belongs.

  According to this method, it is possible to determine with high accuracy which of the plurality of classes the new data indicating the real world information belongs from the probability derived for each class.

The class discrimination program according to the present invention is:
A class determination program that causes a computer to function as a class determination device that determines which of a plurality of classes corresponding to at least one known data each of novel data indicating real world information,
In the h-th layer, the novel data is divided into h × h areas (where “h” is an integer from value 1 to value H and “H” is an integer greater than or equal to value 2). A local feature extraction module that extracts a plurality of higher-order local feature vectors from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer,
An average vector of a plurality of higher-order local feature vectors extracted by the local feature extraction module from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and the plurality of higher-order M-th order correlation vectors from the first order to the M-th order between local feature vectors (where “M” is an integer greater than or equal to value 1 and “m” is an integer between value 1 and value M). A feature vector obtaining module that obtains a feature vector of each of the regions based on an element constituting the average vector and an element constituting the m-th order correlation vector;
For each class, the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer is projected onto the latent space by conversion corresponding to the predetermined region. Projection points obtained by projecting the obtained projection points and the feature vectors of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the regions. Based on the feature vector of the i-th region in the h-th layer of the known data (where “i” is an integer from the value 1 to the value h ² ) in the h-th layer of the novel data. The probability that i = 1 to i = h ^{2 of} the probability that the feature vector of the i-th region appears and the sum from the first layer to the H-th layer is derived as the probability that the feature vector of the novel data appears from the class. A rate derivation module;
A class setting module that sets a class having a maximum probability derived by the probability derivation module as a class to which the novel data belongs;
Is provided.

  According to the computer in which the class discrimination program is installed, it is possible to determine with high accuracy which of the plurality of classes the novel data indicating the real world information belongs from the probability derived for each class.

It is a schematic block diagram which shows the robot apparatus which concerns on one Example of this invention. It is explanatory drawing for demonstrating the process which produces | generates the feature vector which shows the characteristic of the whole image data. It is a flowchart which shows an example of a feature vector generation routine. It is a flowchart which shows an example of a class discrimination | determination routine. It is explanatory drawing for demonstrating the procedure which discriminate | determines to which of a some class new data belongs. It is explanatory drawing for demonstrating the procedure which discriminate | determines to which of a some class new data belongs. It is a graph which shows the evaluation result with the feature expression of the feature vector produced | generated by the feature vector production | generation method of this invention. It is a graph which shows the evaluation result of the effectiveness of the class identification method of this invention. It is a schematic block diagram of the image data processing system which concerns on a modification.

  Next, the form for implementing this invention is demonstrated using an Example.

  FIG. 1 is a schematic configuration diagram of a robot apparatus 20 according to an embodiment of the present invention. A robot apparatus 20 shown in FIG. 1 is a so-called humanoid robot having artificial intelligence, and includes an imaging unit 21 corresponding to a human eye, a sound collecting unit 22 corresponding to a human ear, a manipulator and a leg corresponding to a human hand. A number of actuators 23 for moving a movable part such as a part, a sound generation unit (not shown), a control computer 30 functioning as artificial intelligence, and the like are included. The control computer 30 includes an unillustrated CPU, ROM, RAM, graphic processor (GPU), graphic memory (VRAM), system bus, various interfaces, an external storage device such as a hard disk drive and a flash memory drive (SSD), and the like. The control computer 30 includes an input / output processing unit 31, a feature amount processing unit 32, a learning processing unit in cooperation with one or both of the hardware and software such as a feature amount generation program and a class determination program according to the present invention. 33, a discrimination processing unit 34, a retrieval processing unit 35, a main control unit 36, and the like are constructed. The control computer 30 is connected to a data storage device 40 that stores image data, audio data, and the like, a feature amount storage device 41, and a learning information storage device 42.

The input / output processing unit 31 processes information input / output to / from the robot apparatus 20 via the imaging unit 21, the sound collection unit 22, and the like. When acquired, the audio data from the sound collection unit 22 is appropriately processed and given to the main control unit 36. The feature amount processing unit 32 detects feature points (key points) of images and sounds from image data acquired by the imaging unit 21 and sound data acquired by the sound collection unit 22 by performing feature point detection using a grid, for example. (Selection) and, for example, by executing feature description at each feature point using the SIFT descriptor (see 1 and 2 in FIG. 2), a plurality of higher-order local feature vectors V _{k are} obtained from the target data. Are extracted and stored in the feature quantity storage device 41. Hereinafter, the dimension of the higher-order local feature vector V _k is assumed to be “d” (where “d” is an integer of 2 or more). Further, if the number of higher-order local feature vectors V _k extracted from certain image data is “p” (where “p” is an integer greater than or equal to 2), “k” is from value 1 to value p. It becomes an integer up to. Here, the feature description using the SIFT descriptor is a 128-dimensional local region from a P × P pixel region centered on the feature point while spacing the feature points of the monochrome image by L pixels. Feature vector (Gray-SIFT) is extracted. For color images, each feature point of the color image is described independently for each RGB, and local features extracted for each RGB are combined. A 384-dimensional local feature vector (RGB-SIFT) is generated. In the embodiment, in order to improve the robustness to the scale, the local feature vectors extracted from both the P = 16 region and the P = 36 region are enumerated to obtain the final higher-order local feature vector V _k. It is said. Further, the feature quantity processing unit 32 generates a feature vector X _j indicating the global feature of the image data or the audio data based on the extracted higher-order local feature vector V _k (see 3 in FIG. 2). ), And stored in the feature amount storage device 41. In addition, the feature quantity processing unit 32 extracts a feature vector indicating the feature of the data from metadata about symbols appearing in the image in correspondence with the image data or the sound data or symbols indicating the meaning of the sound. And stored in the feature amount storage device 41.

The learning processing unit 33 generates and updates learning information necessary for the processing of the discrimination processing unit 34 and the retrieval processing unit 35 by executing principal component analysis using the higher-order local feature vector V _k and the feature vector X _j. And stored in the learning information storage device 42. The discrimination processing unit 34 is a novel image data captured by the imaging unit 21 (an unannotated image to which no symbol or metadata indicating what is shown in the image associated with the image data or meaning of the sound is added). Data) and the like belong to a plurality of classes corresponding to a plurality of known image data and the like (symbols indicating a meaning of a plurality of image data classified as the same kind). Further, the discrimination processing unit 34 performs annotation on unannotated image data and unannotated audio data using learning information stored in the learning information storage device 42. The retrieval processing unit 35 executes search processing (retrieval) of unannotated image data and unannotated audio data based on symbols. The main control unit 36 determines the operation mode of the robot apparatus 20 based on the command from the input / output processing unit 31, the processing result of the discrimination processing unit 34, the processing result of the retrieval processing unit 35, and the like. Control.

Next, a procedure for generating the feature vector X _s indicating the global feature of the novel image data I _s captured by the imaging unit 21 in the robot apparatus 20 of the embodiment will be described. FIG. 3 is a flowchart showing an example of a feature vector generation routine executed by the feature amount processing unit 32 of the control computer 30 in order to generate the feature vector X _s of the novel image data I _s .

At the start of the feature vector generation routine of FIG. 3, the feature quantity processing unit 32 inputs data necessary for generating the feature vector X _s such as the novel image data I _s and the projection matrix U _dl , and a predetermined storage area (memory). (Step S100). The projection matrix U _dl is one piece of learning information stored in the learning information storage device 42, and is based on the known image data (including learning data) stored in the data storage device 40 by the learning processing unit 33. It is obtained in advance based on the extracted d-th order local feature vector. Specifically, the projection matrix U _dl is a projection matrix onto a principal component space of dl order (dl <d, eg, dl = 30) that is lower than the order d of the local feature vector, Assume that there is known image data I ^(j) (where “N” is an integer greater than or equal to 2 and “j” is an integer from value 1 to value N), and one known image data I ^{(j )} extracted from the p ^(j) number of d next local feature vectors ^{_{V k (j) = (v}} 1, ..., v d) and then, the average vector of the p ^(j) number of d next local feature vectors V _k Is μ ^(j) shown in the following equation (17), and the autocorrelation matrix of p ^(j) d-order local feature vectors V _k ^(j) is R ^(j) shown in the following equation (18). When the autocorrelation matrix of the entire d-th order local feature vector extracted from the known image data is R _all shown in the following equation (19), it is obtained as a solution to the eigenvalue problem of the following equation (20). Is.

After the data input at step S100, the feature quantity processing unit 32 initializes a variable h to a value 1 indicating the number of hierarchies when dividing the novel image data I _s hierarchically (step S110). Here, it is assumed that the novel image data I _s is divided into h × h areas in the h-th layer, which is a hierarchy corresponding to the variable h. However, “h” is an integer from value 1 to value H, and “H” is an integer greater than or equal to value 2. After the process of step S110, the feature quantity processing unit 32 performs the feature point detection by the grid and the feature description at each feature point using the SIFT descriptor, thereby generating the novel image data I _s in the h-th layer h. A plurality of (in the embodiment, the same number for each area) d-order local feature vector V _k ^{(h, i)} is extracted from each of the areas obtained by xh division (equal to h * h ⁾ , and a predetermined storage area (In the embodiment, the data is stored in the memory and the external storage device, that is, the feature amount storage device 41) (step S120). However, “i” is an integer from the value 1 to the value h ² (the number of the area after h × h division), and the subscript (h, i) is derived from the i-th area in the h-th layer. It shows that. In addition, since the novel image data I _s is divided into 1 × 1 areas in the first layer, when h = 1, in step S120, a plurality of d from the entire novel image data I _s. The next local feature vector V _k ^(1,1) is extracted. Then, the feature amount processing unit 32 performs the same calculation as the above equation (17) for each of the first to h ^2nd regions of the h-th layer, and a plurality of d-order local regions extracted from the respective regions. An average vector μ _Xs ^{(h, i)} of the feature vector V _k ^{(h, i)} is derived and stored in a predetermined storage area (memory) (step S130).

Then, the feature quantity processing unit 32, after initializing a predetermined variable m to a value 1 (step S140), for each area from the first th h layer to the ^second first h, extracted from each region M-th order correlation vectors between the plurality of d-th order local feature vectors V _k ^{(h, i)} (where “m” is an integer from value 1 to value M, and “M” is an integer greater than or equal to value 1) Is derived (step S150). In step S150, the feature quantity processing unit 32, if it is m = 1, for each area from the first th h layer to the ^second first h, the formula h-th layer i-th of according (18) An autocorrelation matrix R ^{(h, i)} of a plurality of d-order local feature vectors V _k ^{(h, i)} extracted from the region is obtained, and the autocorrelation matrix R ^{(h, i)} is input in step S100. projection matrix U _dl and its transpose matrix U _dl ^T diagonal matrix based on the _{^{U dl T R (h, i}} ) obtains the U _dl, the diagonal matrix _{^{U dl T R (h, i}} ) on the U _dl triangular The elements of the matrix are enumerated to obtain a primary correlation vector upper (U _dl ^T R ^{(h, i)} U _dl ), and then stored in a predetermined storage area (memory). Here, as described above, the projection matrix U _dl is a projection matrix onto the principal component space of the dl order (for example, dl = 30) that is lower than the order d (for example, d = 128 or 384) of the local feature vector. It is.

When the m-th order correlation vector upper (U _dl ^{TR (h, i)} U _dl ) is derived in step S150, the feature amount processing unit 32 determines whether or not the variable m is the maximum value M. (Step S160) If the variable m is less than the maximum value M, the variable m is incremented (Step S170), and the process of Step S150 is executed again. Even when the maximum value M of the variable m is 2 or more, it is preferable to perform dimensional compression of the appropriate d-th order local feature vector when deriving the m-th order correlation vector. When it is determined in step S160 that the variable m is the maximum value M, the feature amount processing unit 32 performs d according to the following equation (21) for each of the first to h ^2nd regions of the h-th layer. An element constituting the average vector μ _Xs ^{(h, i)} of the second-order local feature vector V _k ^{(h, i)} and an element constituting the m-th order correlation vector upper (U _dl ^T R ^{(j + 1)} U _dl ) The feature vectors X _s ^{(h, i)} of the novel image data I _s are generated by enumeration in order, and stored in a predetermined storage area (in the embodiment, the memory and the external storage device, that is, the feature amount storage device 41) ( Step S180). After the process of step S180, the feature amount processing unit 32 determines whether or not the variable h is the maximum value H (for example, H = 3 in the embodiment) (step S190), and the variable h is less than the maximum value H. If so, the variable h is incremented (step S200), and the processing after step S120 is executed again. If it is determined in step S190 that the variable h is the maximum value H, the feature vectors of the respective regions obtained by dividing the novel image data I _s by h × h in each layer from the first layer to the H-th layer. X _s ^{(h, i)} has been acquired, and this routine ends at that stage.

Subsequently, a plurality of classes C ₁ to novel image data I _s captured by the imaging unit 21 corresponding to a plurality of known image data, respectively, in the robot apparatus 20 of the _{embodiment, ..., C g, ...,} C G ( where " G ”is an integer from value 1 to value G, and“ G ”is an integer greater than or equal to value 2). FIG. 4 is a flowchart illustrating an example of a class determination routine executed by the determination processing unit 34 of the control computer 30 in order to determine which of the plurality of classes C _{1 to} C _G the novel image data I _s belongs to. .

Here, the class discrimination routine illustrated in FIG. 4 is constructed based on the framework of probabilistic linear discriminant analysis. In the framework of probabilistic linear discriminant analysis, the number of known image data (hereinafter referred to as “sample data”) extracted as a sample from class C _g is n, and the j th (where “j”) belongs to class C _g. the feature vector of the sample data is an integer of from the value 1 to the value n) and X _j ^g, an average vector of feature vectors X _j ^g sample data belonging to the class C _g and X ^-g (provided that hereby In the description and claims, the superscript bar indicates the upper line), the average vector of the feature vectors X _j ^g of all sample data is μ _x , and the intra-class covariance matrix is shown in the following equation (22): sigma and _w, a class outside covariance matrix when the sigma _b shown in the following equation (23), generalized eigenvalue problem of the following formula (24) is formulated. By solving the eigenvalue problem of Equation (24), a projection matrix W that is a transformation for projecting the feature vector X _j ^g to the latent space can be obtained. However, “Λ” in the equation (24) is a diagonal matrix obtained by sequentially arranging eigenvalues as discrimination criteria diagonally. Note that if the number of sample data is not sufficiently large with respect to the dimension of the feature vector, as shown in the following equation (25), the intra-class covariance matrix Σ _w obtained from the equation (23) is excessive. A regularization term γI may be added to suppress learning (where “γ” is an experimentally determined parameter). By using the projection matrix W thus obtained, a projection point (vector) u ^{(h, i)} in the latent space of the feature vector X ^{(h, i)} obtained by executing the above-described feature vector generation routine is expressed by the following equation ( 26). Then, by using the structure shown in the formula (22) to (26) or the like, a probability projection point u _s feature vectors X _s of the novel image data I _s from a class C _g appears p (u _s | C _g ) Can be derived according to the following equation (27). However, character 1 ... n subscript in the formula (27) indicates that from the first 1~n th sample data belonging to the class C _g, "Ψ" is the variance of the latent variables in the following formula (28) is there.

When the probabilistic linear discriminant analysis is used, the probability p (u _s | C _g ) that the projection point u _s of the feature vector X _s of the novel image data I _s appears from a certain class C _g is obtained according to the above equation (27). However, in the robot apparatus 20 of the embodiment, the probability linear discriminant analysis is performed so that the probability that the novel image data I _s (feature vector X _s ) appears from a certain class C _g can be derived with higher accuracy. An extension to multiplex the latent space is introduced. That is, in the robot apparatus 20 of the embodiment, the feature vector of the i-th region in the h-th layer of the j-th sample data belonging to the class C _g is X _j ^{g (h, i),} and belongs to the class C _g . The average vector of feature vectors X _j ^{g (h, i)} of the i-th region in the h-th layer of sample data is X ^{−g (h, i),} and ⁱ in the h-th layer of all sample data belonging to class C _g The average vector of the feature vectors in the ^ith region is μ _x ^{(h, i),} and the intra-class covariance matrix for the ith region in the h-th layer is Σ _w ^{(h, i)} shown in the following equation (29). When the out-of-class covariance matrix for the i-th region in the h-th layer is Σ _b ^{(h, i)} shown in the following equation (30), for each i-th region in the h-th layer ( 31), the feature vector of the i-th region in the h-th layer is obtained by solving the eigenvalue problem "Lambda ^(h projection matrix W ^{(h, i)} as a transformation for projecting the corresponding the latent space is derived in advance for each i-th region in the h layer (wherein ^{(31), i )} "Is a diagonal matrix obtained by arranging eigenvalues as discrimination criteria in order diagonally). Then, the robot apparatus 20, for each of a plurality of classes C _{1 to} C _G , features vectors X ^{(h) of} regions obtained by dividing each sample data by h × h in each layer from the first layer to the Hth layer. ^{, i)} the projection matrix W ^(h, a projection projected point obtained by u ^{(h, i)} the latent space by ^i), h × a novel image data I _s in layers from the first layer to the second H layer h divided for each region obtained by a feature vector X _s ^{(h, i)} the projection matrix W ^{(h, i)} projection points obtained by projecting the latent space by u _s ^{(h, i)} and the following formula ( 32) from the feature vector X ^{(h, i)} of the i-th region in the h-th layer of the sample data to the feature vector X _s ⁽ in the h-th layer of the novel image data I _s ^{) h, i)} is characterized Baie novel image data I _s i = 1 to the probability of occurrence of the sum from i = h ² and the first layer to the second H layer from the class C _g This is derived as the probability p (X _s | C _g ) that the Couttle X _s appears. However, the subscript (h, i) in the equation (32) indicates that it is derived from the i-th region in the h-th layer, the subscript s indicates that it is derived from novel data, and the subscript C _g is the class. It indicates that it belongs to C _g, subscript 1 ... n indicates that from the first 1~n th sample data belonging to the class C _g, "alpha ^h" is the h layer obtained experimentally in advance in the The weights to be given to “Z ^{(h, i) Cg} ” and “Θ ^{(h, i)} ” in the equation (32) are as shown in the following equations (33) and (34). 33) u ^{− (h} , ^{j) Cg} is the average of the projection points u ^(h , ^{j) Cg} of the feature vector X ^(h , ^{j) Cg} belonging to the class C _g , and in Equations (33) and (34) “Ψ ^{(h, i)} ” is the variance of the latent variable shown in the following equation (35), and Λ ^{(h, i)} in equation (35) is in the i-th region in the h-th layer. This is a diagonal matrix obtained by arranging the eigenvalues that are solutions of the eigenvalue problem in a diagonal order. 5 and 6 schematically show how the probability p (X _s | C _g ) is derived in the robot apparatus 20 of the embodiment. The formula (32) is the probability the feature vector X _s of the novel image data I _s appears from class C _g p | is indicative of log-likelihood (X _s C _g) (weighted log likelihood), by deforming as shown equation (32) into equation (36), it is possible to derive a probability that the feature vector X _s of the novel image data I _s appears from a certain class C _g. The class discrimination routine of FIG. 4 derives the probability p (X _s | C _g ) for each of the classes C _{1 to} C _G and discriminates the class C _g having the maximum probability (X _s | C _g ). To be executed.

Now, at the start of the class discrimination routine of FIG. 4, the discrimination processing unit 34 is obtained in each layer from the first layer to the Hth layer obtained by executing the above-described feature vector generation routine and stored in a predetermined storage area. Necessary for class determination of novel image data I _s such as feature vector X _s ^{(h, i) of} each region obtained by dividing h × h of novel image data I _s and learning information stored in learning information storage device 42 Correct data is input and stored in a predetermined storage area (memory) (step S300). The learning information input in step S300 is derived for each region obtained by dividing the sample data by the learning processing unit 33 in each layer from the first layer to the H-th layer and stored in the learning information storage device 42. Projecting a plurality of stored projection matrices W ^(h , ^j) and feature vectors X ^{(h, i)} of each region obtained by dividing each sample data in each layer from the first layer to the H layer Projection points (vectors) u ^{(h, i)} obtained by projecting into the latent space by the matrix W ^(h , ^j) , from the first layer to the H layer, which are obtained in advance and stored in the learning information storage device 42 Weight α ^{h and the} like. In an embodiment, Class C ₁ -C n pieces of sample data for each _G are predetermined as a sample for a class discrimination, projection matrix W ^{(h, j)} is the sample data feature vector X _j ^{g (h , i) and the} like, and are stored in the learning information storage device 42 in advance. Further, the projection point u ^{(h, i)} of each sample data is obtained by the learning processing unit 33 after the projection matrix W ^(h , ^j) is derived, and then the learning processing unit 33 converts the same conversion equation as the above equation (26). And stored in the learning information storage device 42.

After the data input process of step S300, the discrimination processing unit 34 calculates the projection point u _s ^{(h, i)} in the latent space of all feature vectors X _s ^{(h, i)} for the novel image data I _s by the above equation (26 ^). ) Is derived based on the input feature vector X _s ^{(h, i)} , the projection matrix W ^(h , ^j), etc., and stored in a predetermined storage area (memory) (step S310). Further, the discrimination processing unit 34 initializes a variable g for identifying the above-described class to a value 1 (step S320), and initializes a variable h indicating the number of the above-described layers to a value 1 (step S330). Further, a variable i indicating a region number in the h-th layer is initialized to a value 1 (step S340). Next, the discrimination processing unit 34 calculates the value of the term q in the above formula (36) using the information input in step S300 (step S350), and adds the value Q to sequentially add the value of the term q. = Q + q is derived and stored in a predetermined storage area (memory) (step S360). After the process of step S360, the determination processing unit 34 determines whether or not the variable i is the maximum value h ² (the total number of regions in the h-th layer) (step S370), and the variable i is less than the maximum value h ^2. If so, the variable i is incremented (step S380), and the processes of steps S350 and S360 are executed again.

If it is determined in step S370 that the variable i is the maximum value h ² , at that stage, the feature vector X ^{(h, i)} of each region in the h-th layer of each sample data is used to generate the new image data I _s . The sum of the probabilities that the feature vectors X _s ^{(h, i)} of each region in the h layer appear is derived. That is, when h = 1, when the affirmative determination is made in step S370, the value Q is obtained from each sample data itself (a feature vector X _j ^g ) in a certain class C _{g as} can be seen from FIG. The sum of the probabilities that the novel image data I _s itself (feature vector X _s ) appears. Also, if it is h = 2, when the affirmative determination is made in step S370, the value Q, as can be seen from FIG. 6, a region (feature vectors in the second layer of each sample data in a class C _g X ^{(2, i))} from the area (of the feature vector X ^{(2, i)} the area in the second layer of the novel image data I _s corresponding to) (feature vector X _s ^{(2, i))} appears Indicates the sum of probabilities. Accordingly, when determining that the variable i is the maximum value h ² in step S370, the discrimination processing unit 34 derives a value P = P + Q to derive the sum of the values Q from the first layer to the H layer. It is stored in a predetermined storage area (memory) (step S390), and it is further determined whether or not the variable h is the maximum value H (step S400). If the variable h is less than the maximum value H, the determination processing unit 34 increments the variable h (step S410) and executes the processes of steps S340 to S390 again.

When it is determined in step S400 that the variable h is the maximum value H, at that stage, a new value is obtained from the feature vector X ^{(h, i)} of the i-th region in the h-th layer of sample data for a certain class C _g. The sum from the first layer to the Hth layer of the probability that the feature vector X _s ^{(h, i)} of the i-th region in the h-th layer of the image data I _s appears is derived. Accordingly, when determining that the variable h is the maximum value H in step S370, the discrimination processing unit 34 sets the probability p (X _s | C _g ) that the feature vector X _s of the novel image data appears from the class C _g. It is set to exp (−P / 2) and stored in a predetermined storage area (memory) (step S420), and it is further determined whether or not the variable g is the maximum value G (step S430). If the variable g is less than the maximum value G, the determination processing unit 34 increments the variable g (step S440), and executes the processes of steps S330 to S420 again. When the variable g is determined to be the maximum value G in step S430, the derivation of the probability p (X _s | C _g ) for all the classes C _g is completed. When determining that the variable g is the maximum value G in step S430, the determination processing unit 34 obtains a class C _gmax that maximizes the obtained probability p (X _s | C _g ), and the novel image data I _s is obtained. In order to be able to identify that it belongs to the class C _gmax , for example, metadata such as a symbol is added to the novel image data I _s and its feature vector X _s (step S450), and this routine is terminated. According to the class determination routine described so far, it is possible to determine with high accuracy which of the plurality of classes C _{1 to} C _G the novel image data I _s belongs to.

As described above, the robot apparatus 20 of the embodiment, the feature vector generating routine of Fig. 3, novel image data I _s p extracted from the ^(j) number of d following local features as data indicating a real-world information M-th order correlation vectors from the first order to the M-th order between the elements constituting the average vector μ ^(j) of the vector V _k ^(j) and p ^(j) d-th order local feature vectors V _k ^(j) feature vector X _s indicating the novel image data I _s overall characteristics is obtained based on the elements that make up the. Here, the m-th order correlation vector can be obtained by a calculation process that is significantly lighter than, for example, clustering, and the correlation between an important feature point and its surrounding feature points, that is, the local feature It represents the distribution information well. As a result, in the robot apparatus 20 according to the embodiment, a large number of relatively high-order local feature vectors V _k ^(j) extracted from the novel image data I _s (or known image data) while greatly reducing the calculation cost. Based on this, it is possible to quickly generate a feature vector X _s as a feature quantity that accurately represents the entire feature (the novel image data I _s or the known image data) (high in feature expression).

In the robot apparatus 20 of the embodiment, when deriving the m-th order correlation vector, the d-order local feature vector V _k ^(j using the projection matrix U _dl on the dl-order principal component space that is lower than the d-order. ⁾ Dimension compression is performed. As a result, when the dimension d of the d-th order local feature vector V _k ^(j) is higher, it is possible to reduce the calculation cost associated with the derivation of the m-th order correlation vector, and further express the characteristics of the entire data by dimensional compression. In addition, unnecessary local features can be removed. Then, when performing such dimensional compression of the d-th order local feature vector V _k ^(j) , novel image data is obtained by _obtaining a projection matrix U _dl using N pieces of data I ^(j) in advance. When I ^{(j + 1)} appears, the feature vector X ^{(j + 1)} of the novel data I ^{(j + 1)} can be quickly acquired.

Furthermore, when determining which of the plurality of classes C _{1 to} C _G the novel image data I _s belongs to, the robot apparatus 20 according to the embodiment includes a plurality of sample data and a plurality of sample data in each layer from the first layer to the H layer. and elements that make up the average vector of a plurality of d next local feature vectors extracted each novel image data I _s from each of the areas obtained by h × h division, first among the plurality of higher-order local feature vectors The feature vectors X ^{(h, i)} and X _s ^{(h, i)} of the respective regions acquired based on the elements constituting the m-th order correlation vectors from the first order to the M-th order are used. Such feature vectors X ^{(h, i)} and X _s ^{(h, i)} can be obtained at a low calculation cost as described above, and can well represent the features of the target region. Then, the robot apparatus 20 of the embodiment, for each of a plurality of classes C ₁ -C _G, of the respective sample data at each layer from the first layer up to the H layer each region obtained by h × h divided feature vectors Projection point u ^{(h, i)} obtained by projecting X ^{(h, i)} onto the latent space by projection matrix W ^{(h, i)} and novel image data I _{s in} each layer from the first layer to the H layer To the projection point u _s ^{(h, i)} obtained by projecting the feature vector X _s ^{(h, i)} of each region obtained by dividing h × h into the latent space by the projection matrix W ^{(h, i)} Based on the feature vector X _s ^{(h, i)} of the i-th region in the h-th layer of the novel image data I _s from the feature vector X ^{(h, i)} of the i-th region in the h-th layer of the sample data. I = 1 to i = h ² and the total from the first layer to the H-th layer is the feature vector X of the novel image data I _s from the class C _g. Derived as the probability of appearance of _s .

In this way, feature vectors X ^{(h, i)} and X _s ^{(h, i)} , which can be acquired at a low calculation cost and have high feature expression ^, and the latent space are multiplexed for probabilistic linear discriminant analysis. By using a technique that introduces the extension of the function, the probability p (X _s | C _g ) that the feature vector X _s of the novel image data I _s appears from a certain class C _g can be derived with higher accuracy. Therefore, in the robot apparatus 20 according to the embodiment, it is determined from the probability p (X _s | C _g ) derived for each class C _g whether the novel image data I _s belongs to the plurality of classes C _{1 to} C _G. It becomes possible to discriminate with high accuracy. In the framework of probabilistic linear discriminant analysis, learning is completed by solving the generalized eigenvalue problem once, the calculation cost for the number of samples is linear, the memory usage is very small, and the number of classes increases. But basically the calculation cost does not change. Therefore, the class discrimination method according to the present invention, which introduces the extension of multiplexing the latent space to the stochastic linear discriminant analysis, can be applied to a large-scale problem. Reduction is possible.

As described above, the robot apparatus 20 of the embodiment, by storing the learning information necessary for learning information storage device 42, immediately feature vectors required from novel image data I _s obtained by the imaging unit 21 In addition to generating X _{s and the} like, it is possible to determine with high accuracy and promptness which of the plurality of classes C _{1 to} C _G the novel image data I _s belongs to. This makes it possible to cause the robot apparatus 20 to determine the acquired real-world information, that is, what is seen or heard, at high speed and with high accuracy, and makes the autonomous action of the robot apparatus 20 closer to human action. In addition, the intelligence level of the robot apparatus 20 can be further improved.

  Here, with respect to the feature vector generation method according to the present invention and the Bag-of-Keypoints method, which is an existing feature vector generation method, p number of d-order local feature vectors extracted from each of N pieces of image data. Based on this, the calculation cost required to generate the feature vector of each image data is evaluated by dividing into two processes of pre-processing and feature vector generation processing in each method. Here, in the feature vector generation method according to the present invention, the projection matrix derivation process necessary for dimensional compression of the d-th order local feature vector by principal component analysis corresponds to preprocessing, and the processing of steps S130 to S180 in FIG. This corresponds to vector generation processing. In the Bag-of-Keypoints method, the process of clustering d-order local feature vectors and deriving cluster representative vectors (visual words) corresponds to pre-processing, and assigns local features to the closest “visual words”. Corresponds to the feature vector generation process.

First, considering the order of calculation costs in the pre-processing of the feature vector generation method and the Bag-of-Keypoints method according to the present invention, the calculation cost order Oa in the pre-processing of the feature vector generation method according to the present invention is Oa∝p · If the number of representative vectors (visual words) is “V” while N · d ² , the calculation cost order Oa _bag in the pre-processing of the Bag-of-Keypoints method is Oa _bag ∝p · N · V · d. In general, the number V of “visual words” in the Bag-of-Keypoints method is larger than the dimension d of the local feature vector, and the value V increases as the problem scale increases. It can be said that the calculation cost in the preprocessing is considerably lower than that of the Bag-of-Keypoints method. The order of memory usage Om in the pre-processing of the feature vector generation method according to the present invention is OmＯd ² , whereas the order of memory usage Om _bag in the pre-processing of the Bag-of-Keypoints method is , Om _bag ∝p · N · d. In general, the number N of image data is larger than the dimension d of the local feature vector, and the value N increases as the scale of the problem increases. Therefore, the memory usage in the preprocessing of the feature vector generation method according to the present invention is It can be said that it is considerably less than that of -of-Keypoints method. Then, assuming that N = 800, p = 600, d = 128, and V = 1500, the inventors performed preprocessing in the feature vector generation method according to the present invention using an existing general-purpose personal computer, and the Bag-of-Keypoints method. In the Bag-of-Keypoints method, the pre-processing (clustering and derivation of visual words) may take approximately 18 hours, whereas the method according to the present invention Pre-processing was completed in about 90 seconds.

Further, considering the calculation cost order in the feature vector generation method of the present invention and the feature vector generation processing of the Bag-of-Keypoints method, the calculation cost order Of in the feature vector generation processing of the present invention is Of∝p · d ^On the other hand, the order Of _bag of the calculation cost in the feature vector generation process of the Bag-of-Keypoints method is Of _bag ∝p · V · d. As described above, the number V of “visual words” in the Bag-of-Keypoints method is larger than the dimension d of the local feature vector, and the value V increases as the problem size increases. It can be said that the calculation cost in the generation process is considerably lower than that of the Bag-of-Keypoints method. Then, assuming that p = 600, d = 128, and V = 1500, the inventors performed the feature vector generation processing of the present invention and the feature vector generation processing in the Bag-of-Keypoints method using an existing general-purpose personal computer. When executed, in the Bag-of-Keypoints method, it took about 860 msec to generate a feature vector for one image data, whereas the feature vector generation processing of the present invention performed a feature of one image data. The vector was generated in about 60 msec. From these examination results, it can be said that the feature vector generation method according to the present invention is extremely superior to the existing method in terms of calculation cost.

  Subsequently, the feature expression degree of the feature vector generated by the feature vector generation method according to the present invention is evaluated. In order to evaluate such feature expression, the present inventors have developed four types of local feature descriptors such as edge histogram, HSV color histogram, “Gray-SIFT” and “RGB-SIFT”, and a feature vector generation method according to the present invention. And a combination of the above-described four types of local feature descriptors and a technique (hereinafter referred to as “Mean”) that uses an average vector (0th-order correlation vector) of the four types of local feature descriptors and higher-order local feature vectors. The scene discrimination of the data set called “OT8” was executed using the combination, and the scene discrimination rate was compared between the feature vector generation method according to the present invention and “Mean” for each local feature descriptor. FIG. 7 shows a comparison result of scene discrimination rates. “OT8” has 8 classes of scenes (color images): “coast”, “forest”, “mountain”, “open country”, “highway”, “inside city”, “tall building”, “street” (See A. Oliva and A. Torallba. Modeling the shape of the scene: A holistic representation of the spatial). A 72-dimensional gradient direction histogram was used as the edge histogram, and local features were extracted from the gray image. As the color histogram, a standard 84 dimension (H: 36 dimension, S: 32 dimension, V: 16 dimension) in the HSV space was used. For these local feature descriptors, the local feature extraction window was fixed at 10 × 10 pixels and L = 5. Furthermore, for SIFT descriptors, the local feature extraction window was fixed to 16 × 16 pixels and L = 5, and dimension compression by principal component analysis was performed in consideration of higher dimensions than other descriptors. (Dl = 30). When the feature vector generation method according to the present invention is applied, the maximum order M of the correlation vector is set to M = 1.

  As can be seen from FIG. 7, using any of the above four types of local feature descriptors, the feature vector obtained by the feature vector generation method of the present invention is used, compared with the case where “Mean” is used. It can be seen that the scene discrimination performance is greatly improved. The evaluation results shown in FIG. 7 are obtained by using the m-th order correlation vector when generating the feature vector of the image data, and thus the correlation between the important feature points of the image and the surrounding feature points, that is, the distribution information of the local features is the feature. It shows that it is well reflected in the vector.

  Next, the effectiveness of the class discrimination method according to the present invention using the feature vector generated by the feature vector generation method according to the present invention is evaluated. In order to evaluate the effectiveness, the inventors of the present invention used a class discrimination method according to the present invention using the feature vector (H = 3) and a class discrimination method based only on probabilistic linear discriminant analysis using the feature vector. (H = 1: Reference) and a plurality of existing methods, scene discrimination of the above three types of data sets “OT8”, “LSP15”, and “Caltech-101” is executed. The scene discrimination rate was compared. As for the existing methods, “NO-SI”: one not including image position information (Spatial Information) and “SI”: one including image position information were appropriately prepared. Here, “LSP15” is the total of 8 class scenes in “OT8” and “bed room”, “kitchen”, “living room”, “store”, “suburb”, “industrial”, and “office” In Proc. IEEE Conf. Computer Vision A dataset containing 15 classes of scenes (monochrome images) (S. Lazebnik, C. Schmid, and J. Ponce. and Pattern Recognition, 2006.). “Caltech-101” is a data set including 102 classes of 101 object classes and background classes (L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models). from few training examples: an incremental bayesian approach tested on 101 object categories. In Proc. See IEEE CVPR Workshop on Generative-Model Based Vision, 2004.). When evaluating the effectiveness, “RGB-SIFT” for “OT8” and “Caltech-101” and “Gray-SIFT” for “LSP15” are used as local feature descriptors. The local feature vectors extracted from both the 36 × 36 region were enumerated as final high-order local feature vectors. In addition, as the existing methods, the methods [A] to [G] described in the following documents are used. The method [A] is to estimate the “part-based” “generative model” of the image by “CRF (Conditional Random Field) and perform segmentation and identification of the image at the same time. Compared to the bag-of-keypoints method, Method B and Method C perform local feature extraction using the SIFT descriptor and the bag-of-keypoints method, as well as “SVM (Support Vector). Class) ”etc. FIG. 8 shows a comparison result of scene discrimination rates.

Method [A]: Y. Wang and S. Gong. Conditional random field for natural scene categorization. In Proc. British Machine Vision Conference, 2007.
Method [B]: A. Bosch, A. Zisserman, and X. Mu ・ noz. Scene classification using a hybrid generative / discriminative approach. IEEE Trans. Pattern Analysis and Machine Intelligence, pages 712 ・ 727, 2008.
Method [C]: S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
Method [D]: O. Boiman, E. Shechtman, and M. Irani. In defense of nearest-neighbor based image classification. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
Method [E]: H. Zhang, AC Berg, M. Maire, and J. Malik. SVM-KNN: Discriminative nearest neighbor classification for visual category recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, volume 2, pages 2126 ・ 2136, 2006.
Method [F]: K. Grauman and T. Darrell. The pyramid match kernel: Efficient learning with sets of features. Journal of Machine Learning Research, 8: 725 ・ 760, 2007.
Method [G]: N. Herv´e and N. Boujemaa. Image annotation: which approach for realistic databases? In Proc. ACM International Conference on Image and Video Retrieval, 2007

  As can be seen from the comparison results in FIG. 8, with regard to “OT8” and “LSP15”, the class discrimination method (H = 3 in FIG. 8) according to the present invention in which the extension of multiplexing of latent space is introduced into the stochastic linear discriminant analysis is The scene discrimination rate exceeding the score of the existing method is recorded. Also for “Caltech-101”, the class discrimination method according to the present invention (H = 3 in FIG. 8) records a scene discrimination rate comparable to at least the score of the existing method. Further, Method C with H = 1 (discrimination rate 41.2%) corresponds to the most standard Bag-of-keypoints method. Regarding “Caltech-101”, the class discrimination method according to the present invention is as follows. The scene discrimination rate greatly exceeding the score of Method C with H = 1 is recorded, and this point confirms that the class discrimination method according to the present invention is extremely excellent in practical use. Although the methods [D] and [F] each have a high score for “Caltech-101”, both methods require matching of local features, and enormous calculation costs are required. In addition, it requires memory usage and cannot be put to practical use. Further, as can be seen from FIG. 8, the class discrimination method (H = 1: reference) based only on the probabilistic linear discriminant analysis using the feature vector generated by the feature vector generation method according to the present invention is based on the existing method “SI”. Although it does not reach the score, the scene discrimination rate exceeding the score of the existing method “NO SI” is recorded. This is because both the feature vector generated by the feature vector generation method according to the present invention is excellent in the feature expression level and the high effectiveness of the latent space multiplexing in the class discrimination. Don't be. In any case, from the comparison result of FIG. 8, the class discrimination method according to the present invention using the feature vector generated by the feature vector generation method according to the present invention enables a high-speed calculation process while being very simple. And it will be understood that it gives results that are comparable or better than existing methods.

  Note that the scope of application of the present invention is not limited to the robot apparatus 20 as described above, and the present invention is capable of discriminating an object existing in the front of a vehicle, etc. It may be applied to an in-vehicle image recognition device. The present invention can also be applied to an image data processing system as illustrated in FIG. The image data processing system 200 shown in the figure manages a data storage device 210 that stores a large number of image data and word group data in a database, a database on the data storage device 210, and annotations for new image data. The management computer 300 enables database retrieval (retrieval) and the like. The management computer 300 includes a CPU, ROM, RAM, system bus, various interfaces, storage devices and the like (not shown). The management computer 300 can be accessed from the terminal 500 via a network such as the Internet. It can be done. Further, as shown in FIG. 9, the management computer 300 includes hardware such as a CPU, a ROM, a RAM, various interfaces, and a storage device, and a preinstalled feature quantity generation program and class determination program according to the present invention. The search robot 310, the data reception unit 320, the image feature amount extraction unit 330, the word feature amount extraction unit 340, the learning processing unit 350, the annotation processing unit 360, and the search query reception unit 370 are cooperated with one or both of various software. The retrieval processing unit 380, the result output unit 390, and the like are constructed as functional blocks. Further, a feature amount storage device 400 and a learning information storage device 410 are connected to the management computer 300.

  The search robot 310 of the management computer 300 collects data including images that are not stored in the database of the data storage device 210 via a network or the like, and updates the database. The data receiving unit 320 inputs at least one word (metadata indicating input of image data by a human hand using various input means or metadata appearing in an image of the image data in association with the image data). The input of word group data indicating (symbol) is received, and the received data is stored in the data storage device 210. The image feature quantity extraction unit 330 extracts an image feature quantity indicating the feature of the data from the image data, and stores it in the feature quantity storage device 400. That is, the image feature quantity extraction unit 330 calculates the features of the entire image data based on the elements constituting the average vector of the above-mentioned higher-order local feature vectors and the elements constituting the m-th order correlation vector between the higher-order local feature vectors. The indicated feature vector is acquired. The word feature amount extraction unit 340 extracts a word feature amount indicating the feature of the data from the word group data and stores it in the feature amount storage device 400. The learning processing unit 350 learns the relationship between the image data and the word group data by using a plurality of combinations of the image feature amount and the word feature amount, and assigns a word group as metadata to the unannotated image data. The learning information necessary for the search (retrieval) of the unannotated image data based on is acquired, and the acquired learning information is stored in the learning information storage device 410. The learning processing unit 350 generates learning information necessary for class determination of the feature vector and novel image data. The annotation processing unit 360 executes annotation for unannotated image data and class determination of novel image data. The search query receiving unit 370 receives input of at least one word (symbol) as a search query from the terminal 500 or the like. The retrieval processing unit 380 executes a retrieval process (retrieval) of image data including unannotated image data based on the search query accepted by the search query acceptance unit 370. The result output unit 390 outputs the processing result of the retrieval processing unit 380 to the terminal 500 or the like. By applying the present invention to such an image data processing system 200, it is possible to reduce the calculation cost required for generating image feature quantities (feature vectors) and improve the class discrimination performance of novel image data. As a result, the performance of the entire system can be improved.

  The embodiments of the present invention have been described above using the embodiments. However, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the present invention. Needless to say.

  INDUSTRIAL APPLICABILITY The present invention is useful in the information processing field in which a feature vector of one piece of data indicating real world information is handled and whether or not new data indicating real world information belongs to a plurality of classes.

  20 robot apparatus, 21 imaging unit, 22 sound collecting unit, 23 actuator, 30 control computer, 31 input / output processing unit, 32 feature amount processing unit, 33 learning processing unit, 34 discrimination processing unit, 35 retrieval processing unit, 36 main control Unit, 40 data storage device, 41 feature amount storage device, 42 learning information storage device, 200 image data processing system, 210 data storage device, 300 management computer, 310 search robot, 320 data reception unit, 330 image feature amount extraction unit, 340 word feature amount extraction unit, 350 learning processing unit, 360 annotation processing unit, 370 search query reception unit, 380 retrieval processing unit, 390 result output unit, 400 feature amount storage device, 410 learning information storage device.

Claims (10)

A feature quantity generation device that generates a feature vector that represents a feature of the entire data using a plurality of higher-order local feature vectors extracted from one data that represents real world information,
Average acquisition means for acquiring an average vector of the plurality of higher-order local feature vectors;
First-order to M-th order m-order correlation vectors between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to value 1, and “m” is an integer between value 1 and value M) Correlation acquisition means for acquiring
Feature vector obtaining means for obtaining the feature vector based on elements constituting the average vector obtained by the average obtaining means and elements constituting the m-th order correlation vector obtained by the correlation obtaining means;
A feature amount generating apparatus. In the feature-value production | generation apparatus of Claim 1,
When p d-order local feature vectors extracted from one data I indicating real world information are V _k = (v ₁ ,..., V _d ) (where “p” and “d” are respectively The value is an integer greater than or equal to 2, and “k” is an integer from the value 1 to the value p), and the average acquisition means calculates the average vector μ of the p d-order local feature vectors V _k as follows: The correlation acquisition unit acquires the autocorrelation matrix R of the p d-order local feature vectors V _k according to the following equation (2) and enumerates elements of the upper triangular matrix of the autocorrelation matrix R Then, the primary correlation vector upper (R) is acquired, and the feature vector acquisition means, when the feature vector is X, the elements of the average vector μ and the primary correlation vector upper (R) according to the following equation (3): ) To list the feature vector. Feature value generating unit that acquires Le X.
In the feature-value production | generation apparatus of Claim 1,
The correlation acquisition unit is a feature quantity generation device that acquires the m-th order correlation vector with dimensional compression of the higher-order local feature vector by principal component analysis. In the feature-value production | generation apparatus of Claim 3,
Assume that there are N pieces of data I ^(j) indicating real world information (where “N” is an integer greater than or equal to value 2 and “j” is an integer between value 1 and value N), The p ^(j) d-th order local feature vectors extracted from the data I ^(j) are V _k ^(j) = (v ₁ ,..., V _d ) (where “p ^(j) ” and “d” Are integers of value 2 or more, and “k” is an integer from value 1 to value p), and the average of the p ^(j) d-order local feature vectors V _k acquired by the average acquisition means The vector is μ ^(j) shown in the following equation (4), the autocorrelation matrix of the p ^(j) d-order local feature vectors V _k ^(j) is R ^(j) shown in the following equation (5), When the autocorrelation matrix of the entire d-th order local feature vector extracted from the N pieces of data is R _all shown in the following equation (6) and the novel data is I ^{(j + 1)} , the correlation obtaining unit , A projection matrix U _dl to dl next principal component space than d following obtained by solving the eigenvalue problem of 7) is low order, extracted from the novel data ^{I (j + 1) p (} j + 1) Obtaining a diagonal matrix U _dl ^T R ^{(j + 1)} U _dl based on the autocorrelation matrix R ^{(j + 1) of the} number of d-order local feature vectors V _k ^{(j + 1)} , and the diagonal matrix U _dl ^T R ^{(j + 1)} U _dl upper triangular matrix elements are enumerated to obtain a primary correlation vector upper (U _dl ^T R ^{(j + 1)} U _dl ), and the feature vector acquisition means includes: The elements constituting the average vector μ ^{(j + 1)} of the p ^{(j + 1)} d-order local feature vectors V _k ^{(j + 1)} and the first-order correlation vector upper (U _dl ^T A feature quantity generation device that obtains a feature vector X ^{(j + 1)} of novel data I ^{(j + 1)} by enumerating elements constituting R ^{(j + 1)} U _dl ).
A feature amount generation method for generating a feature vector indicating a feature of the entire data using a plurality of higher-order local feature vectors extracted from one data indicating real world information,
The average vector of the plurality of higher-order local feature vectors and the m-th order correlation vector from the first order to the M-th order between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to 1) , “M” is an integer from value 1 to value M),
A feature quantity generation method for acquiring the feature vector based on elements constituting the acquired average vector and elements constituting the acquired m-th order correlation vector. A feature amount generation program that causes a computer to function as a device that generates a feature vector indicating the characteristics of the entire one data using a plurality of higher-order local feature vectors extracted from one data indicating real world information,
An average acquisition module for acquiring an average vector of the plurality of higher-order local feature vectors;
First-order to M-th order m-order correlation vectors between the plurality of higher-order local feature vectors (where “M” is an integer greater than or equal to value 1, and “m” is an integer between value 1 and value M) A correlation acquisition module for acquiring
The feature vector is acquired based on an element constituting an average vector acquired by the average acquisition module and an element constituting an m-th order correlation vector of the plurality of higher-order local feature vectors acquired by the correlation acquisition module. A feature vector acquisition module,
A feature generation program comprising: A class discriminating apparatus for discriminating to which one of a plurality of classes each corresponding to at least one known data each of novel data indicating real world information belongs,
Each of the novel data and the known data in the h-th layer is h × h (where “h” is an integer from value 1 to value H, and “H” is an integer greater than or equal to value 2). The feature vector derived for each region based on the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer is set as a latent space. Conversion storage means for storing conversion for projecting;
Assuming that the novel data is divided into h × h regions in the h-th layer, a plurality of higher-order local features from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer Local feature extraction means for extracting vectors;
An average vector of a plurality of higher-order local feature vectors extracted by the local feature extraction means from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and the plurality of higher-order M-th order correlation vectors from the first order to the M-th order between local feature vectors (where “M” is an integer greater than or equal to value 1 and “m” is an integer between value 1 and value M). And a feature vector acquisition means for acquiring a feature vector of each of the regions based on an element constituting the average vector and an element constituting the m-th order correlation vector,
For each class, it is obtained by projecting the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer into the latent space by the transformation corresponding to the region. A projection point, and a projection point obtained by projecting the feature vector of each region obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the region; Based on the feature vector of the i-th area (where “i” is an integer from 1 to h ² ) in the h-th layer of the known data, the i-th in the h-th layer of the novel data Probability derivation for deriving the sum of the probability from the first layer to the H-th layer from i = 1 to i = h ² and the probability that the feature vector of the new data appears from the class Means,
Class setting means for setting a class having a maximum probability derived by the probability deriving means as a class to which the novel data belongs;
A class discrimination device comprising: In the class discrimination device according to claim 7,
In the transformation for the i-th region in the h-th layer, the number of classes is G (where “G” is an integer equal to or greater than 2), and the class is C _g (where “g” is The number of sample data that is known data extracted as a sample from class C _g is n (where “n” is an integer greater than or equal to value 1), A feature vector of the i-th region in the h-th layer of the j-th sample data (where “j” is an integer from 1 to n) belonging to the class C _g is represented by X _j ^{g (h, i )} , The average vector of the feature vectors X _j ^{g (h, i)} of the i-th region in the h-th layer of the sample data belonging to the class C _g is X ^{−g (h, i),} and all the samples belonging to the class C _g an average vector of feature vectors of the i-th region in the h layer of the sample data mu _x ^{(h, i)} And, i-th equation intraclass covariance matrix for region (9) to indicate sigma _w ^{(h, i)} in the h layer and then, a class outside covariance matrix for the i-th region in the h layer When Σ _b ^{(h, i)} shown in the following equation (10), it is a projection matrix W ^{(h, i)} obtained by solving the eigenvalue problem of the following equation (11) (however, “Λ ^{(h, i)} ” is a diagonal matrix obtained by arranging eigenvalues as discrimination criteria in order diagonally)
When the feature vector is X, the projection matrix is W, and the projection point of the feature vector X is u, it is obtained by dividing each of the sample data in each layer from the first layer to the H layer. Projection points of feature vectors of each region and projection points of feature vectors of each region obtained by dividing the novel data in each layer from the first layer to the H layer are derived according to the following equation (12):
When the feature vector of the novel data is X _s , the probability p (X _s | C _g ) that the feature vector X _s appears from the class C _g is a class discrimination derived based on the following equation (13): apparatus. However, the subscript (h, i) in equation (13) indicates that it is derived from the i-th region in the h-th layer, the subscript s indicates that it is derived from novel data, and the subscript C _g is the class. It indicates that it belongs to C _g, subscript 1 ... n indicates that from the first 1~n th sample data belonging to the class C _g, "alpha ^h" is the weight to be given to the h layer “Z ^{(h, i) Cg} ” and “Θ ^{(h, i)} ” in the equation (13) are as shown in the following equations (14) and (15), and u ^{− ( h} , ^{j) Cg} is the average of the projection points u ^(h , ^{j) Cg} of the feature vector X ^(h , ^{j) Cg} belonging to the class C _g , and “Ψ ^{(h, i} ) in the equations (14) and (15) ⁾ "is the variance of the latent variables in the following equation (16), lambda in the equation (16) ^{(h, i)} is the solution of the eigenvalue problem in the i-th region in the h layer It is a diagonal matrix obtained by arranging eigenvalues diagonally in order.
A class discrimination method for discriminating to which one of a plurality of classes corresponding to at least one known data each new data indicating real world information belongs,
Each of the novel data and the known data in the h-th layer is h × h (where “h” is an integer from value 1 to value H, and “H” is an integer greater than or equal to value 2). Is converted into a potential space based on the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer. Derived for each region,
Assuming that the novel data is divided into h × h regions in the h-th layer, a plurality of higher-order local features from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer Extract the vector,
An average vector of a plurality of higher-order local feature vectors extracted from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and a first vector between the plurality of higher-order local feature vectors. M-order correlation vectors from the first order to the M-th order (where “M” is an integer greater than or equal to value 1, and “m” is an integer from value 1 to value M) and the average Obtaining a feature vector of each of the regions based on an element constituting the vector and an element constituting the m-th order correlation vector;
For each class, it is obtained by projecting the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer into the latent space by the transformation corresponding to the region. A projection point, and a projection point obtained by projecting the feature vector of each region obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the region; Based on the feature vector of the i-th area (where “i” is an integer from 1 to h ² ) in the h-th layer of the known data, the i-th in the h-th layer of the novel data th derives a sum of i = 1 of the probability that the feature vector appears in the area from the i = h ² and the first layer to the second H layer as the probability that the feature vector of the novel data from the class appears,
A class discrimination method for setting a class having the maximum derived probability as a class to which the novel data belongs. A class determination program that causes a computer to function as a class determination device that determines which of a plurality of classes corresponding to at least one known data each of novel data indicating real world information,
In the h-th layer, the novel data is divided into h × h areas (where “h” is an integer from value 1 to value H and “H” is an integer greater than or equal to value 2). A local feature extraction module that extracts a plurality of higher-order local feature vectors from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer,
An average vector of a plurality of higher-order local feature vectors extracted by the local feature extraction module from each of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer, and the plurality of higher-order M-th order correlation vectors from the first order to the M-th order between local feature vectors (where “M” is an integer greater than or equal to value 1 and “m” is an integer between value 1 and value M). A feature vector obtaining module that obtains a feature vector of each of the regions based on an element constituting the average vector and an element constituting the m-th order correlation vector;
For each class, the feature vector of each region obtained by dividing each of the known data in each layer from the first layer to the H-th layer is projected onto the latent space by conversion corresponding to the predetermined region. Projection points obtained by projecting the obtained projection points and the feature vectors of the regions obtained by dividing the novel data in each layer from the first layer to the H-th layer onto the latent space by the transformation corresponding to the regions. Based on the feature vector of the i-th region in the h-th layer of the known data (where “i” is an integer from the value 1 to the value h ² ) in the h-th layer of the novel data. The probability that i = 1 to i = h ^{2 of} the probability that the feature vector of the i-th region appears and the sum from the first layer to the H-th layer is derived as the probability that the feature vector of the novel data appears from the class. A rate derivation module;
A class setting module that sets a class having a maximum probability derived by the probability derivation module as a class to which the novel data belongs;
Class discrimination program with