JP5207870B2 - Dimension reduction method, pattern recognition dictionary generation device, and pattern recognition device - Google Patents

Description

  The present invention relates to a feature space dimension reduction method, a pattern recognition dictionary generation device, and a pattern recognition device.

  An object of the pattern recognition apparatus is to identify a category to which an input pattern belongs using information in a pattern recognition dictionary. The pattern recognition dictionary is created in advance by the pattern recognition dictionary generation device using the dictionary generation pattern group. The pattern recognition dictionary holds information for calculating the similarity between an input pattern and each category. Specifically, a vector space in which an identification function is defined and parameters of the identification function are defined. It holds information for conversion to the above vector value.

  The pattern recognition apparatus holds a pattern recognition dictionary, and performs input pattern identification processing using the dictionary. The pattern recognition apparatus calculates the similarity between the input pattern and each category using the pattern recognition dictionary, and identifies the category to which the pattern belongs based on the calculation result. See Non-Patent Document 1 for such a pattern recognition device.

Hereinafter, a general dictionary generation procedure by the pattern recognition dictionary generation device will be described.
The pattern recognition dictionary generation device uses the dictionary generation pattern group to create a pattern recognition dictionary. First, the pattern recognition dictionary generation device converts a dictionary generation pattern group into a dictionary generation feature pattern group that is a vector group in a vector space by feature extraction. A vector extracted by feature extraction is called a feature vector, and a vector space to which the feature vector belongs is called a feature space. For example, pixel features, contour features, gradient features, and the like are widely used as feature extraction methods (Non-Patent Document 2).

  Next, the pattern recognition dictionary generation apparatus often performs feature space dimension reduction (feature selection) for the purpose of high accuracy, high speed, and memory saving in identification processing. In dimension reduction, a subspace of the feature space that approximates the distribution of categories in the feature space is selected. The subspace information is stored as a projection matrix from the feature space to the subspace. The subspace is generally different for each category, but a common subspace may be selected for all categories. Therefore, the dimension of the subspace may be different for each category. Hereinafter, the partial space generated by the dimension reduction is referred to as a feature space, and a vector on the partial space is referred to as a feature vector.

  Next, the pattern recognition dictionary generation device projects the dictionary generation feature pattern group onto the learning feature pattern group on the feature space.

  Next, the pattern recognition dictionary generation device generates, by learning, an identification function for identifying a category on the feature space using the learning feature pattern group. The discrimination function exists for each category, and the value at the point indicated by the feature vector on the feature space is the similarity between the category corresponding to the discrimination function and the feature vector. The discriminant function for each category is created by learning using a learning feature pattern group so that the loss function representing the loss due to misclassification becomes small. There are various methods for learning the identification function. For example, Non-Patent Documents 1 to 4 are detailed.

  Next, a procedure in which the pattern recognition apparatus identifies an input pattern using a pattern recognition dictionary will be described. The pattern recognition device calculates the similarity of each category of the input pattern, and identifies the category to which the pattern belongs based on the result.

  The pattern recognition apparatus first converts an input pattern into a vector value by feature extraction, as in the pattern recognition dictionary generation apparatus. Then, using the projection matrix stored in the pattern recognition dictionary, it is converted into a feature vector in a lower dimensional feature space. Next, using the identification function parameters stored in the pattern recognition dictionary, the value of the identification function at the point represented by the feature vector is calculated, and the similarity to each category of the input pattern is obtained. A recognition result is output based on the calculation result.

  Next, dimension reduction (or feature selection) in creating a pattern recognition dictionary will be described. The feature space obtained by the dimension reduction is preferably selected as a partial space of the original feature space that is advantageous for identification by sufficiently approximating the distribution of patterns belonging to each category and separating the categories. Statistical methods such as principal component analysis and linear discriminant are widely used as the dimension reduction method. See, for example, Non-Patent Document 2 for dimension reduction techniques such as principal component analysis and linear discriminant method.

  In dimension reduction by the principal component analysis method, the partial space is selected based on the maximum variance criterion, so that the pattern distribution to be analyzed can be approximated well. Specifically, the partial space is obtained by solving the eigenvalue problem of the covariance matrix of all or part of the learning pattern in the feature space. The feature space after the dimension reduction is obtained as a p-dimensional subspace based on eigenvectors corresponding to the upper p eigenvalues of the covariance matrix. When selecting a partial space common to all categories, all learning patterns are analyzed. When a different partial space is selected for each category, only learning patterns belonging to the category may be selected for each category and may be the analysis target.

  The linear discriminant method is a dimension reduction method in consideration of discrimination from other categories. This is a method of selecting a subspace that maximizes the intra-category variance / inter-category variance ratio.

Pattern recognition, Kenichiro Ishii, Nobuyoshi Ueda, Eisaku Maeda, Hiroshi Murase, Ohmsha, 1998. Character Recognition Systems: A Guide for Students and Practitioners, Mohammed Cheriet, Nawwaf Kharma, Cheng-lin Liu and Ching Suen, Wiley-Interscience, 2007. Pattern Classification: A Unified View of Statistical and Neural Approaches, J. Schurmann, Wiley-Interscience, 1996. Class-specific feature polynomial classifier for pattern classification and its application to handwritten numeral recognition, Cheng-Lin Liu and Hiroshi Sako, Pattern Recogn., 2006.

  Statistical methods such as principal component analysis and linear discriminant analysis are widely used as conventional dimension reduction methods. However, there are the following problems.

  The principal component analysis method is a method of selecting a subspace based on a maximum variance criterion. Therefore, the selected subspace is a subspace that well preserves information on the distribution of the analysis target. However, in the principal component analysis method, the subspace is selected using only the statistical distribution to be analyzed. Therefore, the selected partial space is not necessarily advantageous for identification. Therefore, it is not always an effective method when it is necessary to reduce dimensions for identification with other categories.

  The linear discriminant analysis method can select only a maximum of c-1 discriminant axes in the case of c category classification problems. Therefore, there is a problem that it is not possible to flexibly select a discrimination axis effective for identification.

  According to the dimension reduction method of the present invention, a feature pattern group input step for inputting a dictionary generation feature pattern group on the feature space and a coefficient value of a quadratic function on the feature space corresponding to each of the c categories are set. Initial coefficient setting step, and the coefficient of the quadratic function is reduced by a gradient descent method or stochastic gradient descent so that the value of the loss function is reduced by identifying the feature pattern group for dictionary generation using the quadratic function as an identification function. A coefficient correction step for correcting by a method; for each of the c categories, an eigenvector of a symmetric matrix of a quadratic form of a quadratic term of the quadratic function and a coefficient vector of a primary term of the quadratic function A basis vector deriving step for selecting one or more vectors from among them, and for each of the c categories, a feature space based on the vector selected in the basis vector deriving step A projection matrix derivation step for generating a projection matrix to a dictionary subspace which is a subspace, and the projection matrix to c dictionary subspaces corresponding to each of c categories, or each of c categories A projection matrix output step of outputting the corresponding projection matrix and the coefficient of the quadratic function.

  The dimension reduction method of the present invention may include a feature pattern group input step in which a feature space corresponding to each of the c categories and a feature pattern group for dictionary generation are input. In addition, a feature pattern group input step may be provided in which one feature space common to c categories and one dictionary generation feature pattern group are input. The feature space may be a vector space, and the feature pattern group for dictionary generation may be a set of labeled vector values indicating the category to which the vector space belongs. For the case where the feature space corresponding to each of the c categories and the feature pattern group for dictionary generation are input, see Non-Patent Document 4, for example.

  In the dimension reduction method of the present invention, in the coefficient correction step, the square error function when each of the dictionary generating feature pattern groups is identified using a quadratic function is used as a loss function, and the value of the loss function is reduced. The gradient descent method or the stochastic gradient descent method may be used for correction.

  In the projection matrix deriving step, the eigenvector corresponding to the top n largest values of the eigenvalues of the coefficient vector and the symmetric matrix or the top n number of absolute values of the eigenvalues of the coefficient vector and the symmetric matrix are larger than the threshold value h. A projection matrix onto a partial space of the feature space generated by the eigenvector may be generated. Alternatively, a subspace of the feature space generated by the eigenvector corresponding to the top n large eigenvalues of the symmetric matrix or the eigenvector corresponding to the top n eigenvalues of the symmetric matrix larger than the threshold value h A projection matrix may be generated.

  The pattern recognition dictionary generation device of the present invention extracts an input unit for inputting a dictionary generation pattern group, and extracts a dictionary generation feature pattern group on a feature space corresponding to each of c categories from the dictionary generation pattern group A feature extraction unit, a feature pattern group input process for inputting a feature pattern group for dictionary generation on the feature space, and an initial coefficient for setting a value of a coefficient of a quadratic function on the feature space corresponding to each of the c categories The coefficient of the quadratic function is corrected by the gradient descent method or the stochastic gradient descent method so that the value of the loss function is reduced by the setting process and the feature pattern group for dictionary generation using the quadratic function as the discriminant function. Coefficient correction processing to be performed, for each of the c categories, one of a eigenvector of a symmetric matrix of a quadratic form of a quadratic term of the quadratic function and a coefficient vector of a primary term of the quadratic function Basis vector derivation process for selecting the upper vector, and projection for generating a projection matrix for each of the c categories to a dictionary subspace that is a subspace of the feature space based on the vector selected in the basis vector derivation process The matrix derivation process, the projection matrix to c dictionary subspaces corresponding to each of c categories, or the projection matrix and the coefficients of the quadratic function corresponding to each of c categories are output. A dictionary subspace generation unit that generates the projection matrix to the dictionary subspace by dimension reduction processing of the feature space including projection matrix output processing, and learning by projecting a dictionary generation feature pattern group to the dictionary subspace by the projection matrix A learning feature pattern group generation unit that generates a feature pattern group, and an identification function that generates an identification function that identifies c categories using the learning feature pattern group. It has a function generator, and an output unit for outputting the parameters of the discriminant function the projection matrix.

  The discriminant function generating unit generates, for each of the c categories, a restricted quadratic function in which the quadratic function is restricted to a dictionary subspace, and each of the learning feature pattern groups is assigned to the restricted quadratic function as the discriminant function. Using the square error function at the time of category determination as a loss function, and using the gradient descent method or the probabilistic gradient descent method to correct the coefficient of the restricted quadratic function as the discriminant function so that the square error value becomes small It may be generated. Further, for each of the c categories, a quadratic function limited to the dictionary subspace may be generated as an identification function.

  The pattern recognition apparatus of the present invention sets a value of a coefficient of a quadratic function on a feature space corresponding to each of c categories, a feature pattern group input process for inputting a dictionary generating feature pattern group on the feature space. Initial coefficient setting process, the coefficient of the quadratic function is gradient descent or stochastic gradient descent so that the value of the loss function by identifying the characteristic pattern group for dictionary generation using the quadratic function as an identification function becomes small For each of the c categories, the coefficient correction process to be corrected by the eigenvector of the symmetric matrix of the quadratic form of the quadratic term of the quadratic function and the coefficient vector of the primary term of the quadratic function Basis vector derivation process for selecting one or more vectors, and for each of the c categories, a dictionary part that is a subspace of the feature space based on the vector selected in the basis vector derivation process Projection matrix derivation processing for generating a projection matrix to space, projection matrix to c dictionary subspaces corresponding to each of c categories, or projection matrix corresponding to each of c categories and the above-mentioned quadratic A recognition dictionary storage unit for storing parameters of the discriminant function on the dictionary subspace generated by the dimension reduction process of the feature space including the projection matrix output process for outputting the coefficient of the function and the projection matrix, and a recognition target An input unit for inputting the input pattern, a feature extraction unit for extracting a vector value on the feature space from the input pattern, and a projection vector obtained by projecting the input pattern onto the dictionary subspace using the projection matrix for each of the c categories For each of the c categories, a value at a point represented by the projection vector of the discriminant function is calculated, and the input pattern belongs to With that identification section that identifies a category, and an output unit that outputs the identification result of the identifying portion.

  The feature space dimension reduction method according to the present invention can select a partial space of a feature space that is advantageous for identification for the purpose of high accuracy, high speed, and memory saving in a pattern recognition apparatus. As a result, a pattern recognition dictionary with a small capacity can be created in the pattern recognition apparatus. In the identification processing using the pattern recognition dictionary, a high recognition rate, high speed, and memory saving can be realized.

Embodiments of the present invention will be described below.
[Example 1]
FIG. 1 is a block diagram showing an example of a pattern recognition dictionary generating apparatus that generates a pattern recognition dictionary using the dimension reduction method of the present invention. The pattern recognition dictionary generation device includes an input device 11, an arithmetic device 12, a pattern recognition recognition dictionary 13, a display device 15, and a pattern database (DB) 16. The arithmetic unit 12 includes a pattern recognition dictionary creating means for generating a pattern recognition dictionary using the dimension reduction method according to the present invention.

  The input device 11 is a device such as a keyboard and mouse for inputting commands and a scanner for inputting images. The arithmetic unit 12 includes a CPU, a memory, a storage device, and the like, reads a dictionary generation pattern group such as an input image and sound, generates a pattern recognition dictionary, and stores the pattern recognition dictionary in the recognition dictionary 13. Specifically, the arithmetic unit 12 includes a feature extraction unit 17, a dictionary subspace generation unit 18, a learning feature pattern group generation unit 19, and a discrimination function generation unit 20, and the functions of these units are executed by a program. The recognition dictionary 13 is a dictionary database that stores a pattern recognition dictionary created by the arithmetic device 12. The display device 15 is a device such as a display that appropriately displays the processing content of the arithmetic device 12. The display device 15 may not be provided. The pattern DB 16 stores the pattern input by the input device 11. The pattern DB 16 stores a dictionary generation pattern group used by the arithmetic unit 12 to create a pattern recognition dictionary.

  FIG. 2 is a flowchart showing an outline of the pattern recognition dictionary creating means executed by the arithmetic unit 12. Note that. The feature of the present invention is the dictionary subspace generation step 23. In the dictionary subspace generation step 23, the dimension of the feature space is reduced by the dimension reduction method of the present invention. A detailed processing flow in the dictionary subspace generation step 23 is shown in FIG.

  In the input step 21, a dictionary generation pattern group is input by a user or a program executed by the arithmetic device 12. The dictionary generation pattern group is a set of a set of a pattern indicating a pattern such as an image or sound and a category to which the pattern belongs, and is prepared in advance for generating a dictionary. Hereinafter, the number of categories is c (c is a natural number). Here, the category is, for example, a number from 0 to 9 in the case of number recognition. At this time, c = 10.

  In the feature extraction step 22, the input pattern is converted into a vector value. The feature extraction step 22 is executed by the feature extraction unit 17. The pattern recognition dictionary generation device converts each of the dictionary generation pattern groups into a vector value on the vector space by the process of the feature extraction step 22. As a result, the dictionary generation pattern group is converted into a set of vector values on the vector space and their associated labels. This vector space is called a feature space, a vector value is called a feature vector, and a set of feature vectors generated from a dictionary generation pattern group and its associated labels is called a dictionary generation feature pattern group. The converted dictionary generation feature pattern group may be stored in a storage device or the like. The feature space may be obtained as one feature space common to all categories, or there may be a feature space for each category and a total of c feature spaces may be obtained. When a feature space exists for each category, the input pattern is also converted into a feature vector on each feature space. Therefore, in the case of c categories, an expression as a feature vector on c feature spaces can be obtained for one pattern. In addition, when a feature space exists for each category, the number of dimensions of the feature space may be different for each category.

  Below, the typical example of the method of converting a pattern into a vector value is demonstrated. However, the method of converting a pattern into a vector value is not limited to this typical example, and various methods are used. Here is an example. The process in this example is a flow of obtaining a feature vector on the feature space through pattern normalization, feature extraction, and dimension reduction of the original feature space.

  First, patterns such as images and sounds are normalized in order to suppress variations in individual patterns and to align the pattern conditions. Then, it is converted into a vector on a vector space by a feature extraction method. However, normalization may be omitted. The vector space is called an original feature space, and the vector is called an original feature vector. For example, in the case of character recognition, linear normalization, nonlinear normalization, moment normalization, etc. are used as normalization methods, and pixel feature extraction, contour feature extraction, gradient feature extraction, etc. are used as feature extraction methods. For details, see Non-Patent Document 2. Next, features are extracted from each pattern of the dictionary generation pattern group prepared in advance to generate original feature vectors. A set of original feature vectors generated from a dictionary generation pattern group and a label set indicating its category is referred to as a dictionary generation original feature pattern group.

  Next, the dimension reduction of the original feature space is performed using a conventional dimension reduction method such as a principal component analysis method or a linear discrimination method. However, this dimension reduction may be omitted. If omitted, the original feature space is set as the feature space and the original feature vector is set as the feature vector, and the process proceeds to the dictionary partial space generation step 23 as the next processing. Here, two examples of conventional dimension reduction methods using principal component analysis are given.

  The first is an example of dimension reduction for obtaining a feature space for each category. By performing principal component analysis only on patterns belonging to category k of the original feature pattern group for dictionary generation, a subspace approximating the distribution of category k can be obtained. This partial space is defined as a feature space corresponding to category k, and a dictionary generation feature pattern group corresponding to category k is obtained by projecting all of the original feature pattern groups for dictionary generation on the original feature space onto the feature space. In this case, a feature space and a dictionary generation feature pattern group exist corresponding to each of the c categories. For this, see Non-Patent Document 4, for example.

  The second is an example in which one feature space common to all categories is obtained as a result of dimension reduction. This feature space is obtained as a partial space that approximates the distribution of the entire pattern by performing principal component analysis on the entire original feature pattern group for dictionary generation. The partial space is defined as a feature space, and a dictionary generation feature pattern group obtained by projecting a dictionary generation original feature pattern group on the original feature space onto the feature space is defined as a dictionary generation feature pattern group. In this case, the feature space is one common to all categories.

Next, an example of the process of the feature extraction step 22 will be specifically described by taking a character recognition problem of recognizing numbers 0 to 9 in an image as an example. The dictionary generation pattern group is a set composed of N images (N is a natural number) of a set of images in which numbers are drawn and correct labels representing numbers drawn in the images. First, each image included in the dictionary generation pattern group is converted into a vector value on a d-dimensional vector space (original feature space) by a feature extraction method. Here, pixel features will be described as an example of a feature extraction method. It is assumed that the character image is, for example, a gray scale image of 16 × 16 pixels vertically and horizontally, and one pixel is represented by 256 gradations of integer values from 0 to 255. At this time, since the image is represented by 16 × 16 = 256 integer values, the image can be represented by a 256-dimensional vector having the values of each pixel as components. This 256-dimensional vector space is the original feature space, and the 256-dimensional vector is the original feature vector. As a result, the dictionary generation pattern group is converted into a set composed of a set (x _i , ω _i ) of the 256-dimensional vector x _i and the correct label ω _i as shown in the equation (1). For example, when the vector x _j is a vector converted from an image pattern in which the numeral 3 is drawn, ω _j = 3. This set is called an original feature pattern group for dictionary generation.

  Next, the dimension of the original feature space is reduced by a conventional dimension reduction method such as a principal component analysis method or a linear discrimination method. However, the dimension reduction by the conventional dimension reduction method is not performed, and the original feature space may be the feature space and the original feature vector may be the feature vector, and the process may proceed to the dictionary subspace generation step 23 as the next processing. Here, a dimension reduction method for obtaining a feature space for each category by principal component analysis for the original feature pattern group for dictionary generation belonging to the category for each category, and for the entire original feature pattern group for dictionary generation Two examples of dimension reduction methods that obtain one feature space common to all categories by principal component analysis are described.

First, a dimension reduction method by principal component analysis for the original feature pattern group for dictionary generation belonging to the category will be described for each category as a first example. In the principal component analysis for each category, the principal component analysis is performed only for the original feature pattern group for dictionary generation belonging to the category. This means that a partial space having a large distribution of the distribution of patterns belonging to the category is selected, and a partial space that closely approximates the distribution of the category is obtained. Specifically, it is obtained by solving the eigenvalue problem of the covariance matrix Σ _k of the pattern corresponding to the category k shown in Equation (2). Here, μ _k in the equation (2) is an average vector of the pattern distribution of the category defined by the equation (3). The algorithm for solving eigenvalue problems, Numerical Recipes in C:. The Art of Scientific Computing, William H. Press, Brian P. Flannery, Saul A. Teukolsky and William T.Vetterling, Cambridge University Press; 2 nd ed, 1992 ( See Reference 1). So [Phi _ki as the eigenvector corresponding to the upper m eigenvalues of the covariance matrix sigma _k Equation (4) (i = 1,2, ... m) and put. At this time, a partial space spanned by Φ _ki (i = 1, 2,..., M) is selected as a feature space corresponding to category k. The projection matrix onto this subspace is an m × d matrix Q _k defined by equation (5). Here, in equation (5), the upper right t of Q _k indicates that a transposed matrix of the Q _k. The same applies to the following equations. The original feature vector x on the original feature space is projected onto the feature vector y _k on the feature space corresponding to the category k using Q _k as shown in Equation (6).

As described above, the feature space corresponding to the category k is an m-dimensional feature space defined by the m × d projection matrix Q _k . The vector x on the d-dimensional original feature space is expressed in total c ways by the vector on the m-dimensional feature space corresponding to each category k by Equation (6). Here, C _k in equation (6) is a constant for normalizing the distribution scale.

Next, a dimension reduction method by principal component analysis for the entire original feature pattern group for dictionary generation as a second example will be described. In this case, the principal component analysis is performed on all the original feature pattern groups for dictionary generation. This means that a partial space expressing a component having a large variance of all distributions of the original feature pattern group for dictionary generation is selected, and a partial space that closely approximates the state of distribution of the entire pattern is obtained. Specifically, it is obtained by solving the eigenvalue problem of the covariance matrix Σ shown in Equation (7). Here, the average μ in the equation (7) is defined by the equation (8). Let the eigenvector corresponding to the upper m eigenvalues of the covariance matrix Σ be Φ _i (i = 1, 2,... M) as in equation (9). At this time, a partial space spanned by Φ _i (i = 1, 2,... M) is selected as the feature space. The projection matrix onto this partial space is an m × d matrix Q defined by Equation (10). The original feature vector x on the original feature space is expressed as a vector y on the feature space using Q as shown in Equation (11). As described above, the feature space after the dimension reduction becomes a feature space defined by the m × d projection matrix Q. Here, C in Equation (11) is a constant for normalizing the distribution scale.

The dimension-reduced m-dimensional feature space obtained by the dimension reduction method based on principal component analysis for the entire dictionary-generated feature pattern group according to the second example is defined by an m × d projection matrix Q. On the other hand, the feature space after dimension reduction obtained by the dimension reduction method by principal component analysis for the feature pattern group for dictionary generation according to the first example is divided into 10 categories corresponding to the number categories 0 to 9. A feature space that exists and corresponds to each category is defined by a projection matrix Q ₀ , Q ₁ ,..., Q ₉ .

The dimension reduction method according to the second example can be regarded as a special case where Q = Q ₀ = Q ₁ =... = Q _{9 in the first} example.

  As described above, having one feature space common to categories can be regarded as a special case in which a feature space exists for each of the c categories. Therefore, hereinafter, the description will be continued based on the case where a feature space exists for each of the c categories, and the case of one feature space common to the categories will be treated as a special case. When the original feature space is a feature space and the original feature vector is a feature vector, the projection matrix Q is Q = I (unit matrix).

This is the end of the description of the typical example of the process of the feature extraction step 22.
In the feature extraction step 22, information necessary for converting the pattern into a feature vector is stored in the recognition dictionary 13 for pattern recognition. The pattern recognition apparatus uses this information to convert the input pattern into a feature vector by the same method. The information stored in the recognition dictionary 13 for pattern recognition includes, for example, the normalization method and feature extraction method used, and a projection matrix for projecting from the original feature space to the feature space.

  Next, in the dictionary subspace generation step 23, the dimension of the feature space of each category is reduced using the dimension reduction method according to the present invention. The dictionary subspace generation step 23 is executed in the dictionary subspace generation unit 18.

  The dictionary subspace generation step 23 receives the feature pattern group for dictionary generation as input, performs dimension reduction of the feature space for each of the c categories by the dimension reduction method according to the present invention, and the feature space corresponding to each category. Get the subspace. This subspace is called a dictionary subspace. In the dictionary subspace generation step 23, a projection matrix from the feature space to the dictionary subspace is obtained as an output. The output projection matrix is stored in the recognition dictionary 13 for pattern recognition.

  The first feature of the present invention lies in the dictionary subspace generation step 23, and the feature space dimension reduction method is different from the conventional one. In the dictionary subspace generation step 23, a subspace advantageous for identification can be selected from among the subspaces of the feature space as the dictionary subspace.

  FIG. 4 is a flowchart showing an outline of the processing of the dictionary subspace generation step 23 executed by the arithmetic device 12.

  The feature of the present invention resides in a dimension reduction portion that reduces the dimension of the feature space by a method different from the conventional principal component analysis method, linear discrimination method, and the like. Before describing the processing flow shown in FIG. 4, first, the features of the present invention will be briefly described by taking the two-category discrimination problem in a two-dimensional space as an example. Thereafter, the idea of the present invention will be described using a conceptual diagram of a neural network.

  In order to simplify the description, as an example, consider the problem of selecting a one-dimensional feature space in the two-category identification problem of category 51 shown in FIG. 5 and category 61 shown in FIG. 6 distributed in a two-dimensional space.

  First, an example of dimension reduction by the principal component analysis method which is a conventional method will be shown. When principal component analysis is performed on category 61, the x-axis and y-axis shown in FIG. 6 are obtained as directions indicated by the eigenvectors of the covariance matrix. Since the variance of category 61 is larger in the x-axis direction, the x-axis in FIG. 6 is selected as the discrimination axis in dimension reduction by principal component analysis. At this time, the discriminant plane separating the two categories is a straight line perpendicular to the x-axis. For example, the straight line 71 in FIG. 7 can be selected as the discrimination surface. However, in this case, many patterns of the category 61 are distributed on both the left and right sides of the discriminating surface 71, and it can be seen that it is not effective for identifying two categories.

  Next, an example of dimension reduction according to the present invention will be described. In the method according to the present invention, a quadric surface that separates two categories is first created. A quadric surface that separates both categories is, for example, a quadratic curve 81 as shown in FIG. The quadric surface is represented by a contour surface of a quadratic function. This quadratic function is created by learning with a neural network as a quadratic function for identifying both categories. Next, the eigenvector of the quadratic matrix of the quadratic term of the quadratic function created by learning and the coefficient vector of the primary term are derived. Next, one or more vectors are selected from the eigenvectors of the quadratic form matrix of the quadratic term of the obtained quadratic function and the coefficient vectors of the primary term according to a certain criterion. In this example, consider selecting one of the eigenvectors. In the case of the quadratic curve 81, the direction represented by the eigenvector is the direction represented by the straight line 91 and the straight line 92 in FIG. Due to the nature of the quadratic function, the straight line 91 and the straight line 92 are symmetrical axes of the contour surface 81 of the quadratic function.

  As an example, assume that a straight line 91 is selected. In this case, the straight line 91 serves as a discrimination axis, and a straight line perpendicular to the straight line 91 serves as an identification surface. For example, the straight line 101 in FIG. At this time, the identification surface 101 can be an identification surface that separates the category 51 and the category 61. At this time, the category 51 pattern is distributed in the upper part of the identification surface 101 and the category 61 pattern is distributed in the lower part, so that it is possible to perform an advantageous identification as compared with the case of principal component analysis.

Next, the basic idea of the dimension reduction method according to the present invention will be described using a neural network. First, on the feature space corresponding to each category, a quadratic function f _k (x) such as Expression (12) that identifies a pattern belonging to the category and a pattern not belonging to the category is configured. Here, y _k = Q _k x (formula (13)). The quadratic function can be expressed by a neural network as shown in FIG.

First, the weights w _kij , w _ki , w _kc (i, j = 1,..., _{M) of the} quadratic function are learned. At this time, all values of f ₁ (x),..., F _c (x) are calculated for the original feature vector x, and when the largest value is f _k (x), It is determined that the pattern represented by the feature vector x belongs to the category k. In this determination, the weights w _kij , w _ki , w _kc (i, j = 1,..., _M ) are set so that the value of the loss function _indicating the loss when the dictionary generation pattern group is identified and misidentified becomes small. To learn. A specific example of learning will be described in the description of the processing of coefficient correction step 43 described later.

Next, by converting the feature vector using a certain matrix R _k as shown in Expression (14), the neural network can be reconfigured as shown in FIG. 12 without changing the output of the neural network. A method for deriving the matrix R _k will be described later.

Since the weights λ _i and z _{i in} FIG. 12 depend on k indicating the category, they should actually be expressed as λ _ki and z _ki , but in order to avoid complexity, the subscript k is Omitted. This is also omitted in FIG. 13 later.

Next, connections that do not affect the output of the neural network are removed. An example is given below. Now, it is assumed that the coupling weights λ _k1 ,..., Λ _km are arranged in descending order of absolute values. Here, the coupling from λ _{kn + 1} to λ _{km in} the lower part of the white circle in FIG. 13 is considered to be weak coupling. At this time, even if this connection is deleted, it is assumed that the influence on the output of the neural network is small. Therefore, if the coupling of this part is removed, the neural network can be approximated to the structure of the original neural network only by the part connected to the input unit corresponding to z _k1 ,..., Z _kn , z _k0 , 1. . This means that only n + 1 features z _k1 ,..., Z _kn , z _k0 hold sufficient information for identification. In the dimension reduction, the dimension is reduced from the m dimension to the feature dimension corresponding to the n + 1 features.

  Hereinafter, the processing flow will be described with reference to FIG. In the feature pattern group input step 41, a user or a program executed by the arithmetic unit 12 makes a feature vector group with a belonging category label on the feature space corresponding to each of c categories, that is, a feature pattern group for dictionary generation. Is entered. However, when the feature space is common to all categories, the dictionary generating feature pattern group is also a feature vector group on one feature space common to all categories.

The initial coefficient setting step 42 generates, for each of the c categories, a quadratic function on the feature space corresponding to the category. For example, when the feature vector y _k on the feature space of category k is expressed as in equation (15), the quadratic function on the feature space is expressed as in equation (16). Here, k is a subscript indicating a category, and m is the number of dimensions of the feature space corresponding to the category k. What is shown in Expression (17) is a coefficient of the quadratic function (Expression (16)). The quadratic function of Expression (16) can be defined by setting the coefficient shown in Expression (17). This quadratic function can be expressed by a neural network as shown in FIG.

  A value generated by a random number may be set as the coefficient of the quadratic function represented by Expression (16) (Expression (17)). For example, the origin of the feature space is set in advance so that the average of the feature pattern group for dictionary generation on the feature space is the origin of the feature space, and the variance of the feature pattern group for dictionary generation is 1. Suppose. At this time, for example, the coefficient shown in Expression (17) is set by a random number generated by uniform distribution in the range of −0.05 to 0.05. Alternatively, a random number may be generated by a Gaussian distribution with a variance of 0.1 centered at 0, and the value may be set to a coefficient represented by Expression (17). A random number is generated for each coefficient and set to the initial value of the coefficient.

  In the coefficient correction step 43, the coefficient of the quadratic function is corrected by the gradient descent method so that the value of the loss function by the identification using the quadratic function becomes small. Hereinafter, the coefficient correction process will be described using Equations (15) to (29).

First, an example of category identification using a quadratic function will be described. A feature vector obtained by projecting an original feature vector in the original feature space extracted from an input pattern such as an image or sound onto a feature space corresponding to category i is expressed as y _i (Expression (15)). Next, the values of quadratic functions f ₁ (y ₁ ),..., F _c (y _c ) (formula (16)) in the feature vector expressing the pattern on the feature space of each category are calculated. Identification is performed by regarding the value of the quadratic function f _i (y _i ) (equation (16)) in the feature vector y _i (equation (15)) as the similarity to the category i of the input pattern. When the maximum value among the calculated c values f ₁ (y ₁ ),..., F _c (y _c ) is f _k (y _k ), it is identified that the category to which the input pattern belongs is k. .

The loss function is the sum of the partial loss functions. The partial loss function is a function representing a loss due to misidentification when one pattern included in the dictionary generation pattern group is identified, and is defined for each pattern of the dictionary generation pattern group. The partial loss function is a function that depends on the weights w _kij , w _ki , w _kc (i, j = 1,..., _M ). An example of the loss function will be described. A set of a set of original feature vectors and correct labels expressing a pattern group for generating a dictionary including N patterns on the original feature space is given by Equation (1). The target variable l _k ^s (equation (18)) is set to 1 when the s-th pattern belongs to the category k, and is set to 0 otherwise. At this time, the partial loss function due to misidentification of the sth pattern is defined by, for example, Expression (19). U _{k in} equation (19) is defined by equation (20) using the sigmoid function in equation (21). This is when the value u _k (x _s ) in the s-th pattern of the function u _k (equation (20)) corresponding to the category k is close to 1 when the s-th pattern x _s belongs to the category k. This is equivalent to assuming that it has been correctly identified and that it is erroneously identified when it is close to zero. Further, the equation (22) to which the regularization term is added may be used as the partial loss function. Here, b is called load attenuation. The loss function E _k is given by the sum of partial loss functions as shown in equation (23). Equation (23) is called a square error function.

  Next, a correction formula for the coefficient of the quadratic function will be described. The coefficient is corrected by the gradient descent method or the stochastic gradient descent method so that the loss function (equation (23)) becomes small.

First, an example of the gradient descent method will be described. In the gradient descent method, the repetition coefficient is corrected by the gradient method so that the loss function (equation (23)) calculated from the feature pattern group for dictionary generation becomes small. When a vector in which the coefficients are collected is set as in Expression (24), the coefficient value θ _k (t + 1) by the t + 1th correction is given by Expression (25). Here, a is a learning rate rate. For example, a takes a value approximately equal to the scale of variance when t = 0, and is set to be smaller by a certain number so as to become 0 at t corresponding to the last correction. Since the right side of Equation (25) is calculated using all the dictionary generation feature pattern groups, the gradient descent method is batch learning using all dictionary generation feature pattern groups.

  Next, an example of the stochastic gradient descent method will be described. In the gradient descent method, since the derivative of the loss function (Equation (23)) appears on the right side of Equation (25), the right side of Equation (25) must be calculated using all the feature patterns for dictionary generation. Don't be. In the stochastic gradient descent method, the coefficient is corrected by equation (26) using a partial loss function instead of a loss function. Therefore, the correction is sequentially performed for each pattern in the dictionary generation feature pattern group. The correction is performed by the equation (26) each time the dictionary generation feature pattern group is given randomly or sequentially. The number of corrections is such that the entire dictionary generating feature pattern group is circulated about 40 times, for example. When the derivative of the partial loss function in equation (26) is specifically written for each coefficient, equations (27), (28), and (29) are obtained.

  In the following, in order to avoid complication, the subscript k attached to indicate the category may be omitted.

  In the basis vector deriving step 44, for each of the c categories, the eigenvector of the matrix in the quadratic form of the quadratic term of the quadratic function created in the initial coefficient setting step 42 and the coefficient correcting step 43, Derive the coefficient vector of the term. When the quadratic function is expressed as in Expression (16), the quadratic term and the primary term are expressed as Expressions (30) and (31), respectively. The quadratic form of the quadratic function (equation (16)) and the quadratic form of the quadratic function (equation (30)) is given by the symmetric matrix shown in equation (32). Is given by equation (33). Using equations (32) and (33), the quadratic function (equation (16)) can be expressed as equation (34).

Next, the quadratic function of Expression (16) is given in an expression based on the eigenvector of the matrix shown in Expression (32). The eigenvector of the equation (32) is represented by p _k1 , p _k2 ,..., _Km as in the equation (35), and the matrix P _k is given as in the equation (36). When Expression (32) is expressed using the vector of Expression (35) as a basis, a diagonal matrix like Expression (37) is obtained. Since the matrix P _k is an orthogonal matrix, Expression (16) is expressed as Expression (39) using Expressions (37) and (38). Using equations (37), (38), and (39), equation (16) becomes equation (40). Further, when the equation (41) is used and the equation (42) is used, the equation (16) can also be expressed as the equation (43). For example, Reference Document 1 provides a detailed algorithm for deriving the eigenvectors of the symmetric matrix (Formula (32)). In the case of the example of FIG. 9, the eigenvectors of the quadratic matrix of the quadratic term of the quadratic function (16) are a straight line 91 and a straight line 92.

  The projection matrix deriving step 45 includes, for each category, an eigenvector (formula (35)) of the quadratic form of the quadratic term of the quadratic function obtained in the basis vector deriving step 44 and a coefficient vector (formula of the primary term). (33)), one or more vectors are selected, and a subspace generated by the selected vectors is generated as a dictionary subspace. A dictionary subspace exists for each category. In a projection matrix deriving step 45, a projection matrix from the feature space to the dictionary subspace is obtained for each category.

  In the following, first, a selection reference example is shown. After that, a method for obtaining a projection matrix to the dictionary subspace generated by the selected vector group will be described.

A reference example of vector selection in the projection matrix derivation 45 is shown.
First, an example of a selection criterion from eigenvectors of a quadratic matrix (formula (35)) of a quadratic term of a quadratic function is shown.

Example 1: When the absolute value of the i-th coefficient (equation (44)) in the quadratic term of equation (43) is greater than or equal to a predetermined threshold h as shown in equation (45), All corresponding vectors p _ki are selected. The threshold value h may be determined as a value of about 0.01 to 0.5, for example, when the variance scale is 1, depending on the number of dimensions to be reduced.

Example 2: Select n items in descending order of the absolute value of the i-th coefficient (equation (44)) in the quadratic term of equation (43), and, as in example 1, select the eigenvector p _ki corresponding thereto. select. The value of the natural number n may be determined in advance.

  Example 3: The i-th coefficient (equation (44)) in the quadratic term of equation (43) is divided into positive and negative ones, and positive and negative weights are alternated from the one with the largest absolute value. Select n.

These meanings will be described in the case of Example 1. In the case of Example 1, this corresponds to approximating only the main part by ignoring a term having a small coefficient λ _ii as the expression (46) as having little contribution to recognition. However, this approximation is an approximation when no coefficient vector is selected. When a coefficient vector is selected according to the criteria described later, it is approximated as shown in Equation (47).

Next, a reference example for selecting the coefficient vector w ₁ (formula (33)) of the first-order term of the quadratic function is shown. However, if equation (33) is a zero vector, no coefficient vector is selected.

Example 1: First, an eigenvector of a quadratic matrix of a quadratic term of a quadratic function is selected according to any of the above criteria. Thereafter, a coefficient vector component orthogonal to the subspace generated by the selected eigenvector is obtained. A coefficient vector is selected when the aforementioned orthogonal component is equal to or greater than a certain threshold value h. The value of the threshold value h may be determined in advance. The threshold value h is generally different from h in the vector selection criterion example 1 among the eigenvectors of the quadratic matrix of the quadratic term of the quadratic function described above.
Example 2: Select coefficient vectors for all categories.
Example 3: No coefficient vector is selected for all categories.

  The above is a reference example of coefficient vector selection. The eigenvectors of the quadratic matrix of the quadratic term of the quadratic function are the eigenvectors of the equation (32), so there are m. The coefficient vector of the first-order term of the quadratic function is given by the equation (33) and is one. These m + 1 vectors are generally different. In the projection matrix deriving step 45, the eigenvector of the quadratic form of the quadratic term of the quadratic function and the coefficient vector of the primary term are selected according to the selection criteria as described above and other criteria. Select one or more vectors. A subspace generated by the selected one or more vectors is a dictionary subspace.

Next, a method for obtaining a projection matrix to a subspace generated by the selected vector group in the projection matrix deriving step 45 will be described. In the following, the selected eigenvectors are set as q _k1 , q _k2 ,..., Q _kn as in equation (48). Since these are eigenvectors of a symmetric matrix, they form an orthonormal system. However, since the coefficient vector (Equation (33)) is not necessarily orthogonal to these, the vector obtained by normalizing the components of the coefficient vector (Equation (33)) orthogonal to the eigenvector (Equation (48)) Ask for. When equation (49) is not 0, the normalized component of the coefficient vector orthogonal to the space generated by the eigenvector is given by equation (50).

First, when the coefficient vector is not selected, the subspace to be obtained is a space generated by the vectors q _k1 , q _k2 ,..., Q _kn of the equation (48), and therefore the projection matrix is given by the equation (51). . Similarly, when the equation (49) is 0, the matrix R _k of the equation (51) is a projection matrix.

Next, when a coefficient vector is selected and Equation (49) is not 0, the subspace to be _obtained is generated by the vectors q _k1 , q _k2 ,..., Q _kn of Equation (48) and the vector r _k of Equation (50). The Therefore, the projection matrix is given by the matrix R _{k in} equation (52).

  The projection matrix output step 46 stores or outputs information for projecting the feature space vector to the dictionary subspace in the pattern recognition dictionary 13 for each of the c categories. For example, a projection matrix (Formula (51) or Formula (52)) is stored. Further, the projection matrix output step 46 recognizes the coefficients of the quadratic function obtained by projecting the quadratic function generated in the initial coefficient setting step 42 and the coefficient correction step 43 into the dictionary subspace for each of the c categories. It may be stored in the dictionary 13 or output.

When the transformation from the original feature vector to the feature vector is expressed by the projection matrix Q _k as shown in Equation (13), the matrix product R _k Q _k is calculated and stored in the recognition dictionary 13. Keep it. Thereby, at the time of identification, a projection vector on the dictionary subspace can be obtained by calculating R _k Q _k x after extracting the original feature vector x.

  Returning to FIG. 2, the learning feature pattern group generation step 24 projects each of the dictionary generation feature pattern groups into the dictionary subspace by the projection matrix for each category. The projection of each category of the dictionary generation feature pattern group generated in this way onto the dictionary subspace is called a learning feature pattern group. Step 24 is executed by the learning feature pattern group generation unit 19.

  The discriminant function generation step 25 learns the discriminant function using the learning feature pattern group on the dictionary subspace generated for each category. An identification function is created for each category. The discriminant function is generated by learning so that the value of the vector representing the pattern on the dictionary subspace of the category is the pattern similarity to the corresponding category. As a learning method of the discriminant function, for example, a nearest neighbor method, a learning vector quantization method, a projection distance method, a support vector machine, a neural network, or the like can be used. For details, see Non-Patent Document 2. Step 25 is executed by the discriminant function generator 20.

  The output step 26 stores the parameters of the discrimination function created in the discrimination function generation step 25 in the pattern recognition recognition dictionary 13.

  The features of the pattern recognition dictionary generating apparatus of the present invention will be described below. In the present invention, the dimension reduction technique in the dictionary subspace generation step 23 in FIG. 2 is different from the conventional one. In a conventional pattern recognition dictionary generation device, dimension reduction is performed by a method such as a principal component analysis method or a linear discriminant method. However, in the pattern recognition dictionary generation apparatus according to the present invention, the dimension of the feature space is reduced by a series of processes shown in FIG. Prior to these series of processes, the feature extraction step 22 may perform dimension reduction using a conventional dimension reduction technique. Although conventional dimension reduction may be omitted, it is possible to speed up the subsequent processing by executing the conventional dimension reduction and reducing the dimensions to some extent in advance.

The following shows the recognition rate and the time required for identification per character in the dimension reduction method by the principal component analysis method, which is a conventional dimension reduction method, and the dimension reduction method of the present invention in a handwritten kana recognition experiment. Measured and compared. In the experiment, 75,000 learning samples and 22,631 test samples different from the learning samples were used. A part of the test sample is shown in FIG. As the classifier, a polynomial classifier using class-specific features and excluding the projection distance was used. See Non-Patent Document 4 for this.
Method Feature dimension Recognition rate (%) Identification time (ms)
Conventional dimension reduction method 60 98.10 2.544
Conventional dimension reduction method 100 98.43 7.342
Conventional dimension reduction method 200 98.72 18.164
Dimensional reduction method according to the present invention 15 98.63 0.873
Dimensional reduction method according to the present invention 10 98.47 0.629

  It can be seen that the dimension reduction method according to the present invention enables recognition with high speed and high accuracy in a feature space of a lower dimension than the conventional one.

[Example 2]
The configuration diagram of the pattern recognition apparatus in the present embodiment is the same as that in FIG. The input device 11 is a device such as a keyboard and a mouse for inputting commands and the like, and a pattern input device such as a scanner and a camera. The arithmetic device 12 reads the input image, voice pattern, etc., and uses information stored in the pattern recognition recognition dictionary 13 to determine which category the input pattern is close to. Recognize based on. The computing device 12 may include the pattern recognition dictionary creating means of the first embodiment. The arithmetic device 12 includes a CPU, a memory, a storage device, and the like.

  The pattern recognition recognition dictionary 13 is a dictionary database that is referred to when the arithmetic device 12 recognizes an input pattern. The pattern recognition dictionary stored in the pattern recognition recognition dictionary 13 is a pattern recognition dictionary created using the dimension reduction method of the present invention. The display device 15 is a device such as a display for displaying the recognition result of the input pattern by the arithmetic device 12.

  The pattern DB 16 stores the pattern input by the input device 11. The pattern DB 16 may store a dictionary generation pattern group DB used by the arithmetic device 12 to create the pattern recognition recognition dictionary 13, pattern data to be recognized by the arithmetic device 12, and the like. Further, the pattern may not be directly input from the input device.

  FIG. 3 is a processing flow of pattern recognition means executed by the arithmetic unit 12 of this embodiment.

In the input step 31, a pattern to be recognized is input by a user or a program executed by the arithmetic device 12. In the feature vector extraction step 32, the input pattern is converted into a feature vector on the feature space. The input / output for converting the input pattern into the feature vector is the same as the input / output for converting the input pattern into the feature vector in the feature extraction step 22 in FIG. That is, the output for the input pattern is the same as the output of the feature extraction step 22 for the input pattern. Information necessary for conversion may be stored in the recognition dictionary 13. For example, in the example of the first embodiment, when the projection matrix from the original feature space to the feature space is obtained as Q _k , the original feature vector x is extracted from the pattern, and then output by the projection matrix Q _k . To obtain a feature vector Q _k x.

  The projection vector generation step 33 projects the feature vector generated by the feature vector extraction step 32 to the dictionary subspace generated by the dictionary subspace generation step 23 for each category. A projection matrix for projection onto the dictionary subspace is created by the dictionary subspace generation step 23 and stored in the pattern recognition recognition dictionary 13. A projection vector generation step 33 creates a projection vector on the dictionary subspace for each category. The feature of the pattern recognition apparatus of this embodiment is that a projection vector onto the dictionary subspace generated by the dimension reduction method of the present invention is used.

The transformation from the original feature vector to the feature vector is expressed by the projection matrix Q _k as shown in Equation (13), and the product R _k Q _k with the projection matrix R _k from the feature space to the dictionary subspace is the recognition dictionary. 13, the projection vector on the dictionary subspace can be obtained by obtaining R _k Q _k x from the original feature vector x.

  The identification step 34 uses the projection vector of each category created in the projection vector generation step 33 and the parameter of the identification function for each category created in the discrimination function generation step 25 and stored in the pattern recognition dictionary 13. The similarity for each category of the input pattern is obtained. The similarity to each category is calculated as the value of the category identification function at the point represented by the projection vector of the input pattern projected onto the dictionary subspace on the category.

  After calculating the similarity for all categories, the category to which the input pattern belongs is determined. Usually, it is determined that the input pattern belongs to the category having the highest similarity. In some cases, it may be rejected. For example, the category having the highest similarity among all categories is determined as the category to which the pattern belongs. There is also a technique of determining that the category does not belong to any category and rejecting when the first similarity, which is the highest value among the similarities, is equal to or less than a certain threshold. There is also a method of rejecting as an ambiguous category even if the first similarity and the second similarity are equal to or less than a certain threshold value with the second highest similarity value as the second similarity. is there.

  The pattern recognition apparatus of the present invention calculates the similarity using the discrimination function stored in the dictionary created by the dictionary creation apparatus of the present invention. The discriminant function is a function on the dictionary subspace generated by the dimension reduction method according to the present invention.

  In the output step 35, the recognition result in the identification step 34 is output to the display device 15 such as a display, stored in a storage device, or the like.

[Example 3]
An example of an embodiment in which the dictionary partial space generation step 23 is changed in the pattern recognition dictionary generation apparatus of the first embodiment will be described. The apparatus configuration is shown in FIG. 1 as in the first embodiment.

  In the dictionary subspace generation step 23 of the first embodiment, the dimension of the feature space can be moderately reduced by repeatedly reducing the number of small dimensions. As a result, dimensions can be significantly reduced without significantly affecting the recognition accuracy.

A specific method will be described. The dictionary subspace generated by the series of processing from the first feature pattern group input step 41 to the projection matrix output step 46 in the dictionary subspace generation step 23 is called a primary dictionary subspace. First, in the first process is stored in the projection matrix output step 46, projection matrix from the feature space to the primary subspace (referred to as A _1). Next, the primary dictionary subspace is regarded as a feature space, the process returns to the initial value setting step 42, and the processing up to the projection matrix output step 46 is continued. Thereby, dimension reduction of the primary dictionary subspace is performed. The subspace of the primary dictionary subspace generated by this is called a secondary dictionary subspace. In storing step 45 of projection matrix, to store the primary subspace projection matrix to secondary subspace (referred to as A _2). Again, the secondary dictionary subspace may be regarded as a feature space, and the process may continue by returning to the initial value setting step 42. At this time, for example, gradual dimension reduction can be performed by decreasing the number of dimensions by a fixed number. This is repeated t times, and the finally generated t-th dictionary subspace is defined as a dictionary subspace. Projection matrix to the dictionary subspace matrix A _t, ..., obtained by the product of A _1. Further, when storing or outputting the quadratic function, the coefficient of the quadratic function obtained by limiting the quadratic function generated in the t-th process to the dictionary subspace is stored or output.

[Example 4]
In the pattern recognition dictionary generation apparatus according to the first embodiment, an embodiment in which the quadratic function generated in the dictionary subspace generation step 23 is used as an identification function will be described. The apparatus configuration is shown in FIG. 1 as in the first embodiment.

  A quadratic function (Equation (16)) obtained in the dictionary subspace generation step 23 of Embodiment 1 limited to the dictionary subspace can be used as the discriminant function. For example, when the eigenvector selection criterion example 1 in the projection matrix derivation step 45 of the second embodiment is adopted for selecting the eigenvector, the quadratic function in which the quadratic function (equation (16)) is limited to the dictionary subspace is expressed by the equation (46 ) Or equation (47), which is an approximation of the original quadratic function (equation (16)).

  Therefore, the quadratic function restricted to the dictionary subspace may be regarded as an identification function, and the processing may be directly transferred to the output step 26 after the projection matrix storage step 46. As a result, the processing time for creating the pattern recognition dictionary can be greatly reduced. In addition, since the dictionary subspace generation step 23 can select a low-dimensional dictionary subspace, high accuracy, high speed, and memory saving in the pattern recognition apparatus can be realized.

The figure which shows the structure of the dictionary production | generation apparatus for pattern recognition of this invention, and a pattern recognition apparatus. The figure which shows the processing flow of the dictionary generation apparatus for pattern recognition of this invention. The figure which shows the processing flow of the pattern recognition apparatus of this invention. The figure which shows the processing flow of the dimension reduction method of this invention. The distribution map of the 1st category pattern in the two category classification problem for demonstrating the example of dimension reduction. The distribution map of the 2nd category pattern in the 2 category classification problem for demonstrating the example of dimension reduction. The figure which showed the discrimination surface by the conventional method in the two category classification problem for demonstrating the example of dimension reduction. The figure which showed the contour surface of the quadratic function used for the dimension reduction method of this invention in the two-category classification problem for demonstrating the example of dimension reduction. The figure which showed the eigenvector of the quadratic form of the quadratic term of the quadratic function in the dimension reduction method of this invention in the two category classification problem for demonstrating the example of dimension reduction. The figure which showed the discrimination surface by this invention in the two category classification | category problem for demonstrating the example of dimension reduction. The figure which expressed the quadratic function by the neural network. The figure of the neural network after transforming the base of the feature space. Diagram of neural network after dimension reduction. The figure which shows the sample of a handwritten kana character.

Explanation of symbols

11 Input Device 12 Arithmetic Device 13 Recognition Dictionary 15 Display Device 16 Pattern DB
21 Input 22 Feature extraction 23 Dictionary subspace generation 24 Learning feature pattern group generation 25 Discriminant function generation 26 Output 31 Input 33 Projection vector generation 34 Identification 35 Output 41 Feature pattern group input 42 Initial coefficient setting 43 Coefficient correction 44 Base vector derivation 45 Projection matrix derivation 46 Projection matrix output 51 Category 51
61 Category 61
71 Discrimination surface 81 Contour surface of quadratic function 91 Eigenvector 92 Eigenvector 101 Discrimination surface

Claims (20)

A feature pattern group input step for inputting a feature pattern group for dictionary generation on the feature space;
an initial coefficient setting step for setting a value of a coefficient of a quadratic function on the feature space corresponding to each of c categories (c is a natural number);
The coefficient of the quadratic function is reduced in gradient so that the value of the loss function representing the loss due to misidentification when determining the belonging category of each feature pattern group for dictionary generation using the quadratic function as the discriminant function is small. A coefficient correction step to be corrected by a method or a stochastic gradient descent method;
For each of the c categories, one or more vectors are selected from the eigenvectors of the symmetric matrix of the quadratic form of the quadratic term of the quadratic function and the coefficient vector of the primary term of the quadratic function. A basis vector derivation step to select;
For each of the c categories, a projection matrix derivation step for generating a projection matrix into a dictionary subspace spanned by the vectors selected in the basis vector derivation step;
Projecting the projection matrix onto c dictionary subspaces corresponding to each of the c categories, or the projection matrix corresponding to each of the c categories and a coefficient of the quadratic function. A matrix output step;
A feature space dimensionality reduction method characterized by comprising:   The dimension reduction method for the feature space according to claim 1, wherein the feature space is a vector space, and the feature pattern group for dictionary generation is a set of labeled vector values indicating a category to which the vector space belongs. A feature dimension reduction method for feature space.   2. The feature space dimension reduction method according to claim 1, wherein in the initial coefficient setting step, a value generated by a random number is set to a coefficient of the quadratic function corresponding to each of the c categories. Feature space dimension reduction method.   2. The feature space dimension reduction method according to claim 1, wherein, in the coefficient correction step, a square error function obtained by determining each affiliation category of the dictionary generating feature pattern group using the quadratic function as an identification function is a loss function. A feature space dimension reduction method characterized by:   The dimension reduction method of the feature space according to claim 1, wherein in the projection matrix derivation step, the eigenvector corresponding to the top n large absolute values of the eigenvalues of the coefficient vector and the symmetric matrix, or the coefficient vector and the A feature space dimension reduction method, comprising: generating a projection matrix onto a partial space of a feature space generated by the eigenvector corresponding to the top n eigenvalues of a symmetric matrix whose absolute value is larger than a threshold value h.   2. The dimension reduction method for a feature space according to claim 1, wherein in the projection matrix derivation step, the eigenvector corresponding to the top n largest eigenvalues of the symmetric matrix or the eigenvalues of the symmetric matrix are calculated. A feature space dimension reduction method, comprising generating a projection matrix onto a partial space of a feature space generated by the eigenvector corresponding to the top n larger than a threshold value h. An input unit for inputting a dictionary generation pattern group;
A feature extraction unit that extracts a dictionary generation feature pattern group on a feature space corresponding to each of c categories (c is a natural number) from the dictionary generation pattern group;
A feature pattern group input process for inputting the dictionary generating feature pattern group on the feature space; an initial coefficient setting process for setting a value of a coefficient of a quadratic function on the feature space corresponding to each of c categories; The coefficient of the quadratic function is reduced in gradient so that the value of the loss function representing the loss due to misidentification when determining the belonging category of each feature pattern group for dictionary generation using the quadratic function as the discriminant function is small. Or a coefficient correction process corrected by a stochastic gradient descent method, for each of the c categories, an eigenvector of a quadratic symmetric matrix of a quadratic term of the quadratic function, and a linear function of the quadratic function A basis vector deriving process for selecting one or more vectors from the coefficient vector of the term, and the vector selected in the basis vector deriving process for each of the c categories. A projection matrix derivation process for generating a projection matrix to a dictionary subspace spanned by the c, the projection matrix to the c dictionary subspaces corresponding to each of the c categories, or the c categories A dictionary subspace generation unit that generates the projection matrix to the dictionary subspace by a dimension reduction process of a feature space including a projection matrix output process that outputs the projection matrix and the coefficient of the quadratic function corresponding to each of the projection matrix,
A learning feature pattern group generation unit that generates a learning feature pattern group obtained by projecting the dictionary generation feature pattern group onto the dictionary subspace by the projection matrix;
An identification function generator for generating an identification function for identifying the c categories using the learning feature pattern group;
An output unit for outputting the projection matrix and parameters of the discriminant function;
A dictionary recognition apparatus for pattern recognition comprising:   8. The pattern recognition dictionary generating device according to claim 7, wherein the discriminant function generating unit generates a restricted quadratic function in which the quadratic function is limited to the dictionary subspace for each of the c categories, Gradient descent method or probability so that the square error function becomes a loss function with the square error function when determining the belonging category of each of the learning feature pattern groups using the restricted quadratic function as the discriminant function. A pattern recognition dictionary generation device, wherein a coefficient obtained by correcting the coefficient of the limited quadratic function is generated as a discriminant function by a gradient descent method.   8. The pattern recognition dictionary generating device according to claim 7, wherein the discriminant function generating unit generates, for each of the c categories, a discriminant function in which the quadratic function is limited to the dictionary subspace. Characteristic pattern recognition dictionary generator.   8. The pattern recognition dictionary generation device according to claim 7, wherein the feature space is a vector space, and the dictionary generation feature pattern group is a set of labeled vector values indicating a category to which the vector space belongs. Characteristic pattern recognition dictionary generator.   8. The pattern recognition dictionary generating apparatus according to claim 7, wherein the initial coefficient setting process sets a value generated by a random number as a coefficient of the quadratic function corresponding to each of the c categories. Pattern recognition dictionary generator.   8. The pattern recognition dictionary generation apparatus according to claim 7, wherein the coefficient correction processing uses a square error function as a loss function when determining each affiliation category of the dictionary generation feature pattern group using the quadratic function as an identification function. A dictionary recognition apparatus for pattern recognition, characterized in that   8. The pattern recognition dictionary generating apparatus according to claim 7, wherein the projection matrix deriving process includes the eigenvector corresponding to the top n largest absolute values of the eigenvalues of the coefficient vector and the symmetric matrix, or the coefficient vector and the A dictionary generating apparatus for pattern recognition, characterized by generating a projection matrix onto a partial space of a feature space generated by the eigenvectors corresponding to the top n eigenvalues of a symmetric matrix whose absolute value is larger than a threshold value h.   8. The pattern recognition dictionary generating apparatus according to claim 7, wherein the projection matrix derivation process is performed by calculating the eigenvector corresponding to the top n largest eigenvalues of the symmetric matrix or the eigenvalue of the symmetric matrix. A dictionary generating apparatus for pattern recognition, characterized by generating a projection matrix onto a partial space of a feature space generated by the eigenvectors corresponding to the top n larger than a threshold value h. Feature pattern group input processing for inputting a dictionary generation feature pattern group on the feature space, an initial coefficient for setting a value of a coefficient of a quadratic function on the feature space corresponding to each of c categories (c is a natural number) Coefficient of the quadratic function so that the value of the loss function representing the loss due to misidentification when determining the belonging category of each feature pattern group for dictionary generation using the quadratic function as the discriminating function is reduced. For each of the c categories, the eigenvector of the symmetric matrix of the quadratic form of the quadratic term of the quadratic function, and the quadratic A basis vector deriving process for selecting one or more vectors from coefficient vectors of a first-order term of the function, and each of the c categories is selected in the basis vector deriving process. A projection matrix deriving process for generating a projection matrix to a dictionary subspace spanned by vectors, c projection matrices to the c dictionary subspaces corresponding to each of the c categories, or c categories of A parameter of the discriminant function on the dictionary subspace generated by the dimension reduction processing of the feature space including the projection matrix corresponding to each and a projection matrix output processing for outputting the coefficient of the quadratic function, and the projection matrix A recognition dictionary storage for storing
An input unit that inputs an input pattern to be recognized; a feature extraction unit that extracts a vector value on a feature space from the input pattern;
For each of the c categories, a projection vector generation unit that generates a projection vector obtained by projecting an input pattern onto a dictionary subspace using the projection matrix;
For each of the c categories, an identification unit that calculates a value at a point represented by the projection vector of the identification function and identifies a category to which the input pattern belongs based on the calculated value.
An output unit for outputting the identification result in the identification unit;
A pattern recognition apparatus comprising:   16. The pattern recognition apparatus according to claim 15, wherein the feature space is a vector space, and the feature pattern group for dictionary generation is a set of labeled vector values indicating affiliation categories in the vector space. Pattern recognition device.   16. The pattern recognition apparatus according to claim 15, wherein the initial coefficient setting process sets a value generated by a random number as a coefficient of the quadratic function corresponding to each of the c categories. .   16. The pattern recognition apparatus according to claim 15, wherein the coefficient correction processing uses a square error function when a category belonging to each of the dictionary generating feature pattern groups is determined as a loss function using the quadratic function as an identification function. A pattern recognition device.   16. The pattern recognition apparatus according to claim 15, wherein the projection matrix deriving process includes the eigenvector corresponding to the top n largest absolute values of the eigenvalues of the coefficient vector and the symmetric matrix, or the coefficient vector and the symmetric matrix. A pattern recognition apparatus for generating a projection matrix onto a partial space of a feature space generated by the eigenvectors corresponding to the top n eigenvalues whose absolute values are larger than a threshold value h.   16. The pattern recognition apparatus according to claim 15, wherein the projection matrix deriving process is performed such that the eigenvector corresponding to the top n largest eigenvalues of the symmetric matrix or the eigenvalue of the symmetric matrix is greater than a threshold value h. A pattern recognition apparatus for generating a projection matrix onto a partial space of a feature space generated by the eigenvector corresponding to the largest n largest.

Images