Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
  • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/22—Matching criteria, e.g. proximity measures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
  • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
  • G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]

Definitions

• the invention relates to machine learning, and more particularly, to fast regularized least squares classification/regression.
• H is a Reproducing Kernel Hubert Space or RKHS (Aronszajn, 1950) with associated kernel function K
• ⁇ f ⁇ ⁇ is the squared norm in the RKHS
• ⁇ is a regularization constant controlling the tradeoff between fitting the training set accurately and forcing smoothness of/ (in the RKHS norm)
• V(f(x),y) is a loss function representing the price paid when x is seen and f(x) is predicted, but the actual associated value is y.
• Different choices of V give rise to different learning schemes (Evgeniou et al., 2000).
• SVC support vector machine
• Representer Theorem proves that the solution to the Tikhonov minimization problem will have the following form (Sch ⁇ lkopf et al, 2001; Wahba, 1990): n
• the Representer Theorem reduces the infinite dimensional problem of finding a function in an RKHS to the n-dimensional problem of finding the coefficients c,.. For Regularized Least
• Ky K(X ⁇ ) .
• RLSC regularized least squares classifier
• RLS has a serious problem as compared to the more popular SVM.
• direct methods for solving RLS manipulate the entire kernel matrix and therefore require O[n 3 ) time and (even worse)
• One embodiment of the present invention provides a computer implemented methodology for regularized least squares (RLS) classification/regression.
• the method includes receiving a training set of data, and computing a matrix decomposition using the training set (e.g., eigendecomposition or SVD).
• the method further includes receiving a plurality of regularization parameters, and computing coefficients for each regularization parameter using the matrix decomposition.
• the method continues with computing a leave- one-out (LOO) error for each of the regularization parameters, and selecting the regularization parameter with the lowest LOO error.
• the method may further include predicting future data points based on at least one of coefficients c or a hyperplane function w associated with the selected regularization parameter.
• computing a matrix decomposition using the training set includes computing a singular value decomposition (SVD) using the training set, wherein the SVD is the matrix decomposition.
• computing a matrix decomposition using the training set includes forming a kernel matrix using the input training set, and computing an eigendecomposition of the kernel matrix, wherein the eigendecomposition is the matrix decomposition.
• the kernel matrix is represented explicitly, and the method includes storing the kernel matrix, hi another such case, the kernel matrix is represented explicitly, and the method computes the LOO error for all the regularization parameters in O(n s + n 2 d) time and O(n 2 ) space, where n is the number of points in d dimensions of the training set.
• ⁇ , X j is a vector of data points included in the training set and ⁇ is a user-selected bandwidth parameter.
• the kernel matrix can be represented using, for example, matrix vector products (particularly if the matrix is too large to store all at once). In one such case, the matrix vector products are approximated using an Improved Fast Gauss Transform (IFGT).
• IFGT Improved Fast Gauss Transform
• Another embodiment of the present invention provides a machine-readable medium (e.g., one or more compact disks, diskettes, servers, memory sticks, or hard drives) encoded with instructions, that when executed by one or more processors, cause the processor to carry out a process for regularized least squares (RLS) classification/regression.
• This process can be, for example, similar to or a variation of the previously described method.
• Another embodiment of the present invention provides a regularized least squares (RLS) classification/regression system.
• the system functionality can be implemented with a number of means, such as software (e.g., executable instructions encoded on one or more computer- readable mediums), hardware (e.g., gate level logic or one or more ASICs), firmware (e.g., one or more microcontrollers with I/O capability and embedded routines for carrying out the functionality described herein), or some combination thereof, hi one particular case, the system is implemented in a computing environment such as a desktop or laptop computer, with an executable RLS classification module or set of modules stored therein.
• software e.g., executable instructions encoded on one or more computer- readable mediums
• hardware e.g., gate level logic or one or more ASICs
• firmware e.g., one or more microcontrollers with I/O capability and embedded routines for carrying out the functionality described herein
• the system is implemented in a computing environment such as a desktop or laptop computer, with an executable RLS classification module or set of modules stored therein.
• Figure 1 is a block diagram illustrating a computing environment configured in accordance with an embodiment of the present invention.
• FIG. 2 is a block diagram illustrating a regularized least squares (RLS) classifier/regression module configured accordance with an embodiment of the present invention.
• RLS regularized least squares
• Figures 3a-d each illustrates accuracy of an RLS classifier/regression over a range of ⁇ and ⁇ , in accordance with an embodiment of the present invention.
• Figure 4 illustrates
• Figure 5 illustrates an RLS classification/regression methodology configured in accordance with an embodiment of the present invention.
• a regularized least squares classifier is provided.
• classifiers are programmed or otherwise configured to receive input values (such as values of an object's features or a situation's characteristics), and to produce as an output a discrete label related to those input values.
• input values such as values of an object's features or a situation's characteristics
• output a discrete label related to those input values.
• Face recognition, object tracking, room navigation, medical image analysis, and voice recognition are example applications in which classification is applied.
• Classifiers may be implemented as fixed classifiers or learning classifiers, hi addition, note that the techniques described herein can also be applied to RLS regression problems, as will be appreciated in light of this disclosure.
• Gaussian kernels of the form K[x, X j are considered, where ⁇ is a user- selected bandwidth parameter.
• ⁇ is a user- selected bandwidth parameter.
• LOO can be effectively used as a "gold standard" for cross-validation: it is almost unbiased and the variance associated with a procedure such as 10-fold cross-validation can be quite high, with different partitionings of the dataset yielding substantially different results.
• LOO is less important, but still desirable and can be used as discussed herein.
• one embodiment of the present invention makes RLS more practical by showing how, for several RLS related tasks, the regularization parameter ⁇ can be varied over a wide range of values for very little computational cost, simultaneously building classifiers and computing their LOO values on the training set.
• the choice of ⁇ in a regularization scheme is an important one, as a poor choice can lead to an inaccurate classifier.
• RLS and SVM
• SVM SVM
• the ability to consider a large number of ⁇ values simultaneously translates directly into computational savings. The difference is especially important when RLS is used as a component in a feature selection problem (e.g., face or object recognition), or when it is also necessary to select kernel hyperparameters.
• One methodology described herein begins with, for moderate size problems (where the 0(n 2 ) kernel matrix can be stored and manipulated), finding both the "solution” c and the LOO values for a large number of values of ⁇ with a slowdown of only a small constant factor (about 3) compared to solving for a single ⁇ "directly.” Then, the methodology is used to conduct large-scale experiments on a number of benchmark datasets, where both ⁇ and ⁇ are varied over a wide range, to show via empirical observation that accurate classification can often be obtained using quite large values of ⁇ . Also discussed herein is how to exploit the ability to make ⁇ large to solve large problems, and a number of computational examples.
• Figure 1 is a block diagram illustrating a computing environment 10 configured in accordance with an embodiment of the present invention.
• the computing environment 10 includes a processor 101 operatively coupled via a bus 119 to a memory 107, a storage device 103 for storing an RLS classifier module 105 (and other executable code, such as an operating system and applications), a keyboard 113, a graphics adapter 109, a pointing device 117, and a network adapter 115.
• a display 111 is operatively coupled to the graphics adapter 109.
• the processor 101 may be any processor or set of processors capable of executing various operating systems (e.g., UNIX) and applications/drivers (e.g., MATLAB) of the computing environment 10.
• the memory 107 may be, for example, firmware ROM, RAM, and/or flash memory, and holds instructions and data used by the processor 101 (e.g., 512 MByte or higher).
• the storage device 103 is a hard disk drive in one embodiment (e.g., 10 GByte or higher), but can also be any other device capable of persistently storing data, such as a memory stick and/or a solid-state memory device.
• the storage device 103 can hold multiple files containing executable code and/or data, as is typically done.
• the computing environment 10 operates to load an executable file into memory 107 and execute it using the processor 101.
• the RLS classifier module 105 is stored in storage device 103 as executable code, and is loaded into memory 107 for execution by the processor 101 as one or more processes.
• the files stored on the storage device 103 can be, for example, in the MATLAB format (sometimes referred to as M-files). Other file formats may also be stored on storage device 103, such as .EXE files, .DLL files, .DSfI files and any other file types necessary for the computing environment to properly operate. Such files are typical of computing environments that employ Microsoft operating systems. Other file formats, such as those utilized by Apple Macintosh and UTSfTX based computers will be apparent in light of this disclosure.
• the pointing device 117 may be a mouse, track ball, or other such user input device, and is used in combination with the keyboard 113 to allow the user to interact with the computing environment 10 (e.g.
• the graphics adapter 109 displays images and other information on the display 111.
• the network adapter 115 communicatively couples the computing environment 10 with an external network such as the Internet or LAN, or combination thereof (e.g., via conventional wired or wireless technology), if so desired and as is typically done.
• the computing environment 10 is adapted to execute computer program modules for providing RLS functionality described herein, including those of RLS classifier module 105. Structure and functionality of the RLS classifier module 105 will be discussed in further detail with reference to Figures 2 through 5. As will be apparent in light of this disclosure, the computing environment 10 can be implemented via processing systems such as a general purpose desktop or laptop computer, or a special-purpose processing system configured to carryout the RLS classification (or RLS regression) as described herein, hi one particular case, the computing environment 10 is implemented with an IBM T42 laptop with a 1.7 GHz Pentium-M processor and 2 GBytes of RAM. Other computationally intensive processes can be limited as needed to reserve processing power for RLS processing as discussed herein, although a standard user desktop environment can also be running.
• processing systems such as a general purpose desktop or laptop computer, or a special-purpose processing system configured to carryout the RLS classification (or RLS regression) as described herein, hi one particular case, the computing environment 10 is implemented with an IBM T42 laptop with a 1.7
• FIG. 2 is a block diagram illustrating modules within the RLS classifier/regression module 105 configured in accordance with an embodiment of the present invention.
• module 105 includes an input module 201, a kernel matrix generator 203, an eigendecomposition/SVD module 205, a coefficient (c) computation module 207, a leave-one-out (LOO) error computation module 209, a regularization parameter ( ⁇ ) selection module 211, a prediction module 213, and an output module 215.
• each of these modules is implemented with executable software (e.g., C, C++, or other suitable instruction set) for providing the specified functionality.
• modules can also be implemented in hardware (e.g., gate-level logic), firmware (e.g., microcontroller configured with embedded routines for carrying out each module's functionality), or some combination thereof.
• firmware e.g., microcontroller configured with embedded routines for carrying out each module's functionality
• modules illustrated herein represent one embodiment of the present invention.
• Other embodiments may include additional and/or different modules and functionality.
• other embodiments may lack modules described herein and/or distribute the described tfJL ⁇ functionality among the modules in a different manner.
• the input module 201 may be further configured to format and structure the input dataset as needed (if at all), in preparation for generating the kernel matrix generation of pairwise distances.
• the kernel matrix generator 203 is programmed or otherwise configured to form a kernel matrix (K) of the pairwise distances, where the kernel matrix K is size O(n 2 ). Note that the kernel matrix generator 203 does not apply to the case of a linear kernel, where a singular value decomposition process can be used, in accordance with one embodiment of the present invention, and as will be discussed in turn. [0031] For an RLSC algorithm in accordance with an embodiment of the present invention, the leave-one-out (L ⁇ ) error values are given explicitly via the formula:
• the LOO values can be computed given the training weights c (also referred to as coefficients c; in general, c represents training points on the functional expansion) and the diagonal of the inverse of the regularized kernel matrix G.
• K QKQj
• c( ⁇ ) can be computed in O(n ⁇ ).
• LOO error can also be computed in O(n?) time.
• a single entry of G( ⁇ ) "1 can be computed in 0 ⁇ n) time, as will be discussed with reference to module 207.
• diag(G -1 ) can be computed, and compute LOO error, in ⁇ 9( « 2 ) time.
• the coefficient c computation module 207 is programmed or otherwise configured to receive a collection of regularization parameters ( ⁇ ), and to compute coefficients c for each ⁇ using the matrix decomposition computed by module 205.
• the coefficient c computation module 207 operates to apply the following formulae to compute c for each ⁇ :
• the eigendecomposition/SVD module 205 is programmed or otherwise configured to compute the eigendecomposition of the kernel matrix K. Alternatively, or in addition to
• the eigendecomposition/SVD module 205 can be configured to compute the singular value decomposition (SVD) of the data matrix (the training set of data).
• V(S 2 + AIy 1 V X'Y can be obtained from an eigendecomposition of
• module 207 operates to form c and diag(G -1 ) as previously discussed.
• the leave-one-out (LOO) error computation module 209 is programmed or otherwise configured to receive the collection of regularization parameters
• the diag [G ) regularization parameter ( ⁇ ) selection module 211 is programmed or otherwise configured to then select the ⁇ with the lowest LOO error (e.g., based on comparison of values).
• the prediction module 213 is programmed or otherwise configured to predict future data points based on the RLS function associated (e.g., based on expansion coefficients c or hyperplane function w) associated with the best ⁇ .
• the output module 215 is programmed or otherwise configured to output an RLS prediction based on an input data set.
• the output module 215 may be further configured to format and structure the output prediction data as needed (if at all), in preparation for displaying or otherwise reporting that data.
• the time savings provided by this embodiment is best illustrated with an example, hi more detail, and with reference to Table 1, consider the 4192 point, 14 dimension Galaxy Dim dataset. For both the full dataset and a random subset of half the dataset, the LOO error values were computed using both the conventional direct method and an eigenvalue decomposition in accordance with an embodiment of the present invention, and compare the timings. The timing results reported in Table 1 are approximate (e.g., +/- 5%), and the order of timings that are close may vary accordingly.
• Table 1 Timings (in seconds) for computing LOO values using direct linear algebra on the Galaxy Dim dataset.
• the eigenvalue approach to LOO error computation in accordance with an embodiment of the present invention has an advantage, and this advantage grows as the number of ⁇ s increases.
• the relative advantage of the eigenvalue decomposition grows (for sufficiently many ⁇ s), because the per ⁇ cost is O(n 2 ) rather than O( ⁇ 3 ).
• An empirical approach can be used to performance test the RLS classifier/regression module 105, where two different tests are performed to check the accuracy of the LOO computation.
• the first test is a spot check, where for some small subset of the data, each point x is explicitly removed from the training set, a classifier is built on the rest of the data, and the classifier is tested on x.
• the other check is to perturb the data slightly, where all values (the training data, the y values, ⁇ and ⁇ ) are perturbed by multiplying each number by an independent random variate of the form (1 + e), where e is a Gaussian random variable with mean 0 and variance 10 ⁇ 13 .
• a row of Q can be multiplied by T(T + 7J) '1 and then dot the result by the same row of Q, in O(r) time per element.
• the total time to find the approximate diag(K(K + 7J) ⁇ l ) at the r th iteration given Q and T is O(nr).
• a slightly slower but simpler approach can be used where a full eigendecomposition of T is formed.
• the cost is O(nr + r 3 ), and the difference is not noticeable because r is much smaller than -Jn .
• the Improved Fast Gauss Transform can be used to form (approximate) matrix-vector products Kx, in accordance with one embodiment of the present invention.
• Other approaches such as the Nystrom method, can also be used.
• the IFGT picks a set of centers and computes the matrix-vector product Kx using a sum of truncated Hermite expansions around the centers.
• the IFGT is treated mostly as a black box - it is given a matrix A, a vector x, and a parameter p, and the IFGT returns a "p th order" approximation to Ax.
• the norm of an operator A is defined as consider the relative norms of the operator (K - IFGT(K)) and K; operator norms are computed, for example, by performing 20 iterations of an Arnoldi process to find to high accuracy the largest eigenvalue of the operator.
• Galaxy Dim is especially suited to this example because of its size (4192 points); it is large enough to allow one to reasonably see the asymptotics by playing with a subset of it, but small enough to allow one to compare methods that do and do not need the entire kernel matrix K.
• Table 1 the total time used to compute the LOO values for 25 different ⁇ s was 38.1 seconds for a 2048 point subset of the data, and 311.1 seconds for the entire 4192 point dataset.
• the timing is essentially linear: the IFGT is linear in the dataset size, but slightly more iterations were required for convergence for the larger dataset. Lower order IFGTs use substantially less time per IFGT iteration, but may provide insufficient accuracy. [0051] Now, consider the 8124 point, 22 dimensional dataset Mushroom, for which it is typically not practical to work directly with the entire kernel matrix.
• ⁇ e.g. 10" 16 to 1O +16
• the best ⁇ was computed to be Ie-Il, leading to a LOO accuracy of 88.8%.
• ⁇ e.g. 10" 16 to 1O +16
• Figure 5 illustrates an RLS classification/regression methodology configured in accordance with an embodiment of the present invention. The method can be carried out, for example, by the RLS classification/regression module 105 previously discussed with reference to Figures 1 and 2.
• the method includes receiving 501 a training set of data. It the case of a linear kernel, step 503 is skipped and the method continues to step 505. Otherwise, the method continues with forming 503 a kernel matrix of pairwise distances using the input training set. Note that the method may further include receiving an actual dataset to be processed, once the RLS classifier/regression module is trained as described herein.
• the method continues with computing 505 the eigendecomposition of the kernel matrix, or computing 505 the SVD of the data matrix (in the case of a linear kernel), as previously explained.
• One particular embodiment of the present invention is adapted to carryout both eigendecomposition and SVD (e.g., user-configurable).
• the method further includes receiving 507 a collection of regularization parameters ( ⁇ ).
• ⁇ regularization parameters
• the range and resolution of the ⁇ collection which can be provided manually (e.g., input by user) or automatically (e.g., read from a previously stored filed or hard coded or computed dynamically based on user input specifying a target range and resolution) will depend on factors such as the desired accuracy and robustness of the classifier.
• the collection of ⁇ s ranges from 10" 16 to 1O +16 and includes a resolution of 200 to 400 intermediate ⁇ values evenly spaced therebetween.
• Such a broad range allows for convergence (e.g., ranges from too small to too big) and provides sufficient resolution without requiring excessive computation to achieve that convergence.
• the method continues with computing 509 coefficients (c) for each ⁇ using the matrix decomposition computed in step 505 (whether using eigendecomposition or SVD).
• c U(S 2 + AIy 1 U 1 Y + Z -1 U x U[Y (for the linear kernel case). Note that diag(G ⁇ l ) can also be computed here, as previously discussed.
• the method continues with selecting 513 the ⁇ with the lowest LOO error. As previously explained, this selection of the appropriate ⁇ is not trivial and enables an accurate classifier.
• the method may then proceed with predicting 515 future data points based on the RLS function associated with the best ⁇ (e.g., based on coefficients c or hyperplane function w associated with the best ⁇ ).
• the actual classification application i.e., what is being predicted, such as face, objects, sounds, etc
• regression application i.e., size of number being predicted, such as lifespan, salary, etc

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Evolutionary Computation (AREA)
• General Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• Physics & Mathematics (AREA)
• Software Systems (AREA)
• Evolutionary Biology (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Bioinformatics & Computational Biology (AREA)
• Medical Informatics (AREA)
• Computing Systems (AREA)
• Mathematical Physics (AREA)
• Complex Calculations (AREA)

Priority Applications (1)

Applications Claiming Priority (4)

Publications (2)

Family

ID=37900503

Family Applications (1)

Country Status (3)

Cited By (1)

Families Citing this family (13)

Family Cites Families (4)

• 2006
  • 2006-09-27 US US11/535,921 patent/US7685080B2/en not_active Expired - Fee Related
  • 2006-09-28 JP JP2008533701A patent/JP4635088B2/ja not_active Expired - Fee Related
  • 2006-09-28 WO PCT/US2006/038199 patent/WO2007038765A2/en not_active Ceased

Also Published As

Similar Documents

Legal Events