WO1999040521A1 - Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne - Google Patents

Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne Download PDF

Info

Publication number
WO1999040521A1
WO1999040521A1 PCT/DK1999/000052 DK9900052W WO9940521A1 WO 1999040521 A1 WO1999040521 A1 WO 1999040521A1 DK 9900052 W DK9900052 W DK 9900052W WO 9940521 A1 WO9940521 A1 WO 9940521A1
Authority
WO
WIPO (PCT)
Prior art keywords
weight
cell values
values
training
column
Prior art date
Application number
PCT/DK1999/000052
Other languages
English (en)
Inventor
Thomas Martini JØRGENSEN
Christian Linneberg
Original Assignee
Intellix A/S
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from EP98201910A external-priority patent/EP0935212B9/fr
Application filed by Intellix A/S filed Critical Intellix A/S
Priority to IL13733799A priority Critical patent/IL137337A0/xx
Priority to JP2000530867A priority patent/JP2002503002A/ja
Priority to PL99343114A priority patent/PL343114A1/xx
Priority to CA002318502A priority patent/CA2318502C/fr
Priority to NZ506053A priority patent/NZ506053A/xx
Priority to AU22656/99A priority patent/AU756987B2/en
Publication of WO1999040521A1 publication Critical patent/WO1999040521A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation

Definitions

  • the present invention relates generally to n-tuple or RAM based neural network classification systems and, more particularly, to n-tuple or RAM based classification systems having weight vectors with element values being determined during a training process.
  • a known way of classifying objects or patterns represented by electric signals or binary codes and, more precisely, by vectors of signals applied to the inputs of neural network classification systems lies in the implementation of a so-called learning or training phase.
  • This phase generally consists of the configuration of a classification network that fulfils a function of performing the envisaged classification as efficiently as possible by using one or more sets of signals, called learning or training sets, where the membership of each of these signals in one of the classes in which it is desired to classify them is known.
  • This method is known as supervised learning or learning with a teacher.
  • a subclass of classification networks using supervised learning are networks using memory-based learning.
  • one of the oldest memory-based networks is the "n- tuple network” proposed by Bledsoe and Browning (Bledsoe, W.W. and Browning, I, 1959, "Pattern recognition and reading by machine", Proceedings of the Eastern Joint Computer Conference, pp. 225-232) and more recenly described by Morciniec and Rohwer (Morciniec, M. and Rohwer, R.,1996, "A theoretical and experimental account of n-tuple classifier performance", Neural Comp., pp. 629-642).
  • n- tuple networks which is also known as “RAM networks” or “weightless neural 2 networks”
  • learning may be accomplished by recording features of patterns in a random-access memory (RAM), which requires just one presentation of the training set(s) to the system.
  • RAM random-access memory
  • LUTs Look Up Tables
  • Each LUT may probe a subset of a binary input data vector.
  • the bits to be used are selected at random.
  • the sampled bit sequence is used to construct an address. This address corresponds to a specific entry (column) in the LUT.
  • the number of rows in the LUT corresponds to the number of possible classes. For each class the output can take on the values 0 or 1.
  • a value of 1 corresponds to a vote on that specific class.
  • an input vector is sampled, the output vectors from all LUTs are added, and subsequently a winner takes all decision is made to classify the input vector.
  • the output values may initially be set to 0. For each example in the training set, the following steps should then be carried out:
  • each training pattern By use of such a training strategy it may be guaranteed that each training pattern always obtains the maximum number of votes. As a result such a network makes no misclassification on the training set, but ambiguous decisions may occur.
  • the generalisation capability of the network is directly related to the number of input bits for each LUT. If a LUT samples all input bits then it will act as a pure memory device and no generalisation will be provided. As the number of input bits is reduced the generalisation is increased at an expense of an increasing number of ambiguous decisions. Furthermore, the classification and generalisation performances of a LUT are highly dependent on the actual subset of input bits probed. The purpose of an 3
  • inhibitor This method deals with the problem that in many situations two different classes might only differ in a few of their features. In such a case, an example outside the training set has a high risk of sharing most of its features with an incorrect class. So, in order to deal with this problem it becomes necessary to weight different features differently for a given class. Thus, a method is suggested where the network includes inhibition factors for some classes of the addressed columns. Here, a confidence measure is introduced, and the inhibition factors are calculated so that the confidence after inhibition corresponds to a desired level.
  • the result of the preferred inhibition scheme is that all addressed LUT cells or elements that would be set to 1 in the simple system are also set to 1 in the modified version, but in the modified version column cells being set to 1 may further comprise information of the number of times the cell has been visited by the training set. However, some of the cells containing 0's in the simple system will have their contents changed to negative values in the modified network. In other words, the 4 conventional network is extended so that inhibition from one class to another is allowed.
  • the cells of the LUTs are given different values which might be considered a sort of "weighting". However, it is only cells which have not been visited by the training set that are allowed to be suppressed by having their values changed from 0 to a negative value. There is no boosting of cells having positive values when performing classification of input data. Thus, very well performing LUTs or columns of LUTs might easily drown when accompanied by the remaining network.
  • a method for training a computer classification system which can be defined by a network comprising a number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of possible classes and further comprising a number of columns being addressed by signals or elements of sampled training input data examples, each column being defined by a vector having cells with values, said method comprising determining the column 5 vector cell values based on one or more training sets of input data examples for different classes so that at least part of the cells comprise or point to information based on the number of times the corresponding cell address is sampled from one or more sets of training input examples, and determining weight cell values corresponding to one or more column vector cells being addressed or sampled by the training examples.
  • LUTs Look Up Tables
  • a method of determining weight cell values in a computer classification system which can be defined by a network comprising a number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of possible classes and further comprising a number of column vectors with at least part of said column vectors having corresponding weight vectors, each column vector being addressed by signals or elements of a sampled training input data example and each column vector and weight vector having cells with values being determined based on one or more training sets of input data examples for different classes, said method comprising determining the column vector cell values based on the training set(s) of input examples so that at least part of said values comprise or point to information based on the number of times the corresponding cell address is sampled from the set(s) of training input examples, and determining weigiit vector cell values corresponding to one or more column vector cells.
  • LUTs Look Up Tables
  • the weight cell values are determined based on the information of at least part of the determined column vector cell values and by use of at least part of the training set(s) of input examples.
  • the training input data examples may preferably be presented to the network as input signal vectors.
  • determination of the weight cell values is performed so as to allow weighting of one or more column vectors cells of positive value and/or to allow boosting of one or more column vector cells during a classification process.
  • the weight cell values may be determined so as to allow suppressing of one or more column vector cells during a classification process. 6
  • the present invention also provide a method wherein the determination of the weight cell values allows weighting of one or more column vector cells having a positive value (greater than 0) and one or more column vector cells having a non- positive value (lesser than or equal to 0).
  • the determination of the weight cells allows weighting of any column vector cell.
  • the determination of these values may comprise initialising one or more sets of weight cells corresponding to at least part of the column cells, and adjusting at least part of the weight cell values based on the information of at least part of the determined column cell values and by use of at least part of the training set(s) of input examples.
  • the weight cell values it is preferred that these are arranged in weight vectors corresponding to at least part of the column vectors.
  • the column cell values should be determined.
  • at least part of the column cell values are determined as a function of the number of times the corresponding cell address is sampled from the set(s) of training input examples.
  • the information of the column cells may be determined so that the maximum column cell value is 1 , but at least part of the cells have an associated value being a function of the number of times the corresponding cell address is sampled from the training set(s) of input examples.
  • the column vector cell values are determined and stored in storing means before the adjustment of the weight vector cell values.
  • a preferred way of determining the column vector cell values may comprise the training steps of
  • the present invention also covers embodiments where the information of the column cells is determined by alternative functions of the number of times the cell has been addressed by the input training set(s).
  • the cell information does not need to comprise a count of all the times the cell has been addressed, but may for example comprise an indication of when the cell has been visited zero times, once, more than once, and/or twice and more than twice and so on.
  • weight cell values may be determined for one or more column cells, but in a preferred embodiment all column vectors have corresponding weight vectors.
  • the initialisation may comprise setting each weight cell value to a predetermined specific cell value. These values may be different for different cells, but all weight cell values may also be set to a predetermined constant value. Such a value may be 0 or 1 , but other values may be preferred.
  • the adjustment of the weight cell values may comprise the steps of determining a global quality value based on at least part of the weight and column vector cell values, determining if the global quality value fulfils a required quality criterion, and adjusting at least part of the weight cell values until the global quality criterion is fulfilled.
  • the adjustment process may also include determination of a local quality value for each sampled training input example, with one or more weight cell adjustments being performed if the local quality value does not fulfil a specified or required local 8 quality criterion for the selected input example.
  • the adjustment of the weight cell values may comprise the steps of
  • steps (b)-(d) of the above mentioned adjustment process may be carried out for all examples of the training set(s).
  • the local and/or global quality value may be defined as functions of at least part of the weight and/or column cells.
  • the global and/or the local quality criterion may also be functions of the weight and/or column cells.
  • the quality criterion or criteria need not be a predetermined constant threshold value, but may be changed during the adjustment iteration process.
  • the present invention 9 also covers embodiments in which the quality criterion or criteria is/are given by constant threshold values.
  • the adjusted weight cell values are preferably stored after each adjustment, and when the adjustment process includes the determination of a global quality value, the step of determination of the global quality value may further be followed by separately storing the hereby obtained weight cell values or classification system configuration values if the determined global quality value is closer to fulfil the global quality criterion than the global quality value corresponding to previously separately stored weight cell values or configuration values.
  • a main reason for training a classification system according to an embodiment of the present invention is to obtain a high confidence in a subsequent classification process of an input example of an unknown class.
  • the parameters used for determining the output value includes both values of addressed weight cells and addressed column cells.
  • the output value may be determined as a first summation of all the addressed weight cell values corresponding to column cell values greater than or equal to a predetermined value.
  • the step of determining an output value comprises determining a first summation of all the addressed weight cell values corresponding to column cell values greater than or equal to a predetermined value, determining a second summation of all the addressed weight cell values, and determining the output value by dividing the first summation by the second summation.
  • the predetermined value may preferably be set to 1.
  • the present invention also provides training and classification systems according to the above described methods of training and classification.
  • a system for training a computer classification system which can be defined by a network comprising a stored number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of possible classes and further comprising a number of columns being addressed by signals or elements of sampled training input data examples, each column being defined by a vector having cells with values, said system comprising input means for receiving training input data examples of known classes, means for sampling the received input data examples and addressing column vectors in the stored set of n-tuples or LUTs, 1 1 means for addressing specific rows in the set of n-tuples or LUTs, said rows corresponding to a known class, storage means for storing determined n-tuples or LUTs, means for determining column vector cell values so as to comprise or point to information based on the number of times the corresponding cell address is sampled from the training set(s) of input examples,
  • the present invention also provides a system for determining weight cell values of a classification network which can be defined by a stored number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of the number of possible classes and further comprising a number of column vectors with at least part of said column vectors having corresponding weight vectors, each column vector being addressed by signals or elements of a sampled training input data example and each column vector and weight vector having cell values being determined during a training process based on one or more sets of training input data examples, said system comprising: input means for receiving training input data examples of known classes, means for sampling the received input data examples and addressing column vectors and corresponding weight vectors in the stored set of n-tuples or LUTs, means for addressing specific rows in the set of n-tuples or LUTs, said rows corresponding to a known class, storage means for storing determined n-tuples or LUTs,
  • the means for determining the weight cell values is adapted to determine these values based on the information of at least part of the determined column vector cell values and by use of at least part of the training set(s) of input examples.
  • the means for determining the weight cell values is adapted to determine these values so as to allow weighting of one or more column cells of positive value 12 and/or to allow boosting of one or more column cells during a classification process.
  • the determining means may furthermore, or alternatively, be adapted to determine the weight cell values so as to allow suppressing of one or more column vector cells during a classification process.
  • the weight determining means may be adapted to determine the weight cell values so as to allow weighting of one or more column vector cells having a positive value (greater than 0) and one or more column vector cells having a non-positive value (lesser than or equal to 0).
  • the means may further be adapted to determine the weight cell values so as to allow weighting of any column cell. It is also preferred that the means for determining the weight cell values is adapted to determine these values so that the weight cell values are arranged in weight vectors corresponding to at least part of the column vectors.
  • the means for determining the weight cell values may comprise means for initialising one or more sets of weight vectors corresponding to at least part of the column vectors, and means for adjusting weight vector cell values of at least part of the weight vectors based on the information of at least part of the determined column vector cell values and by use of at least part of the training set(s) of input examples.
  • the column cell values should be determined in order to determine the weight cell values.
  • the means for determining the column vector cell values is adapted to determine these values as a function of the number of times the corresponding cell address is sampled from the set(s) of training input examples.
  • the means for determining the column vector cell values may be adapted to determine these cell values so that the maximum value is 1 , but at least part of the cells have an associated value being a function of the number of times the corresponding cell address is sampled from the training set(s) of input examples.
  • the means for determining the column vector cell values is adapted to increment the value or vote of the cells of the addressed column vector(s) corresponding to the row(s) of the known class, said value preferably being incremented by one.
  • the means for initialising the weight vectors is adapted to setting the weight cell values to one or more predetermined values.
  • the means for adjusting the weight vector cell values is adapted to determine a global quality value based on at least part of the weight and column vector cell values, determine if the global quality value fulfils a required global quality criterion, and adjust at least part of the weight cell values until the global quality criterion is fulfilled.
  • the means for adjusting the weight vector cell values may be adapted to
  • a) determine a local quality value corresponding to a sampled training input example, the local quality value being a function of at least part of the addressed weight and column vector cell values,
  • the means for adjusting the weight vector cell values may further be adapted to stop the iteration process if the global quality criterion is not fulfilled after a given number of iterations.
  • the means for storing n-tuples or LUTs comprises means for storing adjusted weight cell values and separate means for storing best so far weight cell values or best so far classification system configuration values.
  • the means for adjusting the weight vector cell values may further be adapted to replace previously separately stored best so far weight cell values with obtained adjusted weight cell values if the determined global quality value is closer to fulfil the global quality criterion than the global quality value corresponding to previously separately stored best so far weight values.
  • the system may always comprise the "best so far" system configuration.
  • a system for classifying input data examples of unknown classes into at least one of a plurality of classes comprising: storage means for storing a number or set of n-tuples or Look Up Tables (LUTs) with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of the number of possible classes and further comprising a number of column vectors with corresponding weight vectors, each column vector being addressed by signals or elements of a sampled input data example and each column vector and weight vector having cells with values being determined during a training process based on one or more sets of training input data examples, said system further comprising: input means for receiving an input data example to be classified, means for sampling the received input data example and addressing columns and corresponding weight vectors in the stored set of n-tuples or LUTs, means for addressing specific rows in the set of n-tuples or LUTs, said rows corresponding to a specific class, means for
  • the output determining means comprises means for producing a first summation of all the addressed weight vector cell values corresponding to a specific class and corresponding to column vector cell values greater than or equal to a predetermined value. It is also preferred that the output determining means further comprises means for producing a second summation of all the addressed weight vector cell values corresponding to a specific class, and means for determining the output value by dividing the first summation by the second summation.
  • the cell values of the column and weight vectors of the classification system according to the present invention are determined by use of a training system according to any of the above described systems. Accordingly, these cell values may be determined during a training process according to any of the above described methods.
  • Fig. 1 shows a block diagram of a RAM classification network with Look Up Tables (LUTs),
  • Fig. 2 shows a detailed block diagram of a single Look Up Table (LUT) according to an embodiment of the present invention
  • Fig. 3 shows a block diagram of a computer classification system according to the present invention
  • Fig. 4 shows a flow chart of a learning process for LUT column cells according to an embodiment of the present invention
  • Fig. 5 shows a flow chart of a learning process for weight cells according to a first embodiment of the present invention
  • Fig. 6 shows a flow chart of a learning process for weight cells according to a second embodiment of the present invention.
  • Fig. 7 shows a flow chart of a classification process according to the present invention.
  • N x Number of examples in the training set X . x : The j'th example from a given ordering of the training set X . y : A specific example (possible outside the training set).
  • C(.v) Class label corresponding to example .v (the true class).
  • C R Runner Up Class obtained by classification.
  • A(.x) Leave-one-out cross-validation classification for example x .
  • N c Number of training classes corresponding to the maximum number of rows in a LUT. 17
  • N ⁇ ⁇ Number of LUTs.
  • N C0l Number of different columns that can be addressed in a specific LUT
  • S c The set of training examples labelled class C.
  • w ⁇ C Weight for the cell addressed by the i'th column and the C'th class.
  • v. r Entry counter for the cell addressed by the i'th column and the C'th class.
  • a, ( ) Index of the column in the i'th LUT being addressed by example y .
  • v Vector containing all v If . elements of the LUT network.
  • Fig. 1 shows a block diagram of a RAM classification network with Look Up Tables (LUTs), and Fig. 2, which shows a detailed block diagram of a single Look Up Table (LUT) according to an embodiment of the present invention.
  • a RAM-net or LUT-net consists of a number of Look Up Tables (LUTs) (1.3). Let the number of LUTs be denoted N LUT .
  • An example of an input data vector y to be classified may be presented to an input module (1.1) of the LUT network. Each LUT may sample a part of the input data, where different numbers of input signals may be sampled for different LUTs (1.2) (in principle it is also possible to have one LUT sampling the whole input space).
  • the outputs of the LUTs may be fed (1.4) to an output module (1.5) of the RAM classification network. 18
  • Fig. 2 it is shown that for each LUT the sampled input data (2.1) of the example presented to the LUT-net may be fed into an address selecting module (2.2).
  • the address selecting module (2.2) may from the input data calculate the address of one or more specific columns (2.3) in the LUT. As an example, let the index of the column in the i'th LUT being addressed by an input example y be calculated as
  • a ( (y) The number of addressable columns in a specific LUT may be denoted N C0I , and varies in general from one LUT to another.
  • the information stored in a specific row of a LUT may correspond to a specific class C (2.4).
  • the maximum number of rows may then correspond to the number of classes, N c .
  • every column within a LUT contains two sets of cells. The number of cells within each set corresponds to the number of rows within the LUT.
  • the first set of cells may be denoted column vector cells and the cell values may correspond to class specific entry counters of the column in question.
  • the other set of cells may be denoted weight cells or weight vector cells with cell values which may correspond to weight factors, each of which may be associated with one entry counter value or column vector cell value.
  • the entry counter value for the cell addressed by the i'th column and class C is denoted v., (2.5).
  • the weight value for the cell addressed by the i'th column and class C is denoted ⁇ r, ( (2.6).
  • the v /( - and w ⁇ ( . -values of the activated LUT columns (2.7) may be fed (1 .4) to the output module (1.5), where a vote number may be calculated for each class and where finally a winner-takes-all (WTA) decision may be performed.
  • WTA winner-takes-all
  • VoteNo(C, ) i « ⁇ c/ o/j(v ⁇ ( - ) ⁇ C , »v II ⁇ ( T) ⁇ C , v ⁇ 2 ( - )-£ ., ⁇ t- ⁇ j ( . ) ⁇ C v a , ⁇ rr i ⁇ > c> w « , l ⁇ r i - .c)
  • VoteNo(C ) An example of a sensible choice of VoteNo(C ) is the following expression:
  • ⁇ describes the set of LUTs making up the whole LUT network.
  • S r denotes the set of training examples labelled class C.
  • the special case with all w ⁇ c -values set to 1 gives the traditional LUT network,
  • Figure 3 shows an example of a block diagram of a computer classification system according to the present invention.
  • a source such as a video camera or a database provides an input data signal or signals (3.0) describing the example to be classified.
  • These data are fed to a pre-processing module (3.1 ) of a type which can extract features, reduce, and transform the input data in a predetermined manner.
  • An example of such a pre-processing module is a FFT-board (Fast Fourier Transform).
  • the transformed data are then fed to a classification unit (3.2) comprising a RAM network according to the present invention.
  • the classification unit (3.2) outputs a ranked classification list which might have associated confidences.
  • the classification unit can be implemented by using software to programme a 20 standard Personal Com puter or programming a hardware device, e.g. using programmable gate arrays combined with RAM circuits and a digital signal processor. These data can be interpreted in a post-processing device (3.3), which could be a computer module combining the obtained classifications with other relevant information. Finally the result of this interpretation is fed to an output device (3.4) such as an actuator.
  • a post-processing device 3.3
  • an output device such as an actuator.
  • the flow chart of Fig. 4 illustrates a one pass learning scheme or process for the determination of the column vector entry counter or cell distribution, v, r -distribution
  • step 4 for the remaining LUTs (4.7, 4.8).
  • the network may be used for classifying an unknown input data example.
  • the classification is performed by determining the class having a maximum vote number, VoteNo, where VoteNo is given by the expression
  • VoteNo can be defined to be 0.
  • FIG. 7 shows a block diagram of the operation of a computer classification system in which a classification process (7.0) is performed.
  • the system acquires one or more input signals (7.1 ) using e.g. an optical sensor system.
  • the obtained input data are pre-processed (7.2) in a pre-processing module, e.g. a low-pass filter, and presented to a classification module (7.3) which according to an embodiment of the invention may be a LUT-network.
  • the output data from the classification module is then post-processed in a post-processing module (7.4), e.g. a CRC algorithm calculating a cyclic redundancy check sum, and the result is forwarded to an output device (7.5), which could be a monitor screen.
  • 22 Weight adjustments e.g. a CRC algorithm calculating a cyclic redundancy check sum
  • the initially determined weight cell values will not present the optimal choice of values.
  • an optimisation or adjustment of the weight values should be performed.
  • a local quality function Q (v, w,x, X) may be defined, where v " denotes a vector containing all v r elements of the LUT network, and ⁇ v denotes a vector containing all w l( . elements of the LUT network.
  • the local quality function may give a confidence measure of the output classification of a specific example x . If the quality value does not satisfy a given criterion (possibly dynamically changed during the iterations), the weights ⁇ 7 are adjusted to make the quality value satisfy or closer to satisfying the criterion (if possible).
  • Q G (v, ⁇ v, X)
  • the global quality function may measure the performance of the input training set as a whole.
  • Fig. 5 shows a flow chart for weight cell adjustment or learning according to the present invention.
  • the flow chart of Fig. 5 illustrates a more general adjustment or learning process, which may be simplified for specific embodiments.
  • a leave-one-out cross-validation classification for an input example x of the training set may be calculated as: 23
  • This global quality function measures the number of examples from the training set
  • the global quality criterion may be to satisfy Q c > ⁇ N x , where £ is a parameter determining the fraction of training examples demanded to be correctly classified in a leave-one-out crossvalidation test.
  • An updating scheme for improving Q c can be implemented by the following rules: 24
  • k is a small constant.
  • a feasible choice of k could be one tenth of the mean of the absolute values of the w l( . values.
  • example 1 may be described as:
  • the parameter / runs over the possible values of r. r , 0 ⁇ r. r ⁇ N x .
  • a confidence Conj between the winning class, C,,., and the runner-up class C ⁇ may then be defined as:
  • a value m may be determined by the function:
  • n can vary from 1 to the maximum v, r value within the v vector.
  • a local quality function may now be defined as:
  • n ⁇ thmh is a threshold constant. If Q, ⁇ 0 then the weights ⁇ > are updated to make Q, increase, by adjusting the weights on the runner-up class, C R :
  • the global quality criterion may be based on two quality functions:
  • ⁇ 0 (Q,) is 1 if Q, ⁇ O and 0 if 2 ⁇ 0.
  • ⁇ , and ⁇ are two parameters determining the fractions of training examples demanded to pass a leave-one-out crossvalidation test and the local quality criterion, respectively. If both of these criterions are passed the global quality criterion is passed in which case Q a (v, w, X) is 1, otherwise it is 0. 27
  • example 2 may be described as:
  • a local quality function Q L (v, w,x, X) is defined as a measure of a vote confidence for an input training example .v .
  • the confidence ConJ between the true class, C(x), and the runner-up class C R may be determined as:
  • the confidence can be zero stating that the runner up class has a vote level equal to that of the true class (if one or more classes have a vote level equal to that of the 28 true class we will define one of the classes different from the true one as the runner up class).
  • the local quality function may now be defined as:
  • a threshold value may be determined for the calculated local quality value and if Q L ⁇ Qi hrcsh t tnen tne we ig nts are updated to make Q, increase.
  • Q t mh ., 1 . 1 would De 0-1 stating that the difference between the vote level of the true and that of the runner up class should at least be 10% of the maximum vote level.
  • the weights may be updated by adjusting the weights for the runner-up class, C ⁇ :
  • the small constant k determines the relative change in the weights to be adjusted.
  • the global quality criterion may be to satisfy Q c > ⁇ N x , where ⁇ is a parameter determining the fraction of training examples demanded to be correctly classified in a leave-one-out crossvalidation test.
  • VoteNo cv (C,.v) /e ⁇
  • runner-up class obtained using VoteNo(C y) may be denoted C R (if one or more classes have a vote level equal to that of the true class we will define one of the classes different from the true one as the runner up class).
  • the local quality function Q L (v, w, x, X) may now be defined by a Boolean expression.
  • the two criteria corresponds to demanding the vote level of the true class in a leave-one-out cross-validating test to be larger than A, and the vote level of the runner up class to be below A, with level A, being larger than level A 2 .
  • the VoteNo() function used in this example will have value between 0 and 1 if we restrict the weights to have positive value in which case a possible choice of k values are A, equal to 0.9 and A, equal to 0.6.
  • ⁇ ,. ⁇ , max( ⁇ OT) + * 4 [2 ⁇ 1 0 ( ⁇ ) ) - l] , ⁇ ).
  • a 3 determines the relative change in the weights to be adjusted for the runner up class.
  • One possible choice would be A 3 is 0.1.
  • a feasible choice of A 4 could be one tenth of the mean of the absolute values of the w l( . values.
  • a suitable global quality function may be defined by the summation of the local quality values for all the training input examples:
  • the global quality criterion may be to satisfy Q ⁇ > ⁇ N x , where f is a parameter determining the fraction of training examples demanded to pass the local quality test.
  • VoteNo(C, ) ⁇ U
  • the local quality function and the threshold criteria is now defined so that the answer to the question "is Qlocal OK" will always be no.
  • the weight updating rule is:
  • the global quality function for the a"' iteration may be defined as:
  • ⁇ (.v, ⁇ ) argmax ⁇ / admir ⁇ ,, T ⁇ >.C) ⁇ I + .5 I , . . .I>.C) . c Vie ⁇ /
  • a ⁇ l ⁇ X for setting the ⁇ r, r values may be defined as follows (6.1 , 6.6, 6.7):
  • a global quality function for the classification performance may be calculated (6.3).
  • One possibility for the global quality function is to calculate a cross-validation error:
  • ⁇ (.v, ⁇ , ) arg cm 1 ax c ⁇ + ⁇ V, ⁇ ,. ⁇ V n, ( f ) .c j®l+ ⁇ 5,- ( r ) ⁇ V ", ( ⁇ ) .c j
  • the network having the best quality value Q G may then be stored (6.4, 6.5).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Facsimile Image Signal Circuits (AREA)

Abstract

L'invention concerne un système et un procédé d'apprentissage d'un système de classification informatique qui peut être défini par un réseau comprenant un certain nombre de n-lignes ou des tableaux de recherche (LUT), chaque n-ligne ou LUT comprenant un certain nombre de rangées correspondant à au moins un sous-ensemble de classes possibles et comprenant des colonnes adressées par des signaux ou éléments d'exemples échantillonnés de données constituant les entrées d'apprentissage, chaque colonne étant définie par un vecteur présentant des cellules dotées de valeur. Ce procédé consiste à déterminer les valeurs de cellules du vecteur définissant la colonne en fonction d'un ou plusieurs ensemble(s) d'apprentissage des exemples de données constituant les entrées pour différentes classes de manière qu'au moins une partie des cellules comprennent ou signalent des informations basées sur le nombre de fois que l'adresse des cellules correspondante est échantillonnée à partir d'un ou plusieurs ensemble(s) d'exemples des entrées d'apprentissage et à déterminer les valeurs des cellules de pesage correspondant à une ou plusieurs cellule(s) de vecteur de colonne adressée(s) ou échantillonnée(s) par les exemples d'apprentissage afin de permettre le pesage d'une ou plusieurs cellules du vecteur définissant la colonne positives au cours de la classification, lesdites valeurs de pesage étant déterminées en fonction des informations d'au moins une partie des valeurs des cellules du vecteur définissant la colonne déterminées et à l'aide d'au moins une partie de l'ensemble ou des ensembles d'apprentissage d'exemple d'entrées. Selon une autre variante, l'invention concerne un système et un procédé permettant de déterminer, dans un système de classification neuronal, les valeurs des cellules de pesage correspondant à une ou plusieurs cellules de vecteur déterminées adressées par les exemples d'apprentissage. Cette détermination s'appuie sur les informations d'au moins une partie des valeurs des cellules de vecteur déterminées et elle permet de peser les cellules du vecteur définissant la colonne dont la valeur est positive ou négative. Finalement l'invention concerne un procédé et un système de classification des exemples de données constituant les entrées en une pluralité de classes au moyen de systèmes de classification informatiques.
PCT/DK1999/000052 1998-02-05 1999-02-02 Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne WO1999040521A1 (fr)

Priority Applications (6)

Application Number Priority Date Filing Date Title
IL13733799A IL137337A0 (en) 1998-02-05 1999-02-02 Network system
JP2000530867A JP2002503002A (ja) 1998-02-05 1999-02-02 Ｎ−タプル又はｒａｍベースのニューラルネットワーク分類システム及び方法
PL99343114A PL343114A1 (en) 1998-02-05 1999-02-02 N-tuple or ram based neural network classification system and method
CA002318502A CA2318502C (fr) 1998-02-05 1999-02-02 Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne
NZ506053A NZ506053A (en) 1998-02-05 1999-02-02 N-tuple or ram based neural network classification system which boosts cells having positive column vector cell values and corresponding n-tuples
AU22656/99A AU756987B2 (en) 1998-02-05 1999-02-02 N-tuple or ram based neural network classification system and method

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
DK0162/98 1998-02-05
DK16298 1998-02-05
EP98201910A EP0935212B9 (fr) 1998-02-05 1998-06-09 Système et procédé de classification avec un réseau neuronal basé sur N-Tuple ou RAM
EP98201910.1 1998-06-09

Publications (1)

Publication Number Publication Date
WO1999040521A1 true WO1999040521A1 (fr) 1999-08-12

Family

ID=26063433

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/DK1999/000052 WO1999040521A1 (fr) 1998-02-05 1999-02-02 Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne

Country Status (8)

Country Link
JP (1) JP2002503002A (fr)
CN (1) CN1227608C (fr)
AU (1) AU756987B2 (fr)
CA (1) CA2318502C (fr)
IL (1) IL137337A0 (fr)
NZ (1) NZ506053A (fr)
PL (1) PL343114A1 (fr)
WO (1) WO1999040521A1 (fr)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6995629B2 (ja) * 2018-01-05 2022-01-14 日本電信電話株式会社 演算回路
CN110163334B (zh) * 2018-02-11 2020-10-09 上海寒武纪信息科技有限公司 集成电路芯片装置及相关产品
CN110197274B (zh) * 2018-02-27 2020-08-25 上海寒武纪信息科技有限公司 集成电路芯片装置及相关产品
CN110197264B (zh) * 2018-02-27 2020-08-04 上海寒武纪信息科技有限公司 神经网络处理器板卡及相关产品
CN110197275B (zh) * 2018-02-27 2020-08-04 上海寒武纪信息科技有限公司 集成电路芯片装置及相关产品
CN110197267B (zh) * 2018-02-27 2020-08-04 上海寒武纪信息科技有限公司 神经网络处理器板卡及相关产品

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1992000572A1 (fr) * 1990-06-29 1992-01-09 University College London Dispositifs de traitement neural avec possibilites d'apprentissage

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1992000572A1 (fr) * 1990-06-29 1992-01-09 University College London Dispositifs de traitement neural avec possibilites d'apprentissage

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ALEKSANDER I, CLARKE T J W, BRAGA A P: "BINARY NEURAL SYSTEMS: COMBINING WEIGHTED AND WEIGHTLESS PROPERTIES", INTELLIGENT SYSTEMS ENGINEERING., INSTITUTION OF ELECTRICAL ENGINEERS., STEVENAGE., GB, 21 December 1994 (1994-12-21), GB, pages 211 - 221, XP002941536, ISSN: 0963-9640 *
JOERGENSEN T M, CHRISTENSEN S S, LIISBERG C: "CROSS-VALIDATION AND INFORMATION MEASURES FOR RAM BASED NEURAL NETWORKS", PROGRESS IN NEURAL PROCESSING, XX, XX, vol. 09, 1 February 1998 (1998-02-01), XX, pages 78 - 88, XP002941537 *
JORGENSEN T. M.: "CLASSIFICATION OF HANDWRITTEN DIGITS USING A RAM NEURAL NET ARCHITECTURE.", INTERNATIONAL JOURNAL OF NEURAL SYSTEMS, SINGAPORE, SG, vol. 08., no. 01., 1 February 1997 (1997-02-01), SG, pages 17 - 25., XP000881137, ISSN: 0129-0657, DOI: 10.1142/S0129065797000045 *

Also Published As

Publication number Publication date
IL137337A0 (en) 2001-07-24
CN1290367A (zh) 2001-04-04
AU2265699A (en) 1999-08-23
CA2318502C (fr) 2008-10-07
JP2002503002A (ja) 2002-01-29
NZ506053A (en) 2003-02-28
CN1227608C (zh) 2005-11-16
AU756987B2 (en) 2003-01-30
CA2318502A1 (fr) 1999-08-12
PL343114A1 (en) 2001-07-30

Similar Documents

Publication Publication Date Title
US11531900B2 (en) Imitation learning for machine learning systems with synthetic data generators
Hazlett Kernel balancing
WO2019067960A1 (fr) Développement agressif à l'aide de générateurs coopératifs
Bojer et al. Relevance determination in Learning Vector Quantization.
CN107729999A (zh) 考虑矩阵相关性的深度神经网络压缩方法
Patel et al. Multi-class uncertainty calibration via mutual information maximization-based binning
Sangari et al. Convergence analysis of two loss functions in soft-max regression
Agapitos et al. A survey of statistical machine learning elements in genetic programming
Trabelsi et al. Classification systems based on rough sets under the belief function framework
Tripoliti et al. Modifications of the construction and voting mechanisms of the random forests algorithm
Bohdal et al. Meta-calibration: Learning of model calibration using differentiable expected calibration error
CN112288137A (zh) 一种计及电价和Attention机制的LSTM短期负荷预测方法及装置
Abellan et al. Imprecise classification with credal decision trees
US6393413B1 (en) N-tuple or RAM based neural network classification system and method
AU753822B2 (en) N-tuple or ram based neural network classification system and method
WO1999040521A1 (fr) Procede et systeme de classification dans un reseau neuronal oriente ram ou n-ligne
Domanski et al. Orsa: Outlier robust stacked aggregation for best-and worst-case approximations of ensemble systems
Sweidan et al. Probabilistic Prediction in scikit-learn
Zenkov Estimating the Probability of a Class at a Point by the Approximation of one Discriminant Function
Wilkowski et al. Application of artificial neural networks for real estate valuation
Drew Deep unsupervised domain adaptation for gamma-hadron separation
Wiewel et al. Dirichlet prior networks for continual learning
Utkin et al. Robust regression random forests by small and noisy training data
Sorjamaa et al. Sparse linear combination of SOMs for data imputation: Application to financial database
Wang et al. Function approximation using robust fuzzy-GreyCMAC method

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 99802761.8

Country of ref document: CN

AK Designated states

Kind code of ref document: A1

Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT UA UG UZ VN YU ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
ENP Entry into the national phase

Ref document number: 2318502

Country of ref document: CA

Ref document number: 2318502

Country of ref document: CA

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 137337

Country of ref document: IL

WWE Wipo information: entry into national phase

Ref document number: 22656/99

Country of ref document: AU

WWE Wipo information: entry into national phase

Ref document number: IN/PCT/2000/00231/MU

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 506053

Country of ref document: NZ

NENP Non-entry into the national phase

Ref country code: KR

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
WWG Wipo information: grant in national office

Ref document number: 22656/99

Country of ref document: AU