EP0484344A1 - Reconnaissance des structures - Google Patents

Reconnaissance des structures

Info

Publication number
EP0484344A1
EP0484344A1 EP90909478A EP90909478A EP0484344A1 EP 0484344 A1 EP0484344 A1 EP 0484344A1 EP 90909478 A EP90909478 A EP 90909478A EP 90909478 A EP90909478 A EP 90909478A EP 0484344 A1 EP0484344 A1 EP 0484344A1
Authority
EP
European Patent Office
Prior art keywords
weight
input
vector
weights
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
EP90909478A
Other languages
German (de)
English (en)
Inventor
Philip Charles Woodland
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
British Telecommunications PLC
Original Assignee
British Telecommunications PLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by British Telecommunications PLC filed Critical British Telecommunications PLC
Publication of EP0484344A1 publication Critical patent/EP0484344A1/fr
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/24317—Piecewise classification, i.e. whereby each classification requires several discriminant rules
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/16—Speech classification or search using artificial neural networks

Definitions

  • This invention relates to pattern recognition apparatus or the like using neural networks, and a method of producing such apparatus; in particular, but not exclusively, using neural networks of the multi-layer perceptron (MLP) type.
  • MLP multi-layer perceptron
  • Neural networks of this type in general comprise a plurality of parallel processing units ("neurons"), each connected to receive a plurality of inputs comprising an input vector, each input being connected to one of a respective plurality of weighting units, the weighting factors of which comprise a respective weight vector, the output of the neuron being a scalar function of the input vector and the weight vector.
  • the output is a function of the sum of the weighted inputs.
  • the perceptron illustrated in Figure 1 consists of simple processing units (neurons) arranged in layers connected together via 'weights* (synapses).
  • the output of each unit in a layer is the weighted sum of the outputs from the previous layer.
  • the values of these weights are adjusted so that a pattern on the output layer is 'recognised* by a particular set of output units being activated above a threshold.
  • interest in perceptrons faded in the 1960s, and did not revive again until the mid 1980s, when two innovations gave perceptrons new potential.
  • the first was the provision of a non-linear compression following each neuron, which had the effect that the transformation between layers was also be non-linear. This meant that, in theory at least, such a device was capable of performing complex, non-linear mappings.
  • the second innovation was the invention of a weight-adjustment algorithm known as the 'generalised delta rule*.
  • the new perceptron could have as many layers as necessary to perform its complicated mappings.
  • HLP multi-layer perceptron
  • the generalised delta rule enabled it to learn patterns by a simple error back propagation training process.
  • a pattern to be learned is supplied and latched ('clamped') to the input units of the device, and the corresponding required output is presented to the output units.
  • the weights, which connect input to output via the multiple layers, are adjusted so that the error between the actual and required output is reduced.
  • the standard back propagation training algorithm for these networks employs a gradient descent algorithm with weights and biases adjusted by an amount proportional to.
  • the gradient of the error function with respect to each weight is known as the learning rate.
  • a 'momentum' term is also usually added that smooths successive weight updates by adding a constant proportion of the previous weight update to the current one.
  • an alternative algorithm computes a variable learning rate and momentum smoothing. The adaptation scheme ensures that steep gradients do not cause excessively large steps in weight space, but still permits reasonable step sizes with small gradients. This process is repeated many times for all the patterns in the training set. After an appropriate number of iterations, the MLP will recognise the patterns in the training set. If the data is structured, and if the training set is representative, then the MLP will also recognise patterns not in the training set.
  • the network must learn an underlying mapping from input patterns to output patterns by using a sparse set of examples (training data).
  • This mapping should also be applicable to previously unseen data, i.e. the network should generalise well. This is especially important for pattern classification systems in which the data forms natural clusters in the input feature space, such as speech data.
  • generalisation is defined as the difference between the classification performance on the training data set and that on a test set drawn from a population with the same underlying statistics. Why does a given network fail to generalise? A net is specified by a set of parameters that must be learned from a set of training examples. If the amount of training data available is increased, the better in general will be the weight estimates and the more likely a net will generalise.
  • the standard algorithm used for training multi-layer perceptrons is the error back-propagation algorithm discussed above.
  • the algorithm adjusts network weight and bias values so as to reduce the sum-squared error between the actual network output and some desired output value.
  • the correct output is a vector that represents a particular class.
  • the same class label will be assigned to an input vector whether or not it is close to a class boundary.
  • the back-propagation algorithm if run for a large number of iterations, builds up such large values. It can be seen that generalisation in the presence of noise and limited training data is promoted if smooth decision surfaces are formed in the input feature space. This means that a small change in input values will lead to only a relatively small change in output value. This smoothness can be guaranteed if the connection weight magnitudes are kept to low values. Although it may not be possible for the network to learn the training set to such a high degree of accuracy, the difference between training set and test set performance decreases and test set performance can increase.
  • weight quantisation In any digital hardware implementation of an MLP or other network, the question of weight quantisation must be addressed. It is known that biological neurons do not perform precise arithmetic, so it might be hoped that weights in a neural network would be robust to quantisation. Normally, quantisation takes place after the network has been trained. However, if as described above, the network has built up large weight values, then node output may depend on small differences between large values. This is an undesirable situation in any numerical computation, especially one in which robustness to quantisation errors is required.
  • weight-quantisation is also an area that has not been approached with a view to performing MLP training subject to a criteria that will improve quantisation performance.
  • MLP weights would normally be examined after training and then a suitable quantisation scheme devised. It is true that the prior art technique of limiting the number of training cycles as discussed above, will improve weight quantisation simply because weight values will have not yet grown to large values, but the extent to which it does so depends, as discussed above, on a number of parameters which may be data-related. It is thus not a general-purpose solution to the problem of providing readily quantised weights. We have found that both generalisation performance and robustness to weight quantisation are improved by including explicit weight-range limiting into the MLP training procedure.
  • a method of deriving weight vectors for a neural net comprising:- - vector input means for receiving a plurality of input values comprising an input vector; and vector processing means for generating a plurality of scalar outputs in dependence upon the input vector and respective reference weight vectors, comprising the steps of:- selecting a sequence of sample input vectors (correspoding to predetermined net outputs); generating, using a digital processing device employing relatively high-precision digital arithmetic, an approximation to the scalar outputs which would be produced by the neural net processing means; generating therefrom an approximation to the outputs of the net corresponding to the respective input vectors; and - iteratively modifying the weight vectors so as to reduce the difference between the said approximated net outputs and the predetermined outputs; characterised in that, if the said modifying step would result in
  • Weight limiting also improves general network robustness to numerical inaccuracies - hence weight quantisation performance improves. It is seen here that with suitable weight limits as few as three bits per weight can be used. Large weights can cause a network node to compute small differences between large (weighted) inputs. This in turn gives rise to sensitivity to numerical inaccuracies and leads to a lessening of the inherent robustness of the MLP structure.
  • the weight limited MLP according to the invention is able to deal with inaccuracies in activation function evaluation and low resolution arithmetic better than a network with larger weight values. These factors combine to mean that the technique is useful in any limited precision, fixed-point MLP implementation.
  • the numerical robustness is increased so that, with suitable weight limits, as few as three bits per weight can be used to represent trained weight values.
  • a neural network employing weights derived according to the above method will be distinguishable, in general, from prior art networks because the distribution of the weight values will not be even (i.e. tailing off at high positive and negative weight sizes), but will be skewed towards the maximum level M, with a substantial proportion of weight magnitudes equal to M. It will also, as discussed above, have an improved generalisation performance.
  • the invention thus extends to such a network.
  • Such a network (which need not be a multi-layer perceptron network but could be, for example, a single layer perception), is, as discussed above, useful for speech recognition, but is also useful in visual object recognition and other classification tasks, and in estimation tasks such as echo cancelling or optimisation tasks such as telephone network management.
  • an "untrained" network which includes training means (e.g. a microcomputer) programmed to accept a series of test data input vectors and derive the weights therefrom in the manner discussed above during a training phase.
  • training means e.g. a microcomputer
  • the magnitude referred to is generally the absolute (i.e. ⁇ ) magnitude, and preferably the magnitude of any single weight (i.e. vector component) is constrained not to exceed M.
  • Figure 2 shows schematically a multi-layer perceptron
  • Figure 3 shows schematically a training algorithm according to one aspect of the invention
  • Figure 4a shows schematically an input stage for an embodiment of the invention
  • Figure 4b shows schematically a neuron for an embodiment of the invention
  • - Figure 5 shows schematically a network of such neurons.
  • the following describes an experimental evaluation of the weight range-limiting method of the invention. First, the example problem and associated database are described, followed by a series of experiments and the results. The problem was required to be a "real-world" problem having noisy features but also being of limited dimension.
  • the speech data used was part of a large database collected by British Telecom from long distance telephone talks over the public switched telephone network. It consisted of a single utterance of the words "yes” and “no" from more than 700 talkers. 798 utterances were available for MLP training and a further 620 for testing. The talkers in the training set were completely distinct from the test set talkers. The speech was digitally sampled at 8kHz and manually endpointed. The resulting data set included samples with impulsive switching noise and very high background noise levels.
  • the data was processed by an energy based segmentation scheme into five variable length portions. Within each segment low order LPC analysis was used to produce two cepstral coefficients and the normalised prediction error. The complete utterance was therefore described by a single fifteen dimensional vector.
  • MLPs had a single hidden layer, full connection between the input and hidden layers and full connection between the hidden and output layer. There were no direct input/output connections.
  • the back-propagation training algorithm was used with updating after each epoch, i.e., after every input/output pattern has been presented. This update scheme ensures that the weights are changed in a direction that reduces error over all the training patterns and result of the training procedure does not depend on the order of pattern presentation. All networks used a single output node. During training, the desired output was set to 0.9 for "yes” and to 0.1 for "no". During testing all utterances that gave an output of greater than 0.5 were classified as "yes” and the remainder as "no".
  • Table 7 lists the number of bits per weight required such that the RMS error is less than 5% greater than for unquantised weights for each of the values of M under consideration.
  • a real-world input (speech) 'signal is received, sampled, digitised, and processed to extract a feature vector by an input processing circuit 1.
  • the sampling rate may be 8 KHZ;
  • the digitisation may involve A-law PCM coding;
  • the feature vector may for example comprise a set of LPC coefficients, Fourier Transform coefficients or preferably melfrequency cepstral coefficients.
  • the start and end points of the utterance are determined by end pointing device la, using for example the method described in Wilpon J.G, Rabiner and Martin T: 'An improved word-detection algorithm for telephone quality speech incorporating both syntactic and semantic constraints', AT&T Bell Labs Tech J, 63 (1984) (or any other well known method), and between these points n-dimensional feature vectors X are supplied periodically (for example, every 10-100 msec) to the input node 2 of the net.
  • the feature vectors are conveniently supplied in time-division multiplexed form to a single input 2a connected to common data bus 2b.
  • the input bus is connectable to each neuron 3a, 3b, 3d, in the input layer.
  • each comprises a weight vector ROM 5 storing the 15 weight values.
  • the weighted input value produced at the output of the multiplier 6 is supplied to a digital accumulating adder 7, and added to the current accumulated total.
  • the clock signal also controls a latch 8 on the output of the accumulating adder 7 so that when all weighted input values of an input vector are accumulated, the (scalar) total is latched in latch 8 for the duration of the next cycle. This is achieved by dividing the clock by n.
  • the accumulating adder 7 is then reset to zero (or to any desired predetermined bias value).
  • the total y . w- is supplied to a non linear compression circuit 9 which generates in response an output of compressed range, typically given by the function
  • the compression circuit 9 could be a lookup ROM, the total being supplied to the address lines and the output being supplied from the data lines but is preferably as disclosed in our UK application GB8922528.8 and the article 'Efficient Implementation of piecewise linear activation function for digital VLSI Neural Networks'; Myers et al, Electronics letters, 23 November 1989, Vol 25 no 24.
  • the (latched) scalar output Y of each neuron is connected as an input to the neurons 10a, 10b, 10c of the intermediate or 'hidden' layer. Typically, fewer neurons will be required in the hidden layer.
  • the output values may be multiplexed onto a common intermediate bus 11 (but clocked at a rate of 1/n that of the input values by, for example, a flipflop circuit) in which case each hidden layer neuron 10a, 10b, 10c may be equivalent to the input layer neurons 3a, 3b, 3c, receiving as their input vector the output values of the input layer.
  • a common intermediate bus 11 but clocked at a rate of 1/n that of the input values by, for example, a flipflop circuit
  • each hidden layer neuron 10a, 10b, 10c may be equivalent to the input layer neurons 3a, 3b, 3c, receiving as their input vector the output values of the input layer.
  • the neurons 10a, 10b, 10c of the hidden layer likewise supply scalar outputs which act as inputs to neurons 12a, 12b, of the output layer.
  • Bach receives as its input vector the set of outputs of the hidden layer below, and applies its weight vector (stored, as above, in ROMs) via a multiplier, to produce a net output value.
  • the output layer neurons also use a compression function.
  • the class which corresponds to the output layer neuron producing the largest output value is allocated to the input vector (although other 'decision rules' could be used).
  • ROMs 5 could be realised as areas of a single memory device.
  • the invention may be realised using hybrid analogue digital networks in which digital weights act on analogue signals, using neurons of the type shown in Figs 11 and 12 of WO 89/02134 (assigned to the present applicants).
  • a weight adjusting device 13 is provided; typically a microprocessor operating according to a stored program.
  • the input test patterns are supplied to the input 1, and the initial weight values are adjusted, using an error back-propagation algorithm, to reduce the difference between the net outputs (to which the device 13 is connectable) and the predetermined outputs corresponding to the inputs.
  • the device 13 thus calculates the output error, calculates the necessary weight value increment for each weight; limits the weight magnitudes (if necessary) to M; accesses the weights in each store 5 (which must, of course in this embodiment, be a read/write store, not a ROM), adds the increments and rewrites the new weight values to the stores 5; the method is discussed above. Consequently, the adjusting device 13 is connected to the address busses of all stores 5 and to the net outputs. It is also connected to a source of correct output values; for example, in training a speaker-dependant word recogniser these are derived from a prompt device (not shown) which instructs the speaker to speak a given word. The correct output (say 0.9) for the output neuron corresponding to that word, is supplied to weight adjuster 13 together with correct outputs (say 0.1) for all other output neurons.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Complex Calculations (AREA)
  • Image Analysis (AREA)

Abstract

L'invention se rapporte à un réseau neuronal, qui est soumis à un processus d'apprentissage effectué au moyen de données d'apprentissage et dans lequel les valeurs de pondération sont augmentées jusqu'à une valeur maximum prédéterminée (M), toutes les valeurs de pondération qui dépasseraient M étant mises à une valeur égale à M. L'invention sert à former par processus d'apprentissage des perceptrons multicouches pour la reconnaissance de la parole, et permet d'obtenir des valeurs de pondération qui sont plus facilement quantifiées, assurant ainsi une plus grande robustesse au niveau du fonctionnement.
EP90909478A 1989-06-30 1990-06-29 Reconnaissance des structures Ceased EP0484344A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB898915085A GB8915085D0 (en) 1989-06-30 1989-06-30 Pattern recognition
GB8915085 1989-06-30

Publications (1)

Publication Number Publication Date
EP0484344A1 true EP0484344A1 (fr) 1992-05-13

Family

ID=10659352

Family Applications (1)

Application Number Title Priority Date Filing Date
EP90909478A Ceased EP0484344A1 (fr) 1989-06-30 1990-06-29 Reconnaissance des structures

Country Status (7)

Country Link
EP (1) EP0484344A1 (fr)
JP (1) JPH04506424A (fr)
CA (1) CA2063426A1 (fr)
FI (1) FI916155A0 (fr)
GB (2) GB8915085D0 (fr)
HK (1) HK132896A (fr)
WO (1) WO1991000591A1 (fr)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0683781B2 (ja) * 1986-03-14 1994-10-26 住友化学工業株式会社 薄片状物質の製造方法
GB9116255D0 (en) * 1991-07-27 1991-09-11 Dodd Nigel A Apparatus and method for monitoring
US5621858A (en) * 1992-05-26 1997-04-15 Ricoh Corporation Neural network acoustic and visual speech recognition system training method and apparatus
DE4300159C2 (de) * 1993-01-07 1995-04-27 Lars Dipl Ing Knohl Verfahren zur gegenseitigen Abbildung von Merkmalsräumen
US11106973B2 (en) 2016-03-16 2021-08-31 Hong Kong Applied Science and Technology Research Institute Company Limited Method and system for bit-depth reduction in artificial neural networks

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0327817B1 (fr) * 1988-01-11 1996-08-28 Yozan Inc. Système de conversion de forme associative et méthode d'adaptation de ce système

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO9100591A1 *

Also Published As

Publication number Publication date
CA2063426A1 (fr) 1990-12-31
GB2253295B (en) 1993-11-03
GB9127502D0 (en) 1992-03-11
JPH04506424A (ja) 1992-11-05
WO1991000591A1 (fr) 1991-01-10
HK132896A (en) 1996-08-02
GB2253295A (en) 1992-09-02
FI916155A0 (fi) 1991-12-30
GB8915085D0 (en) 1989-09-20

Similar Documents

Publication Publication Date Title
EP0623914A1 (fr) Système de la reconnaissance des mots isolés indépendant de celui qui parle utilisant un réseau de neurones
EP0333798B1 (fr) Appareil de reconnaissance de formes
Bhama et al. Single layer neural networks for linear system identification using gradient descent technique
KR0173923B1 (ko) 다층구조 신경망을 이용한 음소 분할 방법
WO1995034063A1 (fr) Procede de decoupage d'une sequence de trames de donnees
WO1993013519A1 (fr) Systeme expert composite
CN111899757A (zh) 针对目标说话人提取的单通道语音分离方法及系统
EP1576580A1 (fr) Procede permettant d'optimiser l'execution d'un reseau neural dans un systeme de reconnaissance vocale et consistant a sauter de maniere conditionnee un nombre variable de trames
US5745874A (en) Preprocessor for automatic speech recognition system
EP0484344A1 (fr) Reconnaissance des structures
Chang et al. Mandarin tone recognition by multi-layer perceptron
JPH0540497A (ja) 話者適応音声認識装置
Woodland Weight limiting, weight quantisation and generalisation in multi-layer perceptrons
Patgiri et al. A class of neuro-computational methods for assamese fricative classification
Lee et al. Recurrent neural networks for speech modeling and speech recognition
Renals et al. Learning phoneme recognition using neural networks
WO2000008634A1 (fr) Procedes et appareil d'evaluation de phonemes au moyen de reseaux neuronaux
Jou et al. Mandarin syllables recognition based on one class one net neural network with modified selective update algorithm
Mafra et al. Text independent automatic speaker recognition using selforganizing maps
JPH08297495A (ja) 信号パターン認識装置の学習方法及び学習装置
Reynolds et al. Spoken letter recognition with neural networks
Czyzewski et al. Multilayer knowledge base system for speaker-independent recognition of isolated words
Sankar et al. Noise immunization using neural net for speech recognition
Costin et al. Tips on speaker recognition by autoregressive parameters and connectionist methods
Kasper et al. Strategies for reducing the complexity of a rnn based speech recognizer

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 19911220

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE CH DE DK ES FR GB IT LI LU NL SE

17Q First examination report despatched

Effective date: 19940513

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN REFUSED

18R Application refused

Effective date: 19970208