EP0583217B1 - Optimisation d'un réseau neuronal utilisant rétropropagation - Google Patents

Optimisation d'un réseau neuronal utilisant rétropropagation Download PDF

Info

Publication number
EP0583217B1
EP0583217B1 EP93650028A EP93650028A EP0583217B1 EP 0583217 B1 EP0583217 B1 EP 0583217B1 EP 93650028 A EP93650028 A EP 93650028A EP 93650028 A EP93650028 A EP 93650028A EP 0583217 B1 EP0583217 B1 EP 0583217B1
Authority
EP
European Patent Office
Prior art keywords
network
training
performance
training method
hidden
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP93650028A
Other languages
German (de)
English (en)
Other versions
EP0583217A3 (fr
EP0583217A2 (fr
Inventor
John Mitchell
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hitachi Europe Ltd
Original Assignee
Hitachi Europe Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Europe Ltd filed Critical Hitachi Europe Ltd
Publication of EP0583217A2 publication Critical patent/EP0583217A2/fr
Publication of EP0583217A3 publication Critical patent/EP0583217A3/fr
Application granted granted Critical
Publication of EP0583217B1 publication Critical patent/EP0583217B1/fr
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections

Definitions

  • Feedforward neural networks have been used in many applications, both using discrete and continuous data. In the discrete case, they have been applied to many problems such as pattern recognition/classification which may involve, for example the grading of apples from photographic images, texture analysis of paper quality, or automatic recognition of hand-written postal codes. Application areas where continuous valued data is used include time series prediction and nonlinear regression.
  • One of the most useful aspects of neural networks in these applications is the potential of a suitably trained network to generalise, i.e. to achieve good performance on new examples not used in training of the network. For example, in applying neural networks to recognition of hand-written postal code digits, training is performed using handwriting from a small group of people. It would be useful for postcode recognition if the network also performs well on handwriting samples other than those not used in training of the network.
  • the actual learning processes themselves may be classified as gradient based or perturbation based.
  • the information to adjust the weight values of the network is determined from the gradient of the error measure with respect to the various parameters.
  • Perturbation processes use a different approach. They involve making a small change in the parameters of the network, using a random perturbation. Then the resulting network is checked, and if it is better, the parameters of the network are kept. If not, the change is undone and a new one tried.
  • An example of a stochastic perturbation process is the "global optimisation" process.
  • the invention is directed towards providing a method and apparatus to provide for optimising a feedforward neural network and also to improving efficiency of training.
  • a training method for a feedforward neural network having hidden units comprising the steps of transmitting input stimuli to the network and adjusting connection weights in response to monitoring the network output signals, for that training method, the steps including dynamically interpreting the network performance during training by interpretation of the transfer function of an individual hidden unit using the immediately connecting weights of that hidden unit; continuously generating a dynamic indicator of that network performance; comparing the dynamic indicator to a desired dynamic indicator of performance; and interrupting the training method when the dynamic indicator falls below the desired dynamic indicator of performance; characterised in that the method comprises the additional steps when training is interrupted of:
  • the invention provides an apparatus for a training method of a feedforward neural network, the apparatus comprising means for transmitting stimuli to the network and means for adjusting connection weights in response to monitoring the network output signals, means for dynamically interpreting performance of the network during training by interpretation of the transfer function of an individual hidden unit using the immediately connecting weights of that hidden unit; means for continuously generating a dynamic indicator of the network performance; means for comparing the dynamic indicator to a desired dynamic indicator of performance; and means for interrupting the training method when the dynamic indicator falls below the desired dynamic indicator of performance, characterised in that the apparatus further comprises:
  • Fig. 1 there is illustrated a training process or method which involves optimising the structure of a feedforward neural network which has hidden units by determination of the optimum number of hidden units.
  • the network has many inputs, a single output and a single layer of hidden units. This could be used for solution of real valued approximation problems such as nonlinear regression applications.
  • the method is indicated by the reference numerals 1 to 5, inclusive.
  • Fig. 2 a typical feedforward neural network 6 is shown.
  • the procedure involved in training a neural network using the backpropagation learning method is as follows.
  • the architecture being considered here is the many input, single output, single hidden layer neural network. This is to be used on real-valued approximation problems.
  • the input to the network is the vector x i
  • the desired output of the network is the scalar y i .
  • This set of examples is repeatedly presented to the network, and the connecting weight values adjusted so as to reduce some error measure measuring the difference between the network output on an example (say N) and the desired output of the network y i .
  • the usual error measure used in this case is the mean square error:
  • the apparatus of the invention comprises the following components:-
  • step 1 of the method training of the neural network is carried out by the neural network processor 15. Training may be carried out by any suitable iterative process, such as backpropagation.
  • step 2 of the method dynamic interpretation of the structure of the neural network is carried out during training.
  • dynamic interpretation of the network at the microscopic level is carried out. This involves interpretation of the transfer function of a hidden unit using only the immediately connecting weights of that hidden unit. Thus, the processing requirements for such dynamic interpretation are relatively small.
  • Step 2 also involves displaying performance indicators generated during dynamic interpretation. Dynamic interpretation is described in more detail below.
  • step 5 a decision is made as to whether or not the particular unit is to be removed from the network. If so, the neural network processor removes the unit, training resumes, and steps 1 to 5 are repeated. Such a decision may alternatively be automatically made by the processor 15 without the need for static interpretation, if the geometrical indicator from dynamic interpretation is clear enough.
  • step 2 interpretation of a hidden unit does not require information relating to other hidden units of the network.
  • this information generally comprises weights connected directly to the hidden unit and boundaries of the data. Accordingly, there is a relatively small processing requirement of the dynamic interpreter and training of a network is not slowed down to any great extent.
  • a feedforward network with a single hidden layer of units and a single output unit performs the following type of transformation from input to output:
  • the structure also contains bias terms ⁇ i and ⁇ , which can be interpreted as weight values to the hidden or output unit from a processing unit which always outputs 1. They are intended to give the structure more flexibility in training.
  • the power of this structure comes from the fact that the units within it process information in a non-linear fashion.
  • each of the hidden units has a transfer function ⁇ i
  • the output unit has a transfer function ⁇ .
  • projection methods can be used to relate the role of the hidden unit to the input space.
  • the derivative of the transfer function is asymptotically zero. This restriction is not a severe one, and most of the transfer functions proposed for feedforward networks satisfy it. In fact since gradient descent methods use the derivative of the transfer function in calculating the change to the weights, it is necessary for this condition to hold for the teaching method not to be unstable.
  • the transfer function is then projected onto a subset of the input space, which is termed a geometrical primitive of the hidden unit. The following example shows how this is done in the case of the sigmoidal transfer function.
  • ⁇ is the bias term of the output unit.
  • K is a or b depending on whether step B3 or B4 were followed.
  • C is the contributory estimate of the Nth hidden unit.
  • f N (x) and f N -1 (x) are related by: where ⁇ is the maximum value of the derivative of the transfer function of the output unit. This relates to the contributory estimate of the hidden unit explicitly to an upper bound for the change in state induced by the removal of that unit.
  • the step 4 of static interpretation provides more detailed information about the network.
  • the mean squared error of the network before and after the removal of a unit may be determined.
  • y i is the required value
  • f i is the network output.
  • the generalisation error can be investigated by calculating (26) for examples outside the training set but within the scope of the required problem.
  • Part (1) can be easily translated into a high-level control conditions for the learning manager, namely, "initially, use A".
  • a control condition to determine when to switch processes is based on the dynamics of the learning. Such control conditions are used to detect situations when features associated with a given learning method hold. In this embodiment the feature is that process A is no longer converging. This is achieved using the moving average detector of the equation (30).
  • Figure 8 shows the complete example.
  • the example shows the rules as they might appear within a system as a set of commands.
  • the moving average control condition is defined.
  • the control condition is attached to method A.
  • this control condition will act as a monitor to decide when it should be stopped.
  • the final two lines apply method A and then B.
  • the method of transfer from A to B is determined by the control condition.
  • the important point is that a control condition is used in deciding what to do next. This may be to switch learning processes, or to end learning.

Claims (9)

  1. Méthode d'apprentissage pour un réseau neuronal à réaction anticipative (6) ayant des unités cachées (8) comprenant les étapes consistant à transmettre des excitations d'entrée au réseau et à ajuster des pondérations de connexion en réponse au contrôle des signaux de sortie du réseau pour cette méthode d'apprentissage, les étapes incluant interpréter dynamiquement (2) la performance du réseau durant l'apprentissage par l'interprétation de la fonction de transfert d'une unité cachée individuelle (8) en utilisant les pondérations de connexion immédiate de cette unité cachée (8); générer continuellement un indicateur dynamique (20) de la performance de ce réseau; comparer l'indicateur dynamique (2) avec un indicateur dynamique désiré de performance; et interrompre la méthode d'apprentissage lorsque l'indicateur dynamique (20) tombe en dessous de l'indicateur dynamique désiré de performance; caractérisée en ce que, lorsque l'apprentissage est interrompu, la méthode comprend les étapes supplémentaires consistant à:
    générer un indicateur statique de la performance de l'unité cachée (8) en réalisant une interprétation statique de la performance globale du réseau avec et sans l'unité cachée; et
    altérer la structure interne du réseau en réponse à l'interprétation statique de la performance de l'unité cachée.
  2. Méthode d'apprentissage telle que revendiquée dans la revendication 1, dans laquelle les fonctions de transfert continues et monotones ayant des dérivées asymptotiquement de zéro, sont interprétées.
  3. Méthode d'apprentissage telle que revendiquée dans l'une quelconque des revendications précédentes, dans laquelle l'indicateur dynamique (20) est géométrique et est affiché.
  4. Méthode d'apprentissage telle que revendiquée dans l'une quelconque des revendications précédentes, dans laquelle les interprétations dynamique et statique comprennent la mise en relation de l'opération d'une unité cachée avec les données d'entrée.
  5. Méthode d'apprentissage telle que revendiquée dans l'une quelconque des revendications précédentes, dans laquelle les informations globales se rapportant à la performance du réseau sont interprétées durant l'interprétation statique (4).
  6. Méthode d'apprentissage telle que revendiquée dans l'une quelconque des revendications précédentes, comprenant les étapes ultérieures consistant à:
    mémoriser les caractéristiques de différentes méthodes d'apprentissage (29);
    sélectionner une méthode d'apprentissage initiale (29) et exécuter l'apprentissage selon la méthode (29);
    contrôler dynamiquement une caractéristique de la méthode d'apprentissage (29);
    évaluer la caractéristique contrôlée selon une condition de contrôle; et
    sélectionner une méthode d'apprentissage différente (29) pour des étapes d'apprentissage subséquentes selon la condition de contrôle.
  7. Méthode d'apprentissage telle que revendiquée dans la revendication 6, dans laquelle une pluralité de conditions de contrôle sont utilisées pour évaluer des caractéristiques contrôlées, une condition de contrôle étant spécifique à une méthode d'apprentissage, et une autre condition de contrôle étant spécifique à toutes les méthodes.
  8. Appareil pour une méthode d'apprentissage d'un réseau neuronal à réaction anticipative (6) ayant des unités cachées (8), l'appareil comprend un moyen (15) pour transmettre des excitations au réseau (6); un moyen (15) pour ajuster des pondérations de connexion en réponse au contrôle des signaux de sortie du réseau, un moyen (16) pour interpréter dynamiquement la performance du réseau (6) pendant l'apprentissage par interprétation de la fonction de transfert d'une unité cachée individuelle (8) en utilisant les pondérations de connexion immédiate de cette unité cachée (8); un moyen pour générer continuellement un indicateur dynamique (20) de la performance du réseau; un moyen pour comparer l'indicateur dynamique (20) avec un indicateur dynamique désiré de performance; et un moyen pour interrompre la méthode d'apprentissage lorsque l'indicateur dynamique (20) tombe en dessous de l'indicateur dynamique désiré de performance, caractérisé en ce que l'appareil comprend en outre:
    un moyen de communication entre l'indicateur dynamique (20) et un interpréteur statique (17);
    un moyen dans l'interpréteur statique (17) pour générer un indicateur statique de la performance de l'unité cachée (8) par la mise en oeuvre d'une interprétation statique de la performance globale du réseau, avec et sans l'unité cachée lorsque l'apprentissage est interrompu; et
    un moyen pour altérer la structure de réseau en réponse à l'interprétation statique de la performance de l'unité cachée.
  9. Appareil pour mettre en oeuvre une méthode d'apprentissage tel que revendiqué dans la revendication 8, incluant:
    une pluralité de méthodes d'apprentissage (29); et
    l'appareil comprend en outre un contrôleur d'apprentissage (30) comprenant:
    un moyen pour mémoriser des caractéristiques des méthodes d'apprentissage;
    un moyen pour amorcer l'apprentissage du réseau selon une méthode d'apprentissage sélectionnée;
    un moyen pour contrôler dynamiquement une caractéristique d'apprentissage;
    un moyen pour évaluer la caractéristique contrôlée selon une condition de contrôle; et
    un moyen pour sélectionner une méthode d'apprentissage différente ou pour des étapes d'apprentissage subséquentes selon la condition de contrôle.
EP93650028A 1992-08-11 1993-07-29 Optimisation d'un réseau neuronal utilisant rétropropagation Expired - Lifetime EP0583217B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IE922575 1992-08-11
IE257592 1992-08-11

Publications (3)

Publication Number Publication Date
EP0583217A2 EP0583217A2 (fr) 1994-02-16
EP0583217A3 EP0583217A3 (fr) 1995-03-15
EP0583217B1 true EP0583217B1 (fr) 2000-05-10

Family

ID=11039732

Family Applications (1)

Application Number Title Priority Date Filing Date
EP93650028A Expired - Lifetime EP0583217B1 (fr) 1992-08-11 1993-07-29 Optimisation d'un réseau neuronal utilisant rétropropagation

Country Status (3)

Country Link
EP (1) EP0583217B1 (fr)
JP (1) JPH07302249A (fr)
DE (1) DE69328596T2 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7412426B2 (en) 2003-06-26 2008-08-12 Neuramatix Sdn. Bhd. Neural networks with learning and expression capability

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7020595B1 (en) * 1999-11-26 2006-03-28 General Electric Company Methods and apparatus for model based diagnostics
JP6948851B2 (ja) * 2016-06-30 2021-10-13 キヤノン株式会社 情報処理装置、情報処理方法
JP7047283B2 (ja) * 2017-08-24 2022-04-05 富士通株式会社 情報処理装置、方法、及びプログラム
JP6831347B2 (ja) * 2018-04-05 2021-02-17 日本電信電話株式会社 学習装置、学習方法および学習プログラム
CN110147872B (zh) * 2018-05-18 2020-07-17 中科寒武纪科技股份有限公司 编码存储装置及方法、处理器及训练方法
JP2022095999A (ja) * 2019-04-19 2022-06-29 国立大学法人北海道大学 ニューラル計算装置、および、ニューラル計算方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5093899A (en) * 1988-09-17 1992-03-03 Sony Corporation Neural network with normalized learning constant for high-speed stable learning
WO1991002315A1 (fr) * 1989-08-01 1991-02-21 E.I. Du Pont De Nemours And Company Procedes relatifs a la configuration d'un reseau de traitement reparti en parallele

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7412426B2 (en) 2003-06-26 2008-08-12 Neuramatix Sdn. Bhd. Neural networks with learning and expression capability
US7778946B2 (en) 2003-06-26 2010-08-17 Neuramatix SDN.BHD. Neural networks with learning and expression capability

Also Published As

Publication number Publication date
EP0583217A3 (fr) 1995-03-15
DE69328596D1 (de) 2000-06-15
DE69328596T2 (de) 2001-01-04
JPH07302249A (ja) 1995-11-14
EP0583217A2 (fr) 1994-02-16

Similar Documents

Publication Publication Date Title
Nauck Neuro-fuzzy systems: review and prospects
De Veaux et al. A comparison of two nonparametric estimation schemes: MARS and neural networks
Andrychowicz et al. Learning to learn by gradient descent by gradient descent
US6119112A (en) Optimum cessation of training in neural networks
EP1504413A2 (fr) Maintenance et generation d'un modele de reseau neural automatique
Suykens et al. NL/sub q/theory: checking and imposing stability of recurrent neural networks for nonlinear modeling
Hicham et al. Comparative study of neural networks algorithms for cloud computing CPU scheduling
Wei et al. Artificial neural network based predictive method for flood disaster
EP0583217B1 (fr) Optimisation d'un réseau neuronal utilisant rétropropagation
Nayak et al. Tlbo-fln: Teaching-learning based optimization of functional link neural networks for stock closing price prediction
Jat et al. Applications of statistical techniques and artificial neural networks: A review
Lee et al. Design optimization with back-propagation neural networks
McCormack Neural networks in the petroleum industry
JP4267726B2 (ja) 制御装置における動作信号と操作量との関係を決定する装置、制御装置、データ生成装置、入出力特性決定装置及び相関関係評価装置
Dennis Landscape aware algorithm configuration
Zhu et al. CausalDyna: Improving Generalization of Dyna-style Reinforcement Learning via Counterfactual-Based Data Augmentation
Guo et al. Temporal difference learning applied to sequential detection
Baisch et al. A neural fuzzy system to evaluate software development productivity
Blows Reinforcement learning for telescope optimisation
Cullinan Revisiting the society of mind: Convolutional neural networks via multi-agent systems
Sarma et al. Neural networks and their applications in industry
EP4195106A1 (fr) Inférence de poids contrainte pour l'apprentissage distribué en ligne
Escovedo et al. Neuroevolutionary models based on quantum-inspired evolutionary algorithms
Gervino Complex Environment Exploration
Jones et al. Optimization techniques applied to neural networks: Line search implementation for back propagation

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): DE FR GB IE

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): DE FR GB IE

17P Request for examination filed

Effective date: 19950622

17Q First examination report despatched

Effective date: 19980610

RTI1 Title (correction)

Free format text: OPTIMISATION OF FEEDFORWARD NEURAL NETWORKS

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB IE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 69328596

Country of ref document: DE

Date of ref document: 20000615

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed
REG Reference to a national code

Ref country code: GB

Ref legal event code: IF02

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IE

Payment date: 20040531

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20040726

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20040728

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20040929

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20050729

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20050729

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060201

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20050729

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20060331

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20060331