CN108734291A - A kind of pseudo label generator using correctness feedback training neural network - Google Patents
A kind of pseudo label generator using correctness feedback training neural network Download PDFInfo
- Publication number
- CN108734291A CN108734291A CN201810516876.XA CN201810516876A CN108734291A CN 108734291 A CN108734291 A CN 108734291A CN 201810516876 A CN201810516876 A CN 201810516876A CN 108734291 A CN108734291 A CN 108734291A
- Authority
- CN
- China
- Prior art keywords
- neural network
- pseudo label
- correctness
- feedback
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
- G06F18/2178—Validation; Performance evaluation; Active pattern learning techniques based on feedback of a supervisor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Abstract
The present invention proposes method of the utilization to the correctness feedback training neural network of neural network inferred results, design a pseudo label generator, corresponding pseudo label is generated according to the feedback to neural network inferred results correctness, it is subsequently used for the training of neural network, solves the problems, such as that the training sample that scene obtains under embedded scene does not have label.
Description
Technical field
Technical field of the present invention is artificial neural network field.
Background technology
Deep neural network is since it is in image classification, the outstanding behaviours quilt of speech recognition and natural language processing etc.
It is widely used in every field.In built-in field, since the operation especially training process of deep neural network needs largely
Calculating, and embedded device, such as smart mobile phone, wearable device etc. again have comparison stringent performance and power consumption limit, because
This seldom directly applies deep neural network on devices.Many Embedded Applications are all that task and data are sent to cloud
Result is returned to equipment after completion task by the neural network model at end by internet again.It is existing however as the progress of technology
It is much being devised and implemented for Embedded neural network SoC, while more mobile phone vendor commercial cities are to built in smart mobile phone
Dedicated for the AI engines of deep learning so that directly become possibility using deep neural network on embedded device.
Although many smart mobile phones and wearable device have the accelerator module dedicated for neural network now, by
It is limited in the power consumption and performance of embedded device, the performance of these neural network accelerators can not be competent at extensive and big data
The training of amount.Therefore, a relatively good scheme is exactly first extensive to neural network progress big on a high performance large-scale group of planes
The training of data volume, then training gained model is moved to by the methods of compressing, distilling on embedded device.
The use of the embedded devices such as smart mobile phone and wearable device is all more personalized, passes through public big number
The data of someone can not may be well used for according to the obtained neural network model of training.Therefore, it is necessary to use equipment
Itself collected personalized data carries out retraining to moving to embedded device epineural network.
One feature of mobile embedded type equipment is exactly to be equipped with a large amount of sensor, can acquire making for user in real time
Use data.For some tasks based on neural network, such as image classification and voice recognition etc., user is using neural network
When can just collect user data.Since data are that scene obtains, to neural network on embedded device
Training process is gradually carried out along with deduction process.
Although the mobile embedded type equipments such as smart mobile phone and wearable device can real-time gathered data, be difficult from
Dynamic gives one correct label of collected data, and if going manually to specify label to data by user, it can influence
User experience.However, since the neural network on mobile embedded is gathered data in use, then while user
The collected data of institute will not be initiatively given to specify an accurate label, but they are often to the prediction knot of neural network
Fruit provides the feedback of a correctness.Such as the speech-to-text application based on neural network, if neural network model
The transformation result of mistake is outputed, then user may delete the word of these mistakes, and if exporting result correctly not
Meeting.The correctness how to be provided in such a scenario using user is fed back to train neural network, and there is presently no relevant
Research.
Invention content
The present invention proposes a kind of method fed back in the correctness using user to train neural network, as shown in Figure 1,
Neural network infers training examples, then export prediction as a result, at this time user can be to the pre- of neural network
It surveys result to be judged, and makes corresponding reaction.For example, if user operate in next step, illustrate nerve net
The prediction of network is correct；If user has carried out delete operation, illustrate that the prediction of neural network is wrong, therefore root
The correctness of neural network prediction result can be obtained according to the different operation of user.Then pseudo label generator is utilized and is obtained just
True property feedback generates pseudo label, is used for the training of neural network.
Neural network for classification can generally use softmax layers to be used as output layer, and prediction distribution and true point
The cross entropy of cloth carrys out back-propagation as loss function and reaches the destination of study to update weight.As illustrated in fig. 2, it is assumed that last
The output of one layer of hidden layer is yj, j ∈ { 1,2 ..., n }, then softmax layers pass through functionOutput is mapped to
The section of [0,1], therefore softmax layers of output can regard probability point of the neural network to sample classification prediction result as
Cloth.
For cross entropy loss function, formula is as follows：
Wherein, p is really to be distributed, and can be obtained by the label of training sample, for the model of n classification, it is assumed that sample
True tag is the i-th class, thenAnd q is the prediction point of classification of the neural network model to training sample
Cloth corresponds to softmax layers of output result.
For a n classification problem, neural network exports a n-dimensional vector at softmax layers, indicates the general of prediction classification
Rate is distributed, and the classification of the item representative of maximum probability is exactly the classification of neural network prediction, such as i-th of element value in output vector
Maximum, then the classification results of neural network prediction, which are exactly sample, belongs to the i-th class；True tag is also a n-dimensional vector, wherein
Element value corresponding to true classification is 1, and other elements value is all that j-th of element is 1 in 0, such as true tag, then illustrating
The true classification of sample is jth class.
Under the application scenarios of correctness feedback, the prediction result of the neural network obtained according to user feedback is divided into two
Class predicts correct and prediction error.In both cases, pseudo label generator constructs the pseudo label vector of a n dimension respectively,
Training for neural network.
For predicting correct sample, as classification corresponding to true tag with the classification of neural network prediction is
, thus to be directly inferred to its true tag.For the sample of prediction error, it is assumed that the classification of neural network prediction is i-th
Class, then since prediction is wrong, so the value in pseudo label corresponding to the i-th class is 0, the value corresponding to other classifications is equal to
The prediction probability of this class adds prediction probability being averaged on n-1 items of the i-th class.Fig. 3 is described in detail using mathematical formulae
The specific method that pseudo label generates.
Description of the drawings
Fig. 1, which is fed back for pseudo label generator using correctness, generates pseudo label to train the general illustration of neural network.
Fig. 2 is the neural network schematic diagram using softmax output layers.
Fig. 3 is the definition graph that pseudo label generator generates pseudo label method.
Specific implementation mode
Fig. 1 illustrates the flow that neural network is trained using the method for the present invention.Neural network is to input first
Training examples are inferred, the classification results predicted；According to the correctness of prediction classification results, user can carry out accordingly
Feedback, then the pseudo label generator in the present invention will be according to the corresponding pseudo label of the correctness of user feedback generation；Then,
Neural network carries out error back propagation using pseudo label, to improve the accuracy of network model.
Claims (2)
1. a kind of correctness using to neural network inferred results feeds back the method to train neural network, it is characterised in that：
User provides the inferred results of neural network the feedback of one correctness, and then the feedback be used to generate corresponding with sample
Pseudo label, reuse pseudo label and neural network be trained.
2. obtaining generating algorithm using pseudo label as described in claim 1, it is characterised in that：If feedback the result is that " pushing away
It is disconnected correct ", then pseudo label is exactly true tag, that class probability for corresponding to inferred results is 1；If the result of feedback
It is " inferring mistake ", then the probability of that classification in pseudo label corresponding to inferred results is exactly 0, then by such deduction
Probability is averagely added to the upper of other classification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810516876.XA CN108734291A (en) | 2018-05-25 | 2018-05-25 | A kind of pseudo label generator using correctness feedback training neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810516876.XA CN108734291A (en) | 2018-05-25 | 2018-05-25 | A kind of pseudo label generator using correctness feedback training neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108734291A true CN108734291A (en) | 2018-11-02 |
Family
ID=63936262
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810516876.XA Pending CN108734291A (en) | 2018-05-25 | 2018-05-25 | A kind of pseudo label generator using correctness feedback training neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108734291A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583321A (en) * | 2019-02-19 | 2020-08-25 | 富士通株式会社 | Image processing apparatus, method and medium |
| CN112101083A (en) * | 2019-06-17 | 2020-12-18 | 辉达公司 | Object detection with weak supervision using one or more neural networks |
-
2018
- 2018-05-25 CN CN201810516876.XA patent/CN108734291A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583321A (en) * | 2019-02-19 | 2020-08-25 | 富士通株式会社 | Image processing apparatus, method and medium |
| CN112101083A (en) * | 2019-06-17 | 2020-12-18 | 辉达公司 | Object detection with weak supervision using one or more neural networks |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11314806B2 (en) | Method for making music recommendations and related computing device, and medium thereof | |
| Salman et al. | Weather forecasting using deep learning techniques | |
| Zhou et al. | Stacked extreme learning machines | |
| EP4018390A1 (en) | Resource constrained neural network architecture search | |
| Ma et al. | Adaptive-step graph meta-learner for few-shot graph classification | |
| CN109345302A (en) | Machine learning model training method, device, storage medium and computer equipment | |
| CN109885756B (en) | CNN and RNN-based serialization recommendation method | |
| CN111666406B (en) | Short text classification prediction method based on word and label combination of self-attention | |
| Yang et al. | Recomputation of the dense layers for performance improvement of dcnn | |
| CN110619050B (en) | Intention recognition method and device | |
| CN107247753B (en) | A kind of similar users choosing method and device | |
| CN112015902B (en) | Least-order text classification method under metric-based meta-learning framework | |
| CN110245228A (en) | The method and apparatus for determining text categories | |
| CN115659254A (en) | Power quality disturbance analysis method for power distribution network with bimodal feature fusion | |
| CN108734291A (en) | A kind of pseudo label generator using correctness feedback training neural network | |
| CN112131261A (en) | Community query method and device based on community network and computer equipment | |
| CN108920727A (en) | Compact visual in vision retrieval describes sub- deep neural network and generates model | |
| CN107807919A (en) | A kind of method for carrying out microblog emotional classification prediction using random walk network is circulated | |
| CN114626550A (en) | Distributed model collaborative training method and system | |
| CN110489616A (en) | A kind of search ordering method based on Ranknet and Lambdamart algorithm | |
| CN103559510B (en) | Method for recognizing social group behaviors through related topic model | |
| Wang et al. | An adaptive learning rate schedule for SIGNSGD optimizer in neural networks | |
| WO2021062219A1 (en) | Clustering data using neural networks based on normalized cuts | |
| CN113010774B (en) | Click rate prediction method based on dynamic deep attention model | |
| CN116403608A (en) | Speech emotion recognition method based on multi-label correction and space-time collaborative fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20181102 |