CN107609566A - Dual network deep learning method based on a small amount of personalized sample - Google Patents

Abstract

The present invention relates to a double-network deep learning method based on a small number of personalized samples, the double-network includes a reconstruction network and a deep network, and the method includes: collecting a small amount of personalized samples; wherein, the personalized samples include sample data and Its label; use the sample data in the personalized sample to train the reconstruction network, and then input the label in the personalized sample into the trained reconstruction network to generate new reconstruction data and its label; based on the reconstruction from The new reconstruction data of network and label thereof train described depth network; The data to be tested is input in the depth network through training, obtain the output of sigmoid function, then through a soft maximum regression from described sigmoid function Select one of the results in the output as a personalized response.

Description

A dual-network deep learning approach based on a small number of personalized samples

technical field

The invention relates to the fields of communication and data processing, in particular to a double-network deep learning method based on a small number of personalized samples.

Background technique

With the rapid development of network technology, the capacity, personalization and diversity of data are increasing rapidly, but the algorithm complexity of data processing is difficult to improve, relying on personal experience and manual operations to describe data, label data, select features, extract features, The method of data processing has been difficult to meet the rapidly growing demand for personalized data, and how to efficiently process personalized data has become an urgent problem.

Research breakthroughs in deep learning methods have pointed out a direction worth exploring for solving data processing problems. Deep learning can automatically extract features from big data, and obtain good processing results through massive sample training. In fact, the rapid growth of big data and the research of deep learning complement each other. On the one hand, the rapid growth of big data requires a method for efficiently processing massive amounts of data. On the other hand, the training of deep learning systems requires massive sample data.

However, personalized data is mainly generated by users at the edge of the network. The current method is to collect the data first, and then analyze and process the data on the cloud server. This will generate a large amount of data transmission. At the same time, for the end user, the response result obtained from the server is obtained from the response of the model trained on the global data, and the response accuracy for personalized users is low. If the deep learning network is constructed on the terminal, the deep learning network cannot be trained due to the small amount of terminal personalized user data. This is the big data dilemma that exists in the current network.

At present, the Hadoop architecture developed by Apache, the parameter server architecture developed by Baidu, and the Mariana architecture developed by Tencent are all attempts for distributed large-scale and deep learning systems. By splitting the model, the data is split These systems solve the problem of large-scale data processing to a certain extent. However, the above distributed systems process large-scale models and data on multiple machines, and there is a large amount of data communication between multiple machines, which leads to the fact that these distributed systems can only be applied to local area networks at present. However, if end users want to get a response, they must first transmit the data to the central cluster, and then the central cluster will send the generated response to the end user, so the user-side delay response is very high.

In wide-area networks, there are problems such as network bandwidth limitations, big data dilemmas, and individual needs. If the problems of high communication costs and few edge user data in the distributed big data real-time processing system can be solved, it can effectively Apply the distributed big data real-time processing system to the wide area network to improve network utilization and user experience.

Therefore, for a small number of personalized samples, it is necessary to provide a distributed data processing method that is more suitable for wide-area networks to solve the problem of large transmission of training data and model parameters, and when there is personalized information in distributed data. Difficult to raise issues.

Contents of the invention

The purpose of the present invention is to overcome the defect of the existing distributed data processing method that the processing accuracy is difficult to improve due to the small number of edge user sample data, thereby providing a dual-network deep learning method based on a small number of personalized samples.

In order to achieve the above object, the present invention provides a dual-network deep learning method based on a small number of personalized samples, the dual-network includes a reconstructed network and a deep network, and the method includes:

Step 1), collecting a small amount of personalized samples; wherein, the personalized samples include sample data and tags thereof;

Step 2), using the sample data in the personalized samples to train the reconstructed network, and then input the labels in the personalized samples into the trained reconstructed network to generate new reconstructed data and labels thereof;

Step 3), based on step 2) obtained new reconstruction data from the reconstruction network and its label training the deep network;

Step 4), input the data to be tested into the trained deep network to obtain the output of the S-shaped function, and then select a result from the output of the S-shaped function through a soft maximum regression as a personalized response.

In the above technical solution, the number of the small number of personalized samples is recorded as m, m≈M/J, where M is the total number of samples when using big data to train a deep network with the same precision, and J is the number of samples reconstructed through a personalized sample. The number of samples out.

In the above technical solution, in step 2), the generated new reconstructed data and its label set are expressed as {x ^r _k , y ^r _k }; where,

x ^r _k represents any reconstructed data, y ^r _k is the label corresponding to the reconstructed data, the value of k is 1≤k≤m×J, m is the number of personalized samples, and J is the The number of reconstructed samples; the relationship between J and the number of personalized samples m is as follows: the smaller the value of m, the larger the value of J, so as to ensure that there are enough reconstructed samples m×J to train the deep network.

In the above technical solution, the reconstruction network is any learning network with the function of randomly reconstructing a large number of samples, including Bayesian network and non-hierarchical neural network.

In the above technical solution, the reconstruction network is a Bayesian procedural learning network with a one-time learning function.

In the above technical solution, the deep network is any high-precision deep learning network including a deep feedforward network, a deep belief network, a deep Boltzmann machine, and a deep convolutional network.

In the above technical solution, said step 4) further includes:

Step 4-1), calculate the output of each neuron through the hidden layer of the deep network, and its calculation formula is:

where, y _j is the output of the jth neuron, f is a given activation function, w _ij is the weight of the connection from the i-th neuron to the j-th neuron, and x _i is the weight from the i-th neuron The input, b _j is the bias weight;

Step 4-2), calculate the output of the sigmoid function; wherein, the expression of the sigmoid function is:

The output of the obtained sigmoid function is a P-dimensional vector (o ₁ ,o ₂ ,…,o _j ,…,o _P ), and each element in the vector o _j =sigmoid(y _j ) is between A real number between 0 and 1, where P is the total number of personalized tags;

Step 4-3), selecting a result from the output of the sigmoid function by soft maximum regression as a personalized response; wherein, the formula of soft maximum regression is as follows:

_The soft maximum regression represents the output of the _sigmoid function _{(o 1 , o 2} ,...,o _j ,...,o _P ) select one o _j as a personalized response.

The advantages of the present invention are:

The present invention converts a small amount of personalized samples into a large amount of data by introducing an auxiliary reconstruction network that supports one-time learning, thereby solving the problem that a small amount of personalized samples cannot directly train a high-precision deep network. At the same time, the method of the present invention effectively avoids the data transmission between the center and the edge. It expands the existing deep learning system and can be applied to the wide area network. It not only solves the problem of personalization of distributed data When processing information, it is difficult to improve the processing accuracy, and improve the transmission cost of the existing distributed big data real-time processing system.

Description of drawings

Fig. 1 is the flow chart of the dual-network deep learning method based on a small amount of individualized samples of the present invention;

Fig. 2 is a schematic structural diagram of the dual network involved in the present invention.

detailed description

The present invention will be further described now in conjunction with accompanying drawing.

As shown in Figure 1 and Figure 2, the dual-network deep learning method based on a small number of personalized samples of the present invention includes:

Step 1), collecting a small amount of personalized samples as the input of the dual network system; wherein, the total number of a small number of personalized samples is recorded as m, and the value range is m≥1; preferably, m≈M/J, where M is The total number of samples when using big data to train a deep network with the same precision, J is the number of samples reconstructed from a personalized sample. The collection of sample data in the collected personalized samples can be written as The value range of i is 1≤i≤m, and the superscript p means personalized.

In this step, in addition to the sample data itself, the collected personalized sample also includes the label corresponding to the sample. Personalized sample labels refer to any data The correct output of The collection of personalized sample labels can be written as The personalized sample collected in this step is a collection of sample data and labels, which can be written as

Step 2), using the sample data of the collected personalized samples to train the reconstruction network in the double network (denoted as Complete one-time learning of a small number of samples, and label the personalized samples (denoted as ) into the trained reconstruction network to generate a large amount of new reconstructed data and their labels, and record the new reconstructed data and their labels as in Any reconstructed data, is the label corresponding to the reconstructed data, the value of k is 1≤k≤m×J, m is the number of personalized samples, J is the number of samples reconstructed from any personalized sample, J and personalized The relationship between the number of samples m is as follows: the smaller the value of m, the larger the value of J, so as to ensure that there are enough reconstructed samples m×J to train the deep network.

Among them, the reconstruction network E _a ⁽ⁱ⁾ in the double network can be any learning network with the function of randomly reconstructing a large number of samples, including Bayesian network and non-hierarchical neural network; the reconstruction network E _a ^{(i )} is preferably a Bayesian program learning network with a one-time learning function; the Bayesian program learning network first splits the input data according to the prior knowledge obtained in the background knowledge base, and splits the input data into small Parts and tuples, and then recombine these small parts and tuples to generate new reconstructed data From the multiple generated reconstructed data, select J data with the highest similarity with the original data as the output of the module; where J is a positive integer, which is used to represent the number of samples reconstructed through a personalized sample.

Step 3), based on a large amount of new reconstruction data and its label collection from the reconstruction network Use the existing technology to train the deep network in the double network (denoted as E _d ⁽ⁱ⁾ );

Among them, the deep network E _d ⁽ⁱ⁾ in the dual network can be any kind of high-precision deep learning network, mainly including deep feedforward network, deep belief network, deep Boltzmann machine, and deep convolutional network.

Step 4), input personalized samples for testing or new data from the outside into the trained deep network E _d ⁽ⁱ⁾ , get the output of the sigmoid function, and then go through a soft maximum regression from the sigmoid function Select a result from the output as a personalized response.

Among them, the expression of the sigmoid function is:

Among them, y _j is the output of the jth neuron, and its value is calculated through the hidden layer of the deep network. The specific calculation formula is as follows:

where f is a given activation function, w _ij is the connection weight from i-th neuron to j-th neuron, x _i is the input from i-th neuron, and b _j is the bias weight.

The output of the sigmoid function is a P-dimensional vector (o ₁ ,o ₂ ,…,o _j ,…,o _P ), each element o _j =sigmoid(y _j ) in the vector is between 0 and 1 A real number between , where P is the total number of personalized tags.

Among them, the formula of soft maximum regression is as follows:

_The soft maximum _regression means that _{(o 1 , o 2} ,...,o _j ,...,o _P ) select an o _j as a personalized response.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limit them. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all of them should be included in the scope of the present invention. within the scope of the claims.

Claims (7)

1. a kind of dual network deep learning method based on a small amount of personalized sample, the dual network include reconstructed network and depth Network, this method include：

Step 1), a small amount of personalized sample of collection；Wherein, the personalized sample includes sample data and its label；

Step 2), using the sample data in personalized sample the reconstructed network is trained, then by the mark in personalized sample In reconstructed network of the label input after training, new reconstruct data and its label are generated；

Step 3), based on the new reconstruct data and its label training depth net from reconstructed network obtained by step 2) Network；

Step 4), by data input to be tested into trained depth network, obtain the output of sigmoid function, then pass through One soft maximum returns selects response of the result as personalization from the output of the sigmoid function.

2. the dual network deep learning method according to claim 1 based on a small amount of personalized sample, it is characterised in that institute The number for stating a small amount of personalized sample is designated as m, and m ≈ M/J, M therein are when training the depth network of same precision with big data Total sample number, J is by the number of the sample that property sample reconstructs one by one.

3. the dual network deep learning method according to claim 1 based on a small amount of personalized sample, it is characterised in that In step 2), the new reconstruct data and its set expression of label that are generated are { x^r _k,y^r _k}；Wherein,

x^r _kRepresent any one reconstruct data, y^r _kFor label corresponding to the reconstruct data, k value is 1≤k≤m × J, and m is individual Property number of samples, J is the number of sample reconstructed by any personalized sample；J and personalized sample number m relation It is as follows：M values are smaller, and J value is bigger, to ensure to have enough reconstructed sample m × J to train depth network.

4. the dual network deep learning method according to claim 1 based on a small amount of personalized sample, it is characterised in that institute It is to have random reconstruct great amount of samples including Bayesian network and without any one including leveled neural net to state reconstructed network The learning network of function.

5. the dual network deep learning method according to claim 4 based on a small amount of personalized sample, it is characterised in that institute It is Bayes's programmed instruction programmed learning network with disposable learning functionality to state reconstructed network.

6. the dual network deep learning method according to claim 1 based on a small amount of personalized sample, it is characterised in that institute It is any including depth feedforward network, depth confidence net, depth Boltzmann machine, depth convolutional network to state depth network One kind has high accuracy deep learning network.

7. the dual network deep learning method according to claim 1 based on a small amount of personalized sample, it is characterised in that institute Step 4) is stated to further comprise：

Step 4-1), calculate by the hidden layer of depth network the output of each neuron, its calculation formula is：

<mrow> <msub> <mi>y</mi> <mi>j</mi> </msub> <mo>=</mo> <mi>f</mi> <mrow> <mo>(</mo> <munder> <mo>&amp;Sigma;</mo> <mi>i</mi> </munder> <msub> <mi>w</mi> <mrow> <mi>i</mi> <mi>j</mi> </mrow> </msub> <msub> <mi>x</mi> <mi>i</mi> </msub> <mo>+</mo> <msub> <mi>b</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> </mrow>

Wherein, y_jIt is the output of j-th of neuron, f is a given activation primitive, w_ijIt is from i-th of neuron to j-th The connection weight of neuron, x_iIt is the input from i-th of neuron, b_jIt is biasing weight；

Step 4-2), calculate sigmoid function output；Wherein, the expression formula of sigmoid function is：

<mrow> <mi>s</mi> <mi>i</mi> <mi>g</mi> <mi>m</mi> <mi>o</mi> <mi>i</mi> <mi>d</mi> <mrow> <mo>(</mo> <msub> <mi>y</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mn>1</mn> <mrow> <mn>1</mn> <mo>+</mo> <mi>exp</mi> <mrow> <mo>(</mo> <msub> <mi>y</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> </mrow> </mfrac> <mo>;</mo> </mrow>

The output of resulting sigmoid function is the vector (o of a P dimension₁,o₂,…,o_j,…,o_P), vector in each element o_j=sigmoid (y_j) all it is real number between 0 and 1, P here is the sum of personalized labels；

Step 4-3), pass through soft maximum return and select a result be used as personalized response from the output of the sigmoid function； Wherein, the formula of soft maximum recurrence is as follows：

<mrow> <mi>s</mi> <mi>o</mi> <mi>f</mi> <mi>t</mi> <mi> </mi> <mi>m</mi> <mi>a</mi> <mi>x</mi> <mrow> <mo>(</mo> <msub> <mi>o</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mrow> <mi>exp</mi> <mrow> <mo>(</mo> <msub> <mi>o</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> </mrow> <mrow> <munder> <mo>&amp;Sigma;</mo> <mi>i</mi> </munder> <mi>exp</mi> <mrow> <mo>(</mo> <msub> <mi>o</mi> <mi>i</mi> </msub> <mo>)</mo> </mrow> </mrow> </mfrac> </mrow>

Soft maximum return is represented respectively with (softmax (o₁),softmax(o₂),…,softmax(o_j),…,softmax (o_P)) probability sigmoid function output (o₁,o₂,…,o_j,…,o_P) in choose an o_jResponded as personalization.