CN102982229A - Multi-assortment commodity price expectation data pre-processing method based on neural networks - Google Patents

Abstract

The invention discloses a multi-assortment commodity price expectation data pre-processing method based on neural networks. The best order of magnitude of commodity price data which is obtained from websites is calculated by an improved radical basis function (RBF) neural networks and an improved back propagation (BP) artificial neural networks. The calculated best order of magnitude is used to preprocess normalized order of magnitude of the commodity price. Expectation accuracy of the RBF neural networks and the BP neural networks is improved. Generality of the RBF neural networks and the BP neural networks for expectation of different kinds of commodity prices is improved.

Description

A data preprocessing method for multi-variety commodity price prediction based on neural network

technical field

The invention belongs to the field of data processing, in particular to a neural network-based data preprocessing method for multi-variety commodity price prediction, which can be applied to commodity price prediction data preprocessing in commodity price prediction analysis and commodity sales decision support systems.

Background technique

The commodity price prediction method is the basis of market forecast analysis and commodity production and sales decision-making. It is an important issue in the field of market forecasting and plays a key role in many problems such as commodity production and sales. The data preprocessing method It has a great influence on the generality and accuracy of the forecasting method. Due to the development of network technology and the popularization of online stores, people have paid more and more attention to the research of commodity price forecasting methods in recent years. The problem of commodity price forecasting can be regarded as a problem of data processing and data analysis based on time series, which is divided into three aspects: data acquisition, data processing and forecasting model. It is relatively easy to obtain public price data such as stock market, futures market, and electricity market. The models used for price prediction mainly include least squares regression, neural network, gray Markov chain, wavelet theory, and GM (1,1) model. Aiming at the acquisition method of consumer commodity price data, commodity price data preprocessing method and dynamic price forecasting, from 2010 to 2012, Zhu Quanyin et al. gave commodity sales data extraction and data mining methods and web-based commodity price preprocessing Methods and Dynamic Forecasting Methods (Quanyin Zhu, Yunyang Yan, Jin Ding and Yu Zhang. The Commodities Price Extracting for Shop Online, 2010 International Conference on Future Information Technology and Management Engineering, Changzhou, Jiangsu, Chian, Dec.2010, Vol.2, pp. 317-320; Quanyin Zhu, Yunyang Yan, Jin Ding and Jin Qian. The Case Study for Price Extracting of Mobile Phone SellOnline. IEEE 2nd International Conference on Software Engineering and Service Science, Beijing, Chian, July.2011, pp.281- 295; Quanyin Zhu, Sunqun Cao, Jin Ding and Zhengyin Hah. Research on the Price Forecast without Complete Data based on Web Mining, 2011 Distributed Computing and Applications to Business, Engineering and Science, Wuxi, Jiangsu, Chian, Oct. 102011, pp. 123; Quanyin Zhu, Hong Zhou, Yunyang Yan, Jin Qian and Pei Zhou. Commodities Price Dynamic Trend Analysis Based on Web Mining. The International Conference on Multimedia Information Networking and Security, Shanghai, Chian, No v.2011, pp.524-527; Jianping Deng, Fengwen Cao, Quanyin Zhu, and Yu Zhang. The Web Data Extracting and Application for Shop Online Based on Commodities Classified. Communications in Computer and Information Science, Vol.234(4): 189-197; Quanyin Zhu, Suqun Cao, Pei Zhou, Yunyang Yan, Hong Zhou. Integrated Price Forecast based on Dichotomy Backfilling and Disturbance Factor Algorithm. International Review on Computers and Software, 2011. Vol.6(6): 1089-1093; -yin Zhu, Pei Zhou, Yun-Yang Yan, Yong-Hua Yin. Exchange Rate Forecasting based on Adaptive Sliding Window and RBF NeuralNetwork. International Review on Computers and Software, 2011.Vol.6(7): 1290-1296; Jiajun Zong, Quanyin Zhu. Price Forecasting for Agricultural Products Based on BP and RBF Neural. ICSESS2012, p.607-610; Hong Zhou, Quanyin Zhu, Pei Zhou. A Hybrid Price Forecasting Based on Linear Backfilling and Sliding or view on Compiling Window. International Alg andSoftware, 2011.Vol.6(6): 1131-1134; Wang Hongyan, Zhu Quanyin, Yan Yunyang, Qian Jin. Comparison of Two WEB Mining Algorithms for Commodity Price Data. Microelectronics and Computers. 2011.Vol.28(19): 168 -172).

RBF (Radical Basis Function) neural network:

RBF is a feed-forward neural network, which simulates the neural network structure of the human brain with local adjustments and mutual coverage of the receptive field. It has a strong biological background and the ability to approximate any nonlinear function. It is a feed-forward network with a three-layer structure: the first layer is the input layer, which is composed of signal source nodes. The second layer is the hidden layer. The transformation function of the hidden unit is a locally distributed non-negative nonlinear function, which is radially symmetrical and attenuated to the center point. The number of units in the hidden layer is determined by the needs of the described problem. The third layer is the output layer, and the output of the network is the linear weighting of the hidden unit output. Among them, the input layer node only transmits the input signal to the hidden layer; the basis function of the hidden layer is nonlinear, and it produces a localized response to the input signal, that is, each hidden node has a parameter vector called the center . The center is used to compare with the network input vector to produce a radially symmetric response. Only when the input falls in a small specified area, the hidden node makes a meaningful non-zero response, and the response value is between 0 and 1. Between, the closer the distance between the input and the center of the basis function, the greater the response of the hidden node; the output unit is linear, that is, the output unit performs a linear weighted combination of the output of the hidden node.

BP (Back Propagation) neural network:

BP is a multilayer feed-forward network trained by the error backpropagation algorithm. It can learn and store a large number of input-output pattern mappings without revealing the mathematical equations describing such mappings in advance. Its learning rule is to use the steepest descent method to continuously adjust the weights and thresholds of the network through backpropagation to minimize the sum of squared errors of the network. BP neural network is a three-layer feedforward network, including input layer, hidden layer and output layer. Each neuron in the input layer is responsible for receiving input information from the outside world and passing it to each neuron in the middle layer; the middle layer is the internal information processing layer, which is responsible for information transformation. According to the requirements of information change capability, the middle layer can be designed as a single hidden layer or Multi-hidden layer structure; the information transmitted from the last hidden layer to each neuron in the output layer, after further processing, completes a forward propagation process of learning, and the output layer outputs information processing results to the outside world. When the actual output does not match the expected output, enter the error backpropagation stage. The error passes through the output layer, corrects the weights of each layer according to the error gradient descent method, and then propagates back to the hidden layer and input layer layer by layer. The repeated process of information forward propagation and error back propagation is a process of continuous adjustment of the weights of each layer, and also a process of neural network learning and training. This process continues until the error of the network output is reduced to an acceptable level, or the pre-set up to the specified number of studies.

When the above algorithms are used for price prediction, there is a great deal of uncertainty in both the prediction accuracy and the learning time of the algorithm. The uncertainty of some function parameters in MATLAB, the technical computing language used in the algorithm, increases the uncertainty of the algorithm learning time and prediction accuracy. This uncertainty makes the algorithm used in the prediction of commodity prices There are great limitations in . In order to make better use of the above algorithms, many improved price prediction methods have been proposed: k-means cluster stock price prediction based on BP neural network model; IPO underpricing prediction based on adaptive algorithm of BP neural network; Agricultural product price prediction model based on network time series model; an improved crude oil price prediction based on wavelet transform and RBF neural network; nonlinear time series prediction based on dynamic RBF neural network, etc. Among the improved prediction methods proposed, these prediction methods are highly pertinent and lack universality. The improved prediction methods are only applicable to one commodity or the same type of commodity, and the fixed parameters of the prediction method make the prediction method inflexible. , the accuracy of price prediction cannot be guaranteed when faced with the same type of different commodities. The lack of flexibility and versatility makes these improved forecasting methods unable to meet the urgent needs of the vast number of sellers for market forecast analysis and commodity sales decisions of different types of commodities. The prediction method of different commodity prices, or find a data preprocessing method for different types of commodity prices, so as to obtain better versatility and higher prediction accuracy of the prediction method.

Contents of the invention

The purpose of the present invention is to combine the normalized original data order of magnitude method with the improved RBF neural network and BP neural network prediction method, and utilize the improved RBF neural network and BP neural network to calculate its optimal order of magnitude for the commodity price data of web mining, Use the calculated optimal order of magnitude to preprocess the normalized data magnitude of the commodity price, and then use the improved RBF neural network and BP neural network to predict the commodity price, and improve the performance of the RBF neural network and BP neural network. Prediction accuracy, while improving the versatility of RBF neural network and BP neural network for different commodity price predictions.

The technical solution of the present invention is to preprocess the data excavated from the webpage through the method of normalizing the order of magnitude of the original data, and use the improved RBF neural network and BP neural network to calculate the commodity price on the data set after realizing the normalized order of magnitude The optimal magnitude of the data, using the calculated optimal magnitude to preprocess the normalized data magnitude of the commodity price, and then complete the market price prediction of the commodity.

For the convenience of understanding the scheme of the present invention, at first the theoretical basis of the present invention is described as follows:

In the field of price prediction based on neural network, many improved data preprocessing methods for price prediction have been proposed, and all of them have achieved obvious improvement effects. However, these improved methods are highly targeted, ignoring the flexibility and versatility of the forecasting method, which makes the improved price forecasting method have great limitations. The data preprocessing method of normalizing the order of magnitude of the original data can improve the versatility and prediction accuracy of the prediction method. The normalized original data order of magnitude method, for the price data of a commodity, relatively reduces the fluctuation range of the commodity price data, improves the stability of the prediction method, and improves the accuracy of the prediction method for the price prediction of the commodity; The price data of different commodities relatively reduces the difference between the price data of different commodities. At the same time, for a specific commodity, the fluctuation range of the commodity price data is relatively reduced, which improves the stability of the prediction method and enhances the generality of the prediction method. The improved RBF neural network and BP neural network are used to realize the price prediction of commodities on the normalized price data and obtain higher prediction accuracy of commodity prices.

Specifically, the scheme of the present invention realizes the normalized original data order of magnitude and the commodity price prediction of the improved RBF neural network and BP neural network through the following steps:

Step 1. Extract the name, model, type and price data of the commodities in the webpage, and establish a data set X={A ₁ , A ₂ ,...,A _h } with h commodities, and set the extracted price of the i-th commodity There are n pieces of data, A _i = {x ₁ , x ₂ , ..., x _n }, where i∈[1, h], x ₁ , x ₂ , ..., x _n refers to the item A _i The extracted n price data;

Step 2. Calculate the price magnitudes of i different commodities, and obtain the price magnitudes of different commodities M={b ₁ , b ₂ ,..., b _h };

Step 3. Customize a forecast sample that contains the number of data z, and the total number of predicted prices is D;

Step 4, select the prediction model;

Step 5, when the selected prediction model is RBF neural network, perform steps 6 to 12; when the selected prediction model is BP neural network, perform steps 14 to 21;

Step 6, set the model training function as the newrbe (P, T, SPREAD) function in the technical computing language MATLAB, which is used to design a strict radial basis network, where P is the input vector, T is the target vector, and SPREAD is the distribution of radial basis functions; the model prediction function is the sim('MODEL', PARAMETERS) function in the technical computing language MATLAB, which is used to simulate a neural network, where MODEL is the trained network model, and PARAMETERS is the input vector ;Set the distribution values of j different radial basis functions Spreads={spread ₁ , spread ₂ ,..., spread _j };

Step 7. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i , and obtain

Step 8. Bring the input vector P and the target vector T into the training function newrbe(P, T, SPREAD), train j different networks net _ij =newrbe(P, T, spread _j ), and establish a prediction sample Test=[t ₁ ,t ₂ ,...,t _z ],

Step 9. The j predicted value Y _ij of commodity A _i on day n+1 = sim(net _ij , Test), and the best predicted value of commodity A _i on day n+1 is y _i , y _i ∈ Y _ij ;

Step 10. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the distribution values of the three best forecast radial basis functions of commodity A _i on day n+1 as Bspread _i1 ∈ Spreads, Bspread _i2 ∈ Spreads, Bspread _i3 ∈ Spreads, find the value of the distribution of the best radial basis function $\mathrm{Bspread}=\frac{{\mathrm{<figure-callout\; id="1"\; label="Bspread\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bspread</figure-callout>}}_i*{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Bespread\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bespread</figure-callout>}}_i*{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+{\mathrm{Bspread}}_{i3}*w_3}{{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+w_3};$

Step 11, training invariant network net=newrbe(P, T, Bspread);

Step 12. Bring in the best forecast value y _i as the forecast sample for the next forecast. The method is t ₁ = last forecast sample [t ₁ ] in the new forecast sample [t ₁ , t ₂ , ..., t _z ] _{. _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., t _z- 1 in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t ₁ , t ₂ , .. ., t _z ], t _z in the new prediction sample [t ₁ , t ₂ ,..., t _z ], t _z =y _i , and get a new prediction sample Test=[t ₁ , t ₂ ,. .., t _z ], the predicted value of the product on day n+2 yi=sim(net, Test);

Step 13. Repeat step 12 to obtain all predicted values of commodity A _i ; repeat steps 7 to 12 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M;

Step 14, setting the model training function as the NET=newff(P, T, NEURON) function and NET'=train(NET, P, T) function in the technical computing language MATLAB, wherein the newff() function is used to create a previous Feed BP network, P is the input vector, T is the target vector, NEURON is the number of neurons in the hidden layer, the train() function is used to train a neural network, NET is the created feed-forward BP network; the model prediction function is NET' (Test), wherein Test is a prediction sample; The value Neurons={neuron ₁ , neuron ₂ ,..., neuron _j } of j different hidden layer neuron numbers is set;

Step 15. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i to obtain

Step 16, input vector P, target vector T are brought into training function NET=newff(P, T, NEURON) and NET'=train(NET, P, T), training just j different networks net _ij =newff(P , T, Neurons), net _ij = train(net _ij , P, T); build prediction sample Test=[t ₁ , t ₂ ,..., t _z ],

Step 17. The j predicted value Y _ij of commodity A _i on day n+1 = net _i (Test), assuming that the best predicted value of commodity A _i on day n+1 is y _i , y _i ∈ _{Y ij} ;

Step 18. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the value of the three best predicted hidden layer neurons on day n+1 of commodity A _i as Bneuron _i1 ∈ Neurons, Bneuron _i2 ∈ Neurons, Bneuron _i3 ∈ Neurons, find the value of the optimal number of neurons in the hidden layer $\mathrm{Bneuron}=\frac{{\mathrm{<figure-callout\; id="1"\; label="Bneuron\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bneuron</figure-callout>}}_i*{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Bneuron\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bneuron</figure-callout>}}_i*{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+{\mathrm{Bneuron}}_{i3}*w_3}{{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+w_3};$

Step 19, training invariant network net=newff (P, T, Bneuron), net=train (net, P, T);

Step 20, bring in the best predicted value y _i as the forecast sample for the next forecast, the method is t ₁ in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t _{1 . _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., t _z- 1 in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t ₁ , t ₂ , .. ., t _z ], t _z in the new prediction sample [t ₁ , t ₂ ,..., t _z ], t _z =y _i , and get a new prediction sample Test=[t ₁ , t ₂ ,. .., t _z ], the predicted value y _i =net(Test) of the commodity on the n+2th day;

Step 21. Repeat step 20 to obtain all predicted values of commodity A _i ; repeat steps 15 to 20 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M.

Extracting the name, model, type and price data of the commodity in the webpage mentioned in step 1 refers to extracting the name, model, type and price data of the commodity displayed on the webpage by using any web data extraction algorithm; where x ₁ , x ₂ ,..., x _n can be n pieces of price data extracted from one webpage for the i-th commodity A _i , or can be n pieces of average price data extracted from multiple webpages.

Step 2 is to calculate the price data of any commodity to obtain the magnitude of the commodity price data.

Steps 3 to 5 are for the parameter setting and forecasting model selection of any commodity in price forecasting, where the z value is generally 3, 5, 7, and the D value is generally 3, 7.

The technical computing language MATLAB in step 6 and step 14 is a product of MathWorks, and its version is R2011b.

Steps 6 to 12 are the predicted value of the price data of any commodity on different dates in a webpage under the improved RBF neural network, or the average price data of different dates in multiple webpages under the improved RBF neural network Predictive value.

Steps 14 to 20 are the predicted value of the price data of any commodity on different dates in a webpage under the improved BP neural network, or the average price data of different dates in multiple webpages under the improved BP neural network Predictive value.

The input vector P in step 6, step 8, step 14 and step 16 is the training sample set, and the target vector T is the data set of the training test prediction value.

The preset value of j in step 6 is generally 40, and the preset value of j in step 14 is generally 10.

In step 7 and step 15, the magnitude of the price data of any commodity is normalized to a unified magnitude, the magnitude of the price data of the commodity is the same as the normalized magnitude, and the magnitude of the price data of the commodity is not normalized Quantitative magnitude preprocessing; the magnitude of the commodity price data is different from the normalized magnitude. The magnitude of the commodity price data is preprocessed with normalized magnitude. The magnitude is generally 1, 10, 100, 1000.

Coupling weight w=[2, 4, 2] defined in step 10 and step 18.

Compared with the data preprocessing methods in various price predictions of the prior art, the present invention selects the price data of webpage commodities mined, and uses the improved RBF neural network and BP neural network to calculate the optimal magnitude of the original data of commodity prices, Using the calculated optimal order of magnitude, the original data of the commodity price is processed in a unified normalized order of magnitude; the preprocessing method of the normalized order of magnitude of the original data of the present invention is used for a certain commodity, which reduces the The fluctuation range of commodity price data; for different commodities, the difference between the price data of different commodities is reduced, which makes up for the limitations of the existing price prediction method when the data preprocessing method is applied to the price prediction of different commodities, and improves the prediction The versatility of the method improves the prediction accuracy at the same time.

Description of drawings

Fig. 1 is a flowchart of a specific embodiment of the present invention.

Detailed ways

The technical scheme of the present invention is described in detail below in conjunction with accompanying drawing:

As shown in accompanying drawing 1, embodiment of the present invention carries out according to the following steps:

Step 1. Extract the name, model, type and price data of the commodities in the webpage, and establish a data set X={A ₁ , A ₂ ,...,A _h } with h commodities, and set the extracted price of the i-th commodity There are n pieces of data, A _i = {x ₁ , x ₂ , ..., x _n }, where i∈[1, h], x ₁ , x ₂ , ..., x _n refers to the item A _i The extracted n price data;

Step 2. Calculate the price magnitudes of i different commodities, and obtain the price magnitudes of different commodities M={b ₁ , b ₂ ,..., b _h };

Step 3. Customize a forecast sample that contains the number of data z, and the total number of predicted prices is D;

Step 4, select the prediction model;

Step 5, when the selected prediction model is RBF neural network, perform steps 6 to 12; when the selected prediction model is BP neural network, perform steps 14 to 21;

Step 6, set the model training function as the newrbe (P, T, SPREAD) function in the technical computing language MATLAB, which is used to design a strict radial basis network, where P is the input vector, T is the target vector, and SPREAD is the distribution of radial basis functions; the model prediction function is the sim('MODEL', PARAMETERS) function in the technical computing language MATLAB, which is used to simulate a neural network, where MODEL is the trained network model, and PARAMETERS is the input vector ;Set the distribution values of j different radial basis functions Spreads={spread ₁ , spread ₂ ,..., spread _j };

Step 7. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i , and obtain

Step 8. Bring the input vector P and the target vector T into the training function newrbe(P, T, SPREAD), train j different networks net _ij =newrbe(P, T, spread _j ), and establish a prediction sample Test=[t ₁ ,t ₂ ,...,t _z ],

Step 9. The j predicted value Y _ij of commodity A _i on day n+1 = sim(net _ij , Test), and the best predicted value of commodity A _i on day n+1 is y _i , y _i ∈ Y _ij ;

Step 10. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the distribution values of the three best predicted radial basis functions of commodity A _i on day n+1 as Bspread _i1 ∈ Spreads, Bspreadd _i2 ∈ Spreads, Bspread _i3 ∈ Spreads, find the value of the distribution of the best radial basis function $\mathrm{Bspread}=\frac{{\mathrm{<figure-callout\; id="1"\; label="Bspread\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bspread</figure-callout>}}_i*{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Bespread\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bespread</figure-callout>}}_i*{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+{\mathrm{Bspread}}_{i3}*w_3}{{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+w_3};$

Step 11, training invariant network net=newrbe(P, T, Bspread);

Step 12. Bring in the best forecast value y _i as the forecast sample for the next forecast. The method is t ₁ = last forecast sample [t ₁ ] in the new forecast sample [t ₁ , t ₂ , ..., t _z ] _{. _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., new forecast sample [t ₁ , t ₂ , ..., t _z ]] in t _z-1 = last forecast sample [t ₁ , t ₂ , . t _z in .., t _{z ]} , t _z = y _i in the new prediction sample [t ₁ , t ₂ , ..., t _z ], and get a new prediction sample Test=[t ₁ , t ₂ , ..., t _z , the predicted value y _i =sim(net, Test) of the product on the n+2th day;

Step 13. Repeat step 12 to obtain all predicted values of commodity A _i ; repeat steps 7 to 12 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M;

Step 14, setting the model training function as the NET=newff(P, T, NEURON) function and NET'=train(NET, P, T) function in the technical computing language MATLAB, wherein the newff() function is used to create a previous Feed BP network, P is the input vector, T is the target vector, NEURON is the number of neurons in the hidden layer, the train() function is used to train a neural network, NET is the created feed-forward BP network; the model prediction function is NET' (Test), wherein Test is a prediction sample; The value Neurons={neuron ₁ , neuron ₂ ,..., neuron _j } of j different hidden layer neuron numbers is set;

Step 15. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i to obtain

Step 16, input vector P, target vector T are brought into training function NET=newff(P, T, NEURON) and NET'=train(NET, P, T), training just j different networks net _ij =newff(P , T, Neurons), net _ij = train(net _ij , P, T); build prediction sample Test=[t ₁ , t ₂ ,..., t _z ],

Step 17. The j predicted value Y _ij of commodity A _i on day n+1 = net _i (Test), assuming that the best predicted value of commodity A _i on day n+1 is y _i , y _i ∈ _{Y ij} ;

Step 18. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the value of the three best predicted hidden layer neurons on day n+1 of commodity A _i as Bneuron _i1 ∈ Neurons, Bneuron _i2 ∈ Neurons, Bneuron _i3 ∈ Neurons, find the value of the optimal number of neurons in the hidden layer $\mathrm{Bneuron}=\frac{{\mathrm{<figure-callout\; id="1"\; label="Bneuron\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bneuron</figure-callout>}}_i*{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="Bneuron\; i"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">Bneuron</figure-callout>}}_i*{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+{\mathrm{Bneuron}}_{i3}*w_3}{{\mathrm{<figure-callout\; id="1"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000773631800011.png"\; state="\{\{state\}\}">w</figure-callout>}}_2+w_3};$

Step 19, training invariant network net=newff (P, T, Bneuron), net=train (net, P, T);

Step 20, bring in the best predicted value y _i as the forecast sample for the next forecast, the method is t ₁ in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t _{1 . _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., t _z- 1 in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t ₁ , t ₂ , .. ., t _z ], t _z in the new prediction sample [t ₁ , t ₂ ,..., t _z ], t _z =y _i , and get a new prediction sample Test=[t ₁ , t ₂ ,. .., t _z ], the predicted value y _i =net(Test) of the commodity on the n+2th day;

Step 21. Repeat step 20 to obtain all predicted values of commodity A _i ; repeat steps 15 to 20 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M.

Extracting the name, model, type and price data of the commodity in the webpage mentioned in step 1 refers to extracting the name, model, type and price data of the commodity displayed on the webpage by using any web data extraction algorithm; where x ₁ , x ₂ ,..., x _n can be n pieces of price data extracted from one webpage for the i-th commodity A _i , or can be n pieces of average price data extracted from multiple webpages.

Step 2 is to calculate the price data of any commodity to obtain the magnitude of the commodity price data.

Steps 3 to 5 are for the parameter setting and forecasting model selection of any commodity in price forecasting, where the z value is generally 3, 5, 7, and the D value is generally 3, 7.

The technical computing language MATLAB in step 6 and step 14 is a product of MathWorks, and its version is R2011b.

Steps 6 to 12 are the predicted value of the price data of any commodity on different dates in a webpage under the improved RBF neural network, or the average price data of different dates in multiple webpages under the improved RBF neural network Predictive value.

: Step 14 to step 20 is the predicted value under the improved BP neural network for the price data of any commodity on different dates in a webpage, or the average price data of different dates in multiple webpages under the improved BP neural network predicted value of .

The input vector P in step 6, step 8, step 14 and step 16 is the training sample set, and the target vector T is the data set of the training test prediction value.

The preset value of j in step 6 is generally 40, and the preset value of j in step 14 is generally 10.

In step 7 and step 15, the magnitude of the price data of any commodity is normalized to a unified magnitude, the magnitude of the price data of the commodity is the same as the normalized magnitude, and the magnitude of the price data of the commodity is not normalized Quantitative magnitude preprocessing; the magnitude of the commodity price data is different from the normalized magnitude. The magnitude of the commodity price data is preprocessed with normalized magnitude. The magnitude is generally 1, 10, 100, 1000.

Coupling weight w=[2, 4, 2] defined in step 10 and step 18.

In order to better illustrate the effectiveness of this method, the daily average price data of 8 different RMB exchange rates extracted from the webpage from January 1, 2011 to December 31, 2011 were used as the original data to calculate the original data The order of magnitude is 1, 10 and 100, and the normalization experiment is performed on the order of magnitude of the original data.

In the experimental environment of the improved RBF neural network, without normalization magnitude preprocessing, the experimental results of the original data are: the average error of the Australian dollar is 3.9%, the average error of the Hong Kong dollar is 11.82%, and the average error of the Canadian dollar is 3.9%. The average error is 640.84%, the average error of USD is 21571.04%, the average error of EUR is 1.66%, the average error of JPY is 1.15%, the average error of CHF is 2.77%, and the average error of SGD is 28959.17%. The average error is 6399.04%; when the data magnitude is normalized to 100, the experimental results are: the average error of the Australian dollar is 3.9%, the average error of the Hong Kong dollar is 1.97%, the average error of the Canadian dollar is 640.84%, and the average error of the US dollar is 640.84%. The average error is 21571.04%, the average error of the euro is 1.66%, the average error of the Japanese yen is 1233177%, the average error of the Swiss franc is 2.77%, the average error of the Singapore dollar is 28959.17%, and the average error of the experiment is 160544.8%; When the data magnitude is normalized to 10, the experimental results are: the average error of the Australian dollar is 1.77%, the average error of the Hong Kong dollar is 0.94%, the average error of the Canadian dollar is 0.39%, the average error of the US dollar is 0.11%, and the average error of the euro is 0.94%. The average error is 224438.5%, the average error is 1.05% for Japanese yen, 1.57% for Swiss franc, 0.50% for Singapore dollar, and 28055.6% for the experiment; normalized to 1 in the data magnitude , the experimental results are: the average error of the Australian dollar is 0.98%, the average error of the Hong Kong dollar is 0.94%, the average error of the Canadian dollar is 0.41%, the average error of the US dollar is 0.09%, the average error of the euro is 1.12%, the Japanese yen The average error is 1.29% for the Swiss franc, 1.91% for the Swiss franc, 0.16% for the Singapore dollar, and 0.86% for the experiment. The conclusion is that when the data magnitude is normalized to 1, the best prediction result is obtained, and the average prediction accuracy rate reaches 99.14%.

In the experimental environment of the improved BP neural network, without normalization magnitude preprocessing, the experimental results of the original data are: the average error of the Australian dollar is 1.31%, the average error of the Hong Kong dollar is 0.17%, and the average error of the Canadian dollar is 1.31%. The average error is 0.28%, the average error for USD is 0.26%, the average error for EUR is 1.41%, the average error for JPY is 1.24%, the average error for CHF is 1.65%, and the average error for SGD is 0.21%. The average error is 0.82%; when the order of magnitude is normalized to 100, the experimental results are: the average error of the Australian dollar is 1.31%, the average error of the Hong Kong dollar is 0.24%, the average error of the Canadian dollar is 0.46%, and the average error of the US dollar is 0.03%, the average error of the Euro is 2.21%, the average error of the Japanese Yen is 1.14%, the average error of the Swiss Franc is 1.59%, the average error of the Singapore Dollar is 2.98%, and the average error of the experiment is 1.25%. When level normalization is 10, the experimental results are: the average error of Australian dollar is 1.09%, the average error of Hong Kong dollar is 0.28%, the average error of Canadian dollar is 0.13%, the average error of US dollar is 0.48%, and the average error of euro is 1.38%, the average error of the Japanese Yen is 2.54%, the average error of the Swiss Franc is 1.93%, the average error of the Singapore Dollar is 0.06%, and the average error of the experiment is 0.99%. When the data magnitude is normalized to 1, The experimental results are: the average error of the Australian dollar is 0.39%, the average error of the Hong Kong dollar is 0.18%, the average error of the Canadian dollar is 0.37%, the average error of the US dollar is 0.40%, the average error of the euro is 1.43%, and the average error of the Japanese yen The error is 1.18%, the average error is 1.74% for Swiss francs, 0.28% for Singapore dollars, and 0.75% for experiments. The conclusion is that when the data magnitude is normalized to 1, the best prediction result is obtained, and the average prediction accuracy rate is as high as 99.25%.

The above experimental data shows the generality of this data preprocessing method for different commodities of the same type. In order to illustrate the generality of this data preprocessing method for different types of commodities, 10 different agricultural products extracted from The weekly average price data of 59 weeks in February 2012 was used as the original data, and the order of magnitude of the original data was calculated as 1 and 10, and the normalization preprocessing experiment was carried out on the order of magnitude of the original data.

In the experimental environment of the improved RBF neural network, without normalization magnitude preprocessing, the experimental results of the original data are: the average error of beef is 3149934%, the average error of soybean oil is 17.96%, and the average error of eggs The average error of peanut oil is 1.61%, the average error of peanut oil is 2.89%, the average error of flour is 0.11%, the average error of pork is 542574.4%, the average error of rice is 0.34%, the average error of white sugar is 0.44%, the average error of blended oil is 6.61%, the average error of mutton is 325260%, and the average error of the experiment is 401779.9%; when the data magnitude is normalized to 10, the experimental results are: the average error of beef is 3149934%, and the average error of soybean oil is 17.96% , the average error of eggs is 1.61%, the average error of peanut oil is 2.89%, the average error of flour is 0.12%, the average error of pork is 542574.4%, the average error of rice is 0.34%, and the average error of white sugar is 0.44%, The average error of blended oil is 6.61%, the average error of mutton is 325260%, and the average error of experiment is 401779.9%. When the data level is normalized to 1, the experimental results are: the average error of beef is 2.44%, the The average error is 17.96%, the average error of eggs is 1.61%, the average error of peanut oil is 0.91%, the average error of flour is 0.11%, the average error of pork is 7.35%, the average error of rice is 0.34%, and the average error of white sugar The error is 0.44%, the average error of blended oil is 0.13%, the average error of mutton is 0.41%, and the average error of experiment is 3.17%. The conclusion is that the experiment when the data magnitude is normalized to 1 has achieved the best prediction results, and the average prediction accuracy rate reaches 96.83%.

The invention can be combined with a computer system to automatically complete the forecast of commodity prices.

The present invention creatively proposes a data preprocessing method based on neural network multi-variety commodity price prediction, and applies the data preprocessing method to the preprocessing of commodity price data such as RMB exchange rate and agricultural products, and utilizes the improved RBF neural network Using the BP neural network to predict commodity prices on the preprocessed price data improves the versatility of the prediction method and obtains higher prediction accuracy, which has high practical value.

A neural network-based data preprocessing method for multi-variety commodity price prediction proposed by the present invention can not only be used for data preprocessing in RMB exchange rate and agricultural product production and sales field price prediction, but also can be used for other consumer commodity price prediction time data preprocessing.

Claims (11)

1. A data preprocessing method based on neural network-based multi-species commodity price prediction, characterized in that: utilize improved RBF neural network and BP neural network to calculate its optimal order of magnitude for the commodity price data of webpage mining, and obtain The optimal order of magnitude preprocesses the normalized data magnitude of commodity prices, thereby improving the prediction accuracy of RBF neural network and BP neural network, and also improving the performance of RBF neural network and BP neural network for price prediction of different commodities Versatility, specifically including the following steps: Step 1. Extract the name, model, type and price data of commodities in the webpage, and establish a data set X={X ₁ , A ₂ ,...,A _h } with h commodities, and set the extracted price of the i-th commodity There are n pieces of data, A _i = {x ₁ , x ₂ , ..., x _n }, where i∈[1, h], x ₁ , x ₂ , ..., x _n refers to the item A _i The extracted n price data; Step 2. Calculate the price magnitudes of i different commodities, and obtain the price magnitudes of different commodities M={b ₁ , b ₂ ,..., b _h }; Step 3. Customize a forecast sample that contains the number of data z, and the total number of predicted prices is D; Step 4, select the prediction model; Step 5, when the selected prediction model is RBF neural network, perform steps 6 to 12; when the selected prediction model is BP neural network, perform steps 14 to 21; Step 6, set the model training function as the newrbe (P, T, SPREAD) function in the technical computing language MATLAB, which is used to design a strict radial basis network, where P is the input vector, T is the target vector, and SPREAD is the distribution of radial basis functions; the model prediction function is the sim('MODEL', PARAMETERS) function in the technical computing language MATLAB, which is used to simulate a neural network, where MODEL is the trained network model, and PARAMETERS is the input vector ;Set the distribution values of j different radial basis functions Spreads={spread ₁ , spread ₂ ,..., spread _j }; Step 7. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i , and obtain

Step 8. Bring the input vector P and the target vector T into the training function newrbe(P, T, SPREAD), train j different networks net _ij =newrbe(P, T, spread _j ), and establish a prediction sample Test=[t ₁ ,t ₂ ,...,t _z ],

Step 9. The j predicted value Y _ij of commodity A _i on day n+1 = sim(net _ij , Test), and the best predicted value of commodity A _i on day n+1 is y _i , t _i ∈ Y _ij ; Step 10. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the distribution values of the three best forecast radial basis functions of commodity A _i on day n+1 as Bspread _i1 ∈ Spreads, Bspread _i2 ∈ Spreads, Bspread _i3 ∈ Spreads, find the value of the distribution of the best radial basis function $\mathrm{Bspread}=\frac{{\mathrm{Bspread}}_{i1}*w_1+{\mathrm{Bespread}}_{i2}*w_2+{\mathrm{Bspread}}_{i3}*w_3}{w_1+w_2+w_3};$ Step 11, training invariant network net=newrbe(P, T, Bspread); Step 12. Bring in the best forecast value y _i as the forecast sample for the next forecast. The method is t ₁ = last forecast sample [t ₁ ] in the new forecast sample [t ₁ , t ₂ , ..., t _z ] _{. _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., t _z- 1 in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t ₁ , t ₂ , .. ., t _z ] in t _z , in the new forecast sample [t ₁ , t ₂ ,..., t _z ]], t _z =y _i , get the new forecast sample Test=[t ₁ , t ₂ , ..., t _z ], the predicted value y _i of the product on the n+2th day = sim(net, Test); Step 13. Repeat step 12 to obtain all predicted values of commodity A _i ; repeat steps 7 to 12 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M; Step 14, setting the model training function as the NET=newff(P, T, NEURON) function and NET'=train(NET, P, T) function in the technical computing language MATLAB, wherein the newff() function is used to create a previous Feed BP network, P is the input vector, T is the target vector, NEURON is the number of neurons in the hidden layer, the train() function is used to train a neural network, NET is the created feed-forward BP network; the model prediction function is NET' (Test), wherein Test is a prediction sample; The value Neurons={neuron ₁ , neuron ₂ ,..., neuron _j } of j different hidden layer neuron numbers is set; Step 15. Normalize the order of magnitude of the sales price of commodity A _i to order of magnitude b _i to obtain

Step 16, input vector P, target vector T are brought into training function NET=newff(P, T, NEURON) and NET'=train(NET, P, T), training just j different networks net _ij =newff(P , T, Neurons), net _ij = train(net _ij , P, T); build prediction sample Test=[t ₁ , t ₂ ,..., t _z ],

Step 17. The j predicted value Y _ij of commodity A _i on day n+1 = net _i (Test), assuming that the best predicted value of commodity A _i on day n+1 is y _i , y _i ∈ _{Y ij} ; Step 18. Define the coupling weight W=(w ₁ , w ₂ , w ₃ ), and set the value of the three best predicted hidden layer neurons on day n+1 of commodity A _i as Bneuron _i1 ∈ Neurons, Bneuron _i2 ∈ Neurons, Bneuron _i3 ∈ Neurons, find the value of the optimal number of neurons in the hidden layer $\mathrm{Bneuron}=\frac{{\mathrm{Bneuron}}_{i1}*w_1+{\mathrm{Bneuron}}_{i2}*w_2+{\mathrm{Bneuron}}_{i3}*w_3}{w_1+w_2+w_3};$ Step 19, training invariant network net=newff (P, T, Bneuron), net=train (net, P, T); Step 20, bring in the best predicted value y _i as the forecast sample for the next forecast, the method is t ₁ in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t _{1 . _ _ _ _ _ _ _ _} .., t _z ] in t ₃ , ..., t _z- 1 in the new forecast sample [t ₁ , t ₂ , ..., t _z ] = last forecast sample [t ₁ , t ₂ , .. ., t _z ], t _z in the new prediction sample [t ₁ , t ₂ ,..., t _z ], t _z =y _i , and get a new prediction sample Test=[t ₁ , t ₂ ,. .., t _z ], the predicted value y _i =net(Test) of the commodity on the n+2th day; Step 21. Repeat step 20 to obtain all predicted values of commodity A _i ; repeat steps 15 to 20 to obtain predicted values of all commodities in data set X at different orders of magnitude, and obtain the best predicted order of magnitude O, O∈M. 2. The data preprocessing method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: the name, model, type and price data of the commodity in the web page extracted as described in step 1 refer to , use any web data extraction algorithm to extract the name, model, type and price data of the product displayed on the web page; where x ₁ , x ₂ ,..., x _n can be the i-th product A _i extracted from a web page n pieces of price data, or n pieces of average price data extracted from multiple web pages. 3. A neural network-based data preprocessing method for multi-variety commodity price prediction according to claim 1, characterized in that: step 2 is to calculate the price data of any commodity to obtain the magnitude of the commodity price data. 4. A neural network-based data preprocessing method for multi-species commodity price prediction according to claim 1, characterized in that: Steps 3 to 5 are for parameter setting and prediction of any commodity in price prediction The model is selected, where the z value is generally 3, 5, 7, and the D value is generally 3, 7. 5. The data preprocessing method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: in step 6 and step 14, the technical computing language MATLAB is a product of MathWorks, and its version is R2011b. 6. The data preprocessing method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: step 6 to step 12 is for any one commodity in a webpage for price data of different dates in The predicted value under the improved RBF neural network, or the predicted value of the average price data on different dates in multiple web pages under the improved RBF neural network. 7. The data preprocessing method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: step 14 to step 20 is for any one commodity in a webpage for price data of different dates in The predicted value under the improved BP neural network, or the predicted value of the average price data of different dates in multiple web pages under the improved BP neural network. 8. The data processing method of a kind of neural network-based multi-species commodity price prediction according to claim 1, characterized in that: the input vector P in step 6, step 8, step 14 and step 16 is a training sample set , and the target vector T is the dataset for training and testing predictors. 9. The data processing and sending method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: the preset j value in step 6 is generally 40, and the preset j value in step 14 is generally 40. The value of j is generally 10. 10. The data processing method of a kind of neural network-based multi-variety commodity price prediction according to claim 1, characterized in that: in step 7 and step 15, the order of magnitude of the price data of any commodity is normalized to a unified The magnitude of the commodity's price data is the same as the normalized magnitude, and the magnitude of the commodity's price data is not subjected to normalized magnitude preprocessing; the magnitude of the commodity's price data is different from the normalized magnitude , the order of magnitude of the price data of the product is preprocessed with normalized order of magnitude, and the order of magnitude is generally 1, 10, 100, 1000. 11. A neural network-based data preprocessing method for multi-variety commodity price prediction according to claim 1, characterized in that: the coupling weight w=[2, 4, 2] defined in step 10 and step 18.

Images