CN107832913A - The Forecasting Methodology and system to monitoring data trend based on deep learning - Google Patents
The Forecasting Methodology and system to monitoring data trend based on deep learning Download PDFInfo
- Publication number
- CN107832913A CN107832913A CN201710941098.4A CN201710941098A CN107832913A CN 107832913 A CN107832913 A CN 107832913A CN 201710941098 A CN201710941098 A CN 201710941098A CN 107832913 A CN107832913 A CN 107832913A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- monitoring
- time
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0637—Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
- G06Q10/06375—Prediction of business process outcome or impact based on a proposed change
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Strategic Management (AREA)
- Educational Administration (AREA)
- Quality & Reliability (AREA)
- General Business, Economics & Management (AREA)
- Game Theory and Decision Science (AREA)
- Development Economics (AREA)
- Tourism & Hospitality (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present invention relates to computer control automation field, and in particular to the Forecasting Methodology and system to monitoring data trend based on deep learning, methods described include：Off-line training system extracts some data from real-time streaming data and forms data set, and data set is carried out forecast model is calculated；Real-time on-line system extracts some data from real-time streaming data and forms forecast set, and forecast set is calculated using forecast model, obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is sky.In the present invention, off-line training system obtains forecast model, then real-time on-line system can be monitored the forecast model combination real-time streaming data prediction result of the following flow data of system by analyzing real-time streaming data.This result can free many energy and times as a good reference of following monitoring data trend for our operation maintenance personnel.
Description
Technical field
The present invention relates to computer control automation field, and in particular to based on deep learning to the pre- of monitoring data trend
Survey method and system.
Background technology
Existing O＆M monitoring system is visualized by the time series of statistical history data, allows operation maintenance personnel
The fluctuation of index can be clearly seen so as to analysis result, when the numerical value of display is up to or over designated value, it is believed that the quilt
The system of monitoring occurs abnormal.But current method is only that historical data is shown, to the rule of data itself
Excavation and the analysis of trend in the future be not directed to.
Obvious abnormal, the obvious problem that persistently deviates of some rings than yesterday occurs in the operational indicator data of daily O＆M
Over time the problems such as the achievement data of cycle shift.The configuration of these monitoring depended on engineer experience or lasting substantially in the past
Iterated revision, in addition it is pure manually investigate, still, in face of substantial amounts of monitoring data, a time operation maintenance personnel may be difficult to find
Potential alarm point can not be found where problem or directly.On the other hand, a large amount of monitoring datas caused by monitoring system are not
There is the value for playing it completely, these data can be used to find and excavate may wherein be joined by certain used in us
System.
Conventional method (ARIMA, Holter winter) does not often make us full for the prediction effect of monitoring data trend
Meaning, if a technological difficulties of ARIMA algorithms are exactly the tranquilization of time series, the time series of tranquilization is for prediction result
Quality play vital effect.Another problem is that the result that moving average operation is brought is as obvious in fluctuated sawtooth, is held
The change of wrong report interference is easily caused, then increases the purpose monitoring cycle.
With the development of monitoring system, the standard of O＆M can be realized by formulating Standard of Monitoring and automatically-monitored deployment
Change and automation, final target are desirable to be settled the matter once and for all with intelligentized method.
The content of the invention
The technical problem to be solved in the present invention is, overcomes the shortcomings of existing technology, there is provided pair based on deep learning
The Forecasting Methodology and system of monitoring data trend, it can predict the trend result of following monitoring data from monitoring data,
This result can be that the trend of following monitoring data makees good reference, so as to improve the effect of work for monitoring system operation maintenance personnel
Rate and accuracy.
To reach above-mentioned technical purpose, on the one hand, it is of the present invention based on deep learning to monitoring data trend
Forecasting Methodology, including：
Off-line training system extracts some data from real-time streaming data and forms data set, and data set calculate
To forecast model；
Real-time on-line system extracts some data from real-time streaming data and forms forecast set, and using forecast model to prediction
Collection is calculated, and obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is sky.
On the other hand, the forecasting system to monitoring data trend of the present invention based on deep learning, including：
Off-line training system, data set is formed for extracting some data from real-time streaming data, and data set is carried out
Forecast model is calculated；
Real-time on-line system, forecast set is formed for extracting some data from real-time streaming data, and use forecast model
Forecast set is calculated, obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set
For sky.
In the present invention, off-line training system obtains forecast model, then real-time online system by analyzing real-time streaming data
System can be monitored the forecast model combination real-time streaming data prediction result of the following flow data of system.This result can be with
As a good reference of following monitoring data trend, many energy and times are freed for our operation maintenance personnel.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.
Fig. 1 is the method flow schematic diagram of one embodiment of the invention；
Fig. 2 is the method flow schematic diagram of another embodiment of the present invention；
Fig. 3 is the system structure diagram of one embodiment of the invention；
Fig. 4 is the system structure diagram of another embodiment of the present invention；
Fig. 5 is the structural representation of off-line training system in the embodiment of the present invention；
Fig. 6 is the general frame schematic diagram of the present invention；
Fig. 7 is the workflow diagram of off-line training system in the embodiment of the present invention；
Fig. 8 is a cellular construction figure of computation model in the embodiment of the present invention；
Fig. 9 is the result displaying figure of data set in the embodiment of the present invention；
Figure 10 is the fitting effect of training set and the prediction result of test set displaying figure in the embodiment of the present invention, wherein, it is shallow
Color curve is the fitting effect of training set, and darker curve is the prediction result of test set；
Figure 11 is the result of maximum and minimum value displaying figure in the data set of the embodiment of the present invention, wherein dark to be maximum
The result of value, light color are the result of minimum value.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
As shown in figure 1, as one embodiment, it is provided by the invention based on deep learning to the pre- of monitoring data trend
Survey method, including：
101st, off-line training system extracts some data from real-time streaming data and forms data set, and data set is counted
Calculation obtains forecast model；
102nd, real-time on-line system extracts some data from real-time streaming data and forms forecast set, and uses forecast model pair
Forecast set is calculated, and obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is
It is empty.
As shown in Fig. 2 as another embodiment, it is provided by the invention based on deep learning to monitoring data trend
Forecasting Methodology, including：
201st, distributed post subscribes to message system and real-time streaming data is obtained from monitoring system, and real-time streaming data is held
Distributed file system and real-time on-line system are respectively sent to after longization processing；
202nd, real-time streaming data is sent to off-line training system by distributed file system with preset frequency；
203rd, off-line training system extracts some data from real-time streaming data and forms data set, and data set is counted
Calculation obtains forecast model；
204th, distributed file system obtains forecast model and stored from off-line training system；
205th, real-time on-line system extracts some data from real-time streaming data and forms forecast set, and uses forecast model pair
Forecast set is calculated, and obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is
It is empty；
206th, database connection pool obtains the prediction result of the following flow data of monitoring system from real-time on-line system, and
It is sent to business intelligence instrument board dashboard；
207th, the dashboard is opened up the prediction result of the following flow data of monitoring system with the form of curve
Show.
As shown in Figure 7, it is preferable that the off-line training system extracts some data from real-time streaming data and forms data
Collection, and data set is carried out forecast model is calculated, specifically include：
Some data are extracted from real-time streaming data with preset frequency and are formed data set；
Data set is divided into training set and test set；
Training set input computation model is trained, the computation model is long memory network LSTM in short-term；
Computation model after test set input training is tested, is prediction by the computation model after the training of test
Model.
The real-time on-line system includes but is not limited to：Using data warehouse technology ETL computing engines Spark；
The distributed file system includes but is not limited to：Big data distributed file system HDFS；
The distributed post is subscribed to message system and included but is not limited to：The distributed post of high-throughput subscribes to message system
Unite kafka；
The database connection pool includes but is not limited to：Druid Druid.
As shown in figure 3, as one embodiment, it is provided by the invention based on deep learning to the pre- of monitoring data trend
Examining system, including：
Off-line training system 11, data set is formed for extracting some data from real-time streaming data, and data set is entered
Forecast model is calculated in row；
Real-time on-line system 12, forecast set is formed for extracting some data from real-time streaming data, and use prediction mould
Type is calculated forecast set, obtains the prediction result of the following flow data of monitoring system；The friendship of the data set and forecast set
Integrate as sky.
As shown in figure 4, as another embodiment, it is provided by the invention based on deep learning to monitoring data trend
Forecasting system, including：
Distributed post subscribes to message system 21, for obtaining real-time streaming data from monitoring system, and by real-time fluxion
Distributed file system 22 and real-time on-line system 12 are respectively sent to after being handled according to persistence；
Distributed file system 22, for real-time streaming data to be sent into off-line training system with preset frequency；And from
Forecast model is obtained in off-line training system and is stored；
Off-line training system 11, data set is formed for extracting some data from real-time streaming data, and data set is entered
Forecast model is calculated in row；
Real-time on-line system 12, forecast set is formed for extracting some data from real-time streaming data, and use prediction mould
Type is calculated forecast set, obtains the prediction result of the following flow data of monitoring system；The friendship of the data set and forecast set
Integrate as sky；
Database connection pool 23, the prediction knot of the following flow data for obtaining monitoring system from real-time on-line system 12
Fruit, and it is sent to business intelligence instrument board 24dashboard；
The business intelligence instrument board 24, for by the form of the prediction result curve of the following flow data of monitoring system
It is shown.
As shown in figure 5, a kind of possible structure as off-line training system 11, the off-line training system 11 include：
Acquiring unit 111, data set is formed for extracting some data from real-time streaming data with preset frequency；
Division unit 112, for data set to be divided into training set and test set；
Training unit 113, for training set input computation model to be trained, the computation model is long short-term memory
Network LSTM；
Test cell 114, for the computation model after test set input training to be tested, after the training of test
Computation model be forecast model.
The real-time on-line system 12 includes but is not limited to：Using data warehouse technology ETL computing engines Spark；
The distributed file system 22 includes but is not limited to：Big data distributed file system HDFS；
The distributed post is subscribed to message system 21 and included but is not limited to：The distributed post of high-throughput subscribes to message
System kafka；
The database connection pool includes but is not limited to：Druid Druid.
Technical scheme is described in detail below according to example：
As shown in fig. 6, monitor supervision platform can collect the monitoring data (real-time streaming data) of magnanimity daily：Daily record log, the prison
After controlling collections of the data logging log by tank flume and agent's program agent, into the distributed post of high-throughput
Subscribe to message system kafka.Data by kafka persistences can be stored in big data distributed file system HDFS and using number
According in REPOSITORY TECHNOLOGY ETL computing engines Spark；Off-line training system 11 can periodically obtain data set from HDFS, and by number
Training set and test set are divided into according to collection；Then the real-time streaming data in training set is used to long memory network LSTM models in short-term
(model) it is trained；The LSTM models completed with the real-time streaming data in test set to training are tested, and pass through test
LSTM models can be stored in HDFS.
LSTM models by testing can be obtained from HDFS using data warehouse technology ETL computing engines Spark, so
The real-time streaming data obtained before inputting afterwards from kafka, then will be pre- to be predicted to the following flow data of monitor supervision platform
Result is surveyed to be sent in Druid Druid.
Database connection pool Druid Druid sends prediction result to business intelligence instrument board dashboard；Business intelligence
Prediction result is shown by energy instrument board dashboard with the form of curve.
It is as shown in fig. 7, as follows from the process for getting the long memory network LSTM models in short-term of displaying：
1st, monitoring data obtains and stores (load)；
It will be taken out first in the monitoring data HDFS of monitor supervision platform, line number of going forward side by side value.Specifically, obtained to what HDFS was provided
Take and the get requests with timestamp are sent in data-interface, obtain the monitoring data of continuous 8 days.Monitoring data export is deposited
Enter in csv file, using the csv file as data set.It is to have taken into full account monitoring data itself to choose 8 days this time spans
The periodicity that may contain and forward-backward correlation.
2nd, generation model training dataset；
The window size of training set refers to need the value at several time points to predict the value of future time point.Here we are first
Length of window is used as 3, i.e., carries out model training with t-2, t-1, the time interval of t times, then with t+1 minor ticks to by instructing
Experienced model is tested.Data set format is as follows：X is training set, and Y is test set；Choose in data set preceding 66.7% number
According to as training set, for rear 33.3% data as test set, this is to prevent overfitting.
3rd, LSTM model structure and parameters are set；
An as shown in figure 8, LSTM cellular construction figure.LSTM units are an important components of LSTM models,
For processing time sequence data.xtRepresent input, ht-1Represent the state of last moment, htRepresent the state at current time, W tables
Show weights：
zt=σ (Wz·[ht-1, xt]) (1)
rt=σ (Wr·[ht-1, xt]) (2)
It needs to be determined that the activation primitive (activation function) of LSTM models；In a kind of keras (neutral nets
Framework) in give tacit consent to selection is tanh as activation primitive.Activation primitive between door is typically chosen Sigmoid functions,
Numerical value between Sigmoid functions output 0 to 1, describe that how many amount of each part can pass through.0 represent " mustn't any amount lead to
Cross ", 1 just refers to " allowing any amount to pass through ".It is determined that receive the full Connection Neural Network (fully-connected of LSTM outputs
Neural network) activation primitive be set to linear activation primitives；Determine the rejection rate dropout of each layer network node
Value be set to 0.2；The calculation of error is used as using mean square error (mean squared error)；Calculated using RMSprop
Iteration update scheme of the method as weight parameter, the algorithm have preferably convergence effect generally for RNN networks.
The epoch (total exercise wheel number) of selected model training is by 50 and batch size (training takes sample number every time)
For 1.The dimensionality reduction that one layer of full articulamentum of common neural network is used for output result is added behind the output of LSTM layers.
4th, the training of LSTM models；
LSTM models after training set X input parameters are provided with, obtain the LSTM models that training is completed；Then will survey
Examination collection Y imports the LSTM models of training completion as parameter, can obtain training set Y predicted value Y1.By training set Y prediction
Value Y1, made comparisons with training set Y actual legitimate reading Y2, the quality of the LSTM models of training completion can be obtained.Will be whole
The file that parameter that individual training pattern has been trained is preserved into h5py forms (keras master patterns form) be used for later prediction or
Among calling.
5th, model and result preserve
It is used to offline partial data in the LSTM models deposit HDFS for the h5py form types that the training of standard is completed handle.
6th, result is shown
Real-time on-line system 12 obtains real-time streaming data from kafka, and using training the LSTM models of completion to enter in HDFS
Row calculates, and the data deposit Druid of calculating shows to go out figure.
As shown in Figure 10, preceding 66.7% light-colored part of curve is the fitting obtained after training set is inputted to LSTM models
Effect curve；Rear 33.3% dark parts of curve are to obtain prediction result curve after test set is inputted to LSTM models.Will
Figure 10 and Fig. 9 are contrasted, it is clear that the fitting effect curve of training set is before the result curve with data set 66.7% portion
Divide is that height overlaps.After seeing the result curve of the prediction result curve of test set and data set
33.3% part is also what height overlapped, and corresponding data is concentrated to Wave crest and wave trough and the week of this part of initial data well
Phase property predicts out.
As shown in figure 11, result figure is made to extreme situation；Add maximum of continuous eight day data in synchronization
Value includes in figure with minimum value as extreme case with reference to checking, and by the result curve of this 8 days maximums and minimum value.And
Figure 10 and Figure 11 are compared, it can be seen that the fitting effect curve of training set almost wraps plus the prediction result curve of test set
It is contained in the upper limit (result curve of dark maximum) and lower limit (result curve of light minimum value).This shows, in some poles
In the case of end, the prediction result of trained LSTM models is still stable.Operation maintenance personnel have reason according to predict come song
The time point that line judges the tendency of actual monitored curve and may alarmed.
Thus prove, trained LSTM models can be very good to show the prediction effect of real-time streaming data.Therefore, transport
Dimension personnel during actual prediction, by monitoring data deliver to training after the completion of LSTM models, the monitoring data is made
Prediction, prejudge the overall tendency of monitoring curve according to prediction result curve and recognize the time point for the problem of being likely to occur,
Work is ready before the time point for being likely to occur alarm.
Time series forecasting problem is a kind of relatively complicated prediction modeling problem, and the prediction of regression analysis model
Difference, time series models are to rely on the sequencing of event generation, an equal amount of input model after being worth change order
Caused result is different.Therefore, the present invention is predicted from LSTM models to monitoring data.LSTM models are a kind of
RNN (Recognition with Recurrent Neural Network) variant, it is exactly the valve node that each layer is with the addition of beyond RNN structures the characteristics of LSTM.This
A little valves can open or close, the knot that the memory state (state of network before) for will determine that prototype network exports in this layer
Whether fruit reaches threshold value so as to be added in the calculating of this current layer.Valve node utilizes sigmoid functions by the memory of network
State calculates as input；The valve is exported if output result reaches threshold value and is multiplied with the result of calculation of current layer as under
One layer of input；The output result is forgotten if threshold value is not reaching to.The weight that each layer includes valve node all can
Updated in the training process of model backpropagation each time.The memory function of LSTM models is exactly to be realized by these valve nodes
's.When valve is opened, before the training result of model will be associated with current model and calculate, and closed when valve
When before result of calculation just no longer influence current calculating.Therefore, early stage is realized by the switch can of control valve
The influence of sequence pair final result.There are many bags to directly invoke to build LSTM models, such as Kears in Python,
Tensorflow, Theano etc..The machine learning framework that we are realized from Keras as model definition of the present invention and algorithm,
The backend of selection is TensorFlow.Keras is that one of deep learning framework TensorFlow is refined and encapsulated, can
With help, we quickly build and realized a desired neural network model.
LSTM is higher for the forecasting accuracy of monitoring data trend relative to existing prediction algorithm, existing in engineering
The Forecasting Methodology of utilization such as HolterWinter, ARIMA, the value that 3Sigma methods provide in actual prediction often and are paid no attention to
To think, the relevance between context is not strong, if operation maintenance personnel is directly analyzed in the predicted value provided or early warning,
Effect may be larger with actual deviation.LSTM can catch the contact that may be present of time series context well, and utilize
This contact is made prediction.We want using the model forward-backward correlation and data in itself existing relevance for ours
Monitoring data makes rational prediction.LSTM gives to the tendency of Future Data and intuitively predicted, the result of trend prediction can
To provide good reference for operation maintenance personnel, in the case of alarm is likely to occur, operation maintenance personnel can take pin earlier
Measure to property, it is effective to reduce the number alarmed, alleviate the work load of operation maintenance personnel.Basically reach and do not failed to report and not
The balance of wrong report, while the model trained according to the substantial amounts of monitoring data of history has very strong extensive and predictive ability, does not have to
Go to add data renewal training in real time, it is not required that labor intensive material resources remove long term maintenance.The work in later stage only needs to update
Model be substituted into prediction module.After the human cost of monitoring saves, we can more put into intelligence
In the middle of the research that can be monitored.
It should be understood that the particular order or level of the step of during disclosed are the examples of illustrative methods.Based on setting
Count preference, it should be appreciated that during the step of particular order or level can be in the feelings for the protection domain for not departing from the disclosure
Rearranged under condition.Appended claim to a method gives the key element of various steps with exemplary order, and not
It is to be limited to described particular order or level.
In above-mentioned detailed description, various features combine in single embodiment together, to simplify the disclosure.No
This open method should be construed to reflect such intention, i.e. the embodiment of theme claimed needs to compare
The more features of feature clearly stated in each claim.On the contrary, as appended claims is reflected
Like that, the present invention is in the state fewer than whole features of disclosed single embodiment.Therefore, appended claims
It is hereby expressly incorporated into detailed description, wherein each claim is alone as the single preferred embodiment of the present invention.
To enable any technical staff in the art to realize or using the present invention, disclosed embodiment being entered above
Description is gone.To those skilled in the art；The various modification modes of these embodiments will be apparent from, and this
The General Principle of text definition can also be applied to other embodiments on the basis of the spirit and scope of the disclosure is not departed from.
Therefore, the disclosure is not limited to embodiments set forth herein, but most wide with principle disclosed in the present application and novel features
Scope is consistent.
Described above includes the citing of one or more embodiments.Certainly, in order to above-described embodiment is described and description portion
The all possible combination of part or method is impossible, but it will be appreciated by one of ordinary skill in the art that each implementation
Example can do further combinations and permutations.Therefore, embodiment described herein is intended to fall into appended claims
Protection domain in all such changes, modifications and variations.In addition, with regard to the term used in specification or claims
"comprising", the mode that covers of the word are similar to term " comprising ", just as " including " solved in the claims as link word
As releasing.In addition, the use of any one term "or" in the specification of claims is to represent " non-exclusionism
Or ".
Those skilled in the art will also be appreciated that the various illustrative components, blocks that the embodiment of the present invention is listed
(illustrative logical block), unit, and step can pass through the knot of electronic hardware, computer software, or both
Conjunction is realized.To clearly show that the replaceability of hardware and software (interchangeability), above-mentioned various explanations
Property part (illustrative components), unit and step universally describe their function.Such work(
Can be that specific application and the design requirement of whole system are depended on to realize by hardware or software.Those skilled in the art
Various methods can be used to realize described function, but this realization is understood not to for every kind of specific application
Beyond the scope of protection of the embodiment of the present invention.
Various illustrative logical blocks described in the embodiment of the present invention, or unit can by general processor,
Digital signal processor, application specific integrated circuit (ASIC), field programmable gate array or other programmable logic devices, discrete gate
Or the design of transistor logic, discrete hardware components, or any of the above described combination is come the function described by realizing or operate.General place
It can be microprocessor to manage device, and alternatively, the general processor can also be any traditional processor, controller, microcontroller
Device or state machine.Processor can also be realized by the combination of computing device, such as digital signal processor and microprocessor,
Multi-microprocessor, one or more microprocessors combine a Digital Signal Processor Core, or any other like configuration
To realize.
The step of method or algorithm described in the embodiment of the present invention can be directly embedded into hardware, computing device it is soft
Part module or the combination of both.Software module can be stored in RAM memory, flash memory, ROM memory, EPROM storages
Other any form of storaging mediums in device, eeprom memory, register, hard disk, moveable magnetic disc, CD-ROM or this area
In.Exemplarily, storaging medium can be connected with processor, to allow processor to read information from storaging medium, and
Write information can be deposited to storaging medium.Alternatively, storaging medium can also be integrated into processor.Processor and storaging medium can
To be arranged in ASIC, ASIC can be arranged in user terminal.Alternatively, processor and storaging medium can also be arranged at use
In different parts in the terminal of family.
In one or more exemplary designs, above-mentioned function described by the embodiment of the present invention can be in hardware, soft
Part, firmware or any combination of this three are realized.If realized in software, these functions can store and computer-readable
On medium, or with one or more instruction or code form be transmitted on the medium of computer-readable.Computer readable medium includes electricity
Brain storaging medium and it is easy to so that allowing computer program to be transferred to other local telecommunication medias from a place.Storaging medium can be with
It is that any general or special computer can be with the useable medium of access.For example, such computer readable media can include but
It is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or other
What can be used for carrying or store with instruct or data structure and it is other can be by general or special computer or general or specially treated
The medium of the program code of device reading form.In addition, any connection can be properly termed computer readable medium, example
Such as, if software is to pass through a coaxial cable, fiber optic cables, double from a web-site, server or other remote resources
Twisted wire, Digital Subscriber Line (DSL) or with defined in being also contained in of the wireless way for transmitting such as infrared, wireless and microwave
In computer readable medium.Described disk (disk) and disk (disc) include Zip disk, radium-shine disk, CD, DVD, floppy disk
And Blu-ray Disc, disk is generally with magnetic duplication data, and disk generally carries out optical reproduction data with laser.Combinations of the above
It can also be included in computer readable medium.
Above-described embodiment, the purpose of the present invention, technical scheme and beneficial effect are carried out further
Describe in detail, should be understood that the embodiment that the foregoing is only the present invention, be not intended to limit the present invention
Protection domain, within the spirit and principles of the invention, any modification, equivalent substitution and improvements done etc., all should include
Within protection scope of the present invention.
Claims (10)
1. a kind of Forecasting Methodology to monitoring data trend based on deep learning, it is characterised in that methods described includes：
Off-line training system extracts some data from real-time streaming data and forms data sets, and data set be calculated pre-
Survey model；
Real-time on-line system extracts some data from real-time streaming data and forms forecast set, and forecast set is entered using forecast model
Row calculates, and obtains the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is sky.
2. the Forecasting Methodology to monitoring data trend according to claim 1 based on deep learning, it is characterised in that institute
State off-line training system and some data composition data sets are extracted from real-time streaming data, and data set is carried out prediction is calculated
Model, specifically include：
Some data are extracted from real-time streaming data with preset frequency and are formed data set；
Data set is divided into training set and test set；
Training set input computation model is trained, the computation model includes long memory network LSTM models in short-term；
Computation model after test set input training is tested, is prediction mould by the computation model after the training of test
Type.
3. the Forecasting Methodology to monitoring data trend according to claim 1 or 2 based on deep learning, its feature exist
In, before the off-line training system extracts some data composition data sets from real-time streaming data, in addition to：
Real-time streaming data is sent to off-line training system by distributed file system with preset frequency；
It is described that data set is carried out after forecast model is calculated, in addition to：
Distributed file system obtains forecast model and stored from off-line training system.
4. the Forecasting Methodology to monitoring data trend according to claim 3 based on deep learning, it is characterised in that institute
State before real-time streaming data is sent to off-line training system by distributed file system with preset frequency, in addition to：
Distributed post subscribes to message system and real-time streaming data is obtained from monitoring system, and real-time streaming data persistence is handled
After be respectively sent to distributed file system and real-time on-line system；The real-time streaming data for being sent to distributed file system is formed
Set be sent to real-time on-line system real-time streaming data form set occur simultaneously for sky；
It is described that forecast set is calculated using forecast model, the prediction result of the following flow data of monitoring system is obtained, afterwards
Also include：
Database connection pool obtains the prediction result of the following flow data of monitoring system from real-time on-line system, and is sent to business
Industry Intelligent instrument panel dashboard；
The prediction result of the following flow data of monitoring system is shown by the dashboard with the form of curve.
5. the Forecasting Methodology to monitoring data trend according to claim 4 based on deep learning, it is characterised in that
The real-time on-line system includes but is not limited to：Using data warehouse technology ETL computing engines Spark；
The distributed file system includes but is not limited to：Big data distributed file system HDFS；
The distributed post is subscribed to message system and included but is not limited to：The distributed post of high-throughput subscribes to message system
kafka；
The database connection pool includes but is not limited to：Druid Druid.
6. a kind of forecasting system to monitoring data trend based on deep learning, it is characterised in that the system includes：
Off-line training system, data set is formed for extracting some data from real-time streaming data, and data set is calculated
Obtain forecast model；
Real-time on-line system, forecast set is formed for extracting some data from real-time streaming data, and using forecast model to pre-
Survey collection to be calculated, obtain the prediction result of the following flow data of monitoring system；The common factor of the data set and forecast set is sky.
7. the forecasting system to monitoring data trend according to claim 6 based on deep learning, it is characterised in that institute
Stating off-line training system includes：
Acquiring unit, data set is formed for extracting some data from real-time streaming data with preset frequency；
Division unit, for data set to be divided into training set and test set；
Training unit, for training set input computation model to be trained, the computation model includes long memory network in short-term
LSTM models；
Test cell, for the computation model after test set input training to be tested, pass through the calculating after the training of test
Model is forecast model.
8. the forecasting system to monitoring data trend based on deep learning according to claim 6 or 7, its feature exist
In the system also includes：
Distributed file system, for real-time streaming data to be sent into off-line training system with preset frequency；And instructed from offline
Practice and forecast model is obtained in system and is stored.
9. the forecasting system to monitoring data trend according to claim 8 based on deep learning, it is characterised in that institute
Stating system also includes：
Distributed post subscribes to message system, for obtaining real-time streaming data from monitoring system, and real-time streaming data is lasting
Distributed file system and real-time on-line system are respectively sent to after change processing；It is sent to the real-time fluxion of distributed file system
Occur simultaneously according to the set of composition with being sent to the set that the real-time streaming data of real-time on-line system is formed for sky；
Database connection pool, the prediction result of the following flow data for obtaining monitoring system from real-time on-line system, and pass
Deliver to business intelligence instrument board dashboard；
The dashboard, for the prediction result of the following flow data of monitoring system to be shown with the form of curve.
10. the forecasting system to monitoring data trend according to claim 9 based on deep learning, it is characterised in that
The real-time on-line system includes but is not limited to：Using data warehouse technology ETL computing engines Spark；
The distributed file system includes but is not limited to：Big data distributed file system HDFS；
The distributed post is subscribed to message system and included but is not limited to：The distributed post of high-throughput subscribes to message system
kafka；
The database connection pool includes but is not limited to：Druid Druid.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710941098.4A CN107832913A (en) | 2017-10-11 | 2017-10-11 | The Forecasting Methodology and system to monitoring data trend based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710941098.4A CN107832913A (en) | 2017-10-11 | 2017-10-11 | The Forecasting Methodology and system to monitoring data trend based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107832913A true CN107832913A (en) | 2018-03-23 |
Family
ID=61647788
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710941098.4A Pending CN107832913A (en) | 2017-10-11 | 2017-10-11 | The Forecasting Methodology and system to monitoring data trend based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107832913A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109555977A (en) * | 2018-11-23 | 2019-04-02 | 水联网技术服务中心（北京）有限公司 | The equipment and recognition methods of leak noise measuring |
| CN109783512A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110460454A (en) * | 2018-05-04 | 2019-11-15 | 上海伽易信息技术有限公司 | Network equipment port intelligent fault prediction technique and principle based on deep learning |
| CN110618911A (en) * | 2019-08-15 | 2019-12-27 | 中国平安财产保险股份有限公司 | Data monitoring method and device, storage medium and server |
| CN110851488A (en) * | 2019-09-26 | 2020-02-28 | 贵阳信息技术研究院（中科院软件所贵阳分部） | Multi-source-based multi-modal data fusion analysis processing method and platform |
| CN111159135A (en) * | 2019-12-23 | 2020-05-15 | 五八有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111199257A (en) * | 2020-01-10 | 2020-05-26 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Fault diagnosis method and device for high-speed rail driving equipment |
| CN112507003A (en) * | 2021-02-03 | 2021-03-16 | 江苏海平面数据科技有限公司 | Internet of vehicles data analysis platform based on big data architecture |
| CN112651785A (en) * | 2020-12-31 | 2021-04-13 | 中国农业银行股份有限公司 | Real-time monitoring method and system for transaction amount |
| CN113128741A (en) * | 2020-01-10 | 2021-07-16 | 阿里巴巴集团控股有限公司 | Data processing method, device, system, equipment and readable storage medium |
| CN113254344A (en) * | 2021-06-07 | 2021-08-13 | 吉林大学 | Novel computing engine test platform, device and system |
| CN114626627A (en) * | 2022-03-28 | 2022-06-14 | 王大成 | Monitoring and early warning system for carbon emission in area |
| CN115953738A (en) * | 2023-03-02 | 2023-04-11 | 上海燧原科技有限公司 | Monitoring method, device, equipment and medium for image recognition distributed training |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260794A (en) * | 2015-10-12 | 2016-01-20 | 上海交通大学 | Load predicting method of cloud data center |
| CN106934497A (en) * | 2017-03-08 | 2017-07-07 | 青岛卓迅电子科技有限公司 | Wisdom cell power consumption real-time predicting method and device based on deep learning |
-
2017
- 2017-10-11 CN CN201710941098.4A patent/CN107832913A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260794A (en) * | 2015-10-12 | 2016-01-20 | 上海交通大学 | Load predicting method of cloud data center |
| CN106934497A (en) * | 2017-03-08 | 2017-07-07 | 青岛卓迅电子科技有限公司 | Wisdom cell power consumption real-time predicting method and device based on deep learning |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110460454B (en) * | 2018-05-04 | 2022-02-08 | 上海伽易信息技术有限公司 | Intelligent network equipment port fault prediction method based on deep learning |
| CN110460454A (en) * | 2018-05-04 | 2019-11-15 | 上海伽易信息技术有限公司 | Network equipment port intelligent fault prediction technique and principle based on deep learning |
| CN109555977A (en) * | 2018-11-23 | 2019-04-02 | 水联网技术服务中心（北京）有限公司 | The equipment and recognition methods of leak noise measuring |
| CN109783512A (en) * | 2018-12-13 | 2019-05-21 | 平安科技（深圳）有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110618911A (en) * | 2019-08-15 | 2019-12-27 | 中国平安财产保险股份有限公司 | Data monitoring method and device, storage medium and server |
| CN110851488A (en) * | 2019-09-26 | 2020-02-28 | 贵阳信息技术研究院（中科院软件所贵阳分部） | Multi-source-based multi-modal data fusion analysis processing method and platform |
| CN111159135A (en) * | 2019-12-23 | 2020-05-15 | 五八有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111199257A (en) * | 2020-01-10 | 2020-05-26 | 中国铁道科学研究院集团有限公司电子计算技术研究所 | Fault diagnosis method and device for high-speed rail driving equipment |
| CN113128741A (en) * | 2020-01-10 | 2021-07-16 | 阿里巴巴集团控股有限公司 | Data processing method, device, system, equipment and readable storage medium |
| CN112651785A (en) * | 2020-12-31 | 2021-04-13 | 中国农业银行股份有限公司 | Real-time monitoring method and system for transaction amount |
| CN112651785B (en) * | 2020-12-31 | 2023-12-08 | 中国农业银行股份有限公司 | Transaction amount real-time monitoring method and system |
| CN112507003A (en) * | 2021-02-03 | 2021-03-16 | 江苏海平面数据科技有限公司 | Internet of vehicles data analysis platform based on big data architecture |
| CN113254344A (en) * | 2021-06-07 | 2021-08-13 | 吉林大学 | Novel computing engine test platform, device and system |
| CN114626627A (en) * | 2022-03-28 | 2022-06-14 | 王大成 | Monitoring and early warning system for carbon emission in area |
| CN115953738A (en) * | 2023-03-02 | 2023-04-11 | 上海燧原科技有限公司 | Monitoring method, device, equipment and medium for image recognition distributed training |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832913A (en) | The Forecasting Methodology and system to monitoring data trend based on deep learning | |
| CN109754605B (en) | Traffic prediction method based on attention temporal graph convolution network | |
| CN113887616B (en) | Real-time abnormality detection method for EPG connection number | |
| Thai-Nghe et al. | Deep learning approach for forecasting water quality in IoT systems | |
| CN109407654B (en) | Industrial data nonlinear causal analysis method based on sparse deep neural network | |
| CN108197739A (en) | A kind of urban track traffic ridership Forecasting Methodology | |
| CN114422381A (en) | Communication network flow prediction method, system, storage medium and computer equipment | |
| CN111242344A (en) | Intelligent water level prediction method based on cyclic neural network and convolutional neural network | |
| CN102130783A (en) | Intelligent alarm monitoring method of neural network | |
| CN109471698B (en) | System and method for detecting abnormal behavior of virtual machine in cloud environment | |
| CN107622308A (en) | A kind of generating equipment parameter method for early warning based on DBN networks | |
| CN113126676B (en) | Livestock and poultry house breeding environment parameter intelligent control system | |
| CN108009632A (en) | Confrontation type space-time big data Forecasting Methodology | |
| CN114169638A (en) | Water quality prediction method and device | |
| CN114118507A (en) | Risk assessment early warning method and device based on multi-dimensional information fusion | |
| CN112733440A (en) | Intelligent fault diagnosis method, system, storage medium and equipment for offshore oil-gas-water well | |
| Cheng et al. | Analysis and forecasting of the day-to-day travel demand variations for large-scale transportation networks: a deep learning approach | |
| CN107944550A (en) | A kind of space-time big data Forecasting Methodology based on detection type depth network | |
| CN114896882A (en) | Data-driven green digital twinning construction system and method | |
| CN108768750A (en) | Communication network failure localization method and device | |
| CN115271186A (en) | Reservoir water level prediction early warning method based on delay factor and PSO RNN Attention model | |
| CN114584406A (en) | Industrial big data privacy protection system and method for federated learning | |
| CN113807951A (en) | Transaction data trend prediction method and system based on deep learning | |
| CN105976125A (en) | Dynamic layered early warning modeling method for food safety risk | |
| CN116822722A (en) | Water level prediction method, system, device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180323 |