CN105427193A - Device and method for big data analysis based on distributed time sequence data service - Google Patents
Device and method for big data analysis based on distributed time sequence data service Download PDFInfo
- Publication number
- CN105427193A CN105427193A CN201510962481.9A CN201510962481A CN105427193A CN 105427193 A CN105427193 A CN 105427193A CN 201510962481 A CN201510962481 A CN 201510962481A CN 105427193 A CN105427193 A CN 105427193A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- storer
- time series
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000007405 data analysis Methods 0.000 title claims abstract description 17
- 238000012544 monitoring process Methods 0.000 claims abstract description 113
- 230000008569 process Effects 0.000 claims abstract description 43
- 230000010354 integration Effects 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims description 19
- 238000007418 data mining Methods 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 11
- 238000012549 training Methods 0.000 claims description 10
- 238000004148 unit process Methods 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 238000012423 maintenance Methods 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000009434 installation Methods 0.000 claims description 3
- 230000008520 organization Effects 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 abstract description 13
- 238000013500 data storage Methods 0.000 abstract description 7
- 238000004458 analytical method Methods 0.000 description 12
- 238000005065 mining Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000010223 real-time analysis Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 230000032683 aging Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- CVOFKRWYWCSDMA-UHFFFAOYSA-N 2-chloro-n-(2,6-diethylphenyl)-n-(methoxymethyl)acetamide;2,6-dinitro-n,n-dipropyl-4-(trifluoromethyl)aniline Chemical compound CCC1=CC=CC(CC)=C1N(COC)C(=O)CCl.CCCN(CCC)C1=C([N+]([O-])=O)C=C(C(F)(F)F)C=C1[N+]([O-])=O CVOFKRWYWCSDMA-UHFFFAOYSA-N 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000003921 oil Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Water Supply & Treatment (AREA)
- Tourism & Hospitality (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Primary Health Care (AREA)
- Marketing (AREA)
- Human Resources & Organizations (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Software Systems (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
A device for big data analysis based on a distributed time sequence data service comprises a monitoring sensor, a monitoring data storage, an equipment data storage, a data integration unit, a time sequence storage, a calculation unit, a wireless terminal and a data service terminal, can process numerous real-time data rapidly, efficiently and timely, and can ensure safe, stable and efficiency running of equipment.
Description
Technical field
The present invention relates to monitoring of equipment analytical applications field, be specifically related to a kind of large data analysis set-up of serving based on distributed time series data and method.
Background technology
Along with the develop rapidly of computer technology, the data of every profession and trade increase rapidly, data quantitative change increasing, type also gets more and more, data structure be also tending towards complicated, traditional database not only each equipment independently place, and need larger space for its deployment, there is shortcomings such as not easily disposing, cost is higher, the General Requirements of user can not be met.
Time series data is the time series data of band time tag, its typical feature be produce frequency fast, depend critically upon acquisition time, measuring point multiple data quantity is large.In power industry; in order to ensure device security, stablizing, run efficiently; usually Real-Time Monitoring can be carried out to the running status of the various kinds of equipment such as generating, power transformation; gather and obtain the basis that a large amount of time series datas can be used as the senior application such as equipment running status assessment, equipment operation failure early warning, equipment dependability analysis; thus; how fast, efficiently, in time to process magnanimity real time data, be the key subjects that the heavy assets industries such as electric power, chemical industry, oil, iron and steel face always.
In power industry, history service data collection and analysis, instant analysis that is real-time or near-realtime data are contents important in informatization process in power industry, it needs complete set, stablizes, agrees with the solution of the large data analysis set-up of practical business scene, provides reliable and stable bottom data to support to real-time analysis class business scenarios such as equipment fault early-warnings.
In recent years, along with IT technology fast developments such as cloud computing, large data, machine learning, data minings, distributed storage, high-performance calculation all obtain key breakthrough in theoretical research and engineering practice aspect, and it take Hadoop as large data processing and the application solution of representative that industry emerges a collection of.
Hadoop is a distributed system architecture, comprise distributed file system HDFS (HadoopDistributedFileSystem), distributed memory system HBase, several cores such as parallel computation programming model MapReduce, it greatly can simplify the processing procedure of large-scale data, but it is at functional completeness, operation stability aspect has some limitations, and follow the actual demand of power business scene to there is deviation based on some commercial large data platforms that Hadoop derives, thus, the business demand of depth analysis research power industry, build a kind of large data analysis set-up of serving based on distributed time series data, there is profound significance and stronger value.
Summary of the invention
The object of the invention is to overcome the deficiencies in the prior art, a kind of large data analysis set-up of serving based on distributed time series data and method are provided, magnanimity real time data can be processed fast, efficiently, in time, ensure device security simultaneously, stablize, run efficiently.
The invention provides a kind of large data analysis set-up of serving based on distributed time series data, comprise monitoring sensor, Monitoring Data storer, device data storer, Data Integration unit, time series data storer, computing unit, wireless terminal and data, services terminal, wherein Monitoring Data storer respectively with monitoring sensor and Data Integration unit, Data Integration unit also distinguishes connection device data-carrier store and time series data storer, Data Integration unit, time series data storer, computing unit is connected successively with data, services terminal, data, services terminal also respectively with time series data storer, monitoring sensor is connected with wireless terminal,
The monitoring of equipment data collected for obtaining monitoring of equipment data in real time or quasi real time, and are transferred to Monitoring Data storer by monitoring sensor;
Monitoring Data storer, for monitoring of equipment data being stored, and in the mode that streaming exports, exports monitoring of equipment data to Data Integration unit;
Device data storer, for storing conventional device data, and in the mode of batch signatures, exports the device data of routine to Data Integration unit;
Data Integration unit, mode for accessing with streaming receives the monitoring of equipment data sent from Monitoring Data storer and the mode accessed with batch, by the device data of the routine in predefined operation plan automatic acquisition device data storer, wherein Data Integration unit also comprises pretreatment unit, carry out the pre-service of the cleaning of data, filtration, conversion for the monitoring of equipment data that will receive and conventional device data with preprocessing rule, and export pretreated data to time series data storer;
Time series data storer, for storing pretreated data and configuration data, and visiting frequency is high, that performance requirement is high data centralization is cached to the internal memory in time series data storer;
Computing unit, for the data driving scheduling engine to call and receive time series data storer to store, and foundation in advance programmed processing logic processes the data called and receive, training formed data mining model, also for by the data back after computing unit process to time series data storer and/or data, services end;
Data, services end, comprise data, services end processor, interface unit and display device, wherein data, services end processor is used for directly reading data from time series data storer and/or receiving from the data after computing unit process, and carry out analyzing and processing, result after process is shown by display device, by interface unit, the result after process is sent to wireless terminal simultaneously;
Wireless terminal, for receive from data, services end send process after result, and can wireless transmission control command to data, services end, data, services termination controls monitoring sensor after receiving control command, adjustment monitoring sensor data acquiring frequency.
Further, described monitoring sensor is the information acquisition sensor be installed on monitoring equipment.
Further, described monitoring sensor is camera and/or the temperature detector of monitoring equipment installation region.
Further, also comprise the manual input device be connected with Data Integration unit, for the Input Monitor Connector device data when implementing quarantine measures because of safety requirements or do not support data access.
Further, configuration data is the business implication data of the device data describing monitoring of equipment data and/or routine, storage organization data and/or processing logic data.
Further, the data that described visiting frequency is high, performance requirement is high refer to recent Monitoring Data, conventional device data, and history achievement data, model metadata and preprocessing rule data that concern rate is higher.
Further, described computing unit also runs for calling and receive equipment in time series data storer the new time series data produced, and repeats training process to new time series data, upgrades data mining model.
Further, the result after the process of data, services end is fault pre-alarming result and/or load prediction results.
Further, described wireless terminal is notebook computer, panel computer and/or mobile phone.
The present invention also provides a kind of large data analysing method of large data analysis set-up of serving based on distributed time series data, in turn includes the following steps:
(1) initialization, the initial parameter of setting data service end, the sampling period controlling monitoring sensor according to the initial parameter that sets is 6 times per second, and the sampling time is 1 minute, and average the data of sampling in 1 minute A;
(2) under same initial parameter condition, repeat step (1) 3 time, try to achieve mean value B, C, D of 3 times respectively;
(3) average after mean value A, B, C, D being sued for peace P again:
If A.
then monitoring sensor stable performance, enters step (4);
If B.
then monitoring sensor unstable properties, then enter step (1);
(4) obtain monitoring of equipment data in real time or quasi real time, and store after the monitoring of equipment data collected are transferred to Monitoring Data storer, the mode exported with streaming, exports monitoring of equipment data to Data Integration unit;
(5) in the mode of batch access, by the device data of the routine in predefined operation plan automatic acquisition device data storer, monitoring of equipment data and conventional device data are carried out the pre-service of the cleaning of data, filtration, conversion with preprocessing rule, and export pretreated data to time series data storer and store;
(6) by recent Monitoring Data, conventional device data, and higher history achievement data, model metadata and the preprocessing rule data centralization of concern rate is cached to the internal memory in time series data storer;
(7) scheduling engine is driven to call and receive the data of time series data storer storage by computing unit, and foundation in advance programmed processing logic processes the data called and receive, training forms data mining model, by the data back after computing unit process to time series data storer and/or data, services end;
(8) directly read data from time series data storer and/or receive from the data after computing unit process, and carry out analyzing and processing, result after process is shown by display device, by interface unit, the result after process is sent to wireless terminal simultaneously;
(9) result after the process sent from data, services end by wireless terminal reception, determine whether send control command to data, services end according to the result after process, data, services termination controls monitoring sensor after receiving control command, the data acquiring frequency of adjustment monitoring sensor, result wherein after process is fault pre-alarming result and/or load prediction results, determines that whether sending control command meets to data, services end according to the result after process:
A., when fault pre-alarming result and/or load prediction results are normal, the data acquiring frequency of monitoring sensor is reduced;
B. when fault pre-alarming result and/or load prediction results are abnormal, improve the data acquiring frequency of monitoring sensor, and repeat step (1)-(9), give the alarm at data, services end simultaneously, the display device of data, services end shows fault pre-alarming result and/or load prediction results in real time, and notifies maintenance personal.
Large data analysis set-up of the present invention and method, can realize:
1) with stable, increase income distributed memory system and parallel computation service for core reliably, efficiently, store and the encapsulation of requirements for access orientation for heavy assets industry time series data, support for the real-time analysis class business scenarios such as equipment fault early-warning provide reliable and stable bottom data;
2) real-time and punctual image data, ageing height, and optimal design data acquiring frequency, collecting efficiency is high, but the low usefulness of efficiency is high, and apparatus function is powerful, monitoring and maintenance personal can be made in Long-distance Control and watch-dog state, immediately process, process ageing higher, and make to decrease equipment loss because shorten the processing time, saved cost;
3) for the reliability of system data, devise average data and confirm scheme, make Monitoring Data more reliable and more stable, and by the real-time status adjustment monitoring frequency according to equipment, alleviate the working load of device, longer service life, performance is more stable.
Accompanying drawing explanation
The large data analysis set-up structural representation of Fig. 1
Embodiment
The following detailed description of specific embodiment of the invention; what be necessary to herein means out is; below implement just to further illustrate for of the present invention; limiting the scope of the invention can not be interpreted as; some nonessential improvement and adjustment that this art skilled person makes the present invention according to the invention described above content, still belong to protection scope of the present invention.
The invention provides a kind of large data analysis set-up of serving based on distributed time series data, as shown in Figure 1, comprise monitoring sensor 1, Monitoring Data storer 2, device data storer 3, Data Integration unit 4, time series data storer 5, computing unit 6, wireless terminal 8 and data, services terminal 7, wherein Monitoring Data storer 2 respectively with monitoring sensor 1 and Data Integration unit 4, Data Integration unit 4 also distinguishes connection device data-carrier store 3 and time series data storer 5, Data Integration unit 4, time series data storer 5, computing unit 6 is connected successively with data, services terminal 7, data, services terminal 7 also respectively with time series data storer 5, monitoring sensor 1 is connected with wireless terminal 8,
Monitoring sensor, for obtaining monitoring of equipment data in real time or quasi real time, and the monitoring of equipment data collected are transferred to Monitoring Data storer, monitoring sensor is the information acquisition sensor be installed on monitoring equipment, it can also be the sensor such as camera, temperature detector of monitoring equipment installation region, Monitoring Data storer can be real-time by monitoring of equipment data store, and with streaming export mode, export monitoring of equipment data to Data Integration unit.
Device data storer is for storing conventional device data, and in the mode of batch signatures, export the device data of routine to Data Integration list, wherein conventional device data comes from system configuration management, be mainly used in the business implication of description business datum, storage organization and processing logic, generally produce in the system configuration stage.
Data Integration unit, mode for accessing with streaming receives the monitoring of equipment data sent from Monitoring Data storer and the mode accessed with batch, by the device data of the routine in predefined operation plan automatic acquisition device data storer, the data collected are obtained by various ways such as batch access, streaming access, artificial importings by Data Integration unit, also directly can connect collection point and obtain Monitoring Data.The data of access before storing, necessary pre-service can be carried out, utilize pre-configured preprocessing rule to carry out cleaning, filter, the operation such as conversion, data are through Data Integration or be directly stored in time series data storer, wait the high data of some access frequencys for some rules, be generally stored in data cached in, for some history service data, the data that access frequency is little, after Data Integration, are generally stored in business datum; For some data prediction rules of system definition, computation rule, the data such as model data, are generally stored in configuration data; Data access service directly reads data by data access interface.No matter be business datum or configuration data, larger difference is there is in its visiting frequency, performance requirement in concrete business scenario, the data that, performance requirement high for visiting frequency is high, system is concentrated to be cached in Installed System Memory, and these business datums be buffered and configuration data are referred to as data cached.Generally speaking, the data access frequency such as the history index that recent business datum, concern rate are higher, model metadata, data prediction rule are higher, can regard as data cached.Data storage can provide basic guarantee for data query service, can provide input for online calculation services and off-line analysis service, also supports the write-back of corresponding result of calculation simultaneously.The database related in time series data storer 5 mainly contains distributed file system HDFS (HadoopDistributedFileSystem), columnar database HBase (HadoopDatabase), memory database Redis, relational database Oracle etc.Oracle database is mainly used in store configuration data and partial service data, HDFS is as the distributed file system unit of large data platform bottom, for the HBASE on upper strata provides support, also can non-sequential part directly in storage service data, HBASE be a high reliability, high-performance, towards row, telescopic distributed memory system, be mainly used in the time preamble section in storage service data, Redis is a key-value storage system based on internal memory, is mainly used in depositing data cached here.
Computing unit can utilize the managerial experience of industry specialists to research and analyse mining algorithm in conjunction with Principle of Statistics, relevant historical data is run for input with power equipment, training forms data mining model, different sample datas can form different data mining models (example: distinguish by season), run for equipment the new time series data produced and can repeat training process, carry out the Continual Improvement of data mining model; The data mining model created can participate in line computation, adopts mode in real time or quasi real time to analyze every evaluation index of power equipment.The training process of mining model has related to batch and has calculated, and is realized by batch computational tasks; The application process of mining model has related to streaming calculating, is realized by streaming computational tasks; In addition, two kinds of computation schemas can also be used for realizing appraisal of equipment index, voice semantics recognition, text semantic analysis etc. the calculation task irrelevant with mining model.
Batch computational tasks is driven by scheduling engine, reads in business historical data from data storage areas, and foundation in advance programmed processing logic calculates, and result of calculation can be written back to data storage area, also directly externally can be provided by off-line analysis service; Streaming computational tasks is also driven by scheduling engine, data access in a streaming manner from data store, foundation in advance programmed processing logic calculates, and result of calculation can be written back to data storage area, also directly externally can be provided by online calculation services.
Computational tasks is for defining (also referred to as industry node) topological structure and the actuating logic of calculation task, be similar to workflow (Workflow), complete in the job design device that its definition procedure can provide in system, from the visual angle of computing engines, each jobs node corresponds to a computing unit (ComputeUnit), and the programmed logic that computing unit is corresponding is referred to as operator (Transformation).System provides visual modeling tool, preset abundant data processing and data display operator, and open operator development specifications, supports the secondary development of practical business scene simultaneously.
Data, services end can realize the encapsulation to the large data analysis set-up types of functionality of serving based on distributed time series data, can realize data access service, online calculation services and off-line analysis service.Data access service directly reads data from data storage area, and its reciprocal process does not relate to data and calculates, and can be further subdivided into configuration information access services, interactive inquiry service, typical apply scene mainly comprehensive inquiry, visual presentation etc.The common time span of online calculation services is between hundreds of millisecond to several seconds, and high concurrent and need quick response analysis result, typical apply scene comprises fault pre-alarming, load prediction etc.The time span of off-line analysis service is between several tens minutes to a few hours, and be mainly used in multidimensional statistics prediction, quasi real time analysis and the data mining such as cluster, classification application, typical apply scene comprises Fault Pattern Recognition, steady working condition analysis etc.The interactive mode of data, services comprises synchronous, asynchronous two kinds, and online calculation services adopts synchronous mode usually, and off-line analysis service adopts asynchronous mode usually, and asynchronous mode can introduce the transmission of messenger service middleware adapter computing mode and result of calculation information.
Wireless terminal can be notebook computer, panel computer and/or mobile phone, monitoring personnel or maintenance job personnel can by wireless terminal long-range realize mutual with data, services end, can the monitoring result that pushes of real-time query initiatively or passive reception data, services end, and also can pass through other part of the long-range manipulation data, services end of wireless terminal and device, realize long-range real-time manipulation, monitoring, in addition for there is abnormal situation, also can process timely.
The present invention also provides a kind of large data analysing method of large data analysis set-up of serving based on distributed time series data, in turn includes the following steps:
(1) initialization, the initial parameter of setting data service end, the sampling period controlling monitoring sensor according to the initial parameter that sets is 6 times per second, and the sampling time is 1 minute, and average the data of sampling in 1 minute A;
(2) under same initial parameter condition, repeat step (1) 3 time, try to achieve mean value B, C, D of 3 times respectively;
(3) average after mean value A, B, C, D being sued for peace P again:
If A.
then monitoring sensor stable performance, enters step (4);
If B.
then monitoring sensor unstable properties, then enter step (1);
(4) obtain monitoring of equipment data in real time or quasi real time, and store after the monitoring of equipment data collected are transferred to Monitoring Data storer, the mode exported with streaming, exports monitoring of equipment data to Data Integration unit;
(5) in the mode of batch access, by the device data of the routine in predefined operation plan automatic acquisition device data storer, monitoring of equipment data and conventional device data are carried out the pre-service of the cleaning of data, filtration, conversion with preprocessing rule, and export pretreated data to time series data storer and store;
(6) by recent Monitoring Data, conventional device data, and higher history achievement data, model metadata and the preprocessing rule data centralization of concern rate is cached to the internal memory in time series data storer;
(7) scheduling engine is driven to call and receive the data of time series data storer storage by computing unit, and foundation in advance programmed processing logic processes the data called and receive, training forms data mining model, by the data back after computing unit process to time series data storer and/or data, services end;
(8) directly read data from time series data storer and/or receive from the data after computing unit process, and carry out analyzing and processing, result after process is shown by display device, by interface unit, the result after process is sent to wireless terminal simultaneously;
(9) result after the process sent from data, services end by wireless terminal reception, determine whether send control command to data, services end according to the result after process, data, services termination controls monitoring sensor after receiving control command, the data acquiring frequency of adjustment monitoring sensor, result wherein after process is fault pre-alarming result and/or load prediction results, determines that whether sending control command meets to data, services end according to the result after process:
A., when fault pre-alarming result and/or load prediction results are normal, the data acquiring frequency of monitoring sensor is reduced;
B. when fault pre-alarming result and/or load prediction results are abnormal, improve the data acquiring frequency of monitoring sensor, and repeat step (1)-(9), give the alarm at data, services end simultaneously, the display device of data, services end shows fault pre-alarming result and/or load prediction results in real time, and notifies maintenance personal.
Large data analysis set-up of serving based on distributed time series data of the present invention and method are completed by the cooperation of software and hardware device, but be not limited thereto, and under certain condition, also can be realized by the mode of hardware completely.
Although for illustrative purposes; describe illustrative embodiments of the present invention; but it should be appreciated by those skilled in the art that; when not departing from scope of invention disclosed in claims and spirit; the change of various amendment, interpolation and replacement etc. can be carried out in form and details; and all these change the protection domain that all should belong to claims of the present invention; and application claims protection each department of product and method in each step, can combine with the form of combination in any.Therefore, be not intended to limit the scope of the invention to the description of embodiment disclosed in the present invention, but for describing the present invention.Correspondingly, scope of the present invention not by the restriction of above embodiment, but is limited by claim or its equivalent.
Claims (10)
1. a large data analysis set-up of serving based on distributed time series data, comprise monitoring sensor, Monitoring Data storer, device data storer, Data Integration unit, time series data storer, computing unit, wireless terminal and data, services terminal, wherein Monitoring Data storer respectively with monitoring sensor and Data Integration unit, Data Integration unit also distinguishes connection device data-carrier store and time series data storer, Data Integration unit, time series data storer, computing unit is connected successively with data, services terminal, data, services terminal also respectively with time series data storer, monitoring sensor is connected with wireless terminal, it is characterized in that:
The monitoring of equipment data collected for obtaining monitoring of equipment data in real time or quasi real time, and are transferred to Monitoring Data storer by monitoring sensor;
Monitoring Data storer, for monitoring of equipment data being stored, and in the mode that streaming exports, exports monitoring of equipment data to Data Integration unit;
Device data storer, for storing conventional device data, and in the mode of batch signatures, exports the device data of routine to Data Integration unit;
Data Integration unit, mode for accessing with streaming receives the monitoring of equipment data sent from Monitoring Data storer and the mode accessed with batch, by the device data of the routine in predefined operation plan automatic acquisition device data storer, wherein Data Integration unit also comprises pretreatment unit, carry out the pre-service of the cleaning of data, filtration, conversion for the monitoring of equipment data that will receive and conventional device data with preprocessing rule, and export pretreated data to time series data storer;
Time series data storer, for storing pretreated data and configuration data, and visiting frequency is high, that performance requirement is high data centralization is cached to the internal memory in time series data storer;
Computing unit, for the data driving scheduling engine to call and receive time series data storer to store, and foundation in advance programmed processing logic processes the data called and receive, training formed data mining model, also for by the data back after computing unit process to time series data storer and/or data, services end;
Data, services end, comprise data, services end processor, interface unit and display device, wherein data, services end processor is used for directly reading data from time series data storer and/or receiving from the data after computing unit process, and carry out analyzing and processing, result after process is shown by display device, by interface unit, the result after process is sent to wireless terminal simultaneously;
Wireless terminal, for receive from data, services end send process after result, and can wireless transmission control command to data, services end, data, services termination controls monitoring sensor after receiving control command, adjustment monitoring sensor data acquiring frequency.
2. device as claimed in claim 1, is characterized in that: described monitoring sensor is the information acquisition sensor be installed on monitoring equipment.
3. device as claimed in claim 1, is characterized in that: described monitoring sensor is camera and/or the temperature detector of monitoring equipment installation region.
4. the device as described in any one of Claims 2 or 3, is characterized in that: also comprise the manual input device be connected with Data Integration unit, for the Input Monitor Connector device data when implementing quarantine measures because of safety requirements or do not support data access.
5. device as claimed in claim 4, is characterized in that: configuration data is the business implication data of the device data describing monitoring of equipment data and/or routine, storage organization data and/or processing logic data.
6. device as claimed in claim 5, it is characterized in that: the data that described visiting frequency is high, performance requirement is high refer to recent Monitoring Data, conventional device data, and history achievement data, model metadata and preprocessing rule data that concern rate is higher.
7. device as claimed in claim 6, it is characterized in that: described computing unit also runs for calling and receive equipment in time series data storer the new time series data produced, and training process is repeated to new time series data, data mining model is upgraded.
8. device as claimed in claim 7, is characterized in that: the result after the process of data, services end is fault pre-alarming result and/or load prediction results.
9. the device as described in any one of claim 1 or 8, is characterized in that: described wireless terminal is notebook computer, panel computer and/or mobile phone.
10. utilize a large data analysing method for the device as described in any one of the claims 1-9, it is characterized in that, in turn include the following steps:
(1) initialization, the initial parameter of setting data service end, the sampling period controlling monitoring sensor according to the initial parameter that sets is 6 times per second, and the sampling time is 1 minute, and average the data of sampling in 1 minute A;
(2) under same initial parameter condition, repeat step (1) 3 time, try to achieve mean value B, C, D of 3 times respectively;
(3) average after mean value A, B, C, D being sued for peace P again:
If A.
then monitoring sensor stable performance, enters step (4);
If B.
then monitoring sensor unstable properties, then enter step (1);
(4) obtain monitoring of equipment data in real time or quasi real time, and store after the monitoring of equipment data collected are transferred to Monitoring Data storer, the mode exported with streaming, exports monitoring of equipment data to Data Integration unit;
(5) in the mode of batch access, by the device data of the routine in predefined operation plan automatic acquisition device data storer, monitoring of equipment data and conventional device data are carried out the pre-service of the cleaning of data, filtration, conversion with preprocessing rule, and export pretreated data to time series data storer and store;
(6) by recent Monitoring Data, conventional device data, and higher history achievement data, model metadata and the preprocessing rule data centralization of concern rate is cached to the internal memory in time series data storer;
(7) scheduling engine is driven to call and receive the data of time series data storer storage by computing unit, and foundation in advance programmed processing logic processes the data called and receive, training forms data mining model, by the data back after computing unit process to time series data storer and/or data, services end;
(8) directly read data from time series data storer and/or receive from the data after computing unit process, and carry out analyzing and processing, result after process is shown by display device, by interface unit, the result after process is sent to wireless terminal simultaneously;
(9) result after the process sent from data, services end by wireless terminal reception, determine whether send control command to data, services end according to the result after process, data, services termination controls monitoring sensor after receiving control command, the data acquiring frequency of adjustment monitoring sensor, result wherein after process is fault pre-alarming result and/or load prediction results, determines that whether sending control command meets to data, services end according to the result after process:
A., when fault pre-alarming result and/or load prediction results are normal, the data acquiring frequency of monitoring sensor is reduced;
B. when fault pre-alarming result and/or load prediction results are abnormal, improve the data acquiring frequency of monitoring sensor, and repeat step (1)-(9), give the alarm at data, services end simultaneously, the display device of data, services end shows fault pre-alarming result and/or load prediction results in real time, and notifies maintenance personal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510962481.9A CN105427193B (en) | 2015-12-17 | 2015-12-17 | A kind of big data analysis device and method based on distributed time series data service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510962481.9A CN105427193B (en) | 2015-12-17 | 2015-12-17 | A kind of big data analysis device and method based on distributed time series data service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105427193A true CN105427193A (en) | 2016-03-23 |
| CN105427193B CN105427193B (en) | 2019-11-12 |
Family
ID=55505375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510962481.9A Active CN105427193B (en) | 2015-12-17 | 2015-12-17 | A kind of big data analysis device and method based on distributed time series data service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105427193B (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202566A (en) * | 2016-08-02 | 2016-12-07 | 山东鲁能软件技术有限公司 | A kind of magnanimity electricity consumption data mixing based on big data storage system and method |
| CN106294829A (en) * | 2016-08-17 | 2017-01-04 | 东莞前沿技术研究院 | A kind of Internet of Things database construction method and device |
| CN106446091A (en) * | 2016-09-13 | 2017-02-22 | 北京协力筑成金融信息服务股份有限公司 | Preprocessing method and device for multi-source time series data |
| CN106776907A (en) * | 2016-11-30 | 2017-05-31 | 江苏省邮电规划设计院有限责任公司 | A kind of smart city mass data collection optimization method |
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN106874483A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned quality of data evaluation and test based on big data technology |
| CN107276912A (en) * | 2016-04-07 | 2017-10-20 | 华为技术有限公司 | Memory, message processing method and distributed memory system |
| CN108681556A (en) * | 2018-04-08 | 2018-10-19 | 华中科技大学 | The access method and its system of distributed instruction numeric field data |
| CN108803538A (en) * | 2018-07-31 | 2018-11-13 | 山东超越数控电子股份有限公司 | A kind of factory's monitoring System and method for based on time series database |
| CN109219118A (en) * | 2018-11-27 | 2019-01-15 | 上海交通大学 | A kind of distributed consistent state estimation method based on sequence wake-up mechanism |
| CN109274093A (en) * | 2018-10-19 | 2019-01-25 | 北京中电普华信息技术有限公司 | A kind of electric power resource scheduling system |
| CN109521372A (en) * | 2018-10-24 | 2019-03-26 | 安徽立卓智能电网科技有限公司 | A kind of distributed new and network data analysis and diagnosis system |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
| CN110837516A (en) * | 2019-11-07 | 2020-02-25 | 恩亿科(北京)数据科技有限公司 | Data cutting and connecting method and device, computer equipment and readable storage medium |
| CN111581049A (en) * | 2020-04-08 | 2020-08-25 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for monitoring running state of distributed system |
| CN111966695A (en) * | 2020-10-23 | 2020-11-20 | 山东鼎滏软件科技有限公司 | Time sequence database system for oil and gas field production |
| CN113590724A (en) * | 2021-08-03 | 2021-11-02 | 厦门至恒融兴信息技术股份有限公司 | Data index comprehensive management and visual evaluation method and system |
| CN113627681A (en) * | 2021-08-25 | 2021-11-09 | 平安国际智慧城市科技股份有限公司 | Data prediction method and device based on prediction model, computer equipment and medium |
| CN118550920A (en) * | 2024-07-24 | 2024-08-27 | 浙江浙能数字科技有限公司 | Calculation method and device based on time sequence data calculation points |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103199628A (en) * | 2013-04-28 | 2013-07-10 | 国家电网公司 | Real-time power equipment monitoring system based on OSGA (Open Grid Service Architecture) grid technology |
-
2015
- 2015-12-17 CN CN201510962481.9A patent/CN105427193B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103199628A (en) * | 2013-04-28 | 2013-07-10 | 国家电网公司 | Real-time power equipment monitoring system based on OSGA (Open Grid Service Architecture) grid technology |
Non-Patent Citations (2)
| Title |
|---|
| 王德文等: "电力用户侧大数据分析与并行负荷预测", 《中国电机工程学报》 * |
| 王继业等: "智能配用电大数据需求分析与应用研究", 《中国电机工程学报》 * |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107276912B (en) * | 2016-04-07 | 2021-08-27 | 华为技术有限公司 | Memory, message processing method and distributed storage system |
| CN107276912A (en) * | 2016-04-07 | 2017-10-20 | 华为技术有限公司 | Memory, message processing method and distributed memory system |
| CN106202566A (en) * | 2016-08-02 | 2016-12-07 | 山东鲁能软件技术有限公司 | A kind of magnanimity electricity consumption data mixing based on big data storage system and method |
| CN106294829A (en) * | 2016-08-17 | 2017-01-04 | 东莞前沿技术研究院 | A kind of Internet of Things database construction method and device |
| CN106446091A (en) * | 2016-09-13 | 2017-02-22 | 北京协力筑成金融信息服务股份有限公司 | Preprocessing method and device for multi-source time series data |
| CN106776907A (en) * | 2016-11-30 | 2017-05-31 | 江苏省邮电规划设计院有限责任公司 | A kind of smart city mass data collection optimization method |
| CN106776907B (en) * | 2016-11-30 | 2019-08-02 | 中通服咨询设计研究院有限公司 | A kind of smart city mass data collection optimization method |
| CN106874482A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned data prediction based on big data technology |
| CN106874483A (en) * | 2017-02-20 | 2017-06-20 | 山东鲁能软件技术有限公司 | A kind of device and method of the patterned quality of data evaluation and test based on big data technology |
| CN108681556A (en) * | 2018-04-08 | 2018-10-19 | 华中科技大学 | The access method and its system of distributed instruction numeric field data |
| CN108681556B (en) * | 2018-04-08 | 2021-01-29 | 华中科技大学 | Distributed instruction domain data access method and system |
| CN108803538A (en) * | 2018-07-31 | 2018-11-13 | 山东超越数控电子股份有限公司 | A kind of factory's monitoring System and method for based on time series database |
| CN109274093A (en) * | 2018-10-19 | 2019-01-25 | 北京中电普华信息技术有限公司 | A kind of electric power resource scheduling system |
| CN109521372A (en) * | 2018-10-24 | 2019-03-26 | 安徽立卓智能电网科技有限公司 | A kind of distributed new and network data analysis and diagnosis system |
| CN109521372B (en) * | 2018-10-24 | 2020-11-27 | 安徽立卓智能电网科技有限公司 | Distributed new energy grid-connected data analysis and diagnosis system |
| CN109219118A (en) * | 2018-11-27 | 2019-01-15 | 上海交通大学 | A kind of distributed consistent state estimation method based on sequence wake-up mechanism |
| CN109219118B (en) * | 2018-11-27 | 2021-08-24 | 上海交通大学 | Distributed consistent state estimation method based on sequence awakening mechanism |
| CN109740941A (en) * | 2019-01-04 | 2019-05-10 | 北京环境特性研究所 | Military products data processing system |
| CN110837516A (en) * | 2019-11-07 | 2020-02-25 | 恩亿科(北京)数据科技有限公司 | Data cutting and connecting method and device, computer equipment and readable storage medium |
| CN111581049A (en) * | 2020-04-08 | 2020-08-25 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for monitoring running state of distributed system |
| WO2021203635A1 (en) * | 2020-04-08 | 2021-10-14 | 北京百度网讯科技有限公司 | Distributed system running state monitoring method and apparatus, device, and storage medium |
| CN111581049B (en) * | 2020-04-08 | 2023-10-10 | 北京百度网讯科技有限公司 | Distributed system running state monitoring method, device, equipment and storage medium |
| CN111966695B (en) * | 2020-10-23 | 2021-01-08 | 山东鼎滏软件科技有限公司 | Time sequence database system for oil and gas field production |
| CN111966695A (en) * | 2020-10-23 | 2020-11-20 | 山东鼎滏软件科技有限公司 | Time sequence database system for oil and gas field production |
| CN113590724A (en) * | 2021-08-03 | 2021-11-02 | 厦门至恒融兴信息技术股份有限公司 | Data index comprehensive management and visual evaluation method and system |
| CN113627681A (en) * | 2021-08-25 | 2021-11-09 | 平安国际智慧城市科技股份有限公司 | Data prediction method and device based on prediction model, computer equipment and medium |
| CN118550920A (en) * | 2024-07-24 | 2024-08-27 | 浙江浙能数字科技有限公司 | Calculation method and device based on time sequence data calculation points |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105427193B (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105427193A (en) | Device and method for big data analysis based on distributed time sequence data service | |
| CN105608758B (en) | A kind of big data analysis platform device and method calculated based on algorithm configuration and distributed stream | |
| CN105608144B (en) | A kind of big data analysis stage apparatus and method based on multilayered model iteration | |
| CN107943668B (en) | Computer server cluster log monitoring method and monitor supervision platform | |
| CN106202566A (en) | A kind of magnanimity electricity consumption data mixing based on big data storage system and method | |
| CN112462696A (en) | Intelligent manufacturing workshop digital twin model construction method and system | |
| CN106874483A (en) | A kind of device and method of the patterned quality of data evaluation and test based on big data technology | |
| CN106874482A (en) | A kind of device and method of the patterned data prediction based on big data technology | |
| CN105184886A (en) | Cloud data center intelligence inspection system and cloud data center intelligence inspection method | |
| CN112181960A (en) | Intelligent operation and maintenance framework system based on AIOps | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN111383130A (en) | Full life cycle management and control platform, mobile operation and maintenance client and terminal | |
| CN106407429A (en) | File tracking method, device and system | |
| KR20190106369A (en) | Programmable Logic Controller pattern analysis system for improving lifetime and productivity of factory automation facilities, and method thereof | |
| WO2024066683A1 (en) | Industrial internet operating system and product processing method | |
| CN115391444A (en) | Heterogeneous data acquisition and interaction method, device, equipment and storage medium | |
| Kun et al. | Application of big data technology in scientific research data management of military enterprises | |
| CN113111261A (en) | Data processing method of cloud platform, cloud platform and panoramic analysis system | |
| CN112732802A (en) | Enterprise data mining system and method based on big data | |
| CN104573074A (en) | High-speed calculating and analyzing method based on hospital data | |
| CN111125450A (en) | Management method of multilayer topology network resource object | |
| CN113610225A (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| CN104021079B (en) | Real-time monitoring and analyzing system and method for host system data | |
| CN113837281B (en) | Metallurgical factory Internet platform and data regeneration method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 250101 5th floor, block B, Yinhe building, 2008 Xinluo street, high tech Zone, Jinan City, Shandong Province Patentee after: Shandong luruan Digital Technology Co.,Ltd. Address before: 250101 5th floor, block B, Yinhe building, 2008 Xinluo street, high tech Zone, Jinan City, Shandong Province Patentee before: SHANDONG LUNENG SOFTWARE TECHNOLOGY Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder |