CN106951464A - Based on the personalized early warning mechanism big data computational methods of storm user orienteds - Google Patents
Based on the personalized early warning mechanism big data computational methods of storm user orienteds Download PDFInfo
- Publication number
- CN106951464A CN106951464A CN201710108940.6A CN201710108940A CN106951464A CN 106951464 A CN106951464 A CN 106951464A CN 201710108940 A CN201710108940 A CN 201710108940A CN 106951464 A CN106951464 A CN 106951464A
- Authority
- CN
- China
- Prior art keywords
- data
- alarm
- storm
- module
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The present invention discloses one kind based on the personalized early warning mechanism big data computational methods of storm user orienteds, comprises the concrete steps that:User by interface according to equipment of itself set alarming logic, after setting successfully, system will according to agreement data structure storage alarming logic;The data structure of storage is read in data cache module timing, and storm alarm module combination alarming logics read the real time data sent from terminal with server gateway and carry out logic judgment;If alert data, will further determine in redis whether there is alarm history, if updating respective record in the presence of if, if one alarm logging of insertion in the absence of if;If it is not, the alarm history in redis is judged, if in the presence of representing to alarm has released, then deletes this record.Processing speed of the present invention is fast, supports mass data processing, and user individual early warning scheme has been used in logic in processing, and user can define alarm scheme according to the actual conditions of oneself equipment.
Description
Technical field
The present invention relates to one kind based on the personalized early warning mechanism big data computational methods of storm user orienteds, belong to data
Processing technology field.
Background technology
With the development of Internet technology, the technology such as Cloud Server, cloud storage is maked rapid progress, and the big data epoch come
Face, in order to it is more preferable, excavate valuable information faster, give client more preferable Consumer's Experience, traditional batch-compute model
These demands can not be met, the big data technology such as distributed treatment, cluster, Stream Processing is born therewith.Storm is used as mesh
One of preceding most fiery Stream Processing framework, in the big data processing platform that may be used on all trades and professions.In construction machinery industry, with
Sensor technology, the development of data acquisition technology, the floor data that terminal device can be gathered is more and more, data volume also with
Sharp increase, how therefrom to extract useful data, faster analyze useful information and be presented to user, be at present urgently
The problem of solution.
At present, operating mode early warning technology in construction machinery industry, typically using flow chart of data processing as shown in Figure 1, service
Device gateway is responsible for obtaining initial data from terminal device;Parsing module parses floor data from gateway;Alarm module is to parsing
The data that module is obtained are processed;Relevant database carries out data storage to alert data, is obtained for foreground interface
Display.This technology can solve traditional data processing and data storage problem, but when there is mass data and need processing, meter
Bottleneck can be all run into calculation machine performance and data storage, can only be configured by improving server, buy expensive hardware facility etc.
Mode solves the matter of great urgency, but also can only temporarily solve problem, it is impossible to tackle the problem at its root;And type of alarm is relative
It is fixed, it is impossible to according to the demand of user, allow user to change alarm scheme at any time, user experience is poor.
The content of the invention
The problem of existing for above-mentioned prior art, the present invention provides a kind of based on the personalized early warning of storm user orienteds
Mechanism big data computational methods, can fundamentally solve the drawbacks of traditional solution is present, more flexible more practical.
To achieve these goals, one kind that the present invention is used is based on the big number of the personalized early warning mechanism of storm user orienteds
According to computational methods, including data preprocessing module, data resolution module, data memory module and alert process module;Data are pre-
Processing module includes server gateway and kafka message queues, and kafka message queues are responsible for gateway data and parsing data
Bridge, server gateway from obtain terminal send initial data deposit kafka topic in, be used as data resolution module
The producer of processing;Data resolution module is storm parsing modules, is responsible for reading data from kafka topic, and right
Initial data in topic is parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, logarithm
According to being parsed one by one;Data memory module includes data cache module, cache database update module and history alarm storage
Module;Alert process module is storm alarm modules, on the one hand, User Defined alarm side is obtained from data cache module
Case, on the other hand, analyzes alarming logic, and analyze alarming result according to these alarm schemes;Based on storm towards with
Family personalization early warning mechanism big data computational methods are comprised the concrete steps that:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to about
The parameter and threshold value of fixed data structure storage alarming logic;
Step 2: the data structure stored in the read step from database one of data cache module timing, and be used for
Next step storm alarm modules;Wherein, the time of timing can be set by user oneself;
Step 3: parameter and threshold value and service of the storm alarm modules with reference to needed for reading alarm from data cache module
Device gateway reads the real time data sent from terminal and carries out logic judgment;
Step 4: the judgement by step 3 determines whether this data is alert data, if alert data, it will enter
One step judges the alarm history that whether there is equipment in redis, if there is then update redis in respective record, such as
Fruit is not present then inserts an alarm logging into redis;If not alert data, the alarm in redis is equally judged
History, if it does, representing that alarm has been released, then deletes this record from redis.
The cache database update module carries out real-time, interactive using caching technology redis with interface.
The history alarm memory module carries out distributed storage using hadoop hdfs technologies.
Compared with prior art, the present invention contain data preprocessing module, data resolution module, alert process module and
Data memory module;Using caching preloading technology in data processing, set according to User Defined alarm scheme, to operating mode
Carry out Realtime Alerts.In framework type selecting, increased income frame using the big data such as more popular hadoop, kafka, storm at present
Frame, wherein, storm distributive types computational methods are used to handle condition alarm, and this method can quickly carry out big data in real time
Processing, is supported extending transversely.What hadoop, kafka, storm technology were used is all distributed deployment, hardware side when data volume is big
Face, it is only necessary to select to set up cluster between the server of general performance, multiple servers, supports dilatation at any time, solves
Mass data storage problem.On the other hand, real-time data memory carries out being handed in real time with interface using caching technology redis
Mutually, digital independent performance is greatly improved, the hdfs technologies that historical data employs hadoop carry out distributed storage.Meanwhile,
In terms of Consumer's Experience, using data preload technology, support User Defined alarming logic, user can according to self-condition,
The early warning scheme for being adapted to equipment of itself is set up, and alert if can be dynamically set, can be promptly and accurately in equipment availability
Monitoring device operation, the work of correct Instructing manufacture.Compared with traditional warning system processing method, this method is more flexible,
Processing speed faster, supports mass data processing, while in processing in logic, having used user individual early warning scheme, user
Alarm scheme can be defined according to the actual conditions of oneself equipment.
Brief description of the drawings
Fig. 1 is conventional architectures schematic diagram;
Fig. 2 is new architecture schematic diagram of the invention;
Fig. 3 is big data cluster schematic diagram of the present invention;
Fig. 4 is alarming logic schematic flow sheet of the invention.
Embodiment
The invention will be further described below in conjunction with the accompanying drawings.
As shown in Figures 2 to 4, it is a kind of based on the personalized early warning mechanism big data computational methods of storm user orienteds, including
Data preprocessing module, data resolution module, data memory module and alert process module;Data preprocessing module includes service
Device gateway and kafka message queues, kafka message queues are responsible for the bridge of gateway data and parsing data, server net
Close from the topic for obtaining the initial data deposit kafka that terminal is sent, the producer handled as data resolution module;Number
It is storm parsing modules according to parsing module, is responsible for reading data from kafka topic, and to the initial data in topic
Parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, and data are parsed one by one;Number
Include data cache module, cache database update module and history alarm memory module according to memory module;Alert process module
For storm alarm modules, on the one hand, User Defined alarm scheme is obtained from data cache module, on the other hand, according to this
A little alarm schemes analyze alarming logic, and analyze alarming result;It is big based on the personalized early warning mechanism of storm user orienteds
Method for computing data is comprised the concrete steps that:
First, big data cluster is built.In the server pre-established dispose hadoop cluster, storm clusters,
Kafka clusters and corresponding Service Database and redis databases, as shown in figure 3, showing for big data clustered deploy(ment) of the present invention
It is intended to, specific dispositions method refers to Apache official websites.
Secondly, storm is created topological (topology).Topology is actual Business Processing framework, comprising multiple
Processing unit (Tuple), Tuple is divided into spout and bolt types again, and spout is responsible for creation data, refers specifically to from kafka
Data source is read, bolt is responsible for consumption data, refers specifically to processing data source.Each processing module mentioned above is at bolt
The form of expression of reason.It is responsible for invention creates spout and two bolt, spout from the topic receive datas in kafka
According to source (being named as Originaltopic), the data that as gateway is sent, and serialize in pairs as;Two bolt correspondences two
Module:Parsing module and alarm module.
Then, alarm module is implemented.In the present invention, alarm module is specific Service Processing Module, as shown in figure 4, tool
Body logic is as follows:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to about
The parameter and threshold value of fixed data structure storage alarming logic;Such as:It is xml document, file that the present invention, which defines data store organisation,
In set the content in multiple nodes, each node to contain the corresponding specific logic of parsing, one defined in the present invention<
function>Node is user-defined alarming logic, and content is " var result=parseInt
(Waterlevelalarm)==1 ", the literary style represents, when Waterlevelalarm values are for 1, to start " water-level alarm ", after
Platform program will load the xml document, and parse the content in the node of the xml document, assign the variable in program as parameter knot
Close the expression formula and determine whether alarm.
Step 2: the data structure stored in the read step from database one of data cache module timing, and be used for
Next step storm alarm modules;Wherein, the time of timing (can be changed) by user oneself setting in configurable file, the present invention
Definition to the parameter is read out by the way of configuration file, and user can be according to the modification to configuration file, setting time
Cycle.
Step 3: parameter and threshold value and service of the storm alarm modules with reference to needed for reading alarm from data cache module
Device gateway reads the real time data sent from terminal and carries out logic judgment;
Step 4: the judgement by step 3 determines whether this data is alert data, if alert data, it will enter
One step judges the alarm history that whether there is equipment in redis, if there is then update redis in respective record, such as
Fruit is not present then inserts an alarm logging into redis;If not alert data, the alarm in redis is equally judged
History, if it does, representing that alarm has been released, then deletes this record from redis.
It is finally to submit topology.After the code logic of two modules of completion is write in storm, jar bags are broken into, in service
The server where storm nimbus nodes is installed in selection in device, and operation storm submits topology order, and now, newly-built opens up
Flutter to operate in process always, ceaselessly perform specifying for task.
It is an advantage of the current invention that storm streamings are calculated, its high-performance, low latency, high fault freedom are adapted to real-time early warning
The processing of data;Using distributed system, support that server is extending transversely, need to only increase common server section when data volume is big
Point;Using Open Framework, economic performance;Users' personal allocation alarm scheme, user can be according to equipment of itself actual conditions, certainly
Define alarm scheme;Data storage can there is provided data access speed using redis cache databases.
The cache database update module carries out real-time, interactive using caching technology redis with interface.
The history alarm memory module carries out distributed storage using hadoop hdfs technologies.
Storm is as most fiery Stream Processing method at present in the present invention, carry in whole alarm data parsing and
The vital task of alert process, because the characteristics such as its high-performance, low latency, fault-tolerance can fast and accurately processing data.hadoop、
What kafka, storm technology were used is all distributed deployment, supports that server is extending transversely, increase by one is only needed to when data volume is big
As performance server, cluster is set up between multiple servers, dilatation at any time is supported, solves mass data storage and ask
Topic.On the other hand, real-time data memory carries out carrying out real-time, interactive with interface using caching technology redis, greatly improves number
According to reading performance, the hdfs technologies that historical data employs hadoop carry out distributed storage.Meanwhile, in terms of Consumer's Experience,
Technology is preloaded using data, User Defined alarming logic is supported, user can set up according to self-condition and be adapted to equipment of itself
Early warning scheme, and alert if can be dynamically set, monitoring device operation that can be promptly and accurately in equipment availability, just
True Instructing manufacture work.
Generally speaking, the system contains data preprocessing module, data resolution module, alert process module and data and deposited
Store up module;Using caching preloading technology in data processing, set, operating mode is carried out real according to User Defined alarm scheme
Alarm.In framework type selecting, using the big data Open Framework such as more popular hadoop, kafka, storm at present, its
In, storm distributive types computational methods are used to handle condition alarm, and this method can quickly carry out big data processing in real time,
Support extending transversely.Compared to traditional warning system processing method, this method is more flexible, and processing speed faster, supports magnanimity number
According to processing, while in processing in logic, used user individual early warning scheme, user can be according to the actual feelings of oneself equipment
Condition, defines alarm scheme.
Claims (3)
1. one kind is based on the personalized early warning mechanism big data computational methods of storm user orienteds, it is characterised in that pre- including data
Processing module, data resolution module, data memory module and alert process module;
Data preprocessing module includes server gateway and kafka message queues, and kafka message queues are responsible for gateway data
Bridge with parsing data, server gateway is used as number from the topic for obtaining the initial data deposit kafka that terminal is sent
The producer handled according to parsing module;
Data resolution module is storm parsing modules, is responsible for reading data from kafka topic, and to the original in topic
Beginning data are parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, and data are carried out one by one
Parsing;
Data memory module includes data cache module, cache database update module and history alarm memory module;
Alert process module is storm alarm modules, on the one hand, User Defined alarm side is obtained from data cache module
Case, on the other hand, analyzes alarming logic, and analyze alarming result according to these alarm schemes;
Based on comprising the concrete steps that for the personalized early warning mechanism big data computational methods of storm user orienteds:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to agreement
The parameter and threshold value of data structure storage alarming logic;
Step 2: the data structure stored in the read step from database one of data cache module timing, and for next
Walk storm alarm modules;Wherein, the time of timing can be set by user oneself;
Step 3: parameter and threshold value and server net of the storm alarm modules with reference to needed for reading alarm from data cache module
Close the real time data progress logic judgment for reading and sending from terminal;
Step 4: the judgement by step 3 determines whether this data is alert data, will be further if alert data
Judge whether there is the alarm history of equipment in redis, if there is the respective record then updated in redis, if not
In the presence of then one alarm logging of insertion into redis;If not alert data, the alarm history in redis is equally judged,
If it does, representing that alarm has been released, then this record is deleted from redis.
2. one kind according to claim 1 is based on the personalized early warning mechanism big data computational methods of storm user orienteds, its
It is characterised by, the cache database update module carries out real-time, interactive using caching technology redis with interface.
3. one kind according to claim 1 is based on the personalized early warning mechanism big data computational methods of storm user orienteds, its
It is characterised by, the history alarm memory module carries out distributed storage using hadoop hdfs technologies.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710108940.6A CN106951464A (en) | 2017-02-27 | 2017-02-27 | Based on the personalized early warning mechanism big data computational methods of storm user orienteds |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710108940.6A CN106951464A (en) | 2017-02-27 | 2017-02-27 | Based on the personalized early warning mechanism big data computational methods of storm user orienteds |
Publications (1)
Publication Number | Publication Date |
---|---|
CN106951464A true CN106951464A (en) | 2017-07-14 |
Family
ID=59467728
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710108940.6A Pending CN106951464A (en) | 2017-02-27 | 2017-02-27 | Based on the personalized early warning mechanism big data computational methods of storm user orienteds |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106951464A (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107508888A (en) * | 2017-08-25 | 2017-12-22 | 同方(深圳)云计算技术股份有限公司 | A kind of car networking service platform |
CN108449235A (en) * | 2018-05-07 | 2018-08-24 | 苏州德姆斯信息技术有限公司 | Equipment alarm calculation processing system and processing method |
CN110035096A (en) * | 2018-01-12 | 2019-07-19 | 中科院微电子研究所昆山分所 | A kind of vehicle early warning processing system and early warning system based on Storm |
CN110677276A (en) * | 2019-09-09 | 2020-01-10 | 杭州玖欣物联科技有限公司 | System for realizing multi-user hot deployment supported by industrial internet data processing |
CN112134860A (en) * | 2020-09-09 | 2020-12-25 | 深圳中兴网信科技有限公司 | Processing method and device of environmental monitoring data, air micro-station and storage medium |
CN117076699A (en) * | 2023-10-13 | 2023-11-17 | 南京奥看信息科技有限公司 | Multi-picture acceleration processing method and device and electronic equipment |
CN117149897A (en) * | 2023-10-31 | 2023-12-01 | 成都交大光芒科技股份有限公司 | Big data alarm information hierarchical display system and method based on double-buffer technology |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101282342A (en) * | 2008-05-30 | 2008-10-08 | 腾讯科技(深圳)有限公司 | Method and system for fetching network contents |
CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
WO2016137848A4 (en) * | 2015-02-23 | 2016-12-08 | Prophecy Sensors, Llc | Internet of things based determination of machine reliability and automated maintainenace, repair and operation (mro) logs |
-
2017
- 2017-02-27 CN CN201710108940.6A patent/CN106951464A/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101282342A (en) * | 2008-05-30 | 2008-10-08 | 腾讯科技(深圳)有限公司 | Method and system for fetching network contents |
CN104618343A (en) * | 2015-01-06 | 2015-05-13 | 中国科学院信息工程研究所 | Method and system for detecting website threat based on real-time log |
WO2016137848A4 (en) * | 2015-02-23 | 2016-12-08 | Prophecy Sensors, Llc | Internet of things based determination of machine reliability and automated maintainenace, repair and operation (mro) logs |
CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107508888A (en) * | 2017-08-25 | 2017-12-22 | 同方(深圳)云计算技术股份有限公司 | A kind of car networking service platform |
CN110035096A (en) * | 2018-01-12 | 2019-07-19 | 中科院微电子研究所昆山分所 | A kind of vehicle early warning processing system and early warning system based on Storm |
CN108449235A (en) * | 2018-05-07 | 2018-08-24 | 苏州德姆斯信息技术有限公司 | Equipment alarm calculation processing system and processing method |
CN108449235B (en) * | 2018-05-07 | 2023-12-12 | 苏州德姆斯信息技术有限公司 | Equipment alarm calculation processing system and processing method |
CN110677276A (en) * | 2019-09-09 | 2020-01-10 | 杭州玖欣物联科技有限公司 | System for realizing multi-user hot deployment supported by industrial internet data processing |
CN112134860A (en) * | 2020-09-09 | 2020-12-25 | 深圳中兴网信科技有限公司 | Processing method and device of environmental monitoring data, air micro-station and storage medium |
CN117076699A (en) * | 2023-10-13 | 2023-11-17 | 南京奥看信息科技有限公司 | Multi-picture acceleration processing method and device and electronic equipment |
CN117076699B (en) * | 2023-10-13 | 2023-12-12 | 南京奥看信息科技有限公司 | Multi-picture acceleration processing method and device and electronic equipment |
CN117149897A (en) * | 2023-10-31 | 2023-12-01 | 成都交大光芒科技股份有限公司 | Big data alarm information hierarchical display system and method based on double-buffer technology |
CN117149897B (en) * | 2023-10-31 | 2024-01-26 | 成都交大光芒科技股份有限公司 | Big data alarm information hierarchical display system and method based on double-buffer technology |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106951464A (en) | Based on the personalized early warning mechanism big data computational methods of storm user orienteds | |
CN107577805B (en) | Business service system for log big data analysis | |
CN110263921B (en) | Method and device for training federated learning model | |
CN105824744B (en) | A kind of real-time logs capturing analysis method based on B2B platform | |
CN111652380A (en) | Method and system for adjusting and optimizing algorithm parameters aiming at machine learning algorithm | |
CN110362544A (en) | Log processing system, log processing method, terminal and storage medium | |
US11132293B2 (en) | Intelligent garbage collector for containers | |
US20210097431A1 (en) | Debugging and profiling of machine learning model training | |
CN109726004B (en) | Data processing method and device | |
CN110929489A (en) | Form generation and form data processing method and system | |
WO2019001359A1 (en) | Data processing method and data processing apparatus | |
CN106815254A (en) | A kind of data processing method and device | |
US10019505B2 (en) | Method and apparatus for creating data cube in streaming manner based on distributed system | |
CN103701635A (en) | Method and device for configuring Hadoop parameters on line | |
US20230368028A1 (en) | Automated machine learning pre-trained model selector | |
US20200394448A1 (en) | Methods for more effectively moderating one or more images and devices thereof | |
CN111523324A (en) | Training method and device for named entity recognition model | |
WO2019109798A1 (en) | Method, device, terminal and storage medium for loading resource | |
CN112507330A (en) | Malicious software detection system based on distributed sandbox | |
CN103914531A (en) | Method and device for processing data | |
CN108073582B (en) | Computing framework selection method and device | |
US11468365B2 (en) | GPU code injection to summarize machine learning training data | |
CN116383521B (en) | Subject word mining method and device, computer equipment and storage medium | |
US20190340540A1 (en) | Adaptive continuous log model learning | |
CN111130882A (en) | Monitoring system and method of network equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170714 |