CN106951464A - Based on the personalized early warning mechanism big data computational methods of storm user orienteds - Google Patents

Based on the personalized early warning mechanism big data computational methods of storm user orienteds Download PDF

Info

Publication number
CN106951464A
CN106951464A CN201710108940.6A CN201710108940A CN106951464A CN 106951464 A CN106951464 A CN 106951464A CN 201710108940 A CN201710108940 A CN 201710108940A CN 106951464 A CN106951464 A CN 106951464A
Authority
CN
China
Prior art keywords
data
alarm
storm
module
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710108940.6A
Other languages
Chinese (zh)
Inventor
张启亮
王飞
朱晓委
张毅
邵世龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Xugong Information Technology Ltd By Share Ltd
Original Assignee
Jiangsu Xugong Information Technology Ltd By Share Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Xugong Information Technology Ltd By Share Ltd filed Critical Jiangsu Xugong Information Technology Ltd By Share Ltd
Priority to CN201710108940.6A priority Critical patent/CN106951464A/en
Publication of CN106951464A publication Critical patent/CN106951464A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/252Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3089Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3093Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

The present invention discloses one kind based on the personalized early warning mechanism big data computational methods of storm user orienteds, comprises the concrete steps that:User by interface according to equipment of itself set alarming logic, after setting successfully, system will according to agreement data structure storage alarming logic;The data structure of storage is read in data cache module timing, and storm alarm module combination alarming logics read the real time data sent from terminal with server gateway and carry out logic judgment;If alert data, will further determine in redis whether there is alarm history, if updating respective record in the presence of if, if one alarm logging of insertion in the absence of if;If it is not, the alarm history in redis is judged, if in the presence of representing to alarm has released, then deletes this record.Processing speed of the present invention is fast, supports mass data processing, and user individual early warning scheme has been used in logic in processing, and user can define alarm scheme according to the actual conditions of oneself equipment.

Description

Based on the personalized early warning mechanism big data computational methods of storm user orienteds
Technical field
The present invention relates to one kind based on the personalized early warning mechanism big data computational methods of storm user orienteds, belong to data Processing technology field.
Background technology
With the development of Internet technology, the technology such as Cloud Server, cloud storage is maked rapid progress, and the big data epoch come Face, in order to it is more preferable, excavate valuable information faster, give client more preferable Consumer's Experience, traditional batch-compute model These demands can not be met, the big data technology such as distributed treatment, cluster, Stream Processing is born therewith.Storm is used as mesh One of preceding most fiery Stream Processing framework, in the big data processing platform that may be used on all trades and professions.In construction machinery industry, with Sensor technology, the development of data acquisition technology, the floor data that terminal device can be gathered is more and more, data volume also with Sharp increase, how therefrom to extract useful data, faster analyze useful information and be presented to user, be at present urgently The problem of solution.
At present, operating mode early warning technology in construction machinery industry, typically using flow chart of data processing as shown in Figure 1, service Device gateway is responsible for obtaining initial data from terminal device;Parsing module parses floor data from gateway;Alarm module is to parsing The data that module is obtained are processed;Relevant database carries out data storage to alert data, is obtained for foreground interface Display.This technology can solve traditional data processing and data storage problem, but when there is mass data and need processing, meter Bottleneck can be all run into calculation machine performance and data storage, can only be configured by improving server, buy expensive hardware facility etc. Mode solves the matter of great urgency, but also can only temporarily solve problem, it is impossible to tackle the problem at its root;And type of alarm is relative It is fixed, it is impossible to according to the demand of user, allow user to change alarm scheme at any time, user experience is poor.
The content of the invention
The problem of existing for above-mentioned prior art, the present invention provides a kind of based on the personalized early warning of storm user orienteds Mechanism big data computational methods, can fundamentally solve the drawbacks of traditional solution is present, more flexible more practical.
To achieve these goals, one kind that the present invention is used is based on the big number of the personalized early warning mechanism of storm user orienteds According to computational methods, including data preprocessing module, data resolution module, data memory module and alert process module;Data are pre- Processing module includes server gateway and kafka message queues, and kafka message queues are responsible for gateway data and parsing data Bridge, server gateway from obtain terminal send initial data deposit kafka topic in, be used as data resolution module The producer of processing;Data resolution module is storm parsing modules, is responsible for reading data from kafka topic, and right Initial data in topic is parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, logarithm According to being parsed one by one;Data memory module includes data cache module, cache database update module and history alarm storage Module;Alert process module is storm alarm modules, on the one hand, User Defined alarm side is obtained from data cache module Case, on the other hand, analyzes alarming logic, and analyze alarming result according to these alarm schemes;Based on storm towards with Family personalization early warning mechanism big data computational methods are comprised the concrete steps that:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to about The parameter and threshold value of fixed data structure storage alarming logic;
Step 2: the data structure stored in the read step from database one of data cache module timing, and be used for Next step storm alarm modules;Wherein, the time of timing can be set by user oneself;
Step 3: parameter and threshold value and service of the storm alarm modules with reference to needed for reading alarm from data cache module Device gateway reads the real time data sent from terminal and carries out logic judgment;
Step 4: the judgement by step 3 determines whether this data is alert data, if alert data, it will enter One step judges the alarm history that whether there is equipment in redis, if there is then update redis in respective record, such as Fruit is not present then inserts an alarm logging into redis;If not alert data, the alarm in redis is equally judged History, if it does, representing that alarm has been released, then deletes this record from redis.
The cache database update module carries out real-time, interactive using caching technology redis with interface.
The history alarm memory module carries out distributed storage using hadoop hdfs technologies.
Compared with prior art, the present invention contain data preprocessing module, data resolution module, alert process module and Data memory module;Using caching preloading technology in data processing, set according to User Defined alarm scheme, to operating mode Carry out Realtime Alerts.In framework type selecting, increased income frame using the big data such as more popular hadoop, kafka, storm at present Frame, wherein, storm distributive types computational methods are used to handle condition alarm, and this method can quickly carry out big data in real time Processing, is supported extending transversely.What hadoop, kafka, storm technology were used is all distributed deployment, hardware side when data volume is big Face, it is only necessary to select to set up cluster between the server of general performance, multiple servers, supports dilatation at any time, solves Mass data storage problem.On the other hand, real-time data memory carries out being handed in real time with interface using caching technology redis Mutually, digital independent performance is greatly improved, the hdfs technologies that historical data employs hadoop carry out distributed storage.Meanwhile, In terms of Consumer's Experience, using data preload technology, support User Defined alarming logic, user can according to self-condition, The early warning scheme for being adapted to equipment of itself is set up, and alert if can be dynamically set, can be promptly and accurately in equipment availability Monitoring device operation, the work of correct Instructing manufacture.Compared with traditional warning system processing method, this method is more flexible, Processing speed faster, supports mass data processing, while in processing in logic, having used user individual early warning scheme, user Alarm scheme can be defined according to the actual conditions of oneself equipment.
Brief description of the drawings
Fig. 1 is conventional architectures schematic diagram;
Fig. 2 is new architecture schematic diagram of the invention;
Fig. 3 is big data cluster schematic diagram of the present invention;
Fig. 4 is alarming logic schematic flow sheet of the invention.
Embodiment
The invention will be further described below in conjunction with the accompanying drawings.
As shown in Figures 2 to 4, it is a kind of based on the personalized early warning mechanism big data computational methods of storm user orienteds, including Data preprocessing module, data resolution module, data memory module and alert process module;Data preprocessing module includes service Device gateway and kafka message queues, kafka message queues are responsible for the bridge of gateway data and parsing data, server net Close from the topic for obtaining the initial data deposit kafka that terminal is sent, the producer handled as data resolution module;Number It is storm parsing modules according to parsing module, is responsible for reading data from kafka topic, and to the initial data in topic Parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, and data are parsed one by one;Number Include data cache module, cache database update module and history alarm memory module according to memory module;Alert process module For storm alarm modules, on the one hand, User Defined alarm scheme is obtained from data cache module, on the other hand, according to this A little alarm schemes analyze alarming logic, and analyze alarming result;It is big based on the personalized early warning mechanism of storm user orienteds Method for computing data is comprised the concrete steps that:
First, big data cluster is built.In the server pre-established dispose hadoop cluster, storm clusters, Kafka clusters and corresponding Service Database and redis databases, as shown in figure 3, showing for big data clustered deploy(ment) of the present invention It is intended to, specific dispositions method refers to Apache official websites.
Secondly, storm is created topological (topology).Topology is actual Business Processing framework, comprising multiple Processing unit (Tuple), Tuple is divided into spout and bolt types again, and spout is responsible for creation data, refers specifically to from kafka Data source is read, bolt is responsible for consumption data, refers specifically to processing data source.Each processing module mentioned above is at bolt The form of expression of reason.It is responsible for invention creates spout and two bolt, spout from the topic receive datas in kafka According to source (being named as Originaltopic), the data that as gateway is sent, and serialize in pairs as;Two bolt correspondences two Module:Parsing module and alarm module.
Then, alarm module is implemented.In the present invention, alarm module is specific Service Processing Module, as shown in figure 4, tool Body logic is as follows:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to about The parameter and threshold value of fixed data structure storage alarming logic;Such as:It is xml document, file that the present invention, which defines data store organisation, In set the content in multiple nodes, each node to contain the corresponding specific logic of parsing, one defined in the present invention< function>Node is user-defined alarming logic, and content is " var result=parseInt (Waterlevelalarm)==1 ", the literary style represents, when Waterlevelalarm values are for 1, to start " water-level alarm ", after Platform program will load the xml document, and parse the content in the node of the xml document, assign the variable in program as parameter knot Close the expression formula and determine whether alarm.
Step 2: the data structure stored in the read step from database one of data cache module timing, and be used for Next step storm alarm modules;Wherein, the time of timing (can be changed) by user oneself setting in configurable file, the present invention Definition to the parameter is read out by the way of configuration file, and user can be according to the modification to configuration file, setting time Cycle.
Step 3: parameter and threshold value and service of the storm alarm modules with reference to needed for reading alarm from data cache module Device gateway reads the real time data sent from terminal and carries out logic judgment;
Step 4: the judgement by step 3 determines whether this data is alert data, if alert data, it will enter One step judges the alarm history that whether there is equipment in redis, if there is then update redis in respective record, such as Fruit is not present then inserts an alarm logging into redis;If not alert data, the alarm in redis is equally judged History, if it does, representing that alarm has been released, then deletes this record from redis.
It is finally to submit topology.After the code logic of two modules of completion is write in storm, jar bags are broken into, in service The server where storm nimbus nodes is installed in selection in device, and operation storm submits topology order, and now, newly-built opens up Flutter to operate in process always, ceaselessly perform specifying for task.
It is an advantage of the current invention that storm streamings are calculated, its high-performance, low latency, high fault freedom are adapted to real-time early warning The processing of data;Using distributed system, support that server is extending transversely, need to only increase common server section when data volume is big Point;Using Open Framework, economic performance;Users' personal allocation alarm scheme, user can be according to equipment of itself actual conditions, certainly Define alarm scheme;Data storage can there is provided data access speed using redis cache databases.
The cache database update module carries out real-time, interactive using caching technology redis with interface.
The history alarm memory module carries out distributed storage using hadoop hdfs technologies.
Storm is as most fiery Stream Processing method at present in the present invention, carry in whole alarm data parsing and The vital task of alert process, because the characteristics such as its high-performance, low latency, fault-tolerance can fast and accurately processing data.hadoop、 What kafka, storm technology were used is all distributed deployment, supports that server is extending transversely, increase by one is only needed to when data volume is big As performance server, cluster is set up between multiple servers, dilatation at any time is supported, solves mass data storage and ask Topic.On the other hand, real-time data memory carries out carrying out real-time, interactive with interface using caching technology redis, greatly improves number According to reading performance, the hdfs technologies that historical data employs hadoop carry out distributed storage.Meanwhile, in terms of Consumer's Experience, Technology is preloaded using data, User Defined alarming logic is supported, user can set up according to self-condition and be adapted to equipment of itself Early warning scheme, and alert if can be dynamically set, monitoring device operation that can be promptly and accurately in equipment availability, just True Instructing manufacture work.
Generally speaking, the system contains data preprocessing module, data resolution module, alert process module and data and deposited Store up module;Using caching preloading technology in data processing, set, operating mode is carried out real according to User Defined alarm scheme Alarm.In framework type selecting, using the big data Open Framework such as more popular hadoop, kafka, storm at present, its In, storm distributive types computational methods are used to handle condition alarm, and this method can quickly carry out big data processing in real time, Support extending transversely.Compared to traditional warning system processing method, this method is more flexible, and processing speed faster, supports magnanimity number According to processing, while in processing in logic, used user individual early warning scheme, user can be according to the actual feelings of oneself equipment Condition, defines alarm scheme.

Claims (3)

1. one kind is based on the personalized early warning mechanism big data computational methods of storm user orienteds, it is characterised in that pre- including data Processing module, data resolution module, data memory module and alert process module;
Data preprocessing module includes server gateway and kafka message queues, and kafka message queues are responsible for gateway data Bridge with parsing data, server gateway is used as number from the topic for obtaining the initial data deposit kafka that terminal is sent The producer handled according to parsing module;
Data resolution module is storm parsing modules, is responsible for reading data from kafka topic, and to the original in topic Beginning data are parsed according to protocol definition;Wherein, storm data processing mechanism is Stream Processing, and data are carried out one by one Parsing;
Data memory module includes data cache module, cache database update module and history alarm memory module;
Alert process module is storm alarm modules, on the one hand, User Defined alarm side is obtained from data cache module Case, on the other hand, analyzes alarming logic, and analyze alarming result according to these alarm schemes;
Based on comprising the concrete steps that for the personalized early warning mechanism big data computational methods of storm user orienteds:
Step 1: user sets alarming logic by interface according to equipment of itself, after setting successfully, system will be according to agreement The parameter and threshold value of data structure storage alarming logic;
Step 2: the data structure stored in the read step from database one of data cache module timing, and for next Walk storm alarm modules;Wherein, the time of timing can be set by user oneself;
Step 3: parameter and threshold value and server net of the storm alarm modules with reference to needed for reading alarm from data cache module Close the real time data progress logic judgment for reading and sending from terminal;
Step 4: the judgement by step 3 determines whether this data is alert data, will be further if alert data Judge whether there is the alarm history of equipment in redis, if there is the respective record then updated in redis, if not In the presence of then one alarm logging of insertion into redis;If not alert data, the alarm history in redis is equally judged, If it does, representing that alarm has been released, then this record is deleted from redis.
2. one kind according to claim 1 is based on the personalized early warning mechanism big data computational methods of storm user orienteds, its It is characterised by, the cache database update module carries out real-time, interactive using caching technology redis with interface.
3. one kind according to claim 1 is based on the personalized early warning mechanism big data computational methods of storm user orienteds, its It is characterised by, the history alarm memory module carries out distributed storage using hadoop hdfs technologies.
CN201710108940.6A 2017-02-27 2017-02-27 Based on the personalized early warning mechanism big data computational methods of storm user orienteds Pending CN106951464A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710108940.6A CN106951464A (en) 2017-02-27 2017-02-27 Based on the personalized early warning mechanism big data computational methods of storm user orienteds

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710108940.6A CN106951464A (en) 2017-02-27 2017-02-27 Based on the personalized early warning mechanism big data computational methods of storm user orienteds

Publications (1)

Publication Number Publication Date
CN106951464A true CN106951464A (en) 2017-07-14

Family

ID=59467728

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710108940.6A Pending CN106951464A (en) 2017-02-27 2017-02-27 Based on the personalized early warning mechanism big data computational methods of storm user orienteds

Country Status (1)

Country Link
CN (1) CN106951464A (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107508888A (en) * 2017-08-25 2017-12-22 同方(深圳)云计算技术股份有限公司 A kind of car networking service platform
CN108449235A (en) * 2018-05-07 2018-08-24 苏州德姆斯信息技术有限公司 Equipment alarm calculation processing system and processing method
CN110035096A (en) * 2018-01-12 2019-07-19 中科院微电子研究所昆山分所 A kind of vehicle early warning processing system and early warning system based on Storm
CN110677276A (en) * 2019-09-09 2020-01-10 杭州玖欣物联科技有限公司 System for realizing multi-user hot deployment supported by industrial internet data processing
CN112134860A (en) * 2020-09-09 2020-12-25 深圳中兴网信科技有限公司 Processing method and device of environmental monitoring data, air micro-station and storage medium
CN117076699A (en) * 2023-10-13 2023-11-17 南京奥看信息科技有限公司 Multi-picture acceleration processing method and device and electronic equipment
CN117149897A (en) * 2023-10-31 2023-12-01 成都交大光芒科技股份有限公司 Big data alarm information hierarchical display system and method based on double-buffer technology

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101282342A (en) * 2008-05-30 2008-10-08 腾讯科技(深圳)有限公司 Method and system for fetching network contents
CN104618343A (en) * 2015-01-06 2015-05-13 中国科学院信息工程研究所 Method and system for detecting website threat based on real-time log
CN104944240A (en) * 2015-05-19 2015-09-30 重庆大学 Elevator equipment state monitoring system based on large data technology
WO2016137848A4 (en) * 2015-02-23 2016-12-08 Prophecy Sensors, Llc Internet of things based determination of machine reliability and automated maintainenace, repair and operation (mro) logs

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101282342A (en) * 2008-05-30 2008-10-08 腾讯科技(深圳)有限公司 Method and system for fetching network contents
CN104618343A (en) * 2015-01-06 2015-05-13 中国科学院信息工程研究所 Method and system for detecting website threat based on real-time log
WO2016137848A4 (en) * 2015-02-23 2016-12-08 Prophecy Sensors, Llc Internet of things based determination of machine reliability and automated maintainenace, repair and operation (mro) logs
CN104944240A (en) * 2015-05-19 2015-09-30 重庆大学 Elevator equipment state monitoring system based on large data technology

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107508888A (en) * 2017-08-25 2017-12-22 同方(深圳)云计算技术股份有限公司 A kind of car networking service platform
CN110035096A (en) * 2018-01-12 2019-07-19 中科院微电子研究所昆山分所 A kind of vehicle early warning processing system and early warning system based on Storm
CN108449235A (en) * 2018-05-07 2018-08-24 苏州德姆斯信息技术有限公司 Equipment alarm calculation processing system and processing method
CN108449235B (en) * 2018-05-07 2023-12-12 苏州德姆斯信息技术有限公司 Equipment alarm calculation processing system and processing method
CN110677276A (en) * 2019-09-09 2020-01-10 杭州玖欣物联科技有限公司 System for realizing multi-user hot deployment supported by industrial internet data processing
CN112134860A (en) * 2020-09-09 2020-12-25 深圳中兴网信科技有限公司 Processing method and device of environmental monitoring data, air micro-station and storage medium
CN117076699A (en) * 2023-10-13 2023-11-17 南京奥看信息科技有限公司 Multi-picture acceleration processing method and device and electronic equipment
CN117076699B (en) * 2023-10-13 2023-12-12 南京奥看信息科技有限公司 Multi-picture acceleration processing method and device and electronic equipment
CN117149897A (en) * 2023-10-31 2023-12-01 成都交大光芒科技股份有限公司 Big data alarm information hierarchical display system and method based on double-buffer technology
CN117149897B (en) * 2023-10-31 2024-01-26 成都交大光芒科技股份有限公司 Big data alarm information hierarchical display system and method based on double-buffer technology

Similar Documents

Publication Publication Date Title
CN106951464A (en) Based on the personalized early warning mechanism big data computational methods of storm user orienteds
CN107577805B (en) Business service system for log big data analysis
CN110263921B (en) Method and device for training federated learning model
CN105824744B (en) A kind of real-time logs capturing analysis method based on B2B platform
CN111652380A (en) Method and system for adjusting and optimizing algorithm parameters aiming at machine learning algorithm
CN110362544A (en) Log processing system, log processing method, terminal and storage medium
US11132293B2 (en) Intelligent garbage collector for containers
US20210097431A1 (en) Debugging and profiling of machine learning model training
CN109726004B (en) Data processing method and device
CN110929489A (en) Form generation and form data processing method and system
WO2019001359A1 (en) Data processing method and data processing apparatus
CN106815254A (en) A kind of data processing method and device
US10019505B2 (en) Method and apparatus for creating data cube in streaming manner based on distributed system
CN103701635A (en) Method and device for configuring Hadoop parameters on line
US20230368028A1 (en) Automated machine learning pre-trained model selector
US20200394448A1 (en) Methods for more effectively moderating one or more images and devices thereof
CN111523324A (en) Training method and device for named entity recognition model
WO2019109798A1 (en) Method, device, terminal and storage medium for loading resource
CN112507330A (en) Malicious software detection system based on distributed sandbox
CN103914531A (en) Method and device for processing data
CN108073582B (en) Computing framework selection method and device
US11468365B2 (en) GPU code injection to summarize machine learning training data
CN116383521B (en) Subject word mining method and device, computer equipment and storage medium
US20190340540A1 (en) Adaptive continuous log model learning
CN111130882A (en) Monitoring system and method of network equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20170714