CN110275763A - A kind of alert data processing system and method - Google Patents
A kind of alert data processing system and method Download PDFInfo
- Publication number
- CN110275763A CN110275763A CN201910385528.8A CN201910385528A CN110275763A CN 110275763 A CN110275763 A CN 110275763A CN 201910385528 A CN201910385528 A CN 201910385528A CN 110275763 A CN110275763 A CN 110275763A
- Authority
- CN
- China
- Prior art keywords
- granularity
- service server
- server
- alert data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to a kind of alert data processing system and methods.The system comprises: load-balanced server is configured and the alert data of access is loaded to dispatch server according to load balancing；Dispatch server group is configured and is the alert data granularity of division received according to granularity rule and determines granularity routing, the alert data and its granular information are forwarded to the service server in granularity routing iinformation；And service server group, after being configured in the alert data received and its granular information, the context data of the alert data is loaded according to the granular information, and the alert data is handled according to the context data in real time.Present invention efficiently solves the problems for the utilization of resources unevenness that former data center encounters, alert data amount handled by service server effectively, neatly can be adjusted and be distributed, the scene that contextual dependency is strong between data volume is big, requirement of real-time is high and data is adapted to.
Description
Technical field
The present invention relates to a kind of technical field of data processing, particularly, are related to a kind of alert data processing system and method.
Background technique
The development of computer software and hardware is so that Parallel and distributed computation becomes more and more popular, however computer hardware resource is sharp
It is happened occasionally with unbalanced phenomenon, load-balancing technique is a pass for improving resource utilization ratio and Parallel Computing Performance
Key technology.
Load-balancing technique can be divided into static and two classes of dynamic.If load can determine before runtime and before operation
Load partition has been carried out, then has belonged to static load balancing problem；It is drawn if can only measure at runtime and load and be dynamically determined load
Point, then belong to dynamic load leveling problem.
Alert data is one of numerous machine datas, and there is timeliness to require high, contextual dependency by force and data
Measure the features such as big.Especially in data center's scene, need to receive the alert data information of multi-source, in order to improve alert receipt
The resource utilization of server often uses dynamic load equilibrium technology, builds the load-balanced server of software or hardware
To solve the problems, such as this class.
But the load-balancing algorithm used in conventional load balancing technique, such as polling algorithm, random algorithm, minimum load
Priority algorithm and longest processing time priority algorithm etc., not can solve alert data timeliness is high, contextual dependency is strong etc.
Problem；On in terms of the resource-sharing, the data of conventional load balancing technique are synchronous and shared often rely on data center, Ge Gejie
Point needs to obtain required context data by data center, in the background of magnanimity alert data when handling alert data
Under, data center has become where the bottleneck of system.
Summary of the invention
For the technical problems in the prior art, the invention proposes alert data processing system and method, to
Solve the bottleneck problem when handling magnanimity alert data.
In order to solve the above-mentioned technical problems, the present invention provides a kind of alert data processing system, including: load is equal
Weigh server, is configured and the alert data of access is loaded to dispatch server according to load balancing；Dispatch server group,
Be configured and be the alert data granularity of division received according to granularity rule and determine granularity routing, by the alert data and its
Granular information is forwarded to the service server in granularity routing iinformation；And service server group, it is configured in the report received
After alert data and its granular information, the context data of the alert data is loaded according to the granular information, on described
Context data handles the alert data in real time.
Wherein, the load-balanced server includes two servers using two-node cluster hot backup mechanism, is configured respectively
Active and standby load-balanced server.
Wherein, the dispatch server group includes multiple dispatch servers, wherein the dispatch server includes: granularity
Division module is configured for determining the alert data according to the raw information of the granularity rule and the alert data
Granularity；Granularity routing module, is configured for determining the granularity routing of the corresponding granularity, granularity routing iinformation include granularity and
Corresponding service server；And data forwarding module, alert data will be received by, which being configured for, is forwarded to granularity routing
Service server in information.
Wherein, the granularity routing module further include: the first query unit is configured for according to current granularity, inquiry
Granularity routing iinformation in memory obtains service server corresponding with current granularity；And computing unit, it is configured in memory
When not getting service server corresponding with current granularity, new road is calculated according to the load resource of service server
By to obtain service server corresponding with current granularity.
Wherein, the computing unit is configured to complete following steps: calculating the load point of each service server
Number；It is clustered according to the numerical values recited of load score, obtains multiple cluster values；Minimum load is carried out to the multiple cluster value
It is preferential to calculate, determine multiple candidate Traffic servers；And according to random algorithm, from the multiple candidate Traffic server really
A fixed service server.
Wherein, the computing unit is configured, and the load score of each service server: root is calculated by following steps
The maximum processing capability value of the service server is calculated according to the basic resource information of service server；According to the business service
The operation resource information of device calculates the service server current ability free time ratio；With the maximum for calculating the service server
The product of processing ability value and current ability free time ratio obtains the load score of the service server.
Wherein, the computing unit is configured, and the operation resource information of service server is periodically obtained, according to newest
Operation resource information calculate the load score of the service server current ability free time ratio and the service server.
Wherein, at least one dispatch server in the dispatch server group includes: granularity routing adjustment module, through matching
The load resource for calculating each service server in service server group is set, granularity road is adjusted according to the load resource
By.
Wherein, the service server includes: context data loading module, for loading institute according to the granular information
State the context data of alert data；With alert data processing module, for according to the context number for loading the alert data
According to the processing alert data.
Wherein, the context data loading module includes: the second query unit, for being inquired according to the granular information
Context data in service server storage inside module；First loading unit, for having in the storage inside module
When context data matched with the granular information that receives, the context data is loaded from storage inside module；
Second loading unit, the matched context number of the granular information for not having and receiving in the storage inside module
According to when, the matched context data of the granular information that obtains and receive from data-base cluster.
Wherein, the service server further include: monitoring resource module, for monitoring the business in the service server
Whether resource matches with current operation resource；Request module is changed with granularity, in currently operation resource and service resources
When not matching that, granularity change request is sent to dispatch server.
Wherein, the service server further include: storage inside module, for storing alert data corresponding with granularity
Context data.
The alert data processing system further include: data synchronized clusters server is configured in synchronization system
The data mode and maintenance granularity data of each server processes；And data-base cluster, for being stored in storage alert data
Context data.
The present invention also provides a kind of alert data processing method, including: load-balanced server by the report of access
Data payload is warned to dispatch server；The dispatch server according to granularity rule be the alert data granularity of division that receives simultaneously
It determines that granularity routes, and the alert data and its granular information is forwarded to the service server in granularity routing iinformation；With
And after the service server receives the alert data and its granular information, the alarm is loaded according to the granular information
The context data of data, and the alert data is handled according to the context data in real time.
Wherein, when the alert data of access is loaded to dispatch server by the load-balanced server, according to configuration
Load balancing, or random manner is used, the alert data of access is loaded to a scheduling in dispatch server group
Server.
Wherein, the load-balanced server includes two servers using two-node cluster hot backup mechanism, when main load balancing
When server fail, the alert data of access is loaded into dispatch server by standby load-balanced server.
Wherein, the dispatch server is the alert data granularity of division received according to granularity rule and determines granularity road
By the step of include: granularity of query rule, according to granularity rule and the raw information of the alert data, determine the report
The granularity of alert data；The granularity of query routing iinformation in the memory of the dispatch server obtains grain corresponding with the granularity
Degree routing, the granularity routing includes service server corresponding with granularity；And do not have in the memory of the dispatch server
When having the granularity routing of the corresponding granularity, new granularity is calculated according to the load resource of service server and is routed, new granularity
Routing includes service server corresponding with granularity.
Wherein, when determining the granularity of the alert data, if not corresponding with the raw information of the alert data
Granularity, create new granularity；And new granularity is calculated according to the load resource of service server and is routed, and records the grain
The incidence relation of degree and the service server in the routing of new granularity.
Wherein, before the service server that the alert data is forwarded in the granularity routing iinformation, institute is determined
State whether service server can be used；When the service server is unavailable, calculated according to the load resource of service server
It is routed to new granularity；The alert data is forwarded to the service server in new granularity routing iinformation.
Wherein, by the service server in former granularity routing iinformation granular information and corresponding alert data up and down
Literary data are synchronized to the service server in new granularity routing.
Wherein, the raw information of the alert data include device IP, equipment region, machinery equipment characteristic information or
One or more feature fields defined in business information.
Wherein, the granularity rule include: according in business information feature field, device IP and/or equipment region it is true
Determine granularity；Or granularity is determined by the granularity rule that hot can be updated and dynamically come into force that regulation engine is realized.
Wherein, the step of routing of new granularity being calculated according to the load resource of service server includes: to calculate each
The load score of service server；It is clustered according to the numerical values recited of load score, obtains multiple cluster values；To the multiple
Cluster value carries out minimum load and preferentially calculates, and determines multiple candidate Traffic servers；And according to random algorithm, from the multiple
A service server is determined in candidate Traffic server.
Wherein, the step of calculating the load score of each service server includes: the basis money according to service server
Source information calculates the maximum processing capability value of the service server；It is calculated according to the operation resource information of the service server
The service server current ability free time ratio；And maximum processing capability value and the current energy of the calculating service server
The product of power free time ratio obtains the load score of the service server.
The alert data processing method may further comprise: the operation resource letter for periodically obtaining service server
Breath calculates the service server current ability free time ratio and the service server according to newest operation resource information
Load score.
The alert data processing method further comprises: periodically granularity of query and service server incidence relation
Whether change.
Wherein, the step of service server loads the context data of the alert data according to the granular information
Include:
Granular information in the storage inside module of inquiry business server has and receives in the storage inside module
When the matched context data of the granular information arrived, the context data is loaded from storage inside module；And
When the matched context data of the granular information for not having and receiving in the storage inside module, from number
According to the matched context data of the granular information for loading and receiving in the cluster of library.
The alert data processing method further include: the service server when handling the alert data in real time,
When currently operation resource and service resources do not match that, the granularity of the service server is changed or split.
Wherein, the change or the step of splitting the granularity of the service server include: the service server to tune
It spends server and sends granularity change request；After receiving the granularity change request, the dispatch server is according to the business
The basic resource data and operation data of server determine change or split the granularity of the service server；Changing the industry
When the granularity of business server, the current business server in the granularity routing of the granularity is become into new operational processor；It is tearing open
When dividing the granularity of the service server, according to the split rule, the granularity is split as multiple seed degree, and respectively multiple
Corresponding service server is calculated in seed degree, to determine corresponding granularity routing for multiple seed degree.
Wherein, when the current business server in the granularity routing by the granularity is changed to new operational processor,
Further include: the context data of the alert data in current business server is synchronized to the new operational processor；From working as
The context data of the alert data of the corresponding granularity is deleted in preceding service server；With by the granularity and business service
Current business processor in the incidence relation of device is changed to new operational processor.
Wherein, when splitting the granularity of the service server, further includes: deleted from current business server with it is described
The context data of the corresponding alert data of granularity；The incidence relation of the granularity and service server is deleted, and records seed
It spends and the incidence relation of the service server in the routing of corresponding granularity.
Present invention efficiently solves the utilization of resources unevenness problems that former data center encounters, and can effectively, neatly adjust
Alert data amount handled by whole and distribution service server；Between adaptation data volume is big, requirement of real-time is high and data up and down
The strong scene of literary dependence；Set expandability is good, availability is high, no Single Point of Faliure hidden danger, runs within safeguards system 7*24 hours.
Detailed description of the invention
In the following, the preferred embodiment of the present invention will be described in more detail in conjunction with attached drawing, in which:
Fig. 1 is the block diagram of alarm processing method according to an embodiment of the invention and system；
Fig. 2 is the functional block diagram of dispatch server according to an embodiment of the invention；
Fig. 3 is the functional block diagram of service server according to an embodiment of the invention；
Fig. 4 is the flow chart of alert data processing method according to an embodiment of the invention；
Fig. 5 is the process flow diagram according to an embodiment of the invention for the alert data granularity of division received
Fig. 6 is that new routing is calculated in the load resource according to an embodiment of the invention according to service server
Process flow diagram；And
Fig. 7 is the specific process flow diagram of change or fractionation granularity request according to an embodiment of the invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
In the following detailed description, the specific embodiment for being used to illustrate the application as the application a part may refer to
Each Figure of description.In the accompanying drawings, similar appended drawing reference describes substantially similar component in different drawings.This Shen
Each specific embodiment please has carried out description detailed enough following, so that having the general of ability domain-dependent knowledge and technology
Logical technical staff can implement the technical solution of the application.It should be appreciated that can also be using other embodiments or to the application
Embodiment carry out structure, logic or electrical property change.
The present invention provides a kind of alert data processing method and system based on load balancing and adaptive granularity division,
Wherein, the schematic diagram of the system is as shown in Figure 1, the alert data processing system includes load-balanced server group 1, scheduling
Server group 2 and service server group 3.Wherein, the load-balanced server group 1 is configured for the alert data that will be accessed
Load to dispatch server.As a specific embodiment, the load equalization server group 1 is two and uses two-node cluster hot backup mechanism
Active and standby server FC.By configuring two-node cluster hot backup mechanism, make two-server FC respectively as active and standby server according to negative
Carry balance policy or randomly, dynamically by HTTP the or HTTPS assignment of traffic of entrance into different dispatch server DC, because
And improve the application availability and responding ability of system entirety.
The dispatch server group 2 includes multiple dispatch server DC, and dispatch server DC is configured, according to granularity rule
For the alert data granularity of division received, and the alert data and its granular information are forwarded to service server BC.
Service server group 3 include multiple service server BC, service server BC be configured receive alert data and its
Granular information loads the context data of the alert data according to the granular information, real-time according to the context data
Handle the alert data.
As shown in Fig. 2, being the functional block diagram of dispatch server.The dispatch server DC include: granularity division module 21,
Granularity routing module 22 and data forwarding module 23.Wherein, granularity division module 21 is configured for according to the granularity rule
And the raw information of the alert data, determine the granularity of the alert data.In one embodiment, in the alert data
Raw information can be one kind or more defined in device IP, equipment region, machinery equipment characteristic information or business information
Kind feature field.The IP address for sending the alarm front end of the alert data is described in the device IP；It is wrapped in business information
Multiple feature fields are included, such as represent the field of returning the vehicle to the garage and knock off of the type of service of the alert data, according to the field, can learn this
Alert data is from belonging to the business of returning the vehicle to the garage and knock off.Equipment region geographic area as belonging to the alarm front end for sending alert data.This
Invention is provided with granularity division standard in granularity rule, such as is divided according to equipment region: granularity can be south China, North China etc.
Geographic area；It is also possible to the administrative region of different stage, such as Henan Province, Hebei province, Beijing, Shanghai City, Xi'an；May be used also
With according to delineation of activities, for example, granularity can be divided according to the feature field of business information.It returns the vehicle to the garage and knock off for example, granularity can be
Business, guarantor sell business, financial business.It is also possible to the combination of above two standard, such as business of returning the vehicle to the garage and knock off (Beijing), business of returning the vehicle to the garage and knock off
(Shanghai), guarantor sell business (Beijing), protect and sell business (Shanghai).Region can be determined by inquiring the table of comparisons by IP address, pass through industry
Field in business information can determine corresponding business.
The granularity rule is storable on data synchronized clusters server Z, and dispatch server DC can be synchronous from data
Dynamic acquisition on cluster server Z.Furthermore it is also possible to which increasingly complex rule is arranged, regular setting and division can pass through rule
Then engine implementation, regulation engine can carry out flexible heat and update, and dynamic comes into force, and for details, reference can be made to drools, siddhi etc.
Technology.
The granularity rule is also provided with the method for creating new granularity, for example, if current granularity rule only gives
The granularity of Beijing and Shanghai City, when receiving from the alert data of the alarm front end of other regions, by alarm front end
Region be created as a granularity and be recorded in data synchronized clusters server Z.
Granularity routing module 22 is configured for determining granularity routing corresponding with the granularity, that is, determines described in processing
The service server BC of the alert data of granularity.Wherein, the granularity routing is including granularity and for handling the report of the granularity
Alert data service server BC.The granularity routing, which is shown, is assigned to specific business service for the alert data of the granularity
Device BC.Wherein, in one embodiment, the alert data of a granularity is assigned to only a service server BC processing.Due to
Real time data amount is very big, if using storage disk or using other data sharing technologies, requires the bigger cost of consumption (more
More servers), the present invention uses an equipment first, and a service server BC as the aforementioned carries out data sharing and data
Interaction splits rule according to granularity, by grain if a service server BC handles endless or service server BC failure
Degree split, by after fractionation part granularity and corresponding context data be synchronized on other service servers BC.Granularity
Routing module 22 judges whether that having corresponding granularity routes according to fixed granularity: corresponding with the granularity having existed
Granularity routing in the case where, by inquiry can obtain, if currently without granularity corresponding with the granularity route,
The service server BC of the alert data of a processing granularity is determined by calculating.Specifically, granularity routing module 22
Including the first query unit 221 and computing unit 222.Wherein, the first query unit 221 is configured for according to current granularity,
Granularity data in audit memory or data synchronized clusters server Z obtains service server BC corresponding with current granularity, from
And the corresponding granularity routing of the granularity is obtained, and send it to data forwarding module 23.Wherein, the first query unit 221
The memory of preferential query scheduling server DC, when not corresponding to the granularity routing of the granularity in the memory of dispatch server,
Data synchronized clusters server Z is inquired again.Computing unit 222 obtains service server from data synchronized clusters server Z
Service server corresponding with current granularity is calculated according to the load resource of service server in load resource data, thus
Granularity routing corresponding with the granularity has been obtained, and has sent it to data forwarding module 23.Wherein, the process of calculating will be rear
It is described in detail in continuous method.It should be noted that computing unit 222 can in order to improve the real-time calculating speed of granularity routing
To precompute the parameter value for obtaining new granularity routing, since the parameter value is the load resource number by service server
According to what is obtained, thus, computing unit 222 periodically obtains the load resource of service server, and according to service server
Load resource obtains corresponding parameter value, the use when needing to calculate granularity routing, to ensure that currently available parameter value
Meet the present load resource situation of service server.
Since the load resource situation of service server can change, the data volume of the alert data of a granularity can be with
Can become very big due to various reasons, this just will appear the inadequate resource of service server with allow elsewhere should before granularity
Alert data, at this time, it may be necessary to carry out the adjustment of granularity routing.Thus, in one embodiment, one in dispatch server group
It include granularity routing adjustment module 24 in dispatch server DC1, for calculating each service server in service server group
Load resource, according to the load resource adjust granularity routing.
After data forwarding module 23 receives granularity routing, alert data is forwarded to the industry in the granularity routing iinformation
Be engaged in server B C.
Due to there is multiple dispatch servers in system, thus, different function can be distributed dispatch server.Have at one
It in body embodiment, in N platform dispatch server, is used to determine each service server BC pressure condition for one therein, adjusts
Whole grain degree forwarding strategy, the i.e. function of completion granularity routing adjustment module 24, hereon referred to as first kind dispatch server.By other
N-1 platform dispatch server is completed according to granularity routing forwarding alert data, i.e. completion granularity division module 21, granularity routing module
22 and data forwarding module 23, hereon referred to as the second class dispatch server.New granularity routing is needed in the second class dispatch server
When, it is communicated with first kind dispatch server, is calculated by first kind dispatch server and newly route and be supplied to first kind scheduling clothes
Business device.Since the second class dispatch server is absorbed in the forwarding of alert data, first kind dispatch server is absorbed in granularity routing
With the maintenance of rule, thus the treatment effeciency of each dispatch server is improved.
As shown in figure 3, being the functional block diagram of service server.The service server BC include data reporting module 30,
Context data loading module 31 and alert data processing module 32.Wherein, data reporting module 30 is used for sync database Z
Registration resource information, and periodically operation resource information and service resources information are submitted to sync database Z.For the ease of adjusting
The load resource that server obtains service server BC is spent, granularity routing is reasonably created, service server BC after actuation, is counted
Current resource situation is registered on data synchronized clusters server Z according to reporting module 30, the resource and operating condition are divided into:
Basic resource information: including but is not limited to CPU model, memory size, disk revolution and network bandwidth.
It runs resource information: including but is not limited to CPU usage, memory usage, network interface card flow information, disk I/O use
Rate.
Service resources information: include but is not limited to it is current or the alarm statistical information that will handle and store.
Basic resource information provides once in registration, mentions with running resource information and service resources property information cycle
It is sent on data synchronized clusters server Z.In order to show whether service server BC can be used, be set, such as the time-to-live of data
The resource and operating condition information that fruit service server BC is provided are (such as: not carrying out more than 15 seconds more than the time-to-live being arranged
Update) indicate that the service server BC for not providing information is unavailable.
The context data loading module 31 is used to load the context of the alert data according to the granular information
Data.Specifically, the context data loading module 31 includes the second query unit 311, the first loading unit 312 and second
Loading unit 313.Wherein, it receives after the alert data of dispatch server DC and granular information, the second query unit
311 according to the granular information in current granularity inquiry business server B C memory；Judge whether to have and receive in memory
The matched context data of granular information.If so, then sending load instruction, the first load to the first loading unit 312
Unit 312 loads the context data from memory.If the grain for not having and receiving in service server BC memory
The context data of information matches is spent, then sends load instruction to the second loading unit 313, the second loading unit 313 is from data
The context data that the granularity for obtaining and receiving in the cluster DB of library matches.Service server BC in the present invention is big memory
Server can store the context data of a large amount of alert data, ensure the alert data of a granularity at one with this
It is handled on service server BC.Make external storage when thus avoiding using external storage equipment storage context data
The problem of as bottleneck.
Alert data processing module 32 handles the alert data according to the context data for loading the alert data.Its
In, the alarm context data refers to the multistage alert data loaded on a service server BC.Service server
The multistage alert data can be judged as a significant alarm according to rule by BC.For example, working as the CPU of some equipment
When utilization rate is greater than 90%, alert data is sent from alarm front end to this system, is at this time the beginning of alarm, if be received again by
Identical alarm, it is believed that alarm always exists.If until receiving alarm recovery, (data are small in equipment CPU usage
The message issued when 80%), it can determine that the equipment is recovered normal.From receive first alert data to receive report
The alert a plurality of alarm restored in this period, is a series of alarms, by the processing of service server BC, by this several sections alarm numbers
According to merging into a significant alarm: including time started, end time, sample rate etc..And it is corresponding according to warning message
Triggering and elimination rule, the triggering or elimination alarmed.For alert data processing module 32 in after treatment, delay batch is same
Step alarm context data is into data-base cluster DB, to reduce the processing pressure of data-base cluster DB.
In addition, the service server BC further includes monitoring resource module 33 and granularity change request module 34.The industry
Whether business server B C is matched by the service resources that monitoring resource module 33 monitors oneself with current operation resource, i.e., currently
Operation resource whether handle the business currently distributed enough.If determining that current operation resource is tight by monitoring resource module 33
And service resources when not matching that, request module 34 is changed from granularity and sends granularity change request to dispatch server DC1,
To reduce the pressure of the service server BC.
After dispatch server DC1 receives the granularity change request, granularity routing adjustment module 24 is according to business service
Current business resource, basic resource and the operation resource of device BC judges that carrying out granularity change or granularity splits.If necessary to more
Change granularity, then changes the routing iinformation of current granularity, that is, determine the alarm for handling the granularity by other service servers BC
Data；It is split if it is granularity, i.e., according to the split rule, which is split as multiple seed degree, original grain is deleted after fractionation
The relevant information of degree, and handled for the seed degree after splitting as newly-increased granularity.To make the service server BC of resource anxiety
The granularity alert process work undertaken is handled by other service servers BC.
It in the present invention, further include the pipe of data synchronized clusters server Z and data-base cluster DB for cluster in system
Reason, the storage etc. of synchronization, data including process.Handle the dotted line of server connection relationship that this layer is dealt with relationship as shown in figure 1
It is shown.Dispatch server DC and service server BC is connected with data synchronized clusters server Z and data-base cluster DB respectively
Communication, by the data mode of each process of data synchronized clusters server Z synchronized clusters server.For example, working as dispatch service
When device DC creates a new granularity, by the new granularity and its granularity route record into data synchronized clusters server Z.Example again
Such as, service server BC submits operation resource to sync database Z to sync database Z registration resource information, and periodically
Information and service resources information.It is stored with granularity data in synchronized clusters server Z, such as the granularity and its routing, granularity of creation
With the incidence relation of service server and granularity rule etc..Data-base cluster DB is used to be stored in the context number of storage alert data
According to.In a specific embodiment, the data synchronized clusters server Z is by based on distributed application program
Zookeeper is realized, specifically refers to the relevant technologies of Zookeeper to realize.
As shown in figure 4, being the flow chart of alert data processing method provided by the invention, specifically include:
The alert data of access is loaded to dispatch server by step S1, load-balanced server.Wherein, load balancing takes
It is engaged in being configured with load balancing in device, Http the or Https flow load for that will access is suitable, different into system
Dispatch server, avoid flow while being mapped on a dispatch server and causing the load excessive of dispatch server
The problem of.
Step S2, the dispatch server are the alert data granularity of division received according to granularity rule and determine granularity
The alert data and its granular information are forwarded to the service server in granularity routing by routing.Wherein, the report to receive
The step of warning data granularity of division and forwarding is specifically as shown in Figure 5:
Step S21, granularity of query rule.
Step S22 judges whether to determine described according to the granularity rule and the raw information of the alert data
The granularity of alert data.If granularity not corresponding with the alert data in not current granularity rule, in step
S28 creates new granularity.For example, only from the granularity of the alert data in Beijing and Shanghai City in current granularity rule, but
It is the alert data that is currently received from Shandong Province, does not have the granularity of corresponding Shandong Province in granularity rule at this time, thus, this
When need to create a new granularity.If the granularity of the alert data can be determined according to the granularity rule, in step
S23 determines the granularity of the corresponding alert data.For example, according in the alert data " returning the vehicle to the garage and knock off " field and device IP can be with
Determine that the granularity is "-Beijing of returning the vehicle to the garage and knock off ".Wherein, the granularity rule can store on dispatch server, or be stored in
In synchronized clusters server Z.
Step S24, granularity of query routing iinformation.Dispatch server handle alert data forwarding during, so
Many granularity routing iinformations are had recorded in its memory.After granularity has been determined, the granularity routing iinformation in its memory is inquired, from
And to determine whether having corresponding granularity routing.
Step S25 judges whether there is granularity corresponding with granularity routing, if had corresponding with the granularity
Granularity routing, then go to step S26；If not having granularity corresponding with the granularity to route, in step S29, according to industry
New granularity routing is calculated in the load resource of business server, then executes step S27.
Step S26, judges whether the service server in granularity routing can be used.Specifically judgment method is implemented at one
In example, the resource data of the service server BC in data synchronized clusters server Z can be checked by dispatch server DC more
New situation, if the resource data of the service server BC in data synchronized clusters server Z is in time-to-live section without more
Newly, then it is assumed that the service server BC is unavailable.In another embodiment, data synchronized clusters server Z monitoring business
The update status of the resource data of server, and safeguard the whether available state of all service server BC.Collect when data are synchronous
When the resource data that group server Z monitors some service server BC does not update in time-to-live section, synchronized by data
Cluster server Z mark the service server state be it is unavailable, if there is updating, then label is available.Dispatch server
DC can know the available mode for the service server BC currently to be inquired by inquiring the state.By above-mentioned processing, if
The service server BC is unavailable, return step S29, and new granularity road is calculated according to the load resource of service server
By；If the service server BC can be used, step S27 is turned to.
Step S27 by service server BC that the alert data is forwarded in granularity routing iinformation and terminates.
Wherein, the new granularity of creation in step S28 can create new granularity according to creation rule, wherein the wound
Rule is built for example, being a granularity by the corresponding areal setting of the device IP for the alert data being currently received.For example, current
Alert data is from the alarm front end of Shandong Province, then it is Shandong Province that the IP address in alert data is corresponding, at this point, by Shandong
Save the granularity new as one.After creating a new granularity, it is recorded in data synchronized clusters server Z.
In step S29, the step of new granularity routes is calculated such as Fig. 6 institute according to the load resource of service server
Show, specifically include:
Step S291 calculates the load score of each service server.Wherein, it specifically includes: according to service server
The basic resource information of BC calculates the maximum processing capability value Max of the service server；According to the service server BC's
It runs resource information and calculates the service server current ability free time ratio Percent；With according to formula m=max*
Percent calculates the maximum processing capability value of the service server and the product of current ability free time ratio, obtains the industry
The load score m of business server.
Step S292 is clustered according to the numerical values recited of load score m, obtains multiple cluster values；
Step S293 carries out minimum load to the multiple cluster value and preferentially calculates, determines multiple candidate Traffic servers；
And
Step S294 determines a service server according to random algorithm from the multiple candidate Traffic server.
So that it is determined that corresponding with granularity service server is to get having arrived granularity routing.
It after having obtained granularity routing, is stored in the memory of dispatch server, and it is synchronous that data are recorded simultaneously
In cluster server Z.
Step S3 after the service server receives the alert data and its granular information, believes according to the granularity
Breath loads the context data of the alert data, and handles the alert data in real time according to the context data.
Specifically, whether the service server is inquired first in its memory alert data corresponding with the granularity
Context data, if so, then loading the context data from memory.If the institute for not having and receiving in the memory
State the matched context data of granular information, from the granular information for loading and receiving in data-base cluster DB it is matched on
Context data.Simultaneously to the incidence relation of the data synchronized clusters server Z granularity updated and service server.
The service server monitors its current operation resource and business during handling the alert data in real time
Whether resource matches, i.e., whether has enough resources to handle the alert data.When the service server is without foot
Enough resources come when handling alert data thereon, need change or split granularity.Specific process flow is as shown in Figure 7:
Step S301, the service server BC send granularity change request to dispatch server DC1.In the present invention,
It include a first kind dispatch server DC1 in dispatch server group, for safeguarding granularity routing and granularity rule.When business takes
When the current operation inadequate resource of business device BC is with processing business resource, granularity change request is sent to dispatch server DC1.
Step S302, after receiving the granularity change request, the dispatch server DC1 is according to the service server
Basic resource data and operation data determine change or split the granularity of the service server.
Step S3031 calculates a determining new industry by carrying out load when changing the granularity of the service server
Business server.
The granularity is corresponded to the current business server in granularity routing and is changed to new business processing by step S3032
Device.
Step S3033, by the context data of the corresponding alert data of the granularity from current business server sync to
The new service server.
According to the split rule, the granularity is split as when splitting the granularity of the service server by step S3041
Multiple seed degree.For example, the granularity of province to be split into the granularity in multiple cities, as previously mentioned, splitting the granularity of Shandong Province by helping
The multiple seed degree in Nanshi, Qingdao City, Yantai City ... etc..
Corresponding service server is calculated in step S3042, respectively multiple seed degree, to determine for multiple seed degree
Corresponding seed degree routing.
Step S3043 is routed according to the seed degree, by original alert data to the industry being synchronized in the routing of seed degree
Business server.
Step S305 modifies the granularity data in each server.Specifically: when changing granularity, taken from current business
The granular information and context data are deleted in business device, and modify corresponding granularity road in data synchronized clusters server Z
By the incidence relation of information, granularity and service server.When splitting granularity, from current service server described in deletion
Granular information (granularity and granularity routing) and context data, it is synchronous to be recorded data for seed degree and its seed degree routing iinformation
Cluster server Z records the incidence relation of the seed degree and service server.
The present invention implements load balancing in load-balanced server and dispatch server respectively, can efficiently solve
The utilization of resources unevenness problem that former data center encounters, reduces the expense of former data center server.
The present invention uses adaptive granularity division mode and division rule, can effectively, neatly adjust and distribution business
Alert data amount handled by business server.
The present invention uses the service server of big memory, efficiently solves and needs in magnanimity alert data treatment process
When context data and after treatment need to store context data, bottleneck and data center's pressure that external storage occurs
Big problem.
Each server division of labor in the present invention is clear, thus favorable expandability, each role can dynamically be added
And reduction.
System availability provided by the invention is high, no Single Point of Faliure hidden danger, runs within safeguards system 7*24 hours.
Above-described embodiment is used for illustrative purposes only, and is not limitation of the present invention, in relation to the general of technical field
Logical technical staff can also make a variety of changes and modification without departing from the present invention, therefore, all equivalent
Technical solution also should belong to scope disclosed by the invention.
Claims (31)
1. a kind of alert data processing system, including:
Load-balanced server is configured and the alert data of access is loaded to dispatch server according to load balancing；
Dispatch server group is configured and is the alert data granularity of division received according to granularity rule and determines that granularity routes,
The alert data and its granular information are forwarded to the service server in granularity routing iinformation；And
Service server group after being configured in the alert data received and its granular information, is loaded according to the granular information
The context data of the alert data handles the alert data according to the context data in real time.
2. alert data processing system according to claim 1, wherein the load-balanced server includes two uses
The server of two-node cluster hot backup mechanism is configured respectively active and standby load-balanced server.
3. alert data processing system according to claim 1, wherein the dispatch server group includes multiple scheduling clothes
It is engaged in device, wherein the dispatch server includes:
Granularity division module is configured for determining the report according to the raw information of the granularity rule and the alert data
The granularity of alert data；
Granularity routing module, is configured for determining the granularity routing of the corresponding granularity, granularity routing iinformation include granularity and
Corresponding service server；With
Data forwarding module is configured for that the service server that alert data is forwarded in granularity routing iinformation will be received.
4. alert data processing system according to claim 3, wherein the granularity routing module further include:
First query unit is configured for according to current granularity, and the granularity routing iinformation in audit memory obtains and current grain
Spend corresponding service server；With
Computing unit is configured when not getting service server corresponding with current granularity in memory, is taken according to business
New routing is calculated in the load resource of business device, to obtain service server corresponding with current granularity.
5. alert data processing system according to claim 4, wherein the computing unit is configured to complete following
Step:
Calculate the load score of each service server；
It is clustered according to the numerical values recited of load score, obtains multiple cluster values；
Minimum load is carried out to the multiple cluster value preferentially to calculate, and determines multiple candidate Traffic servers；And
According to random algorithm, a service server is determined from the multiple candidate Traffic server.
6. alert data processing system according to claim 5, wherein the computing unit is configured, and passes through following step
Suddenly the load score of each service server is calculated:
The maximum processing capability value of the service server is calculated according to the basic resource information of service server；
The service server current ability free time ratio is calculated according to the operation resource information of the service server；With
The maximum processing capability value of the service server and the product of current ability free time ratio are calculated, the business clothes are obtained
The load score of business device.
7. alert data processing system according to claim 6, wherein the computing unit is configured, and is periodically obtained
It is empty to calculate the service server current ability according to newest operation resource information for the operation resource information for taking service server
The load score of not busy ratio and the service server.
8. alert data processing system according to claim 4, wherein at least one of described dispatch server group is adjusted
Spending server includes:
Granularity routing adjustment module, is configured for calculating the load resource of each service server in service server group,
Granularity routing is adjusted according to the load resource.
9. alert data processing system according to claim 1, wherein the service server includes:
Context data loading module, for loading the context data of the alert data according to the granular information；With
Alert data processing module, for handling the alert data according to the context data for loading the alert data.
10. alert data processing system according to claim 9, wherein the context data loading module includes:
Second query unit, for according to the context number in the granular information inquiry business server internal memory module
According to；
First loading unit, the matched context of the granular information for having and receiving in the storage inside module
When data, the context data is loaded from storage inside module；
Second loading unit, the granular information for not having and receiving in the storage inside module are matched up and down
When literary data, the matched context data of the granular information that obtains and receive from data-base cluster.
11. alert data processing system according to claim 9, wherein the service server further include:
Monitoring resource module, for monitoring whether the service resources in the service server match with current operation resource；
With
Granularity changes request module, for being sent to dispatch server when currently operation resource and service resources do not match that
Granularity change request.
12. alert data processing system according to claim 9, wherein the service server further include:
Storage inside module, for storing the context data of alert data corresponding with granularity.
13. alert data processing system according to claim 1, wherein further include:
Data synchronized clusters server is configured for the data mode of each server processes and maintenance granularity in synchronization system
Data；And
Data-base cluster, for being stored in the context data of storage alert data.
14. a kind of alert data processing method, including:
The alert data of access is loaded to dispatch server by load-balanced server；
The dispatch server is the alert data granularity of division received according to granularity rule and determines granularity routing, and by institute
It states alert data and its granular information is forwarded to service server in granularity routing iinformation；And
After the service server receives the alert data and its granular information, the report is loaded according to the granular information
The context data of alert data, and the alert data is handled according to the context data in real time.
15. alert data processing method according to claim 14, wherein the load-balanced server is by the report of access
When alert data payload arrives dispatch server, according to the load balancing of configuration, or using random manner, by the alarm of access
Data payload is to a dispatch server in dispatch server group.
16. alert data processing method according to claim 14 or 15, wherein the load-balanced server includes two
A server using two-node cluster hot backup mechanism will by standby load-balanced server when main load-balanced server breaks down
The alert data of access loads to dispatch server.
17. alert data processing method according to claim 14, wherein the dispatch server is according to granularity rule
The alert data granularity of division that receives and the step of determining granularity routing includes:
Granularity of query rule determines the alert data according to the granularity rule and the raw information of the alert data
Granularity；
The granularity of query routing iinformation in the memory of the dispatch server obtains granularity routing corresponding with the granularity, institute
Stating granularity routing includes service server corresponding with granularity；And
When not corresponding to the granularity routing of the granularity in the memory of the dispatch server, according to the load of service server
Resource Calculation obtains new granularity routing, and new granularity routing includes service server corresponding with granularity.
18. alert data processing method according to claim 17, wherein when determining the granularity of the alert data,
If granularity not corresponding with the raw information of the alert data, creates new granularity；And
New granularity routing is calculated according to the load resource of service server, and records in the granularity and the routing of new granularity
The incidence relation of service server.
19. alert data processing method according to claim 17, wherein the alert data is being forwarded to the grain
It spends before the service server in routing iinformation, determines whether the service server can be used；The service server can not
Used time is calculated new granularity according to the load resource of service server and routes；The alert data is forwarded to new granularity road
By the service server in information.
20. alert data processing method according to claim 19, wherein by the business service in former granularity routing iinformation
The context data of granular information and corresponding alert data in device is synchronized to the service server in new granularity routing.
21. alert data processing method described in 7 or 18 according to claim 1, wherein the original packet of the alert data
Include device IP, equipment region, one or more feature fields defined in machinery equipment characteristic information or business information.
22. alert data processing method according to claim 21, wherein the granularity rule includes: to be believed according to business
Feature field, device IP and/or equipment region in breath determine granularity；Or by regulation engine realize can heat update and dynamic
The granularity rule to come into force determines granularity.
23. alert data processing method described in 7 or 18 or 19 according to claim 1, wherein according to the load of service server
Resource Calculation obtain new granularity routing the step of include:
Calculate the load score of each service server；
It is clustered according to the numerical values recited of load score, obtains multiple cluster values；
Minimum load is carried out to the multiple cluster value preferentially to calculate, and determines multiple candidate Traffic servers；And
According to random algorithm, a service server is determined from the multiple candidate Traffic server.
24. alert data processing method according to claim 23, wherein calculate the load point of each service server
Several steps include:
The maximum processing capability value of the service server is calculated according to the basic resource information of service server；
The service server current ability free time ratio is calculated according to the operation resource information of the service server；And
The maximum processing capability value of the service server and the product of current ability free time ratio are calculated, the business clothes are obtained
The load score of business device.
25. alert data processing method according to claim 24, wherein may further comprise: and periodically obtain industry
The operation resource information of business server calculates the service server current ability idle ratio according to newest operation resource information
The load score of example and the service server.
26. alert data processing method described in 7 or 18 or 19 according to claim 1, wherein further comprise: periodically
Whether granularity of query changes with service server incidence relation.
27. alert data processing method according to claim 14, wherein the service server is believed according to the granularity
Breath loads the step of context data of the alert data and includes:
Granular information in the storage inside module of inquiry business server has and receives in the storage inside module
When the matched context data of the granular information, the context data is loaded from storage inside module；And
When the matched context data of the granular information for not having and receiving in the storage inside module, from database
The matched context data of the granular information for loading and receiving in cluster.
28. alert data processing method described in 4 or 27 according to claim 1, wherein further include:
The service server is not matched that when handling the alert data in real time in currently operation resource and service resources
When, change or split the granularity of the service server.
29. alert data processing method according to claim 28, wherein the change splits the service server
Granularity the step of include:
The service server sends granularity change request to dispatch server；
After receiving granularity change request, the dispatch server according to the basic resource data of the service server and
Operation data determines change or splits the granularity of the service server；
When changing the granularity of the service server, the current business server in the granularity routing of the granularity is become newly
Operational processor；
When splitting the granularity of the service server, according to the split rule, the granularity is split as multiple seed degree, and divide
Not Wei multiple seed degree corresponding service server is calculated, to determine the routing of corresponding granularity for multiple seed degree.
30. alert data processing method according to claim 29, wherein working as in the granularity routing by the granularity
When preceding service server is changed to new operational processor, further includes:
The context data of alert data in current business server is synchronized to the new operational processor；
The context data of the alert data of the corresponding granularity is deleted from current business server；With
Current business processor in the incidence relation of the granularity and service server is being changed to new operational processor.
31. alert data processing method according to claim 29, wherein in the granularity for splitting the service server
When, further includes:
The context data of alert data corresponding with the granularity is deleted from current business server；
The incidence relation of the granularity and service server is deleted, and records seed degree and the business service in the routing of corresponding granularity
The incidence relation of device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910385528.8A CN110275763A (en) | 2019-05-09 | 2019-05-09 | A kind of alert data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910385528.8A CN110275763A (en) | 2019-05-09 | 2019-05-09 | A kind of alert data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110275763A true CN110275763A (en) | 2019-09-24 |
Family
ID=67959797
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910385528.8A Pending CN110275763A (en) | 2019-05-09 | 2019-05-09 | A kind of alert data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110275763A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579590A (en) * | 2019-09-29 | 2021-03-30 | 北京国双科技有限公司 | Data processing method, device, equipment and storage medium |
-
2019
- 2019-05-09 CN CN201910385528.8A patent/CN110275763A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579590A (en) * | 2019-09-29 | 2021-03-30 | 北京国双科技有限公司 | Data processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10999201B2 (en) | Dynamic advertisement routing | |
| JP2022106826A5 (en) | ||
| WO2020147331A1 (en) | Micro-service monitoring method and system | |
| CN108712464A (en) | A kind of implementation method towards cluster micro services High Availabitity | |
| CN111277629A (en) | High-availability-based web high-concurrency system and method | |
| CN103761309A (en) | Operation data processing method and system | |
| CN101820384A (en) | Method and device for dynamically distributing cluster services | |
| CN102694868A (en) | Cluster system implementation and task dynamic distribution method | |
| CN110020061B (en) | Smart city space-time information cloud platform service engine | |
| CN102984184B (en) | The service load balancing method and device of a kind of distributed system | |
| CN112953982B (en) | Service processing method, service configuration method and related device | |
| CN110580198B (en) | Method and device for adaptively switching OpenStack computing node into control node | |
| CN110213203A (en) | Network dispatching method, device and computer storage medium | |
| CN108600316A (en) | Data managing method, system and the equipment of cloud storage service | |
| CN113010313A (en) | Load balancing method and device, electronic equipment and computer storage medium | |
| CN107819632A (en) | A kind of dynamic load leveling group system based on performance monitoring system and Docker Swarm | |
| CN112492022A (en) | Cluster, method, system and storage medium for improving database availability | |
| CN108875035A (en) | The date storage method and relevant device of distributed file system | |
| CN114422331B (en) | Disaster recovery switching method, device and system | |
| CN104484228A (en) | Distributed parallel task processing system based on Intelli-DSC (Intelligence-Data Service Center) | |
| CN105282045B (en) | A kind of distributed computing and storage method based on consistency hash algorithm | |
| CN111258760A (en) | Platform management method, system, device and storage medium | |
| CN110275763A (en) | A kind of alert data processing system and method | |
| CN111352726A (en) | Streaming data processing method and device based on containerized micro-service | |
| CN107180034A (en) | The group system of MySQL database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |