Background technology
Data screening algorithm will be applied in the environment of multi-user, distribution and wide area networking, in its data
For the heart often in network tip, resource is big, and interconnection bandwidth is low, local to use there is also certain constraints policy between resource and user
Family can directly manage local resource, and long-distance user then needs to be connected to resource-side by comprehensively utilizing server.
The characteristics of different from general distributed system " decentralization ", data screening algorithm requirement have a management and control/
Role is arbitrated, the concentration to complete to rule is examined, to realize the screening to regular legitimacy.This just needs us to design one
Centralized management scheme.Distributed wide area networking is not very reliable, there are interruption, bandwidth is low, packet loss, big delay the shortcomings of, such as
Fruit Distributed-solution designs unreasonable, for global coherency, it is possible to must sacrifice local availability, or even cause to use
Family is using the obstacle on local resource, and local availability is preferentially the primary demand of user.
CAP (Consistency, Availability, Partition tolerance) law is taught that:At one
In distributed system, these three guarantees of uniformity, availability and subregion fault-tolerance can not be met at the same time.It is for distribution
System, subregion fault-tolerance is basic demand, therefore design can only make balance between uniformity and availability, and here it is have CP
System, AP systems and BASE model systems【I.e. basically available (Basically Available), flexible consistent (Soft
State) with final consistent (Eventually consistent)】The reason for Deng design pattern.
Data Consistency mostlys come from more copies in distributed system (referring mainly to geographic distribution here),
Modification i.e. in system there are multiple copies, data would generally carry out on the different copies.In order to solve one in distribution
Sex chromosome mosaicism is caused, existing several typical algorithms mainly have:Two-phase commitment algorithm, Lamport bakerys algorithm, Paxos algorithms,
Method of synchronization that Gossip algorithm and the optimistic lock principle of use are realized etc..Three of Google in 2006 are on " chubby lock clothes
The paper of business " points out that other distributed algorithms are all considered as the specialization form of Paxos.In consideration of it, only introduce below
Paxos algorithms once:
Paxos algorithms are that distributed consensus algorithm is used for solving how a distributed system reaches with regard to some value (resolution)
Into it is consistent the problem of.One typical scene is, in a distributed data base system, if the original state one of each node
Cause, each node is carried out the identical sequence of operation, then they can finally obtain a consistent state.To ensure each section
Point performs identical command sequence, it is necessary to perform one " consistency algorithm " in each instruction to ensure that each node is seen
Instruction it is consistent.Paxos algorithms are the consistency algorithms of a more efficient, and traditional distributed lock algorithm or not quite alike
, such as RA algorithms, the locking and de-locking on it and unit have one it is maximum what is common is that have it is apparent " request -- etc.
Treat " process, and relatively independent between Paxos algorithms each examples, election process can be with parallel execution, the example that occurs afterwards
It is that the example that can first occur again is decided by vote before, the sensation of not obvious " request -- wait ".Specific in application,
Distributed lock algorithm is more the sensation of " mutual exclusion ", and consistency algorithm is more the sensation of " synchronization ".Whole Paxos algorithms
It can be regarded as multiple intersection execution and interactional Two-phase commitment algorithm, the algorithm have the characteristics that:
1st, 3 points based on message transmission are it is assumed that i.e. no Byzantium general problem (Byzantine failure);As long as
Time enough is waited, message will be transferred to;And the Congressman on Paxos islands is will not to oppose determining for other Congressmen proposition
View.
2nd, still can normally work in the case where nodal point number is less than half failure, node fails can be at any time
Occur normally to perform without influencing algorithm;Before proposer proposes a motion, first have to and be enough to form majority
Acceptors communicates, and obtains the last motion received of their progress.
3rd, whole algorithm is basically to be carried out around motion numbering:Proposer is busy with selecting the numbering of bigger to carry
Friendship proposal, whether the numbering that acceptor then compares the proposal of submission has been maximum, as long as numbering determines, institute is right
The value answered is also determined that.
4th, a distributed problem is converted into a single-point problem, and the robustness of single-point is ensured by election mechanism,
Its solution is to elect a proposer to make leader, and all proposal are submitted by leader, when
Leader delay machine when at once elect other Leader again.
5th, proposer, acceptor, learner cannot be promised to undertake and necessarily occurred when others needs in Paxos algorithms,
Also it can not promise to undertake approval resolution or transmit the time of message.
Paxos algorithms are widely used in distributed consensus, but are needed towards our algorithm application
Ask, still there are following problem to be difficult to overcome for Paxos algorithms:
1st, the message in algorithm may lose during (storage forwarding) is transmitted, and Paxos algorithms
Message is then required all must reliably to reach.
2nd, in the case of wide area network failure, this resource user remains to manage this resource apparatus for algorithm requirement, but
Paxos algorithms can not be worked normally in the case where number of nodes is more than half failure conditions, also can not just realize this resource user to this money
The management of source device.
3rd, algorithm needs regular center to examine rule, and Paxos algorithms have the characteristics that decentralization, section
It is completely reciprocity between point, it is not necessary to any Centroid, so examination & approval function can not be performed.
4th, algorithm plays the part of " grand justice " role using the regular center of centralization, and Paxos algorithms using random or
Timeout mechanism realizes election algorithm.
The content of the invention
The present invention provides a kind of distributed consensus implementation method of data screening algorithm, avoids centralized fashion
Cause the global disabled situation of system to occur because the node at regular center is abnormal, while also ensure that all rule for creating and coming into force
Then by examination & approval.
Technical scheme is as follows:
A kind of distributed consensus implementation method of data screening algorithm, comprises the following steps:
Step 1: subscription client by WebService by rule set RuSend to user agent, user agent is by RuSolution
Analyse as Ru={ M1Ru, M2Ru..., MnRu, wherein M represents means, MiRuIt is unique by database for atomic rule, user agent
Property the constraint storage rule on centralized and distributed respectively;
Step 2: DDS middlewares are passed through as theme, distributed user agency using the resource of atomic rule and means classification
" publication/subscription " pattern is by MiRuIt is delivered to corresponding resource broker service device;At the same time, centralized user agent also by
Atomic rule is committed to regular center by DDS middlewares;If deliver failure if asynchronous return mistake, successful synchronization state after
Afterflow journey;
Step 3: rule centrally through database unique constraints storage rule, builds strictly all rules view, while to rule
Then examined;Distributed resource agency is put in storage the atomic rule received, and submits them to centralized resources agency
Enter library storage, while all atomic rules are deployed to corresponding resource apparatus;
Step 4: approval results are acted on behalf of and delivered by regular center by DDS middlewares to centralized resources;Lost if delivered
Lose then asynchronous return mistake, successful synchronization state and then continue flow;
Step 5: centralized resources agency corrects atomic rule state according to approval results, and by revised rule
State submits to distributed resource agency and regular center respectively;Distributed resource agency waits resource apparatus to return to deployment message
Examination & approval message is returned with centralized Resource Broker, if resource apparatus and centralized Resource Broker return to validation message, is distributed
Formula Resource Broker acts on behalf of Feedback Rule to distributed user and comes into force;Otherwise, if resource apparatus and centralized Resource Broker wherein
Any one returns to Pending The Entry Into Force message, and distributed resource agency then acts on behalf of Feedback Rule Pending The Entry Into Force to distributed user;
Step 6: distributed resource agency is come into force to distributed user agency and centralized Resource Broker Feedback Rule respectively
Whether message;Meanwhile centralized resources are acted on behalf of to message whether regular center feedback taking effect rules, regular center is again to centralization
User agent's Feedback Rule comes into force situation;
Step 7: distributed user is acted on behalf of to centralized user agent's co-ordination principle uniformity, subscription client is from distribution
Formula user agent obtains the state of rule, mutually coordinated by two kinds of frameworks of Distribution and Centralization, to reach rule state
Final consistency.
Beneficial effects of the present invention:
1st, the present invention is directed to the application scenarios of multi-user, distribution and wide area networking, analyzes data screening algorithm
Application demand, have studied relevant distributed consensus algorithm, drawn one can use engineering solution substitute science solution knot
By finally giving a specific Project Realization scheme.
2nd, in the environment that more Autonomous Domains are integrated, the thought of internal inconsistency is not transmitted externally based on information, basic
It is " autonomous uniformity " in domain by the consistency problem Mandatory Decomposition on multiple domains on the premise of meeting user's needs, and finally
Ensure two domains at borderline " coordinative coherence ", so can it is one big the problem of spatial decomposition and defeat in detail.
3rd, using a kind of solution of " centralization+distributed " double trestle structures, two stacks in wide area network aspect at the same time
Operation, it is non-interference, on endpoint reconcile two stacks on result inconsistent state, realize concentrate examination & approval function and part can
It is preferential with property.
4th, with audit generation examination & approval for condition of grant, support part can use precedence constraint.Audit needs people to participate in, for a long time
Hang-up is unfavorable for uniformity reconciliation, after being analysed in depth to user demand, iting is found that examination & approval in advance completely can be subsequent
Audit substituted, and this replacement can make the realization of local availability preference strategy more unsophisticated.
Embodiment
The invention will be described further below in conjunction with the accompanying drawings.
In order to avoid centralized fashion causes the global disabled situation of system to occur because the node at regular center is abnormal,
Again ensure at the same time it is all create the rule to come into force by examination & approval, centralized and distributed be combined this paper presents a kind of
Double trestle structures (see Fig. 1).Wherein, thick dashed line represents the connection of distributed structure/architecture, and heavy line represents the connection of centralized architecture, is
System uses Message-Passing Model.
System is made of user agent, regular four layers of center, Resource Broker and resource apparatus, wherein user agent and resource
Agency is using way to manage that is centralized and distributed and depositing.In new algorithm, centralized user agent, regular center
Acted on behalf of with centralized resources and form centralized architecture, and distributed user agency, the equity interconnection of distributed resource agency are formed
Distributed structure/architecture, centralization are run, are not interfere with each other in communication aspect parallel with distributed double stacks.
During increase rule, the Rule content of establishment is sent to distributed user and acted on behalf of by user, by it directly by under rule
Distributed resource agency is sent to, then rule is delivered to corresponding resource apparatus, rule need not examine first in distributed structure/architecture
Come into force temporarily;Centralized user agent then sends rule to regular center, and received rule is examined by regular center,
And approval results are handed down to centralized resources agency, which is issued dividing in same level by centralization again after receiving message
Cloth;Distributed resource agency is modified the state of rule according to approval results, and by revised rule state to disappear
Breath form passes to distributed user agency, centralized resources agency and resource apparatus at the same time;Centralized resources are acted on behalf of message
Be transferred to centralized user agent step by step upwards, same level it is distributed to centralized coordinative coherence;User is then from distribution
Formula user agent knows specifying information whether taking effect rules.
During deletion rule, distributed structure/architecture issues the Rule content of deletion step by step, and then need not again in centralized architecture
The rule of deletion is examined, is directly issued step by step, it is at the same time, same on user agent and Resource Broker layer
Level it is distributed to centralized coordinative coherence;User still knows the specific feelings of redundant rule elimination from distributed user Agency
Condition.
In algorithm, whether distributed or centralized architecture, creates or during deletion rule, message is all by upper
Down transmit step by step, and message whether taking effect rules or deletion is then to transmit step by step from lower to upper.
Whole algorithm framework is designed based on following principle:
1. functionally in a distributed manner based on, in effectiveness based on centralization
Strictly all rules does not have to the first approval results for coming into force temporarily, waiting rule center of examination & approval in distribution, if center is examined
Criticize not by the rule to come into force in resource being modified to Pending The Entry Into Force state, and reason feedback is to terminal user.Accordingly even when
In the case that regular center is abnormal, distributed structure/architecture operates as usual, and system still can normally complete the increase of rule, delete
Work, so that it is guaranteed that system functionally and is not affected by too big influence.
In new algorithm, centralized and distributed two sets of solutions exist at the same time in wide area network aspect,
Both ends (resource-side and user terminal) while coordinate the inconsistent of both effectiveness, based on the regular center of centralization.
2. not transmitting the thought of internal inconsistency externally based on information, the reliable delivery step by step of message is realized
2012, Poulin proposed a kind of thought that identity is not transmitted in SOA (Services Oriented Achitecture) and cloud,
Given this thought, security and managerial demand based on system, in cascaded form network environment, tandem node will need to transmit
Message encapsulated on the border of cascade, by cascade (dividing domain) coherency management, ensure two domains border uniformity, from
And a big consistency problem is tied under small environment.
In this rule system, user encapsulates and transmits on the two border with wide area internetwork message using networking, wide area network
Also only encapsulate and transmit on the two border with resource group internetwork message, and user can not then be disappeared with resource networking by border
Breath transmits, and the network environment entirely cascaded is thus divided into two points of domains, the consistent performance on domain border accesses guarantee, disappears
Breath does not transmit the inconsistency of inside externally.
3. with audit generation examination & approval for condition of grant, support part can use preference strategy
Required according to original system, the examination & approval function of executing rule is not only wanted at regular center, can also be realized to rule
Audit, if it will audit and examination & approval function separately forms two kinds of management modes, will destroy the local availability of system, be
Avoid this problem, this rule system with generation examination & approval of auditing for condition of grant so that the part for supporting system can use preferentially
Strategy.
First, the implementation method of algorithm
1st, state machine migration pattern, the coherence request of implementation rule distributed deployment are passed through
The state of state machine description has two kinds of major classes, logic when one kind is task run.Second of type is algorithm logic,
Using logic state machine, clearly characteristic describes complicated algorithm.
This rule system uses first kind state machine, by the migration of state machine come the consistent of implementation rule distributed deployment
Property.The state of this state diagram is typically that task is waiting some condition to reach, that is, task is in " wait state ".Usually
Task is to push directly on the engine of state machine running in itself, and the event of task inspection state machine itself reaches situation and divided
Hair.The transition of task status reach triggering by some event, it is desirable to which synchronization mechanism has readjustment ability between task, so as in bar
Part is pressed into event when setting up into state machine events queue.
User is by regular RuIt is issued to user agent, user agent creates rule and by R on ituResolve to Ru={ M1Ru,
M2Ru..., MnRu(wherein M represents means, MiRuFor atomic rule), user agent feeds back according to regular center and Resource Broker
Atomic rule status message carry out regulation rule state:During increase rule, as long as there is an atomic rule MiRuCome into force user i.e.
Rule R on agencyuState be adjusted to come into force, and to the concrete condition of the other Pending The Entry Into Force atomic rules of user's feedback;Delete rule
When then, only all atomic rule MiRuAll delete successfully, user agent just will rule R thereonuState be adjusted to delete,
For the not deleted atomic rule in part, then using persistence deleting mechanism untill it is all deleted successfully.
This rule system uses two sets of state machine modes, its atomic rule state machine transition graph is as shown in Fig. 2, wherein each layer
Level is expressed as:Distributed user Agent layer-DUL, centralized user agent layer-CUL, regular central core-RCL, distribution
Formula Resource Broker layer-DPL;Centralized resources Agent layer-CPL, processing equipment layer-EL.Migration is defined as:Trigger event E |
State operates (outgoing event E ');Trigger event defines:Level _ event-action occurs for event;In addition to local operation is completed, send
Message need to determine that other side receives and stores, and at this moment just the state of rule can be considered as stable state.
When to create rule exemplified by the state machine migration flow of atomic rule, in distributed structure/architecture, distributed user generation
After reason receives the rule creation U_create message that user sends, i.e., the atomic rule is 1. created in the database, and will
The rule state is set to DUL_created (establishment);2. export DUL_create message to distributed resource to act on behalf of, it is thus necessary to determine that
DPL is received and stored, while DUL_create message is sent to centralized user agent.Defined according to state transition, Fig. 2 is used
" U_create | UP1 (DUL_create) " represents aforesaid operations.After distributed resource agency receives DUL_create message,
1. creating the atomic rule in the database, and the rule state is set to DPL_created;2. DPL_create message is sent out
Send to CPL, while send to EL, it is thus necessary to determine that EL is received and stored.Acted on behalf of similar to distributed user, we use " DUL_
Create | PP1 (DPL_create) " represent aforesaid operations.After resource apparatus receives DPL_create message, in resource apparatus
The atomic rule is created, with " DPL_create | EP1 " is represented.
In centralized architecture, centralized user agent receives the DUL_create message from distributed user agency
Afterwards, the atomic rule is 1. created in the database, and the rule state is set to CUL_created;2. CUL_create is disappeared
Breath is sent to regular center, it is thus necessary to determine that RCL is received and stored.Aforesaid operations use " DUL_create | UCP1 (CUL_
Create) " represent.After regular center receives CUL_create message, the atomic rule is 1. created in the database, and this is advised
Then state is set to RCL_created;2. RCL_create message is sent to CPL, it is thus necessary to determine that CPL is received and stored.With
" CUL_create | RCP1 (RCL_create) " is represented.Centralized resources are acted on behalf of after RCL_create message is received, and are 1. existed
The atomic rule is created in database, and the rule state is set to CPL_created;2. CPL_create is sent to same
The distribution of layer.Represented herein with " RCL_create | PCP1 (CPL_create) ".
When creating rule, if regular center is broken down, centralized resources agency can not receive RCL_create message,
But the DPL_create message from distributed resource agency can be received;If there is abnormal, distribution in wan communication
Resource Broker may can not receive DUL_create message, but has no effect on and receive the centralized resources agency of same layer and send
CPL_create message.
To sum up, by the contrast and amendment of two sets of state machines of Distribution and Centralization, this rule system is realized to apply and needed
Ask concentration examination & approval function and the local availability of middle proposition preferential.
2nd, the publish/subscribe model based on DDS, realizes reliable communication under wide area network distributed environment
Publisher and subscriber are associated by theme in publish/subscribe model, and both sides need not know about other side wherein,
Need not be online at the same time, realize the multidimensional loose coupling of communicating pair time, space and data communication.
Existing Publish/Subscribe Middleware product is very much, as Java Message Service (Java message service,
JMS) specification solves the interconnection problem between each vendor product substantially, but the specification is used in distributed real time environment and also deposits
In many problems:It is realized using Java, have impact on the real-time of system;QoS constraints are not proposed in JMS, user cannot
Communication mode is specified according to their needs, limits the flexibility of communication;Master is preserved using fixed node server in JMS
Topic or queue, it is impossible to equally loaded or failure transfering node server, presence server bottleneck and security risk.
Data distributing (data distribution service, DDS) be OMG after CORBA specifications are released, specially
Door is directed to the latest specification that data publication/subscription model is promulgated in distributed real-time systems, it is purely data-centered
To carry out data distribution, and resource situation, the expectation degree to resource, Network status etc. are all described with qos parameter, significantly
Enhance real-time and the flexibility of communication.
3rd, the reconciliation mechanism based on checkpoint, realizes the uniformity revision to rule state
Data screening algorithm needs to ensure that the permanent of local data can use, cannot lose, once loss of data is run into,
The situation of storage disk array corrupted, above-mentioned requirements do not ensure that this requires we have to carry out reconciliation.
Reconciliation is from regular center to distributed user agency, centralized user agent, distributed resource is acted on behalf of and resource generation
Reason centralization initiates reconciliation request, is obtained according to checkpoint points from this in certain time (before checkpoint points
Time of reconciliation was not carried out in checkpoint points) the rule state sequence completed, then with the rule at regular center
Then status switch is compared.When rule state sequence order occurs inconsistent, " establishment " state is subject to user agent,
" coming into force ", " Pending The Entry Into Force " and " deletion " state is subject to Resource Broker.