CN103294558B - A kind of MapReduce dispatching method supporting dynamic trust evaluation - Google Patents
A kind of MapReduce dispatching method supporting dynamic trust evaluation Download PDFInfo
- Publication number
- CN103294558B CN103294558B CN201310206615.5A CN201310206615A CN103294558B CN 103294558 B CN103294558 B CN 103294558B CN 201310206615 A CN201310206615 A CN 201310206615A CN 103294558 B CN103294558 B CN 103294558B
- Authority
- CN
- China
- Prior art keywords
- entity
- trust
- trust value
- value
- schedulable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of MapReduce dispatching method supporting dynamic trust evaluation.The method include the steps that 1) by intrasystem each entity division, set up a kind of tree；2) initialize described tree, trust threshold, submission threshold value on host node, inherit the factor, feedback factor, the trust value of each entity；3) system calculates the trust threshold needed for this operation according to the job property of submitted to operation；Then look up trust value more than some entities of this trust threshold as schedulable entity, perform this operation；4) system carries out integrity verification to the execution result of schedulable entity, if by checking, then increasing the trust value of this entity, otherwise then reduce its trust value；5) change according to the trust value of entity, by feedback mechanism, father's entity trusts value of this entity is successively updated, until the root of tree.The present invention substantially increases the credibility of result of calculation.
Description
Technical field
The invention belongs to the security fields of cloud computing environment, relate to a kind of MapReduce dispatching method supporting dynamic trust evaluation,
It is mainly used on the MapReduce Computational frame of Hadoop.
Background technology
Hadoop is not only a distributed file system for storage, and is to be designed to be made up of universal computing device
Large-scale cluster on perform the framework of Distributed Application, MapReduce is the Computational frame of Hadoop.MapReduce framework
For processing the calculating of mass data in distributed parallel environment.One Task-decomposing is become the fine-grained subtask of more part by it,
After these subtasks are scheduled between idle process node and quickly process, merge life eventually through specific rule
Becoming final result, it processes model and is somewhat similarly to the decomposition in tradition programming model and inductive method.MapReduce model
Distributed arithmetic is abstracted into two steps of Map and Reduce, thus realizes efficient Distributed Application.Wherein Map step
The responsible Key-Value according to user's input is to generating intermediate object program, and intermediate object program uses the form of Key-Value pair equally.
All of intermediate object program is then merged by Reduce step according to Key, then generates final result.And developer needs to do
Map and the Reduce function logics realizing oneself exactly, be then forwarded to MapReduce running environment.
The feature of MapReduce causes the execution result of each subtask can affect final result of calculation.Cloud computing at present
Environment is the most complicated, and publicly-owned cloud and privately owned cloud merge, and distributed computing system is more and more open so that assailant is organic
Can take advantage of, external attacker obtains certain or the control of multiple cloud computing clustered node by various attack meanses, and inside is attacked
The existence of the person of hitting, has all threatened the calculating safety of MapReduce.Attack to MapReduce, can be summarized as two kinds and take advantage of
Deceive form:
Deception 1 hypothesis map function is f, calculates the allocated split D of participant (mapper), it is desirable to in all D
Record X ∈ D={x1..., xn, all carry out f (x) and calculate.Assailant's only subset D to D ' in record carry out f calculating,
And claim and all records have been carried out map calculating, it is complete by heartbeat notice master task.
Deception 2 hypothesis map function is f, calculates the allocated split D of participant (mapper), it is desirable to in all D
Record x ∈ D={x1..., xn, all carry out f (x) and calculate.Assailant carries out g calculating to the record in D, and claims
All records are carried out f calculating, has been complete by heartbeat notice master task.
According to the difference of the means attacked, assailant can be divided into following three classes.
The first kind is crude and rash attack.Assailant controls mapper, always returns the result of mistake.For this kind of attack, if right
Result is the most simply verified, it is possible to find the existence attacked.This attack, is also to be easiest to penetrate and avoid
's.
Equations of The Second Kind is common attack.Assailant returns the result of mistake with certain probability.First kind attack is that this class is attacked
Extreme case (probability is 100%).This attack is relatively common, and attacking than the first kind has higher success rate, is also more difficult to know
Broken.This type of is attacked, the way of voting can be used to take the integrity of the result, namely distribute multiple mapper
Simultaneously to a split computing, output can be compared after completing by map, if the result of multiple mapper all as,
Then thinking that result is correct, if there being Different Results, then choosing the most result of occurrence number as correct result.Lacking of this way
Point is to bring great performance to be lost.
3rd class is quick-witted attack.Assailant assumes there is trust systems, in a period of time in the Job scheduling mechanism of MapReduce
Return correct result the most always, until system loosens the supervision to it, trust assailant the most completely, can uncensored acceptance
The result that assailant provides.Now, assailant just can return the result of mistake.This attack means is the most cunning, is difficult to most
Take precautions against.
The attack pattern that assailant takes may be more increasingly complex than above-mentioned.Assailant may control multiple node, and this is more to hold
The easily inspection of avoidance system.According to whether the attack for there being cooperation, Attack Scenarios can be divided into again following two.
The first is non-conspiracy attack.The attack pattern before discussed substantially belongs to this type of.Under the model of non-conspiracy, assailant
Only control a worker, or control multiple worker, but each worker is independently to hold in attack process
OK, not cooperation and interaction between worker.Such as, an assailant controls two worker, two worker and is divided
Joining the same map operation performing on same split, the two worker all returns the result of mistake, but the two is wrong
Result is but different by mistake.
The second is conspiracy attack.Assailant controls multiple worker, and the behavior of one of them worker depends on closes with it
The behavior of other worker of scheme.Exchange and the communication of information can be there is between worker.This attack is mainly tackled system and is used
Replication or voting carries out result integrity verification.Replication is a special case of voting, and system is divided
Joining two worker and perform same calculating, if two worker return result difference, the most at least one is wrong.
Such as, assailant controls two worker, master after worker distribution task, two worker it is known that they
Identical input split whether is assigned.If identical, they can return identical result, but this result is
Mistake, thus avoid the risk being detected.The difficulty that conspiracy attack is implemented is higher, and successful probability is less, but
It is to bring the biggest threat.
The existence more than attacked so that make a set of can ensure that and calculate credible framework, a kind of believable MapReduce
Trust metrics and scheduling strategy are particularly important, this set strategy should compatible existing application program, it is contemplated that how to exist
Believable result of calculation is obtained on incredible node.The following is can find at present with result of calculation integrity in MapReduce
Protection and trustworthy scheduling policy-related (noun) Patent.
Publication No. US20090651100, invention entitled " SUSPICIOUS NODE DETECTION AND RECOVERY IN
MAPREDUCE COMPUTING " a kind of detection of disclosure of the invention and correct the method for malicious node, the method in cloud computing environment
Employing suspicion index (suspicion index), arranges suspicion threshold value, whether checks the suspicion index of node in tasks carrying
Exceeding this threshold value, if exceeding, carrying out the recovery performed, the result that otherwise recipient node performs.
Although this patent also relates to the detection of malicious node, detect malicious node by arranging of suspicion threshold value, and according to this
Recovering but the integrity protection aspect that do not calculates for MapReduce of malicious node is provided, its focus of attention more
In more common attack pattern.
Patent No. 200910311687.X, invention entitled " a kind of self adaptation job scheduling method based on MapReduce "
Invention provide a kind of based on calculating node Practical Calculation ability, there is adaptive task and divide and the method for task scheduling.This invention
Relate to MapReduce self adaptation job scheduling method in Distributed Parallel Computing field, comprise the following steps: that MapReduce calculates
Each calculates the Capability index of node list CPU core；Calculate the block size of MapReduce operation；Scheduling node is to newly entering
The data of MapReduce operation divide；The data chunk of MapReduce operation is dynamically dressed up task by scheduling node, point
Dispensing respectively calculates node；Dynamic statistics respectively calculates the resource utilization of node, if resource utilization is less than thresholding, recalculates
The block size of MapReduce operation.
This patent for the calculating in distributed system and storage resource, by its according to resource utilization dispatch, solution be
The problem of calculated performance in MapReduce.Although be also under MapReduce framework, task to be scheduling, but this patent
Not being a patent from security standpoint, what it was paid attention to is the efficiency solving Distributed Application.
Summary of the invention
For integrity and the credible guarantee problem of MapReduce result of calculation, current not Patents relates to.But
Along with MapReduce is increasingly widely applied in big data processing field, it is provided that a kind of believable mechanism ensures result of calculation
Credibility becomes the demand urgently met.The present invention is directed to the demand, propose one based on existing MapReduce Computational frame
Supporting the dispatching method of dynamic trust evaluation, it is it is important that give, on result integrity verification scheme flexibly, the reality that is scheduled
The dynamic trust evaluation system of body.Result integrity verification scheme uses test checking (Quiz) and checkpoint verification (Checkpoint)
Method combines, and the two is complementary to one another, it is to avoid erroneous judgement and failing to judge, and is the technical foundation of whole system, and two kinds of verification methods can
Use with selectivity, it is provided that motility；Trust evaluation system result based on above-mentioned integrity verification, to trusting, this is abstract
Concept quantifies, and by inheritance mechanism and feedback mechanism, trust value is carried out dynamic evaluation；Scheduling strategy then based on the former two,
To trust in the scheduler adding MapReduce as schedule considerations factor, and different operating is arranged trust value threshold value, thus
Reach the optimum balance of performance and trust.
Three main points in the elaboration invention of following emphasis:
One, result integrity verification: include result integrity verification based on Quiz and result integrity based on Checkpoint.
Quiz and normal calculating task are as broad as long, and only the result of calculation of Quiz can be verified.Result based on Quiz is complete
Integrity verification basic thought is insertion Quiz in normal tasks, and the participant of calculating cannot be distinguished by the existence of Quiz.By right
The checking of Quiz result, client can select the result accepted or refusal calculates.Its specific practice is, if a task is big
Little for s, client chooses the task of t size, is then sent to the task of t+m=s size calculate executor, wherein m
It is Quiz.After this task completes, client checks that the result of calculation of hiding Quiz is the most correct.The result of only Quiz is correct,
The result this time calculated just can be accepted.Otherwise, all result of calculations can be abandoned by client, reschedules and calculates.Quiz number
There is provided according to by user, or by system according to user-defined function or rule stochastic generation automatically.Quiz data are directly passed through generation
Reason is inserted in the input data of mapper, after map has calculated, and checking that the result of calculation of Quiz is compared.Checking
After completing, Quiz to be rejected, positioned the position of Quiz by the pattern match that map is exported key/value, then will
It is deleted from file.
In result integrity verification based on Checkpoint, system uses redundant computation, same task is distributed to not same
Two Worker of one node perform.When Worker calculate go to certain ad-hoc location of input file time (such as the 1st row,
100th row, the 1000th row), and arrive certain key position (as a file can not store Worker output number completely
According to when needing to write new file), suspend the execution of Worker, be set as Checkpoint.For performing the two of same task
Individual Worker, chooses corresponding Checkpoint, and data with existing is calculated cryptographic Hash, then this Checkpoint of system comparison
The cryptographic Hash of Shi Erzhe.If two cryptographic Hash are identical, accepting the result of this Checkpoint, calculating proceeds；If two
Individual cryptographic Hash is different, and the most at least a result is wrong, stops the execution of two Worker the most immediately, again chooses two
Individual Worker performs this task.
System uses the proof scheme that Quiz and Checkpoint combines, and when a task starts, two worker distribute simultaneously
Identical normal tasks and Quiz task, carry out the checking of Checkpoint during tasks carrying, if being verified, then
Tasks carrying carries out the checking of Quiz after completing.If the comparison of all Checkpoint cryptographic Hash is all identical, and Quiz
Result of calculation consistent with desired correct result, then be considered as, by result integrity verification, being otherwise considered as result integrity verification
Failure.Checkpoint scheme can find non-conspiracy attack and the existence of part conspiracy attack as early as possible, for conspiracy attack, by
Do not rely on Task Duplication in Quiz scheme, the existence that this kind is attacked can be stopped.And this combine advantage be prevented effectively from right
Failing to judge of various attacks.
Two, trust evaluation: by cloud computing system from data center, calculate node and press to each entity of process difference level of trust
According to physics or inclusion relation in logic, set up a kind of tree-shaped partition structure.One entity (father's entity) can comprise some
Individual other entities (sporophore), as a cluster comprises multiple calculating node, one calculates node and comprises multiple process.One reality
Body (sporophore) may only be contained in some other entity (father's entity) (as a process be contained in one calculate node, one
Individual calculating node is contained in a cluster etc.).Fig. 4 is the tree in a typical cloud computing system between entity.
Trust evaluation system is entered trusting this abstract conception according to tree attribute and their historical behavior of entity in system
Row quantifies.Trust in invention has three features: one, can inherit and feed back.Inheritance mechanism refers to, when a new entity adds
When entering system, its initial trust value is inherited in the entity comprising it, the trust value * of the trust value of the entity being i.e. newly added=father's entity
Inherit the factor；Feedback mechanism then refers to, the trust value variation of a sporophore influences whether that the trust value of his father's entity, i.e. father are real
The trust value variable quantity * feedback factor of the former trust value+sporophore of new trust value=father's entity of body.Two, it is dynamic for trusting
, the trust value of any entity is assessed in it runs and changes, and the trust value of related entities is by inheriting and feedback simultaneously
Mechanism is interactional, and in system is run, trust value is decayed over time, has such as spent 1 hour, and trust value subtracts
1, but due to the existence of other mechanism, trust value entirety is also likely to be increase.Three, trust has life cycle, different real
The trust life cycle of body is different.Restarting of system, the configuration that system is new, system manager's definition etc. can result in new life
The life cycle.
One entity starts its life cycle when adding system, causes the initialization of trust value.System has been run management role
Particular entity, including host node (master node), the trust value initial value of checking node (verifier node) is the most logical
Crossing default value to determine, service provides and empirically gives one basic value；For other by system administration entity (as
Mapper) its initial value is affected by inheritance mechanism, and the trust value of father's entity is given to sporophore after being multiplied by factor of influence.?
In the execution of entity, Quiz and Checkpoint scheme can carry out integrity verification to performing result, is trusted by checking
Reward, increase the trust value of this entity, otherwise then punish, reduce the trust value of this entity.The trust value change of this entity,
The trust value of his father's entity is affected by feedback mechanism, the like, until top entity.
Three, scheduling strategy: the scheduling strategy of system, on the basis of the original scheduling strategy of Hadoop, add trust because of
Element, it is ensured that all tasks can be performed reliably, and system is different with the service that user buys by task according to the input of user
Being divided into different priorities, the mission critical high to integrity sensitive degree, priority scheduling is trusting high physically execution.System
Trust threshold and submission threshold value are set, to meet client's demand to different degree of beliefs.
The scheduling strategy of the present invention includes lazy (Lazy Committing) mechanism of submitting to of task, task rollback mechanism and trust
Scheduling mechanism.Lazy submission (Lazy Committing) mechanism refers to that the result of calculation of an entity is submitted to the most at once, but
First put into global buffer (GRB), along with the execution of computing, newly verify that (Quiz verifies and Checkpoint by result
Checking) if the correct execution always of this entity, can constantly accumulate trust value, until exceed submission threshold value (submit to threshold value be by
The empirical value that system is arranged, is defaulted as unified value, but can be separately provided certain entity), the result produced before this entity is
Can submit to, i.e. the submission of mapper task in Hadoop, notice master tasks carrying completes, this entity (worker) after submission
Enter new life cycle.If certain entity is found to have cheating in result integrity verification, its trust value is according to system
Punitive measures be reduced, when the trust value of this entity is for, time negative, i.e. showing that this entity can not be trusted again, and this thread is terminated,
New thread is filled in thread pool wait scheduling.Here it is rollback is machine-processed, rollback mechanism is along with rescheduling.
Trusting scheduling mechanism uses trust threshold as the requirement of scheduling.Trust threshold is a specific trust value, service carry
Setting according to customer demand for business, job scheduling when, scheduler only can be higher than the trust of operation trust threshold to trust value
Entity distribution task.Trust threshold is the highest, and the entity meeting Job execution condition is the fewest, performs result the most credible.System is divided
Join the more resource that calculates to the high entity of trust value, and as far as possible by entity low for job assignment low for trust threshold to trust value,
To reach the balancing the load between different entities.Meanwhile, cloud service provider externally provides calculating service, and trust threshold can conduct
One standard of price.Require that high user can select to buy more expensive service to obtain higher trust threshold to credible.
Compared with prior art, the positive effect of the present invention is:
One, providing the scheme of two kinds of different result integrity verifications based on Quiz with Checkpoint, client can basis
Scheme is selected by application-specific scene, and can be easily added into new result integrity verification scheme as trust evaluation
Foundation, be not required to general frame is modified, it is provided that motility；
Two, conventional technology often only using CPU, disk etc. as the influence factor of scheduling, trust quantification while of the present invention,
Using trusting as the factor affecting job scheduling, can effectively prevent the attack for calculating safety including conspiracy attack,
And reach performance and believable balance；
Three, existing MapReduce program is not required to amendment or only needs the least amendment can run on the present invention, have upper and
Compatibility under to.
Accompanying drawing explanation
Fig. 1 is the method flow diagram of the present invention；
Fig. 2 is the constructive ways flow chart of Quiz；
Fig. 3 is the comparison of Quiz and removes flow chart；
Fig. 4 is the tree trusting collection of a typical system.
Detailed description of the invention
Below in conjunction with the accompanying drawings the concrete grammar of the present invention is explained in further detail.
The method flow of the present invention is as it is shown in figure 1, the Map Reduce system that dynamic evaluation is trusted in this support is tieed up by master node
Protecting system correctly runs the information of needs, including the tree of trusted entities, system blacklist, trust threshold, submits threshold value to,
Inheriting the factor, feedback factor, the trust value etc. of each trusted entities, wherein the trust value of trusted entities is as entity attribute data
One field of structure is dynamically revised during job scheduling.System is run and is first initialized above-mentioned parameter when starting, foundation
The trust value of each entity is calculated and stores by inheritance mechanism.Initial value has default reference value, and by the shadow of inheritance mechanism
Ring.The calculating of trust value occurs on master.First InitCredit () function calls GetDomainParent ()
Function, obtains father's entity of entity to be calculated.Afterwards by GetDomainCredit () function, obtain the letter of father's entity
Appoint value.The trust value of entity to be calculated is that the trust value * of the father's entity comprising it inherits the factor.Note, when reality to be calculated
Father's entity of body on the blacklist time, above-mentioned calculating will not be carried out, but indirect assignment will be-1, i.e. add blacklist.
When user submits operation to, system calculates desired parameters (trust threshold of operation, job priority) according to job property,
Scheduling strategy traversal reads each entity trusts value, finds trust value to be more than the entity of trust threshold, can scheduling entity set.?
In all schedulable entities, sort from small to large according to trust value, forward enough of selected and sorted from the entity of sequence
Entity so that the estimated value of the computing capability of these entities can reach the requirement of client, then operation is physically pressed at these
Distribute according to former Hadoop Scheduling factors (such as distance).
Task starts after being assigned to perform, and result is carried out integrity verification.Result integrity verification uses based on Quiz
With the scheme of Checkpoint, the combination of Quiz and checkpoint, by the initial data of performed operation and Hadoop
Add Agent layer ProxyReader between middle class RecordReader and ensure that the worker performing same map task is (adjustable
Degree entity) input data be the same.
In result integrity verification based on Quiz, the constructive ways of Quiz has two kinds as shown in Figure 2, one be client directly to
System provides Quiz data, and another kind is to provide rule, system produce by randomizer according to rule, for Quiz
Each field, first confirm that type and scope, system is stochastic generation in the range of being given.The insertion of Quiz is by ProxyReader
Realizing, system adds the ProxyReader that acts on behalf of, ProxyReader of a digital independent between RecordReader and data source
Working identical with original RecordReader, new RecorReader is equivalent to one layer of packaging of ProxyReader, it is ensured that mapper
The API reading data does not change.Adding Hook Function in ProxyReader, function triggers according to certain probability, when
When function triggers, the output of ProxyReader adds Quiz data (i.e. in map input).RecordReader handle
ProxyReader is as unique data source, it is ensured that the different Worker input performing same task is identical.Quiz calculates
After completing, according to key/value to and expected result carry out fuzzy matching, and reject (otherwise Quiz result according to matching result
Subsequent calculations can be affected).The comparison of Quiz and removal are as shown in Figure 3.Quiz after a map tasks carrying completes by master
Verify.
The setting of Checkpoint depends on trigger function CheckpointTrigger () and Freeze () function.?
Adding CheckpointTrigger function in RecordReader, Worker reads record by RecordReader class, when
When going to checkpoint, CheckpointTrigger () calls Freeze () function, Freeze () function to
RecordReader sends PAUSE signal, suspends its reading to input file, and waits that the record read completes to calculate,
Complete calculating and the comparison of cryptographic Hash afterwards.
The trust value of each entity, according to the above results integrity verification, is dynamically adjusted by trust evaluation system.If certain is real
Body is by result integrity verification (Quiz checking and Checkpoint checking are the most correct), then triggering CalCredit () function increases
The trust value of this entity, the former trust value of the trust value that this entity is new=this entity+trust award value, trusting award value is system
Definition on the occasion of.The change of the trust value of one entity, affects the trust value of his father's entity by feedback mechanism.System is fed back
Mechanism is realized by CreditFeedback () function.It is to trigger this function whenever the trust value of an entity changes
Perform.Dividing according to the logical level of entity, this function performs from bottom to top.Assume the trust value change of an entity
A, CreditFeedback () are triggered.It finds father's entity of this entity, father's entity first with GetDomainParent ()
New trust value=old trust value+a* feedback factor.If the root that this father's entity is tree (does not has the entity of father's entity),
Then feedback terminates, and otherwise proceeds, until reaching root.The change of the trust value of all entities, have impact on again under scheduling strategy
Behavior once.Otherwise, if certain entity is not over result integrity verification, (Quiz checking and Checkpoint verify at least
Have a failure), then it is considered as attack and is present in inside this entity, this entity is put on the blacklist, this result integrity verification
This entity after system is got rid of and attacked, can be taken down from blacklist by result as the reference of attack analysis.Whole system is strong
Cohesion, it is achieved that to support the MapReduce scheduling strategy of dynamic trust evaluation.
Additionally, system setup time attenuation function.The historical behavior pair that some entities (such as node) life cycle is longer, more long
Current reference meaning weakens, and the trust value of this entity can run along with system, and each a period of time deducts on current trust value declines
Depreciation.Pad value is also rule of thumb to be arranged by system.
Scheduling strategy depends on global buffer (Global Result Buffer) and rollback function, it is achieved that lazy submission (Lazy
Committing) mechanism and task rollback mechanism.In native system, the result of calculation of mapper is passed through after being not immediately
Shuffle process consigns to reducer, but is first placed on global buffer (GRB, Global Result Buffer)
In, GRB is stored in master node.It is placed through function RegisterGRB () to complete.RegisterGRB () is not carried out really
Data move, simply register in GRB.The calculating of Worker (mapper performs the Map operation in MapReduce)
Result is placed on the local disk of node, and master is not aware that the actual location of data, the registration entries being stored in GRB
The conversion of actual storage locations in disk of position and data is completed by the mapping being positioned in the TaskTracker from node.
After having registered, GRB exists for record the item not submitting result to.When the trust value of a worker accumulation exceedes submission threshold value
Time, the record data in GRB belonging to it will be submitted to by function TrustedSubmit (), the shuffle after carrying out
And reduce operation.If this worker is not over result integrity verification, it is put on the blacklist, belongs to this worker's
The content in GRB after a upper submission point all can be abandoned.RollBack () function is according to the record in GRB, coupling system
Daily record, it is judged which or which task is not correctly completed.After finding these tasks, then give notice to master,
Inform that these tasks of master need to reschedule, these tasks are put into the scheduling waited in task pool next time.
Claims (11)
1. support a MapReduce dispatching method for dynamic trust evaluation, the steps include:
1) it is different to trust set by each entity division in Map Reduce system, sets up the tree-shaped knot of Map Reduce system
Structure；
2) on the master node of Map Reduce system, described tree, trust threshold, submission threshold value, succession are initialized
The factor, feedback factor, the trust value of each entity；
3) Map Reduce system calculates the trust threshold needed for this operation according to the job property of submitted to operation；Then from described
In tree, lookup trust value is more than some entities of this trust threshold as schedulable entity, performs this operation；
4) Map Reduce system uses the proof scheme combined based on Quiz and Checkpoint to hold described schedulable entity
Row result carries out integrity verification, if by checking, then increasing the trust value of this schedulable entity, otherwise then reduce
The trust value of this schedulable entity；Wherein, the method for integrity verification by: first the initial data of submission operation
And perform map operation entity between add an Agent layer ensure to perform same map task schedulable entity input
Data are the same；Then during this Job execution, carry out Checkpoint checking, if Checkpoint checking
Pass through, then carry out Quiz checking after this Job execution completes；Wherein, by described Agent layer, Quiz data are added
In inputting to map, carry out described Quiz checking；
5) Map Reduce system changes according to the trust value of entity, by feedback mechanism to the upper layer entity trust value of this entity by
Layer is updated, until the root of described tree.
2. the method for claim 1, it is characterised in that the execution result of this operation is first put into a caching by described schedulable entity
In, when the trust value of this schedulable entity exceedes the submission threshold value of setting, this work that this schedulable entity produces before submitting to
Industry performs result, and notice master node tasks has performed.
3. method as claimed in claim 2, it is characterised in that be found in described integrity verification if as certain schedulable entity
There is cheating, reduce the trust value of this schedulable entity, when the trust value of this schedulable entity is for, time negative, terminating this adjustable
Degree entity also adds a new trusted entities and performs this operation.
4. method as claimed in claim 2, it is characterised in that described caching is a global buffer being positioned at master node；Slow
The method depositing described execution result is: described execution result is stored on the local disk of schedulable entity, and by described execution
Result is registered in this global buffer；Then on master node, set up the position of registration entries and perform that result is actual deposits
Storage space put between mapping.
5. method as claimed in claim 4, it is characterised in that if the trust value that schedulable entity performs this operation accumulation exceedes setting
Submission threshold value time, the record data in described global buffer belonging to this schedulable entity are submitted, notify master
Node tasks has performed.
6. method as claimed in claim 4, it is characterised in that described master node includes a blacklist；If schedulable is real
This schedulable entity not over integrity verification, is then added described blacklist by the execution result of body, abandons the described overall situation and delays
Rush the data after the upper submission point belonging to this schedulable entity in district.
7. method as claimed in claim 2, it is characterised in that the method for the trust value initializing each entity described is: first set
The trust value initial value of the root of described tree；Then utilize inheritance mechanism, the trust value of father's entity is multiplied by a succession because of
It is given to its sporophore after son, successively calculates the trust value initial value of each sporophore.
8. method as claimed in claim 7, it is characterised in that described master node includes a blacklist, if certain entity
The trust value indirect assignment of this entity on described blacklist, is then-1 by father's entity.
9. the method for claim 1, it is characterised in that in Map Reduce system, each entity is according to physics or in logic
Inclusion relation is divided into different trust set, sets up described tree；Wherein, father's entity bag in described tree
Containing several sporophore, a sporophore is only contained in some father's entity.
10. the method for claim 1, it is characterised in that Map Reduce system calculates according to job property and submitted to work
The priority of industry, the entity high to the job assignment trust value that priority is high；Each trusted entities has a life cycle, no
Trust life cycle with trusted entities is different；The trust value of described trusted entities is decayed over time.
11. the method for claim 1, it is characterised in that the described upper layer entity trust value by feedback mechanism to this entity
The method being successively updated is: the trust value variable quantity of the sporophore of entity is multiplied by a feedback factor, then real plus this
The former trust value of body is as the new trust value of this entity.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310206615.5A CN103294558B (en) | 2013-05-29 | 2013-05-29 | A kind of MapReduce dispatching method supporting dynamic trust evaluation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310206615.5A CN103294558B (en) | 2013-05-29 | 2013-05-29 | A kind of MapReduce dispatching method supporting dynamic trust evaluation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103294558A CN103294558A (en) | 2013-09-11 |
| CN103294558B true CN103294558B (en) | 2016-09-21 |
Family
ID=49095486
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310206615.5A Expired - Fee Related CN103294558B (en) | 2013-05-29 | 2013-05-29 | A kind of MapReduce dispatching method supporting dynamic trust evaluation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103294558B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103532967B (en) * | 2013-10-23 | 2017-02-08 | 北京工业大学 | Trust quantification method based on subject access behavior |
| CN103699441B (en) * | 2013-12-05 | 2017-07-18 | 深圳先进技术研究院 | The MapReduce report task executing method of task based access control granularity |
| CN105450560B (en) | 2014-08-14 | 2019-02-12 | 华为技术有限公司 | A kind of method of channel feedback transmission |
| CN105991596B (en) * | 2015-02-15 | 2020-11-20 | 中兴通讯股份有限公司 | Access control method and system |
| CN106406990B (en) * | 2016-08-26 | 2018-11-09 | 湘潭大学 | A kind of job stacking-reso urce matching method and system with security constraint |
| CN107741879A (en) * | 2017-10-19 | 2018-02-27 | 郑州云海信息技术有限公司 | A kind of big data processing method and its device |
| CN110139278B (en) * | 2019-05-20 | 2020-08-04 | 西安安盟智能科技股份有限公司 | Method of safety type collusion attack defense system under Internet of vehicles |
| CN110909390B (en) * | 2019-12-03 | 2022-06-24 | 北京百度网讯科技有限公司 | Task auditing method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101311898A (en) * | 2008-07-02 | 2008-11-26 | 北京大学 | Software element reliability evaluation method and system |
| CN101576944A (en) * | 2008-11-20 | 2009-11-11 | 武汉大学 | Computer secure startup system based on trusted platform module |
-
2013
- 2013-05-29 CN CN201310206615.5A patent/CN103294558B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101311898A (en) * | 2008-07-02 | 2008-11-26 | 北京大学 | Software element reliability evaluation method and system |
| CN101576944A (en) * | 2008-11-20 | 2009-11-11 | 武汉大学 | Computer secure startup system based on trusted platform module |
Non-Patent Citations (1)
| Title |
|---|
| 《基于MapReduce的并行Web服务自动组合》;黄龙涛等;《电子学报》;20120731;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103294558A (en) | 2013-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103294558B (en) | A kind of MapReduce dispatching method supporting dynamic trust evaluation | |
| CN110915166B (en) | Block chain | |
| Yao et al. | Fog resource provisioning in reliability-aware IoT networks | |
| US11017388B2 (en) | Cryptographically assured zero-knowledge cloud service for composable atomic transactions | |
| US11356282B2 (en) | Sending cross-chain authenticatable messages | |
| US10623269B2 (en) | Operator fusion management in a stream computing environment | |
| US9129108B2 (en) | Systems, methods and computer programs providing impact mitigation of cyber-security failures | |
| US10810055B1 (en) | Request simulation for ensuring compliance | |
| Wang et al. | Viaf: Verification-based integrity assurance framework for mapreduce | |
| US10740139B2 (en) | Method and system for performing hyperconvergence using blockchains | |
| EP2932381A1 (en) | Computer-implemented method, system and computer program product for deploying an application on a computing resource | |
| US20120188249A1 (en) | Distributed graph system and method | |
| Zhou et al. | Privacy regulation aware process mapping in geo-distributed cloud data centers | |
| CN110555079B (en) | Data processing method, device, equipment and storage medium | |
| Cameron et al. | Rule-based peer-to-peer framework for decentralised real-time service oriented architectures | |
| CN109191287A (en) | A kind of sharding method, device and the electronic equipment of block chain intelligence contract | |
| CN110457128B (en) | Task allocation method, device and system | |
| EP3049959A1 (en) | Processing a hybrid flow associated with a service class | |
| Netaji et al. | A comprehensive survey on container resource allocation approaches in cloud computing: State-of-the-art and research challenges | |
| Levitin et al. | Co-residence data theft attacks on N-Version programming-based cloud services with task cancelation | |
| KR102304954B1 (en) | Peer node, method for processing information executed on peer node and blockchain platform system | |
| Bissiriou et al. | Towards secure tag-MapReduce framework in cloud | |
| CN107851101A (en) | The processing based on stream cluster with key/value storage checkpoint | |
| Arantes et al. | Probabilistic byzantine tolerance scheduling in hybrid cloud environments | |
| Yusuf et al. | Energy aware parallel scheduling techniques for network-on-chip based systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160921 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |