CN103186418B - A kind of task distribution method and system - Google Patents
A kind of task distribution method and system Download PDFInfo
- Publication number
- CN103186418B CN103186418B CN201110456174.5A CN201110456174A CN103186418B CN 103186418 B CN103186418 B CN 103186418B CN 201110456174 A CN201110456174 A CN 201110456174A CN 103186418 B CN103186418 B CN 103186418B
- Authority
- CN
- China
- Prior art keywords
- task
- task message
- file
- message file
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Abstract
The present invention relates to a kind of task distribution method, comprise the following steps: generate task message file, and be stored in task message catalogue；For each Task Processing Unit, if being in idle state, then timing scan task message catalogue, whether inquiry exists pending and not have the task message file being locked；If it is present obtain described task message file and lock；Task Processing Unit processes described task message file.The invention still further relates to a kind of task dissemination system, including task message file creating apparatus and some Task Processing Units of being connected with task message file creating apparatus；Task message file creating apparatus is used for generating task message file, and is stored in task message catalogue；Task Processing Unit is for obtaining when idle state pending and not have the task message file being locked and process the task message file of this acquisition.The method of the invention and system are capable of the load balancing of Task Processing Unit.
Description
Technical field
The present invention relates to a kind of task distribution method and system, be primarily adapted for use in the distribution of website orientation task
Field.
Background technology
Along with the development of Internet technology, the content quantity exponentially of news portal website increases, and is
Extending the reading of netizen, website designer also can place the rope of related content in other pages, region
Draw link, attract netizen to read.Along with acquisition channel abundant, context is increasingly wider, content is sent out
The page quantity that cloth is to be updated drastically expands, and the problem brought therewith is how to ensure a large amount of renewal of the page
Under situation, the page generates passage will face immense pressure, thus cause generation task quickly not respond.
The method of such issues that solve at present mainly has two kinds, and monitoring generates channel status distribution task peace
All distributing task, monitoring generates the method for channel status distribution task typically has two ways, i.e. by hard
Part monitoring server state and by program monitoring server state distributed tasks.Both approaches has
Oneself obvious shortcoming.
By the method for dedicated hardware monitoring of tools server state, advantage is that task distribution is accurate, permissible
It is accurately controlled generation task and is distributed to some generation passage, it is ensured that each generates the relative of channel load
Equilibrium.Shortcoming the most clearly, if use hardware device monitoring server state, (example with high costs
As, F5 load-balancing device price about 200,000 yuans).
And using program monitoring server state, monitoring/task distributing programs must frequently obtain server and bear
Load state, owing to the task amount of page generation task is huge, general medium-sized or above portal website generates the page
Quantity is about 1000 pages per second.Therefore want to accomplish task distribution accurately, monitoring/task distribution journey
Sequence must obtain tasks carrying situation and the load pressure of service end in real time, distributes for it if meeting condition again
Task.The limitation that this mechanism is brought is, monitoring programme poll time must be the least, and guarantee distributes
Accuracy, poll frequency is the highest, and service end expense also can improve therewith.Additionally generate server
Generation task such as needs to specify, therefore in the face of a large amount of concurrent update at the to be monitored/task distributing programs
In the case of, the bottleneck of systematic function is easy to concentrate on monitoring programme.And if taking trunking mode to improve
The concurrency of monitoring programme, thus can carry the accuracy of the complexity of monitoring programme and task allocation algorithms
Go out high requirement.
And the method for mean allocation task is that successively or distribution in certain sequence is a certain by each generation task
Individual generation passage.Advantage be realize simple, shortcoming the most clearly, i.e. it is difficult to ensure that each generates passage
The relative equilibrium of task.Owing to the renewal of the page quantity of each task has uncertainty, by content-data
And the impact of adduction relationship between the page, can cause that some generation task is actual have updated substantial amounts of static page,
And some task only have updated a small amount of several page.Therefore the mode of mean allocation task is likely to produce
Raw this situation, some generates passage and has just received one and update the generation task that the page is a lot, and other
The task of generating passage only needs to update a small amount of page, when new task is distributed to this passage again, and
When this passage has also been not carried out the more new task of last time, the accumulation of task, load can be caused to increase.And
Other generate passage and are not allocated to task simultaneously, are in idle state so that hardware server resource
It is not used effectively.
Summary of the invention
For defect present in prior art, the technical problem to be solved is to provide a kind of negative
Carry task distribution method and the system of equilibrium.
For solving above-mentioned technical problem, the technical solution used in the present invention is as follows:
A kind of task distribution method, comprises the following steps:
Generate task message file, and be stored in task message catalogue；
For each Task Processing Unit, if being in idle state, then timing scan task message mesh
Record, whether inquiry exists pending and not have the task message file being locked；If it is present obtain
Take described task message file and lock；
Task Processing Unit processes described task message file.
A kind of task dissemination system, including task message file creating apparatus and with described task message file
Some Task Processing Units that generating means connects；
Described task message file creating apparatus is used for generating task message file, and is stored in task message
Catalogue；Described Task Processing Unit for obtaining pending and the most locked appointing when idle state
Business message file also processes the task message file of this acquisition.
The method of the invention and system, multiple Task Processing Units only the most just can be got
Task, after task quantity is more than Task Processing Unit number, each Task Processing Unit is appointed in process
Business, and there is no idle Task Processing Unit, it can be considered that the distribution of task meets each task
The requirement of processing means load balancing.Simultaneously because use file system store tasks message file, from soft
Part aspect solves the problems referred to above, reduces hardware cost and implementation cost.
Accompanying drawing explanation
Fig. 1 is the structured flowchart of task dissemination system in detailed description of the invention；
Fig. 2 is the flow chart of task distribution method in detailed description of the invention；
Fig. 3 is the processing procedure that task message file is stored in detailed description of the invention task message catalogue
Flow chart；
Fig. 4 is the process flowchart after processing the failure of task message file in detailed description of the invention.
Detailed description of the invention
Task distribution policy mainly considers following three aspect factor: task balance distribution, task are concurrently located
Reason and unusual condition process.Task balance being distributed, processing task when there is multiple Task Processing Unit
Time, for avoiding task amount distribution unbalanced, according to distribution policy, newly submitting to of task is assigned to load
Perform on the lightest Task Processing Unit, so that each Task Processing Unit all can maintain essentially identical
Work under the state of load.For task concurrent processing, when having at multiple Task Processing Unit wait simultaneously
During reason task, task is only capable of distributing to one of them Task Processing Unit, i.e. task will not be by other
Business processing means performs again.Processing for unusual condition, process task must possess transactional, i.e. appoint
Business will not be accidentally deleted, and causes processing task termination owing to mistake is abnormal in task processes
Or after failure, task can be labeled and wait sub-distribution again, without losing.The present invention consider above-mentioned
On the basis of three factors, utilize file lock mechanism and the strategy robbing task that task is distributed to light load
Task Processing Unit.Below in conjunction with the accompanying drawings the detailed description of the invention of the present invention is described in detail.
As it is shown in figure 1, the task dissemination system in present embodiment includes task message file creating apparatus
1, the some Task Processing Units 2 being connected with task message file creating apparatus 1 and failure handling device (figure
Not shown in 1).It is single that described task message file creating apparatus 1 includes that memory element 11, first locks
Unit 12, copied cells 13 conciliate lock unit 14.Described Task Processing Unit 2 include scanning element 21,
Query unit 22, acquiring unit 23, lock cell 24, processing unit 25 and deletion unit 26.
Task message file creating apparatus 1 is used for generating task message file, and is stored in task message mesh
Record.Wherein: memory element 11 is for being stored in temp directory by task message with document form；First lock
Cell 12 is used for locking task message file；Copied cells 13 is for will be stored in appointing of temp directory
Business message file copies to task message catalogue；Solve lock unit 14 to be used for after replicating successfully, release appointing
The locking of business message file.
Task Processing Unit 2 is for obtaining when idle state pending and being not have locked of task to disappear
Cease file and process the task message file of this acquisition.Wherein: scanning element 21 is for when idle state
Timing scan task message catalogue；Whether query unit 22 exists pending for query task message list
And there is no locked task message file；Acquiring unit 23 is for waiting to locate when task message catalogue exists
Reason and when there is no locked task message file, acquisition task message file；Second lock cell 24
Task message file for locking and acquiring；Processing unit 25 is used for processing task message file.First lock
Cell 12 and the second lock cell 24 lock the mode of task message file: in task message list
Set up the message lock file of task message file to be locked.Delete unit 26 for disappearing in the task of process
After breath, first delete the task message file of task message catalogue, then delete the message of this task message file
Lock file.
Failure handling device is after processing the failure of task message file at Task Processing Unit 2, and it is right to release
The locking of this task message file also records the frequency of failure；If the frequency of failure exceedes setting threshold value, then will
This task message file dumps to mission failure catalogue.
As in figure 2 it is shown, the method using said system to realize task distribution comprises the following steps:
(1) task message file creating apparatus 1 generates task message file, and is stored in task message mesh
Record.
First task message is stored in one in the form of a file can by each Task Processing Unit 2 all
In the file system that can have access to, will task message catalogue be arranged in described file system, it is common that
A block space on NAS (Network-attached storage, the network storage).
As it is shown on figure 3, in present embodiment, task message file is stored in the mistake of task message catalogue
Journey is as follows:
1. task message is stored in temp directory with document form by memory element 11.
2. the first lock cell 12 locks this task message file.
The method of locking task message file is: set up task message literary composition to be locked in task message list
The message lock file of part.Described message lock file identical with the filename of the task message file locked by it,
Extension name is different.Message lock file can set expiry date, exceedes the expiry date message lock literary composition of setting
Part just lost efficacy.Expiry date can determine according to concrete application, in website orientation field, typically sets
It is set to 30 minutes.
3. copied cells 13 will be stored in the task message file of temp directory and copies to task message catalogue.
4., after replicating successfully, solve lock unit 14 and release the locking to this task message file.
(2) for each Task Processing Unit 2, if being in idle state, then scanning element 21
Timing scan task message catalogue, query unit 22 inquire about whether exist pending and the most locked
Task message file；If it is present acquiring unit 23 obtains this task message file, the second locking is single
Unit 24 locks this task message file.
Each Task Processing Unit 2 after a task, is in idle state, now scans on having processed
Unit 21 meeting timing scan task message catalogue, query unit 22 searches whether that pending task disappears
Breath file.Generally the automatic regular polling time can respond requirement according to system, is set as 1 second.
After scanning element 21 scans some task message file, query unit 22 is searched this task and is disappeared
Cease the (identical with task message file with the message lock file whether having same file name under catalogue of file
Filename, different extension name).2 process of Task Processing Unit do not have message lock file or message
The task message file of lock file expiration, thus ensure that this task will not be executed once again.Only extremely
In the case of there is the possibility that task is performed a plurality of times, but the lowest according to reality test probability of detection, can neglect
Slightly disregard.
(3) Task Processing Unit 2 processes task message file.
The task definition of Task Processing Unit 2 loading tasks message file, and perform to appoint according to task definition
Business.Task Processing Unit 2, after having processed task message, is first deleted task message mesh by deletion unit 26
The task message file of record, then delete the message lock file of this task message file.
As shown in Figure 4, if Task Processing Unit 2 processes the failure of task message file, then by unsuccessfully
Reason device is deleted the message lock file of this task message file and records the frequency of failure；Now etc. message is effective
Phase crosses after date, and this task can be merely re-executed.If the frequency of failure exceedes setting threshold value, then by this task
Message file dumps to mission failure catalogue, system manager analyze and process.Threshold value is preferably 3.
Obviously, those skilled in the art can carry out various change and modification without deviating from this to the present invention
The spirit and scope of invention.So, if these amendments of the present invention and modification belong to right of the present invention and want
Ask and within the scope of equivalent technology, then the present invention is also intended to comprise these change and modification.
Claims (13)
1. a task distribution method, it is characterised in that comprise the following steps:
Generate task message file, and be stored in task message catalogue；Described by the storage of task message file
Process in task message list comprises the following steps:
Task message is stored in temp directory with document form；
Lock this task message file；
The task message file that will be stored in described temp directory copies to described task message catalogue；
After replicating successfully, release the locking to this task message file；
For each Task Processing Unit, if being in idle state, then timing scan task message mesh
Record, whether inquiry exists pending and not have the task message file being locked；If it is present obtain
Take described task message file and lock；
Task Processing Unit processes described task message file.
2. task distribution method as claimed in claim 1, it is characterised in that locking task message file
Method be: task message list set up task message file to be locked message lock file；Described
File is identical from the filename of the task message file locked by it, extension name is different for message lock.
3. task distribution method as claimed in claim 2, it is characterised in that: described message lock file sets
There is expiry date, exceed described expiry date message lock file and lost efficacy.
4. task distribution method as claimed in claim 3, it is characterised in that whether described inquiry exists
The method of the most locked task message file is: search whether do not have message lock file or disappear
The task message file that breath lock file lost efficacy.
5. task distribution method as claimed in claim 1, it is characterised in that: described timing scan time
Between be spaced apart 1 second.
6. task distribution method as claimed in claim 2, it is characterised in that: processing task message
After, first delete the task message file of task message catalogue, then delete the message lock of this task message file
File.
7. task distribution method as claimed in claim 1, it is characterised in that: if Task Processing Unit
Process the failure of task message file, then release the locking to described task message file and record the frequency of failure；
If the frequency of failure exceedes setting threshold value, then this task message file is dumped to mission failure catalogue.
8. task distribution method as claimed in claim 7, it is characterised in that: the described threshold value that sets is as 3.
9. a task dissemination system, it is characterised in that: include task message file creating apparatus (1)
And the some Task Processing Units (2) being connected with described task message file creating apparatus (1)；
Described task message file creating apparatus (1) is used for generating task message file, and is stored in task
Message list；Described Task Processing Unit (2) for obtain when idle state pending and not by
The task message file locked the task message file processing this acquisition；
Described task message file creating apparatus (1) includes for task message being stored in document form
The memory element (11) of temp directory；For locking first lock cell (12) of task message file；
The duplication of described task message catalogue is copied to for will be stored in the task message file of described temp directory
Unit (13)；And for after replicating successfully, release the solution lock unit to task message file locking
(14)。
10. task dissemination system as claimed in claim 9, it is characterised in that: described task processes dress
Put (2) and include the scanning element (21) for the timing scan task message catalogue when idle state；With
Inquiry that is pending and that do not have locked task message file whether is there is in query task message list
Unit (22)；For when task message catalogue exists pending and does not has locked task message literary composition
During part, obtain the acquiring unit (23) of task message file；Task message file for locking and acquiring
The second lock cell (24)；And for processing the processing unit (25) of task message file.
11. task dissemination systems as claimed in claim 10, it is characterised in that described first locking is single
The mode of unit (12) and the second lock cell (24) locking task message file is: at task message mesh
The message lock file of task message file to be locked is set up in record.
12. task dissemination systems as claimed in claim 11, it is characterised in that: described task processes dress
Put (2) also to include, for after having processed task message, first deleting the task message literary composition of task message catalogue
Part, then delete the deletion unit (26) of the message lock file of this task message file.
The 13. task dissemination systems as according to any one of claim 9～12, it is characterised in that: institute
System of stating also includes failure handling device, for processing task message literary composition at described Task Processing Unit (2)
After part failure, release the locking to this task message file and record the frequency of failure；If the frequency of failure surpasses
Cross setting threshold value, then this task message file is dumped to mission failure catalogue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110456174.5A CN103186418B (en) | 2011-12-30 | 2011-12-30 | A kind of task distribution method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110456174.5A CN103186418B (en) | 2011-12-30 | 2011-12-30 | A kind of task distribution method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103186418A CN103186418A (en) | 2013-07-03 |
| CN103186418B true CN103186418B (en) | 2016-08-17 |
Family
ID=48677596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110456174.5A Expired - Fee Related CN103186418B (en) | 2011-12-30 | 2011-12-30 | A kind of task distribution method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103186418B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103414756B (en) * | 2013-07-18 | 2016-08-24 | 华为技术有限公司 | A kind of task distribution method, distribution node and system |
| CN105450769A (en) * | 2015-12-10 | 2016-03-30 | 浪潮通用软件有限公司 | File transmission method and device |
| CN106773805A (en) * | 2016-11-15 | 2017-05-31 | 努比亚技术有限公司 | Task message processing system and method |
| CN107688500B (en) * | 2017-07-26 | 2021-12-24 | 创新先进技术有限公司 | Distributed task processing method, device, system and equipment |
| CN108228349A (en) * | 2017-12-26 | 2018-06-29 | 北京市商汤科技开发有限公司 | For handling the method for task, system and storage medium |
| CN108924214A (en) * | 2018-06-27 | 2018-11-30 | 中国建设银行股份有限公司 | A kind of load-balancing method of computing cluster, apparatus and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102016872A (en) * | 2008-05-08 | 2011-04-13 | 微软公司 | Controlling access to documents using file locks |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9213586B2 (en) * | 2009-03-18 | 2015-12-15 | Sas Institute Inc. | Computer-implemented systems for resource level locking without resource level locks |
| CN102012840A (en) * | 2010-12-23 | 2011-04-13 | 中国农业银行股份有限公司 | Batch data scheduling method and system |
-
2011
- 2011-12-30 CN CN201110456174.5A patent/CN103186418B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102016872A (en) * | 2008-05-08 | 2011-04-13 | 微软公司 | Controlling access to documents using file locks |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103186418A (en) | 2013-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9244928B1 (en) | Systems and methods for directory snapshot | |
| CN103186418B (en) | A kind of task distribution method and system | |
| US20220229803A1 (en) | Automatic archiving of data store log data | |
| US8335769B2 (en) | Executing replication requests for objects in a distributed storage system | |
| CN108804253B (en) | Parallel operation backup method for mass data backup | |
| CA2676593C (en) | Scalable secondary storage systems and methods | |
| US8527459B2 (en) | System and method for data replication between heterogeneous databases | |
| US20110161281A1 (en) | Distributed Transaction Management in a Distributed Shared Disk Cluster Environment | |
| JP5724735B2 (en) | Database update control device, database management system, and database update control program | |
| US8880488B1 (en) | Efficient extent-based B-tree insertion | |
| US10437788B2 (en) | Automatic detection, retry, and resolution of errors in data synchronization | |
| US8825606B1 (en) | Community based restore of computer files | |
| AU2008201421A1 (en) | Removal | |
| JP5686034B2 (en) | Cluster system, synchronization control method, server device, and synchronization control program | |
| CN110188103A (en) | Data account checking method, device, equipment and storage medium | |
| US9710532B2 (en) | Method for avoiding conflicts in database cluster | |
| CN112654978A (en) | Method, equipment and system for checking data consistency in distributed heterogeneous storage system in real time | |
| CN103577546A (en) | Method and equipment for data backup, and distributed cluster file system | |
| CN101673374A (en) | Bill processing method and device | |
| CN111338834B (en) | Data storage method and device | |
| JP5331050B2 (en) | Data synchronization system, data synchronization method, information processing apparatus, information processing method, and program | |
| US20110153769A1 (en) | Apparatus and method for accessing a metadata | |
| CN109753245B (en) | Multi-disk load balancing asynchronous read-write scheduling method and device | |
| US10078680B2 (en) | Method for streaming transactions in database cluster | |
| Wang et al. | Deister: A light-weight autonomous block management in data-intensive file systems using deterministic declustering distribution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160817 Termination date: 20161230 |