CN108205470A - A kind of distribution ad data calculating task management system and method - Google Patents
A kind of distribution ad data calculating task management system and method Download PDFInfo
- Publication number
- CN108205470A CN108205470A CN201611188392.4A CN201611188392A CN108205470A CN 108205470 A CN108205470 A CN 108205470A CN 201611188392 A CN201611188392 A CN 201611188392A CN 108205470 A CN108205470 A CN 108205470A
- Authority
- CN
- China
- Prior art keywords
- task
- data calculating
- calculating task
- library
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000009826 distribution Methods 0.000 title claims abstract description 16
- 239000000284 extract Substances 0.000 claims description 37
- 238000007726 management method Methods 0.000 claims description 25
- 238000004064 recycling Methods 0.000 claims description 10
- 238000013468 resource allocation Methods 0.000 claims description 10
- 230000004048 modification Effects 0.000 claims description 9
- 238000012986 modification Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 8
- 241001269238 Data Species 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000015556 catabolic process Effects 0.000 abstract description 3
- 230000009885 systemic effect Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 12
- 230000008901 benefit Effects 0.000 description 11
- 238000000605 extraction Methods 0.000 description 10
- 238000004821 distillation Methods 0.000 description 8
- 239000004744 fabric Substances 0.000 description 6
- 238000004321 preservation Methods 0.000 description 6
- 230000001419 dependent effect Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001035 drying Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0241—Advertisements
- G06Q30/0247—Calculate past, present or future revenues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Finance (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Game Theory and Decision Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of distributed ad data calculating task management system and methods.The system includes：Task creation server, task library, distributed server cluster and multiple client；Task creation server, suitable for ad data calculating task is saved in task library；Task library, suitable for storing the ad data calculating task that the task creation server is created；The ad data calculating task extracted suitable for extracting ad data calculating task from task library, and is sent to distributed server cluster by client；Distributed server cluster, the ad data calculating task sent suitable for operation client.As it can be seen that the drawbacks of the invention avoids systemic breakdown is caused during single client failure；Ensure that multiple tasks are performed simultaneously simultaneously, improve system running speed；And system resource is made to have obtained effective distribution and make full use of, and then realize system load balancing.
Description
Technical field
The present invention relates to distributed proccessing fields, and in particular to a kind of distribution ad data calculating task management system
System and method.
Background technology
With the WEB application more and more universal diversification with business, distributed task management system has obtained users
Favor.But existing distributed management system has the following problems, cause system resource do not obtain effectively distribute and
It makes full use of, and then makes system load out of balance.
Existing distributed management system there are the problem of it is as follows：(1) a centre management father node is only set to system
Interior child node publication control instruction, causes synchronization that can only perform an instruction, is not had so as to the resource of system
The utilization of effect, once and the centre management father node break down, whole system will be out of service.(2) distributed management system
The task that part of nodes performs in system is more, and the task that part of nodes performs is few, leads to part of nodes in system at runtime,
The problems such as his node in a dormant state, leads to system resource waste, load imbalance long-term existence.
Invention content
In view of the above problems, it is proposed that the present invention overcomes the above problem in order to provide one kind or solves at least partly
State the distributed task management system and method for problem.
One side according to the present invention provides a kind of distributed ad data calculating task management system, wherein, it should
Ad data calculating task management system includes：Task creation server, task library, distributed server cluster and multiple clients
End；
Task creation server, suitable for ad data calculating task is saved in task library；
Task library, suitable for storing the ad data calculating task that the task creation server is created；
Client, suitable for extracting ad data calculating task, and the ad data calculating task that will be extracted from task library
It is sent to distributed server cluster；
Distributed server cluster, the ad data calculating task sent suitable for operation client.
Optionally, the distributed server cluster includes：Distributed resource manager YARN, suitable for being sent to client
Ad data calculating task be scheduled, run with being assigned in the respective server in cluster.
Optionally, in the distributed server cluster, YARN is using container DOCKER in the cluster on server for not
Different running environment needed for same ad data calculating task structure operation.
Optionally, the distributed resource manager YARN, suitable for the ad data calculating task in cluster is run money
Source is divided into multigroup, and one of which is preferential group, the correspondence markings ad data calculating task of assigned priority information；Work as reception
During to the ad data calculating task that assigned priority information is marked, it is assigned to preferential group and is run；If in preferential group
It is run temporarily without ad data calculating task, then the part ad data calculating task operation resource allocation in preferential group is arrived it
He organizes help, and other organize operation ad data calculating tasks, once there is the operation of ad data calculating task in preferential group, recycling divides
The operation resource of other groups is fitted on to preferential group.
Optionally, the task creation server, suitable for receiving newly-built ad data calculating task by front end page,
Ad data calculating task will be received to be saved in task library；And it is further adapted for receiving modification ad data by front end page
The instruction of calculating task modifies to the respective advertisement data calculating task in task library operation according to the instruction.
Optionally, client, suitable for from task library extract ad data calculating task when, first check task in task library
Table, according to the high ad data calculating task of task list advantage distillation priority, and the ad data calculating task that will be extracted
It is sent to distributed server cluster；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence information of each ad data calculating task.
Optionally, the multiple client includes one or more priority tasks processing clients；
Priority tasks handle client, suitable for extracting the ad data that assigned priority information is marked from task library
Calculating task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client, it is unmarked specified excellent suitable for being extracted from task library
The ad data calculating task of first grade information, and the ad data calculating task extracted is sent to distributed server collection
Group.
Optionally, distributed server cluster, suitable for the operating status of each ad data calculating task run is deposited
It stores up in the ad data calculating task operating status list in task library.
Optionally, client when being further adapted for extracting an ad data calculating task from task library, judges that this is wide
Accuse whether data calculating task depends on other ad data calculating tasks；If the ad data calculating task is independent of it
His ad data calculating task, then directly extract the ad data calculating task from task library and be sent to distributed server
Cluster；If the ad data calculating task depends on other ad data calculating tasks, pass through the advertisement in query task library
Data calculating task operating status list, judge the ad data calculating task rely on other ad data calculating tasks whether
It is finished；The ad data calculating task is extracted from task library if being finished and is sent to distributed server collection
Group；It carries out waiting until other ad data calculating tasks that the ad data calculating task relies on if being not carried out finishing
It extracts the ad data calculating task when being finished from task library again and is sent to distributed server cluster.
Optionally, the task library is appointed suitable for extracting an ad data when the request for receiving a client and calculating
During the request of business, lock token is added for the ad data calculating task, other clients has been avoided to ask to lift the advertisement again
Data calculating task.
According to another aspect of the present invention, a kind of distributed ad data calculating task management method is provided, wherein,
Ad data calculating task is saved in task library by task creation server；
The ad data calculating task that task creation server described in task library storage is created；
Client extracts ad data calculating task from task library, and the ad data calculating task extracted is sent to
Distributed server cluster；
The ad data calculating task that distributed server cluster operation client is sent.
Optionally, client is sent by the distributed resource manager YARN in the distributed server cluster wide
It accuses data calculating task to be scheduled, be run with being assigned in the respective server in cluster.
Optionally, the distributed resource manager YARN is using being different on container DOCKER in the cluster server
Different running environment needed for the structure operation of ad data calculating task.
Optionally, the ad data calculating task operation resource in cluster is divided by the distributed resource manager YARN
Multigroup, one of which is preferential group, the correspondence markings ad data calculating task of assigned priority information；YARN, which works as, to be received
When the ad data calculating task of assigned priority information is marked, it is assigned to preferential group and is run；If in preferential group temporarily
When run without ad data calculating task, then YARN arrives the part ad data calculating task operation resource allocation in preferential group
Other groups help other group operation ad data calculating tasks, once there are the operation of ad data calculating task, recycling in preferential group
The operation resource of other groups is assigned to preferential group.
Optionally, the task creation server receives newly-built ad data calculating task by front end page, will connect
The ad data calculating task received is saved in task library；
The task creation server also receives the instruction of modification ad data calculating task by front end page, according to this
Instruct operation of modifying to the respective advertisement data calculating task in task library.
Optionally, client from task library extract ad data calculating task when, first check task list in task library, according to
The high ad data calculating task of task list advantage distillation priority, and the ad data calculating task extracted is sent to point
Cloth server cluster；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence informations of multiple ad data calculating tasks.
Optionally, the multiple client includes one or more priority tasks processing clients；
Priority tasks processing client extracts the ad data calculating that assigned priority information is marked from task library
Task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client extract unmarked assigned priority from task library
The ad data calculating task of information, and the ad data calculating task extracted is sent to distributed server cluster.
Optionally, distributed server cluster takes office the operating status storage of each ad data calculating task run
It is engaged in the ad data calculating task operating status list in library.
Optionally, this method further comprises：
When client extracts an ad data calculating task from task library, whether the ad data calculating task is judged
Dependent on other ad data calculating tasks；
If the ad data calculating task is directly carried independent of other ad data calculating tasks from task library
It takes the ad data calculating task and is sent to distributed server cluster；
If the ad data calculating task depends on other ad data calculating tasks, by wide in query task library
Data calculating task operating status list is accused, judges that other ad data calculating tasks that the ad data calculating task relies on are
It is no to be finished；The ad data calculating task is extracted from task library and be sent to distributed server if being finished
Cluster；It carries out waiting until that other ad datas calculating that the ad data calculating task relies on is appointed if being not carried out finishing
Business extracts the ad data calculating task from task library and is sent to distributed server cluster again when being finished.
Optionally, the task library asking when the request one ad data calculating task of extraction for receiving a client
When asking, lock token is added for the ad data calculating task, asks to lift the ad data meter again to avoid other clients
Calculation task.
A kind of distributed task management system that the present invention is built includes：Task creation server, task library, distributed clothes
Business device cluster and multiple client；Task creation server, suitable for ad data calculating task is saved in task library；Task
Library, suitable for storing the ad data calculating task that the task creation server is created；Client, suitable for being extracted from task library
Ad data calculating task, and the ad data calculating task extracted is sent to distributed server cluster；Distribution clothes
Business device cluster, the ad data calculating task sent suitable for operation client.As it can be seen that the present invention by set multiple client from
The drawbacks of ad data calculating task is extracted in task library, leads to systemic breakdown when avoiding single client failure, into
One step ensure that the stable operation of system；It ensure that multiple tasks are performed simultaneously simultaneously, further improve the operation speed of system
Degree；The ad data calculating task that client is sent is run by setting distributed server cluster, obtains system resource
It effective distribution and makes full use of, and then realize system load balancing.
Above description is only the general introduction of technical solution of the present invention, in order to better understand the technological means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, below the special specific embodiment for lifting the present invention.
Description of the drawings
By reading the detailed description of hereafter preferred embodiment, it is various other the advantages of and benefit it is common for this field
Technical staff will become clear.Attached drawing is only used for showing the purpose of preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.
In the accompanying drawings：
Fig. 1 shows a kind of showing for distributed ad data calculating task management system according to an embodiment of the invention
It is intended to；
Fig. 2 shows a kind of showing for distributed ad data calculating task management method according to an embodiment of the invention
It is intended to.
Specific embodiment
The exemplary embodiment of the disclosure is more fully described below with reference to accompanying drawings.Although the disclosure is shown in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
Completely it is communicated to those skilled in the art.
Fig. 1 shows a kind of showing for distributed ad data calculating task management system according to an embodiment of the invention
It is intended to.As shown in Figure 1, the system includes：Task creation server, task library, distributed server cluster and multiple client；
Task creation server, suitable for ad data calculating task is saved in task library；
Task library, suitable for storing the ad data calculating task that the task creation server is created；
Client, suitable for extracting ad data calculating task, and the ad data calculating task that will be extracted from task library
It is sent to distributed server cluster；
Distributed server cluster, the ad data calculating task sent suitable for operation client.
To make the solution of the present invention clearer, illustrated with reference to a specific example.It is specific at one
In example, first, if the ad data calculating task received includes, " the daily ad margins of calculating " wide pool " ", " calculating " is covered
The daily ad margins of ox " " and " the daily ad margins of calculating " Erie " ", then the task creation server connects described
The ad data calculating task received is saved in the task library.Then, the task library will " calculatings " wide pool " daily wide
Announcement profit ", " the daily ad margins of calculating " Mongolia Ox " " and " the daily ad margins of calculating " Erie " " are stored；In addition,
Client ad margins daily by " wide pool " is calculated ", " the daily ad margins of calculating " Mongolia Ox " " and " calculating " Erie "
Daily ad margins " are extracted from the task library, and 3 ad data calculating tasks of the extraction are sent to
The distributed server cluster.Finally, the distributed server cluster runs the client and sends " calculating " wide pool "
Daily ad margins ", " the daily ad margins of calculating " Mongolia Ox " " and " the daily ad margins of calculating " Erie " ".
In one embodiment of the invention, the distributed server cluster includes：Distributed resource manager YARN,
Ad data calculating task suitable for being sent to client is scheduled, and is run with being assigned in the respective server in cluster.
Such as：The distributed server cluster includes " calculating receiving the ad data calculating task that the client sends
" wide pool " daily ad margins ", " the daily ad margins of calculating " Mongolia Ox " " and " the daily ad margins of calculating " Erie " "
Task after, by the distributed resource manager YARN in the distributed server cluster to above-mentioned ad data calculating task
It is scheduled, and is run in the respective server being assigned in the distributed server cluster, realize effective distribution of resource
With make full use of, and promotion system load balancing.
In one embodiment of the invention, in the distributed server cluster, YARN is being collected using container DOCKER
It is the different running environment needed for different ad data calculating task structure operations on server in group.Such as：It is distributed
The ad data calculating task that server cluster receives includes " the daily ad margins of calculating " wide pool " ", " calculating " Mongolia Ox "
Daily ad margins " and " the daily ad margins of calculating " Erie " ", the distributed resource manager YARN utilize container
The ad data calculating tasks of the DOCKER in the cluster on server to receive builds different needed for operation respectively
Running environment realizes and different environment is isolated, to play the role of resource isolation, and then promote source effective distribution and
It makes full use of, realizes system load balancing.
In one embodiment of the invention, the distributed resource manager YARN, suitable for by the advertisement number in cluster
It is divided into according to calculating task operation resource multigroup, one of which is preferential group, the correspondence markings advertisement number of assigned priority information
According to calculating task；When receiving the ad data calculating task that assigned priority information is marked, it is assigned to preferential group and carries out
Operation；If run in preferential group temporarily without ad data calculating task, the part ad data in preferential group is calculated and is appointed
Business operation resource allocation helps other group operation ad data calculating tasks to other groups, once there is ad data meter in preferential group
Task run is calculated, recycling is assigned to the operation resource of other groups to preferential group.
Such as：Ad data calculating task operation resource in cluster is divided into more by the distributed resource manager YARN
Group, and one of which is determined as preferential group, while the correspondence markings ad data calculating task of assigned priority information, only
Receive the ad data calculating task that assigned priority information is marked, then first by the ad data calculating task
It is assigned in this preferential group and runs.It should be noted that the operation resource preferentially organized is most, processing speed is most fast.Such as：
The ad data calculating task that the YARN is received is including " the daily ad margins of calculating " wide pool " ", " calculating " Mongolia Ox " is daily
Ad margins " and " the daily ad margins of calculating " Erie " ", the distributed resource manager YARN has found wherein " to calculate
" Erie " daily ad margins " are the ad data calculating task that assigned priority information is marked, then the YARN will
" the daily ad margins of calculating " Erie " ", which are assigned in described preferential group, to be run；If temporarily without advertisement number in described preferential group
When being run according to calculating task, then the part ad data calculating task in described preferential group is run resource allocation by the YARN
This two groups of operations are helped to " the daily ad margins of calculating " wide pool " " and " the daily ad margins of calculating " Mongolia Ox " " task groups
Ad data calculating task, once there is the operation of ad data calculating task in described preferential group, recycling is assigned to described " calculate
The operation resource of " wide pool " daily ad margins " and " the daily ad margins of calculating " Mongolia Ox " " task groups to it is described preferentially
Group.
In one embodiment of the invention, the task creation server, suitable for receiving what is created by front end page
Ad data calculating task will receive ad data calculating task and be saved in task library；And it is further adapted for through preceding end page
Face receive modification ad data calculating task instruction, according to the instruction to the respective advertisement data calculating task in task library into
Row modification operation.
Such as：Front end page is set, is convenient to the ad data that the task creation server real-time reception user creates
Calculating task, the ad data calculating task that the user creates include " the daily ad margins of calculating " wide pool " ", " calculate
" Mongolia Ox " daily ad margins " and " the daily ad margins of calculating " Erie " ", while the task creation server will connect
The ad data calculating task that the user received creates is saved in real time in the task library.And user can also pass through
Front end page modifies to the ad data calculating task, described in the task creation server is received by front end page
User changes the instruction of ad data calculating task, according to the instruction to the respective advertisement data calculating task in the task library
It modifies operation, greatly improves flexibility and the practicability of system.
In one embodiment of the invention, client, suitable for from task library extract ad data calculating task when, first look into
See task list in task library, according to the high ad data calculating task of task list advantage distillation priority, and it is wide by what is extracted
It accuses data calculating task and is sent to distributed server cluster；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence information of each ad data calculating task.
For example, the currently stored ad data calculating task list of user is equipped in the task library, and described wide
It accuses each ad data calculating task in data calculating task list and distinguishes correspondence markings precedence information；It is if described wide
Data calculating task list is accused to include：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", priority is minimum；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " ", highest priority；
Ad data calculating task 3：" the daily ad margins of calculating " Erie " ", during priority is；
So when client extracts ad data calculating task from the task library, first check described in the task library
Ad data calculating task table is calculated according to the high ad data of the ad data calculating task table advantage distillation priority and is appointed
Business 2, and the ad data calculating task 2 extracted is sent to distributed server cluster.
In one embodiment of the invention, the multiple client includes one or more priority tasks processing visitors
Family end；
Priority tasks handle client, suitable for extracting the ad data that assigned priority information is marked from task library
Calculating task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client, it is unmarked specified excellent suitable for being extracted from task library
The ad data calculating task of first grade information, and the ad data calculating task extracted is sent to distributed server collection
Group further ensures effectively distributing and make full use of, and then promotion system load balancing for system resource.
Such as：As shown in Figure 1, N number of client includes one or M priority tasks processing client, need
Illustrate, if the ad data calculating task list of the task library includes：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", no priority；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " ", priority；
Client 1 handles client for the priority tasks, and client 2 handles client for no priority tasks；That
Client 1 extracts ad data calculating task 2 from above-mentioned task, and ad data calculating task 2 is sent to described point
Cloth server cluster；Client 2 extracts ad data calculating task 1 from above-mentioned task, and by ad data calculating task 1
It is sent to the distributed server cluster.
In one embodiment of the invention, distributed server cluster, suitable for each ad data run is calculated
The operating status storage of task is in the task run status list in task library, convenient for the fault condition of real-time monitoring system,
It ensure that the normal operation of system, further promote effectively distributing and make full use of for resource.
Such as the task run status list includes：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", operation is completed；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " " wait for operation；
Ad data calculating task 3：" the daily ad margins of calculating " Erie " ", operation troubles
Further, the client when being further adapted for extracting an ad data calculating task from task library, is sentenced
Whether the ad data calculating task of breaking depends on other ad data calculating tasks；If the ad data calculating task is disobeyed
Rely in other ad data calculating tasks, then the ad data calculating task is directly extracted from task library and be sent to distribution
Server cluster；If the ad data calculating task depends on other ad data calculating tasks, by query task library
Ad data calculating task operating status list, judge the ad data calculating task rely on other ad datas calculate appoint
Whether business is finished；The ad data calculating task is extracted from task library if being finished and is sent to distributed clothes
Business device cluster；It carries out waiting until other ad data meters that the ad data calculating task relies on if being not carried out finishing
It calculates when tasks carrying finishes and extracts the ad data calculating task from task library again and be sent to distributed server cluster, into
One step ensure that effectively distributing and make full use of, and then promotion system load balancing for system resource.
Such as：When client extracts an ad data calculating task including " calculating " wide pool " is every from the task library
During it ad margins ", judge whether " the daily ad margins of calculating " wide pool " " depend on other tasks；
" if calculating " wide pool " daily ad margins " are independent of other ad data calculating tasks, then directly from
" the daily ad margins of calculating " wide pool " " are extracted in the task library and are sent to distributed server cluster；
If " the daily ad margins of calculating " wide pool " " include dependent on other ad data calculating tasks, " calculating is " wide
The daily advertising income in pool " " by the ad data calculating task operating status list in query task library, judges " to calculate
Whether " wide pool " daily advertising income " is finished.If the ad data calculating task operating status list includes：
Ad data calculating task 1：" the daily advertising income of calculating " wide pool " ", operation is completed；
Ad data calculating task 2：" the daily advertising income of calculating " Mongolia Ox " " waits for operation；
Ad data calculating task 3：" the daily advertising income of calculating " Erie " ", operation troubles；
So described client extracts " calculating " wide pool " daily ad margins from the task library " and be sent to point
Cloth server cluster；
If the ad data calculating task operating status list includes：
Ad data calculating task 1：" the daily advertising income of calculating " wide pool " ", is currently running；
Ad data calculating task 2：" the daily advertising income of calculating " Mongolia Ox " " waits for operation；
Ad data calculating task 3：" the daily advertising income of calculating " Erie " ", operation troubles；
So " the daily ad margins of calculating " wide pool " " are waited until " the daily ad margins of calculating " wide pool " "
" calculating " wide pool " is daily for " the daily advertising income of calculating " wide the pool " " extraction in task library described in Shi Zaicong that is finished relied on
Ad margins " task and be sent to distributed server cluster.
In one embodiment of the invention, the task library, suitable for when the request extraction one for receiving a client
During the request of a ad data calculating task, lock token is added for the ad data calculating task, has avoided other clients
It asks to lift the ad data calculating task again, avoids same task and be repeatedly executed, further ensure system resource
Effectively distribute and make full use of, and then promotion system load balancing.
Such as：One client sends extraction " the daily ad margins of calculating " wide pool " " task to the task library please
It asks, is that " calculating " wide pool " is daily when the task library receives " the daily ad margins of calculating " wide pool " " task requests
Ad margins " task adds lock token, asks to lift the ad data calculating task again to avoid other clients, further
Repeating for same task is avoided, effectively distributing and make full use of for system resource is further promoted, so as to fulfill being
The load balancing of system.
Fig. 2 shows a kind of schematic diagrames of distributed task management method according to an embodiment of the invention.Such as Fig. 2 institutes
Show, this method includes：
Ad data calculating task is saved in task library by step S200, task creation server.
Step S210, the ad data calculating task that task creation server described in task library storage is created.
Step S220, client extract ad data calculating task, and the ad data extracted is calculated from task library
Task is sent to distributed server cluster.
Step S230, the ad data calculating task that distributed server cluster operation client is sent.
To make the solution of the present invention clearer, illustrated with reference to a specific example.It is specific at one
In example, first, if the ad data calculating task received includes, " the daily ad margins of calculating " wide pool " ", " calculating " is covered
The daily ad margins of ox " " and " the daily ad margins of calculating " Erie " ", then the task creation server connects described
The ad data calculating task received is saved in the task library.Then, the task library will " calculatings " wide pool " daily wide
Announcement profit ", " the daily ad margins of calculating " Mongolia Ox " " and " the daily ad margins of calculating " Erie " " are stored；In addition,
Client ad margins daily by " wide pool " is calculated ", " the daily ad margins of calculating " Mongolia Ox " " and " calculating " Erie "
Daily ad margins " are extracted from the task library, and 3 ad data calculating tasks of the extraction are sent to
The distributed server cluster.Finally, the distributed server cluster runs the client and sends " calculating " wide pool "
Daily ad margins ", " the daily ad margins of calculating " Mongolia Ox " " and " the daily ad margins of calculating " Erie " ".
In one embodiment of the invention, in step S230, by the distribution in the distributed server cluster
Explorer YARN is scheduled the ad data calculating task that client is sent, to be assigned to the respective service in cluster
It is run on device.Such as：The distributed server cluster is in the ad data calculating task for receiving the client and sending
Including " calculating " wide pool " daily ad margins ", " the daily ad margins of calculatings " Mongolia Ox " " and " calculating " Erie " is daily
Ad margins " calculate above-mentioned ad data by the distributed resource manager YARN in the distributed server cluster
Task is scheduled, and is run in the respective server being assigned in the distributed server cluster, realizes the effective of resource
It distributes and makes full use of, and promotion system load balancing.
Further, the distributed resource manager YARN is using being different on container DOCKER in the cluster server
Ad data calculating task structure operation needed for different running environment.Such as：Distributed server cluster receives
Ad data calculating task include " calculatings " wide damp " daily ad margins ", " the daily ad margins of calculating " Mongolia Ox " "
" the daily ad margins of calculating " Erie " ", the distributed resource manager YARN is taken in the cluster using container DOCKER
The ad data calculating task on business device to receive builds the different running environment needed for operation respectively, and realizing will
Different environment are isolated, and to play the role of resource isolation, and then are promoted effectively distributing and make full use of for source, are realized system
System load balancing.
In one embodiment of the invention, the distributed resource manager YARN calculates the ad data in cluster
Task run resource is divided into multigroup, and one of which is preferential group, and the ad data of correspondence markings assigned priority information calculates
Task；YARN is assigned to preferential group and is transported when receiving the ad data calculating task that assigned priority information is marked
Row；If run in preferential group temporarily without ad data calculating task, YARN calculates the part ad data in preferential group
Task run resource allocation helps other group operation ad data calculating tasks to other groups, once there is ad data in preferential group
Calculating task is run, and recycling is assigned to the operation resource of other groups to preferential group, further ensures effectively dividing for system resource
Match and make full use of, to play the role of promotion system load balancing.
Such as：The distributed resource manager YARN task run resource in cluster is divided into it is multigroup, and will wherein
One group be determined as preferential group, while the correspondence markings ad data calculating task of assigned priority information, as long as receiving mark
The ad data calculating task of assigned priority information is remembered, then first that the ad data distribution of computation tasks is excellent to this
First run in group.It should be noted that the operation resource preferentially organized is most, processing speed is most fast.Such as：The YARN is received
The ad data calculating task arrived includes " the daily ad margins of calculating " wide pool " ", " the daily ad margins of calculating " Mongolia Ox " "
" the daily ad margins of calculating " Erie " ", the distributed resource manager YARN have found that wherein " calculating " Erie " is daily
Ad margins " be that the ad data calculating task of assigned priority information is marked, then the YARN is by " calculating " Erie "
Daily ad margins " are assigned in described preferential group and run；If temporarily without ad data calculating task in described preferential group
During operation, then the YARN is by the partial task operation resource allocation in described preferential group to " calculatings " wide damp " daily wide
Announcement profit " and " the daily ad margins of calculating " Mongolia Ox " " task groups help this two groups of operation ad data calculating tasks, once
Have the operation of ad data calculating task in described preferential group, recycling be assigned to " the daily ad margins of calculatings " wide damp " " and
The operation resource of " the daily ad margins of calculating " Mongolia Ox " " task groups is to described preferential group.
In one embodiment of the invention, the task creation server receives newly-built advertisement number by front end page
According to calculating task, the ad data calculating task received is saved in task library；
The task creation server also receives the instruction of modification ad data calculating task by front end page, according to this
Instruct operation of modifying to the respective advertisement data calculating task in task library.
Such as：Front end page is set, is convenient to the ad data that the task creation server real-time reception user creates
Calculating task, the ad data calculating task that the user creates include " the daily ad margins of calculating " wide pool " ", " calculate
" Mongolia Ox " daily ad margins " and " the daily ad margins of calculating " Erie " ", while the task creation server will connect
The ad data calculating task that the user received creates is saved in real time in the task library.And user can also pass through
Front end page modifies to the ad data calculating task, described in the task creation server is received by front end page
User changes the instruction of ad data calculating task, according to the instruction to the respective advertisement data calculating task in the task library
It modifies operation, greatly improves flexibility and the practicability of system.
In one embodiment of the invention, client from task library extract ad data calculating task when, first check appoint
It is engaged in task list in library, according to the high ad data calculating task of task list advantage distillation priority, and the advertisement number that will be extracted
Distributed server cluster is sent to according to calculating task；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence informations of multiple ad data calculating tasks further ensures effectively distributing and make full use of for system resource,
And then promotion system load balancing.
For example, the currently stored ad data calculating task list of user is equipped in the task library, and described wide
It accuses each ad data calculating task in data calculating task list and distinguishes correspondence markings precedence information；It is if described wide
Data calculating task list is accused to include：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", priority is minimum；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " ", highest priority；
Ad data calculating task 3：" the daily ad margins of calculating " Erie " ", during priority is；
So when client extracts ad data calculating task from the task library, first check described in the task library
Ad data calculating task table is calculated according to the high ad data of the ad data calculating task table advantage distillation priority and is appointed
Business 2, and the ad data calculating task 2 extracted is sent to distributed server cluster.
In one embodiment of the invention, the multiple client includes one or more priority tasks processing visitors
Family end；
Priority tasks processing client extracts the ad data calculating that assigned priority information is marked from task library
Task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client extract unmarked assigned priority from task library
The ad data calculating task of information, and the ad data calculating task extracted is sent to distributed server cluster, into
One step ensure that effectively distributing and make full use of, and then promotion system load balancing for system resource.
Such as：As shown in Figure 1, N number of client includes one or M priority tasks processing client, need
Illustrate, if the ad data calculating task list of the task library includes：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", no priority；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " ", priority；
Client 1 handles client for the priority tasks, and client 2 handles client for no priority tasks；That
Client 1 extracts ad data calculating task 2 from above-mentioned task, and ad data calculating task 2 is sent to described point
Cloth server cluster；Client 2 extracts ad data calculating task 1 from above-mentioned task, and by ad data calculating task 1
It is sent to the distributed server cluster.
In one embodiment of the invention, distributed server cluster is by each ad data calculating task run
In operating status storage to the ad data calculating task operating status list in task library, convenient for the failure of real-time monitoring system
Situation ensure that the normal operation of system, further promote effectively distributing and make full use of for resource.
Such as the task run status list includes：
Ad data calculating task 1：" the daily ad margins of calculating " wide pool " ", operation is completed；
Ad data calculating task 2：" the daily ad margins of calculating " Mongolia Ox " " wait for operation；
Ad data calculating task 3：" the daily ad margins of calculating " Erie " ", operation troubles
Further, this method further comprises：
When client extracts an ad data calculating task from task library, whether the ad data calculating task is judged
Dependent on other ad data calculating tasks；
If the ad data calculating task is directly carried independent of other ad data calculating tasks from task library
It takes the ad data calculating task and is sent to distributed server cluster；
If the ad data calculating task depends on other ad data calculating tasks, by wide in query task library
Data calculating task operating status list is accused, judges that other ad data calculating tasks that the ad data calculating task relies on are
It is no to be finished；The ad data calculating task is extracted from task library and be sent to distributed server if being finished
Cluster；It carries out waiting until that other ad datas calculating that the ad data calculating task relies on is appointed if being not carried out finishing
Business extracts the ad data calculating task from task library and is sent to distributed server cluster again when being finished, further
It ensure that effectively distributing and make full use of, and then promotion system load balancing for system resource.
Such as：When client extracts an ad data calculating task including " calculating " wide pool " is every from the task library
During it ad margins ", judge whether " the daily ad margins of calculating " wide pool " " depend on other tasks；
" if calculating " wide pool " daily ad margins " are independent of other ad data calculating tasks, then directly from
" the daily ad margins of calculating " wide pool " " are extracted in the task library and are sent to distributed server cluster；
If " the daily ad margins of calculating " wide pool " " include dependent on other ad data calculating tasks, " calculating is " wide
The daily advertising income in pool " " by the ad data calculating task operating status list in query task library, judges " to calculate
Whether " wide pool " daily advertising income " is finished.If the ad data calculating task operating status list includes：
Ad data calculating task 1：" the daily advertising income of calculating " wide pool " ", operation is completed；
Ad data calculating task 2：" the daily advertising income of calculating " Mongolia Ox " " waits for operation；
Ad data calculating task 3：" the daily advertising income of calculating " Erie " ", operation troubles；
So described client extracts " calculating " wide pool " daily ad margins from the task library " and be sent to point
Cloth server cluster；
If the ad data calculating task operating status list includes：
Ad data calculating task 1：" the daily advertising income of calculating " wide pool " ", is currently running；
Ad data calculating task 2：" the daily advertising income of calculating " Mongolia Ox " " waits for operation；
Ad data calculating task 3：" the daily advertising income of calculating " Erie " ", operation troubles；
So " the daily ad margins of calculating " wide pool " " are waited until " the daily ad margins of calculating " wide pool " "
" calculating " wide pool " is daily for " the daily advertising income of calculating " wide the pool " " extraction in task library described in Shi Zaicong that is finished relied on
Ad margins " task and be sent to distributed server cluster.
In one embodiment of the invention, the task library is when request one advertisement of extraction for receiving a client
During the request of data calculating task, lock token is added for the ad data calculating task, is asked again to avoid other clients
Lift the ad data calculating task, avoid same task and be repeatedly executed, further ensure effectively dividing for system resource
Match and make full use of, and then promotion system load balancing.
Such as：One client sends extraction " the daily ad margins of calculating " wide pool " " task to the task library please
It asks, is that " calculating " wide pool " is daily when the task library receives " the daily ad margins of calculating " wide pool " " task requests
Ad margins " task adds lock token, asks to lift the ad data calculating task again to avoid other clients, further
Repeating for same task is avoided, further promotes effectively distributing and make full use of for system resource；So as to fulfill being
The load balancing of system.
All in all, a kind of distributed ad data calculating task management system that the present invention is built includes：Task creation
Server, task library, distributed server cluster and multiple client；Task creation server, suitable for ad data is calculated
Task is saved in task library；Task library, suitable for storing the ad data calculating task that the task creation server is created；
Client suitable for extracting ad data calculating task from task library, and the ad data calculating task extracted is sent to point
Cloth server cluster；Distributed server cluster, the ad data calculating task sent suitable for operation client.As it can be seen that this
Invention avoids single client failure by the way that multiple client is set to extract ad data calculating task from task library
When the drawbacks of leading to systemic breakdown, further ensure the stable operation of system；It ensure that multiple tasks are performed simultaneously simultaneously, into
One step improves the speed of service of system；The ad data meter that client is sent is run by setting distributed server cluster
Calculation task makes system resource obtain effective distribution and make full use of, and then realize system load balancing.
It should be noted that：
Algorithm and display be not inherently related to any certain computer, virtual bench or miscellaneous equipment provided herein.
Various fexible units can also be used together with teaching based on this.As described above, required by constructing this kind of device
Structure be obvious.In addition, the present invention is not also directed to any certain programmed language.It should be understood that it can utilize various
Programming language realizes the content of invention described herein, and the description done above to language-specific is to disclose this hair
Bright preferred forms.
In the specification provided in this place, numerous specific details are set forth.It is to be appreciated, however, that the implementation of the present invention
Example can be put into practice without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this description.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of each inventive aspect,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the method for the disclosure should be construed to reflect following intention：I.e. required guarantor
Shield the present invention claims the more features of feature than being expressly recited in each claim.More precisely, as following
Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,
Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim is in itself
Separate embodiments all as the present invention.
Those skilled in the art, which are appreciated that, to carry out adaptively the module in the equipment in embodiment
Change and they are arranged in one or more equipment different from the embodiment.It can be the module or list in embodiment
Member or component be combined into a module or unit or component and can be divided into addition multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it may be used any
Combination is disclosed to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so to appoint
Where all processes or unit of method or equipment are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Profit requirement, abstract and attached drawing) disclosed in each feature can be by providing the alternative features of identical, equivalent or similar purpose come generation
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments means in of the invention
Within the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointed
One of meaning mode can use in any combination.
The all parts embodiment of the present invention can be with hardware realization or to be run on one or more processor
Software module realize or realized with combination thereof.It will be understood by those of skill in the art that it can use in practice
Microprocessor or digital signal processor (DSP) realize distributed ad data calculating task according to embodiments of the present invention
The some or all functions of some or all components in management system.The present invention is also implemented as performing here
The some or all equipment or program of device of described method are (for example, computer program and computer program production
Product).Such program for realizing the present invention can may be stored on the computer-readable medium or can have one or more
The form of signal.Such signal can be downloaded from internet website to be obtained either providing or to appoint on carrier signal
What other forms provides.
It should be noted that the present invention will be described rather than limits the invention, and ability for above-described embodiment
Field technique personnel can design alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference mark between bracket should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" before element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.If in the unit claim for listing equipment for drying, several in these devices can be by same hardware branch
To embody.The use of word first, second, and third does not indicate that any sequence.These words can be explained and run after fame
Claim.
The invention discloses A1, a kind of distributed ad data calculating task management system, wherein, which calculates
Task management system includes：Task creation server, task library, distributed server cluster and multiple client；
Task creation server, suitable for ad data calculating task is saved in task library；
Task library, suitable for storing the ad data calculating task that the task creation server is created；
Client, suitable for extracting ad data calculating task, and the ad data calculating task that will be extracted from task library
It is sent to distributed server cluster；
Distributed server cluster, the ad data calculating task sent suitable for operation client.
A2, the method as described in claim A1, wherein, the distributed server cluster includes：Distributed resource pipe
Device YARN is managed, the ad data calculating task suitable for being sent to client is scheduled, to be assigned to the respective service in cluster
It is run on device.
A3, the system as described in claim A2, wherein,
In the distributed server cluster, YARN is using being different advertisements on container DOCKER in the cluster server
Different running environment needed for the structure operation of data calculating task.
A4, the system as described in claim A1, wherein,
The distributed resource manager YARN is more suitable for the ad data calculating task operation resource in cluster is divided into
Group, one of which are preferential group, the correspondence markings ad data calculating task of assigned priority information；It is marked when receiving
During the ad data calculating task of assigned priority information, it is assigned to preferential group and is run；If temporarily without wide in preferential group
The operation of data calculating task is accused, then the part ad data calculating task operation resource allocation in preferential group is organized into help to other
Other group operation ad data calculating tasks, once there is the operation of ad data calculating task in preferential group, recycling is assigned to other
The operation resource of group is to preferential group.
A5, the system as described in claim A1, wherein,
The task creation server suitable for receiving newly-built ad data calculating task by front end page, will receive
It is saved in task library to ad data calculating task；And it is further adapted for receiving modification ad data by front end page and calculating appointing
The instruction of business modifies to the respective advertisement data calculating task in task library operation according to the instruction.
A6, the system as described in claim A1, wherein,
Client, suitable for from task library extract ad data calculating task when, first check task list in task library, according to appoint
The high ad data calculating task of table advantage distillation priority of being engaged in, and the ad data calculating task extracted is sent to distribution
Formula server cluster；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence information of each ad data calculating task.
A7, the system as described in claim A1, wherein, the multiple client includes one or more preferential in charge of a grade
Business processing client；
Priority tasks handle client, suitable for extracting the ad data that assigned priority information is marked from task library
Calculating task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client, it is unmarked specified excellent suitable for being extracted from task library
The ad data calculating task of first grade information, and the ad data calculating task extracted is sent to distributed server collection
Group.
A8, the system as described in claim A1, wherein,
Distributed server cluster, suitable for storing the operating status of each ad data calculating task run to task
In ad data calculating task operating status list in library.
A9, the system as described in claim A8, wherein,
Client when being further adapted for extracting an ad data calculating task from task library, judges the ad data
Whether calculating task depends on other ad data calculating tasks；If the ad data calculating task is independent of other advertisements
Data calculating task then directly extracts the ad data calculating task from task library and is sent to distributed server cluster；
If the ad data calculating task depends on other ad data calculating tasks, pass through the ad data meter in query task library
Task run status list is calculated, judges whether other ad data calculating tasks that the ad data calculating task relies on have performed
Finish；The ad data calculating task is extracted from task library and be sent to distributed server cluster if being finished；Such as
Fruit is not carried out finishing, and carries out waiting until that other ad data calculating tasks that the ad data calculating task relies on have performed
Bi Shizai extracts the ad data calculating task from task library and is sent to distributed server cluster.
A10, the system as described in claim A1, wherein,
The task library, suitable for when the request for the request one ad data calculating task of extraction for receiving a client
When, lock token is added for the ad data calculating task, other clients has been avoided to ask to lift ad data calculating again
Task.
The invention also discloses B11, a kind of distributed ad data calculating task management method, wherein,
Ad data calculating task is saved in task library by task creation server；
The ad data calculating task that task creation server described in task library storage is created；
Client extracts ad data calculating task from task library, and the ad data calculating task extracted is sent to
Distributed server cluster；
The ad data calculating task that distributed server cluster operation client is sent.
B12, the method as described in claim B11, wherein,
The ad data sent by the distributed resource manager YARN in the distributed server cluster to client
Calculating task is scheduled, and is run with being assigned in the respective server in cluster.
B13, the method as described in claim B12, wherein,
The distributed resource manager YARN is using being different advertisement numbers on container DOCKER in the cluster server
According to the different running environment needed for calculating task structure operation.
B14, the method as described in claim B11, wherein,
The distributed resource manager YARN by cluster ad data calculating task operation resource be divided into it is multigroup,
In one group be preferential group, the correspondence markings ad data calculating task of assigned priority information；YARN, which works as to receive, to be marked
During the ad data calculating task of assigned priority information, it is assigned to preferential group and is run；If temporarily without wide in preferential group
The operation of data calculating task is accused, then the part ad data calculating task in preferential group is run resource allocation to other groups by YARN
Other group operation ad data calculating tasks are helped, once there is the operation of ad data calculating task in preferential group, recycling is assigned to
The operation resource of other groups is to preferential group.
B15, the method as described in claim B11, wherein,
The task creation server receives newly-built ad data calculating task by front end page, wide by what is received
Data calculating task is accused to be saved in task library；
The task creation server also receives the instruction of modification ad data calculating task by front end page, according to this
Instruct operation of modifying to the respective advertisement data calculating task in task library.
B16, the method as described in claim B11, wherein,
Client from task library extract ad data calculating task when, task list in task library is first checked, according to task list
The high ad data calculating task of advantage distillation priority, and the ad data calculating task extracted is sent to distributed clothes
Business device cluster；
Wherein, ad data calculating task currently stored in task library and corresponding preservation are saved in task list
The precedence informations of multiple ad data calculating tasks.
B17, the method as described in claim B11, wherein, the multiple client includes one or more priority
Task handles client；
Priority tasks processing client extracts the ad data calculating that assigned priority information is marked from task library
Task, and the ad data calculating task extracted is sent to distributed server cluster；
Other clients in addition to priority tasks handle client extract unmarked assigned priority from task library
The ad data calculating task of information, and the ad data calculating task extracted is sent to distributed server cluster.
B18, the method as described in claim B11, wherein,
Distributed server cluster will be in the operating status storage to task library for each ad data calculating task that run
Ad data calculating task operating status list in.
B19, the method as described in claim B18, wherein, this method further comprises：
When client extracts an ad data calculating task from task library, whether the ad data calculating task is judged
Dependent on other ad data calculating tasks；
If the ad data calculating task is directly carried independent of other ad data calculating tasks from task library
It takes the ad data calculating task and is sent to distributed server cluster；
If the ad data calculating task depends on other ad data calculating tasks, by wide in query task library
Data calculating task operating status list is accused, judges that other ad data calculating tasks that the ad data calculating task relies on are
It is no to be finished；The ad data calculating task is extracted from task library and be sent to distributed server if being finished
Cluster；It carries out waiting until that other ad datas calculating that the ad data calculating task relies on is appointed if being not carried out finishing
Business extracts the ad data calculating task from task library and is sent to distributed server cluster again when being finished.
B20, the method as described in claim B11, wherein,
The task library is when the request of ad data calculating task is extracted in the request for receiving client
The ad data calculating task adds lock token, asks to lift the ad data calculating task again to avoid other clients.
Claims (10)
1. a kind of distribution ad data calculating task management system, wherein, ad data calculating task management system includes：
Task creation server, task library, distributed server cluster and multiple client；
Task creation server, suitable for ad data calculating task is saved in task library；
Task library, suitable for storing the ad data calculating task that the task creation server is created；
Client suitable for extracting ad data calculating task from task library, and the ad data calculating task extracted is sent
To distributed server cluster；
Distributed server cluster, the ad data calculating task sent suitable for operation client.
2. the method for claim 1, wherein the distributed server cluster includes：Distributed resource manager
YARN, the ad data calculating task suitable for being sent to client are scheduled, to be assigned in the respective server in cluster
Operation.
3. system as claimed in claim 2, wherein,
In the distributed server cluster, YARN is using being different ad datas on container DOCKER in the cluster server
Different running environment needed for calculating task structure operation.
4. the system as claimed in claim 1, wherein,
The distributed resource manager YARN, it is multigroup suitable for the ad data calculating task operation resource in cluster is divided into,
One of which is preferential group, the correspondence markings ad data calculating task of assigned priority information；Finger is marked when receiving
When determining the ad data calculating task of precedence information, it is assigned to preferential group and is run；If temporarily without advertisement in preferential group
Data calculating task is run, then the part ad data calculating task in preferential group is run resource allocation helps it to other groups
He organizes operation ad data calculating task, once there is the operation of ad data calculating task in preferential group, recycling is assigned to other groups
Operation resource to preferential group.
5. the system as claimed in claim 1, wherein,
The task creation server suitable for receiving newly-built ad data calculating task by front end page, will receive wide
Data calculating task is accused to be saved in task library；And it is further adapted for receiving modification ad data calculating task by front end page
Instruction modifies to the respective advertisement data calculating task in task library operation according to the instruction.
6. a kind of distribution ad data calculating task management method, wherein,
Ad data calculating task is saved in task library by task creation server；
The ad data calculating task that task creation server described in task library storage is created；
Client extracts ad data calculating task from task library, and the ad data calculating task extracted is sent to distribution
Formula server cluster；
The ad data calculating task that distributed server cluster operation client is sent.
7. method as claimed in claim 6, wherein,
The ad data that client is sent is calculated by the distributed resource manager YARN in the distributed server cluster
Task is scheduled, and is run with being assigned in the respective server in cluster.
8. the method for claim 7, wherein,
The distributed resource manager YARN is using being different ad data meters on container DOCKER in the cluster server
Different running environment needed for the structure operation of calculation task.
9. method as claimed in claim 6, wherein,
The distributed resource manager YARN by cluster ad data calculating task operation resource be divided into it is multigroup, wherein one
Group is preferential group, the correspondence markings ad data calculating task of assigned priority information；YARN when receive be marked it is specified
During the ad data calculating task of precedence information, it is assigned to preferential group and is run；If temporarily without advertisement number in preferential group
It is run according to calculating task, then the part ad data calculating task operation resource allocation in preferential group is organized help by YARN to other
Other group operation ad data calculating tasks, once there is the operation of ad data calculating task in preferential group, recycling is assigned to other
The operation resource of group is to preferential group.
10. method as claimed in claim 6, wherein,
The task creation server receives newly-built ad data calculating task, the advertisement number that will be received by front end page
It is saved in task library according to calculating task；
The task creation server also receives the instruction of modification ad data calculating task by front end page, according to the instruction
It modifies to the respective advertisement data calculating task in task library operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611188392.4A CN108205470A (en) | 2016-12-20 | 2016-12-20 | A kind of distribution ad data calculating task management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611188392.4A CN108205470A (en) | 2016-12-20 | 2016-12-20 | A kind of distribution ad data calculating task management system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108205470A true CN108205470A (en) | 2018-06-26 |
Family
ID=62603230
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611188392.4A Pending CN108205470A (en) | 2016-12-20 | 2016-12-20 | A kind of distribution ad data calculating task management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108205470A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111338790A (en) * | 2020-02-12 | 2020-06-26 | 中山大学 | High-throughput computing task management method and system |
| CN117076555A (en) * | 2023-05-08 | 2023-11-17 | 芜湖本初子午信息技术有限公司 | Distributed task management system and method based on calculation |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915254A (en) * | 2011-08-02 | 2013-02-06 | 中兴通讯股份有限公司 | Task management method and device |
| KR20130074227A (en) * | 2011-12-26 | 2013-07-04 | 텔코웨어 주식회사 | Deistributed data management system and method thereof |
| CN104468638A (en) * | 2013-09-12 | 2015-03-25 | 北大方正集团有限公司 | Distributed data processing method and system |
| CN104657214A (en) * | 2015-03-13 | 2015-05-27 | 华存数据信息技术有限公司 | Multi-queue multi-priority big data task management system and method for achieving big data task management by utilizing system |
| CN104794003A (en) * | 2015-02-04 | 2015-07-22 | 汉鼎信息科技股份有限公司 | Large data analysis system integrating real-time mode and non-real-time mode |
-
2016
- 2016-12-20 CN CN201611188392.4A patent/CN108205470A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915254A (en) * | 2011-08-02 | 2013-02-06 | 中兴通讯股份有限公司 | Task management method and device |
| KR20130074227A (en) * | 2011-12-26 | 2013-07-04 | 텔코웨어 주식회사 | Deistributed data management system and method thereof |
| CN104468638A (en) * | 2013-09-12 | 2015-03-25 | 北大方正集团有限公司 | Distributed data processing method and system |
| CN104794003A (en) * | 2015-02-04 | 2015-07-22 | 汉鼎信息科技股份有限公司 | Large data analysis system integrating real-time mode and non-real-time mode |
| CN104657214A (en) * | 2015-03-13 | 2015-05-27 | 华存数据信息技术有限公司 | Multi-queue multi-priority big data task management system and method for achieving big data task management by utilizing system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111338790A (en) * | 2020-02-12 | 2020-06-26 | 中山大学 | High-throughput computing task management method and system |
| CN111338790B (en) * | 2020-02-12 | 2023-07-04 | 中山大学 | High-throughput computing task management method and system |
| CN117076555A (en) * | 2023-05-08 | 2023-11-17 | 芜湖本初子午信息技术有限公司 | Distributed task management system and method based on calculation |
| CN117076555B (en) * | 2023-05-08 | 2024-03-22 | 深圳市优友网络科技有限公司 | Distributed task management system and method based on calculation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108182111A (en) | Task scheduling system, method and apparatus | |
| CN104834561B (en) | A kind of data processing method and device | |
| CN102426602B (en) | Scoped database connections | |
| CN104915285B (en) | A kind of container process monitoring method, apparatus and system | |
| CN106888254A (en) | A kind of exchange method between container cloud framework based on Kubernetes and its each module | |
| CN109729106B (en) | Method, system and computer program product for processing computing tasks | |
| CN110325968A (en) | System upgrade management in distributed computing system | |
| CN104461239B (en) | A kind of information interacting method and device | |
| CN106529682A (en) | Method and apparatus for processing deep learning task in big-data cluster | |
| CN107315627A (en) | A kind of method and apparatus of automatic configuration data warehouse parallel task queue | |
| US9836330B2 (en) | Virtual resource management tool for cloud computing service | |
| CN110532059B (en) | Quota management method and device for K8s cluster management software | |
| CN108829469A (en) | A kind of application program page methods of exhibiting and device | |
| CN106453501A (en) | Method and apparatus for modifying configuration information of service | |
| CN107370796A (en) | A kind of intelligent learning system based on Hyper TF | |
| CN109818810A (en) | A kind of access server connection optimization method, access server and communication system | |
| CN109343972A (en) | Task processing method and terminal device | |
| WO2016077146A1 (en) | Application assignment reconciliation and license management | |
| CN108153877A (en) | Data dictionary methods of exhibiting, device, terminal device and storage medium | |
| CN108112268A (en) | Management and the relevant load balancer of automatic expanded set | |
| CN108205470A (en) | A kind of distribution ad data calculating task management system and method | |
| CN109800078B (en) | Task processing method, task distribution terminal and task execution terminal | |
| CN107357640A (en) | Request processing method and device, the electronic equipment in multi-thread data storehouse | |
| CN110457559A (en) | Distributed data crawls system, method and storage medium | |
| CN105975329A (en) | Creating method and device of virtual machine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180626 |
|
| RJ01 | Rejection of invention patent application after publication |