CN105335219A - Distribution-based task scheduling method and system - Google Patents
Distribution-based task scheduling method and system Download PDFInfo
- Publication number
- CN105335219A CN105335219A CN201410323841.6A CN201410323841A CN105335219A CN 105335219 A CN105335219 A CN 105335219A CN 201410323841 A CN201410323841 A CN 201410323841A CN 105335219 A CN105335219 A CN 105335219A
- Authority
- CN
- China
- Prior art keywords
- task
- frame
- hard disk
- accessed
- allocated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention is suitable for the technical field of data storage, and provides a distribution-based task scheduling method and system. The method comprises the following steps: setting an accessed datanode threshold value of each rack in a distributed cluster; obtaining an amount of the datanodes accessed in each rack, and judging whether the amount of the datanodes accessed in each rack at present exceeds the threshold value or not; and if the amount of the datanodes accessed in each rack at present exceeds the threshold value, distributing a new distributed task to other racks, or scheduling the new task to a task queue to wait. On the basis of a control algorithm based on IO (Input/Output) access, the single-cabinet power of cold storage data access and the energy consumption of an integral cold data center can be controllable, the relationship between flash memory media service and energy consumption is fully utilized, and the characteristics of data distribution type storage/access are combined to lower cold data backup cluster expense.
Description
Technical field
The present invention relates to technical field of data storage, particularly relate to a kind of based on distributed method for scheduling task and system.
Background technology
Along with evolution and the commercialization practice of " large data " theory and correlation technique, data have become one of most important assets of Internet firm.Have the design degree of correlation of several important feature and storage backup cluster very high in large concept data, namely data value density is relatively low, and data value uncertainty is relatively high, and data volume is large.Which dictates that store data needs according to data importance, access performance, accessed frequency, the features such as data redundancy requirement provide data service capabilities targetedly.And back up cluster and assume responsibility for last that prevent all loss of datas and ensure, need to take into full account actual demand from several aspects such as data content, application characteristic, service ability, resource consumptions.
Traditional data backup cluster is typically employed in the mode of line cluster, near line backup, offline backup.The data of different life are sought survival as required and are placed in different cluster, and each level cluster meets time difference for data access.Such as online cluster is real time access (close to 5 ~ 10ms level); Near line cluster and online cluster, by network link UNICOM, exist that to close be importing and exporting of data, and data access time is as the criterion real-time mode (depending on desired data amount and the duration importing online cluster, from minute to a hour rank); Off-line cluster and near line cluster are similarly the relation of data exporting, and desired data access needs to preengage preparation in advance, are prepare and the access time usually, as shown in Figure 1 with sky.
Along with the growth year by year of online data, require that the data storage capacity of near line cluster and off-line cluster also can be increasing accordingly, therefore near line and off-line cluster, the demand in extensibility, holistic cost also can become principal contradiction.Meanwhile along with cloud computing ability capability improving and be employed as this reduction year by year, the dimension excavated for data value and demand also may make the demand of accessing full dose data more frequent and urgent, so near line and off-line cluster in overall usability, overall performance aspect it is also proposed requirement.
Wherein online cluster selects the suitable datanodeorchunkserver (directory management node or block server) having calculating and storage capacity concurrently according to its Distributed Architecture.Near line backup adopts and the similar solution of online cluster usually, but by the cutting of Equipments Setting and use relatively inexpensive storage medium, such as near line SATA big capacity hard disk, cloud dish, filing dish etc., reduce TCO (the TotalCostOwnership total cost of ownership).
Offline backup adopts tape-shaped medium's to store usually, coordinates band special library management software to implement.
In the face of the data of current PB rank up to a hundred store, file, backup requirements, nearline storage cluster and the offline backup cluster adopting the business such as tape library to store software and hardware integration scheme are supporting extendability, access performance demand, and unit capacity cost aspect all runs into challenge in various degree.
With regard near line cluster, Internet firm has abandoned costliness substantially, there is the commercial NAS (NetworkAttachedStorage of limitation in capacity extension and behavior extension, the network storage equipment) equipment, the substitute is the distributed type assemblies scheme based on computer server framework.And usually distributed type assemblies scheme is used more common must be scheme based on Hadoop distributed file system, wherein directory management node adopts Large Copacity near line SATA hard disc and cloud dish or filing dish usually; These storage mediums are still that micro-precise electric control machinery magnetic arm coordinates the conventional hard of perpendicular recording magnetic storage medium to realize in essence, monomer power consumption mainly consumes in the motor, the electric-controlled mechanical magnetic arm seek operation that drive magnetic disc to rotate, and magnetic head read-write operation current work consumes, common 3.5 cun of 7200rpm hard disk idle power consumptions are about 7W, and full load runs power consumption more than 10 watts; 5400rpm slow-speed of revolution hard disk nominal power consumption is about 7W, and idle power consumption is at 4.5 ~ 5W, and the power consumption of 10000RPM and 15000RPM hard disk is higher.
For mechanical hard disk background power consumption, (idle still needs to keep disk to rotate, namely consuming electric energy conversion is mechanical energy) for energy resource consumption, and produce heat in this process and need system-level refrigeration means to take away heat simultaneously, therefore for utilizing the datanode of mechanical hard disk magnetic medium scheme (directory management node) scheme to need its Capex (CapitalExpenditure of actuarial on a large scale, i.e. capital expenditures) and Opex (OperatingExpense, operation cost), and be not 24*7 real time access near line cluster, read to write less more, without planning random write, the use feature having planning order to write, continue in this part overall plan to use the scheme of mechanical hard disk medium to need to drop into a large amount of equipment purchasing expenses in capex aspect, in lifecycle of industry cluster, need to pay a large amount of rack spaces take paid rent, consume a large amount of electric energy simultaneously.
In summary, obviously there is inconvenience and defect in prior art in actual use, so need a new scheme to carry out satisfied new system to the demand of low-power consumption.
Summary of the invention
For above-mentioned defect, the object of the present invention is to provide a kind of based on distributed method for scheduling task and system, the energy consumption that the control algolithm mainly solved based on IO access realizes the Single Cabinet power of cold storage data access and the cold data center of entirety is controlled, it makes full use of flash media service and energy consumption relation, in conjunction with the feature of the distributed storage/access of data, reduce cold data backup cluster cost.
To achieve these goals, the invention provides a kind of based on distributed method for scheduling task, described method comprises:
The accessed task computation Node B threshold of each frame in distributed type assemblies is set;
Obtain task computation number of nodes accessed in each frame, and judge in frame, whether current accessed task computation number of nodes exceeds described threshold value;
If current accessed task computation number of nodes exceedes described threshold value in described frame, then the new task of distribution is assigned in other frame or described new task is dispatched to task queue and wait for.
The present invention is corresponding provides a kind of based on distributed task scheduling system, and described system comprises:
Configuration module, for arranging the accessed task computation Node B threshold of each frame in distributed type assemblies;
Acquisition module, for obtaining task computation number of nodes accessed in each frame,
Analysis module, for judging in described frame, whether current accessed task computation number of nodes exceeds described threshold value;
Distribution module, for when accessed task computation number of nodes current in frame exceeds described threshold value, is then assigned to the new task of distribution in other frame or described new task is dispatched to task queue and wait for.
The present invention by arranging the accessed task computation Node B threshold of each frame in distributed type assemblies, task computation number of nodes accessed in each frame of Real-time Obtaining, and judges in frame, whether current accessed task computation number of nodes exceeds threshold value; If exceed, to be assigned to the new task of distribution in other frame or new task to be dispatched to task queue and wait for, and the task of being dispensed to this frame waits for that the accessed task computation number of nodes of this frame performs lower than continuing after threshold value one preset ratio again.Thus it is controlled to realize the energy consumption at Single Cabinet power and overall data center by the storage data access controlling each frame, thus make it possible to adopt the hard disk protected can be applied in cold data backup cluster, reduce cold data backup cluster cost.
Accompanying drawing explanation
Fig. 1 is the data exporting schematic diagram of online cluster, near line cluster and off-line cluster in prior art;
Fig. 2 is a kind of frame diagram based on distributed task scheduling system of the present invention;
Fig. 3 is the process flow diagram in an embodiment of the present invention, I/O request being integrated with IO queue;
Fig. 4 is that in an embodiment of the present invention, the I/O request in IO queue is carried out the process flow diagram distributed by directory management node;
Fig. 5 is a kind of process flow diagram based on distributed method for scheduling task of the present invention.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.
In the application one typically configuration, the equipment of terminal, service network and trusted party include one or more processor (CPU), input/output interface, network interface and internal memory.
Internal memory may comprise the volatile memory in computer-readable medium, and the forms such as random access memory (RAM) and/or Nonvolatile memory, as ROM (read-only memory) (ROM) or flash memory (flashRAM).Internal memory is the example of computer-readable medium.
Computer-readable medium comprises permanent and impermanency, removable and non-removable media can be stored to realize information by any method or technology.Information can be computer-readable instruction, data structure, the module of program or other data.The example of the storage medium of computing machine comprises, but be not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic RAM (DRAM), the random access memory (RAM) of other types, ROM (read-only memory) (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc ROM (read-only memory) (CD-ROM), digital versatile disc (DVD) or other optical memory, magnetic magnetic tape cassette, tape magnetic rigid disk stores or other magnetic storage apparatus or any other non-transmitting medium, can be used for storing the information can accessed by computing equipment.According to defining herein, computer-readable medium does not comprise non-temporary computer readable media (transitorymedia), as data-signal and the carrier wave of modulation.
In the prior art, in cold data backup cluster (near line cluster), adopt the scheme of Hadoop distributed file system, wherein directory management node adopts Large Copacity near line SATA hard disc and cloud dish or filing dish usually, these storage mediums are still that micro-precise electric control machinery magnetic arm coordinates the conventional hard of perpendicular recording magnetic storage medium to realize in essence, mechanical hard disk background power consumption is comparatively large for energy resource consumption, and produces heat in this process and need system-level refrigeration means larger to take away heat simultaneously.And near line cluster is not 24*8 real time access, read to write less more, without planning random write, the use feature having planning order to write, the scheme of mechanical hard disk medium is used to need to drop into a large amount of equipment purchasing expenses, in lifecycle of industry cluster, need to pay a large amount of rack spaces take paid rent, consume a large amount of electric energy simultaneously.
Therefore the technical matters that the present invention mainly solves is, the energy consumption that the control algolithm based on IO access realizes the Single Cabinet power of cold storage data access and the cold data center of entirety is controlled.
As shown in Figure 2, the present invention is a kind of based on distributed task scheduling system 100, and system 100 comprises configuration module 11, acquisition module 12, analysis module 13 and distribution module 14.Preferably, system 100 also comprises and merges module 15.
Configuration module 10, for arranging the accessed task computation Node B threshold of each frame in distributed type assemblies.Such as, according to the default power consumption of frame and the default power consumption of each task computation node, the threshold value of the task computation number of nodes that can simultaneously work in computer rack.For hadoop distributed type assemblies, hadoop cluster generally includes task management node (namenode) and task computation node (datanode), each node is a server, and each node comprises a disk memory array be made up of multiple disk.In one embodiment of the invention, when designing cold data backup cluster, complete machine frame memory node design specification can adopt: meet every rack-U242.5 cun of dish bit density, single channel XeonorATOMorARM64 mainboard orSOC plate, 32 ~ 64GB internal memory, be no less than 24portSATA/SAS port, 10,000,000,000 network interface * 2; Single node is fully loaded with power consumption 180W ~ about 200W (idle power consumption is no more than 60W); 48U space can hold 1152 solid state hard discs, with 0.5TB average size, can support 576TB.Single chassis theoretical power density 9 ~ 10kw; Actual telecommunication standard 4.5kw machine of renting carries use.Data distribution8 and data-accessing tasks distribution are no more than the half of any physical frame memory node, and whole node idle power consumption * nodes=complete machine frame idle power consumption is positioned within 4.5kw safe range.Thus the task computation Node B threshold that can simultaneously work can be gone out according to complete machine frame idle power consumption and whole node idle power consumption calculation.
Acquisition module 12, for obtaining task computation number of nodes accessed in each frame.
Analysis module 13, for judging in frame, whether current accessed task computation number of nodes exceeds threshold value.
Distribution module 14, for when accessed task computation number of nodes current in frame exceeds threshold value, is then assigned to the new task of distribution in other frame or new task is dispatched to task queue and wait for.In addition, distribution module 14 is also for when accessed task computation number of nodes current in frame exceeds threshold value, the task of being dispensed to this frame is waited for, and the accessed task computation number of nodes of this frame performs lower than continuing after threshold value one preset ratio again, and the task scheduling being also about to be dispensed to this frame is waited for task queue.
In the present invention, configuration module 11, acquisition module 12, analysis module 13 and distribution module 14 all can be arranged in directory management node (namenode) and run, the residing physical machine rack position of directory management node perceived task computation node (datanode), perception cluster interior framework information, in each frame of perception current accessed (read or write or background scheduling produce I/O) task computation number of nodes, arrange threshold values to control: namely the quantity of accessed task computation node exceeds the frame of threshold values simultaneously, no longer accept new scheduler task, new task enters task queue and waits for or be dispensed to other frames, having started of task waits for that the accessed task computation number of nodes of current frame performs lower than continuing after threshold values certain proportion again.Thus the energy consumption that the present invention realizes Single Cabinet power and overall data center by the storage data access controlling each frame is controlled, thus make it possible to adopt the hard disk protected can be applied in cold data backup cluster, reduce cold data backup cluster cost.
Preferably, the threshold value of hard disk quantity of configuration module 11 also for can work according to the power consumption calculation of each hard disk in the default power consumption of task computation node and task computation node simultaneously.This hard disk is preferably solid state hard disc, and solid state hard disc has less power consumption and the fast advantage of read or write speed.
Preferably, acquisition module 12 also for reading task to be allocated in task queue, and calculates the hard disk quantity of this task needs to be allocated activation, and obtains the hard disk quantity worked in task computation node.This task to be allocated is generally I/O request, and this task queue is IO queue.In one embodiment, this IO queue adopts FIFO (FirstInputFirstOutput, First Input First Output) scheduling mode, and acquisition module needs to calculate the hard disk quantity that the I/O request being positioned at head of the queue needs to activate.This task queue also can be that other fly fifo queue, corresponding, also can adopt other task scheduling modes.
Preferably, analysis module 13 is also for judging that the hard disk quantity that working in task computation node and task to be allocated need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can work in computing node simultaneously.
Preferably, when distribution module 14 also needs the hard disk quantity sum activated to be less than or equal to the hard disk quantity that can simultaneously work in computing node for the hard disk quantity that working in task computation node and task to be allocated, task to be allocated is fallen out and is dispensed in this task computation node and perform, otherwise this task to be allocated dormancy Preset Time in task queue.Acquisition module 11 also after dormancy Preset Time, again reads task to be allocated when the hard disk quantity worked in task computation node is less than the hard disk quantity that can work in task computation node simultaneously for this task to be allocated in task queue from task queue.Preferably, system 100 also comprises and merges module 15, and the task merging that the user for getting submits to is to task queue.
Thus the present invention can realize individual task node energy consumption according to the data memory access in each task computation node is controlled, thus the hard disk protected can be adopted can be applied in cold data backup cluster, reduce cold data backup cluster cost.
In one embodiment of the invention, when carrying out task scheduling, first the I/O request that user submits to is obtained, again this I/O request is integrated with in IO queue, and calculate the hard disk quantity that the I/O request being positioned at head of the queue in IO queue needs to activate, then the hard disk quantity that can simultaneously work in the hard disk quantity activated, the hard disk quantity worked in frame and frame is needed to determine whether this I/O request to be dispensed to this frame according to I/O request.Concrete, judge that the hard disk quantity that working in frame and I/O request need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can simultaneously work in frame, if the I/O request being then positioned at head of the queue goes out right, and be dispensed in frame and perform.If the hard disk quantity worked in this frame and I/O request need the hard disk quantity sum activated to be greater than the hard disk quantity that can work in frame simultaneously, this I/O request can be dispensed to other frames.And this frame dormancy Preset Time, perform when the hard disk quantity worked in frame is less than the hard disk quantity that can work in frame simultaneously and judge next time, namely judge that the hard disk quantity that working in frame and I/O request need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can simultaneously work in frame.
Fig. 3 is the process flow diagram in an embodiment of the present invention, I/O request being integrated with IO queue.This flow process comprises:
Step S301, obtains the I/O request that user submits to.This concrete step also comprises recording user ID, IO read-write type, goes out cryptographic hash according to I/O request address computation.
Step S302, by I/O request with merge IO queue the element of tail is compared.
Step S303, judges the type of this I/O request, if read request then enters step S304, if write request then enters step S3207.
Step S304, judges whether carry out reading to merge, is enter step S305, otherwise enters step S306.
Step S305, I/O request is incorporated into IO queue to tail, this I/O request enter read merge read in queue.
Step S306, directly joins the team this I/O request and waits for.
Step S307, judges whether carry out writing merging, is enter step S308, otherwise enters step S309.
Step S308, I/O request is incorporated into IO queue to tail, this I/O request enter read merge read in queue.
Step S309, directly joins the team this I/O request and waits for.
Fig. 4 is the process flow diagram I/O request in IO queue being carried out in an embodiment of the present invention distributing.This flow process comprises:
Step S401, according to power consumption setting, calculates the solid state hard disc quantity MAS that can simultaneously work in frame.
Step S402, obtains the solid state hard disc quantity NAS of work at present in frame.
Step S403, reads the I/O request of head of the queue but does not go out team from the IO queue merged.
Step S404, calculates the solid state hard disc quantity N that this I/O request needs to activate.
Step S405, judges whether N and NAS sum is less than or equal to MAS, if then perform step S406, otherwise performs step S407.
Step S406, is dispensed to this frame by this I/O request.
Step S407, this frame dormancy Preset Time.Enter step S408.
Step S408, obtains the solid state hard disc quantity NAS of this frame work at present again.
Step S409, judges whether the NAS again obtained is less than MAS, if then get back to step S403, no person gets back to step S407.
In the diagram, when this N and NAS sum is greater than MAS, this I/O request can be dispensed to other frames, such as, when being write request, can by the node of data stored in other; Also can wait for that the solid state hard disc quantity NAS of this frame work at present re-executes this I/O request when being less than the solid state hard disc quantity MAS that can simultaneously work in frame, such as, when I/O request is read request, due to data be stored in this frame time, can only wait for that the accessed task computation number of nodes of this frame performs lower than continuing after threshold value one preset ratio again.
Fig. 5 is a kind of process flow diagram based on distributed method for scheduling task of the present invention, and it is realized by system as shown in Figure 2, the method:
Step S501, arranges the accessed task computation Node B threshold of each frame in distributed type assemblies.This step is realized by configuration module 11 as shown in Figure 2.Preferably, this step also comprises: according to the default power consumption of described frame and the default power consumption of each task computation node, the threshold value of the task computation number of nodes that can simultaneously work in computer rack.
Step S502, obtains task computation number of nodes accessed in each frame, and judges in frame, whether current accessed task computation number of nodes exceeds threshold value.This step is realized jointly by acquisition module 12 as shown in Figure 2 and analysis module 13.
Step S503, if current accessed task computation number of nodes exceedes threshold value in frame, is then assigned to the new task of distribution in other frame or new task is dispatched to task queue and wait for.Preferably, if current accessed task computation number of nodes exceedes described threshold value in frame, the task of being dispensed to this frame waits for that the accessed task computation number of nodes of this frame performs lower than continuing after threshold value one preset ratio again.This step is realized by distribution module 14 as shown in Figure 2.
Preferably, the method also comprises: according to the threshold value of the hard disk quantity that the power consumption calculation of each hard disk in the default power consumption of task computation node and task computation node can work simultaneously; Read task to be allocated in task queue, and calculate the hard disk quantity of this task needs to be allocated activation; Obtain the hard disk quantity worked in task computation node; Judge that the hard disk quantity that working in task computation node and task to be allocated need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can simultaneously work in computing node; Perform if then task to be allocated is fallen out and be dispensed in this task computation node, otherwise this task to be allocated dormancy Preset Time in task queue.Preferably, also comprise after the step of this task to be allocated dormancy Preset Time in task queue: again from task queue, read task to be allocated when the hard disk quantity worked in described task computation node is less than the hard disk quantity that can work in task computation node simultaneously.Preferably, also comprise before reading the step of task to be allocated in task queue: obtain the task that user submits to, and by task merging to task queue.
Some online clusters adopt solid state hard disc at present, and solid state hard disc is eliminated after crossing guarantor usually.But solid state hard disc is after server system crosses guarantor in 3 years, and wherein most SSD is still in upstate; Solid state hard disc idle power consumption is less than 1W simultaneously, and access power consumption is in the electrical specification of about 3 ~ 5W, and the performance speciality of read access performance ultra-traditional mechanical hard disk far away; What consider that SSD performance in cooling requirements, space density and year failure rate is all better than mechanical hard disk can operation management performance.The present invention proposes a kind of reasonably resources circulation and optimizes operational version, by the solid state hard disc of " on line retirement ", design take frame as the memory node unit of dimension, coordinate the Data distribution8 based on control strategy and access control dispatching algorithm, make full use of IDC frame physical space density and single chassis power density, the overall cost reducing nearline storage cluster.
In sum, the present invention is by arranging the accessed task computation Node B threshold of each frame in distributed type assemblies, task computation number of nodes accessed in each frame of Real-time Obtaining, and judge in frame, whether current accessed task computation number of nodes exceeds threshold value; If exceed, no longer for this frame distributes new task, and the task of being dispensed to this frame waits for that the accessed task computation number of nodes of this frame performs lower than continuing after threshold value one preset ratio again.Thus it is controlled to realize the energy consumption at Single Cabinet power and overall data center by the storage data access controlling each frame, thus make it possible to adopt the hard disk protected can be applied in cold data backup cluster, reduce cold data backup cluster cost.
Certainly; the present invention also can have other various embodiments; when not deviating from the present invention's spirit and essence thereof; those of ordinary skill in the art are when making various corresponding change and distortion according to the present invention, but these change accordingly and are out of shape the protection domain that all should belong to the claim appended by the present invention.
Claims (14)
1. based on a distributed method for scheduling task, it is characterized in that, described method comprises:
The accessed task computation Node B threshold of each frame in distributed type assemblies is set;
Obtain task computation number of nodes accessed in each frame, and judge in frame, whether current accessed task computation number of nodes exceeds described threshold value;
If current accessed task computation number of nodes exceedes described threshold value in described frame, then the new task of distribution is assigned in other frame or described new task is dispatched to task queue and wait for.
2. task scheduling according to claim 1, is characterized in that, described method also comprises:
If current accessed task computation number of nodes exceedes described threshold value in frame, the task of being dispensed to this frame waits for that the accessed task computation number of nodes of this frame performs lower than continuing after described threshold value one preset ratio again.
3. method for scheduling task according to claim 1, is characterized in that, the step arranging the accessed task computation Node B threshold of each frame in distributed type assemblies comprises:
According to the default power consumption of described frame and the default power consumption of each task computation node, the threshold value of the task computation number of nodes that can simultaneously work in computer rack.
4. method for scheduling task according to claim 3, is characterized in that, described method also comprises:
According to the threshold value of the hard disk quantity that the power consumption calculation of each hard disk in the default power consumption of described task computation node and described task computation node can work simultaneously.
5. method for scheduling task according to claim 4, is characterized in that, described method also comprises:
Read task to be allocated in task queue, and calculate the hard disk quantity of described task needs to be allocated activation;
Obtain the hard disk quantity worked in described task computation node;
Judge that the hard disk quantity that working in described task computation node and described task to be allocated need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can simultaneously work in described computing node;
Perform if then described task to be allocated is fallen out and is dispensed in this task computation node, if not then described task to be allocated dormancy Preset Time in described task queue.
6. method for scheduling task according to claim 5, it is characterized in that, also comprise after the step of described task to be allocated dormancy Preset Time in described task queue: again from described task queue, read task to be allocated when the hard disk quantity worked in described task computation node is less than the hard disk quantity that can work in described task computation node simultaneously.
7. method for scheduling task according to claim 5, is characterized in that, in described reading task queue task to be allocated step before also comprise:
Obtain the task that user submits to, and by described task merging to described task queue.
8. based on a distributed task scheduling system, it is characterized in that, described system comprises:
Configuration module, for arranging the accessed task computation Node B threshold of each frame in distributed type assemblies;
Acquisition module, for obtaining task computation number of nodes accessed in each frame,
Analysis module, for judging in described frame, whether current accessed task computation number of nodes exceeds described threshold value;
Distribution module, for when accessed task computation number of nodes current in frame exceeds described threshold value, is then assigned to the new task of distribution in other frame or described new task is dispatched to task queue and wait for.
9. task scheduling system according to claim 8, it is characterized in that, if described distribution module also exceeds described threshold value for accessed task computation number of nodes current in frame, the task of being dispensed to this frame is waited for, and the accessed task computation number of nodes of this frame performs lower than continuing after described threshold value one preset ratio again.
10. task scheduling system according to claim 8, it is characterized in that, described configuration module also for according to the default power consumption of described frame and the default power consumption of each task computation node, the threshold value of the task computation number of nodes that can simultaneously work in computer rack.
11. task scheduling systems according to claim 10, it is characterized in that, the threshold value of hard disk quantity of described configuration module also for can work according to the power consumption calculation of each hard disk in the default power consumption of described task computation node and described task computation node simultaneously.
12. task scheduling systems according to claim 11, it is characterized in that, described acquisition module also for reading task to be allocated in task queue, and calculates the hard disk quantity of described task needs to be allocated activation, and obtains the hard disk quantity worked in described task computation node;
Described analysis module is also for judging that the hard disk quantity that working in described task computation node and described task to be allocated need the hard disk quantity sum activated whether to be less than or equal to the hard disk quantity that can work in described computing node simultaneously;
Described distribution module is used for when the hard disk quantity that working in described task computation node and described task to be allocated need the hard disk quantity sum activated to be less than or equal to the hard disk quantity that can simultaneously work in described computing node, described task to be allocated being fallen out and being dispensed in this task computation node performing, otherwise described task to be allocated dormancy Preset Time in described task queue.
13. task scheduling systems according to claim 12, it is characterized in that, described acquisition module also after dormancy Preset Time, again reads task to be allocated when the hard disk quantity worked in described task computation node is less than the hard disk quantity that can work in described task computation node simultaneously for described task to be allocated in described task queue from described task queue.
14. task scheduling systems according to claim 12, is characterized in that, described system also comprises merging module, the task that the user for getting submits to, and by described task merging to described task queue.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410323841.6A CN105335219A (en) | 2014-07-08 | 2014-07-08 | Distribution-based task scheduling method and system |
| HK16106716.2A HK1218795A1 (en) | 2014-07-08 | 2016-06-12 | Distributed method and system for scheduling tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410323841.6A CN105335219A (en) | 2014-07-08 | 2014-07-08 | Distribution-based task scheduling method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105335219A true CN105335219A (en) | 2016-02-17 |
Family
ID=55285776
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410323841.6A Pending CN105335219A (en) | 2014-07-08 | 2014-07-08 | Distribution-based task scheduling method and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105335219A (en) |
| HK (1) | HK1218795A1 (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106681808A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Task scheduling method and device |
| CN106951456A (en) * | 2017-02-24 | 2017-07-14 | 广东广信通信服务有限公司 | A kind of memory database system and data handling system |
| CN107544251A (en) * | 2017-09-25 | 2018-01-05 | 清华大学 | A kind of minimum based on Robust distributed model always drags the Single Machine Scheduling method of phase |
| CN107623711A (en) * | 2016-07-15 | 2018-01-23 | 北京金山云网络技术有限公司 | The distribution method and device of primary node and secondary node in a kind of cluster |
| CN107679197A (en) * | 2017-10-10 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of optimization method and device of file truncation |
| CN108073349A (en) * | 2016-11-08 | 2018-05-25 | 北京国双科技有限公司 | The transmission method and device of data |
| CN108289086A (en) * | 2017-01-10 | 2018-07-17 | 阿里巴巴集团控股有限公司 | Request processing method and device, server |
| CN109521971A (en) * | 2018-11-26 | 2019-03-26 | 广东小天才科技有限公司 | Cross-machine-room-based data read-write separation method and system |
| CN109669775A (en) * | 2018-12-10 | 2019-04-23 | 平安科技（深圳）有限公司 | Distributed task dispatching method, system and storage medium |
| CN110287025A (en) * | 2019-06-19 | 2019-09-27 | 深圳前海微众银行股份有限公司 | A kind of resource allocation methods, device and equipment |
| WO2019223444A1 (en) * | 2018-05-22 | 2019-11-28 | 杭州海康威视数字技术股份有限公司 | Data storage system |
| CN110650206A (en) * | 2019-09-29 | 2020-01-03 | 北京天融信网络安全技术有限公司 | IO flow control method and device in distributed storage system and storage medium |
| CN110661824A (en) * | 2018-06-28 | 2020-01-07 | 阿里巴巴集团控股有限公司 | Flow regulation and control method and device for servers in distributed cluster |
| CN112114971A (en) * | 2020-09-28 | 2020-12-22 | 中国建设银行股份有限公司 | Task allocation method, device and equipment |
| CN113010287A (en) * | 2021-03-16 | 2021-06-22 | 恩亿科(北京)数据科技有限公司 | Small distributed task processing system, method, electronic device and storage medium |
| CN113723618A (en) * | 2021-08-27 | 2021-11-30 | 南京星环智能科技有限公司 | SHAP optimization method, equipment and medium |
| CN115033370A (en) * | 2022-08-09 | 2022-09-09 | 北京得瑞领新科技有限公司 | Method and device for scheduling flash memory tasks in storage equipment, storage medium and equipment |
| CN115981811A (en) * | 2022-12-19 | 2023-04-18 | 杭州新迪数字工程系统有限公司 | Task scheduling method, system, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090106571A1 (en) * | 2007-10-21 | 2009-04-23 | Anthony Low | Systems and Methods to Adaptively Load Balance User Sessions to Reduce Energy Consumption |
| CN101916209A (en) * | 2010-08-06 | 2010-12-15 | 华东交通大学 | Cluster task resource allocation method for multi-core processor |
| CN102096603A (en) * | 2009-12-14 | 2011-06-15 | 中国移动通信集团公司 | Task decomposition control method in MapReduce system and scheduling node equipment |
| CN102739785A (en) * | 2012-06-20 | 2012-10-17 | 东南大学 | Method for scheduling cloud computing tasks based on network bandwidth estimation |
| CN103019853A (en) * | 2012-11-19 | 2013-04-03 | 北京亿赞普网络技术有限公司 | Method and device for dispatching job task |
| CN103281376A (en) * | 2013-05-31 | 2013-09-04 | 武汉大学 | Method for automatic caching construction of massive timing sequence remote-sensing images in cloud environment |
| US8555014B1 (en) * | 2007-12-27 | 2013-10-08 | Emc Corporation | Automatic access management of clients to a storage system |
-
2014
- 2014-07-08 CN CN201410323841.6A patent/CN105335219A/en active Pending
-
2016
- 2016-06-12 HK HK16106716.2A patent/HK1218795A1/en unknown
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090106571A1 (en) * | 2007-10-21 | 2009-04-23 | Anthony Low | Systems and Methods to Adaptively Load Balance User Sessions to Reduce Energy Consumption |
| US8555014B1 (en) * | 2007-12-27 | 2013-10-08 | Emc Corporation | Automatic access management of clients to a storage system |
| CN102096603A (en) * | 2009-12-14 | 2011-06-15 | 中国移动通信集团公司 | Task decomposition control method in MapReduce system and scheduling node equipment |
| CN101916209A (en) * | 2010-08-06 | 2010-12-15 | 华东交通大学 | Cluster task resource allocation method for multi-core processor |
| CN102739785A (en) * | 2012-06-20 | 2012-10-17 | 东南大学 | Method for scheduling cloud computing tasks based on network bandwidth estimation |
| CN103019853A (en) * | 2012-11-19 | 2013-04-03 | 北京亿赞普网络技术有限公司 | Method and device for dispatching job task |
| CN103281376A (en) * | 2013-05-31 | 2013-09-04 | 武汉大学 | Method for automatic caching construction of massive timing sequence remote-sensing images in cloud environment |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107623711A (en) * | 2016-07-15 | 2018-01-23 | 北京金山云网络技术有限公司 | The distribution method and device of primary node and secondary node in a kind of cluster |
| CN108073349A (en) * | 2016-11-08 | 2018-05-25 | 北京国双科技有限公司 | The transmission method and device of data |
| CN108073349B (en) * | 2016-11-08 | 2021-02-12 | 北京国双科技有限公司 | Data transmission method and device |
| CN106681808A (en) * | 2016-12-01 | 2017-05-17 | 北京奇虎科技有限公司 | Task scheduling method and device |
| CN108289086A (en) * | 2017-01-10 | 2018-07-17 | 阿里巴巴集团控股有限公司 | Request processing method and device, server |
| CN108289086B (en) * | 2017-01-10 | 2020-11-24 | 阿里巴巴集团控股有限公司 | Request processing method and device and server |
| CN106951456A (en) * | 2017-02-24 | 2017-07-14 | 广东广信通信服务有限公司 | A kind of memory database system and data handling system |
| CN107544251A (en) * | 2017-09-25 | 2018-01-05 | 清华大学 | A kind of minimum based on Robust distributed model always drags the Single Machine Scheduling method of phase |
| CN107544251B (en) * | 2017-09-25 | 2020-05-08 | 清华大学 | Single machine scheduling method for minimizing total stall period based on distributed robust model |
| CN107679197A (en) * | 2017-10-10 | 2018-02-09 | 郑州云海信息技术有限公司 | A kind of optimization method and device of file truncation |
| WO2019223444A1 (en) * | 2018-05-22 | 2019-11-28 | 杭州海康威视数字技术股份有限公司 | Data storage system |
| CN110661824A (en) * | 2018-06-28 | 2020-01-07 | 阿里巴巴集团控股有限公司 | Flow regulation and control method and device for servers in distributed cluster |
| CN110661824B (en) * | 2018-06-28 | 2022-04-12 | 阿里巴巴集团控股有限公司 | Flow control method of server in distributed cluster and storage medium |
| CN109521971A (en) * | 2018-11-26 | 2019-03-26 | 广东小天才科技有限公司 | Cross-machine-room-based data read-write separation method and system |
| CN109669775A (en) * | 2018-12-10 | 2019-04-23 | 平安科技（深圳）有限公司 | Distributed task dispatching method, system and storage medium |
| WO2020119029A1 (en) * | 2018-12-10 | 2020-06-18 | 平安科技（深圳）有限公司 | Distributed task scheduling method and system, and storage medium |
| CN110287025A (en) * | 2019-06-19 | 2019-09-27 | 深圳前海微众银行股份有限公司 | A kind of resource allocation methods, device and equipment |
| CN110650206A (en) * | 2019-09-29 | 2020-01-03 | 北京天融信网络安全技术有限公司 | IO flow control method and device in distributed storage system and storage medium |
| CN110650206B (en) * | 2019-09-29 | 2022-03-11 | 北京天融信网络安全技术有限公司 | IO flow control method and device in distributed storage system and storage medium |
| CN112114971A (en) * | 2020-09-28 | 2020-12-22 | 中国建设银行股份有限公司 | Task allocation method, device and equipment |
| CN113010287A (en) * | 2021-03-16 | 2021-06-22 | 恩亿科(北京)数据科技有限公司 | Small distributed task processing system, method, electronic device and storage medium |
| CN113723618A (en) * | 2021-08-27 | 2021-11-30 | 南京星环智能科技有限公司 | SHAP optimization method, equipment and medium |
| CN113723618B (en) * | 2021-08-27 | 2022-11-08 | 南京星环智能科技有限公司 | SHAP optimization method, equipment and medium |
| CN115033370A (en) * | 2022-08-09 | 2022-09-09 | 北京得瑞领新科技有限公司 | Method and device for scheduling flash memory tasks in storage equipment, storage medium and equipment |
| CN115033370B (en) * | 2022-08-09 | 2022-11-18 | 北京得瑞领新科技有限公司 | Method and device for scheduling flash memory tasks in storage equipment, storage medium and equipment |
| CN115981811A (en) * | 2022-12-19 | 2023-04-18 | 杭州新迪数字工程系统有限公司 | Task scheduling method, system, electronic equipment and storage medium |
| CN115981811B (en) * | 2022-12-19 | 2024-03-15 | 上海新迪数字技术有限公司 | Task scheduling method, system, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| HK1218795A1 (en) | 2017-03-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105335219A (en) | Distribution-based task scheduling method and system | |
| US9798642B2 (en) | Method for allocating a server amongst a network of hybrid storage devices | |
| US10031671B2 (en) | Method, apparatus, and system for calculating identification threshold to distinguish cold data and hot data | |
| CN110008149A (en) | Fusion memory part and its operating method | |
| US8645730B2 (en) | Systems and methods to improve power efficiency in hybrid storage clusters | |
| US8626902B2 (en) | Modeling and reducing power consumption in large IT systems | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN104272386A (en) | Reducing power consumption by migration of data within tiered storage system | |
| CN104462432B (en) | Adaptive distributed computing method | |
| CN102508789A (en) | Grading storage method for system | |
| US10592337B1 (en) | Systems and methods for distributing information across distributed storage devices | |
| CN103530388A (en) | Performance improving data processing method in cloud storage system | |
| CN102117248A (en) | Caching system and method for caching data in caching system | |
| CN103442070A (en) | Private cloud storage resource allocation method based on statistical prediction | |
| US8856442B2 (en) | Method for volume management | |
| CN105681052B (en) | A kind of power-economizing method for the storage of data center's distributed document | |
| US10228856B2 (en) | Storage space management in a thin provisioned virtual environment | |
| CN110321331A (en) | The object storage system of storage address is determined using multistage hash function | |
| US9075606B2 (en) | Storage apparatus and method of determining device to be activated | |
| CN101364138B (en) | Power supply management method and device of solid hard disk storage system | |
| US8966293B1 (en) | Method and apparatus for allocation of power consumption of a computer storage system to its tenants | |
| CN104951242A (en) | Method and device for automatically repositioning data among storage arrays | |
| Ganesh | Data center energy management | |
| Li et al. | Distributed heterogeneous storage based on data value | |
| CN103605795A (en) | Metadata-based file storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160217 |
|
| RJ01 | Rejection of invention patent application after publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: WD Ref document number: 1218795 Country of ref document: HK |