CN114356514A - Scheduling method and device of data processing task, storage medium and electronic device - Google Patents

Scheduling method and device of data processing task, storage medium and electronic device Download PDF

Info

Publication number
CN114356514A
CN114356514A CN202111470968.7A CN202111470968A CN114356514A CN 114356514 A CN114356514 A CN 114356514A CN 202111470968 A CN202111470968 A CN 202111470968A CN 114356514 A CN114356514 A CN 114356514A
Authority
CN
China
Prior art keywords
scheduling
data processing
storage unit
task
scheduler
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111470968.7A
Other languages
Chinese (zh)
Inventor
朱兆生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Priority to CN202111470968.7A priority Critical patent/CN114356514A/en
Publication of CN114356514A publication Critical patent/CN114356514A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Power Sources (AREA)

Abstract

The invention discloses a data processing task scheduling method, a data processing task scheduling device, a storage medium and an electronic device. Wherein, the method comprises the following steps: sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring a scheduling task and an operation attribute of a logic storage unit; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit. The invention solves the technical problems of increased scheduling cost and resource waste of an energy-saving filing system in the related art due to the fact that an application program APP can not sense the power-up and power-down scheduling information of a disk.

Description

Scheduling method and device of data processing task, storage medium and electronic device
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for scheduling a data processing task, a storage medium, and an electronic apparatus.
Background
In the related art, for the energy-saving filing system, the purpose of saving energy is usually achieved by periodically powering up and down a part of the disk, and the part is realized by a separate scheduling logic. The disk can be read and written only when being powered on, if the application cannot sense the scheduling logic, the operability cannot be predicted and judged, so that only a dump area system can be introduced to temporarily store data in the dump area for background relocation, and the cost and the resource waste are caused. For example, in a cold data processing scenario, in order to effectively reduce cost, an application in the energy-saving archive system imports data into the energy-saving archive system, the speed of writing cold data into the energy-saving archive system affects the idle cost of the system, and the performance of read-back affects the direct use experience of a user.
For example, in a power-saving filing system based on power-up and power-down, since the application APP cannot sense the power-up and power-down scheduling of the disk, data needs to be placed in a temporary dump area, which causes additional overhead and causes the performance and delay to be constrained by the dump area, resulting in increased cost and wasted resources.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a scheduling method and device of a data processing task, a storage medium and an electronic device, which are used for at least solving the technical problems of increased scheduling cost and resource waste of an energy-saving filing system in the related art due to the fact that an application program APP cannot sense the power-up and power-down scheduling information of a disk.
According to an aspect of the embodiments of the present invention, there is provided a method for scheduling a data processing task, including: sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring a scheduling task and an operation attribute of a logic storage unit; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
According to another aspect of the embodiments of the present invention, there is also provided a scheduling apparatus for data processing tasks, including: a sending module, configured to send a data processing task to a scheduler, where the data processing task at least includes: archiving tasks and read-back tasks; the system comprises an acquisition module, a scheduling module and a processing module, wherein the acquisition module is used for acquiring scheduling tasks and operation attributes of a logical storage unit, and the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; the generating module is used for generating a control instruction based on the scheduling task and the operation attribute; and the scheduling module is used for controlling and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, the apparatus where the computer-readable storage medium is located is controlled to execute any one of the above scheduling methods for data processing tasks.
According to another aspect of the embodiments of the present invention, there is also provided a processor, where the processor is configured to execute a program, where the program executes any one of the above scheduling methods for data processing tasks.
According to another aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including: a processor; and a memory, connected to the processor, for providing instructions to the processor for processing the following processing steps: step 1, sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; step 2, acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; step 3, generating a control instruction based on the scheduling task and the operation attribute; and 4, sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
In an embodiment of the present invention, by sending a data processing task to a scheduler, the data processing task at least includes: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit, and a mode of aggregating data by dividing a disk is achieved, so that an application program can sense the up-down power scheduling information of the disk, thereby realizing the technical effects of improving the scheduling rate of the data processing task, reducing the scheduling cost and wasting resources, and further solving the technical problems that the energy-saving filing system in the related art has increased scheduling cost and wasting resources because the application program APP can not sense the up-down power scheduling information of the disk.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 shows a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing a scheduling method of data processing tasks;
FIG. 2 is a flow chart of a method of scheduling data processing tasks according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an alternative partitioning of disk resources into logical storage units according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a scenario of a scheduling method of data processing tasks according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a data processing task scheduling apparatus according to an embodiment of the present invention;
fig. 6 is a block diagram of a computer terminal according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
First, some terms or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:
JBOD (just a Bunch Of disks): a disk expansion cabinet;
erasure Coding (EC). A data protection method of coding fault tolerance;
datacontainer (dc): a logical storage unit, e.g., a data container, selects a combination of a set of disks from the JBOD that meets certain location constraints, and is also a power-up/down scheduling and storage unit. The data guarantees the reliability in a multi-copy or EC mode in the DC, and the moving cost of the data in the DC is lower than that in the DC;
application-aware scheduling: application aware DC and scheduling logic are supported to maximize resource utilization efficiency;
energy-conserving filing system: and dividing the multiple groups of JBOD disks into multiple groups of DCs according to constraints, and performing power-on and power-off scheduling by taking the DCs as units to achieve the aim of saving energy.
Example 1
In accordance with an embodiment of the present invention, there is provided an embodiment of a method for scheduling data processing tasks, it being noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
The method provided by the embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Fig. 1 shows a hardware configuration block diagram of a computer terminal (or mobile device) for implementing a scheduling method of data processing tasks. As shown in fig. 1, the computer terminal 10 (or mobile device 10) may include one or more (shown as 102a, 102b, … …, 102 n) processors 102 (the processors 102 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a Universal Serial BUS (USB) port (which may be included as one of the ports of the BUS), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration and is not intended to limit the structure of the electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
It should be noted that the one or more processors 102 and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuit may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computer terminal 10 (or mobile device). As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).
The memory 104 may be used to store software programs and modules of application software, such as program instructions/data storage devices corresponding to the scheduling method of data processing tasks in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the scheduling of data processing tasks is implemented. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).
In the foregoing operating environment, the present invention provides a method for scheduling data processing tasks as shown in fig. 2, and fig. 2 is a flowchart of a method for scheduling data processing tasks according to an embodiment of the present invention, and as shown in fig. 2, the method includes:
step S202, sending a data processing task to a scheduler, wherein the data processing task at least includes: archiving tasks and read-back tasks;
step S204, acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets;
step S206, generating a control instruction based on the scheduling task and the operation attribute;
step S208 is to send a control command to the scheduler, where the control command is used to control the scheduler to process the data processing task by scheduling the logical storage unit.
Optionally, the scheduling method for the data processing task provided in the embodiment of the present application may be, but is not limited to, applied to an archive storage scenario, for example, an energy-saving archive system and a cold archive storage system, and as an optional embodiment, the method may be implemented in the following products, for example: deep archive storage series products, cloud archive storage products, search engine archive storage products, and the like.
In an alternative embodiment, for example, the method is applied to an energy-saving archive system, and the energy-saving archive system uses the logical storage unit (DC, e.g., data container) as a scheduling unit, and achieves the energy-saving goal by periodically powering on and powering off a predetermined portion of all the disk resources.
Optionally, as shown in fig. 3, a data processing task is sent to the scheduler by a client of the energy-saving archiving system, where the data processing task at least includes: archiving tasks and read-back tasks; wherein, the archiving task is a processing task of archiving data, and the read-back task is a processing task of reading back data, for example, as shown in fig. 3, DC 0: reading a file set and writing the file set; DC 1: reading a file set and writing the file set.
Optionally, the scheduler is a DC scheduler shown in fig. 3, as shown in fig. 3, the multiple logical storage units DC0, DC1, DC2, DC5, DC7, and the like are obtained by pre-dividing disk resources in multiple disk expansion cabinets, and the scheduling task includes: and powering on and off a planning task, wherein the operational attributes comprise at least one of: as shown in fig. 3, when scheduling is performed according to a top-bottom scheduling task, it may be, but is not limited to, as schematically shown in fig. 3, that the operation attribute corresponding to the logical storage unit DC0 or DC1 is a read attribute, the operation attribute corresponding to the logical storage unit DC5 or DC7 is a write attribute, and the attribute corresponding to the logical storage unit DC2 is an internal repair attribute.
As an alternative embodiment, in order to improve the processing performance of write-in and read-back between end-to-end, and eliminate the overhead of the dump area (i.e. there is no need to introduce a temporary dump area for reading and writing data, and the archive background logic does not need to move user data out of or into the dump area), due to the particularity of the energy-saving archive system, the application program may obtain an archive or read-back plan in advance and send it to the archive backend scheduler, so as to convert the archive and read-back into a DC read-write operation task with priority. And the DC scheduler acquires and combines DC read-write operation tasks sent by all the clients and schedules the DC according to the priority. The scheduling plan is determined within a certain time, the scheduling result is the DC of the next power-on cycle, the DC is divided into 3 types, and the DC is applied with reading, writing and internal repairing, and is updated to a DC state table. The application program can periodically acquire the scheduling plan from the DC state table, because the scheduling plan is determined, the application program can schedule the subsequent power-on time workload according to the scheduling plan, such as resource allocation and scheduling for archiving and readback, and so on. In addition, the data needing to be read back simultaneously in the future are applied to be written into the same DC in an aggregation mode. By sensing DC and scheduling planning, the application can achieve the purpose of directly and efficiently using hard disk resources.
The embodiment of the application provides a scheduling method of a data processing task, which comprises a new disk resource dividing method, wherein disk resources are divided into logic storage units, namely scheduling and storage units (DCs), so that application senses DC information and scheduling plans, a scheduling system can optimize the resource use efficiency, the application can cooperate with scheduling to achieve the optimal use of resources, the application can aggregate data, and the associated data are written in the same DC, so that the small-scale read-back lower delay can be reduced.
In an embodiment of the present invention, by sending a data processing task to a scheduler, the data processing task at least includes: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit, and a mode of aggregating data by dividing a disk is achieved, so that an application program can sense the up-down power scheduling information of the disk, thereby realizing the technical effects of improving the scheduling rate of the data processing task, reducing the scheduling cost and wasting resources, and further solving the technical problems that the energy-saving filing system in the related art has increased scheduling cost and wasting resources because the application program APP can not sense the up-down power scheduling information of the disk.
In an optional embodiment, before sending the data processing task to the scheduler, the method further includes:
step S302, dividing the disk resources in a plurality of disk expansion cabinets into a plurality of logical storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving filing system.
As shown in fig. 3, since the plurality of disk expansion cabinets are distributed in the rack server under the plurality of switches of the energy-saving filing system, before sending the data processing task to the scheduler, the disk resources in the plurality of disk expansion cabinets may be divided into a plurality of above-mentioned logical storage units, for example, logical storage unit 1, logical storage unit 2, and logical storage unit 3, and so on.
A selectable partitioning process for partitioning the disk resources in the multiple disk expansion cabinets into the multiple logical storage units may be:
diskj: the jth disk;
rackja jth rack server;
dckthe kth DC;
repeat
selecting out the rack under the same switch from the rest of the racksj0And rackj1
repeat
Rack from the same switchj0And rackj1In all JBODs, disk resources which are not selected are left;
are added to dc in sequencekUntil the data security requirements cannot be met;
Until dckit comprises n blocks of magnetic disks, and the magnetic disk,
until adds some DC to all of the rack disks.
In an optional embodiment, the dividing the disk resources in the plurality of disk expansion cabinets into a plurality of logical storage units includes:
step S402, dividing the disk resource into a plurality of logical storage units according to preset constraints.
Optionally, the preset constraint includes at least one of: the disk management system comprises a first constraint, a second constraint and a third constraint, wherein the first constraint is used for describing that disk resources under the same logical storage unit correspond to a predetermined number of rack servers, the second constraint is used for describing that the number of the disk resources contained in each logical storage unit is multiple, and the third constraint is used for describing that each logical storage unit selects the disk resources in each disk expansion cabinet.
As an alternative embodiment, a Rack server Rack with a plurality of disk expansion cabinets JBOD distributed under a plurality of switches in the archive cluster can be divided into DC groups according to the following constraints: a first constraint, namely 2 Rack servers Rack with disks below the same DC located under the same switch, optionally, the first constraint is to reduce the cost of data movement inside the DC; the second constraint is that the number of disks contained in each DC is n, so that enough DC number in the cluster can be scheduled; and the third constraint is that each DC only selects a few disks in each JBOD, so that the data security of JBOD maintenance scenes is ensured.
In addition, it should be noted that after the disk resources are divided into DC groups, the background scheduler may periodically power up and power down the unit according to DC, and provide an interface to expose the scheduling plan to the application program.
In an alternative embodiment, obtaining the scheduling task and the operation attribute of the logical storage unit includes:
step S502, obtaining the power-up and power-down planning task corresponding to each of the plurality of logic storage units, and obtaining the operation attribute corresponding to each of the plurality of logic storage units, where the power-up and power-down planning task is used to plan a next power-up time interval of the logic storage unit.
In an optional embodiment, controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task according to the scheduling task and the operation attribute includes:
step S602, obtaining the next power-on time interval of the logic storage unit determined by the power-on and power-off planning task;
step S604, correspondingly controlling the scheduler to execute scheduling of the logical storage unit to process the data processing task according to the next power-on time interval of the logical storage unit and the operation attribute corresponding to each of the logical storage units.
As an optional embodiment, the power-up and power-down planning task corresponding to each of the plurality of logic storage units is obtained, the operation attribute corresponding to each of the logic storage units is obtained, and the scheduler is correspondingly controlled to execute scheduling of the logic storage unit to process the data processing task according to a next power-up time interval of the logic storage unit and the operation attribute corresponding to each of the logic storage units; as shown in fig. 4, the content of the data processing task includes: and applying reading processing, applying writing processing and internal repair processing of the logic storage unit. It should be noted that, since the scheduling plan is determined within a certain time, the scheduling result is the DC of the next power-on cycle, which can be roughly divided into 3 types, i.e., read, write, and DC internal repair.
In addition, as also shown in fig. 4, the application program may also periodically obtain the scheduler plan, and since the scheduler plan is determined, the application program may schedule the load of the next power-on time according to the scheduler plan, for example, resource allocation and scheduling for archiving and readback, and so on. In addition, the application program can be aggregated and written into the same DC for the data needing to be read back simultaneously in the future.
In an optional embodiment, controlling the scheduler to perform scheduling of the logical storage unit to process the data processing task includes:
step S702, determining the priority of the data processing task;
step S704, scheduling the logical storage unit to process the data processing task according to the priority.
In the embodiment of the present application, the scheduler is controlled to execute scheduling of the logical storage unit to process the data processing task, that is, the scheduler is controlled to schedule the logical storage unit to process the data processing task according to the determined priority of the data processing task.
By the embodiment of the application, the concept of the scheduling and storage unit is introduced, and the scheduling plan of the scheduling and storage unit is exposed to the application. The application can periodically acquire the dispatching plan of the DC and perform IO planning according to the operation attribute of the DC so as to achieve the effect of direct and efficient use. Also, a scheduling and storage unit concept is introduced and exposes DC packets and scheduling plans to applications. The temporary unloading area is avoided, and the application can use the disk resources more transparently and efficiently under the condition of power-on and power-off through sensing DC and scheduling.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
Example 2
According to an embodiment of the present invention, there is further provided an embodiment of an apparatus for implementing the scheduling method for data processing tasks, fig. 5 is a schematic structural diagram of a scheduling apparatus for data processing tasks according to an embodiment of the present invention, and as shown in fig. 5, the apparatus includes: a sending module 500, an obtaining module 502, a generating module 504 and a scheduling module 506, wherein:
a sending module 500, configured to send a data processing task to a scheduler, where the data processing task at least includes: archiving tasks and read-back tasks; an obtaining module 502, configured to obtain a scheduling task and an operation attribute of a logical storage unit, where the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; a generating module 504, configured to generate a control instruction based on the scheduling task and the operation attribute; a scheduling module 506, configured to control sending of a control instruction to the scheduler, where the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit.
It should be noted here that the sending module 500, the obtaining module 502, the generating module 504, and the scheduling module 506 correspond to steps S202 to S208 in embodiment 1, and the four modules are the same as the corresponding steps in the implementation example and the application scenario, but are not limited to the disclosure in embodiment 1. It should be noted that the above modules may be operated in the computer terminal 10 provided in embodiment 1 as a part of the apparatus.
In this embodiment of the present invention, the sending module 500 is configured to send a data processing task to a scheduler, where the data processing task at least includes: archiving tasks and read-back tasks; an obtaining module 502, configured to obtain a scheduling task and an operation attribute of a logical storage unit, where the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; a generating module 504, configured to generate a control instruction based on the scheduling task and the operation attribute; the scheduling module 506 is configured to control sending of a control instruction to the scheduler, where the control instruction is used to control the scheduler to process the data processing task by scheduling the logical storage unit, so as to achieve a manner of aggregating data by dividing a disk, and enable an application to sense the power-up and power-down scheduling information of the disk, thereby achieving a scheduling rate of the data processing task, and reducing scheduling cost and resource waste, and further solving technical problems of increased scheduling cost and resource waste of an energy-saving filing system in the related art due to the fact that the application APP cannot sense the power-up and power-down scheduling information of the disk.
It should be noted that, reference may be made to the relevant description in embodiment 1 for a preferred implementation of this embodiment, and details are not described here again.
Example 3
According to an embodiment of the present invention, there is further provided an embodiment of an electronic apparatus, which may be any one of computing devices in a computing device group. The electronic device includes: a processor and a memory, wherein:
a processor; and a memory, connected to the processor, for providing instructions to the processor for processing the following processing steps: step 1, sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; step 2, acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; step 3, generating a control instruction based on the scheduling task and the operation attribute; and 4, sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
In an embodiment of the present invention, by sending a data processing task to a scheduler, the data processing task at least includes: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit, and a mode of aggregating data by dividing a disk is achieved, so that an application program can sense the up-down power scheduling information of the disk, thereby realizing the technical effects of improving the scheduling rate of the data processing task, reducing the scheduling cost and wasting resources, and further solving the technical problems that the energy-saving filing system in the related art has increased scheduling cost and wasting resources because the application program APP can not sense the up-down power scheduling information of the disk.
It should be noted that, reference may be made to the relevant description in embodiment 1 for a preferred implementation of this embodiment, and details are not described here again.
Example 4
According to an embodiment of the present invention, there may be provided an embodiment of a computer terminal, which may be any one computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced with a terminal device such as a mobile terminal.
Optionally, in this embodiment, the computer terminal may be located in at least one network device of a plurality of network devices of a computer network.
In this embodiment, the computer terminal may execute the program code of the following steps in the scheduling method of the data processing task: sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
Alternatively, fig. 6 is a block diagram of a computer terminal according to an embodiment of the present invention. As shown in fig. 6, the computer terminal may include: one or more processors 602 (only one of which is shown), memory 604, and a peripherals interface 606.
The memory may be configured to store software programs and modules, such as program instructions/modules corresponding to the scheduling method and apparatus for data processing tasks in the embodiments of the present invention, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the scheduling method for data processing tasks is implemented. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the computer terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor can call the information and application program stored in the memory through the transmission device to execute the following steps: sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
Optionally, the processor may further execute the program code of the following steps: and dividing the disk resources in a plurality of disk expansion cabinets into a plurality of logic storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving filing system.
Optionally, the processor may further execute the program code of the following steps: dividing the disk resources into a plurality of logical storage units according to preset constraints, wherein the preset constraints include at least one of the following: the method comprises the steps of a first constraint, a second constraint and a third constraint, wherein the first constraint is used for describing that the disk resources under the same logic storage unit correspond to the same switch; the second constraint is used for describing that the disk resources under the same logical storage unit correspond to a predetermined number of the rack servers, and the third constraint is used for describing that the number of the disks contained in each logical storage unit is multiple.
Optionally, the processor may further execute the program code of the following steps: and obtaining the power-on and power-off planning task corresponding to each of the plurality of logic storage units, and obtaining the operation attribute corresponding to each of the plurality of logic storage units, wherein the power-on and power-off planning task is used for planning a next power-on time interval of the logic storage unit.
Optionally, the processor may further execute the program code of the following steps: acquiring a next power-on time interval of the logic storage unit determined by the power-on and power-off planning task; correspondingly controlling the scheduler to execute scheduling of the logic storage unit to process the data processing task according to the next power-on time interval of the logic storage unit and the operation attribute corresponding to each logic storage unit, where the content of the data processing task includes: and applying reading processing, applying writing processing and internal repair processing of the logic storage unit.
Optionally, the processor may further execute the program code of the following steps: determining the priority of the data processing task; and scheduling the logic storage unit to process the data processing task according to the priority.
The embodiment of the invention provides a scheme for scheduling data processing tasks. Sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit, so that a mode of aggregating data by dividing a disk is achieved, and an application program can sense the upper and lower electric power scheduling information of the disk, thereby realizing the scheduling rate of the data processing task, reducing the scheduling cost and the technical effect of resource waste, and further solving the technical problems that the energy-saving filing system in the related art has increased scheduling cost and resource waste because the application program APP can not sense the upper and lower electric power scheduling information of the disk.
It can be understood by those skilled in the art that the structure shown in fig. 6 is only an illustration, and the computer terminal may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 6 is a diagram illustrating a structure of the electronic device. For example, the computer terminal may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 6, or have a different configuration than shown in FIG. 6.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the computer-readable storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
Example 5
Embodiments of a computer-readable storage medium are also provided according to embodiments of the present invention. Optionally, in this embodiment, the computer-readable storage medium may be configured to store the program code executed by the scheduling method for data processing tasks provided in embodiment 1.
Optionally, in this embodiment, the computer-readable storage medium may be located in any one of a group of computer terminals in a computer network, or in any one of a group of mobile terminals.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks; acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets; generating a control instruction based on the scheduling task and the operation attribute; and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: and dividing the disk resources in a plurality of disk expansion cabinets into a plurality of logic storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving filing system.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: dividing the disk resources into a plurality of logical storage units according to preset constraints, wherein the preset constraints include at least one of the following: the method comprises the steps of a first constraint, a second constraint and a third constraint, wherein the first constraint is used for describing that the disk resources under the same logic storage unit correspond to the same switch; the second constraint is used for describing that the disk resources under the same logical storage unit correspond to a predetermined number of the rack servers, and the third constraint is used for describing that the number of the disks contained in each logical storage unit is multiple.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: and obtaining the power-on and power-off planning task corresponding to each of the plurality of logic storage units, and obtaining the operation attribute corresponding to each of the plurality of logic storage units, wherein the power-on and power-off planning task is used for planning a next power-on time interval of the logic storage unit.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: acquiring a next power-on time interval of the logic storage unit determined by the power-on and power-off planning task; correspondingly controlling the scheduler to execute scheduling of the logic storage unit to process the data processing task according to the next power-on time interval of the logic storage unit and the operation attribute corresponding to each logic storage unit, where the content of the data processing task includes: and applying reading processing, applying writing processing and internal repair processing of the logic storage unit.
Optionally, in this embodiment, the computer readable storage medium is configured to store program code for performing the following steps: determining the priority of the data processing task; and scheduling the logic storage unit to process the data processing task according to the priority.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or a predetermined number of units or more may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned computer-readable storage media comprise: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims (11)

1. A method for scheduling data processing tasks, comprising:
sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks;
acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets;
generating a control instruction based on the scheduling task and the operation attribute;
sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
2. The method according to claim 1, wherein the method is applied to an energy-saving archiving system, and the energy-saving archiving system achieves an energy-saving goal by using the logical storage units as scheduling units and periodically powering on and powering off a predetermined part of the logical storage units in all the disk resources.
3. The method of claim 1, wherein prior to sending the data processing task to the scheduler, the method further comprises:
and dividing the disk resources in the plurality of disk expansion cabinets into a plurality of logic storage units, wherein the plurality of disk expansion cabinets are distributed in rack servers under a plurality of switches of the energy-saving filing system.
4. The method of claim 3, wherein the partitioning of disk resources in a plurality of disk expansion cabinets to a plurality of the logical storage units comprises:
dividing the disk resources into a plurality of logical storage units according to preset constraints, wherein the preset constraints comprise at least one of the following: the disk management system comprises a first constraint, a second constraint and a third constraint, wherein the first constraint is used for describing that disk resources under the same logical storage unit correspond to a predetermined number of rack servers, the second constraint is used for describing that a plurality of disk resources are contained in each logical storage unit, and the third constraint is used for describing that each logical storage unit selects disk resources in each disk expansion cabinet.
5. The method of claim 1, wherein the scheduling a task comprises: a power up and down planning task, the operational attributes including at least one of: reading attribute, writing attribute and internal repair attribute, and acquiring scheduling task and operation attribute of the logic storage unit, wherein the method comprises the following steps:
and acquiring the power-on and power-off planning task corresponding to each logic storage unit in the plurality of logic storage units, and acquiring the operation attribute corresponding to each logic storage unit, wherein the power-on and power-off planning task is used for planning the next power-on time interval of the logic storage unit.
6. The method of claim 1, wherein the scheduling a task comprises: a power up and down planning task, the operational attributes including at least one of: reading attribute, writing attribute, internal repair attribute, controlling the scheduler to execute and schedule the logic storage unit to process the data processing task according to the scheduling task and the operation attribute, including:
acquiring the next power-on time interval of the logic storage unit determined by the power-on and power-off planning task;
correspondingly controlling the scheduler to execute scheduling of the logic storage unit to process the data processing task according to the next power-on time interval of the logic storage unit and the operation attribute corresponding to each logic storage unit, wherein the content of the data processing task comprises: applying a read process, applying a write process, and applying an internal repair process to the logical storage unit.
7. The method of claim 1, wherein controlling the scheduler to perform scheduling of the logical storage unit to process the data processing task comprises:
determining a priority of the data processing task;
and scheduling the logic storage unit to process the data processing task according to the priority of the data processing task.
8. A scheduling apparatus of data processing tasks, comprising:
a sending module, configured to send a data processing task to a scheduler, where the data processing task at least includes: archiving tasks and read-back tasks;
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring scheduling tasks and operation attributes of a logical storage unit, and the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets;
the generating module is used for generating a control instruction based on the scheduling task and the operation attribute;
and the scheduling module is used for controlling and sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
9. A computer-readable storage medium comprising a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method of scheduling data processing tasks of any of claims 1 to 7.
10. A processor for running a program, wherein the program when running performs the method of scheduling data processing tasks of any of claims 1 to 7.
11. An electronic device, comprising:
a processor; and
a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:
step 1, sending a data processing task to a scheduler, wherein the data processing task at least comprises: archiving tasks and read-back tasks;
step 2, acquiring scheduling tasks and operation attributes of a logical storage unit, wherein the logical storage unit is obtained by pre-dividing disk resources in a plurality of disk expansion cabinets;
step 3, generating a control instruction based on the scheduling task and the operation attribute;
and 4, sending a control instruction to the scheduler, wherein the control instruction is used for controlling the scheduler to process the data processing task by scheduling the logic storage unit.
CN202111470968.7A 2021-12-03 2021-12-03 Scheduling method and device of data processing task, storage medium and electronic device Pending CN114356514A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111470968.7A CN114356514A (en) 2021-12-03 2021-12-03 Scheduling method and device of data processing task, storage medium and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111470968.7A CN114356514A (en) 2021-12-03 2021-12-03 Scheduling method and device of data processing task, storage medium and electronic device

Publications (1)

Publication Number Publication Date
CN114356514A true CN114356514A (en) 2022-04-15

Family

ID=81097066

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111470968.7A Pending CN114356514A (en) 2021-12-03 2021-12-03 Scheduling method and device of data processing task, storage medium and electronic device

Country Status (1)

Country Link
CN (1) CN114356514A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020144057A1 (en) * 2001-01-30 2002-10-03 Data Domain Archival data storage system and method
US20060053250A1 (en) * 2004-09-03 2006-03-09 Nobuyuki Saze Storage system and data management device for storage system
US20070079087A1 (en) * 2005-09-29 2007-04-05 Copan Systems, Inc. User interface for archival storage of data
US20070079086A1 (en) * 2005-09-29 2007-04-05 Copan Systems, Inc. System for archival storage of data
US7734867B1 (en) * 2002-05-17 2010-06-08 Hewlett-Packard Development Company, L.P. Data storage using disk drives in accordance with a schedule of operations
KR20160004161A (en) * 2014-07-02 2016-01-12 삼성전자주식회사 Method for task scheduling and Electronic device using the same
US20160092109A1 (en) * 2014-09-30 2016-03-31 Emc Corporation Performance of de-clustered disk array
CN108255431A (en) * 2018-01-11 2018-07-06 中国人民解放军国防科技大学 Low-power-consumption filing and analyzing system based on strategy and capable of achieving unified management
CN113448939A (en) * 2020-03-24 2021-09-28 北京有限元科技有限公司 Data archiving and storing method and device and storage medium

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020144057A1 (en) * 2001-01-30 2002-10-03 Data Domain Archival data storage system and method
US7734867B1 (en) * 2002-05-17 2010-06-08 Hewlett-Packard Development Company, L.P. Data storage using disk drives in accordance with a schedule of operations
US20060053250A1 (en) * 2004-09-03 2006-03-09 Nobuyuki Saze Storage system and data management device for storage system
US20070079087A1 (en) * 2005-09-29 2007-04-05 Copan Systems, Inc. User interface for archival storage of data
US20070079086A1 (en) * 2005-09-29 2007-04-05 Copan Systems, Inc. System for archival storage of data
KR20160004161A (en) * 2014-07-02 2016-01-12 삼성전자주식회사 Method for task scheduling and Electronic device using the same
US20160092109A1 (en) * 2014-09-30 2016-03-31 Emc Corporation Performance of de-clustered disk array
CN108255431A (en) * 2018-01-11 2018-07-06 中国人民解放军国防科技大学 Low-power-consumption filing and analyzing system based on strategy and capable of achieving unified management
CN113448939A (en) * 2020-03-24 2021-09-28 北京有限元科技有限公司 Data archiving and storing method and device and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PABLO LLOPIS 等: "Survey of Energy-Efficient and Power-Proportional Storage Systems", 《THE COMPUTER JOURNAL》, vol. 57, no. 7, 11 June 2013 (2013-06-11), pages 1017 - 1032 *
刘公仆: "基于SAN存储系统管理模块的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 03, 15 March 2011 (2011-03-15), pages 137 - 48 *

Similar Documents

Publication Publication Date Title
CN104239132B (en) A kind of wake up the method for alignment, device and terminal up
CN104885055A (en) Method and device for application data synchronization
CN102708064A (en) Multi-phase resume from hibernate
CN103793238A (en) Rack and method for simultaneously updating multiple basic input output systems thereof
CN102981990A (en) Data transmission method and device of terminal equipment with Androild platform
CN104050248A (en) File storage system and storage method
CN103399781A (en) Cloud server and virtual machine management method thereof
CN102870374B (en) Load-sharing method and apparatus, and veneer,
CN112506427A (en) Method, system, equipment and medium for reducing power consumption of solid state disk
CN111984557A (en) Data processing method, device and system
CN110908485A (en) Method and system for dynamically allocating and optimizing power resources and machine readable medium
US20070022218A1 (en) Network-attached storage device having a connection to a local user device
CN104919425A (en) Systems and methods for device-to-cloud message delivery
US10965148B1 (en) Datacenter backup power management
CN107209712A (en) Storage system, storage device and disk scheduling method
CN104079610A (en) Cloud server, and method and system for cloud synchronization of application software
CN101853185A (en) Blade server and service dispatching method thereof
CN114356514A (en) Scheduling method and device of data processing task, storage medium and electronic device
CN103375420A (en) Equipment cabinet system and fan control system and control method thereof
CN102681650B (en) The storage system of a kind of Energy control power-economizing method and correspondence thereof
CN106843890A (en) Sensor network, node and its operation method based on intelligent decision
CN111158595A (en) Enterprise-level heterogeneous storage resource scheduling method and system
CN110650546A (en) File transmission method, device, storage medium and terminal
CN107590087B (en) Electronic equipment and hard disk access method of electronic equipment
CN103209084B (en) The method of uninterrupted power supply and control power distribution unit thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination