CN113127158B - Method and device for executing data processing task - Google Patents
Method and device for executing data processing task Download PDFInfo
- Publication number
- CN113127158B CN113127158B CN201911394801.XA CN201911394801A CN113127158B CN 113127158 B CN113127158 B CN 113127158B CN 201911394801 A CN201911394801 A CN 201911394801A CN 113127158 B CN113127158 B CN 113127158B
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- executor
- data processing
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Abstract
The embodiment of the disclosure discloses a method and a device for executing a data processing task, and relates to the technical field of cloud computing. One embodiment of the method comprises the following steps: responding to the detection that the unexecuted task exists in the database, acquiring the task to be processed, and configuring a current executor identifier for the task to be processed; executing the task to be processed, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp, and the time stamp is used for identifying monitoring time when the task to be processed is executed; and sending task completion information to the database in response to executing the task to be processed. According to the method, the running state feedback information of the task to be processed is directly stored in the database, so that the running state of the task to be processed is monitored in time, and the execution efficiency of the task to be processed is improved.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of cloud computing, in particular to a method and a device for executing a data processing task.
Background
A job engine is a data processing system that allows a user to define jobs, initiate jobs, and query the results of job execution. Common job engines typically include a database, a task dispatcher, and a plurality of task executors. The database is used for storing tasks to be processed; the task distributor is used for calling the task to be processed from the database and distributing the task to be processed to the corresponding task executor.
When a single task executor fails (such as a network interrupt or restart), the task on the failed task executor is rescheduled to other task executors for execution by the task distributor.
Disclosure of Invention
The embodiment of the disclosure provides a method and a device for executing a data processing task.
In a first aspect, embodiments of the present disclosure provide a method for performing a data processing task, the method comprising: responding to the detection that the unexecuted task exists in the database, acquiring the task to be processed, and configuring a current executor identifier for the task to be processed; executing the task to be processed, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp, and the time stamp is used for identifying monitoring time when the task to be processed is executed; and sending task completion information to the database in response to executing the task to be processed.
In some embodiments, the database includes task expiration times corresponding to the tasks to be processed, wherein the task expiration times are used to characterize a time required to perform the tasks to be processed, and the performing the tasks to be processed includes: and distributing a first data processing thread for the task to be processed according to the task expiration time and the data processing capacity of the task to be processed.
In some embodiments, the operation state feedback information further includes task progress information, and the task progress information includes at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed and the process position label of the task to be processed.
In some embodiments, the above method comprises: deleting the task to be processed in response to the fact that the current executor of the task to be processed is identified as other executors; and monitoring task expiration time and running state feedback information of other tasks to be processed in the database.
In a second aspect, embodiments of the present disclosure provide a method for performing a data processing task, the method comprising: monitoring task expiration time and running state feedback information of a task to be processed in a database, wherein the task expiration time is used for representing time required for identifying execution completion of the task to be processed, the running state feedback information comprises a time stamp, and the time stamp is used for identifying monitoring time when the task to be processed is executed; and responding to the task expiration time later than the time stamp, continuously monitoring the task expiration time and running state feedback information of the task to be processed, and otherwise, executing the task to be processed.
In some embodiments, the task to be processed is marked with a corresponding current executor identifier, and the executing the task to be processed includes: and modifying the current executor identifier into a local executor identifier.
In some embodiments, the operation state feedback information further includes task progress information, and the task progress information includes at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed, the process position label of the task to be processed, and the executing of the task to be processed, including: determining the residual data processing capacity of the task to be processed according to the task progress information; and distributing a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount.
In a third aspect, embodiments of the present disclosure provide an apparatus for performing data processing tasks, the apparatus comprising: the task acquisition unit is used for responding to the fact that the unexecuted task to be processed exists in the database, and is configured to acquire the task to be processed and configure a current executor identifier for the task to be processed; the first execution unit is configured to execute the task to be processed, and send running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp, and the time stamp is used for identifying monitoring time when the task to be processed is executed; and a completion information transmitting unit configured to transmit task completion information to the database in response to execution of completion of the task to be processed.
In some embodiments, the database includes task expiration times corresponding to the tasks to be processed, wherein the task expiration times are used to characterize a time required to execute the tasks to be processed, and the first execution unit includes: and the first execution subunit is configured to allocate a first data processing thread for the task to be processed according to the task expiration time and the data processing amount of the task to be processed.
In some embodiments, the operation state feedback information further includes task progress information, and the task progress information includes at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed and the process position label of the task to be processed.
In some embodiments, the apparatus comprises: a task deleting unit configured to delete the task to be processed in response to the identification of the current executor of the task to be processed as other executors; and the first task monitoring unit is configured to monitor task expiration time and running state feedback information of other tasks to be processed in the database.
In a fourth aspect, embodiments of the present disclosure provide an apparatus for performing data processing tasks, the apparatus comprising: a second task monitoring unit configured to monitor task expiration time of a task to be processed in a database and running state feedback information, wherein the task expiration time is used for representing and identifying time required for executing the task to be processed, the running state feedback information comprises a time stamp, and the time stamp is used for identifying the monitoring time when the task to be processed is executed; and the second execution unit is configured to respond to the task expiration time later than the time stamp, continuously monitor the task expiration time and the running state feedback information of the task to be processed, and otherwise, execute the task to be processed.
In some embodiments, the task to be processed is marked with a corresponding current executor identifier, and the second execution unit includes: an identifier modification subunit configured to modify the current actuator identifier to a local actuator identifier.
In some embodiments, the operation state feedback information further includes task progress information, and the task progress information includes at least one of: the process information of the task to be processed, the current instruction of the task to be processed, the process position label of the task to be processed, and the second execution unit includes: a remaining data processing amount determination subunit configured to determine a remaining data processing amount of the task to be processed according to the task progress information; and the second execution subunit is configured to allocate a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount.
In a fifth aspect, embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a memory having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to perform the method for performing data processing tasks of the first aspect or the method for performing data processing tasks of the second aspect.
In a sixth aspect, embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method for performing data processing tasks of the first aspect or the method for performing data processing tasks of the second aspect described above.
The method and the device for executing the data processing task, provided by the embodiment of the disclosure, acquire the task to be processed after detecting that the task to be processed which is not executed exists in the database, and configure a current executor identifier for the task to be processed; then executing the task to be processed, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time; and finally, after the task to be processed is executed, sending task completion information to the database. According to the method and the device, the running state feedback information of the task to be processed is directly stored in the database, so that the execution state of the task to be processed is monitored in time, and the execution efficiency of the task to be processed is improved.
Drawings
Other features, objects and advantages of the present disclosure will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;
FIG. 2 is a flow chart of one embodiment of a method for performing data processing tasks according to the present disclosure;
FIG. 3 is a flow chart of yet another embodiment of a method for performing data processing tasks according to the present disclosure;
FIG. 4a is a schematic diagram of an application scenario corresponding to the method of FIG. 2 for performing data processing tasks;
FIG. 4b is a schematic illustration of an application scenario corresponding to the method for performing data processing tasks of FIG. 3;
FIG. 5 is a schematic structural view of one embodiment of an apparatus for performing data processing tasks according to the present disclosure;
FIG. 6 is a schematic structural view of another embodiment of an apparatus for performing data processing tasks according to the present disclosure;
fig. 7 is a schematic diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, without conflict, the embodiments of the present disclosure and features of the embodiments may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
FIG. 1 illustrates an exemplary system architecture 100 of a method for performing data processing tasks or an apparatus for performing data processing tasks to which embodiments of the present disclosure may be applied.
As shown in fig. 1, system architecture 100 may include an active task executor group 101, a standby task executor group 102, a network 103, and a database 104. Network 103 is the medium used to provide communication links between active task executor group 101, standby task executor group 102, and database 104. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
Active task executors within the active task executor group 101 may interact with the database 104 through the network 103, obtain and execute tasks to be processed from the database 104, and the like. The standby task executors in the standby task executor group 102 may interact with the database 104 through the network 103, and monitor the status of the tasks to be processed in the database 104. And when the task to be processed is not executed at the expiration time of the task, replacing the currently active task executor of the task to be processed to execute the task to be processed.
The active task executor group 101 and the standby task executor group 102 may be hardware or software. When the active task executor group 101 and the standby task executor group 102 are hardware, they may be various electronic devices having a display screen and supporting data processing, including but not limited to smart phones, tablet computers, laptop and desktop computers, and the like. When the active task executor group 101 and the standby task executor group 102 are software, they can be installed in the above-listed electronic devices. Which may be implemented as multiple software or software modules (e.g., to provide distributed services), or as a single software or software module, without limitation.
It should be noted that, the method for executing the data processing task provided by the embodiments of the present disclosure may be executed by the active task executor alone, or may also be executed by the active task executor and the standby task executor together. Generally, a method for executing data processing tasks is performed by an active task executor. Accordingly, the means for performing data processing tasks may be provided in the active task executor or in the standby task executor.
The database 104 may be hardware or software. When the database 104 is hardware, it may be implemented as a distributed server cluster formed by a plurality of servers, or as a single server. When database 104 is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module, without limitation.
It should be understood that the number of active task executors, standby task executors, networks, and databases in FIG. 1 are merely illustrative. There may be any number of active task executors, standby task executors, networks, and databases, as desired for an implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method for performing data processing tasks according to the present disclosure is shown. The method for performing data processing tasks comprises the steps of:
In this embodiment, an execution body of the method for executing a data processing task (for example, an active task executor in the active task executor group 101 shown in fig. 1) may monitor a task to be processed in the database through a wired connection manner or a wireless connection manner. It should be noted that the wireless connection may include, but is not limited to, 3G/4G connections, wiFi connections, bluetooth connections, wiMAX connections, zigbee connections, UWB (Ultra Wideband) connections, and other now known or later developed wireless connection means.
In the prior art, when a certain task executor fails, a task to be processed cannot be processed normally, and a user side of the task to be processed can also sense abnormal processing procedures of the task to be processed. The task dispatcher is typically aware of the running state time of the task to be processed after exceeding the task expiration time. Then, the task distributor needs to transfer the task to be processed executed by the failed task executor to other task executors. Because other task executors are also processing the task to be processed, after receiving the task to be processed of the fault task executor, the workload of the other task executors is increased, the processing time for processing the task to be processed is increased, and the processing efficiency of the task to be processed is reduced.
The execution subject of the application can monitor the tasks to be processed in the database. When the fact that the unexecuted task to be processed exists in the database is detected, the fact that the task to be processed is not executed is indicated. The execution main body can acquire the task to be processed and configure a current executor identifier for the task to be processed. The current executor identifier is used for representing a task executor executing the task to be processed, and may be information such as a number of an execution subject in the active task executor group 101, a hardware number of the execution subject, and the like, which may be specifically determined according to actual needs.
After the execution main body acquires the task to be processed, the task to be processed can be executed according to the requirement of the task to be processed. In order to facilitate effective execution of the task to be processed, the execution body may send running state feedback information corresponding to the task to be processed to the database at intervals of a set time. The running state feedback information comprises a time stamp, and the time stamp is used for identifying the monitoring time when the task to be processed is executed. In this way, the standby task executor in the standby task executor group 102 can timely learn the processing state of the task to be processed, so that when the execution main body fails, the standby task executor timely takes over the task to be processed through the running state feedback information, and timely and effective processing of the task to be processed is ensured.
In some optional implementations of this embodiment, the database includes task expiration times corresponding to the tasks to be processed. The task expiration time may be used to characterize the time required to identify the completion of the task to be processed. and
The executing the task to be processed may include: and distributing a first data processing thread for the task to be processed according to the task expiration time and the data processing capacity of the task to be processed.
The amount of data to be processed may vary from one task to another. Meanwhile, the task to be processed can be configured with corresponding task expiration time according to user requirements or other conditions. Therefore, after the execution body acquires the task to be processed, the execution body can allocate a first data processing thread for the task to be processed according to the task expiration time and the data processing amount of the task to be processed. The number of threads of the first data processing thread is determined according to the requirement. When the execution body has redundant threads, other tasks to be processed can be acquired. Therefore, the method is beneficial to improving the processing efficiency of the task to be processed and the use efficiency of the execution main body.
In some optional implementations of this embodiment, the running state feedback information may further include task progress information. The task progress information may include at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed and the process position label of the task to be processed.
In addition to the time stamp, the running state feedback information may further include task progress information of a task to be processed, an instruction currently executed by the task to be processed, a process position tag of the task to be processed, and the like. The task progress information may record the specific content of the execution subject to execute the task to be processed in more detail. When the execution main body fails, the standby task executor can know the specific occurrence condition of the task to be processed through the task progress information, and does not need to process the task to be processed again from the beginning, so that the processing efficiency of the task to be processed is improved.
And step 203, in response to executing the task to be processed, sending task completion information to the database.
After the execution main body smoothly executes the task to be processed under the condition of no fault, the execution main body can send task completion information to the database. In this way, execution of the task to be processed is completed.
As can be seen from the above description, when executing the task to be processed, the present application does not need to go through the task distributor, but directly obtains and monitors the task to be processed through the database, and the running state of the task to be processed can also be directly reflected in the running state feedback information corresponding to the task to be processed. Therefore, when the execution main body fails and can not process the task to be processed in time, other standby task executors can process the task to be processed in time, and therefore the processing efficiency of the task to be processed is ensured.
In some optional implementations of the present embodiment, the method may include the steps of:
and firstly, deleting the task to be processed in response to the fact that the current executor of the task to be processed is identified as other executors.
When the execution body fails, it is generally impossible to continue executing the task to be processed. Thus, execution of the task to be processed may still not be completed after the task expiration time of the task to be processed is exceeded. The standby task executor can replace the execution main body to execute the task to be processed when the execution main body cannot execute the task to be processed. When the executing body resumes normal operation from the failure, it is necessary to first check whether the current executor identification of the task to be processed is also the identification of the executing body. If yes, the fault time is not long, and the execution main body can continue to execute the task to be processed; if the current executor of the task to be processed is identified as other executors, the fault time of the execution main body is too long, and the corresponding task expiration time is exceeded. And, the task to be processed has been executed by the standby task executor. At this time, the execution body may delete the task to be processed, and release the thread occupied by the task to be processed, so as to prepare for other subsequent tasks to be processed.
And secondly, monitoring task expiration time and running state feedback information of other tasks to be processed in the database.
After deleting the task to be processed, the execution body may continue to monitor the task expiration time and running state feedback information of other tasks to be processed in the database. When it is determined that there is an unexecuted task in the database through the task expiration time and the running state feedback information, the execution subject may execute the task to be processed. At this time, the execution body is shifted from the active task executor to the standby task executor. Thus, the processing efficiency of the task to be processed is improved.
With further reference to FIG. 3, a flow 300 of yet another embodiment of a method for performing a data processing task is shown. The flow 300 of the method for performing data processing tasks comprises the steps of:
In this embodiment, the execution body of the method for executing the data processing task (e.g., the standby task executor in the standby task executor group 102 shown in fig. 1) may monitor the task to be processed in the database through a wired connection manner or a wireless connection manner.
In general, the task to be processed is executed by the active task executor in embodiment 2, and the execution body of this embodiment is in a state in which the task to be processed is not executed. The execution body can monitor task expiration time and running state feedback information of the task to be processed in the database so as to know the state of the task to be processed in time. Meanwhile, the operation state of the active task executor in embodiment 2 can also be reflected by the task expiration time and operation state feedback information of the task to be processed.
And step 302, in response to the task expiration time being later than the time stamp, continuing to monitor the task expiration time and the running state feedback information of the task to be processed, otherwise, executing the task to be processed.
When the monitoring task expiration time of the executing body is later than the time stamp, the task to be processed is processed. The task expiration time being later than the time stamp may refer to a temporal succession. For example, the task expiration time is: XX year XX month XX day 11:00; the time stamp is: XX year XX month XX day 10:30. The execution body can continuously monitor the task expiration time and the running state feedback information of the task to be processed. When the monitoring task expiration time of the execution subject is earlier than the time stamp, the active task executor executing the task to be processed cannot normally execute the task to be processed. At this time, the executing body may execute the task to be processed, and the identity of the executing body is also converted into an active task executor by the standby task executor. It can be seen that the identities of the standby task executor and the active task executor in the application can be converted according to whether to execute the task to be processed.
In some optional implementations of this embodiment, the task to be processed is marked with a corresponding current executor identifier, and executing the task to be processed may include: and modifying the current executor identifier into a local executor identifier.
From the above description, it is clear that, in order to flag the execution state of the task to be processed, the task to be processed may be marked with the current actuator identification. The current actuator identification may be used to identify the actuator currently executing the task to be processed. When the execution subject is to execute the task to be processed, the current executor identifier of the task to be processed can be modified to be a local executor identifier (i.e. the identifier of the execution subject in this embodiment), which is better than avoiding that the task to be processed is executed by multiple executors at the same time.
In some optional implementations of this embodiment, the running state feedback information may further include task progress information. The task progress information may include at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed and the process position label of the task to be processed. And executing the task to be processed, which may include the following steps:
And a first step of determining the residual data processing amount of the task to be processed according to the task progress information.
As can be seen from the above description, the execution body of embodiment 2 sends the running state feedback information to the database when executing the task to be processed. The task progress information in the running state feedback information can mark the specific execution condition of the task to be processed. The execution body of the embodiment may determine the execution progress of the task to be processed according to the task progress information, and determine the remaining data throughput of the task to be processed according to the task progress information.
And a second step of distributing a second data processing thread for the task to be processed according to the task expiration time and the residual data processing capacity.
After obtaining the residual data processing amount, the execution main body can allocate a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount. Therefore, the task to be processed can be processed in time, and the processing efficiency of the task to be processed is improved.
With continued reference to fig. 4a, fig. 4a is a schematic illustration of an application scenario corresponding to the method of fig. 2 for performing data processing tasks. In the application scenario of fig. 4a, the active task executor 1011 in the active task executor group 101 monitors that a task a to be processed exists in the database 104. Thereafter, the active task executor 1011 executes the task a to be processed and sends the database 104 the operation state feedback information-1 of the corresponding database 104. The standby task executor 1021 in the standby task executor group 102 may monitor the execution status of the task a to be processed (i.e., the task expiration time and the running status feedback information-1 of the task a to be processed). After the active task executor 1011 performs the task a to be processed, task completion information corresponding to the task a to be processed is sent to the database 104.
Fig. 4b is another schematic illustration of an application scenario corresponding to the method of fig. 3 for performing data processing tasks. In the application scenario of fig. 4b, the active task executor group 101 fails when executing the pending task a. When the standby task executor 1021 in the standby task executor group 102 monitors that the task expiration time of the task a to be processed is earlier than the time stamp in the running state feedback information-1, it may be considered that the active task executor 1011 has failed, and cannot continue to execute the task a to be processed. The standby task executor 1021 may obtain the task expiration time and the running state feedback information-1 of the task a to be processed from the database 104, and continue to execute the task a to be processed. At this time, the identity of the standby task executor 1021 is converted into an active task executor, and the identity of the active task executor 1011 is converted into a standby task executor.
The method provided by the embodiment of the disclosure acquires the task to be processed after detecting that the task to be processed which is not executed exists in the database, and configures a current executor identifier for the task to be processed; then executing the task to be processed, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time; and finally, after the task to be processed is executed, sending task completion information to the database. According to the method and the device, the running state feedback information of the task to be processed is directly stored in the database, so that the execution state of the task to be processed is monitored in time, and the execution efficiency of the task to be processed is improved.
With further reference to fig. 5, as an implementation of the method shown in the foregoing figures, the present disclosure provides an embodiment of an apparatus for performing data processing tasks, which corresponds to the method embodiment shown in fig. 2, and which may be applied in particular in various electronic devices.
As shown in fig. 5, the apparatus 500 for performing a data processing task of the present embodiment may include: a task acquisition unit 501, a first execution unit 502, and a completion information transmission unit 503. The task obtaining unit 501 is configured to obtain a task to be processed and configure a current executor identifier for the task to be processed, in response to detecting that the task to be processed exists in the database; the first execution unit 502 is configured to execute the task to be processed, and send running state feedback information corresponding to the task to be processed to the database at intervals of set time, where the running state feedback information includes a timestamp, and the timestamp is used to identify a monitoring time when the task to be processed is executed; the completion information transmitting unit 503 is configured to transmit task completion information to the database in response to execution of completion of the task to be processed.
In some optional implementations of this embodiment, the database includes a task expiration time corresponding to the task to be processed, where the task expiration time is used to characterize a time required for executing the task to be processed to complete, and the first execution unit 502 may include: a first execution subunit (not shown in the figure) is configured to allocate a first data processing thread to the task to be processed according to the task expiration time and the data processing amount of the task to be processed.
In some optional implementations of this embodiment, the running state feedback information further includes task progress information, where the task progress information includes at least one of: the process information of the task to be processed, the current executing instruction of the task to be processed and the process position label of the task to be processed.
In some optional implementations of this embodiment, the apparatus 500 for performing data processing tasks may include: a task deletion unit (not shown) and a first task monitoring unit (not shown). The task deleting unit is used for responding to the fact that the current executor of the task to be processed is identified as other executors and is configured to delete the task to be processed; and the first task monitoring unit is configured to monitor task expiration time and running state feedback information of other tasks to be processed in the database.
With further reference to fig. 6, as an implementation of the method shown in the foregoing figures, the present disclosure provides an embodiment of an apparatus for performing data processing tasks, which corresponds to the method embodiment shown in fig. 3, and which may be applied in particular in various electronic devices.
As shown in fig. 6, an apparatus 600 for performing a data processing task of the present embodiment may include: a second task monitoring unit 601 and a second execution unit 602. The second task monitoring unit 601 is configured to monitor task expiration time of a task to be processed in the database and running state feedback information, where the task expiration time is used for representing a time required for identifying execution completion of the task to be processed, and the running state feedback information includes a timestamp, and the timestamp is used for identifying a monitoring time when the task to be processed is executed; and a second execution unit 602 configured to continuously monitor the task expiration time and the running state feedback information of the task to be processed in response to the task expiration time being later than the time stamp, and otherwise, execute the task to be processed.
In some optional implementations of this embodiment, the task to be processed is marked with a corresponding current executor identifier, and the second execution unit 602 may include: an identifier modification subunit (not shown in the figure) configured to modify the current actuator identifier to a local actuator identifier.
In some optional implementations of this embodiment, the running state feedback information further includes task progress information, where the task progress information includes at least one of: the process information of the task to be processed, the instruction currently executed by the task to be processed, the process location tag of the task to be processed, and the second execution unit 602 may include: a remaining data throughput determination subunit (not shown) and a second execution subunit (not shown). The remaining data processing amount determining subunit is configured to determine the remaining data processing amount of the task to be processed according to the task progress information; and the second execution subunit is configured to allocate a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount.
The embodiment also provides an electronic device, including: one or more processors; and a memory having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to perform the method for performing data processing tasks described above.
The present embodiment also provides a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the above-described method for performing data processing tasks.
Referring now to FIG. 7, a schematic diagram of a computer system 700 suitable for use in an electronic device (e.g., an active task executor of the active task executor set 101 of FIG. 1) implementing embodiments of the present disclosure is shown. The electronic device shown in fig. 7 is only one example and should not impose any limitations on the functionality and scope of use of embodiments of the present disclosure.
As shown in fig. 7, the electronic device 700 may include a processing means (e.g., a central processor, a graphics processor, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage means 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
In general, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 shows an electronic device 700 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 7 may represent one device or a plurality of devices as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via communication device 709, or installed from storage 708, or installed from ROM 702. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 701.
It should be noted that, the above-mentioned computer readable medium according to the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the above-mentioned two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In an embodiment of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Whereas in embodiments of the present disclosure, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: responding to the detection that the unexecuted task exists in the database, acquiring the task to be processed, and configuring a current executor identifier for the task to be processed; executing the task to be processed, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp, and the time stamp is used for identifying monitoring time when the task to be processed is executed; and sending task completion information to the database in response to executing the task to be processed.
Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments described in the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes a task acquisition unit, a first execution unit, and a completion information transmission unit. The names of these units do not in any way limit the unit itself, for example, the first execution unit may also be described as "a unit that executes a task to be processed and sends running state feedback information".
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention referred to in this disclosure is not limited to the specific combination of features described above, but encompasses other embodiments in which features described above or their equivalents may be combined in any way without departing from the spirit of the invention. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Claims (8)
1. A method for performing data processing tasks, comprising:
the method comprises the steps of responding to the fact that an unexecuted task to be processed exists in a database through an active task executor, obtaining the task to be processed, and configuring a current executor identifier for the task to be processed;
executing the task to be processed through the active task executor, and sending running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp and task progress information, the time stamp is used for identifying monitoring time when the task to be processed is executed, and the task progress information comprises at least one of the following: the process information of the task to be processed, the current instruction of the task to be processed and the process position label of the task to be processed;
Transmitting task completion information to the database by the active task executor in response to execution of the task to be processed;
monitoring task expiration time and running state feedback information of a task to be processed in a database through a standby task executor, wherein the task expiration time is used for representing time required for executing the task to be processed;
responding to the task expiration time later than the time stamp by the standby task executor, and continuously monitoring the task expiration time and running state feedback information of the task to be processed;
otherwise, executing the task to be processed by the following method:
modifying the current executor identifier to be a local executor identifier; determining the residual data processing capacity of the task to be processed according to the task progress information; and distributing a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount.
2. The method of claim 1, wherein the executing, by the active task executor, the task to be processed comprises:
and distributing a first data processing thread for the task to be processed according to the task expiration time and the data processing amount of the task to be processed.
3. The method according to any one of claims 1 to 2, wherein the method comprises:
deleting the task to be processed in response to the fact that the current executor of the task to be processed is identified as other executors;
and monitoring task expiration time and running state feedback information of other tasks to be processed in the database.
4. An apparatus for performing data processing tasks, comprising:
the task acquisition unit is used for responding to the fact that the unexecuted task to be processed exists in the database, and is configured to acquire the task to be processed through an active task executor, and to configure a current executor identifier for the task to be processed;
the first execution unit is configured to execute the task to be processed through the active task executor, and send running state feedback information corresponding to the task to be processed to the database at intervals of set time, wherein the running state feedback information comprises a time stamp and task progress information, the time stamp is used for identifying monitoring time when the task to be processed is executed, and the task progress information comprises at least one of the following: the process information of the task to be processed, the current instruction of the task to be processed and the process position label of the task to be processed;
A completion information transmitting unit configured to transmit task completion information to the database through the active task executor in response to execution of the task to be processed;
the second task monitoring unit is configured to monitor task expiration time and running state feedback information of a task to be processed in the database through the standby task executor, wherein the task expiration time is used for representing and identifying time required for executing the task to be processed;
the second execution unit is configured to respond to the task expiration time later than the time stamp by the standby task executor and continuously monitor the task expiration time and the running state feedback information of the task to be processed, and otherwise, execute the task to be processed;
wherein the second execution unit includes:
an identification modification subunit configured to modify the current actuator identification to a local actuator identification;
a remaining data processing amount determination subunit configured to determine a remaining data processing amount of the task to be processed according to the task progress information;
and the second execution subunit is configured to allocate a second data processing thread for the task to be processed according to the task expiration time and the residual data processing amount.
5. The apparatus of claim 4, wherein the first execution unit comprises:
and the first execution subunit is configured to allocate a first data processing thread for the task to be processed according to the task expiration time and the data processing quantity of the task to be processed.
6. The apparatus according to any one of claims 4 to 5, wherein the apparatus comprises:
a task deleting unit, configured to delete the task to be processed in response to the current executor of the task to be processed being identified as other executors;
and the first task monitoring unit is configured to monitor task expiration time and running state feedback information of other tasks to be processed in the database.
7. An electronic device, comprising:
one or more processors;
a memory having one or more programs stored thereon,
the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-3.
8. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911394801.XA CN113127158B (en) | 2019-12-30 | 2019-12-30 | Method and device for executing data processing task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911394801.XA CN113127158B (en) | 2019-12-30 | 2019-12-30 | Method and device for executing data processing task |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113127158A CN113127158A (en) | 2021-07-16 |
| CN113127158B true CN113127158B (en) | 2023-07-14 |
Family
ID=76768922
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911394801.XA Active CN113127158B (en) | 2019-12-30 | 2019-12-30 | Method and device for executing data processing task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127158B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114218300B (en) * | 2021-12-23 | 2023-08-08 | 北京百度网讯科技有限公司 | Data processing method, data processing device, data processing system, electronic equipment and computer readable medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009089746A1 (en) * | 2007-12-28 | 2009-07-23 | Huawei Technologies Co., Ltd. | Method, device and system for realizing task in cluster environment |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7395537B1 (en) * | 2003-12-08 | 2008-07-01 | Teradata, Us Inc. | Administering the workload of a database system using feedback |
| CN102880473A (en) * | 2012-09-28 | 2013-01-16 | 五八有限公司 | Method and device for task execution based on quartz framework |
| US11372672B2 (en) * | 2013-02-15 | 2022-06-28 | Apple Inc. | Centralized task scheduling |
| CN104657212B (en) * | 2015-02-13 | 2018-02-16 | 北京亚信数据有限公司 | A kind of method and system of task scheduling |
| CN106095540B (en) * | 2016-05-31 | 2020-01-03 | 上海萌果信息科技有限公司 | Task flow processing method based on Quartz frame |
| US10193762B2 (en) * | 2016-08-11 | 2019-01-29 | Rescale, Inc. | Dynamic optimization of simulation resources |
| CN108628677B (en) * | 2017-03-20 | 2022-12-02 | 北京京东尚科信息技术有限公司 | Distributed task processing system, method and device |
| CN108762900A (en) * | 2018-05-21 | 2018-11-06 | 平安科技（深圳）有限公司 | High frequency method for scheduling task, system, computer equipment and storage medium |
| CN110377406A (en) * | 2019-06-17 | 2019-10-25 | 平安银行股份有限公司 | A kind of method for scheduling task, device, storage medium and server node |
-
2019
- 2019-12-30 CN CN201911394801.XA patent/CN113127158B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009089746A1 (en) * | 2007-12-28 | 2009-07-23 | Huawei Technologies Co., Ltd. | Method, device and system for realizing task in cluster environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113127158A (en) | 2021-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105357038B (en) | Monitor the method and system of cluster virtual machine | |
| US20160275123A1 (en) | Pipeline execution of multiple map-reduce jobs | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN107016480B (en) | Task scheduling method, device and system | |
| CN111813545A (en) | Resource allocation method, device, medium and equipment | |
| CN109743209B (en) | Method and device for transmitting information | |
| US10127092B1 (en) | Method and apparatus for monitoring a message transmission frequency in a robot operating system | |
| US20100228849A1 (en) | Deployment of Asynchronous Agentless Agent Functionality in Clustered Environments | |
| CN111338834B (en) | Data storage method and device | |
| CN110673959A (en) | System, method and apparatus for processing tasks | |
| CN113127158B (en) | Method and device for executing data processing task | |
| CN115328741A (en) | Exception handling method, device, equipment and storage medium | |
| CN107729213A (en) | A kind of background task monitoring method and device | |
| CN111459629A (en) | Azkaban-based project operation method and device and terminal equipment | |
| US20150156340A1 (en) | Information processing system, information processing apparatus, and program | |
| CN112131013A (en) | Monitoring method, device, equipment and storage medium of call chain in distributed system | |
| US20190196880A1 (en) | Method for sharing processing modules between pipelines | |
| CN111444148B (en) | Data transmission method and device based on MapReduce | |
| CN113760522A (en) | Task processing method and device | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN114816477A (en) | Server upgrading method, device, equipment, medium and program product | |
| CN111652691A (en) | Order information processing method and device and electronic equipment | |
| CN115250276A (en) | Distributed system and data processing method and device | |
| CN112000720A (en) | Management method and management system for database connection and database connection pool | |
| US11941432B2 (en) | Processing system, processing method, higher-level system, lower-level system, higher-level program, and lower-level program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |