CN111708799B - Spark task processing method and device, electronic equipment and storage medium - Google Patents
Spark task processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111708799B CN111708799B CN202010366884.8A CN202010366884A CN111708799B CN 111708799 B CN111708799 B CN 111708799B CN 202010366884 A CN202010366884 A CN 202010366884A CN 111708799 B CN111708799 B CN 111708799B
- Authority
- CN
- China
- Prior art keywords
- processed
- task
- spark
- cluster system
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a Spark task processing method, a Spark task processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: sequentially reading the tasks to be processed according to the priorities of the tasks to be processed; and judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed. According to the method and the device for determining the tasks to be processed, whether the tasks to be processed are executed or not is determined according to the running state of the Spark cluster system, the current available resources and the resources required by the tasks to be processed currently, so that the problem that the tasks which are ordered later but require fewer resources cannot be executed due to the fact that the time consumed for ordering the tasks which are prior is too long and the resources are occupied is avoided, and further optimal allocation of the resources can be achieved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a Spark task processing method, device, electronic equipment, and storage medium.
Background
Spark is a general purpose engine that can be used to perform a variety of operations including SQL queries, text processing, machine learning, etc. In big data systems Spark is often used to fulfill the needs of various big data processing. For example, through Spark, a plurality of SQL query statistics tasks are performed, and the common measures adopted at this time are: under the condition that a plurality of tasks are to be processed, a Spark-sub (Spark application deployment tool) can create a Spark context through a unified main method, the tasks are submitted to run simultaneously through multithreading in the Spark context, then after the Spark context receives all the tasks, resources are allocated in sequence through central scheduling to execute each task, and finally the task completion program exits. Or, by adding multiple main methods, multiple tasks are scattered in the main methods, so that multiple spark contexts can be created at the same time, and the spark contexts can schedule and distribute execution of tasks by themselves.
However, both of these schemes suffer from corresponding drawbacks:
for creating a spark context by a unified main method, a processing mode of simultaneously submitting a plurality of task operations by multithreading is not capable of being dynamically scheduled under the conditions that certain tasks consume long time and occupy more resources or the condition that statistics results of certain tasks are needed in a temporary emergency, resources are allocated preferentially, and the tasks are executed preferentially to obtain data. The method can only apply for resources and queue to execute tasks in sequence according to the task sequence in the main method which is designed originally, is not flexible enough and cannot meet the actual demands. For example, when a task that is sequenced first takes too long and takes too much resources, tasks that are sequenced later but require less resources cannot be executed, so that optimal allocation of resources cannot be achieved. For another example, when a certain urgent task needs to be temporarily executed, the task cannot be dynamically scheduled and preferentially executed, so that the task is not flexible enough and cannot meet the actual requirement.
In the processing mode of dispersing a plurality of tasks into a plurality of main methods by adding the plurality of main methods, as one task corresponds to one main method, too many tasks are distributed and too thin, so that the maintenance cost of operation and maintenance personnel is increased.
Disclosure of Invention
Because the existing method has the problems, the embodiment of the invention provides a Spark task processing method, a Spark task processing device, electronic equipment and a storage medium.
In a first aspect, an embodiment of the present invention provides a Spark task processing method, including:
sequentially reading the tasks to be processed according to the priorities of the tasks to be processed;
and judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed.
Further, the Spark task processing method further includes:
if the current read task to be processed is judged not to be executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the current read task to be processed, the next task to be processed is directly judged.
Further, for each currently read task to be processed, judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed, and judging for the next task to be processed, wherein the method specifically comprises the steps of:
judging whether the current running state of the Spark cluster system is normal or not according to the current read task to be processed, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the current read task to be processed or not, if so, executing the current read task to be processed, and judging according to the next task to be processed.
Further, if it is determined that the currently read task to be processed is not executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system, and the resources required by the currently read task to be processed, then the next task to be processed is directly determined, which specifically includes:
if the current available resources of the Spark cluster system are judged not to meet the resources required by the currently read task to be processed, judging the next task to be processed.
Further, the judging for the next task to be processed specifically includes:
judging whether the current running state of the Spark cluster system is normal, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the next task to be processed, if so, executing the next task to be processed, and if not, directly judging the next task to be processed until all the tasks to be processed are traversed.
Further, the Spark task processing method further includes:
and after each detection of the completion of the execution of one task, judging whether the unexecuted task to be processed exists, if so, judging the unexecuted task to be processed according to the priority of the unexecuted task to be processed by the same logic.
Further, judging whether the current running state of the Spark cluster system is normal or not specifically includes:
and judging whether the disk read-write speed of the Spark cluster system is greater than a first threshold value and whether the system throughput is greater than a second threshold value, and if so, determining that the current running state of the Spark cluster system is normal.
Further, the Spark task processing method further includes:
when the task to be processed needs to temporarily adjust the execution sequence, the priority of the corresponding task is modified.
In a second aspect, an embodiment of the present invention further provides a Spark task processing device, including:
the reading module is used for sequentially reading the tasks to be processed according to the priorities of the tasks to be processed;
the processing module is used for judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the Spark task processing method according to the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the Spark task processing method according to the first aspect.
According to the Spark task processing method, device, electronic equipment and storage medium provided by the embodiment of the invention, whether the current task to be processed is executed is determined according to the running state of the Spark cluster system, the current available resources and the resources required by the current task to be processed, so that the problem that the tasks which are ordered later but require less resources cannot be executed due to the fact that the tasks which are ordered earlier take too long and occupy too much resources can be avoided, and the optimal allocation of the resources can be realized.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other drawings can be obtained from these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a Spark task processing method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a Spark task processing device according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following describes the embodiments of the present invention further with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
Fig. 1 shows a flowchart of a Spark task processing method provided by an embodiment of the present invention, and as shown in fig. 1, the Spark task processing method provided by the embodiment of the present invention is applied to a Spark task processing device, and specifically includes the following contents:
step 101: and sequentially reading the tasks to be processed according to the priorities of the tasks to be processed.
Step 102: and judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed.
In this embodiment, the priority of each task to be processed may be determined according to the importance of each task to be processed. The determination may also be performed according to the usability of each task to be processed, which is not limited in this embodiment. For example, assume that, according to the importance of 3 tasks to be processed, the determined task priorities are, in order from high to low: task3> Task2> Task1.
In this embodiment, the current running state of the Spark cluster system may refer to the current disk read-write speed of the Spark cluster system, or may refer to the system throughput of the Spark cluster system, or may refer to both the current disk read-write speed of the Spark cluster system and the system throughput of the Spark cluster system.
In this embodiment, the currently available resource of the Spark cluster system may refer to the currently available memory of the Spark cluster system. The current available memory of the Spark cluster system is: the difference between the total available memory of the Spark cluster system and the memory occupied by all the tasks currently running. For example, the total available memory of the Spark cluster system is mem_origin 90% (mem_origin is the initial memory of the system, and the system memory usage is generally considered to be below 90% and can be normally operated, so the total available memory of the system is mem_origin 90%), and assuming that the tasks currently operated are Task2 and Task3, the current available memory of the Spark cluster system is mem_origin 90% -mem2-mem3. Wherein, mem2 and mem3 are memories occupied by Task2 and Task3 during operation respectively.
In this embodiment, the resources required by each task to be processed refer to resources that need to be occupied when each task to be processed is executed, such as the memory that needs to be occupied when running.
In this embodiment, the execution of each task is not performed only according to the inherent sequence of each task, but the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the tasks which are currently read are comprehensively considered, and whether the tasks which are currently read are executed is dynamically determined, so that the execution of multiple tasks can be more flexibly realized, the execution of multiple tasks can be efficiently completed under the condition that the normal running of the system and the normal execution of each task are ensured, the problem that the tasks which are ordered afterwards but require less resources cannot be executed due to overlong time consumption and excessive resources occupation of the tasks which are ordered before are avoided, and further the optimal allocation of the resources can be realized, and the execution efficiency of the Spark cluster can be effectively improved. Therefore, the embodiment realizes the dynamic adjustment of multiple tasks in one main method without maintaining multiple main methods and increasing personnel maintenance cost, and realizes the maximization of resource utilization by completing the execution of multiple tasks in the shortest time while minimizing the maintenance cost.
According to the Spark task processing method provided by the embodiment of the invention, whether the current task to be processed is executed is determined according to the running state of the Spark cluster system, the current available resources and the resources required by the current task to be processed, so that the problem that the tasks which are ordered later but require less resources cannot be executed due to the fact that the time consumption of the tasks which are ordered earlier is too long and the resources are too much occupied can be avoided, and the optimal allocation of the resources can be realized.
Based on the foregoing embodiment, in this embodiment, the Spark task processing method further includes: when the task to be processed needs to temporarily adjust the execution sequence, the priority of the corresponding task is modified.
In this embodiment, when a task to be processed needs to temporarily adjust the execution sequence, for example, when a temporary urgent task needs to be processed in an urgent manner, the execution sequence may be adjusted in a manner of modifying the priority of the corresponding task.
According to the Spark task processing method provided by the embodiment of the invention, the execution sequence of each task to be processed and the concurrent execution condition of each task to be processed are dynamically determined according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by each task to be processed, and meanwhile, when the task temporarily needs to adjust the execution sequence, the priority of the corresponding task is modified. Therefore, the embodiment of the invention dynamically schedules the tasks according to the available resource condition of the system, the resource demand condition of each task and the running state of the system after the tasks are run, and the task execution sequence is considered to be temporarily adjusted, so that the task priority is temporarily adjusted, and the actual service scene is met. The embodiment of the invention realizes the multi-task dynamic adjustment in one main method without maintaining a plurality of main methods and increasing personnel maintenance cost, and realizes the maximization of resource utilization by completing the multi-task execution in the shortest time while minimizing the maintenance cost.
Based on the foregoing embodiment, in this embodiment, the Spark task processing method further includes:
if the current read task to be processed is judged not to be executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the current read task to be processed, the next task to be processed is directly judged.
In this embodiment, for each currently read task to be processed, whether to execute the currently read task to be processed is determined according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if yes, the currently read task to be processed is executed, and the next task to be processed is determined, otherwise, the next task to be processed is directly determined. For example, assume that the priorities of 3 tasks are, in order from high to low: task3> Task2> Task1, since the Task3 has the highest priority, firstly, the Task3 is read, then, for the read Task3, whether the current running state of the Spark cluster system is normal is judged, if so, whether the current available resources of the Spark cluster system meet the resources required by the Task3 is continuously judged, and if so, the Task3 is executed (if not, whether the current available resources of the Spark cluster system meet the resources required by the Task2 is judged). Assuming that Task3 is executed, whether the current available resources of the Spark cluster system meet the resources required by Task2 is continuously determined by determining that the running state of the system is normal, if yes, task2 is continuously executed (in this case, the system simultaneously executes Task3 and Task2, it is to be noted that if Task2 is executed, it is continuously determined whether the current available resources of the system meet the resources required by Task 1), if the current available resources of the Spark cluster system do not meet the resources required by Task2, it is continuously determined whether the current available resources of the Spark cluster system meet the resources required by Task1, and if yes, task1 is continuously executed (in this case, the system simultaneously executes Task3 and Task 1). Therefore, the embodiment dynamically determines the execution sequence of each task to be processed and the concurrent execution condition of each task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by each task to be processed, so that the embodiment can realize the dynamic scheduling of the tasks to be executed, the multi-task execution can be completed most effectively, the phenomenon that some tasks which consume resources occupy system resources for a long time, the read-write rate of a system disk is reduced or the throughput is reduced is avoided, and meanwhile, some tasks which require less resources cannot be executed normally as early as possible.
Based on the foregoing embodiment, in this embodiment, for each currently read task to be processed, whether to execute the currently read task to be processed is determined according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system, and the resources required by the currently read task to be processed, if yes, executing the currently read task to be processed, and determining for the next task to be processed, which specifically includes:
judging whether the current running state of the Spark cluster system is normal or not according to the current read task to be processed, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the current read task to be processed or not, if so, executing the current read task to be processed, and judging according to the next task to be processed.
If the current read task to be processed is not executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the current read task to be processed, the next task to be processed is directly judged, and the method specifically comprises the following steps:
if the current available resources of the Spark cluster system are judged not to meet the resources required by the currently read task to be processed, judging the next task to be processed.
In this embodiment, the determining for the next task to be processed specifically includes:
judging whether the current running state of the Spark cluster system is normal, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the next task to be processed, if so, executing the next task to be processed, and if not, directly judging the next task to be processed until all the tasks to be processed are traversed.
In this embodiment, it is assumed that the priorities of the 3 tasks are, in order from high to low: task3> Task2> Task1, if Task3 is determined to be normal by determining that the current running state of the Spark cluster system is normal before executing, at this time, it is determined whether the current available resources of the Spark cluster system meet the resources required by Task3, if yes, task3 is executed, if Task3 is determined to be executed, by determining that the running state of the confirmation system is normal, and by determining that the remaining resources of the confirmation system do not meet the resources required by Task2, task2 is not executed, and further determining whether the remaining resources of the system meet the resources required by Task1, and if yes, task1 is executed. In addition, after the Task1 is executed, whether the residual resources of the system meet the resources required by the Task2 is judged again, and if yes, the Task2 is executed.
In this embodiment, it should be noted that, when it is determined that the operation state of the Spark cluster system is abnormal, the currently read task to be processed is not executed any more, so as to avoid that the currently read task to be processed cannot be executed normally.
Based on the foregoing embodiment, in this embodiment, determining whether the current running state of the Spark cluster system is normal specifically includes:
and judging whether the disk read-write speed of the Spark cluster system is greater than a first threshold value and whether the system throughput is greater than a second threshold value, and if so, determining that the current running state of the Spark cluster system is normal.
In this embodiment, the disc read-write speed being greater than the first threshold may be that the disc read-write speed is greater than iostat_origin by 80%. The system throughput being greater than the second threshold may be greater than tps_origin by 90%. The iostat_origin here represents the standard disk read-write speed of the spark cluster system, and the value is a preset value, and the values corresponding to different systems are different. Here tps_origin represents the standard throughput of the spark cluster system, which is a preset value, and the values corresponding to different systems are different.
In this embodiment, the system throughput may be determined by the number of executors that the task starts, the number of cores of each executor, and the duration of execution of the task. For example, the system throughput of the task execution is obtained through a first relationship model; wherein the first relationship model is:
tps＝num_executors*executor_core/dur
wherein tps represents system throughput; num_executors represents the number of executors that the task starts; the executor_core represents the number of cores per executor; dur represents the duration of task execution.
In this embodiment, the following three main factors are known to affect the execution efficiency of Spark task: memory mem, disk read-write speed iostat and system throughput tps, so that the related parameters of the spark cluster environment are collected first and respectively marked as mem_origin, iostat_origin and tps_origin.
Assuming a total of 3 data statistics tasks, the average of the memory index data of these 3 tasks when running on the designated spark cluster is collected first, as shown in table 1 below.
TABLE 1
In addition, assuming the commonality according to the data result and the importance of the data according to the service judgment, the priorities of the 3 tasks are determined as follows: task3> Task2> Task1, whereby level3 = 3 of Task3, level2 = 2 of Task2, level1 = 1 of Task1 is set by default.
The general system memory utilization rate can normally run below 90%, and if the memory utilization rate is higher than 90%, the abnormal risk exists, and time delay and even downtime are possible. Thus during task execution, the available memory mem_uneuse of the computing system:
mem_unuse＝mem_origin*90％-mem
wherein mem is the occupied memory of the Task in operation, for example, the available memory of Task1 in operation, and can be calculated by replacing mem in the above formula with mem 1. And sequentially replacing mem to obtain available memories mem_unwuse 2 and mem_unwuse 3 of the Task2 and the Task3.
In this embodiment, task3 is preferentially executed according to the priority ranking of 3 tasks, and when Task3 is executed, mem_unwuse 3 is obtained, where mem_unwuse 3=mem_origin is 90% -mem3. When mem_unw3 is larger than the memory required by Task2 and Task1 at the same time, task2 is preferentially operated according to the priority rank of 3 tasks.
At this time, the available memory mem_unwuse 2 after the Task2 is run is calculated:
mem_unuse2＝mem_unuse3-mem2
if mem_unwe2 at this time is greater than mem1, then Task1 may be performed; if mem_unwe2 is smaller than mem1, then the system can only run Task2 and Task3 at the same time, task1 can only queue, and execution can continue until the system resources are free.
While the foregoing describes the memory related content, it should be noted that the disk read/write speed needs to be further considered. The known hardware device adopted by the big data system has the disk read-write speed of the iostat_origin, and in general, the iostat in the task execution process reaches 80% of the iostat_origin, so that the computing system is in a good operation range. According to the priority ranking of the 3 tasks, preferentially running Task3, when Task3 runs, obtaining iostat3, if iostat3> = iostat_origin 80%, continuing to run Task2. When the Task2 and the Task3 are simultaneously operated, if iostat3> = iostat_origin 80%, and if iostat2> = iostat_origin 80%, the Task1 is continued to be operated, and so on.
While the foregoing describes the disk read/write speed, it should be noted that the system throughput needs to be further considered. In this embodiment, when a task is submitted to the Spark cluster for execution, the number num_executives of executors that the task starts and the number of cores per executor executor_core are generally specified. According to a system throughput calculation formula:
tps＝num_executors*executor_core/dur
wherein tps represents system throughput; num_executors represents the number of executors that the task starts; the executor_core represents the number of cores per executor; dur represents the duration of task execution. When tps > = tps_origin 90%, the system is judged to be in a good operating range.
Suppose when Spark-submit tasks to Spark cluster execution, task3 is run preferentially according to the priority ranking of 3 tasks. At this time, according to the calculation formula mem_unwuse 3=mem_origin 90% -mem3, mem_unwuse 3 can be obtained, if mem_unwuse 3 is greater than 0, it indicates that the system memory usage is below 90%. Meanwhile, the assumption is that the reading and writing speed of the magnetic disk is obtained after the reading and writing speed of the magnetic disk is obtained, namely, the iostat3> =iostat_origin is 80%, and the reading and writing speed is larger than 80% of a normal value at the moment; meanwhile, according to a system throughput calculation formula, tps3> =tps_origin is obtained, and the system throughput is shown to be in a normal value range. When the three conditions of memory, disk read-write speed and system throughput are all satisfied, the state is normal when the system runs Task3.
On the basis of the above, task2 is continuously operated according to the priority ranking of the 3 tasks, the above processing procedure is repeated, and the result of normal system state is obtained, and the simultaneous operation of Task2 and Task3 is indicated.
When tasks need to be temporarily adjusted in execution sequence and run preferentially, the level value is set to be the maximum. For example, if Task1 is to be executed preferentially, then level1 is set to 4, i.e., level1> level3> level2, and if the system detects a change in level value, level1 is executed preferentially in the new order of priority.
To sum up, a database table spark_data is created and the data is initialized as in table 2 below.
TABLE 2
In this embodiment, a configuration file spark_origin_data_properties may be added, where a system performance reference value is configured: memory mem_origin, disk read/write speed iostat_origin, and system throughput tps_origin.
When the Spark-sub is ready to submit a task to the Spark cluster for execution, firstly, reading the spark_data table to obtain data with the maximum level value for execution. At this time, the first task to be executed is the equivalent of mem_ origin, iostat _origin and tps_origin, which are read from spark_origin_data. When the condition of the memory is met, the mem_unwuse_ realtime, iostat _realtetime and tps_realtetime can be recorded, and at the moment, the system instantiates the task with the maximum level value according to the task_name and submits the task to the Spark cluster to start execution.
And then, continuing to traverse the spark_data table to acquire a task with a second level value row to be executed. If the current available memory meets the memory required by the Task2 during execution and the disk read-write speed of the Task3 during execution is greater than the second threshold, and if the system throughput of the Task3 during execution is greater than the third threshold, the system can continue to operate the Task2. At this time, the task may be instantiated according to the task_name with the level value second largest, and the task may be submitted to the Spark cluster to start execution.
And continuing to cycle and traverse subsequently, when three conditions of the memory, the disk read-write speed and the system throughput cannot be all met, not instantiating the corresponding task, but directly entering the next cycle, and continuing to judge whether the executable task meeting the conditions exists or not. And when the cycle is ended and no task meeting three conditions of memory, disk read-write speed and system throughput exists, the cycle is ended. And when the currently started task is completed, recycling and submitting a new task to execute.
When tasks need to be temporarily adjusted in execution sequence, the level is modified. And when the next cycle starts, extracting data according to the latest level rank of the spark_data table, and performing calculation and cycle judgment.
As can be seen from the above description, in this embodiment, under the normal conditions of the available memory, the disk read-write speed and the system throughput index of the monitoring system, a plurality of tasks can be started simultaneously, and the temporary adjustment of the task execution sequence is also supported, so as to improve the utilization rate of system resources. By adopting the Spark task processing method provided by the embodiment, multiple tasks can be operated at the same time, and the utilization rate of system resources is improved. The method also supports the temporary adjustment of the task execution sequence, reduces the manual maintenance operation cost, is more flexible and meets the actual scene requirement.
Fig. 2 is a schematic structural diagram of a Spark task processing device according to an embodiment of the present invention, where, as shown in fig. 2, the Spark task processing device according to the embodiment of the present invention includes:
a reading module 21, configured to sequentially read the tasks to be processed according to the priorities of the tasks to be processed;
the processing module 22 is configured to determine, for each currently read task to be processed, whether to execute the currently read task to be processed according to a current running state of the Spark cluster system, a current available resource of the Spark cluster system, and a resource required by the currently read task to be processed, if so, execute the currently read task to be processed, and determine for a next task to be processed.
Because the Spark task processing device provided in this embodiment may be used to execute the Spark task processing method provided in the foregoing embodiment, the working principle and the beneficial effects thereof are similar, and will not be described in detail herein.
Based on the same inventive concept, a further embodiment of the present invention provides an electronic device, see fig. 3, comprising in particular: a processor 301, a memory 302, a communication interface 303, and a communication bus 304;
wherein, the processor 301, the memory 302, and the communication interface 303 complete communication with each other through the communication bus 304; the communication interface 303 is used for realizing information transmission between devices;
the processor 301 is configured to invoke a computer program in the memory 302, where the processor executes the computer program to implement all the steps of the Spark task processing method described above, for example, the processor executes the computer program to implement the following steps: sequentially reading the tasks to be processed according to the priorities of the tasks to be processed; and judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed.
Based on the same inventive concept, a further embodiment of the present invention provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements all the steps of the Spark task processing method described above, for example, the processor implements the following steps when executing the computer program: sequentially reading the tasks to be processed according to the priorities of the tasks to be processed; and judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed.
Further, the logic instructions in the memory described above may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, spark task processing device, or network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment of the invention. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, spark task processing device, or network device, etc.) to execute the Spark task processing method described in the various embodiments or some parts of the embodiments.
Furthermore, in the present disclosure, such as "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.
Moreover, in the present invention, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Furthermore, in the description herein, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. The Spark task processing method is characterized by comprising the following steps of:
sequentially reading the tasks to be processed according to the priorities of the tasks to be processed;
judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed;
judging whether the unexecuted task to be processed exists after each task is detected to be executed, if yes, judging the unexecuted task to be processed with the same logic according to the priority of the unexecuted task to be processed;
when the task to be processed needs to temporarily adjust the execution sequence, modifying the priority of the corresponding task;
the judging for the next task to be processed specifically includes:
judging whether the current running state of the Spark cluster system is normal, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the next task to be processed, if so, executing the next task to be processed, and if not, directly judging the next task to be processed until all the tasks to be processed are traversed.
2. The Spark task processing method according to claim 1, further comprising:
if the current read task to be processed is judged not to be executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the current read task to be processed, the next task to be processed is directly judged.
3. The Spark task processing method according to claim 1, wherein for each currently read task to be processed, determining whether to execute the currently read task to be processed according to a current running state of the Spark cluster system, a current available resource of the Spark cluster system, and a resource required by the currently read task to be processed, if yes, executing the currently read task to be processed, and determining for a next task to be processed, including:
judging whether the current running state of the Spark cluster system is normal or not according to the current read task to be processed, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the current read task to be processed or not, if so, executing the current read task to be processed, and judging according to the next task to be processed.
4. The Spark task processing method according to claim 2, wherein if it is determined that the currently read task to be processed is not executed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system, and the resources required by the currently read task to be processed, then the determining directly for the next task to be processed specifically includes:
if the current available resources of the Spark cluster system are judged not to meet the resources required by the currently read task to be processed, judging the next task to be processed.
5. The Spark task processing method according to claim 3, wherein determining whether the current operation state of the Spark cluster system is normal specifically includes:
and judging whether the disk read-write speed of the Spark cluster system is greater than a first threshold value and whether the system throughput is greater than a second threshold value, and if so, determining that the current running state of the Spark cluster system is normal.
6. A Spark task processing device, comprising:
the reading module is used for sequentially reading the tasks to be processed according to the priorities of the tasks to be processed;
the processing module is used for judging whether to execute the currently read task to be processed according to the current running state of the Spark cluster system, the current available resources of the Spark cluster system and the resources required by the currently read task to be processed, if so, executing the currently read task to be processed and judging the next task to be processed;
judging whether the unexecuted task to be processed exists after each task is detected to be executed, if yes, judging the unexecuted task to be processed with the same logic according to the priority of the unexecuted task to be processed;
when the task to be processed needs to temporarily adjust the execution sequence, modifying the priority of the corresponding task;
the judging for the next task to be processed specifically includes:
judging whether the current running state of the Spark cluster system is normal, if so, judging whether the current available resources of the Spark cluster system meet the resources required by the next task to be processed, if so, executing the next task to be processed, and if not, directly judging the next task to be processed until all the tasks to be processed are traversed.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the Spark task processing method according to any one of claims 1 to 5 when executing the computer program.
8. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the Spark task processing method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010366884.8A CN111708799B (en) | 2020-04-30 | 2020-04-30 | Spark task processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010366884.8A CN111708799B (en) | 2020-04-30 | 2020-04-30 | Spark task processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111708799A CN111708799A (en) | 2020-09-25 |
| CN111708799B true CN111708799B (en) | 2023-09-05 |
Family
ID=72536834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010366884.8A Active CN111708799B (en) | 2020-04-30 | 2020-04-30 | Spark task processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111708799B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342493B (en) * | 2021-06-15 | 2022-09-20 | 上海哔哩哔哩科技有限公司 | Task execution method and device and computer equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022670A (en) * | 2015-07-17 | 2015-11-04 | 中国海洋大学 | Heterogeneous distributed task processing system and processing method in cloud computing platform |
| CN108711007A (en) * | 2018-05-16 | 2018-10-26 | 国电南瑞南京控制系统有限公司 | A kind of multitask real-time scheduling method of energy integration collection system |
| CN110413412A (en) * | 2019-07-19 | 2019-11-05 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus based on GPU cluster resource allocation |
| CN110704186A (en) * | 2019-09-25 | 2020-01-17 | 国家计算机网络与信息安全管理中心 | Computing resource allocation method and device based on hybrid distribution architecture and storage medium |
-
2020
- 2020-04-30 CN CN202010366884.8A patent/CN111708799B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105022670A (en) * | 2015-07-17 | 2015-11-04 | 中国海洋大学 | Heterogeneous distributed task processing system and processing method in cloud computing platform |
| CN108711007A (en) * | 2018-05-16 | 2018-10-26 | 国电南瑞南京控制系统有限公司 | A kind of multitask real-time scheduling method of energy integration collection system |
| CN110413412A (en) * | 2019-07-19 | 2019-11-05 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus based on GPU cluster resource allocation |
| CN110704186A (en) * | 2019-09-25 | 2020-01-17 | 国家计算机网络与信息安全管理中心 | Computing resource allocation method and device based on hybrid distribution architecture and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111708799A (en) | 2020-09-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113238838B (en) | Task scheduling method and device and computer readable storage medium | |
| US8959515B2 (en) | Task scheduling policy for limited memory systems | |
| US8793695B2 (en) | Information processing device and information processing method | |
| US7441240B2 (en) | Process scheduling apparatus, process scheduling method, program for process scheduling, and storage medium recording a program for process scheduling | |
| US20160210174A1 (en) | Hybrid Scheduler and Power Manager | |
| US8713573B2 (en) | Synchronization scheduling apparatus and method in real-time multi-core system | |
| US20080066070A1 (en) | Method and system for the dynamic scheduling of jobs in a computing system | |
| US20090276781A1 (en) | System and method for multi-level preemption scheduling in high performance processing | |
| CN107273200B (en) | Task scheduling method for heterogeneous storage | |
| US7920282B2 (en) | Job preempt set generation for resource management | |
| CN107515781B (en) | Deterministic task scheduling and load balancing system based on multiple processors | |
| CN112764904A (en) | Method for preventing starvation of low priority tasks in multitask-based system | |
| CN108509280A (en) | A kind of Distributed Calculation cluster locality dispatching method based on push model | |
| CN111190691A (en) | Automatic migration method, system, device and storage medium suitable for virtual machine | |
| CN107977275B (en) | Task processing method based on message queue and related equipment | |
| CN113032102A (en) | Resource rescheduling method, device, equipment and medium | |
| CN111708799B (en) | Spark task processing method and device, electronic equipment and storage medium | |
| CN114968566A (en) | Container scheduling method and device under shared GPU cluster | |
| CN113391911B (en) | Dynamic scheduling method, device and equipment for big data resources | |
| CN113961353A (en) | Task processing method and distributed system for AI task | |
| CN116860723B (en) | Cross-computing center data migration method | |
| CN109766168B (en) | Task scheduling method and device, storage medium and computing equipment | |
| CN111143063A (en) | Task resource reservation method and device | |
| CN111143210A (en) | Test task scheduling method and system | |
| CN109189581B (en) | Job scheduling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |