CN115617520A

CN115617520A - Resource parameter configuration method and device, electronic equipment and storage medium

Info

Publication number: CN115617520A
Application number: CN202211288909.2A
Authority: CN
Inventors: 竹祯铮; 付庆午
Original assignee: Shenzhen Huantai Technology Co Ltd
Current assignee: Shenzhen Huantai Technology Co Ltd
Priority date: 2022-10-20
Filing date: 2022-10-20
Publication date: 2023-01-17

Abstract

The disclosure provides a resource parameter configuration method, a resource parameter configuration device, electronic equipment and a computer readable storage medium, and relates to the technical field of big data. The resource parameter configuration method comprises the following steps: acquiring a prediction configuration parameter of an execution stage of the current operation according to the resource information of the historical task; based on the prediction configuration parameters of each execution stage, adopting a global resource adjustment strategy to adjust the prediction configuration parameters of each execution stage to obtain reference configuration parameters of each execution stage; and matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determining the target configuration parameters of each execution stage according to the matching result so as to configure the resource parameters based on the target configuration parameters. The method and the device can improve the resource parameter configuration efficiency and improve the resource utilization rate.

Description

Resource parameter configuration method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of big data technologies, and in particular, to a resource parameter configuration method, a resource parameter configuration apparatus, an electronic device, and a computer-readable storage medium.

Background

With the rapid development of the big data technology field, the resource optimization of the computing engine is concerned, and the configuration of the resource parameters is directly related to the allocable resources of the computing engine. If the resource parameter setting of the big data engine Spark is reasonable or not, not only the performance release is influenced, but also the overhead of the cluster resources is influenced.

At present, the resource parameter configuration method relying on manual configuration has the defects of low efficiency and accuracy, and influences the reasonable configuration of computing resources to a certain extent.

Disclosure of Invention

The present disclosure provides a resource parameter configuration method, a resource parameter configuration apparatus, an electronic device, and a computer-readable storage medium, so as to improve efficiency and accuracy of resource parameter configuration at least to a certain extent.

According to a first aspect of the present disclosure, a resource parameter configuration method is provided, including: acquiring a prediction configuration parameter of an execution stage of the current operation according to resource information of a historical task; based on the predicted configuration parameters of each execution stage, adopting a global resource adjustment strategy to adjust the predicted configuration parameters of each execution stage to obtain reference configuration parameters of each execution stage; and matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determining the target configuration parameters of each execution stage according to the matching result so as to configure the resource parameters based on the target configuration parameters.

According to a second aspect of the present disclosure, there is provided a resource parameter configuration apparatus, including: the parameter prediction module is used for acquiring the prediction configuration parameters of the execution stage of the current operation according to the resource information of the historical task; a first parameter processing module, configured to adjust the predicted configuration parameters of each execution stage by using a global resource adjustment policy based on the predicted configuration parameters of each execution stage, to obtain reference configuration parameters of each execution stage; and the second parameter processing module is used for matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, determining the target configuration parameters of each execution stage according to the matching result, and configuring the resource parameters based on the target configuration parameters.

According to a third aspect of the present disclosure, there is provided an electronic device comprising: a processor; and memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the above-described method.

According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the above-mentioned method.

In the technical solutions provided in some embodiments of the present disclosure, on one hand, the predicted configuration parameters of the execution stage of the current job are predicted according to the resource information of the computation engine executing the historical task, so that the predicted configuration parameters increase the performance on the historical resource information, reduce errors to a certain extent, and break through a resource adjustment mechanism for performing global adjustment on the resource parameters of the execution job. On the other hand, the predicted configuration parameters of each execution stage are adjusted by adopting a global resource adjustment strategy, so that the reference configuration parameters of each execution stage can be in smooth transition globally, and the extra expense caused by repeated application and destruction of the container when the resources are subsequently applied is avoided; on the other hand, the reference configuration parameters are matched with the parameter information of the corresponding stage, the target configuration parameters of the execution stage are determined according to the matching result, the stage information corresponding to the target configuration parameters can be ensured not to be overdue, and the accuracy of the target configuration parameters of each execution stage is improved, so that the resource regulation and control process can be accurately carried out subsequently; in addition, the resource parameter configuration method of the embodiment of the disclosure realizes the automatic configuration of the resource parameters of each execution stage based on the resource information of the historical task, and improves the resource parameter configuration efficiency.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty. In the drawings:

fig. 1 shows a schematic diagram of stages involved in a resource parameter configuration scheme of an embodiment of the present disclosure;

fig. 2 schematically shows a resource allocation process diagram of the related art;

FIG. 3 schematically illustrates a flow chart of a resource parameter configuration method in an exemplary embodiment of the present disclosure;

fig. 4 is a schematic diagram illustrating a hierarchical distribution of Spark memories in an exemplary embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of one implementation of obtaining predicted configuration parameters in an exemplary embodiment of the present disclosure;

fig. 6 is a schematic diagram illustrating matching of current memory overhead and preset parameter rules in an exemplary embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of one implementation of obtaining reference configuration parameters in an exemplary embodiment of the present disclosure;

fig. 8 is a schematic diagram illustrating obtaining reference configuration parameters corresponding to predicted memory parameters according to an exemplary embodiment of the disclosure;

FIG. 9 is a diagram schematically illustrating a reference configuration parameter corresponding to each of the plurality of predicted memory parameters corresponding to FIG. 8 according to an exemplary embodiment of the disclosure;

FIG. 10 schematically illustrates a flow chart of one implementation of obtaining target configuration parameters for various execution phases in an exemplary embodiment of the present disclosure;

FIG. 11 is a schematic diagram illustrating one method of obtaining target configuration parameters in an exemplary embodiment of the present disclosure;

FIG. 12 is a flow diagram that schematically illustrates a process for resource parameter configuration based on a resource specification, in an exemplary embodiment of the disclosure;

FIG. 13 is a block diagram schematically illustrating a resource parameter configuration system in an exemplary embodiment of the present disclosure;

FIG. 14 is a schematic diagram illustrating interaction of modules involved in configuring a resource parameter in an exemplary embodiment of the present disclosure;

fig. 15 is a block diagram schematically illustrating a resource parameter configuration apparatus in an exemplary embodiment of the present disclosure;

fig. 16 shows a schematic diagram of an electronic device to which an embodiment of the present disclosure may be applied.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

Fig. 1 shows a schematic diagram of stages involved in a resource parameter configuration scheme according to an embodiment of the present disclosure. The resource parameter configuration scheme comprises resource parameter configuration, resource parameter submission and a resource regulation and control process based on the resource parameters.

And configuring the resource parameters of each execution stage of the current operation in the resource parameter configuration process, and submitting the resource parameters after the configuration is finished so as to perform a resource regulation and control process based on the submitted resource parameters. In the resource parameter configuration process, the execution stage dimension is taken as a logic configuration unit to distinguish the resource requirements of each execution stage, the number of the execution stages of the current operation is obtained according to the actual condition of the current operation, and no special limitation is made on the number.

It should be noted that the resource parameter configuration method in the embodiment of the present disclosure is applicable to configuration of various resource parameters of Spark, such as a memory parameter, a CPU parameter, and a task concurrency amount, and the following description is made with respect to a configuration process of a certain type of resource parameter.

In the related art, as shown in fig. 2, spark starts a dynamic executor distributor at the same time when starting, and the dynamic executor distributor schedules and determines whether to release or apply for an executor based on the currently registered executor (container with a certain resource specification) resources. The dynamic executor distributor describes the Resource information of the executor through a Resource description file and applies for the executor based on the Resource information. And when the Spark submits the phase, the event in the submitting phase is transmitted to the dynamic executor distributor through the event bus, so that the dynamic executor distributor progressively creates executors with corresponding Resource quantity through the Resource Profile of the phase. When the actuator is idle for a certain time, the dynamic actuator distributor releases the actuator. It is worth noting that the above architecture is based on Spark3, and the dynamic resource allocation mechanism has been turned on.

However, the resource scheduling native to Spark has the following drawbacks: according to the resource adjustment mechanism for performing resource parameter adjustment on the whole operation, all actuators share one resource Profile, namely the resource rules of the actuators are the same on the whole. The resource requirements are different at different stages of SQL (Structured Query Language) operation execution, which leads to resource waste to a certain extent; the demands of the operation on the resources are different, the resource parameter configuration cannot be carried out in combination with the resource demands of the operation, the spare configuration parameters are more, the mismatch probability is increased through manual inspection, and higher requirements are put forward for a configurator.

Based on one or more of the problems described above, the present exemplary embodiment provides a resource parameter configuration method. Referring to fig. 3, the resource parameter configuration method may include the following steps S310 to S330:

in step S310, the predicted configuration parameters of the execution phase of the current job are acquired according to the resource information of the historical task.

In an exemplary embodiment of the present disclosure, in order to meet different resource requirements of different jobs, resource information of a historical task may be acquired, where the resource information of the historical task is overhead conditions of various types of resources of the historical task, including but not limited to memory overhead, CPU overhead, concurrent task amount, and the like. The acquisition period of the resource information of the historical tasks can be set according to the actual parameter configuration requirements, and no special limitation is imposed on the acquisition period.

Resource information of historical Spark SQL tasks can be collected in Spark, and reported resource information is collected in a point burying mode. The distributed search and analysis engine Elastic search can be used as a component for data query and analysis, and full-text retrieval is carried out on the basis of the inverted index of the Elastic search to analyze the acquired resource information of the historical tasks.

Taking memory overhead of a history task as an example, fig. 4 shows a Spark memory hierarchy distribution diagram according to an exemplary embodiment of the present disclosure, which includes five layers from a bottom layer to a top layer, and may mainly collect memory settings in the third layer, such as an "execute.memory" parameter corresponding to an on-heap part, an off-heap part corresponding to an "execute.memory.off-heap.size" parameter, and a py memory part corresponding to an "execute.py.memory" parameter of a py Spark. Wherein, because the part outside the heap is the direct memory, can count directly; the in-heap part is managed by a JVM (Java Virtual Machine), the memory in use may be referenced and cannot be recovered, and the memory overhead can be indirectly obtained by establishing a garbage recovery task and counting the remaining memory after executing the garbage recovery task, so that the direct use of the memory index built in the Spark is avoided, because the memory index built in the Spark is statistically obtained in the program, and a certain error exists.

It should be noted that, the above is only an example of obtaining the memory overhead of the historical task, and the embodiments of the present disclosure may obtain the overhead conditions of various resources according to actual needs, which is not listed here.

In step S320, the predicted configuration parameters of each execution stage are adjusted by using a global resource adjustment policy based on the predicted configuration parameters of each execution stage, so as to obtain reference configuration parameters of each execution stage.

In the exemplary embodiment of the present disclosure, the global policy adjustment policy refers to a policy for adjusting the predicted configuration parameters of each execution stage from the global perspective of the current job, the reference configuration parameter refers to a parameter that allows Spark to apply for sufficient resources first and then release the resources in sequence when applying for the resources, and may be determined according to the specific type of the predicted configuration parameter. The predicted configuration information of each execution stage is obtained based on the resource information of the historical task, and from the perspective of resource utilization, the costs of creating and destroying the cluster container need to be further considered, so as to improve the performance of the finally obtained target configuration parameters on improving the resource utilization efficiency.

The predicted configuration parameters of each execution stage may be optimized through an extremum optimization strategy, so that the optimized reference configuration parameters are smoothly transited in each execution stage, which is specifically described below.

On the basis of the predicted configuration parameters of each execution stage, a global resource adjustment strategy is adopted to optimize the predicted configuration parameters of each execution stage from the global perspective, so that fluctuation and uneven distribution of numerical values of the predicted configuration parameters obtained on the basis of resource information of historical tasks are avoided, and further, frequent application and destruction of containers, which are reflected on Spark resource application, on data in an irregular manner are reduced.

Step S330: and matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determining the target configuration parameters of each execution stage according to the matching result so as to configure the resource parameters based on the target configuration parameters.

In an exemplary embodiment of the present disclosure, the current stage parameter information refers to current information of the execution stage, and in order to avoid that the obtained target configuration parameter does not match with the execution stage, the reference configuration parameter of the execution stage may be matched with the current stage parameter information of the execution stage to obtain the target configuration parameter matching with the execution stage.

And determining the current stage parameter information according to the type of the reference configuration parameter. For example, the current phase parameter information may include an execution phase identifier (Stage ID), an execution phase Name (Stage Name), and an elastic Distributed data set identifier (RDD Names), which are not listed.

The reference configuration parameters are checked and adjusted by using the current stage parameter information of each execution stage, so that the target configuration parameters of each execution stage are ensured to be in smooth transition on the whole, and the target configuration parameters are ensured to be adapted to the execution stages, thereby improving the accuracy of the target configuration parameters of each execution stage and being beneficial to the subsequent reasonable resource scheduling based on the target configuration parameters.

In an exemplary embodiment, an implementation is provided for obtaining predicted configuration parameters. Acquiring the predicted configuration parameters of the execution stage of the current job according to the resource information of the historical task may include step S510 and step S520:

step S510: and acquiring current resource occupation information of the execution stage.

The current resource occupation information refers to the actual cost of the current resource in the execution phase, such as the actual memory cost. The current resource occupation information may be index information built in Spark, or information obtained by analyzing the resource occupation information in each execution stage, and a corresponding mode may be selected according to the type of the resource parameter to be configured.

Step S520: and matching the current resource occupation information with preset parameter rule information, and determining a prediction configuration parameter of an execution stage by combining a matching result and the resource information of the historical task.

The preset parameter rule information may be rule information obtained by summarizing a common scenario of resource configuration. Illustratively, the preset parameter rule information corresponding to the Memory parameter includes whether the current resource occupation information is smaller than a preset minimum parameter threshold, whether an analysis result of Spark SQL includes a key node (e.g., broadcast Exchange, in Memory Table Scan) for indicating that the Memory has been optimized, and the like.

In some possible embodiments, in response to that the current resource occupation information does not match the preset parameter rule information, phase resource feature data of the execution phase is acquired according to the resource information of the historical task, and time series prediction is performed according to the phase resource feature data to acquire a predicted configuration parameter of the execution phase.

If the current resource occupation information does not match the preset parameter rule, the prediction configuration parameters of the execution stage can be obtained by adopting a time sequence prediction algorithm. The stage resource feature data is a key feature extracted from the resource information of the historical task and is used for reflecting the resource information of the historical task. The timing prestoring algorithm may adopt ARIMA (differential Integrated Moving Average Autoregressive model), LSTM (Long Short-Term Memory network), and the like.

Taking an ARIMA algorithm as an example, a time sequence of the phase resource characteristic data can be obtained according to the phase resource characteristic data, the time sequence is an unbalanced time sequence, a relatively stable time sequence of the resource characteristic data is obtained by differentially processing the time sequence, and then the time sequence is predicted by using the time sequence of the resource characteristic data to obtain a predicted configuration parameter of the execution phase.

Time series prediction is carried out through the stage resource characteristic data, so that the performance of the prediction configuration parameters on historical resource information is increased, the efficiency of obtaining the prediction configuration parameters is high, and the error of the prediction configuration parameters is reduced to a certain degree.

In some possible embodiments, if the current resource occupation information matches the preset parameter rule information, the predicted configuration parameters of the execution stage may be determined according to the preset parameter rule information. Taking the example that the predicted configuration parameters include predicted memory parameters and the current resource occupation information includes current memory overhead, fig. 6 shows a schematic diagram of matching between the current memory overhead and preset parameter rules according to an exemplary embodiment of the present disclosure.

As shown in fig. 6, the current memory overhead at least includes the in-heap memory after performing a Garbage Collection (GC) and the default reserved memory. The in-heap memory after the GC task is executed is the actual memory overhead in the execution stage, and the reserved memory is reserved by default to a preset basic operation reserved memory. Comparing the current memory overhead with a preset minimum parameter threshold (such as 450 megabits), and if the current memory overhead is less than 450 megabits, directly determining that the predicted memory parameter of the execution stage is 450 megabits. The preset minimum parameter threshold may be set according to an actual situation, and is not particularly limited.

Because the part in the heap is managed by the JVM, the used memory can not be recycled due to reference, and the residual space in the heap can be counted after the GC task by monitoring the GC task to be used as the actual memory overhead. Specifically, the method comprises the steps of monitoring and acquiring the size of the in-pile memory after each GC task through a JMX interface, acquiring the maximum value of the whole operation life cycle, and determining the actual memory cost according to the value. Wherein, the preset proportion is output through the processing module 1 and the processing module 2.

According to the embodiment of the invention, the current resource occupation information is matched with the preset parameter rule information, and the matching result and the resource information of the historical task are combined to determine the prediction configuration parameters in the execution stage, so that the determination efficiency of the prediction configuration parameters is improved, and the relatively accurate configuration parameters are provided to a certain extent.

In an exemplary embodiment, in order to further improve resource utilization, an implementation manner of obtaining the reference configuration parameter is further provided. As shown in fig. 7, based on the predicted configuration parameters of each execution stage, adjusting the predicted configuration parameters of each execution stage by using a global resource adjustment policy, and obtaining the reference configuration parameters of each execution stage may include:

step S710: and generating a prediction memory parameter array according to the execution sequence of the corresponding execution stage by using each prediction memory parameter.

And generating pre-stored memory parameter data, such as 450,1240,3650,2480 … …, according to the execution sequence of the corresponding execution stage according to the obtained predicted memory parameters of each execution stage.

Step S720: and circularly executing the following processes until each predicted memory parameter in the predicted memory parameter array determines the corresponding reference configuration parameter.

Wherein, step S7210: the method comprises the steps of obtaining the maximum prediction memory parameter in a prediction memory parameter array, forming a first trend group according to the maximum prediction memory parameter and the prediction memory parameter which is positioned in front of the maximum prediction memory parameter in the prediction memory parameter array, and forming a second trend group according to the prediction memory parameter which is positioned in back of the maximum prediction memory parameter in the prediction memory parameter array.

Fig. 8 is a schematic diagram illustrating obtaining reference configuration parameters corresponding to predicted memory parameters according to an exemplary embodiment of the present disclosure, where as shown in fig. 8, the abscissa is each execution stage corresponding to a predicted memory parameter in the predicted memory parameter array, and the ordinate is a specific numerical value of each predicted memory parameter in the predicted memory parameter array. And traversing the predicted memory parameters of each execution stage to obtain the maximum predicted memory parameter (parameter A), forming the parameter A and the predicted memory parameters before the parameter A into a first trend group, and forming the predicted memory parameters after the parameter A into a second trend group.

Step S7220: and updating each predicted memory parameter in the first trend group to be the maximum predicted memory parameter corresponding to the first trend group so as to obtain the reference configuration parameter corresponding to each predicted memory parameter in the first trend group.

Referring to fig. 8, the predicted memory parameter before the parameter a is updated to the value corresponding to the parameter a, that is, the value corresponding to the parameter a is used as the reference configuration parameter of each execution stage corresponding to the first trend group.

Step S7230: the second trend group is updated to the prediction memory parameter array, and the step S720 is returned to.

As shown in fig. 8, a predicted memory parameter array is formed according to the predicted memory parameters after the parameter a, and the process returns to step S720 to obtain the maximum predicted memory parameter in the predicted memory parameter data again, which is not described herein again.

Through steps S710 to S7230, the predicted memory parameters of each execution stage are traversed, each predicted memory parameter is divided into a plurality of trend groups, and the reference configuration parameter corresponding to each predicted memory parameter in each trend group is the value of the largest predicted memory parameter in the group.

As shown in fig. 9, which is a schematic diagram of the reference configuration parameters corresponding to the predicted memory parameters corresponding to fig. 8, the reference configuration parameters of each execution stage obtained by adjusting the predicted configuration parameters of each execution stage are in a sequentially decreasing trend, so that the reference configuration parameters of each execution stage are in smooth transition, thereby avoiding additional overhead caused by repeated application and destruction of the container during subsequent resource application, and moreover, by using the maximum predicted memory parameter of each trend group as the reference configuration parameter of each predicted memory parameter of the corresponding trend group, the Spark applies for sufficient resources first when applying for the resources, and then releases the resources in sequence, thereby reducing the initialization frequency of the container and improving the resource utilization rate.

In an exemplary embodiment, an implementation manner of acquiring target configuration parameters of each execution phase is also provided. As shown in fig. 10, matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determining the target configuration parameters of each execution stage according to the matching result may include steps S1010 to S1030:

step S1010: and aiming at any target execution stage, matching the reference configuration parameters of the target execution stage with the corresponding current stage parameter information.

The current stage parameter information is determined according to the type of the parameter to be configured, and includes, but is not limited to, an execution stage identifier, an execution stage name, and an elastic distributed data set identifier.

Step S1020: and if the reference configuration parameters of the target execution stage are completely matched with the corresponding current stage parameter information, determining the reference configuration parameters of the target execution stage as the target configuration parameters of the target execution stage.

If the reference configuration parameters of the target execution stage are completely matched with the corresponding current stage parameter information, it is indicated that the relevant information of the target execution stage is completely applicable to the reference configuration information of the target execution stage, i.e. the reference configuration information is not expired for the target execution stage.

Step S1030: and if the reference configuration parameters of the target execution stage are not completely matched with the corresponding current stage parameter information, determining the target configuration parameters of the target execution stage according to a preset stage matching rule.

In some possible embodiments, if there is a first reference configuration parameter matching the execution stage identifier, matching the execution stage name, and the first reference configuration parameter not matching the flexible distributed data set identifier, a second reference configuration parameter having the same execution stage name as the first reference configuration parameter is obtained, a target reference configuration parameter having the largest value among the second reference configuration parameters is obtained, and the target reference configuration parameter is determined as the target configuration parameter of the target execution stage.

In this case, it is possible that the relevant information of the target execution phase has changed, but the executed SQL may not change much, and the first reference configuration parameter still has reference value. As shown in fig. 11, according to an exemplary embodiment of the disclosure, when the target execution stage is "stage 2", and the first reference configuration parameter matches with the execution stage identifier, the first reference configuration parameter matches with the execution stage name, and the first reference configuration parameter does not match with the flexible distributed data set identifier, all the second reference configuration parameters having the same execution stage name "stage name 01" as "stage 2", including the second reference configuration parameters of "stage 0", "stage 1", "stage 3", and "stage 5", are acquired, and "stage 3" (with the largest value) is determined as the target configuration parameter of "stage 2". Other stages are similar and will not be described again.

Based on this, it is possible to cope with an increase in memory demand caused by a change in the execution plan in the target execution stage.

In some possible embodiments, if there is a match between the third reference configuration parameter and the execution stage identifier, and the third reference configuration parameter is not matched between the execution stage name and the elastic distributed data set identifier, the global reference configuration parameter with the largest value among the reference configuration parameters of each execution stage is obtained, and the global reference configuration parameter is determined as the target configuration parameter of the target execution stage.

Continuing with fig. 11, when the target execution stage is "stage 4", the third reference configuration parameter of "stage 4" matches the execution stage identifier, and the third reference configuration parameter does not match the execution stage name and the flexible distributed data set identifier, the reference configuration parameter of "stage 3" with the largest value among the reference configuration parameters in all the execution stages is obtained as the global reference configuration parameter, and is determined as the target configuration parameter of "stage 4".

And in order to ensure that the execution task of the target execution stage does not report errors, selecting the global reference configuration parameter as the target configuration parameter of the target execution stage so as to ensure the stability of the resource regulation and control process.

In an exemplary embodiment, an implementation manner of resource parameter configuration based on a target configuration parameter is further provided, which may include:

and acquiring current resource data of a current execution stage, wherein the current resource data is the resource data applied by the current execution stage.

And comparing the current resource data with the target configuration parameters of the current execution stage, determining the resource specification of the task executor according to the comparison result, and performing resource parameter configuration processing based on the resource specification.

The current resource data of the current execution stage is the expected resource overhead of the current execution stage, and comprises the resource data which is applied in the current execution stage.

Optionally, in the embodiment of the present disclosure, since the execution stages are used as the logic configuration unit, the application and the release of the resource may be performed in each execution stage, and when the application in the current execution stage is submitted, the resource in the previous stage may not be completely applied, in order to prevent excessive application of the resource, the resource application data corresponding to the pending state may also be counted globally, and the current resource data is determined according to the obtained resource application data and the resource data already applied in the current execution stage. If the resource application is finished, the Spark Listener can be used to monitor whether the resource application is finished, the current resource data is updated according to the monitoring result, and the monitored resource application corresponding to the pending state in the memory overhead of the calculation stage is added into the statistical range of the current resource data, so that the fine control of the resource data is realized.

In some possible embodiments, the resource specification is increased if the current resource data is less than the target configuration parameter for the current execution phase.

In some possible embodiments, the resource specification is reduced if the current resource data is greater than the target configuration parameter of the current execution stage.

In some possible embodiments, if the difference between the current resource data and the target configuration parameter of the current execution stage is within a preset threshold range, the resource specification of the task executor is kept unchanged.

Referring to fig. 12, a flowchart of a resource parameter configuration process based on resource specifications according to an exemplary embodiment of the present disclosure is shown. Taking the memory parameters as an example, fig. 12 may include the following steps:

step 1210: the current resource data is compared to the target configuration parameters for the current execution phase.

Step S1220: and judging whether the current resource data (current memory) has spare memory.

Step S1230: if the current resource data has redundant memory of one actuator, destroying the redundant actuator and correcting the existing concurrency of the actuator.

Step S1240: if the current resource data lacks the memory of one executor, an executor is tried to be added.

Step S1250: if the difference value between the current resource data and the target configuration parameter of the current execution stage is within the preset threshold range, the resource specification of the task executor can be kept unchanged.

The preset threshold range can be set according to actual requirements, and is not particularly limited herein.

Step S1260: and applying for resources according to the processed result of the resource parameter configuration and submitting the resource to the execution stage.

If there is more than one memory of one actuator, the concurrent amount of the existing actuator can be modified to obtain the result after the resource parameter configuration processing. It should be noted that the resource description file in this step is an internal structure in Spark, and is not a file that can be really set by a user for modification, and a resource can be applied based on a result after resource parameter configuration processing.

In a possible embodiment, the preset virtual computing resource configuration information may be obtained in response to that the current resource data is smaller than the target configuration parameter of the current execution stage, and the number of concurrent tasks corresponding to the task executor is adjusted based on the preset virtual computing resource configuration information, so as to perform a resource configuration process according to the adjusted number of concurrent tasks.

In the native Spark, since each register has determined its CPU core (cores) before registering, but the memory required by the tasks at different execution stages is different, the actual memory of the concurrently full executor may not reach the limit, resulting in resource surplus. According to the method and the device, the preset virtual computing resource configuration information is added and used for the newly added CPU core to participate in concurrent computing of the actuator, but the virtual computing resource configuration information does not apply for actual resources, and tasks are additionally submitted to the actuator through a task Scheduler based on the virtual computing resource configuration information, so that the resources of the actuator can be fully utilized.

In addition, after the resource parameter configuration is performed based on the target configuration parameter, the resource at the corresponding execution stage can be regulated and controlled according to the obtained resource parameter configuration information.

Fig. 13 is a block diagram illustrating a resource parameter configuration system according to an exemplary embodiment of the present disclosure. As shown in fig. 13, the resource parameter configuration system of the exemplary embodiment of the present disclosure includes dag Scheduler (dag Scheduler), task Scheduler (task Scheduler), dynamic executor allocator (execution Allocation Manager), resource optimizer (tuning Manager), and external server (remote server). The resource optimizer is used for executing the parameter strategy based on the target configuration parameter, modifying the resource parameter and controlling the resource application and release. The external server can be used for analyzing the historical Spark SQL and acquiring resource information of the historical tasks to obtain stage resource characteristic data and the like. Wherein the architecture is based on Spark3 and has turned on the dynamic resource allocation mechanism.

Fig. 14 shows interaction diagrams of modules involved in resource parameter configuration according to an exemplary embodiment of the present disclosure, and a resource parameter configuration method according to an embodiment of the present disclosure is described below with reference to fig. 13 and 14 based on Spark.

After Spark submits an Application, the Application is constructed into a DAG directed acyclic graph according to the dependency relationship between RDDs, and Spark creates two objects at the Driver end: DAG Scheduler and Task Scheduler.

The DAG Scheduler is used for dividing the DAG into execution stages (such as S1, S2, S3, S4, S5 and the like) according to the width dependency relationship among the RDDs, and submitting the execution stages to the Task Scheduler in the form of a Task Set; the Task Scheduler traverses the Task Set, and sends the Task to the computing node Executor for execution after acquiring the corresponding Task. The parameter processing module sequentially performs processing through each execution stage, the number of the execution stages in fig. 14 is only exemplary, and the number of the execution stages may be determined according to an actual situation, which is not particularly limited in the embodiment of the present disclosure.

Before submitting an execution stage to a resource distributor, a parameter prediction module obtains prediction configuration parameters of the execution stage of a current job according to resource information of a historical task, and based on the prediction configuration parameters of each execution stage, a parameter processing module (comprising a first parameter processing module and a second parameter processing module) adjusts the prediction configuration parameters of each execution stage by adopting a global resource adjustment strategy to obtain reference configuration parameters of each execution stage, so that the reference configuration parameters of each execution stage are matched with the corresponding parameter information of the current stage, and target configuration parameters of each execution stage are determined according to a matching result, so that resource parameter optimization modification is realized, and uniform resource requests are controlled, namely parameter modification and resource requests are carried out on the basis of the target configuration parameters through the parameter execution module.

It should be noted that, the obtaining process of the target configuration parameter before submitting the execution phase to the resource allocator is already described in detail in the foregoing exemplary embodiment, and is not described herein again.

As can be seen from the above, the resource parameters of each execution stage are respectively configured by using the execution stages as the access point to control the use of resources, so that different resource appeal of each execution stage is considered, and the resource waste is reduced while the resource requirements are met; the predicted configuration parameters of each execution stage are adjusted by adopting a global resource adjustment strategy, so that the reference configuration parameters of each execution stage can be in smooth transition on the global, and the additional expense caused by repeated application and destruction of a container when resources are subsequently applied is avoided; predicting configuration parameters of the current operation in the execution stage according to the resource information of the historical task executed by the computing engine, so that the performance of the predicted configuration parameters on the historical resource information is increased, and the error is reduced to a certain degree; matching the reference configuration parameters with the corresponding stage parameter information, determining the target configuration parameters of the execution stages according to the matching result, ensuring that the stage information corresponding to the target configuration parameters is not overdue, and improving the accuracy of the target configuration parameters of each execution stage so as to accurately perform the resource regulation and control process in the following; the resource parameter configuration method of the embodiment of the disclosure realizes the automatic configuration of the resource parameters of each execution stage based on the resource information of the historical task, improves the resource parameter configuration efficiency, has no perception in parameter setting from the perspective of users, reduces the configuration threshold, and greatly improves the working efficiency of employees.

It is noted that the above-mentioned figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.

Further, referring to fig. 15, an exemplary embodiment of the present disclosure provides a resource parameter configuration apparatus 1500, which includes a parameter prediction module 1510, a first parameter processing module 1520 and a second parameter processing module 1530. Wherein:

the parameter prediction module 1510 is configured to obtain a prediction configuration parameter of an execution stage of a current job according to resource information of a historical task;

a first parameter processing module 1520, configured to adjust the predicted configuration parameters of each execution stage by using a global resource adjustment policy based on the predicted configuration parameters of each execution stage, to obtain reference configuration parameters of each execution stage;

the second parameter processing module 1530 is configured to match the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determine the target configuration parameters of each execution stage according to the matching result, so as to configure the resource parameters based on the target configuration parameters.

In an exemplary embodiment, the parameter prediction module 1510 includes:

the acquisition unit is used for acquiring the current resource occupation information of the execution stage;

and the first determining unit is used for matching the current resource occupation information with preset parameter rule information and determining the prediction configuration parameters of the execution stage by combining the matching result and the resource information of the historical task.

In an exemplary embodiment, the first determination unit is configured to: in response to the fact that the current resource occupation information is not matched with the preset parameter rule information, acquiring stage resource characteristic data of an execution stage according to the resource information of the historical task; and performing time series prediction according to the phase resource characteristic data to obtain a prediction configuration parameter of the execution phase.

In an exemplary embodiment, the first determining unit is configured to: and if the current resource occupation information is matched with the preset parameter rule information, determining the prediction configuration parameters of the execution stage according to the preset parameter rule information.

In an exemplary embodiment, the predicted configuration parameters include predicted memory parameters, and the current resource occupancy information includes current memory overhead;

the first determination unit is configured to: if the current memory overhead is matched with the preset parameter rule information, determining a predicted memory parameter of an execution stage according to the preset parameter rule information; the current memory overhead at least comprises the in-heap memory and the default reserved memory after the garbage collection task is executed.

In an exemplary embodiment, the first parameter processing module 1520 is configured to: generating a prediction memory parameter array by each prediction memory parameter according to the execution sequence of the corresponding execution stage; circularly executing the following processes until each predicted memory parameter in the predicted memory parameter array determines the corresponding reference configuration parameter: acquiring the maximum predicted memory parameter in the predicted memory parameter array, forming a first trend group according to the maximum predicted memory parameter and the predicted memory parameter in the predicted memory parameter array, which is positioned before the maximum predicted memory parameter, and forming a second trend group according to the predicted memory parameter in the predicted memory parameter array, which is positioned after the maximum predicted memory parameter; updating each predicted memory parameter in the first trend group to be the maximum predicted memory parameter corresponding to the first trend group so as to obtain a reference configuration parameter corresponding to each predicted memory parameter in the first trend group; and updating the second trend group into a prediction memory parameter array.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: aiming at any target execution stage, matching the reference configuration parameters of the target execution stage with the corresponding parameter information of the current stage; if the reference configuration parameters of the target execution stage are completely matched with the corresponding parameter information of the current stage, determining the reference configuration parameters of the target execution stage as the target configuration parameters of the target execution stage; and if the reference configuration parameters of the target execution stage are not completely matched with the corresponding current stage parameter information, determining the target configuration parameters of the target execution stage according to a preset stage matching rule.

In an exemplary embodiment, the current phase parameter information includes an execution phase identifier, an execution phase name, and a flexible distributed data set identifier; the second parameter processing module 1530 is configured to: if the first reference configuration parameter is matched with the execution stage identifier and the execution stage name and the first reference configuration parameter is not matched with the elastic distributed data set identifier, acquiring a second reference configuration parameter with the same execution stage name as the first parameter configuration parameter; and acquiring the target reference configuration parameter with the largest value in the second reference configuration parameters, and determining the target reference configuration parameter as the target configuration parameter of the target execution stage.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: if the third reference configuration parameter is matched with the execution stage identifier and the third reference configuration parameter is not matched with the execution stage name and the elastic distributed data set identifier, acquiring the global reference configuration parameter with the largest value in the reference configuration parameters of each execution stage; and determining the global reference configuration parameter as a target configuration parameter of the target execution stage.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: acquiring current resource data of a current execution stage, wherein the current resource data is the resource data applied by the current execution stage; and comparing the current resource data with the target configuration parameters of the current execution stage, determining the resource specification of the task executor according to the comparison result, and performing resource parameter configuration processing based on the resource specification.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: if the current resource data is smaller than the target configuration parameter of the current execution stage, increasing the resource specification; if the current resource data is larger than the target configuration parameter of the current execution stage, reducing the resource specification; and if the difference value between the current resource data and the target configuration parameter of the current execution stage is within the preset threshold range, the resource specification of the task executor is kept unchanged.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: responding to the situation that the current resource data is smaller than the target configuration parameter of the current execution stage, and acquiring preset virtual computing resource configuration information; and adjusting the quantity of the concurrent tasks corresponding to the task executor based on the preset virtual computing resource configuration information, so as to perform a resource configuration process according to the adjusted quantity of the concurrent tasks.

In an exemplary embodiment, the second parameter processing module 1530 is configured to: and counting the resource application data corresponding to the pending state in the global state, and determining the current resource data according to the obtained resource application data and the resource data applied in the current execution stage.

In an exemplary embodiment, the resource parameter configuration apparatus further includes:

and the parameter execution module is used for regulating and controlling the resources in the corresponding execution stage according to the acquired parameter configuration information of each resource.

The specific details of each module in the above apparatus have been described in detail in the method section, and details that are not disclosed may refer to the method section, and thus are not described again.

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

An electronic device for the method is also provided in an exemplary embodiment of the present disclosure, and the electronic device may be the above-mentioned imaging device or the server. Generally, the electronic device comprises at least a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the above-mentioned method via execution of the executable instructions.

The following takes the mobile terminal 1600 in fig. 16 as an example, and exemplifies the configuration of the electronic device in the embodiment of the present disclosure. It will be appreciated by those skilled in the art that the configuration of figure 16 can be applied to fixed type devices, in addition to components specifically intended for mobile purposes. In other embodiments, mobile terminal 1600 may include more or fewer components than shown, or combine certain components, or split certain components, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. The interfacing relationship between the various components is shown schematically and is not meant to be a structural limitation of mobile terminal 1600. In other embodiments, the mobile terminal may also interface differently than shown in fig. 16, or a combination of multiple interfaces.

As shown in fig. 16, the mobile terminal 1600 may specifically include: a processor 1601, a memory 1602, a bus 1603, a mobile communication module 1604, an antenna 1, a wireless communication module 1605, an antenna 2, a display 1606, a camera module 1607, an audio module 1608, a power module 1609, a sensor module 1610, and a photovoltaic sensor module 1611.

Processor 1601 may include one or more processing units, such as: the Processor 1601 may include an AP (Application Processor), a modem Processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband Processor, and/or an NPU (Neural-Network Processing Unit), etc.

An encoder may encode (i.e., compress) an image or video to reduce the data size for storage or transmission. The decoder may decode (i.e., decompress) the encoded data for the image or video to recover the image or video data. The mobile terminal 1600 may support one or more encoders and decoders.

The processor 1601 may be connected with the memory 1602 or other components via a bus 1603.

Memory 1602 may be used to store computer-executable program code, which includes instructions. The processor 1601 performs various functional applications and data processing of the mobile terminal 1600 by executing instructions stored in the memory 1602. The memory 1602 may also store application data, such as cache device credential information and the like.

The communication function of the mobile terminal 1600 may be implemented by the mobile communication module 1604, the antenna 1, the wireless communication module 1605, the antenna 2, a modem processor, a baseband processor, and the like. The

antennas

1 and 2 are used for transmitting and receiving electromagnetic wave signals. The mobile communication module 1604 may provide mobile communication solutions of 3G, 4G, 5G, etc. applied to the mobile terminal 1600. The wireless communication module 1605 may provide wireless communication solutions such as wireless local area network, bluetooth, near field communication, etc. applied to the mobile terminal 1600.

The display screen 1606 is used to implement display functions, such as displaying a user interface, images, videos, and the like, and displaying exception notification information. The camera module 1607 is used to implement a shooting function, such as shooting images, videos, etc., to capture images of a scene. The audio module 1608 is used for implementing audio functions, such as playing audio, collecting voice, and the like. The power module 1609 is used to implement power management functions, such as charging a battery, powering a device, monitoring battery status, etc. The sensor module 1610 may include one or more sensors for implementing corresponding sensing functions.

Furthermore, the exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-described method of the present specification is stored. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the above-mentioned "exemplary methods" section of this specification, when the program product is run on the terminal device.

It should be noted that the computer readable media shown in the present disclosure may be computer readable signal media or computer readable storage media or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Furthermore, program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A resource parameter configuration method is characterized by comprising the following steps:

acquiring a prediction configuration parameter of an execution stage of the current operation according to the resource information of the historical task;

based on the prediction configuration parameters of each execution stage, adopting a global resource adjustment strategy to adjust the prediction configuration parameters of each execution stage to obtain reference configuration parameters of each execution stage;

and matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, and determining the target configuration parameters of each execution stage according to the matching result so as to configure the resource parameters based on the target configuration parameters.

2. The method according to claim 1, wherein the obtaining of the predicted configuration parameters of the execution phase of the current job according to the resource information of the historical task comprises:

acquiring current resource occupation information of the execution stage;

and matching the current resource occupation information with preset parameter rule information, and determining the prediction configuration parameters of the execution stage by combining the matching result and the resource information of the historical task.

3. The method according to claim 2, wherein the matching the current resource occupation information with preset parameter rule information, and determining the predicted configuration parameters of the execution stage by combining the matching result and the resource information of the historical task, comprises:

responding to the mismatching of the current resource occupation information and the preset parameter rule information, and acquiring stage resource characteristic data of the execution stage according to the resource information of the historical task;

and performing time series prediction according to the stage resource characteristic data to obtain a prediction configuration parameter of the execution stage.

4. The method according to claim 2, wherein the matching the current resource occupation information with preset parameter rule information, and determining the predicted configuration parameters of the execution stage by combining the matching result and the resource information of the historical task, comprises:

and if the current resource occupation information is matched with the preset parameter rule information, determining the prediction configuration parameters of the execution stage according to the preset parameter rule information.

5. The method of claim 4, wherein the predicted configuration parameters comprise predicted memory parameters, and wherein the current resource occupancy information comprises current memory overhead;

if the current resource occupation information is matched with the preset parameter rule information, determining the prediction configuration parameters of the execution stage according to the preset parameter rule information, including:

if the current memory overhead is matched with the preset parameter rule information, determining the predicted memory parameters of the execution stage according to the preset parameter rule information;

the current memory overhead at least comprises an in-heap memory and a default reserved memory after the garbage collection task is executed.

6. The method of claim 5, wherein the adjusting the predicted configuration parameters of each execution phase by using a global resource adjustment policy based on the predicted configuration parameters of each execution phase to obtain the reference configuration parameters of each execution phase comprises:

generating a prediction memory parameter array by each prediction memory parameter according to the execution sequence of the corresponding execution stage;

circularly executing the following processes until each predicted memory parameter in the predicted memory parameter array determines the corresponding reference configuration parameter:

acquiring the maximum predicted memory parameter in the predicted memory parameter array, forming a first trend group according to the maximum predicted memory parameter and the predicted memory parameter in the predicted memory parameter array before the maximum predicted memory parameter, and forming a second trend group according to the predicted memory parameter in the predicted memory parameter array after the maximum predicted memory parameter;

updating each predicted memory parameter in the first trend group to be the maximum predicted memory parameter corresponding to the first trend group to obtain a reference configuration parameter corresponding to each predicted memory parameter in the first trend group;

and updating the second trend group into the prediction memory parameter array.

7. The method according to claim 1, wherein the matching the reference configuration parameters of each of the execution stages with the corresponding current stage parameter information, and determining the target configuration parameters of each of the execution stages according to the matching result comprises:

aiming at any target execution stage, matching the reference configuration parameters of the target execution stage with the corresponding parameter information of the current stage;

if the reference configuration parameters of the target execution stage are completely matched with the corresponding current stage parameter information, determining the reference configuration parameters of the target execution stage as the target configuration parameters of the target execution stage;

and if the reference configuration parameters of the target execution stage are not completely matched with the corresponding current stage parameter information, determining the target configuration parameters of the target execution stage according to a preset stage matching rule.

8. The method of claim 7, wherein the current phase parameter information comprises an execution phase identifier, an execution phase name, and a flexible distributed data set identifier;

if the reference configuration parameter of the target execution stage is not completely matched with the corresponding current stage parameter information, determining the target configuration parameter of the target execution stage according to a preset stage matching rule, including:

if a first reference configuration parameter is matched with the execution stage identifier, matched with the execution stage name and not matched with the elastic distributed data set identifier, acquiring a second reference configuration parameter with the same execution stage name as the first parameter configuration parameter;

and acquiring a target reference configuration parameter with the maximum value in the second reference configuration parameters, and determining the target reference configuration parameter as a target configuration parameter of the target execution stage.

9. The method of claim 8, wherein if the reference configuration parameter of the target execution stage is not completely matched with the corresponding current stage parameter information, determining the target configuration parameter of the target execution stage according to a preset stage matching rule, further comprising:

if a third reference configuration parameter is matched with the execution stage identifier and the third reference configuration parameter is not matched with the execution stage name and the elastic distributed data set identifier, acquiring a global reference configuration parameter with the largest value in the reference configuration parameters of each execution stage;

and determining the global reference configuration parameters as target configuration parameters of the target execution stage.

10. The method of claim 1, wherein the configuring the resource parameter based on the target configuration parameter comprises:

acquiring current resource data of a current execution stage, wherein the current resource data is the resource data applied by the current execution stage;

and comparing the current resource data with the target configuration parameters of the current execution stage, and determining the resource specification of the task executor according to the comparison result so as to perform resource parameter configuration processing based on the resource specification.

11. The method of claim 10, wherein comparing the current resource data with the target configuration parameters of the current execution stage and determining the resource specification of the task executor according to the comparison result, so as to perform the resource parameter configuration process based on the resource specification, comprises:

if the current resource data is smaller than the target configuration parameter of the current execution stage, increasing the resource specification;

if the current resource data is larger than the target configuration parameter of the current execution stage, reducing the resource specification;

and if the difference value between the current resource data and the target configuration parameter of the current execution stage is within a preset threshold range, the resource specification of the task executor is kept unchanged.

12. The method of claim 10, wherein comparing the current resource data with the target configuration parameters of the current execution stage and determining the resource specification of the task executor according to the comparison result, so as to perform the resource parameter configuration process based on the resource specification, comprises:

responding to the situation that the current resource data are smaller than the target configuration parameters of the current execution stage, and acquiring preset virtual computing resource configuration information;

and adjusting the quantity of the concurrent tasks corresponding to the task executor based on the preset virtual computing resource configuration information, so as to perform a resource configuration process according to the adjusted quantity of the concurrent tasks.

13. The method of claim 10, wherein obtaining current resource data for a current execution phase comprises:

and counting the resource application data corresponding to the pending state in the global state, and determining the current resource data according to the obtained resource application data and the resource data applied in the current execution stage.

14. The method according to any of claims 1 to 13, wherein after said configuring resource parameters based on said target configuration parameters, the method further comprises:

and regulating and controlling the resources of the corresponding execution stage according to the acquired parameter configuration information of each resource.

15. A resource parameter configuration apparatus, comprising:

the parameter prediction module is used for acquiring the prediction configuration parameters of the execution stage of the current operation according to the resource information of the historical task;

a first parameter processing module, configured to adjust the predicted configuration parameters of each execution stage by using a global resource adjustment policy based on the predicted configuration parameters of each execution stage, to obtain reference configuration parameters of each execution stage;

and the second parameter processing module is used for matching the reference configuration parameters of each execution stage with the corresponding current stage parameter information, determining the target configuration parameters of each execution stage according to the matching result, and configuring the resource parameters based on the target configuration parameters.

16. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any of claims 1-14 via execution of the executable instructions.

17. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1 to 14.