CA3182794A1 - Resource adjusting method, device, equipment and storage medium for flink task processing - Google Patents

Abstract

The present invention discloses to resource adjustment for Flink task processing method, apparatus, computer device and storage medium, comprising: obtaining actual usage of each preset resource usage indicator during current Flink task; generating resource usage total score of current Flink task according to actual usage and pre-configured indicator parameter of each resource usage indicator; obtaining adjustment strategy of current Flink task according to the resource usage total score and the mapping relationship between pre-configured resource usage total score and adjustment strategy; adjusting resource allocation of current Flink task according to the adjustment strategy. This application sets plural resource usage indicators, scores resource usage of current Flink task from plural dimensions, and accurately locates adjustment strategy that matches current Flink task based on the mapping relationship between resource usage total score and adjustment strategy, making the obtaining of adjustment stlategy more accurate and adjusting resource more reasonable and accurate.

Description

RESOURCE ADJUSTING METHOD, DEVICE, EQUIPMENT AND STORAGE
MEDIUM FOR FLINK TASK PROCESSING
Technical Field [0001] The present disclosure relates to computer technical field, particularly to a resource adjustment for Flink task processing method, apparatus, computer device and storage medium.
Background

[0002] At present, each TaskManager of the underling Flink framework is set to IC and 2G
by default, user can adjust quantity of Slots (task slots) and memory of the TaskManager through advanced parameters. For example, the current user sets concurrency, mainly by setting a plurality of TaskManagers or setting a plurality of slots through advanced parameters.
The main problem is when user configures concurrency, due to the lack of experience in recourse allocation, a small amount of concurrency is required to solve the problem but user configures too much concurrency, CPU and memory utilization is low, resulting in waste of resources. Therefore, how to rationally adjust resources has become a problem to be solved.

[0003] In traditional technical scheme, when Flink framework performs task processing, the utilization of resources is generally evaluated by the highest value and average value of QPS(Throughput), and resources are adjusted according to evaluation result, which easily leads to evaluation inaccurate, unreasonable adjustment issues.
Summary

[0004] Based on this, it is necessary to provide a source adjustment for Flink task processing method, apparatus, computer device and storage medium.

[0005] A source adjustment method for Flink task processing, the method comprises:

[0006] Obtaining actual usage of each preset resource usage indicator during the execution of current Flink task;

Date Regue/Date Received 2023-01-24

[0007] Generating resource usage total score of current Flint( task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;

[0008] Obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;

[0009] Adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.

[0010] In an embodiment, the above-mentioned indicator parameter includes the maximum value and the indicator highest score, wherein generating resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator, comprising:

[0011] =When the evaluation factor of the resource usage indicator belonging is first factor, generating usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;

[0012] Generating first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score;

[0013] When the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating consumption ratio based on the actual usage of first indicator and the actual usage of second indicator;

[0014] Generating second score according to the consumption ratio and the highest indicator Date Regue/Date Received 2023-01-24 score of the resource usage indicator of second factor;

[0015] Generating resource usage total score of current Flink task according to the second score and the first score.

[0016] In an embodiment, the above-mentioned obtaining adjustment strategy of current Ffink task according to the resource usage total score of current Ffink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy, comprising:

[0017] Obtaining adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score and the adjustment strategy, then obtaining adjustment strategy of current Flink task.

[0018] In an embodiment, wherein the adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, wherein adjusting resource allocation of current Ffink task according to adjustment strategy of current Flink task, comprising:

[0019] Adjusting the resource allocation amount corresponding to each resource usage indicator of current Flink task according to adjustment value corresponding to resource usage indicator to be adjusted.

[0020] In an embodiment, the above-mentioned method also comprises:

[0021] When receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task;

[0022] Alternatively, when detecting of the current Flink task starts to run, executing the step Date Regue/Date Received 2023-01-24 of obtaining actual usage of each preset resource usage indicator during execution of current Ffink task.

[0023] In an embodiment, the method also comprises:

[0024] Comparing actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result;

[0025] Generating detailed resource usage information corresponding to each resource usage indicator according to the comparison result;

[0026] Sending the detailed resource usage information to the terminal for display.

[0027] In an embodiment, the method also comprises:

[0028] Receiving configuration request for the mapping relationship between resource usage total score and adjustment strategy;

[0029] Creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.

[0030] A resource adjustment apparatus for Flint( task processing, wherein, the apparatus comprises:

[0031] An obtaining module configured to obtain actual usage of each preset resource usage indicator during the execution of current Flink task;

[0032] A generating module configured to generate resource usage total score of current Flink Date Recue/Date Received 2023-01-24 task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator.

[0033] A processing module configured to obtain adjustment strategy of current FlMk task according to the resource usage total score of current Mink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;

[0034] An adjusting module configured to adjust resource allocation of current Flink task according to adjustment strategy of current Flink task.

[0035] A computer device, including a memory, a processor and a computer program stored in the memory and run on the processor configured to achieve following steps when the processor executes the computer program:

[0036] Obtaining actual usage of each preset resource usage indicator during the execution of current Flink task;

[0037] Generating resource usage total score of current Mink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;

[0038] Obtaining adjustment strategy of current FlMk task according to the resource usage total score of current Rink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;

[0039] Adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.

[0040] A computer readable storage medium stored with a computer program configured to achieve following steps when the processor executes the computer program:
Date Regue/Date Received 2023-01-24

[0041] Obtaining actual usage of each preset resource usage indicator during the execution of current Flink task;

[0042] Generating resource usage total score of current Ffink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;

[043] Obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;

[0044] Adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.

[0045] The above-mentioned to resource adjustment for Flink task processing method, apparatus, computer device and storage medium, comprising: obtaining actual usage of each preset resource usage indicator during current Flink task; generating resource usage total score of current Flink task according to actual usage and pre-configured indicator parameter of each resource usage indicator; obtaining adjustment strategy of current Flink task according to the resource usage total score and the mapping relationship between pre-configured resource usage total score and adjustment strategy; adjusting resource allocation of current Flink task according to the adjustment strategy. This application sets plural resource usage indicators, scores resource usage of current Flink task from plural dimensions, and accurately locates adjustment strategy that matches current Flink task based on the mapping relationship between resource usage total score and adjustment strategy, making the obtaining of adjustment strategy more accurate, and adjusting resource based on the adjustment strategy, making resource adjustment more accurate.

Date Recue/Date Received 2023-01-24 Drawing Description Detailed Description [0052] In order to make clearer application purposes, technical solutions, and advantages, the present disclosure is further explained in detail with a particular embodiment thereof, and with reference to the drawings. It shall be appreciated that these descriptions are only intended to be illustrative, but not to limit the scope of the disclosure thereto.
[0053] The present application provides source adjustment for Flink task processing method applied in the application environment from Figure 1. As shown in Figure 1, the application environment includes server 100 and terminal end 101, server 100 connects with terminal 101 through network 102 to achieve the source adjustment method for Flink task processing.
[0054] The server 100 is configured to obtain actual usage of each preset resource usage indicator during the execution of current Flink task when receiving the resource adjustment request sent by terminal 101; generate resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator; obtain adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy; adjust resource allocation of current Flink task according to adjustment strategy of current Flink task, the server is also configured to generate detailed resource usage information corresponding to each resource usage indicator according to the actual usage amount and indicator corresponding to each resource usage indicator; and send resource usage detailed information to terminal 101 for display. Wherein, server 100 can be implemented by an independent server or a server cluster composed of a plurality of servers.
[0055] Terminal 101 is configured to send resource adjustment request to server 100 and receive resource usage details corresponding to each resource usage indicator sent by server, display source usage details corresponding to each resource usage indicator on user interface.

Date Regue/Date Received 2023-01-24 Terminal 101 can be but is not limited to various personal computers, laptops, smartphones, tablet computers and portable wearable device.
[0056] Network 102 is configured to achieve the network connection between terminal 101 and server 100, specifically, the network 102 can include various types of wired or wireless networks.
[0057] In an embodiment, as shown in Figure 2, a resource adjustment method for Flink task processing is provided, the method is applied to the server in Figure 1 as an example for illustration, comprising following steps:
[0058] Si!, obtaining actual usage of each preset resource usage indicator during the execution of current Flink task.
[0059] In the present application, the above-mentioned current Flink task can be processing task of the real-time data flow of Flink framework. The above resource usage indicator can include the highest memory usage, the average memory usage, the quantity of FullGC (Full Garbage Collection), FullGC time, the highest TaskManager CPU usage, the average TaskManager CPU usage, task sender Buffer usage, committedOffsets(commit offset(compensation) to a specific location), currentOffsets(current offset location).
[0060] Furthermore, the actual usage of the above-mentioned resource usage indicator is the actual value of each resource usage indicator during the execution of current Flink task. For example, the highest memory usage can be obtained during the execution of current Flink task.
Specifically, the present application can preset a preset time period, obtain the actual usage amount of each resource usage indicator within the preset time period. For example, if the preset time period is one hour, the actual usage of each resource usage indicator within one hour is obtained. A certain period of time on a certain day can also be set to obtain the actual usage of each resource usage indicator. The specific situation is set according to actual requirements, and no specific limitation is made here.

Date Regue/Date Received 2023-01-24 [0061] S12, generating resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator.
[0062] In the present application, different indicator parameters are set for different resource usage indicators. For example, for the average memory usage, setting the indicator parameter as the maximum memory value 16, and the indicator maximum score value is 50.
[0063] In the present application, obtaining score of each resource usage indicator according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator, further, obtaining resource usage total score of current Ffink task according to the score of each resource usage indicator.
[0064] In an embodiment, the above-mentioned indicator parameter includes the maximum value and the indicator highest score, as shown in Figure 3, a process diagram of generating resource usage total score of current Flinlc task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator is provided in an embodiment, comprising following steps:
[0065] S31, when the evaluation factor of the resource usage indicator belonging is first factor, generating usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;
[0066] S32, generating first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score;
[0067] S33, when the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating Date Regue/Date Received 2023-01-24 consumption ratio based on the actual usage of first indicator and the actual usage of second indicator;
[0068] S34, generating second score according to the consumption ratio and the highest indicator score of the resource usage indicator of second factor;
[0069] S35, generating resource usage total score of current Rink task according to the second score and the first score.
[0070] In the present application, a plurality of evaluation factors are preset. The evaluation factors can include memory, CPU, FullGC, Kafka consumption capacity, and back pressure, data skew, etc. One or more resource usage indicators can be set for each evaluation factor. For example, corresponding resource usage indicator can be set for memory, including the highest memory usage and the average memory usage. Corresponding resource usage indicator can be set for CPU, including the highest TaskManager CPU usage ang the average TaskManager CPU usage. Corresponding resource usage indicator can be set for FulIGC, including the number of FullGC and the time of FullGC. For Kafka consumption capacity, corresponding resource usage indicator can be set, including committedOffsets and currentOffsets. For back pressure and data skew, corresponding recourse usage indicator can be set, including the usage of Buffer at the task sending end and the usage of Buffer at the task receiving end.
[0071] Furthermore, the above-mentioned first factor includes memory, CPU, FullGC, back pressure, and data skew. The above-mentioned second factor includes Kafka consumption capacity. The above-mentioned first indicator is committedOffsets. The above-mentioned second indicator is currentOffsets.
[0072] Specifically, for the memory evaluation factor, assuming that indicator parameter corresponding to the memory is set to include a maximum memory value 16, a highest indicator score 50, obtaining the maximum memory usage M under the memory evaluation factor, then the score of memory evaluation factor = M/16*50.
Date Regue/Date Received 2023-01-24 [0073] In the same way, for the CPU evaluation factor, assuming that indicator parameter corresponding to the CPU is set to include a CPU maximum value 100, a highest indicator score 20, obtaining the average TaskManager CPU usage N under the CPU
evaluation factor, then the score of CPU evaluation factor = N/100*20.
[0074] For the FullGC evaluation factor, assuming that indicator parameter corresponding to the FullGC is set to include a maximum value of FullGC number 20, a highest indicator score 5, obtaining actual value of FullGC number S, then the score of FullGC
evaluation factor =
S/20*5.
[0075] For the back pressure and data skew evaluation factor, assuming that corresponding indicator parameter is set to include a highest score of back pressure 5, a maximum value 100, obtaining the actual value of the Buffer usage at the task sending end Y, then the score of back pressure and data skew evaluation factor = Y/100*5.
[0076] For the Kafka consumption capacity evaluation factor, assuming that corresponding indicator parameter is set to a highest score of offset 15, obtaining the actual value of committedOffsets E and the actual value of currentOffsets F, then the score of Kafka consumption capacity evaluation factor = F/E*15.
[0077] Therefore, the above-mentioned first score comprises the score of memory evaluation factor = M/16*50, the score of CPU evaluation factor = N/100*20, the score of FullGC
evaluation factor = S/20*5, the score of back pressure and data skew evaluation factor =
Y/100 5, the score of Kafka consumption capacity evaluation factor = F/E*15.
Then the calculation formula of total resource usage score of current Ffink task is as follows:
[0078] Resource usage total score = M/16*50+N/100*20+S/20*5+Y/100*5+F/E*15.

Date Regue/Date Received 2023-01-24 [0079] In the present application, by setting a plurality of resource usage indicators and obtaining the actual usage of each resource usage indicator in any time period, based on the actual usage, generating the resource usage total score of current Flink task to achieve resource utilization from plural dimensions, making the evaluation of resource utilization more accurate and scientific, further making resource utilization adjustment more reasonable and solving the problem of resource waste caused by unreasonable resource allocation in the existing technology, improving resource utilization.
[0080] What should be noted is that a plurality of resource usage indicators under one evaluation factor can exist. In this case, when calculating the resource usage total score of current Flink task, the actual usage of one of the resource usage indicators can be selected from a plurality of resource usage indicators for calculating. For example, for the resource usage indicators of the highest memory usage and the average memory usage under the memory evaluation factor, when calculating the resource usage total score of current Flink task, the actual value of the highest memory usage can be selected for calculating.
[0081] In another embodiment, all resource usage indicators under each evaluation factor can also be involved in the calculating. For example, the first score corresponding to the highest memory usage and the average memory usage can be used to calculate the resource usage total score of current Flink task.
[0082] S13, obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy.
[0083] In the present application, the above-mentioned mapping relationship between the resource usage total score and the adjustment strategy includes the corresponding relationship between the resource usage total score and the adjustment strategy. Wherein, the above-mentioned adjustment strategy can include the resource usage indicator to be adjusted and the corresponding adjustment value, can also include the reasons for resource shortage or resource Date Regue/Date Received 2023-01-24 excess, and can also include usage details of other resource usage indicators.
[0084] Specifically, the present application can obtain adjustment strategy according to resource usage total score of current Flink task and can obtain the corresponding adjustment strategy according to the score of each evaluation factor of current Flink task at the same time, so that the obtained adjustment strategy is more accurate.
[0085] For example, the resource usage total score of current Flink task is A, the adjustment strategy is matched according to the mapping relationship between the resource usage total score and the adjustment strategy: the CPU is recommended to reduce to 2C and the memory is reduced to 4G.
[0086] In an embodiment, the above-mentioned obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flint( task and the mapping relationship between pre-configured resource usage total score and adjustment strategy, comprising:
[0087] Obtaining adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score and the adjustment strategy, then obtaining adjustment strategy of current Flink task.
[0088] In the present application, the above-mentioned mapping relationship between the resource usage total score and the adjustment strategy includes the corresponding relationship between the resource usage total score and the adjustment strategy. Wherein, the relationship can be one-to-one correspondence between the resource total score and the adjustment strategy, or one-to-one correspondence between the numerical range of the resource usage total score belonging and the adjustment strategy. The former is a score corresponding to an adjustment strategy, and the latter is a score range corresponding to an adjustment strategy, the specific settings can be set according to actual requirements.

Date Regue/Date Received 2023-01-24 [0089] The present application can locate the adjustment strategy more accurately through the present embodiment, making the resource adjustment more accurate.
[0090] S14, adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.
[0091] In the present application, the above-mentioned adjustment strategy can include the resource usage indicator to be adjusted and the corresponding adjustment value, can also include the reasons for resource shortage or resource excess, and can also include usage details of other resource usage indicators.
[0092] In an embodiment, the above-mentioned adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, wherein adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task, comprising:
[0093] Adjusting the resource allocation amount corresponding to each resource usage indicator of current Rink task according to adjustment value corresponding to resource usage indicator to be adjusted.
[0094] In the present application, the adjustment strategy of current Flink task includes the adjustment value corresponding to the resource usage indicator to be adjusted and the resource usage details information of each resource usage indicator.
[0095] For example, the above-mentioned adjustment strategy can be:
[0096] The resource currently applied for task B is 4C CPU and 8G memory, during the statistical period (2021-11-8:10 to 2021-11-8:11), the total score is C, according to the mapping relationship between the total usage score and the adjustment strategy, the matched adjustment strategy is: the actual resource usage rate is low, and resource is recommended to reduce to Date Recue/Date Received 2023-01-24 2C(CPU)4G(Memory).
[0097] Wherein, resource usage detailed information is as follows:
[0098] (1) Memory: applied memory is 8G, the average used memory during statistical period is actually 3G.
[0099] (2) CPU: applied CPU is 4C, actual use during statistical period is 2.
[0100] (3) FullGC: during statistical period, the number of FullGC is 2, the total time length is 5 minutes, not frequent.
[0101] (4) Back pressure, data skew: the average usage rate of outPoolU sage and inPoolUsage is lower than 30% during the statistical period, no back pressure occurred.
[0102] (5) Consumption capacity: the offset consumption capacity is normal, no backlog occurred.
[0103] According to the above-mentioned adjustment strategy, the adjustment strategy includes the adjustment value of resource usage indicator to be adjusted and the detailed resource usage information. Wherein, the resource usage detailed information describes information such as actual usage information of each resource usage indicator and whether the usage is within a reasonable range. According to the above-mentioned adjustment strategy, the memory can be reduced to 4G, the CPU can be reduced to 2C. This application can achieve self-defined resource adjustment according to the adjustment strategy with high flexibility.
[0104] In an embodiment, the method also comprises:
[0105] When receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink Date Regue/Date Received 2023-01-24 task;
[0106] Alternatively, when detecting of the current Flink task starts to run, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task.
[0107] After the current Flink task is created, the present application can trigger the resource adjustment method of the above-mentioned Flink task processing through the adjustment request submitted by client. Also, the present application can automatically trigger the resource adjustment method of Flink task processing when the current Flink task starts to execute.
Generally, each Plink task can execute the above-mentioned resource adjustment method for Flink task processing once during execution. Specifically, when obtaining the actual usage of each resource usage indicator, the actual usage of any time period during task execution or the actual usage of the entire process can be obtained.
[0108] The present application obtains actual usage amount of each resource usage indicator in any time period, generates the adjustment strategy based on the actual usage of each resource usage indicator in any time period, makes the adjustment strategy more accurate and also improves the flexibility of the solution.
[0109] In an embodiment, as shown in Figure 4, the above-mentioned method also comprises:
[0110] S41, comparing actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result;
[0111] S42, generating detailed resource usage information corresponding to each resource usage indicator according to the comparison result;
[0112] S43, sending the detailed resource usage information to the terminal for display.

Date Regue/Date Received 2023-01-24 [0113] Wherein, resource usage detailed information comprises the actual usage amount of each resource usage indicator, whether each actual usage is within a reasonable range and corresponding reasons analysis.
[0114] The above-mentioned generating detailed resource usage information corresponding to each resource usage indicator according to the actual usage amount corresponding to each resource usage indicator and indicator score, can comprise:
[0115] Comparing actual usage corresponding to each resource usage indicator with preset threshold and generating detailed resource usage information according to the comparison result.
[0116] The present application compares the actual usage corresponding to each resource usage indicator with the preset threshold generating detailed resource usage information according to the comparison result. In another embodiment, the actual usage corresponding to each resource usage indicator can be converted into usage rate, comparing the obtained usage rate with a preset threshold.
[0117] Specifically, when the resource usage indicator is the highest memory usage, obtaining the highest memory usage rate according to the highest memory usage, comparing the highest memory usage rate with the corresponding preset threshold. Determining whether the above-mentioned highest memory usage rate is lower than the preset threshold, if yes, the memory usage rate of the entire task is low, the TaskManager memory or the quantity of TaskManager is recommended to be reduced. The above-mentioned preset threshold can be 30%.
[0118] When the resource usage indicator is the number of FullGC, the corresponding preset threshold can be 50%. Specifically, counting the number of FullGC within a preset time period, such as the number of FullGC within an hour and the duration of each FullGC, multiplying the number of FullGC and the duration of each FullGC to compare with statistical period, if multiplication result exceeds 50% of the statistical period, then generating over frequent during Date Regue/Date Received 2023-01-24 the execution of current Ffink task, considering of improving the resource usage details of the TaskManager memory.
[0119] When the resource usage indicator is the highest task TaskManager CPU
usage, the corresponding preset threshold can be 5%. Specifically, the highest TaskManager CPU usage rate can be obtained according to the obtained highest TaskManager CPU usage, comparing the highest TaskManager CPU usage rate with 5%, if lower than 5%, then generating the highest detailed resource usage information with low TaskManager CPU usage rate.
[0120] When the resource usage indicator is the usage amount of task sending end Buffer and the usage amount of the task receiving end Buffer, generating resource usage detailed information, can comprise following steps:
[0121] Obtaining the usage rate of corresponding task sending end Buffer and the usage rate of task receiving end Buffer respectively according to the usage amount of task sending end Buffer and the usage amount of task receiving end Buffer.
[0122] =When the usage rate of ExclusiveBuffer(private Buffer) is lower than 10% and the usage rate of FloatingBuffer(shared Buffer) is higher than 90%, the generated reason analysis information can be data skew.
[0123] When the Buffer occupancy rate of the receiving end inPoolUsage(input capacity) and the sending end outPoolUsage(output capacity) of a Subtask are both lower than 10%, indicating normal.
[0124] When the Buffer occupancy rate of the receiving end inPoolUsage of a Subtask is more than 90%, the Buffer occupancy rate of the sending end outPoolUsage(output capacity) of a Subtask is lower than 10%, then the generated reason analysis information is that conducting the back pressure upstream can be the source of back pressure.

Date Regue/Date Received 2023-01-24 [0125] When the Buffer occupancy rate of the receiving end inPoolUsage and outPoolUsage of a Subtask are both higher than 90%, then generated reason analysis information is under downstream back pressure.
[0126] When the resource usage indicator is committedOffsets and currentOffsets, if each partition is successfully submitted to the current submission offset position of ICafka partition, much lower than the position of offset in the current loaded partition, and indicating that there is backlog, then generating reason analysis information such as whether the resource allocation is reasonable and whether the business data flow has increased.
[0127] Through the present embodiment, the present application generates detailed resource usage information based on the actual usage of each resource usage indicator, sends the detailed information to terminal for display, so that relevant person can analyze data and improve solution flexibility.
[0128] In an embodiment, the above-mentioned method also comprises:
[0129] Receiving configuration request for the mapping relationship between resource usage total score and adjustment strategy;
[0130] Creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.
[0131] In the present application, when an adjustment strategy requires to be added, a configuration request for the mapping relationship between the resource usage total score and the adjustment strategy can be sent through terminal. The configuration request carries the new adjustment strategy and the resource usage total score corresponding to the new adjustment strategy, according to the configuration request, creating a mapping relationship of the resource Date Recue/Date Received 2023-01-24 usage total score corresponding to the newly added adjustment strategy, and the mapping relationship is stored in a preset database. The adjustment strategy of the present application supports medication and addition, and the scheme flexibility is improved.
[0132] In an embodiment, as shown in Figure 5, a resource adjustment apparatus for Flink task processing is provided, comprising: obtaining module II, generating module 12, processing module 13 and adjusting module 14, wherein:
[0133] An obtaining module configured to obtain actual usage of each preset resource usage indicator during the execution of current Flink task;
[0134] A generating module configured to generate resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;
[0135] A processing module configured to obtain adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;
[0136] An adjusting module configured to adjust resource allocation of current Flink task according to adjustment strategy of current Flink task.
[0137] In an embodiment, the above-mentioned indicator parameter includes the maximum value and the indicator highest score, the above-mentioned generating module 12 generates usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor when the evaluation factor of the resource usage indicator belonging is first factor, generates first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score, when the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating Date Recue/Date Received 2023-01-24 consumption ratio based on the actual usage of first indicator and the actual usage of second indicator, generates second score according to the consumption ratio and the highest indicator score of the resource usage indicator of second factor, and generates resource usage total score of current Flink task according to the second score and the first score.
[0138] In an embodiment, the above-mentioned processing module 13 can obtain adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score and the adjustment strategy, then obtaining adjustment strategy of current FIMk task.
[0139] In an embodiment, the above-mentioned adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, the above-mentioned adjusting module 14 can adjust the resource allocation amount corresponding to each resource usage indicator of current Flink task according to adjustment value corresponding to resource usage indicator to be adjusted.
[0140] In an embodiment, the above-mentioned apparatus also comprises receiving module (not shown in Figure), when receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task, alternatively, when detecting of the current Flink task starts to run, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task.
[0141] In an embodiment, the above-mentioned apparatus also comprises comparing module (not shown in Figure), the present comparing module can compare actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result, generate detailed resource usage information corresponding to each resource usage indicator according to the comparison result, sends the detailed resource usage information to the terminal for display.

Date Regue/Date Received 2023-01-24 [0142] In an embodiment, the above-mentioned apparatus also comprises newly addition module (not shown in Figure), the newly addition module can receive configuration request for the mapping relationship between resource usage total score and adjustment strategy, creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.
[0143] In an embodiment, a computer device is provided, the computer device can be a server whose internal structure diagram is shown in Figure 6. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide calculating and control capabilities.
The memory of the computer device includes non-volatile storage medium and internal memory. The memory of non-volatile storage medium has an operation system, computer programs and database. The internal memory provides an environment for the operation system and computer program running in a non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a resource allocation method for compiling a virtual machine.
[0144] In an embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and ran on the processor configured to achieve the following steps when the processor executes the computer program:
obtaining actual usage of each preset resource usage indicator during the execution of current Ffink task; generating resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;
obtaining adjustment strategy of current Flink task according to the resource usage total score of current Mink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;

Date Regue/Date Received 2023-01-24 [0145] Adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.
[0146] In an embodiment, the above-mentioned indicator parameter includes the maximum value and the indicator highest score, the processor executes the computer program to achieve the above-mentioned generating resource usage total score of current Rink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator, specifically achieving following steps:
[0147] When the evaluation factor of the resource usage indicator belonging is first factor, generating usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;
[0148] Generating first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score;
[0149] When the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating consumption ratio based on the actual usage of first indicator and the actual usage of second indicator;
[0150] Generating second score according to the consumption ratio and the highest indicator score of the resource usage indicator of second factor;
[0151] Generating resource usage total score of current Flink task according to the second score and the first score.
[0152] In an embodiment, the processor executes the computer program to achieve the above-mentioned obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource Date Recue/Date Received 2023-01-24 usage total score and adjustment strategy, specifically achieving following steps:
[0153] Obtaining adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score and the adjustment strategy, then obtaining adjustment strategy of current Ffink task.
[0154] In an embodiment, the adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, the processor executes the computer program to achieve the above-mentioned adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task, specifically achieving following steps:
[0155] Adjusting the resource allocation amount corresponding to each resource usage indicator of cm-rent Flink task according to adjustment value corresponding to resource usage indicator to be adjusted.
[0156] In an embodiment, the processor executes the computer program to achieve following steps:
[0157] When receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task;
[0158] Alternatively, when detecting of the current Flink task starts to run, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task.
[0159] In an embodiment, the processor executes the computer program to achieve following steps:

Date Regue/Date Received 2023-01-24 [0160] Comparing actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result;
[0161] Generating detailed resource usage information corresponding to each resource usage indicator according to the comparison result;
[0162] Sending the detailed resource usage information to the terminal for display.
[0163] In an embodiment, the processor executes the computer program to achieve following steps:
[0164] Receiving configuration request for the mapping relationship between resource usage total score and adjustment strategy;
[0165] Creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.
[0166] In an embodiment, a computer readable storage medium is provided, the medium stored with computer program and the processor performs the following steps when executing the computer program: obtaining actual usage of each preset resource usage indicator during the execution of current Flink task; generating resource usage total score of current Mink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator; obtaining adjustment strategy of current Flink task according to the resource usage total score of current Mink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;
adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.
[0167] In an embodiment, the above-mentioned indicator parameter includes the maximum Date Regue/Date Received 2023-01-24 value and the indicator highest score, the processor executes the computer program to achieve the above-mentioned generating resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator, specifically achieving following steps:
[0168] When the evaluation factor of the resource usage indicator belonging is first factor, generating usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;
[0169] Generating first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score;
[0170] When the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating consumption ratio based on the actual usage of first indicator and the actual usage of second indicator;
[0171] Generating second score according to the consumption ratio and the highest indicator score of the resource usage indicator of second factor;
[0172] Generating resource usage total score of current Flink task according to the second score and the first score.
[0173] In an embodiment, the processor executes the computer program to achieve the above-mentioned obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy, specifically achieving following steps:
[0174] Obtaining adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score Date Regue/Date Received 2023-01-24 and the adjustment strategy, then obtaining adjustment strategy of current Ffink task.
[0175] In an embodiment, the adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, the processor executes the computer program to achieve the above-mentioned adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task, specifically achieving following steps:
[0176] Adjusting the resource allocation amount corresponding to each resource usage indicator of current Flink task according to adjustment value corresponding to resource usage indicator to be adjusted.
[0177] In an embodiment, the processor executes the computer program to achieve following steps:
[0178] When receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Ffink task;
[0179] Alternatively, when detecting of the current Flink task starts to run, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task.
[0180] In an embodiment, the processor executes the computer program to achieve following steps:
[0181] Comparing actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result;
[0182] Generating detailed resource usage information corresponding to each resource usage Date Recue/Date Received 2023-01-24 indicator according to the comparison result;
[0183] Sending the detailed resource usage information to the terminal for display.
[0184] In an embodiment, the processor executes the computer program to achieve following steps:
[0185] Receiving configuration request for the mapping relationship between resource usage total score and adjustment strategy;
[0186] Creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.
[0187] The skilled in the art can understand that all or partial of procedures from the above-mentioned methods can be performed by computer program instructions through related hardware, the mentioned computer program can be stored in a non-volatile material computer readable storage medium, this computer can include various embodiment procedures from the abovementioned methods when execution. Any reference to the memory, the storage, the database, or the other media used in each embodiment provided in current application can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programable ROM (PROM), electrically programmable ROM (EPRPMD), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an instruction but not limited to, RAM is available in many forms such as static RAM (SRAM), dynamic RAM
(DRAMD), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SRAM (ESDRAM), synchronal link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Date Regue/Date Received 2023-01-24 [0188] The technical features of the above-mentioned embodiments can be randomly combined, for concisely statement, not all possible combinations of technical feature in the abovementioned embodiments are described. However, if there are no conflicts in the combinations of these technical features, it shall be within the scope of this description.
[0189] The above-mentioned embodiments are only several embodiments in this disclosure and the description is more specific and detailed but cannot be understood as the limitation of the scope of the invention patent. Evidently those ordinary skilled in the art can make various modifications and variations to the disclosure without departing from the spirit and scope of the disclosure. Therefore, the appended claims are intended to be construed as encompassing the described embodiment and all the modifications and variations coming into the scope of the disclosure.

Date Regue/Date Received 2023-01-24

Claims (10)

Claims:

1. A resource adjustment method for Flink task processing, the method comprises:
obtaining actual usage of each preset resource usage indicator during the execution of current Flink task;
generating resource usage total score of current Flink task according to the actnal usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;
obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy; and adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task.

2. The method according to claim 1, the indicator parameter includes the maximum value and the indicator highest score, wherein generating resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator, comprising:
when the evaluation factor of the resource usage indicator belonging is first factor, generating usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;
generating first score according to the usage ratio of the resource usage indicator of each first factor and the indicator highest score;
Date Recue/Date Received 2023-01-24 when the evaluation factor of the resource usage indicator belonging is second factor, the resource usage indicator includes first indicator and second indicator, generating consumption ratio based on the actual usage of first indicator and the actual usage of second indicator;
generating second score according to the consumption ratio and the highest indicator score of the resource usage indicator of second factor; and generating resource usage total score of current Flink task according to the second score and the first score.

3. The method according to claim 1, wherein, obtaining adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy, comprising:
obtaining adjustment strategy matching the range of the resource usage total score of current Flink task according to the mapping relationship between the resource usage total score and the adjustment strategy, then obtaining adjustment strategy of current Flink task.

4. The method according to claim 1, wherein the adjustment strategy of current Flink task includes adjustment value corresponding to resource usage indicator to be adjusted, wherein adjusting resource allocation of current Flink task according to adjustment strategy of current Flink task, comprising:
adjusting the resource allocation amount corresponding to each resource usage indicator of current Flink task according to adjustment value corresponding to resource usage indicator to be adjusted.

Date Regue/Date Received 2023-01-24

5. The method according to claim 1, wherein, the method also comprises:
when receiving resource adjustment request sent by terminal, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task; and altematively, when detecting of the current Flink task starts to run, executing the step of obtaining actual usage of each preset resource usage indicator during execution of current Flink task.

6. The method according to claim 5, wherein, the method also comprises:
comparing actual usage corresponding to each resource usage indicator with preset threshold to obtain comparison result;
generating detailed resource usage information corresponding to each resource usage indicator according to the comparison result; and sending the detailed resource usage information to the terminal for display.

7. The method according to claim 1, wherein, the method also comprises:
receiving configuration request for the mapping relationship between resource usage total score and adjustment strategy; and creating the mapping relationship between the newly added adjustment strategy and the resource usage total score corresponding to the newly added adjustment strategy according to the newly added adjustment strategy in the configuration request and the resource usage total score corresponding to the newly added adjustment strategy.

Date Recue/Date Received 2023-01-24

8. A resource adjustment apparatus for Flink task processing, wherein, the apparatus comprises:
an obtaining module configured to obtain actual usage of each preset resource usage indicator during the execution of current Flink task;
a generating module configured to generate resource usage total score of current Flink task according to the actual usage of each resource usage indicator and pre-configured indicator parameter of each resource usage indicator;
a processing module configured to obtain adjustment strategy of current Flink task according to the resource usage total score of current Flink task and the mapping relationship between pre-configured resource usage total score and adjustment strategy;
and an adjusting module configured to adjust resource allocation of current Flink task according to adjustment strategy of current Flink task.

9. A computer device, including a memory, a processor and a computer program stored in the memory and run on the processor configured to achieve the steps of any methods in claim 1 to 7 when the processor executes the computer program.

10. A computer readable storage medium stored with a computer program configured to achieve the steps of any methods in claim 1 to 7 when the processor executes the computer program.