CN110389842A - A kind of dynamic resource allocation method, device, storage medium and equipment - Google Patents
A kind of dynamic resource allocation method, device, storage medium and equipment Download PDFInfo
- Publication number
- CN110389842A CN110389842A CN201910681471.6A CN201910681471A CN110389842A CN 110389842 A CN110389842 A CN 110389842A CN 201910681471 A CN201910681471 A CN 201910681471A CN 110389842 A CN110389842 A CN 110389842A
- Authority
- CN
- China
- Prior art keywords
- resources
- target job
- adjusted
- parallelism
- degree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The invention discloses a kind of dynamic resource allocation method, device, storage medium and equipment.The described method includes: obtaining the degree of parallelism for having used stock number, available volume of resources and the target job of the total resources in resources bank, target job；The target job is monitored in the execution time in data shuffling stage；If the execution time in the data shuffling stage is greater than preset duration, by the degree of parallelism for having used stock number, available volume of resources and the target job of pre-defined rule adjustment target job；According to the available volume of resources of total resources and the target job adjusted in resources bank, the surplus yield in resources bank is determined；Judge whether available volume of resources adjusted is less than the surplus yield；If available volume of resources adjusted is less than surplus yield, continue to monitor the target job in the execution time in data shuffling stage.The application can adjust degree of parallelism, the Operating ettectiveness of Lai Tigao big data frame by dynamic.
Description
Technical field
This application involves big data analysis and process field, in particular to a kind of dynamic resource allocation method, device, storage
Medium and equipment.
Background technique
With information-based development, the growth of enterprise's data explosion formula to be processed, data volume has all reached TB grades, PB
Grade.In order to support the analysis and processing of so large-scale data, all kinds of big data frames, tool and technology are come into being, Spark
It is one of them.By taking Spark as an example, Spark is a big data processing around speed, ease for use and complicated analysis building
Frame, by using data shuffling (Shuffle) mode in data processing, by " mapping-specification " model (Map-
Reduce it) is promoted to a higher level, and using the processing capacity of internal storage data storage and near real-time, makes its performance
Many times faster than other big data processing techniques.
Currently, for the operation for thering is Shuffle to operate, being normally set up one admittedly when carrying out Activity Calculation using spark
Fixed degree of parallelism parameter can not provide dynamic degree of parallelism parameter adjustment, so that being easy to cause when spark resource distribution is too small
Lead to the problem of job request is less than resource when memory overflows or resource distribution is excessive.Therefore, how dynamically to adjust parallel
Degree, to improve the Operating ettectiveness of big data frame, becomes urgent problem to be solved in the prior art.
Summary of the invention
The purpose of the embodiment of the present application is to provide a kind of dynamic resource allocation method, device, storage medium and equipment, dynamic
Adjust degree of parallelism, the Operating ettectiveness of Lai Tigao big data frame.
In order to achieve the above objectives, the embodiment of the present application provides a kind of dynamic resource allocation method, comprising:
Obtain resources bank in total resources, target job use stock number, available volume of resources and the target work
The degree of parallelism of industry；
The target job is monitored in the execution time in data shuffling stage；
If the execution time in the data shuffling stage is greater than preset duration, by pre-defined rule adjustment target job
With the degree of parallelism of stock number, available volume of resources and the target job；
According to the available volume of resources of total resources and the target job adjusted in resources bank, resources bank is determined
In surplus yield；
Judge whether available volume of resources adjusted is less than the surplus yield；
If available volume of resources adjusted is less than surplus yield, continue to monitor the target job in data shuffling rank
The execution time of section.
Preferably, described by the pre-defined rule available volume of resources distributed of adjustment target job and the target job
Degree of parallelism, comprising:
The available volume of resources is double, will be adjusted to the one third of available volume of resources with stock number, and it is double simultaneously
Row degree；
Judge whether degree of parallelism adjusted is greater than three times for having used stock number adjusted；
If more than then setting three times for having used stock number adjusted for the degree of parallelism.
It preferably, will be described parallel if degree of parallelism adjusted is less than three times for having used stock number adjusted
Spend the value after being set as double.
Preferably, if available volume of resources adjusted is more than or equal to surplus yield, the target is no longer monitored
Operation generates alarm signal in the execution time in data shuffling stage.
Preferably, the preset duration is that duration is completed in the default operation of the target job.
Preferably, the available volume of resources that the total resources in the resources bank, target job are distributed includes CPU number.
The embodiment of the present application also provides a kind of dynamic resource allocation apparatus, comprising:
Data acquisition module, for obtaining the total resources in resources bank, target job has used stock number, available resources
The degree of parallelism of amount and the target job；
Time monitoring module is executed, for monitoring the target job in the execution time in data shuffling stage；
Parameter adjustment module, if the execution time for the data shuffling stage is greater than preset duration, by pre- set pattern
Then adjust the degree of parallelism of available volume of resources and the target job that target job is distributed；
Surplus yield determining module, for according in resources bank total resources and the target job adjusted
Shared stock number determines the surplus yield in resources bank；
First judgment module, for judging whether available volume of resources adjusted is less than the surplus yield；
Loop module continues to monitor the target work if being less than surplus yield for available volume of resources adjusted
Execution time of the industry in the data shuffling stage.
Preferably, the parameter adjustment module includes:
Parameter set unit, for by the available volume of resources it is double, available volume of resources will be adjusted to stock number
One third, and double degree of parallelism；
Second judgment module described adjusted has used the three of stock number for judging whether degree of parallelism adjusted is greater than
Times；
Degree of parallelism setting unit, in the case where being greater than, then setting adjusted for the degree of parallelism and having used money
Three times of source amount.
The embodiment of the present application also provides a kind of computer equipment, including processor and refers to for storage processor to be executable
The memory of order, the processor realize above-mentioned steps when executing described instruction.
The embodiment of the present application also provides a kind of computer readable storage medium, is stored thereon with computer instruction, the finger
Order is performed realization above-mentioned steps.
As can be seen from the technical scheme provided by the above embodiments of the present application, it in the embodiment of the present application, is shuffled by monitoring data
Stock number, available volume of resources and degree of parallelism have been used in the execution time in stage (shuffle stage), dynamic adjustment operation,
To find more particularly suitable configuration parameter, to improve the Operating ettectiveness of big data frame.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The some embodiments recorded in application, for those of ordinary skill in the art, in the premise of not making the creative labor property
Under, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of dynamic resource allocation method flow chart in the embodiment of the present application；
Fig. 2 is a kind of flow chart of data processing figure of dynamic resource allocation in the embodiment of the present application；
Fig. 3 is a kind of flow chart of data processing of the configuration parameter adjustment module of dynamic resource allocation in the embodiment of the present application
Figure；
Fig. 4 is a kind of flow chart of data processing figure of the judgment module of dynamic resource allocation in the embodiment of the present application；
Fig. 5 is a kind of modular structure schematic diagram of dynamic resource allocation apparatus in the embodiment of the present application；
Fig. 6 is the schematic diagram of computer equipment provided by the embodiments of the present application.
Specific embodiment
The application embodiment provides a kind of dynamic resource allocation method, device, storage medium and equipment.
In order to make those skilled in the art better understand the technical solutions in the application, below in conjunction with the application reality
The attached drawing in mode is applied, the technical solution in the application embodiment is clearly and completely described, it is clear that described
Embodiment is only a part of embodiment of the application, rather than whole embodiments.Based on the embodiment party in the application
Formula, every other embodiment obtained by those of ordinary skill in the art without making creative efforts, is all answered
When the range for belonging to the application protection.
In big data frame, the total resources that operation can use is managed in resources bank, is appointed in operation big data
When business, system is that operation distributes corresponding resource, specifically, the resource is usually the quantity of process executor, and each
The CPU number and memory size of executor can also include certainly other resources, be not construed as limiting to this application.Dividing
After complete resource, how degree of parallelism parameter is set, so that degree of parallelism parameter matches with the resource distributed, is made full use of point
The resource matched has very big influence to the Operating ettectiveness of big data frame.
By taking big data frame spark as an example, degree of parallelism is referred in spark operation, the quantity of the task of each stage.
For example, distributing 100 CPU for some operation, then 100 task can be run parallel, therefore, it is necessary at least set degree of parallelism
It is set to the fully effective utilization cluster resource of 100 ability and finally promotes the performance and the speed of service of entire Spark operation.But
In the prior art, usually after distributing resource for operation, a fixed degree of parallelism parameter is set, how to be dynamically adjusted simultaneously
Row degree parameter becomes the key for promoting big data frame Operating ettectiveness.
It is a kind of flow chart of dynamic resource allocation provided by the present application shown in referring to Figure 1 and Figure 2.It specifically includes following
Step:
S101: the total resources in resources bank is obtained, target job has used stock number, available volume of resources and described
The degree of parallelism of target job.
In some embodiments, the total resources in resources bank is determined, i.e., the retrievable resource upper limit can be cluster
Middle CPU sum；The CPU number that can be current operational objective operation with stock number of target job, the available money of target job
Source amount refers to can be with the maximum CPU number of operational objective operation.
S102: the target job is monitored in the execution time in data shuffling stage.
S103: if the execution time in the data shuffling stage is greater than preset duration, make by pre-defined rule adjustment target
Stock number and degree of parallelism shared by industry.
The data shuffling stage, that is, shuffle stage describes data from map task and is output to reduce task input
This section of process, the performance height of shuffle directly affects the performance and handling capacity of entire program, therefore, it is necessary to logarithm
It is monitored according to the execution time in the stage of shuffling, to judge the operating condition of operation.
In some embodiments, the execution time in stage is shuffled by monitoring data, and is compared with preset duration,
If more than preset duration, then stock number shared by operation and degree of parallelism are adjusted；Wherein, preset duration can be target job
Preset completion duration.
In a specific embodiment, the configuration parameter adjustment module in Fig. 2 can be refering to what is shown in Fig. 3, can incite somebody to action
The available volume of resources that target job is distributed is double, and by it is current with stock number be adjusted to available volume of resources three/
One, then double degree of parallelism parameter.Due to degree of parallelism be greater than executor three times when, parallel task will appear serial accumulation
The case where, cannot achieve the maximization of efficiency, therefore, after degree of parallelism parameter is double, it is also necessary to judge it is double after degree of parallelism
Whether parameter is greater than three times current for having used stock number, if the degree of parallelism parameter after double has used stock number greater than current
Three times, then degree of parallelism parameter is set as having used three times of stock number；If the degree of parallelism parameter after double is less than or waits
In having used three times of stock number, then take it is double after degree of parallelism parameter.
For example, target job be 10 CPU with stock number, available volume of resources is 30 CPU, degree of parallelism 10.Prison
The execution time for controlling the data shuffling stage is greater than preset duration and available volume of resources is adjusted to 60 according to the method described above
Current is adjusted to 20 with stock number by CPU, and degree of parallelism is adjusted to 20, it can be seen that degree of parallelism at this time, which is less than, have been used
Therefore three times of stock number can set 20 for degree of parallelism parameter.
S104: according to stock number shared by the target job adjusted, the surplus yield in resources bank is determined.
In some embodiments, it by the interface of resources bank, inputs after the adjustment, the available money that target job is distributed
Source amount, and determine the surplus yield in resources bank.
S105: judge whether available volume of resources adjusted is greater than the surplus yield.
With reference to process shown in Fig. 4.
S106: if available volume of resources adjusted is less than surplus yield, continue to monitor the target job in data
Shuffle the execution time in stage.
For example, the total resources in resources bank is 100 CPU, after the adjustment, the available volume of resources that target job is distributed
For 60 CPU, then surplus yield is 40 CPU.As can be seen that available volume of resources adjusted is less than surplus yield, then
Continue monitoring objective operation in the execution time in data shuffling stage.When available volume of resources adjusted is more than or equal to residue
When stock number, reach the retrievable resource upper limit, exits dynamic tuning, and alert.
Refering to what is shown in Fig. 5, the application also provides a kind of dynamic resource allocation apparatus, which includes:
Data acquisition module 411, for obtaining the total resources in resources bank, target job has used stock number, available
The degree of parallelism of stock number and the target job；
Time monitoring module 412 is executed, for monitoring the target job in the execution time in data shuffling stage；
Parameter adjustment module 413, if the execution time for the data shuffling stage is greater than preset duration, by predetermined
The degree of parallelism of available volume of resources and the target job that rule adjustment target job is distributed；
Surplus yield determining module 414, for according in resources bank total resources and the target adjusted
Stock number shared by operation determines the surplus yield in resources bank；
First judgment module 415, for judging whether available volume of resources adjusted is less than the surplus yield；
Loop module 416 continues to monitor the target if being less than surplus yield for available volume of resources adjusted
Execution time of the operation in the data shuffling stage.
As shown in fig. 6, the application embodiment also provides a kind of computer equipment, including processor and at storage
The memory of device executable instruction is managed, when the processor executes described instruction the step of the realization above method.
The application also provides a kind of computer readable storage medium, is stored thereon with computer instruction, and described instruction is held
The step of above method is realized when row.
The application operation calculating data volume with it is expected do not differ larger and different periods data volume not etc. in the case where,
It will not lead to operation because of intrinsic degree of parallelism parameter configuration there is a situation where memory spilling or can not apply to enough resources,
To reduce the number of operation failure.At the same time, the time for reducing spark resource distribution dynamic retractility passes through monitoring
The execution time of shuffle stage adjusts executor number, finds relatively suitable minExcutors-maxExcutors,
The operational efficiency for improving operation also can automatically adjust the degree of parallelism of operation while dynamic retractility resource distribution.
In the 1990s, the improvement of a technology can be distinguished clearly be on hardware improvement (for example,
Improvement to circuit structures such as diode, transistor, switches) or software on improvement (improvement for method flow).So
And with the development of technology, the improvement of current many method flows can be considered as directly improving for hardware circuit.
Designer nearly all obtains corresponding hardware circuit by the way that improved method flow to be programmed into hardware circuit.Cause
This, it cannot be said that the improvement of a method flow cannot be realized with hardware entities module.For example, programmable logic device
(Programmable Logic Device, PLD) (such as field programmable gate array (Field Programmable Gate
Array, FPGA)) it is exactly such a integrated circuit, logic function determines device programming by user.By designer
Voluntarily programming comes a digital display circuit " integrated " on a piece of PLD, designs and makes without asking chip maker
Dedicated IC chip.Moreover, nowadays, substitution manually makes IC chip, this programming is also used instead mostly " is patrolled
Volume compiler (logic compiler) " software realizes that software compiler used is similar when it writes with program development,
And the source code before compiling also write by handy specific programming language, this is referred to as hardware description language
(Hardware Description Language, HDL), and HDL is also not only a kind of, but there are many kind, such as ABEL
(Advanced Boolean Expression Language)、AHDL(Altera Hardware Description
Language)、Confluence、CUPL(Cornell University Programming Language)、HDCal、JHDL
(Java Hardware Description Language)、Lava、Lola、MyHDL、PALASM、RHDL(Ruby
Hardware Description Language) etc., VHDL (Very-High-Speed is most generally used at present
Integrated Circuit Hardware Description Language) and Verilog.Those skilled in the art also answer
This understands, it is only necessary to method flow slightly programming in logic and is programmed into integrated circuit with above-mentioned several hardware description languages,
The hardware circuit for realizing the logical method process can be readily available.
It is also known in the art that other than realizing controller in a manner of pure computer readable program code, it is complete
Entirely can by by method and step carry out programming in logic come so that controller with logic gate, switch, specific integrated circuit, programmable
Logic controller realizes identical function with the form for being embedded in microcontroller etc..Therefore this controller is considered one kind
Hardware component, and the structure that the device for realizing various functions for including in it can also be considered as in hardware component.Or
Even, can will be considered as realizing the device of various functions either the software module of implementation method can be Hardware Subdivision again
Structure in part.
Device that above-described embodiment illustrates, module can specifically realize by computer chip or entity, or by having certain
The product of function is planted to realize.
For convenience of description, it is divided into various modules when description apparatus above with function to describe respectively.Certainly, implementing this
The function of each module can be realized in the same or multiple software and or hardware when application.
As seen through the above description of the embodiments, those skilled in the art can be understood that the application can
It realizes by means of software and necessary general hardware platform.Based on this understanding, the technical solution essence of the application
On in other words the part that contributes to existing technology can be embodied in the form of software products, in a typical configuration
In, calculating equipment includes one or more processors (CPU), input/output interface, network interface and memory.The computer is soft
Part product may include that some instructions are used so that a computer equipment (can be personal computer, server or network
Equipment etc.) execute method described in certain parts of each embodiment of the application or embodiment.The computer software product can
To be stored in memory, memory may include the non-volatile memory in computer-readable medium, random access memory
(RAM) and/or the forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer
The example of readable medium.Computer-readable medium includes that permanent and non-permanent, removable and non-removable media can be by
Any method or technique come realize information store.Information can be computer readable instructions, data structure, the module of program or its
His data.The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory
(SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only memory
(ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory techniques, CD-ROM are read-only
Memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or
Other magnetic storage devices or any other non-transmission medium, can be used for storage can be accessed by a computing device information.According to
Herein defines, and computer-readable medium does not include of short duration computer readable media (transitory media), such as modulation
Data-signal and carrier wave.
All the embodiments in this specification are described in a progressive manner, same and similar portion between each embodiment
Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for device reality
For applying example, since it is substantially similar to the method embodiment, so being described relatively simple, related place is referring to embodiment of the method
Part explanation.
The application can be used in numerous general or special purpose computing system environments or configuration.Such as: personal computer, clothes
Business device computer, handheld device or portable device, laptop device, multicomputer system, microprocessor-based system, set
Top box, programmable consumer-elcetronics devices, network PC, minicomputer, mainframe computer including any of the above system or equipment
Distributed computing environment etc..
The application can describe in the general context of computer-executable instructions executed by a computer, such as program
Module.Generally, program module includes routines performing specific tasks or implementing specific abstract data types, programs, objects, group
Part, data structure etc..The application can also be practiced in a distributed computing environment, in these distributed computing environments, by
Task is executed by the connected remote processing devices of communication network.In a distributed computing environment, program module can be with
In the local and remote computer storage media including storage equipment.
Although depicting the application by embodiment, it will be appreciated by the skilled addressee that the application there are many deformation and
Variation is without departing from spirit herein, it is desirable to which the attached claims include these deformations and change without departing from the application's
Spirit.
Claims (10)
1. a kind of dynamic resource allocation method characterized by comprising
Obtain the total resources in resources bank, target job has used stock number, available volume of resources and the target job
Degree of parallelism；
The target job is monitored in the execution time in data shuffling stage；
If the execution time in the data shuffling stage is greater than preset duration, money has been used by pre-defined rule adjustment target job
The degree of parallelism of source amount, available volume of resources and the target job；
According to the available volume of resources of total resources and the target job adjusted in the resources bank, resources bank is determined
In surplus yield；
Judge whether available volume of resources adjusted is less than the surplus yield；
If available volume of resources adjusted is less than surplus yield, continue to monitor the target job in the data shuffling stage
Execute the time.
2. the method according to claim 1, wherein it is described by pre-defined rule adjustment target job distributed can
With the degree of parallelism of stock number and the target job, comprising:
The available volume of resources is double, the one third of available volume of resources, and double degree of parallelism will be adjusted to stock number；
Judge whether degree of parallelism adjusted is greater than three times for having used stock number adjusted；
If more than then setting three times for having used stock number adjusted for the degree of parallelism.
3. according to the method described in claim 2, it is characterized in that, if degree of parallelism adjusted is less than described adjusted used
Three times of stock number, the then value after setting double for the degree of parallelism.
4. the method according to claim 1, wherein if available volume of resources adjusted is more than or equal to residue
Stock number then no longer monitors the target job in the execution time in data shuffling stage, and generates alarm signal.
5. the method according to claim 1, wherein the preset duration is the default operation of the target job
Complete duration.
6. the method according to claim 1, wherein total resources, target job in the resources bank are
It include CPU number with stock number, available volume of resources.
7. a kind of dynamic resource allocation apparatus characterized by comprising
Data acquisition module, for obtaining the total resources in resources bank, target job has used stock number, available volume of resources,
And the degree of parallelism of the target job；
Time monitoring module is executed, for monitoring the target job in the execution time in data shuffling stage；
Parameter adjustment module, if the execution time for the data shuffling stage is greater than preset duration, by pre-defined rule tune
The degree of parallelism of available volume of resources and the target job that whole target job is distributed；
Surplus yield determining module, for according in the resources bank total resources and the target job adjusted
Available volume of resources, determine the surplus yield in resources bank；
First judgment module, for judging whether available volume of resources adjusted is less than the surplus yield；
Loop module continues to monitor the target job and exists if being less than surplus yield for available volume of resources adjusted
The execution time in data shuffling stage.
8. device according to claim 7, which is characterized in that the parameter adjustment module includes:
Parameter set unit, for by the available volume of resources it is double, three points of available volume of resources will be adjusted to stock number
One of, and double degree of parallelism；
Second judgment module, for judging whether degree of parallelism adjusted is greater than three times for having used stock number adjusted；
Degree of parallelism setting unit, in the case where being greater than, then setting adjusted for the degree of parallelism and having used stock number
Three times.
9. a kind of computer equipment, including processor and for the memory of storage processor executable instruction, the processing
The step of device realizes any one of claims 1 to 6 the method when executing described instruction.
10. a kind of computer readable storage medium is stored thereon with computer instruction, described instruction, which is performed, realizes that right is wanted
The step of seeking any one of 1 to 6 the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681471.6A CN110389842B (en) | 2019-07-26 | 2019-07-26 | Dynamic resource allocation method, device, storage medium and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910681471.6A CN110389842B (en) | 2019-07-26 | 2019-07-26 | Dynamic resource allocation method, device, storage medium and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110389842A true CN110389842A (en) | 2019-10-29 |
| CN110389842B CN110389842B (en) | 2022-09-20 |
Family
ID=68287614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910681471.6A Active CN110389842B (en) | 2019-07-26 | 2019-07-26 | Dynamic resource allocation method, device, storage medium and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110389842B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538576A (en) * | 2020-04-20 | 2020-08-14 | 支付宝(杭州)信息技术有限公司 | Data resource processing method, device, equipment and storage medium |
| CN111597037A (en) * | 2020-04-15 | 2020-08-28 | 北京文思海辉金信软件有限公司 | Job distribution method and device, electronic equipment and readable storage medium |
| CN111858030A (en) * | 2020-06-17 | 2020-10-30 | 北京百度网讯科技有限公司 | Job resource processing method and device, electronic equipment and readable storage medium |
| CN112463290A (en) * | 2020-11-10 | 2021-03-09 | 中国建设银行股份有限公司 | Method, system, apparatus and storage medium for dynamically adjusting the number of computing containers |
| CN113010551A (en) * | 2021-03-02 | 2021-06-22 | 北京三快在线科技有限公司 | Resource caching method and device |
| CN113391911A (en) * | 2021-07-05 | 2021-09-14 | 中国工商银行股份有限公司 | Big data resource dynamic scheduling method, device and equipment |
| WO2023115931A1 (en) * | 2021-12-21 | 2023-06-29 | 浪潮通信信息系统有限公司 | Big-data component parameter adjustment method and apparatus, and electronic device and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426254A (en) * | 2015-12-24 | 2016-03-23 | 北京轻元科技有限公司 | Graded cloud computing resource customizing method and system |
| US9342355B2 (en) * | 2013-06-20 | 2016-05-17 | International Business Machines Corporation | Joint optimization of multiple phases in large data processing |
| CN106033371A (en) * | 2015-03-13 | 2016-10-19 | 杭州海康威视数字技术股份有限公司 | Method and system for dispatching video analysis task |
| US20170093966A1 (en) * | 2015-09-28 | 2017-03-30 | International Business Machines Corporation | Managing a shared pool of configurable computing resources having an arrangement of a set of dynamically-assigned resources |
| CN109189563A (en) * | 2018-07-25 | 2019-01-11 | 腾讯科技（深圳）有限公司 | Resource regulating method, calculates equipment and storage medium at device |
| CN109324894A (en) * | 2018-08-13 | 2019-02-12 | 中兴飞流信息科技有限公司 | PC cluster method, apparatus and computer readable storage medium |
| CN109413125A (en) * | 2017-08-18 | 2019-03-01 | 北京京东尚科信息技术有限公司 | The method and apparatus of dynamic regulation distributed system resource |
| CN109522100A (en) * | 2017-09-19 | 2019-03-26 | 阿里巴巴集团控股有限公司 | Real-time calculating task method of adjustment and device |
-
2019
- 2019-07-26 CN CN201910681471.6A patent/CN110389842B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9342355B2 (en) * | 2013-06-20 | 2016-05-17 | International Business Machines Corporation | Joint optimization of multiple phases in large data processing |
| CN106033371A (en) * | 2015-03-13 | 2016-10-19 | 杭州海康威视数字技术股份有限公司 | Method and system for dispatching video analysis task |
| US20170093966A1 (en) * | 2015-09-28 | 2017-03-30 | International Business Machines Corporation | Managing a shared pool of configurable computing resources having an arrangement of a set of dynamically-assigned resources |
| CN105426254A (en) * | 2015-12-24 | 2016-03-23 | 北京轻元科技有限公司 | Graded cloud computing resource customizing method and system |
| CN109413125A (en) * | 2017-08-18 | 2019-03-01 | 北京京东尚科信息技术有限公司 | The method and apparatus of dynamic regulation distributed system resource |
| CN109522100A (en) * | 2017-09-19 | 2019-03-26 | 阿里巴巴集团控股有限公司 | Real-time calculating task method of adjustment and device |
| CN109189563A (en) * | 2018-07-25 | 2019-01-11 | 腾讯科技（深圳）有限公司 | Resource regulating method, calculates equipment and storage medium at device |
| CN109324894A (en) * | 2018-08-13 | 2019-02-12 | 中兴飞流信息科技有限公司 | PC cluster method, apparatus and computer readable storage medium |
Non-Patent Citations (7)
| Title |
|---|
| QI ZHANG: "PRISM: Fine-Grained Resource-Aware Scheduling for MapReduce", 《IEEE TRANSACTIONS ON CLOUD COMPUTING》 * |
| SUZHEN WANG: "Spark Load Balancing Strategy Optimization Based on Internet of Things", 《2018 INTERNATIONAL CONFERENCE ON CYBER-ENABLED DISTRIBUTED COMPUTING AND KNOWLEDGE DISCOVERY 》 * |
| 杨忙忙: "Spark数据处理平台中资源动态分配技术研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
| 葛庆宝等: "基于关键阶段分析的Spark性能预测模型", 《计算机系统应用》 * |
| 袁景凌: "《Hadoop核心技术与实验》", 30 April 2017 * |
| 韩海雯: "MapReduce计算任务调度的资源配置优化研究", 《中国博士学位论文全文数据库信息科技辑》 * |
| 黄冬梅: "《案例驱动的大数据原理技术及应用》", 30 November 2018 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597037A (en) * | 2020-04-15 | 2020-08-28 | 北京文思海辉金信软件有限公司 | Job distribution method and device, electronic equipment and readable storage medium |
| CN111597037B (en) * | 2020-04-15 | 2023-06-16 | 中电金信软件有限公司 | Job allocation method, job allocation device, electronic equipment and readable storage medium |
| CN111538576A (en) * | 2020-04-20 | 2020-08-14 | 支付宝(杭州)信息技术有限公司 | Data resource processing method, device, equipment and storage medium |
| CN111538576B (en) * | 2020-04-20 | 2023-05-02 | 支付宝(杭州)信息技术有限公司 | Data resource processing method, device, equipment and storage medium |
| CN111858030A (en) * | 2020-06-17 | 2020-10-30 | 北京百度网讯科技有限公司 | Job resource processing method and device, electronic equipment and readable storage medium |
| CN111858030B (en) * | 2020-06-17 | 2024-03-22 | 北京百度网讯科技有限公司 | Resource processing method and device for job, electronic equipment and readable storage medium |
| CN112463290A (en) * | 2020-11-10 | 2021-03-09 | 中国建设银行股份有限公司 | Method, system, apparatus and storage medium for dynamically adjusting the number of computing containers |
| CN113010551A (en) * | 2021-03-02 | 2021-06-22 | 北京三快在线科技有限公司 | Resource caching method and device |
| CN113391911A (en) * | 2021-07-05 | 2021-09-14 | 中国工商银行股份有限公司 | Big data resource dynamic scheduling method, device and equipment |
| CN113391911B (en) * | 2021-07-05 | 2024-03-26 | 中国工商银行股份有限公司 | Dynamic scheduling method, device and equipment for big data resources |
| WO2023115931A1 (en) * | 2021-12-21 | 2023-06-29 | 浪潮通信信息系统有限公司 | Big-data component parameter adjustment method and apparatus, and electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110389842B (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110389842A (en) | A kind of dynamic resource allocation method, device, storage medium and equipment | |
| US10983895B2 (en) | System and method for data application performance management | |
| US9277003B2 (en) | Automated cloud workload management in a map-reduce environment | |
| Cho et al. | Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters | |
| US8826277B2 (en) | Cloud provisioning accelerator | |
| Hu et al. | Scheduling real-time parallel applications in cloud to minimize energy consumption | |
| CN109144699A (en) | Distributed task dispatching method, apparatus and system | |
| AU2017261531B2 (en) | Prescriptive analytics based activation timetable stack for cloud computing resource scheduling | |
| US10356167B1 (en) | Workload profiling | |
| CN107577694A (en) | A kind of data processing method and equipment based on block chain | |
| CN116167463B (en) | Distributed model training container scheduling method and device for intelligent computing | |
| CN110262847B (en) | Application program starting acceleration method and device and machine-readable storage medium | |
| CN108345644A (en) | A kind of method and device of data processing | |
| CN105786603A (en) | High-concurrency service processing system and method based on distributed mode | |
| CN112748993A (en) | Task execution method and device, storage medium and electronic equipment | |
| US20120254822A1 (en) | Processing optimization load adjustment | |
| CN110795238A (en) | Load calculation method and device, storage medium and electronic equipment | |
| US10635501B2 (en) | Adaptive scaling of workloads in a distributed computing environment | |
| CN109739627A (en) | Dispatching method, electronic equipment and the medium of task | |
| Maroulis et al. | A holistic energy-efficient real-time scheduler for mixed stream and batch processing workloads | |
| CN116521350A (en) | ETL scheduling method and device based on deep learning algorithm | |
| CN110083602A (en) | A kind of method and device of data storage and data processing based on hive table | |
| CN110489392A (en) | Data access method, device, system, storage medium and equipment between multi-tenant | |
| CN111782409B (en) | Task processing method, device and electronic equipment, and risk identification task processing method and device | |
| Koo et al. | Sahws: Iot-enabled workflow scheduler for next-generation hadoop cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |